2021-12-15T17:03:45.060 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-15T17:03:45.062 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} --name sage-2021-12-15_15:59:44-rados-wip-sage2-testing-2021-12-14-1714-distro-basic-smithi --archive /home/teuthworker/archive/sage-2021-12-15_15:59:44-rados-wip-sage2-testing-2021-12-14-1714-distro-basic-smithi/6564320 --owner scheduled_sage@teuthology --verbose -- /home/teuthworker/archive/sage-2021-12-15_15:59:44-rados-wip-sage2-testing-2021-12-14-1714-distro-basic-smithi/6564320/orig.config.yaml 2021-12-15T17:03:45.095 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-12-15T17:03:45.163 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2021-12-15_15:59:44-rados-wip-sage2-testing-2021-12-14-1714-distro-basic-smithi/6564320 branch: wip-sage2-testing-2021-12-14-1714 description: rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} email: null first_in_suite: false job_id: '6564320' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2021-12-15_15:59:44-rados-wip-sage2-testing-2021-12-14-1714-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-sage2-testing-2021-12-14-1714 ceph: conf: global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 cephadm: cephadm_mode: cephadm-package install: ceph: sha1: aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 extra_packages: - cephadm selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-sage2-testing-2021-12-14-1714 sha1: aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 owner: scheduled_sage@teuthology priority: 76 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 sleep_before_teardown: 0 suite: rados suite_branch: wip-sage2-testing-2021-12-14-1714 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 targets: smithi022.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeM83kE49CcQLK33nd43ov+h9aJqojpBFtU44fuvwLmCDeKAuN72JGOMo3jvEtFs4CRoina0PK4JuvuoMwq53zNpMbb5mvTsFoaqkBjqfFdSfjaflbbf3H10ixKTKkRq0o8IqW2WNoirPZprgZX0QTZq35NdSa/rXp8e5m6UFrbEkJfkKF2RBW3o4OC+4SwHO8jfFNsmkqxXqjWxUk1pouFEehnD56odphYXQNcfmE692sIHLL1uSc4ZKzuNNtbeD5fplm0Rj13ox2IwFlQ39BZ4w3DN8FFcMjwU6BzKQIjOJMhhXtP/EAD4NYL35Ne+AjxjRFHtSAG2WfxxtX/8fa98W8Qq8UiXTfyVyGjyhiwealCswDJ9GUtLFN2MsRt1tCM60uqcTet3hIu/E/rOPvPbaUeaXo9VuHm8Nf/e8IjmG8Ou7by/HGWwxeD/enkv/B4FArhnWGEylIYzXTX/dJerteDzSaggsGN02i6SgyJ7ScVUr+DAj6mH7mbEulf2c= smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJo9uz7no9yw9i08yXys1ZBCq7Fwy2EnlSyib1huEdHVf42Iv2A7m4OSJtvv4BwdcLbgBG4Qxv+mdmGTOAzq+t1Xcf6lqzYsRavKy2+9O7+2uAaLttWLIfUUQdlpg4KEhWPek2fJK/EjwRf4BzYr2eLN9jOH6p4xqh17LZDLkiF4Wlhcc55smTyxRCq8jM4hVrc2IH+lr5/AmSheBggYIBWePxquKp9fxJDQvsphdLhr/CxUpqobXc+6qDxRkaJkiCi2ecd9dYApsaCMDYQCRdmfxUlfFzJwP27MKd15yBFPBhtHboITyR/5MBrBzG20i6pM2WFU4C13TDu8vCLaIw9mToG7guYWjLMZyKMpfnrQUT6JbskfZYgWVp7K32zs5s4UC3Y2O+4I0m/kGyh2ZRUvSbfricB28X34O/fgBCvidUmn/SjKqvMbEJEL57wqEZB5uk4VebgOz/3qdmf2ecVaPQXVBYLAtHI44f+UF3oMuwdCYUMTTdlgCXKZwhFKc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - "set -ex\nfor f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x\n\ do\n echo \"rotating key for $f\"\n K=$(ceph auth get-key $f)\n NK=\"\ $K\"\n ceph orch daemon rotate-key $f\n while [ \"$K\" == \"$NK\" ]; do\n\ \ sleep 5\n NK=$(ceph auth get-key $f)\n done\ndone\n" teuthology_branch: master teuthology_sha1: 95a7d4799b562f3bbb5ec66107094963abd62fa1 timestamp: 2021-12-15_15:59:44 tube: smithi user: sage verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 2021-12-15T17:03:45.164 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/qa; will attempt to use it 2021-12-15T17:03:45.164 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/qa/tasks 2021-12-15T17:03:45.165 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-12-15T17:03:45.167 INFO:teuthology.task.internal:Checking packages... 2021-12-15T17:03:45.187 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2' 2021-12-15T17:03:45.188 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-12-15T17:03:45.188 INFO:teuthology.packaging:ref: None 2021-12-15T17:03:45.189 INFO:teuthology.packaging:tag: None 2021-12-15T17:03:45.189 INFO:teuthology.packaging:branch: wip-sage2-testing-2021-12-14-1714 2021-12-15T17:03:45.189 INFO:teuthology.packaging:sha1: aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 2021-12-15T17:03:45.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage2-testing-2021-12-14-1714 2021-12-15T17:03:45.478 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-9665.gaa33f58d 2021-12-15T17:03:45.478 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-12-15T17:03:45.490 INFO:teuthology.task.internal:no buildpackages task found 2021-12-15T17:03:45.491 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-12-15T17:03:45.542 INFO:teuthology.task.internal:Saving configuration 2021-12-15T17:03:45.554 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-12-15T17:03:45.568 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-15T17:03:45.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sage-2021-12-15_15:59:44-rados-wip-sage2-testing-2021-12-14-1714-distro-basic-smithi/6564320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-12-15 16:58:48.595429', 'locked_by': 'scheduled_sage@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBATLzDzXlgYGhtRULVlsljD1oB3W//vq2jypnMfh8Qbu06J2+8mS5v0qC8Ze/m1upP73rOmN1LcWZa8HJKcAqzc='} 2021-12-15T17:03:45.617 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sage-2021-12-15_15:59:44-rados-wip-sage2-testing-2021-12-14-1714-distro-basic-smithi/6564320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-12-15 16:58:48.594088', 'locked_by': 'scheduled_sage@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWQY78iE8d6LEw9DYJuPVwqJT5f8CZHxcQh6xW4sN89x4D/KnR4WoJAMw33iBS97/ngXXwMCkPT/2d7lzKcKEc='} 2021-12-15T17:03:45.617 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-12-15T17:03:45.629 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-12-15T17:03:45.629 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-12-15T17:03:45.630 INFO:teuthology.run_tasks:Running task console_log... 2021-12-15T17:03:45.728 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8e7cadcd08>, signals=[15]) 2021-12-15T17:03:45.729 INFO:teuthology.run_tasks:Running task internal.connect... 2021-12-15T17:03:45.740 INFO:teuthology.task.internal:Opening connections... 2021-12-15T17:03:45.741 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2021-12-15T17:03:45.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-12-15T17:03:45.817 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2021-12-15T17:03:45.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-12-15T17:03:45.886 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-12-15T17:03:45.896 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2021-12-15T17:03:45.917 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2021-12-15T17:03:45.918 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2021-12-15T17:03:45.977 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2021-12-15T17:03:45.977 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="8" 2021-12-15T17:03:45.978 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2021-12-15T17:03:45.978 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2021-12-15T17:03:45.978 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="8" 2021-12-15T17:03:45.979 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el8" 2021-12-15T17:03:45.979 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-15T17:03:45.979 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2021-12-15T17:03:45.979 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-15T17:03:45.980 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2021-12-15T17:03:45.980 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-15T17:03:45.980 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-15T17:03:45.980 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-15T17:03:45.981 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2021-12-15T17:03:46.002 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2021-12-15T17:03:46.022 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2021-12-15T17:03:46.022 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2021-12-15T17:03:46.080 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2021-12-15T17:03:46.080 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8" 2021-12-15T17:03:46.081 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2021-12-15T17:03:46.081 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2021-12-15T17:03:46.081 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8" 2021-12-15T17:03:46.082 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2021-12-15T17:03:46.082 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-15T17:03:46.082 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2021-12-15T17:03:46.082 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-15T17:03:46.083 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2021-12-15T17:03:46.083 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-15T17:03:46.083 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-15T17:03:46.083 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-15T17:03:46.084 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2021-12-15T17:03:46.102 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-12-15T17:03:46.126 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-12-15T17:03:46.148 INFO:teuthology.task.internal:Checking for old test directory... 2021-12-15T17:03:46.149 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2021-12-15T17:03:46.152 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2021-12-15T17:03:46.169 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-12-15T17:03:46.178 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-12-15T17:03:46.178 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2021-12-15T17:03:46.210 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2021-12-15T17:03:46.238 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-12-15T17:03:46.298 INFO:teuthology.run_tasks:Running task kernel... 2021-12-15T17:03:46.315 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-12-15T17:03:46.316 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-12-15T17:03:46.316 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2021-12-15T17:03:46.317 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2021-12-15T17:03:46.336 INFO:teuthology.orchestra.run.smithi022.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:03:46.337 INFO:teuthology.task.kernel:Running kernel on smithi022: 4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:03:46.337 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2021-12-15T17:03:46.339 INFO:teuthology.orchestra.run.smithi122.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:03:46.339 INFO:teuthology.task.kernel:Running kernel on smithi122: 4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:03:46.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2021-12-15T17:03:47.392 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:03:47.393 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:03:47.397 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:03:47.398 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:03:49.118 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - AppStream 32 MB/s | 18 MB 00:00 2021-12-15T17:03:49.216 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - AppStream 29 MB/s | 18 MB 00:00 2021-12-15T17:03:52.917 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - Base 27 MB/s | 16 MB 00:00 2021-12-15T17:03:53.002 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - Base 28 MB/s | 16 MB 00:00 2021-12-15T17:03:55.258 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - Extras 126 kB/s | 16 kB 00:00 2021-12-15T17:03:55.493 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - Extras 153 kB/s | 16 kB 00:00 2021-12-15T17:03:55.586 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - PowerTools 17 MB/s | 3.8 MB 00:00 2021-12-15T17:03:55.854 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - PowerTools 16 MB/s | 3.8 MB 00:00 2021-12-15T17:03:58.870 INFO:teuthology.orchestra.run.smithi022.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-12-15T17:03:58.902 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2021-12-15T17:04:00.253 INFO:teuthology.orchestra.run.smithi022.stdout:Apache Arrow 145 kB/s | 184 kB 00:01 2021-12-15T17:04:00.261 INFO:teuthology.orchestra.run.smithi122.stdout:Apache Arrow 146 kB/s | 184 kB 00:01 2021-12-15T17:04:01.311 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-12-15T17:04:01.345 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-12-15T17:04:03.962 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 415 kB/s | 24 kB 00:00 2021-12-15T17:04:04.178 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 415 kB/s | 24 kB 00:00 2021-12-15T17:04:06.281 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-4.18.0-348.2.1.el8_5.x86_64 is already installed. 2021-12-15T17:04:06.281 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-12-15T17:04:06.383 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:04:06.384 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:04:06.384 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:04:06.507 DEBUG:teuthology.orchestra.run.smithi122:> echo no | sudo yum reinstall kernel || true 2021-12-15T17:04:06.584 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-4.18.0-348.2.1.el8_5.x86_64 is already installed. 2021-12-15T17:04:06.584 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-12-15T17:04:06.682 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:06.683 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:06.699 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:04:06.701 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:04:06.701 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:04:06.834 DEBUG:teuthology.orchestra.run.smithi022:> echo no | sudo yum reinstall kernel || true 2021-12-15T17:04:07.020 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:07.021 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:07.040 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:04 ago on Wed 15 Dec 2021 05:04:03 PM UTC. 2021-12-15T17:04:07.398 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:03 ago on Wed 15 Dec 2021 05:04:04 PM UTC. 2021-12-15T17:04:07.765 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:04:07.766 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:04:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2021-12-15T17:04:07.767 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:04:07.767 INFO:teuthology.orchestra.run.smithi122.stdout:Reinstalling: 2021-12-15T17:04:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-12-15T17:04:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:04:07.767 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:04:07.768 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:04:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:04:07.768 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.0 M 2021-12-15T17:04:07.768 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 0 2021-12-15T17:04:07.769 INFO:teuthology.orchestra.run.smithi122.stderr:Operation aborted. 2021-12-15T17:04:07.846 INFO:teuthology.orchestra.run.smithi122.stdout:Is this ok [y/N]: 2021-12-15T17:04:07.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum reinstall -y kernel || true 2021-12-15T17:04:08.021 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:08.022 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:08.208 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:04:08.209 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:04:08.209 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2021-12-15T17:04:08.209 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:04:08.210 INFO:teuthology.orchestra.run.smithi022.stdout:Reinstalling: 2021-12-15T17:04:08.210 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-12-15T17:04:08.210 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:04:08.210 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:04:08.211 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:04:08.211 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:04:08.211 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 7.0 M 2021-12-15T17:04:08.211 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 0 2021-12-15T17:04:08.212 INFO:teuthology.orchestra.run.smithi022.stderr:Operation aborted. 2021-12-15T17:04:08.278 INFO:teuthology.orchestra.run.smithi022.stdout:Is this ok [y/N]: 2021-12-15T17:04:08.279 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum reinstall -y kernel || true 2021-12-15T17:04:08.376 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:05 ago on Wed 15 Dec 2021 05:04:03 PM UTC. 2021-12-15T17:04:08.473 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:08.473 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:04:08.840 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:04 ago on Wed 15 Dec 2021 05:04:04 PM UTC. 2021-12-15T17:04:09.093 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:04:09.094 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:04:09.095 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2021-12-15T17:04:09.096 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:04:09.096 INFO:teuthology.orchestra.run.smithi122.stdout:Reinstalling: 2021-12-15T17:04:09.097 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-12-15T17:04:09.097 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:04:09.097 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:04:09.098 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:04:09.098 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:04:09.100 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.0 M 2021-12-15T17:04:09.101 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 0 2021-12-15T17:04:09.102 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:04:09.373 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 25 MB/s | 7.0 MB 00:00 2021-12-15T17:04:09.373 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:04:09.374 INFO:teuthology.orchestra.run.smithi122.stdout:Total 25 MB/s | 7.0 MB 00:00 2021-12-15T17:04:09.466 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:04:09.607 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:04:09.608 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:04:09.608 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2021-12-15T17:04:09.608 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:04:09.609 INFO:teuthology.orchestra.run.smithi022.stdout:Reinstalling: 2021-12-15T17:04:09.609 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-12-15T17:04:09.609 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:04:09.609 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:04:09.610 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:04:09.610 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:04:09.610 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 7.0 M 2021-12-15T17:04:09.610 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 0 2021-12-15T17:04:09.610 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:04:09.852 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:04:09.852 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:04:09.983 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:04:10.004 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:04:10.132 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 13 MB/s | 7.0 MB 00:00 2021-12-15T17:04:10.134 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:04:10.134 INFO:teuthology.orchestra.run.smithi022.stdout:Total 13 MB/s | 7.0 MB 00:00 2021-12-15T17:04:10.225 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:04:10.258 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:04:10.453 INFO:teuthology.orchestra.run.smithi122.stdout: Reinstalling : kernel-4.18.0-348.2.1.el8_5.x86_64 1/2 2021-12-15T17:04:10.453 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : kernel-4.18.0-348.2.1.el8_5.x86_64 2/2 2021-12-15T17:04:10.605 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:04:10.606 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:04:10.738 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:04:10.748 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:04:10.789 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-4.18.0-348.2.1.el8_5.x86_64 2/2 2021-12-15T17:04:10.789 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/2 2021-12-15T17:04:11.006 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:04:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: Reinstalling : kernel-4.18.0-348.2.1.el8_5.x86_64 1/2 2021-12-15T17:04:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : kernel-4.18.0-348.2.1.el8_5.x86_64 2/2 2021-12-15T17:04:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 2/2 2021-12-15T17:04:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:04:11.194 INFO:teuthology.orchestra.run.smithi122.stdout:Reinstalled: 2021-12-15T17:04:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:04:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:04:11.195 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:04:11.279 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel --last | head -n 1 2021-12-15T17:04:11.382 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Wed 15 Dec 2021 05:04:10 PM UTC 2021-12-15T17:04:11.383 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:04:11.384 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-12-15T17:04:11.384 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-12-15T17:04:11.384 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-12-15T17:04:11.421 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2021-12-15T17:04:11.436 DEBUG:teuthology.parallel:result is None 2021-12-15T17:04:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-4.18.0-348.2.1.el8_5.x86_64 2/2 2021-12-15T17:04:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/2 2021-12-15T17:04:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 2/2 2021-12-15T17:04:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:04:11.997 INFO:teuthology.orchestra.run.smithi022.stdout:Reinstalled: 2021-12-15T17:04:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:04:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:04:11.998 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:04:12.071 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel --last | head -n 1 2021-12-15T17:04:12.180 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Wed 15 Dec 2021 05:04:11 PM UTC 2021-12-15T17:04:12.181 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-12-15T17:04:12.181 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-12-15T17:04:12.182 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-12-15T17:04:12.182 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-12-15T17:04:12.221 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2021-12-15T17:04:12.245 DEBUG:teuthology.parallel:result is None 2021-12-15T17:04:12.246 INFO:teuthology.run_tasks:Running task internal.base... 2021-12-15T17:04:12.257 INFO:teuthology.task.internal:Creating test directory... 2021-12-15T17:04:12.258 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-12-15T17:04:12.289 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-12-15T17:04:12.309 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-12-15T17:04:12.319 INFO:teuthology.run_tasks:Running task internal.archive... 2021-12-15T17:04:12.344 INFO:teuthology.task.internal:Creating archive directory... 2021-12-15T17:04:12.344 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-12-15T17:04:12.347 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-12-15T17:04:12.382 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-12-15T17:04:12.390 INFO:teuthology.task.internal:Enabling coredump saving... 2021-12-15T17:04:12.392 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-12-15T17:04:12.411 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-12-15T17:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-12-15T17:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-12-15T17:04:12.447 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-12-15T17:04:12.459 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-12-15T17:04:12.461 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-12-15T17:04:12.470 INFO:teuthology.task.internal:Configuring sudo... 2021-12-15T17:04:12.470 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-12-15T17:04:12.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-12-15T17:04:12.532 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-12-15T17:04:12.543 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-12-15T17:04:12.543 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-12-15T17:04:12.557 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-12-15T17:04:12.592 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-12-15T17:04:12.658 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-12-15T17:04:12.708 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-12-15T17:04:12.790 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-12-15T17:04:12.833 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:04:12.834 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-12-15T17:04:12.900 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-12-15T17:04:12.947 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-12-15T17:04:13.006 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-12-15T17:04:13.091 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-12-15T17:04:13.135 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:04:13.135 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-12-15T17:04:13.203 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2021-12-15T17:04:13.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2021-12-15T17:04:13.274 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-12-15T17:04:13.311 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-12-15T17:04:13.696 INFO:teuthology.run_tasks:Running task internal.timer... 2021-12-15T17:04:13.706 INFO:teuthology.task.internal:Starting timer... 2021-12-15T17:04:13.706 INFO:teuthology.run_tasks:Running task pcp... 2021-12-15T17:04:13.748 INFO:teuthology.run_tasks:Running task selinux... 2021-12-15T17:04:13.771 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-12-15T17:04:13.771 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2021-12-15T17:04:13.827 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: [ OK ] 2021-12-15T17:04:13.828 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2021-12-15T17:04:13.889 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: [ OK ] 2021-12-15T17:04:13.891 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-12-15T17:04:13.892 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2021-12-15T17:04:13.913 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2021-12-15T17:04:13.914 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2021-12-15T17:04:13.941 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2021-12-15T17:04:13.942 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2021-12-15T17:04:13.942 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-12-15T17:04:13.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-12-15T17:04:13.983 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-12-15T17:04:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-12-15T17:04:14.016 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-12-15T17:04:14.016 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2021-12-15T17:04:14.057 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2021-12-15T17:04:14.111 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-12-15T17:04:14.122 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-12-15T17:04:14.123 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-12-15T17:04:14.142 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-12-15T17:04:14.144 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-12-15T17:04:14.529 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-12-15T17:04:15.373 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-12-15T17:04:18.307 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-12-15T17:04:18.433 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:18 +0000 (0:00:00.142) 0:00:00.142 **** 2021-12-15T17:04:18.453 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:18 +0000 (0:00:00.012) 0:00:00.155 **** 2021-12-15T17:04:19.843 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi022.front.sepia.ceph.com] 2021-12-15T17:04:19.893 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:19 +0000 (0:00:01.447) 0:00:01.602 **** 2021-12-15T17:04:19.904 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] 2021-12-15T17:04:19.952 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:19 +0000 (0:00:00.058) 0:00:01.661 **** 2021-12-15T17:04:20.177 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-12-15T17:04:20.179 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-12-15T17:04:20.199 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:20 +0000 (0:00:00.246) 0:00:01.908 **** 2021-12-15T17:04:20.205 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-12-15T17:04:20.230 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:20 +0000 (0:00:00.031) 0:00:01.939 **** 2021-12-15T17:04:20.730 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-12-15T17:04:20.731 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-12-15T17:04:20.752 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:20 +0000 (0:00:00.522) 0:00:02.461 **** 2021-12-15T17:04:20.759 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-12-15T17:04:20.783 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:20 +0000 (0:00:00.030) 0:00:02.492 **** 2021-12-15T17:04:20.820 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-12-15T17:04:20.822 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:20.843 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:20 +0000 (0:00:00.060) 0:00:02.552 **** 2021-12-15T17:04:20.860 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:20.883 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:20 +0000 (0:00:00.039) 0:00:02.592 **** 2021-12-15T17:04:21.107 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-12-15T17:04:21.109 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:04:21.129 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:21 +0000 (0:00:00.246) 0:00:02.838 **** 2021-12-15T17:04:21.136 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:04:21.162 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:21 +0000 (0:00:00.032) 0:00:02.871 **** 2021-12-15T17:04:22.017 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-12-15T17:04:22.019 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1639587861.1745927-7526-27428075416943/source", "state": "file", "uid": 0} 2021-12-15T17:04:22.041 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:22 +0000 (0:00:00.879) 0:00:03.751 **** 2021-12-15T17:04:22.050 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1639587861.2074447-7532-265200980588018/source", "state": "file", "uid": 0} 2021-12-15T17:04:22.072 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:22 +0000 (0:00:00.030) 0:00:03.782 **** 2021-12-15T17:04:22.661 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-12-15T17:04:22.662 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-12-15T17:04:22.664 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-12-15T17:04:22.687 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-12-15T17:04:22.689 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-12-15T17:04:22.761 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-12-15T17:04:22.762 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-12-15T17:04:22.775 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-12-15T17:04:22.776 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:22 +0000 (0:00:00.702) 0:00:04.484 **** 2021-12-15T17:04:23.303 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] 2021-12-15T17:04:23.312 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] 2021-12-15T17:04:23.330 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Wednesday 15 December 2021 17:04:23 +0000 (0:00:00.555) 0:00:05.039 **** 2021-12-15T17:04:23.355 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-12-15T17:04:23.366 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-12-15T17:04:23.371 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-12-15T17:04:23.543 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:23 +0000 (0:00:00.212) 0:00:05.252 **** 2021-12-15T17:04:23.563 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:23 +0000 (0:00:00.019) 0:00:05.272 **** 2021-12-15T17:04:24.204 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-12-15T17:04:24.205 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] 2021-12-15T17:04:24.225 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] 2021-12-15T17:04:24.287 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Wednesday 15 December 2021 17:04:24 +0000 (0:00:00.723) 0:00:05.995 **** 2021-12-15T17:04:24.311 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:24 +0000 (0:00:00.022) 0:00:06.018 **** 2021-12-15T17:04:25.020 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-12-15T17:04:25.024 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.200282", "end": "2021-12-15 17:04:25.000355", "rc": 0, "start": "2021-12-15 17:04:24.800073", "stderr": "", "stderr_lines": [], "stdout": "271ee3bd41c26a34f5caf35e3fcae76aca032a42", "stdout_lines": ["271ee3bd41c26a34f5caf35e3fcae76aca032a42"]} changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.191961", "end": "2021-12-15 17:04:25.003718", "rc": 0, "start": "2021-12-15 17:04:24.811757", "stderr": "", "stderr_lines": [], "stdout": "271ee3bd41c26a34f5caf35e3fcae76aca032a42", "stdout_lines": ["271ee3bd41c26a34f5caf35e3fcae76aca032a42"]} 2021-12-15T17:04:25.054 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.741) 0:00:06.759 **** 2021-12-15T17:04:25.084 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.033) 0:00:06.793 **** 2021-12-15T17:04:25.268 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-12-15T17:04:25.270 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.007305", "end": "2021-12-15 17:04:25.251853", "failed_when_result": false, "rc": 0, "start": "2021-12-15 17:04:25.244548", "stderr": "", "stderr_lines": [], "stdout": "c165763fba468068dd0382f5b2240f4498360f4a", "stdout_lines": ["c165763fba468068dd0382f5b2240f4498360f4a"]} 2021-12-15T17:04:25.292 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.208) 0:00:07.001 **** 2021-12-15T17:04:25.300 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013901", "end": "2021-12-15 17:04:25.281346", "failed_when_result": false, "rc": 0, "start": "2021-12-15 17:04:25.267445", "stderr": "", "stderr_lines": [], "stdout": "c165763fba468068dd0382f5b2240f4498360f4a", "stdout_lines": ["c165763fba468068dd0382f5b2240f4498360f4a"]} 2021-12-15T17:04:25.325 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.032) 0:00:07.034 **** 2021-12-15T17:04:25.352 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.376 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.051) 0:00:07.085 **** 2021-12-15T17:04:25.404 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.427 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.051) 0:00:07.136 **** 2021-12-15T17:04:25.465 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-12-15T17:04:25.467 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.489 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.061) 0:00:07.198 **** 2021-12-15T17:04:25.507 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.530 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.041) 0:00:07.239 **** 2021-12-15T17:04:25.558 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-12-15T17:04:25.559 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.583 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.052) 0:00:07.292 **** 2021-12-15T17:04:25.602 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.626 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.043) 0:00:07.336 **** 2021-12-15T17:04:25.656 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-12-15T17:04:25.657 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.692 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.065) 0:00:07.401 **** 2021-12-15T17:04:25.723 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.751 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.059) 0:00:07.460 **** 2021-12-15T17:04:25.768 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-12-15T17:04:25.770 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.793 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.041) 0:00:07.502 **** 2021-12-15T17:04:25.821 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:04:25.844 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:25 +0000 (0:00:00.051) 0:00:07.553 **** 2021-12-15T17:04:26.050 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-12-15T17:04:26.052 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:04:26.082 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:26 +0000 (0:00:00.238) 0:00:07.792 **** 2021-12-15T17:04:26.103 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:04:26.133 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:26 +0000 (0:00:00.050) 0:00:07.842 **** 2021-12-15T17:04:26.384 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-12-15T17:04:26.386 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-12-15T17:04:26.417 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-12-15T17:04:26.595 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-12-15T17:04:26.621 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-12-15T17:04:26.810 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-12-15T17:04:26.822 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-12-15T17:04:27.011 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-12-15T17:04:27.023 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-12-15T17:04:27.213 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-12-15T17:04:27.224 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-12-15T17:04:27.402 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-12-15T17:04:27.434 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-12-15T17:04:27.601 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-12-15T17:04:27.634 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-12-15T17:04:27.802 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-12-15T17:04:27.845 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-12-15T17:04:28.003 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-12-15T17:04:28.046 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-12-15T17:04:28.203 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-12-15T17:04:28.257 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-12-15T17:04:28.403 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-12-15T17:04:28.466 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-12-15T17:04:28.613 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-12-15T17:04:28.667 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-12-15T17:04:28.813 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-12-15T17:04:28.877 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-12-15T17:04:29.015 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-12-15T17:04:29.088 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-12-15T17:04:29.205 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-12-15T17:04:29.300 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-12-15T17:04:29.406 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-12-15T17:04:29.511 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-12-15T17:04:29.606 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-12-15T17:04:29.722 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-12-15T17:04:29.807 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-12-15T17:04:29.933 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-12-15T17:04:30.007 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-12-15T17:04:30.143 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-12-15T17:04:30.207 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-12-15T17:04:30.355 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-12-15T17:04:30.407 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-12-15T17:04:30.553 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-12-15T17:04:30.607 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-12-15T17:04:30.764 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-12-15T17:04:30.797 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-12-15T17:04:30.968 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-12-15T17:04:31.001 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-12-15T17:04:31.180 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-12-15T17:04:31.204 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-12-15T17:04:31.392 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-12-15T17:04:31.403 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-12-15T17:04:31.592 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-12-15T17:04:31.603 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-12-15T17:04:31.802 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-12-15T17:04:31.813 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-12-15T17:04:32.002 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-12-15T17:04:32.014 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-12-15T17:04:32.016 INFO:teuthology.task.ansible.out: 2021-12-15T17:04:32.203 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-12-15T17:04:32.216 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-12-15T17:04:32.416 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-12-15T17:04:32.418 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-12-15T17:04:32.608 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-12-15T17:04:32.619 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-12-15T17:04:32.818 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-12-15T17:04:32.831 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-12-15T17:04:33.011 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-12-15T17:04:33.044 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-12-15T17:04:33.223 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-12-15T17:04:33.245 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-12-15T17:04:33.414 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-12-15T17:04:33.456 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-12-15T17:04:33.615 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-12-15T17:04:33.668 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-12-15T17:04:33.816 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-12-15T17:04:33.880 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-12-15T17:04:34.008 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-12-15T17:04:34.083 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-12-15T17:04:34.219 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-12-15T17:04:34.293 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-12-15T17:04:34.418 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-12-15T17:04:34.502 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-12-15T17:04:34.617 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-12-15T17:04:34.704 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-12-15T17:04:34.810 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-12-15T17:04:34.914 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-12-15T17:04:35.008 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-12-15T17:04:35.123 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-12-15T17:04:35.219 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-12-15T17:04:35.335 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-12-15T17:04:35.419 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-12-15T17:04:35.534 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-12-15T17:04:35.608 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-12-15T17:04:35.746 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-12-15T17:04:35.810 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-12-15T17:04:35.956 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-12-15T17:04:36.009 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-12-15T17:04:36.158 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-12-15T17:04:36.212 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-12-15T17:04:36.369 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-12-15T17:04:36.412 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-12-15T17:04:36.591 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-12-15T17:04:36.613 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-12-15T17:04:36.791 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-12-15T17:04:36.813 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-12-15T17:04:37.002 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1152} ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-12-15T17:04:37.201 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-12-15T17:04:37.213 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-12-15T17:04:37.402 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-12-15T17:04:37.413 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-12-15T17:04:37.604 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-12-15T17:04:37.626 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-12-15T17:04:37.793 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-12-15T17:04:37.836 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-12-15T17:04:37.993 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-12-15T17:04:38.046 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-12-15T17:04:38.193 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-12-15T17:04:38.247 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-12-15T17:04:38.393 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-12-15T17:04:38.457 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-12-15T17:04:38.593 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-12-15T17:04:38.657 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-12-15T17:04:38.792 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-12-15T17:04:38.868 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-12-15T17:04:38.984 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-12-15T17:04:39.078 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-12-15T17:04:39.182 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-12-15T17:04:39.288 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-12-15T17:04:39.385 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-12-15T17:04:39.490 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-12-15T17:04:39.575 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-12-15T17:04:39.701 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-12-15T17:04:39.780 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-12-15T17:04:39.917 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-12-15T17:04:39.970 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-12-15T17:04:40.127 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-12-15T17:04:40.171 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-12-15T17:04:40.338 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-12-15T17:04:40.370 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-12-15T17:04:40.548 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-12-15T17:04:40.581 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-12-15T17:04:40.752 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-12-15T17:04:40.774 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-12-15T17:04:40.952 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-12-15T17:04:40.973 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-12-15T17:04:41.162 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-12-15T17:04:41.175 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-12-15T17:04:41.372 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-12-15T17:04:41.373 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-12-15T17:04:41.574 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-12-15T17:04:41.585 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-12-15T17:04:41.773 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-12-15T17:04:41.784 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-12-15T17:04:41.963 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-12-15T17:04:41.985 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-12-15T17:04:42.164 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-12-15T17:04:42.198 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-12-15T17:04:42.357 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-12-15T17:04:42.410 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-12-15T17:04:42.567 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-12-15T17:04:42.609 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-12-15T17:04:42.756 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-12-15T17:04:42.820 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-12-15T17:04:42.966 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-12-15T17:04:43.020 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-12-15T17:04:43.157 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-12-15T17:04:43.232 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-12-15T17:04:43.358 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-12-15T17:04:43.443 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-12-15T17:04:43.559 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-12-15T17:04:43.643 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-12-15T17:04:43.759 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-12-15T17:04:43.854 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-12-15T17:04:43.949 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-12-15T17:04:44.054 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-12-15T17:04:44.151 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-12-15T17:04:44.267 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-12-15T17:04:44.351 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-12-15T17:04:44.476 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-12-15T17:04:44.550 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-12-15T17:04:44.677 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-12-15T17:04:44.752 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-12-15T17:04:44.888 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-12-15T17:04:44.951 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-12-15T17:04:45.097 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-12-15T17:04:45.140 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-12-15T17:04:45.297 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-12-15T17:04:45.339 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-12-15T17:04:45.508 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-12-15T17:04:45.542 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-12-15T17:04:45.722 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-12-15T17:04:45.754 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-12-15T17:04:45.921 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-12-15T17:04:45.943 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-12-15T17:04:46.131 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-12-15T17:04:46.142 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-12-15T17:04:46.342 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-12-15T17:04:46.343 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-12-15T17:04:46.542 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-12-15T17:04:46.544 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-12-15T17:04:46.732 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-12-15T17:04:46.754 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-12-15T17:04:46.936 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-12-15T17:04:46.977 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-12-15T17:04:47.135 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-12-15T17:04:47.178 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-12-15T17:04:47.326 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-12-15T17:04:47.390 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-12-15T17:04:47.526 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-12-15T17:04:47.600 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-12-15T17:04:47.728 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-12-15T17:04:47.813 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-12-15T17:04:47.919 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-12-15T17:04:48.027 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-12-15T17:04:48.123 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-12-15T17:04:48.239 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-12-15T17:04:48.324 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-12-15T17:04:48.451 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-12-15T17:04:48.515 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-12-15T17:04:48.651 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-12-15T17:04:48.717 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-12-15T17:04:48.863 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-12-15T17:04:48.916 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-12-15T17:04:49.064 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-12-15T17:04:49.107 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-12-15T17:04:49.274 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-12-15T17:04:49.306 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-12-15T17:04:49.486 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-12-15T17:04:49.508 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-12-15T17:04:49.686 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-12-15T17:04:49.698 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-12-15T17:04:49.887 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-12-15T17:04:49.953 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-12-15T17:04:50.098 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-12-15T17:04:50.099 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-12-15T17:04:50.298 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-12-15T17:04:50.299 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-12-15T17:04:50.497 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-12-15T17:04:50.508 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-12-15T17:04:50.705 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-12-15T17:04:50.717 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-12-15T17:04:50.906 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-12-15T17:04:50.928 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-12-15T17:04:51.106 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-12-15T17:04:51.138 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-12-15T17:04:51.304 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-12-15T17:04:51.346 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-12-15T17:04:51.503 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-12-15T17:04:51.546 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-12-15T17:04:51.703 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-12-15T17:04:51.766 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-12-15T17:04:51.891 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-12-15T17:04:51.965 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-12-15T17:04:52.090 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-12-15T17:04:52.175 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-12-15T17:04:52.290 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-12-15T17:04:52.384 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-12-15T17:04:52.501 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-12-15T17:04:52.596 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-12-15T17:04:52.703 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-12-15T17:04:52.799 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-12-15T17:04:52.894 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-12-15T17:04:53.010 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-12-15T17:04:53.095 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-12-15T17:04:53.220 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-12-15T17:04:53.294 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-12-15T17:04:53.431 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-12-15T17:04:53.494 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-12-15T17:04:53.631 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-12-15T17:04:53.686 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-12-15T17:04:53.835 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-12-15T17:04:53.889 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-12-15T17:04:54.045 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-12-15T17:04:54.088 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1238} 2021-12-15T17:04:54.255 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-12-15T17:04:54.288 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1239} 2021-12-15T17:04:54.455 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-12-15T17:04:54.477 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1240} 2021-12-15T17:04:54.668 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-12-15T17:04:54.690 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1241} 2021-12-15T17:04:54.878 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-12-15T17:04:54.900 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1242} 2021-12-15T17:04:55.100 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-12-15T17:04:55.102 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1243} 2021-12-15T17:04:55.300 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1244} 2021-12-15T17:04:55.300 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1238} 2021-12-15T17:04:55.489 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1245} 2021-12-15T17:04:55.511 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1239} 2021-12-15T17:04:55.687 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1246} 2021-12-15T17:04:55.719 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1240} 2021-12-15T17:04:55.887 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1247} 2021-12-15T17:04:55.919 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1241} 2021-12-15T17:04:56.087 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "move_home": false, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "uid": 1248} 2021-12-15T17:04:56.130 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1242} 2021-12-15T17:04:56.288 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "move_home": false, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "uid": 1249} 2021-12-15T17:04:56.331 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1243} 2021-12-15T17:04:56.487 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "move_home": false, "name": "xuwang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2021-12-15T17:04:56.541 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1244} 2021-12-15T17:04:56.688 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "move_home": false, "name": "blaine", "shell": "/bin/bash", "state": "present", "uid": 1251} 2021-12-15T17:04:56.752 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1245} 2021-12-15T17:04:56.889 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "move_home": false, "name": "choffman", "shell": "/bin/bash", "state": "present", "uid": 1252} 2021-12-15T17:04:56.963 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1246} 2021-12-15T17:04:57.079 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/oro", "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "move_home": false, "name": "oro", "shell": "/bin/bash", "state": "present", "uid": 1253} 2021-12-15T17:04:57.174 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1247} 2021-12-15T17:04:57.384 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "move_home": false, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "uid": 1248} 2021-12-15T17:04:57.596 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "move_home": false, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "uid": 1249} 2021-12-15T17:04:57.805 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "move_home": false, "name": "xuwang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2021-12-15T17:04:58.005 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "move_home": false, "name": "blaine", "shell": "/bin/bash", "state": "present", "uid": 1251} 2021-12-15T17:04:58.213 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "move_home": false, "name": "choffman", "shell": "/bin/bash", "state": "present", "uid": 1252} 2021-12-15T17:04:58.421 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/oro", "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "move_home": false, "name": "oro", "shell": "/bin/bash", "state": "present", "uid": 1253} 2021-12-15T17:04:59.458 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/mrfreezeex", "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "name": "mrfreezeex", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-12-15T17:04:59.584 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:59 +0000 (0:00:33.451) 0:00:41.293 **** 2021-12-15T17:04:59.666 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-12-15T17:04:59.701 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:59 +0000 (0:00:00.109) 0:00:41.402 **** 2021-12-15T17:04:59.792 INFO:teuthology.task.ansible.out: 2021-12-15T17:04:59.794 INFO:teuthology.task.ansible.out:TASK [Merge managed_users and managed_admin users] ***************************** 2021-12-15T17:04:59.795 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68 2021-12-15T17:04:59.797 INFO:teuthology.task.ansible.out:c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "al 2021-12-15T17:04:59.799 INFO:teuthology.task.ansible.out:i@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\ny 2021-12-15T17:04:59.804 INFO:teuthology.task.ansible.out:uriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s 2021-12-15T17:04:59.805 INFO:teuthology.task.ansible.out:0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11 2021-12-15T17:04:59.807 INFO:teuthology.task.ansible.out:af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5 2021-12-15T17:04:59.809 INFO:teuthology.task.ansible.out:457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a9 2021-12-15T17:04:59.810 INFO:teuthology.task.ansible.out:5579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b1 2021-12-15T17:04:59.812 INFO:teuthology.task.ansible.out:0be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1 2021-12-15T17:04:59.813 INFO:teuthology.task.ansible.out:fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, { 2021-12-15T17:04:59.815 INFO:teuthology.task.ansible.out:"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f6 2021-12-15T17:04:59.816 INFO:teuthology.task.ansible.out:0b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41 2021-12-15T17:04:59.818 INFO:teuthology.task.ansible.out:cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf 2021-12-15T17:04:59.820 INFO:teuthology.task.ansible.out:2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5 2021-12-15T17:04:59.821 INFO:teuthology.task.ansible.out:299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe80 2021-12-15T17:04:59.823 INFO:teuthology.task.ansible.out:2ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}]}, "changed": false} 2021-12-15T17:04:59.853 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:04:59 +0000 (0:00:00.160) 0:00:41.563 **** 2021-12-15T17:05:00.866 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-12-15T17:05:00.868 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/mrfreezeex", "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "name": "mrfreezeex", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-12-15T17:05:00.879 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-12-15T17:05:00.881 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"after": "271ee3bd41c26a34f5caf35e3fcae76aca032a42", "attempts": 1, "before": "271ee3bd41c26a34f5caf35e3fcae76aca032a42", "changed": false, "remote_url_changed": false} 2021-12-15T17:05:00.910 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:00 +0000 (0:00:01.056) 0:00:42.619 **** 2021-12-15T17:05:00.948 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-12-15T17:05:01.146 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:01 +0000 (0:00:00.235) 0:00:42.855 **** 2021-12-15T17:05:01.215 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-12-15T17:05:01.236 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:01 +0000 (0:00:00.090) 0:00:42.945 **** 2021-12-15T17:05:01.330 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-12-15T17:05:01.333 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-12-15T17:05:01.334 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-12-15T17:05:01.336 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-12-15T17:05:01.338 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-12-15T17:05:01.339 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu" 2021-12-15T17:05:01.340 INFO:teuthology.task.ansible.out:, "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-12-15T17:05:01.341 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-12-15T17:05:01.343 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-12-15T17:05:01.344 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-12-15T17:05:01.346 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-12-15T17:05:01.348 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-12-15T17:05:01.349 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-12-15T17:05:01.352 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70c 2021-12-15T17:05:01.352 INFO:teuthology.task.ansible.out:f55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-12-15T17:05:01.354 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-12-15T17:05:01.356 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}]}, "changed": false} 2021-12-15T17:05:01.379 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:01 +0000 (0:00:00.143) 0:00:43.089 **** 2021-12-15T17:05:01.429 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:01.430 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-12-15T17:05:01.662 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-12-15T17:05:01.893 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-12-15T17:05:02.123 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-12-15T17:05:02.125 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-12-15T17:05:02.362 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-12-15T17:05:02.623 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-12-15T17:05:02.625 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-12-15T17:05:02.666 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi022.front.sepia.ceph.com] => {"after": "271ee3bd41c26a34f5caf35e3fcae76aca032a42", "attempts": 1, "before": "271ee3bd41c26a34f5caf35e3fcae76aca032a42", "changed": false, "remote_url_changed": false} 2021-12-15T17:05:02.688 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:02 +0000 (0:00:01.308) 0:00:44.397 **** 2021-12-15T17:05:02.851 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:02.853 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-12-15T17:05:03.085 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-12-15T17:05:03.086 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-12-15T17:05:03.316 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-12-15T17:05:03.328 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU 2021-12-15T17:05:03.330 INFO:teuthology.task.ansible.out:/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-12-15T17:05:03.558 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-12-15T17:05:03.560 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-12-15T17:05:03.580 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-12-15T17:05:03.800 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-12-15T17:05:03.802 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-12-15T17:05:03.812 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:03.823 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:03.824 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-12-15T17:05:03.826 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-12-15T17:05:04.055 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-12-15T17:05:04.067 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-12-15T17:05:04.298 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-12-15T17:05:04.300 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-12-15T17:05:04.319 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-12-15T17:05:04.322 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-12-15T17:05:04.531 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-12-15T17:05:04.572 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-12-15T17:05:04.762 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-12-15T17:05:04.815 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-12-15T17:05:04.995 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-12-15T17:05:05.060 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-12-15T17:05:05.072 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:05.229 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-12-15T17:05:05.231 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-12-15T17:05:05.313 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-12-15T17:05:05.315 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-12-15T17:05:05.472 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-12-15T17:05:05.474 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-12-15T17:05:05.557 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-12-15T17:05:05.559 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-12-15T17:05:05.568 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:05.579 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:05.717 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-12-15T17:05:05.822 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-12-15T17:05:05.949 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-12-15T17:05:06.065 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-12-15T17:05:06.067 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-12-15T17:05:06.192 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-12-15T17:05:06.193 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-12-15T17:05:06.318 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-12-15T17:05:06.424 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-12-15T17:05:06.552 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-12-15T17:05:06.680 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-12-15T17:05:06.807 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-12-15T17:05:06.924 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-12-15T17:05:07.051 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-12-15T17:05:07.053 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-12-15T17:05:07.168 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-12-15T17:05:07.305 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-12-15T17:05:07.307 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-12-15T17:05:07.411 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-12-15T17:05:07.551 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-12-15T17:05:07.657 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-12-15T17:05:07.658 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-12-15T17:05:07.795 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-12-15T17:05:07.890 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-12-15T17:05:08.060 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-12-15T17:05:08.062 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-12-15T17:05:08.134 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-12-15T17:05:08.303 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-12-15T17:05:08.367 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-12-15T17:05:08.546 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-12-15T17:05:08.599 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-12-15T17:05:08.797 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-12-15T17:05:08.833 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-12-15T17:05:09.053 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-12-15T17:05:09.064 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-12-15T17:05:09.295 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-12-15T17:05:09.306 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-12-15T17:05:09.548 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-12-15T17:05:09.550 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-12-15T17:05:09.559 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-12-15T17:05:09.561 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-12-15T17:05:09.788 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-12-15T17:05:09.799 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-12-15T17:05:09.801 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-12-15T17:05:10.028 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-12-15T17:05:10.039 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-12-15T17:05:10.268 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-12-15T17:05:10.280 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-12-15T17:05:10.510 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-12-15T17:05:10.532 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-12-15T17:05:10.750 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-12-15T17:05:10.752 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-12-15T17:05:10.772 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-12-15T17:05:10.981 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-12-15T17:05:10.983 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-12-15T17:05:10.985 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-12-15T17:05:11.024 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-12-15T17:05:11.223 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-12-15T17:05:11.265 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-12-15T17:05:11.453 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-12-15T17:05:11.517 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-12-15T17:05:11.518 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-12-15T17:05:11.694 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-12-15T17:05:11.768 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-12-15T17:05:11.769 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-12-15T17:05:11.923 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-12-15T17:05:12.007 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-12-15T17:05:12.153 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-12-15T17:05:12.257 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-12-15T17:05:12.392 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-12-15T17:05:12.394 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-12-15T17:05:12.507 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-12-15T17:05:12.624 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-12-15T17:05:12.749 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-12-15T17:05:12.750 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-12-15T17:05:12.853 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-12-15T17:05:12.999 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-12-15T17:05:13.000 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-12-15T17:05:13.002 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-12-15T17:05:13.093 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-12-15T17:05:13.095 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-12-15T17:05:13.249 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-12-15T17:05:13.333 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-12-15T17:05:13.489 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-12-15T17:05:13.564 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-12-15T17:05:13.739 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-12-15T17:05:13.813 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-12-15T17:05:13.814 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-12-15T17:05:13.981 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-12-15T17:05:14.055 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-12-15T17:05:14.223 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-12-15T17:05:14.287 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-12-15T17:05:14.476 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-12-15T17:05:14.478 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-12-15T17:05:14.518 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-12-15T17:05:14.728 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-12-15T17:05:14.760 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-12-15T17:05:14.969 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-12-15T17:05:14.993 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-12-15T17:05:15.222 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-12-15T17:05:15.224 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-12-15T17:05:15.233 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-12-15T17:05:15.462 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-12-15T17:05:15.474 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-12-15T17:05:15.693 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-12-15T17:05:15.725 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-12-15T17:05:15.923 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-12-15T17:05:15.966 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-12-15T17:05:15.968 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-12-15T17:05:16.166 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-12-15T17:05:16.229 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-12-15T17:05:16.398 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-12-15T17:05:16.472 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-12-15T17:05:16.630 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-12-15T17:05:16.715 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-12-15T17:05:16.862 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-12-15T17:05:16.957 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-12-15T17:05:17.095 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-12-15T17:05:17.213 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-12-15T17:05:17.341 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-12-15T17:05:17.457 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-12-15T17:05:17.586 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-12-15T17:05:17.703 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-12-15T17:05:17.819 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-12-15T17:05:17.955 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-12-15T17:05:18.060 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-12-15T17:05:18.206 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-12-15T17:05:18.290 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-12-15T17:05:18.452 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-12-15T17:05:18.526 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-12-15T17:05:18.704 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-12-15T17:05:18.767 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-12-15T17:05:18.769 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-12-15T17:05:18.947 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-12-15T17:05:19.000 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-12-15T17:05:19.202 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-12-15T17:05:19.235 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-12-15T17:05:19.445 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-12-15T17:05:19.466 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-12-15T17:05:19.695 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-12-15T17:05:19.727 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-12-15T17:05:19.729 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-12-15T17:05:19.938 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-12-15T17:05:19.960 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-12-15T17:05:20.179 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-12-15T17:05:20.190 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-12-15T17:05:20.421 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-12-15T17:05:20.433 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-12-15T17:05:20.663 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-12-15T17:05:20.686 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-12-15T17:05:20.897 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-12-15T17:05:20.919 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-12-15T17:05:21.129 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-12-15T17:05:21.172 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-12-15T17:05:21.173 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-12-15T17:05:21.362 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-12-15T17:05:21.415 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-12-15T17:05:21.594 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-12-15T17:05:21.596 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-12-15T17:05:21.657 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-12-15T17:05:21.846 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-12-15T17:05:21.900 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-12-15T17:05:22.077 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-12-15T17:05:22.151 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-12-15T17:05:22.153 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-12-15T17:05:22.320 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-12-15T17:05:22.395 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-12-15T17:05:22.549 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-12-15T17:05:22.645 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-12-15T17:05:22.793 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-12-15T17:05:22.891 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-12-15T17:05:23.026 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-12-15T17:05:23.132 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-12-15T17:05:23.268 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-12-15T17:05:23.270 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-12-15T17:05:23.382 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-12-15T17:05:23.496 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-12-15T17:05:23.623 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-12-15T17:05:23.729 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-12-15T17:05:23.877 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-12-15T17:05:23.961 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-12-15T17:05:24.130 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-12-15T17:05:24.132 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-12-15T17:05:24.219 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-12-15T17:05:24.380 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-12-15T17:05:24.456 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-12-15T17:05:24.626 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-12-15T17:05:24.689 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-12-15T17:05:24.869 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-12-15T17:05:24.923 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-12-15T17:05:25.113 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-12-15T17:05:25.166 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-12-15T17:05:25.366 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-12-15T17:05:25.398 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-12-15T17:05:25.609 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-12-15T17:05:25.631 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-12-15T17:05:25.851 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-12-15T17:05:25.852 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-12-15T17:05:25.873 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-12-15T17:05:26.097 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-12-15T17:05:26.099 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-12-15T17:05:26.340 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-12-15T17:05:26.342 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-12-15T17:05:26.571 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-12-15T17:05:26.593 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-12-15T17:05:26.811 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-12-15T17:05:26.833 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-12-15T17:05:27.044 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-12-15T17:05:27.077 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-12-15T17:05:27.280 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-12-15T17:05:27.339 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-12-15T17:05:27.534 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-12-15T17:05:27.588 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-12-15T17:05:27.798 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-12-15T17:05:27.800 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-12-15T17:05:27.842 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-12-15T17:05:28.021 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-12-15T17:05:28.084 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-12-15T17:05:28.262 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-12-15T17:05:28.336 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-12-15T17:05:28.502 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-12-15T17:05:28.578 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-12-15T17:05:28.737 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-12-15T17:05:28.833 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-12-15T17:05:28.970 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-12-15T17:05:29.075 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-12-15T17:05:29.202 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-12-15T17:05:29.318 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-12-15T17:05:29.436 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-12-15T17:05:29.559 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-12-15T17:05:29.664 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-12-15T17:05:29.811 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-12-15T17:05:29.907 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-12-15T17:05:30.054 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-12-15T17:05:30.138 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-12-15T17:05:30.307 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-12-15T17:05:30.370 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-12-15T17:05:30.557 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-12-15T17:05:30.558 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-12-15T17:05:30.610 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-12-15T17:05:30.815 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-12-15T17:05:30.858 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "keyfi 2021-12-15T17:05:30.860 INFO:teuthology.task.ansible.out:le": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-12-15T17:05:31.104 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-12-15T17:05:31.146 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-12-15T17:05:31.399 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-12-15T17:05:31.410 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-12-15T17:05:31.650 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-12-15T17:05:31.662 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-12-15T17:05:31.663 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-12-15T17:05:31.891 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-12-15T17:05:31.893 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-12-15T17:05:32.125 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-12-15T17:05:32.147 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-12-15T17:05:32.358 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-12-15T17:05:32.391 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-12-15T17:05:32.601 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-12-15T17:05:32.645 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-12-15T17:05:32.844 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-12-15T17:05:32.909 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-12-15T17:05:33.076 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-12-15T17:05:33.161 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-12-15T17:05:33.308 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-12-15T17:05:33.424 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-12-15T17:05:33.550 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-12-15T17:05:33.665 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-12-15T17:05:33.781 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-12-15T17:05:33.918 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "keyfi 2021-12-15T17:05:33.920 INFO:teuthology.task.ansible.out:le": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-12-15T17:05:34.024 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-12-15T17:05:34.161 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-12-15T17:05:34.256 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-12-15T17:05:34.413 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-12-15T17:05:34.487 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-12-15T17:05:34.655 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-12-15T17:05:34.658 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-12-15T17:05:34.730 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-12-15T17:05:34.907 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-12-15T17:05:34.960 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-12-15T17:05:35.154 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-12-15T17:05:35.198 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-12-15T17:05:35.408 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-12-15T17:05:35.440 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-12-15T17:05:35.649 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-12-15T17:05:35.671 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-12-15T17:05:35.901 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-12-15T17:05:35.912 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-12-15T17:05:36.142 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-12-15T17:05:36.143 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-12-15T17:05:36.383 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-12-15T17:05:36.385 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-12-15T17:05:36.614 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-12-15T17:05:36.636 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-12-15T17:05:36.855 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-12-15T17:05:36.876 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-12-15T17:05:37.085 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-12-15T17:05:37.138 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-12-15T17:05:37.328 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAqZvQXoWrH8ZaQ9Y8KK1B9JBCuD8Zspp2mWtjrWXGotKBCSGdDn3QSDEFtwpoK6vXZ3BijhUmrQ1bNUSo59556+T+wpfJAPJjuvB9i7A2lKSX74heuDl0QhJwHdt9mbCtpZ//7+IU0srzTn5qKwJ16GXeomgcyOZmefSGi/sbn0wRUR/UGA/LOJ3EK0nXcxyrv3iDfv75KRL87kaXtPRJRvSlXB5PSuDQtIoXhzMuD4nnh879Gwy0sYE/FUXcbl7LQb+/9rNJLE2POVNjArGv7RMp3y0K3k2/02AjbZvQwhfuIVayzTpmUiGY2Fo6hJRkH8ZT0aYF8uotfweGK18EYw== rsa-key-20151116", "key_options": null, "keyfile": "/home/oro/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oro", "validate_certs": true} 2021-12-15T17:05:37.381 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-12-15T17:05:37.564 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIADWYnXrwweaWs8/3gaUDctlaelfSPRh0Cs3G2SHbER0 mrfreezeex@inferno", "key_options": null, "keyfile": "/home/mrfreezeex/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mrfreezeex", "validate_certs": true} 2021-12-15T17:05:37.649 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-12-15T17:05:37.671 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:37 +0000 (0:00:34.983) 0:01:19.381 **** 2021-12-15T17:05:37.754 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:37.756 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.765 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.767 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.777 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.788 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.791 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.800 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.811 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.813 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:37.864 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:37.866 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-12-15T17:05:38.053 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:38.055 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-12-15T17:05:38.068 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.076 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.118 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:38.120 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-12-15T17:05:38.307 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:38.309 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-12-15T17:05:38.360 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:38.362 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-12-15T17:05:38.540 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-12-15T17:05:38.553 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.564 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.576 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.577 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.588 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.600 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.602 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.611 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.613 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.623 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:38.625 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkM 2021-12-15T17:05:38.627 INFO:teuthology.task.ansible.out:Eik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.637 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.648 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.650 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.660 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.671 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.683 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.685 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.695 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.706 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.708 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.718 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.729 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.731 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.740 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.742 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.753 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.765 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.767 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.777 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.779 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.789 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.800 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.812 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.814 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.824 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.835 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.836 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.847 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.858 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.860 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.870 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.871 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:38.872 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.883 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.894 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.895 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.905 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.918 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.930 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.942 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.951 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.952 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.962 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.964 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.974 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.985 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.987 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.997 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:38.999 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.008 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.009 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.020 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.031 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.033 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.042 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.044 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.054 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.065 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.067 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.077 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.088 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.091 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.100 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.111 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.112 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-12-15T17:05:39.122 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:39.128 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.137 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.145 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.147 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.158 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.169 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.187 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.198 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.200 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.209 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.211 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.220 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.232 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.243 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.254 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.265 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.267 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.277 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.279 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.289 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.300 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.310 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.312 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.322 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.333 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.335 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.344 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.355 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.357 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.367 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:39.369 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn 2021-12-15T17:05:39.370 INFO:teuthology.task.ansible.out:': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.379 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.380 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.391 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.402 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.413 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.424 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.426 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.436 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.437 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.448 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.460 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.470 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.481 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.483 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.492 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.503 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.505 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.515 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.516 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.526 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.537 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.539 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.548 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.550 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.560 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.575 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.616 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:39 +0000 (0:00:01.944) 0:01:21.325 **** 2021-12-15T17:05:39.622 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:39.623 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-12-15T17:05:39.688 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:39.710 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:39 +0000 (0:00:00.093) 0:01:21.419 **** 2021-12-15T17:05:39.854 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:39.855 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-12-15T17:05:39.990 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:39.992 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-12-15T17:05:40.114 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:40.116 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-12-15T17:05:40.178 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:40.179 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-12-15T17:05:40.364 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:40.366 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} TASK [Remove revoked users] **************************************************** ok: [smithi122.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-12-15T17:05:40.564 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-12-15T17:05:40.607 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:40.609 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAqZvQXoWrH8ZaQ9Y8KK1B9JBCuD8Zspp2mWtjrWXGotKBCSGdDn3QSDEFtwpoK6vXZ3BijhUmrQ1bNUSo59556+T+wpfJAPJjuvB9i7A2lKSX74heuDl0QhJwHdt9mbCtpZ//7+IU0srzTn5qKwJ16GXeomgcyOZmefSGi/sbn0wRUR/UGA/LOJ3EK0nXcxyrv3iDfv75KRL87kaXtPRJRvSlXB5PSuDQtIoXhzMuD4nnh879Gwy0sYE/FUXcbl7LQb+/9rNJLE2POVNjArGv7RMp3y0K3k2/02AjbZvQwhfuIVayzTpmUiGY2Fo6hJRkH8ZT0aYF8uotfweGK18EYw== rsa-key-20151116", "key_options": null, "keyfile": "/home/oro/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oro", "validate_certs": true} 2021-12-15T17:05:40.754 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:40.756 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-12-15T17:05:40.848 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-12-15T17:05:40.850 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIADWYnXrwweaWs8/3gaUDctlaelfSPRh0Cs3G2SHbER0 mrfreezeex@inferno", "key_options": null, "keyfile": "/home/mrfreezeex/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mrfreezeex", "validate_certs": true} 2021-12-15T17:05:40.945 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:40 +0000 (0:00:01.235) 0:01:22.655 **** 2021-12-15T17:05:40.954 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:40.955 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-12-15T17:05:41.051 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:41.053 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.062 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.073 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.084 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.086 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.096 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.107 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.108 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.139 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:41.141 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-12-15T17:05:41.334 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-12-15T17:05:41.366 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:41.367 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-12-15T17:05:41.377 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.378 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.535 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:41.537 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-12-15T17:05:41.631 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:41.632 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-12-15T17:05:41.737 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:41.739 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-12-15T17:05:41.863 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:41.865 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-12-15T17:05:41.874 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.886 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.888 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.898 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.908 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.909 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.920 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.931 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.933 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi122.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:41.934 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.945 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.956 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.957 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.968 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.970 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.980 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.991 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:41.992 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.002 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.014 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.016 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.025 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.027 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.037 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.048 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.049 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.059 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.061 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.071 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.082 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.084 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.093 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.095 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.105 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.116 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.118 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.127 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:42.129 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.131 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.141 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.152 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.154 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.164 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.175 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.186 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.189 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.198 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.209 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.211 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.221 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.232 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.234 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.243 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.254 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.256 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.266 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.277 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.279 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.288 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.299 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.310 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.312 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.322 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.324 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi122.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-12-15T17:05:42.333 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-12-15T17:05:42.335 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.344 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.355 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.357 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.367 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.378 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.379 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.390 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.401 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.403 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.412 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.423 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.425 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.434 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.436 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.446 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.457 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.458 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.469 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.480 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.482 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.491 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.502 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.504 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.514 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.516 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.525 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:42.527 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.537 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.539 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.549 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.571 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.573 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.583 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.594 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.596 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.605 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.616 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.618 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.627 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.638 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.640 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.650 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.652 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.661 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.673 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.675 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.685 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.696 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.708 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.709 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.719 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.721 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi122.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.732 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.743 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.745 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.755 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.757 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.767 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.778 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.789 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.791 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.801 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.812 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.814 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.824 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.835 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.837 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.847 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.848 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.860 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.861 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.872 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.883 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.884 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.894 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.896 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.907 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:42.933 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:42.934 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-12-15T17:05:42.955 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:42 +0000 (0:00:02.009) 0:01:24.664 **** 2021-12-15T17:05:43.026 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-12-15T17:05:43.027 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:43.049 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:43 +0000 (0:00:00.093) 0:01:24.758 **** 2021-12-15T17:05:43.109 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:43.111 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-12-15T17:05:43.297 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-12-15T17:05:43.350 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-12-15T17:05:43.505 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-12-15T17:05:43.558 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-12-15T17:05:43.694 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-12-15T17:05:43.758 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-12-15T17:05:43.886 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-12-15T17:05:43.961 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-12-15T17:05:44.087 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-12-15T17:05:44.172 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-12-15T17:05:44.279 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-12-15T17:05:44.375 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-12-15T17:05:44.470 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-12-15T17:05:44.585 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-12-15T17:05:44.671 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-12-15T17:05:44.797 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-12-15T17:05:44.861 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-12-15T17:05:45.007 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-12-15T17:05:45.050 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-12-15T17:05:45.218 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-12-15T17:05:45.245 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-12-15T17:05:45.413 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-12-15T17:05:45.444 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-12-15T17:05:45.614 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-12-15T17:05:45.636 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-12-15T17:05:45.824 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-12-15T17:05:45.824 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-12-15T17:05:46.011 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-12-15T17:05:46.022 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-12-15T17:05:46.220 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-12-15T17:05:46.222 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-12-15T17:05:46.408 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-12-15T17:05:46.430 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-12-15T17:05:46.597 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-12-15T17:05:46.629 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-12-15T17:05:46.796 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-12-15T17:05:46.828 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-12-15T17:05:46.984 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-12-15T17:05:47.038 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-12-15T17:05:47.175 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-12-15T17:05:47.239 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-12-15T17:05:47.368 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-12-15T17:05:47.452 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-12-15T17:05:47.568 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": false, "item": "davidz", "name": "davidz", "state": "absent"} 2021-12-15T17:05:47.652 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-12-15T17:05:47.767 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": false, "item": "wusui", "name": "wusui", "state": "absent"} 2021-12-15T17:05:47.873 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-12-15T17:05:47.957 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": false, "item": "nwatkins", "name": "nwatkins", "state": "absent"} 2021-12-15T17:05:48.072 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-12-15T17:05:48.156 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": false, "item": "sidharthanup", "name": "sidharthanup", "state": "absent"} 2021-12-15T17:05:48.272 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-12-15T17:05:48.346 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": false, "item": "varsha", "name": "varsha", "state": "absent"} 2021-12-15T17:05:48.473 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-12-15T17:05:48.537 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": false, "item": "hmunjulu", "name": "hmunjulu", "state": "absent"} 2021-12-15T17:05:48.672 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-12-15T17:05:48.736 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-12-15T17:05:48.883 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-12-15T17:05:48.935 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-12-15T17:05:49.081 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-12-15T17:05:49.124 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-12-15T17:05:49.290 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-12-15T17:05:49.312 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-12-15T17:05:49.490 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-12-15T17:05:49.511 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-12-15T17:05:49.689 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-12-15T17:05:49.700 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-12-15T17:05:49.888 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-12-15T17:05:49.889 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-12-15T17:05:50.087 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-12-15T17:05:50.099 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-12-15T17:05:50.278 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-12-15T17:05:50.300 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-12-15T17:05:50.468 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-12-15T17:05:50.524 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:50 +0000 (0:00:07.475) 0:01:32.234 **** 2021-12-15T17:05:50.531 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-12-15T17:05:50.711 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-12-15T17:05:50.910 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-12-15T17:05:51.086 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-12-15T17:05:51.088 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "checksum": "09dde075805f5d4bdea4ecc07cef18b8cc5cbdf3", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "2cf30c8356ea33991eb6921c31aaa08a", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1639587950.6223836-17723-58326615074042/source", "state": "file", "uid": 0} 2021-12-15T17:05:51.110 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:51 +0000 (0:00:00.584) 0:01:32.818 **** 2021-12-15T17:05:51.116 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:51.118 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-12-15T17:05:51.180 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-12-15T17:05:51.182 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => { "msg": "Host smithi122.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-12-15T17:05:51.202 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:51 +0000 (0:00:00.093) 0:01:32.911 **** 2021-12-15T17:05:51.284 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-12-15T17:05:51.285 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-12-15T17:05:51.307 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:51 +0000 (0:00:00.104) 0:01:33.016 **** 2021-12-15T17:05:51.324 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:51.326 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": false, "item": "davidz", "name": "davidz", "state": "absent"} 2021-12-15T17:05:51.377 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:51.399 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:51 +0000 (0:00:00.092) 0:01:33.108 **** 2021-12-15T17:05:51.520 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi022.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": false, "item": "wusui", "name": "wusui", "state": "absent"} 2021-12-15T17:05:51.710 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.076442", "end": "2021-12-15 17:05:51.690965", "rc": 0, "start": "2021-12-15 17:05:51.614523", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-12-15T17:05:51.730 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:51 +0000 (0:00:00.331) 0:01:33.440 **** 2021-12-15T17:05:51.737 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:51.739 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": false, "item": "nwatkins", "name": "nwatkins", "state": "absent"} 2021-12-15T17:05:51.791 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:51.818 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:51 +0000 (0:00:00.086) 0:01:33.527 **** 2021-12-15T17:05:51.887 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-12-15T17:05:51.889 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:51.910 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:51 +0000 (0:00:00.092) 0:01:33.619 **** 2021-12-15T17:05:51.927 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:51.929 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": false, "item": "sidharthanup", "name": "sidharthanup", "state": "absent"} 2021-12-15T17:05:51.982 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-12-15T17:05:51.983 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.010 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.100) 0:01:33.720 **** 2021-12-15T17:05:52.092 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-12-15T17:05:52.094 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.114 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.103) 0:01:33.823 **** 2021-12-15T17:05:52.131 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:52.132 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": false, "item": "varsha", "name": "varsha", "state": "absent"} 2021-12-15T17:05:52.184 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.207 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.093) 0:01:33.916 **** 2021-12-15T17:05:52.297 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.319 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.111) 0:01:34.028 **** 2021-12-15T17:05:52.347 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:52.349 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": false, "item": "hmunjulu", "name": "hmunjulu", "state": "absent"} 2021-12-15T17:05:52.400 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-12-15T17:05:52.402 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:52.422 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.103) 0:01:34.132 **** 2021-12-15T17:05:52.512 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-12-15T17:05:52.514 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.534 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.112) 0:01:34.244 **** 2021-12-15T17:05:52.552 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:52.554 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-12-15T17:05:52.607 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.629 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.094) 0:01:34.338 **** 2021-12-15T17:05:52.698 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.723 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.093) 0:01:34.432 **** 2021-12-15T17:05:52.750 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:52.752 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-12-15T17:05:52.793 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-12-15T17:05:52.795 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:52.816 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.093) 0:01:34.525 **** 2021-12-15T17:05:52.884 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-12-15T17:05:52.886 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:52.905 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.089) 0:01:34.615 **** 2021-12-15T17:05:52.953 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:52.955 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-12-15T17:05:52.975 INFO:teuthology.task.ansible.out: 2021-12-15T17:05:52.977 INFO:teuthology.task.ansible.out:TASK [common : Register with subscription-manager.] **************************** skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:52.998 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:52 +0000 (0:00:00.092) 0:01:34.707 **** 2021-12-15T17:05:53.066 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-12-15T17:05:53.068 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.088 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.090) 0:01:34.798 **** 2021-12-15T17:05:53.157 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:53.158 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} TASK [common : List CDN releases available to system] ************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.179 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.090) 0:01:34.889 **** 2021-12-15T17:05:53.248 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.270 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.090) 0:01:34.980 **** 2021-12-15T17:05:53.340 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-12-15T17:05:53.342 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.361 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.091) 0:01:35.071 **** 2021-12-15T17:05:53.367 INFO:teuthology.task.ansible.out: 2021-12-15T17:05:53.369 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi022.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-12-15T17:05:53.422 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:53.445 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.083) 0:01:35.154 **** 2021-12-15T17:05:53.524 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-12-15T17:05:53.526 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.546 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.101) 0:01:35.255 **** 2021-12-15T17:05:53.564 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:53.565 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-12-15T17:05:53.659 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.681 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.135) 0:01:35.390 **** 2021-12-15T17:05:53.761 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi022.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-12-15T17:05:53.784 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.102) 0:01:35.493 **** 2021-12-15T17:05:53.785 INFO:teuthology.task.ansible.out: 2021-12-15T17:05:53.864 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-12-15T17:05:53.865 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.888 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.104) 0:01:35.597 **** 2021-12-15T17:05:53.967 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:53.969 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:53.990 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:53 +0000 (0:00:00.101) 0:01:35.699 **** 2021-12-15T17:05:54.060 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:54.082 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:54 +0000 (0:00:00.091) 0:01:35.791 **** 2021-12-15T17:05:54.150 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:54.172 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:54 +0000 (0:00:00.090) 0:01:35.882 **** 2021-12-15T17:05:54.178 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:54.181 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-12-15T17:05:54.233 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:54.265 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:54 +0000 (0:00:00.092) 0:01:35.974 **** 2021-12-15T17:05:54.334 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-12-15T17:05:54.336 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:54.367 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:54 +0000 (0:00:00.101) 0:01:36.076 **** 2021-12-15T17:05:54.375 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-12-15T17:05:54.377 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-12-15T17:05:54.423 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:54 +0000 (0:00:00.056) 0:01:36.132 **** 2021-12-15T17:05:54.618 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-12-15T17:05:54.619 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:05:54.641 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:54 +0000 (0:00:00.217) 0:01:36.350 **** 2021-12-15T17:05:54.993 INFO:teuthology.task.ansible.out: 2021-12-15T17:05:54.994 INFO:teuthology.task.ansible.out:TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-12-15T17:05:54.996 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "checksum": "09dde075805f5d4bdea4ecc07cef18b8cc5cbdf3", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "2cf30c8356ea33991eb6921c31aaa08a", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1639587954.5178585-18089-94410223419210/source", "state": "file", "uid": 0} 2021-12-15T17:05:55.019 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.377) 0:01:36.728 **** 2021-12-15T17:05:55.098 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi022.front.sepia.ceph.com] => { "msg": "Host smithi022.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-12-15T17:05:55.119 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.100) 0:01:36.828 **** 2021-12-15T17:05:55.198 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-12-15T17:05:55.200 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-12-15T17:05:55.223 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.103) 0:01:36.932 **** 2021-12-15T17:05:55.303 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-12-15T17:05:55.304 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:55.325 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.102) 0:01:37.034 **** 2021-12-15T17:05:55.437 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-12-15T17:05:55.439 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-12-15T17:05:55.626 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-12-15T17:05:55.628 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.065366", "end": "2021-12-15 17:05:55.610411", "rc": 0, "start": "2021-12-15 17:05:55.545045", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-12-15T17:05:55.648 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.323) 0:01:37.358 **** 2021-12-15T17:05:55.717 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:55.739 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.090) 0:01:37.448 **** 2021-12-15T17:05:55.819 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:55.842 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.102) 0:01:37.551 **** 2021-12-15T17:05:55.873 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-12-15T17:05:55.874 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-12-15T17:05:55.914 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.071) 0:01:37.623 **** 2021-12-15T17:05:55.915 INFO:teuthology.task.ansible.out: 2021-12-15T17:05:55.925 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-12-15T17:05:55.926 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:55.959 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:55 +0000 (0:00:00.045) 0:01:37.669 **** 2021-12-15T17:05:56.031 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-12-15T17:05:56.033 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.054 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.094) 0:01:37.763 **** 2021-12-15T17:05:56.123 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-12-15T17:05:56.125 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.144 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.090) 0:01:37.854 **** 2021-12-15T17:05:56.213 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.235 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.090) 0:01:37.944 **** 2021-12-15T17:05:56.312 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-12-15T17:05:56.314 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi022.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:56.334 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.099) 0:01:38.043 **** 2021-12-15T17:05:56.403 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-12-15T17:05:56.405 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.429 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.095) 0:01:38.138 **** 2021-12-15T17:05:56.437 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-12-15T17:05:56.439 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-12-15T17:05:56.492 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.513 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.084) 0:01:38.223 **** 2021-12-15T17:05:56.582 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-12-15T17:05:56.584 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.606 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.092) 0:01:38.316 **** 2021-12-15T17:05:56.676 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-12-15T17:05:56.678 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:56.699 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.092) 0:01:38.408 **** 2021-12-15T17:05:56.769 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-12-15T17:05:56.771 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.793 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.093) 0:01:38.502 **** 2021-12-15T17:05:56.862 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-12-15T17:05:56.864 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:56.886 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.092) 0:01:38.595 **** 2021-12-15T17:05:56.894 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-12-15T17:05:56.895 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-12-15T17:05:56.925 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.038) 0:01:38.634 **** 2021-12-15T17:05:56.963 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-12-15T17:05:56.965 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:56.988 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:56 +0000 (0:00:00.063) 0:01:38.697 **** 2021-12-15T17:05:57.006 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-12-15T17:05:57.008 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.030 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.041) 0:01:38.739 **** 2021-12-15T17:05:57.057 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-12-15T17:05:57.059 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.081 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.051) 0:01:38.790 **** 2021-12-15T17:05:57.099 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-12-15T17:05:57.101 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.123 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.042) 0:01:38.832 **** 2021-12-15T17:05:57.151 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-12-15T17:05:57.153 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.175 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.052) 0:01:38.885 **** 2021-12-15T17:05:57.204 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-12-15T17:05:57.206 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.227 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.052) 0:01:38.937 **** 2021-12-15T17:05:57.266 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-12-15T17:05:57.267 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.290 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.062) 0:01:39.000 **** 2021-12-15T17:05:57.309 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-12-15T17:05:57.311 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.339 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.048) 0:01:39.048 **** 2021-12-15T17:05:57.367 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-12-15T17:05:57.369 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:57.395 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.055) 0:01:39.104 **** 2021-12-15T17:05:57.414 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-12-15T17:05:57.416 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.427 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.452 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.056) 0:01:39.161 **** 2021-12-15T17:05:57.470 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-12-15T17:05:57.472 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.495 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.043) 0:01:39.204 **** 2021-12-15T17:05:57.524 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-12-15T17:05:57.526 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.550 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.054) 0:01:39.259 **** 2021-12-15T17:05:57.567 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-12-15T17:05:57.569 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.595 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.045) 0:01:39.304 **** 2021-12-15T17:05:57.624 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-12-15T17:05:57.626 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.648 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.052) 0:01:39.357 **** 2021-12-15T17:05:57.665 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-12-15T17:05:57.667 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.690 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.042) 0:01:39.399 **** 2021-12-15T17:05:57.719 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-12-15T17:05:57.721 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.743 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.052) 0:01:39.452 **** 2021-12-15T17:05:57.760 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-12-15T17:05:57.762 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.785 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.042) 0:01:39.494 **** 2021-12-15T17:05:57.814 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-12-15T17:05:57.816 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.838 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.052) 0:01:39.547 **** 2021-12-15T17:05:57.859 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.881 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.043) 0:01:39.590 **** 2021-12-15T17:05:57.911 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-12-15T17:05:57.913 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.935 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.054) 0:01:39.645 **** 2021-12-15T17:05:57.954 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-12-15T17:05:57.956 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:57.981 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:57 +0000 (0:00:00.045) 0:01:39.690 **** 2021-12-15T17:05:58.051 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi022.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-12-15T17:05:58.074 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:58 +0000 (0:00:00.093) 0:01:39.784 **** 2021-12-15T17:05:58.167 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-12-15T17:05:58.168 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:58.190 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:58 +0000 (0:00:00.115) 0:01:39.899 **** 2021-12-15T17:05:58.258 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-12-15T17:05:58.260 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:05:58.281 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:58 +0000 (0:00:00.090) 0:01:39.990 **** 2021-12-15T17:05:58.516 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-12-15T17:05:58.518 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.390896", "end": "2021-12-15 17:05:58.500314", "rc": 0, "start": "2021-12-15 17:05:58.109418", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:05:58.539 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:58 +0000 (0:00:00.258) 0:01:40.248 **** 2021-12-15T17:05:58.545 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-12-15T17:05:58.546 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:05:58.569 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:58 +0000 (0:00:00.030) 0:01:40.278 **** 2021-12-15T17:05:59.117 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-12-15T17:05:59.118 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-12-15T17:05:59.584 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-12-15T17:05:59.607 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:05:59 +0000 (0:00:01.037) 0:01:41.316 **** 2021-12-15T17:06:00.143 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi022.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-12-15T17:06:00.270 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-12-15T17:06:00.272 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:00.293 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.685) 0:01:42.002 **** 2021-12-15T17:06:00.375 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-12-15T17:06:00.377 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:00.400 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.106) 0:01:42.109 **** 2021-12-15T17:06:00.470 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-12-15T17:06:00.472 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:00.492 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.092) 0:01:42.202 **** 2021-12-15T17:06:00.562 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:00.584 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.091) 0:01:42.294 **** 2021-12-15T17:06:00.622 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-12-15T17:06:00.624 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-12-15T17:06:00.652 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.067) 0:01:42.362 **** 2021-12-15T17:06:00.660 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-12-15T17:06:00.662 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:00.685 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.032) 0:01:42.394 **** 2021-12-15T17:06:00.725 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-12-15T17:06:00.727 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:00.750 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.065) 0:01:42.460 **** 2021-12-15T17:06:00.820 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:00.841 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.091) 0:01:42.551 **** 2021-12-15T17:06:00.910 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-12-15T17:06:00.911 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:00.931 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:00 +0000 (0:00:00.089) 0:01:42.641 **** 2021-12-15T17:06:01.000 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-12-15T17:06:01.001 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.021 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.089) 0:01:42.731 **** 2021-12-15T17:06:01.090 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-12-15T17:06:01.092 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.101 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.103 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.127 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.106) 0:01:42.837 **** 2021-12-15T17:06:01.197 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-12-15T17:06:01.199 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.220 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.092) 0:01:42.929 **** 2021-12-15T17:06:01.228 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-12-15T17:06:01.230 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-12-15T17:06:01.255 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.033) 0:01:42.963 **** 2021-12-15T17:06:01.282 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-12-15T17:06:01.283 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.306 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.052) 0:01:43.015 **** 2021-12-15T17:06:01.375 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-12-15T17:06:01.377 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.399 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.093) 0:01:43.109 **** 2021-12-15T17:06:01.468 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-12-15T17:06:01.469 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.490 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.090) 0:01:43.199 **** 2021-12-15T17:06:01.549 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:01.574 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:01 +0000 (0:00:00.083) 0:01:43.283 **** 2021-12-15T17:06:02.524 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-12-15T17:06:02.525 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.723326", "end": "2021-12-15 17:06:02.500344", "rc": 0, "start": "2021-12-15 17:06:01.777018", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:06:02.546 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:02 +0000 (0:00:00.972) 0:01:44.255 **** 2021-12-15T17:06:02.729 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:02.753 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:02 +0000 (0:00:00.206) 0:01:44.462 **** 2021-12-15T17:06:03.287 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi122.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-12-15T17:06:03.731 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-12-15T17:06:04.127 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-12-15T17:06:04.128 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:04.149 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:04 +0000 (0:00:01.396) 0:01:45.859 **** 2021-12-15T17:06:04.167 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-12-15T17:06:04.169 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-12-15T17:06:04.193 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:04 +0000 (0:00:00.043) 0:01:45.903 **** 2021-12-15T17:06:04.210 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-12-15T17:06:04.212 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:04.234 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:04 +0000 (0:00:00.040) 0:01:45.943 **** 2021-12-15T17:06:04.302 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-12-15T17:06:04.304 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:04.325 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:04 +0000 (0:00:00.090) 0:01:46.034 **** 2021-12-15T17:06:04.382 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-12-15T17:06:04.384 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:04.405 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:04 +0000 (0:00:00.080) 0:01:46.114 **** 2021-12-15T17:06:04.425 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:04 +0000 (0:00:00.019) 0:01:46.134 **** 2021-12-15T17:06:04.462 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-12-15T17:06:04.465 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:04.486 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:04 +0000 (0:00:00.061) 0:01:46.195 **** 2021-12-15T17:06:04.922 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi122.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-12-15T17:06:05.005 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-12-15T17:06:05.007 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-12-15T17:06:05.028 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:05 +0000 (0:00:00.542) 0:01:46.737 **** 2021-12-15T17:06:05.347 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi122.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-12-15T17:06:05.764 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-12-15T17:06:05.824 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:05 +0000 (0:00:00.796) 0:01:47.534 **** 2021-12-15T17:06:05.863 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-12-15T17:06:05.864 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-12-15T17:06:05.885 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:05 +0000 (0:00:00.060) 0:01:47.594 **** 2021-12-15T17:06:06.078 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-12-15T17:06:06.080 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-12-15T17:06:06.099 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:06 +0000 (0:00:00.214) 0:01:47.809 **** 2021-12-15T17:06:06.136 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-12-15T17:06:06.138 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:06.147 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:06.158 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:06.169 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:06.191 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:06 +0000 (0:00:00.091) 0:01:47.901 **** 2021-12-15T17:06:06.622 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-12-15T17:06:06.624 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:06.645 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:06 +0000 (0:00:00.453) 0:01:48.355 **** 2021-12-15T17:06:07.160 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-12-15T17:06:07.162 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-12-15T17:06:07.622 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-12-15T17:06:07.664 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-12-15T17:06:07.666 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:07.688 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:07 +0000 (0:00:01.042) 0:01:49.397 **** 2021-12-15T17:06:07.715 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:07.737 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:07 +0000 (0:00:00.048) 0:01:49.446 **** 2021-12-15T17:06:08.065 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-12-15T17:06:08.067 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-12-15T17:06:08.091 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:08 +0000 (0:00:00.354) 0:01:49.800 **** 2021-12-15T17:06:08.192 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-12-15T17:06:08.194 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-12-15T17:06:08.217 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:08 +0000 (0:00:00.125) 0:01:49.926 **** 2021-12-15T17:06:08.296 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-12-15T17:06:08.298 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:06:08.320 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:08 +0000 (0:00:00.103) 0:01:50.029 **** 2021-12-15T17:06:08.682 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-12-15T17:06:08.705 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:08 +0000 (0:00:00.384) 0:01:50.414 **** 2021-12-15T17:06:08.743 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-12-15T17:06:08.745 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:08.768 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:08 +0000 (0:00:00.062) 0:01:50.477 **** 2021-12-15T17:06:08.805 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-12-15T17:06:08.807 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-12-15T17:06:08.972 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-12-15T17:06:08.974 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:06:08.994 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:08 +0000 (0:00:00.226) 0:01:50.704 **** 2021-12-15T17:06:09.240 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-12-15T17:06:09.242 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} TASK [common : Check firewalld status] ***************************************** fatal: [smithi122.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-12-15T17:06:09.262 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:09 +0000 (0:00:00.267) 0:01:50.972 **** 2021-12-15T17:06:09.289 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-12-15T17:06:09.291 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:09.311 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:09 +0000 (0:00:00.048) 0:01:51.020 **** 2021-12-15T17:06:09.675 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-12-15T17:06:09.677 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-12-15T17:06:09.701 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:09 +0000 (0:00:00.389) 0:01:51.410 **** 2021-12-15T17:06:09.740 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-12-15T17:06:09.741 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-12-15T17:06:09.761 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:09 +0000 (0:00:00.060) 0:01:51.470 **** 2021-12-15T17:06:09.788 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-12-15T17:06:09.793 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-12-15T17:06:09.813 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:09 +0000 (0:00:00.051) 0:01:51.522 **** 2021-12-15T17:06:09.964 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-12-15T17:06:09.966 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-12-15T17:06:09.988 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:09 +0000 (0:00:00.175) 0:01:51.697 **** 2021-12-15T17:06:10.037 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-12-15T17:06:10.039 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:10.049 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:10.060 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:10.082 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:10 +0000 (0:00:00.093) 0:01:51.791 **** 2021-12-15T17:06:10.441 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-12-15T17:06:10.443 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-12-15 17:03:19 UTC", "ActiveEnterTimestampMonotonic": "21469302", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target var-run.mount tmp.mount local-fs.target time-sync.target sysinit.target basic.target system.slice -.mount network-online.target systemd-journald.socket systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-12-15 17:03:19 UTC", "AssertTimestampMonotonic": "21466721", "Before": "plymouth-quit.service shutdown.target getty@tty1.service multi-user.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-12-15 17:03:19 UTC", "Co 2021-12-15T17:06:10.444 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "21466721", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1316", "ExecMainStartTimestamp": "Wed 2021-12-15 17:03:19 UTC", "ExecMainStartTimestampMonotonic": "21469245", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-12-15 17:03:19 UTC] ; stop_time=[n/a] ; pid=1316 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-12-15 17:03:19 UTC", "InactiveExitTimestampMonotonic": "214693 2021-12-15T17:06:10.446 INFO:teuthology.task.ansible.out:02", "InvocationID": "34309ed53e3f44aeb183b5d19778275f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1316", "MemoryAccounting": "yes", "MemoryCurrent": "679936", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlG 2021-12-15T17:06:10.448 INFO:teuthology.task.ansible.out:roups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-12-15 17:03:19 UTC", "StateChangeTimestampMonotonic": "21469302", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-12-15 17:03:19 UTC", "WatchdogTimestampMonotonic": "21469301", "WatchdogUSec": "0"}} 2021-12-15T17:06:10.468 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:10 +0000 (0:00:00.385) 0:01:52.177 **** 2021-12-15T17:06:10.661 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-12-15T17:06:10.663 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001734", "end": "2021-12-15 17:06:10.639185", "rc": 0, "start": "2021-12-15 17:06:10.637451", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-12-15T17:06:10.684 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:10 +0000 (0:00:00.215) 0:01:52.393 **** 2021-12-15T17:06:11.643 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-12-15T17:06:11.645 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:11.666 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:11 +0000 (0:00:00.982) 0:01:53.375 **** 2021-12-15T17:06:11.693 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** 2021-12-15T17:06:11.695 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:11.716 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:11 +0000 (0:00:00.049) 0:01:53.425 **** 2021-12-15T17:06:12.120 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-12-15T17:06:12.122 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:12.142 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:12 +0000 (0:00:00.426) 0:01:53.852 **** 2021-12-15T17:06:12.202 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-12-15T17:06:12.203 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-12-15T17:06:12.225 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:12 +0000 (0:00:00.082) 0:01:53.934 **** 2021-12-15T17:06:12.698 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-12-15T17:06:12.701 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-12-15T17:06:12.721 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:12 +0000 (0:00:00.495) 0:01:54.430 **** 2021-12-15T17:06:12.759 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi022.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:12.760 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-12-15T17:06:12.781 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:12 +0000 (0:00:00.060) 0:01:54.490 **** 2021-12-15T17:06:12.986 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-12-15T17:06:12.988 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:06:13.009 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:13 +0000 (0:00:00.227) 0:01:54.718 **** 2021-12-15T17:06:13.235 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-12-15T17:06:13.237 INFO:teuthology.task.ansible.out:fatal: [smithi022.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-12-15T17:06:13.258 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:13 +0000 (0:00:00.249) 0:01:54.968 **** 2021-12-15T17:06:13.297 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:06:13.321 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:13 +0000 (0:00:00.062) 0:01:55.030 **** 2021-12-15T17:06:13.810 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-12-15T17:06:13.812 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-12-15T17:06:13.832 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:13 +0000 (0:00:00.511) 0:01:55.542 **** 2021-12-15T17:06:14.144 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-12-15T17:06:14.145 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-12-15 17:02:54 UTC", "ActiveEnterTimestampMonotonic": "21666789", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target network-online.target systemd-tmpfiles-setup.service local-fs.target time-sync.target sysinit.target basic.target system.slice -.mount systemd-journald.socket tmp.mount var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-12-15 17:02:54 UTC", "AssertTimestampMonotonic": "21654977", "Before": "plymouth-quit.service xdm.service shutdown.target getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-12-15 17:02:54 UTC", "Co 2021-12-15T17:06:14.147 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "21654977", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1259", "ExecMainStartTimestamp": "Wed 2021-12-15 17:02:54 UTC", "ExecMainStartTimestampMonotonic": "21666676", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-12-15 17:02:54 UTC] ; stop_time=[n/a] ; pid=1259 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-12-15 17:02:54 UTC", "InactiveExitTimestampMonotonic": "216667 2021-12-15T17:06:14.148 INFO:teuthology.task.ansible.out:89", "InvocationID": "c920dbfd04ca4fc29a5a375fa1af9d91", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127268", "LimitNPROCSoft": "127268", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127268", "LimitSIGPENDINGSoft": "127268", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1259", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlG 2021-12-15T17:06:14.150 INFO:teuthology.task.ansible.out:roups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-12-15 17:02:54 UTC", "StateChangeTimestampMonotonic": "21666789", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203628", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-12-15 17:02:54 UTC", "WatchdogTimestampMonotonic": "21666788", "WatchdogUSec": "0"}} 2021-12-15T17:06:14.171 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:14 +0000 (0:00:00.339) 0:01:55.881 **** 2021-12-15T17:06:14.366 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-12-15T17:06:14.368 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001881", "end": "2021-12-15 17:06:14.348653", "rc": 0, "start": "2021-12-15 17:06:14.346772", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-12-15T17:06:14.389 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:14 +0000 (0:00:00.217) 0:01:56.098 **** 2021-12-15T17:06:15.942 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-12-15T17:06:15.944 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:06:15.964 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:15 +0000 (0:00:01.575) 0:01:57.674 **** 2021-12-15T17:06:51.338 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-12-15T17:06:51.340 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-85.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-85.el8.noarch", "Removed: selinux-policy-3.14.3-83.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-83.el8.noarch"]} 2021-12-15T17:06:51.362 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:51 +0000 (0:00:35.397) 0:02:33.071 **** 2021-12-15T17:06:52.073 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-12-15T17:06:52.076 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-12-15T17:06:52.100 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:52 +0000 (0:00:00.737) 0:02:33.809 **** 2021-12-15T17:06:56.482 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-85.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-85.el8.noarch", "Removed: selinux-policy-3.14.3-83.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-83.el8.noarch"]} 2021-12-15T17:06:56.505 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:56 +0000 (0:00:04.405) 0:02:38.214 **** 2021-12-15T17:06:56.972 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-12-15T17:06:56.974 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-12-15T17:06:56.995 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:06:56 +0000 (0:00:00.489) 0:02:38.704 **** 2021-12-15T17:07:07.859 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-12-15T17:07:07.861 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.564274", "end": "2021-12-15 17:07:07.842126", "failed_when_result": false, "rc": 0, "start": "2021-12-15 17:06:52.277852", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-12-15T17:07:07.890 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:07 +0000 (0:00:10.894) 0:02:49.599 **** 2021-12-15T17:07:08.357 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-12-15T17:07:08.359 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-12-15T17:07:08.380 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:08 +0000 (0:00:00.489) 0:02:50.089 **** 2021-12-15T17:07:08.595 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.021957", "end": "2021-12-15 17:07:08.574276", "rc": 0, "start": "2021-12-15 17:07:08.552319", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:08.617 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:08 +0000 (0:00:00.237) 0:02:50.326 **** 2021-12-15T17:07:08.820 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-12-15T17:07:08.822 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.005653", "end": "2021-12-15 17:07:08.797642", "rc": 0, "start": "2021-12-15 17:07:08.791989", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:08.843 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:08 +0000 (0:00:00.226) 0:02:50.552 **** 2021-12-15T17:07:13.386 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-12-15T17:07:13.388 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.171489", "end": "2021-12-15 17:07:13.362922", "failed_when_result": false, "rc": 0, "start": "2021-12-15 17:06:57.191433", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-12-15T17:07:13.410 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:13 +0000 (0:00:04.566) 0:02:55.119 **** 2021-12-15T17:07:13.897 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-12-15T17:07:13.898 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-12-15T17:07:13.919 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:13 +0000 (0:00:00.509) 0:02:55.628 **** 2021-12-15T17:07:14.150 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.023626", "end": "2021-12-15 17:07:14.126131", "rc": 0, "start": "2021-12-15 17:07:14.102505", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:14.175 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:14 +0000 (0:00:00.256) 0:02:55.884 **** 2021-12-15T17:07:14.392 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-12-15T17:07:14.394 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.017365", "end": "2021-12-15 17:07:14.373987", "rc": 0, "start": "2021-12-15 17:07:14.356622", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:14.415 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:14 +0000 (0:00:00.239) 0:02:56.124 **** 2021-12-15T17:07:24.661 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.623437", "end": "2021-12-15 17:07:24.635652", "rc": 0, "start": "2021-12-15 17:07:09.012215", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:24.684 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:24 +0000 (0:00:10.269) 0:03:06.394 **** 2021-12-15T17:07:24.881 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-12-15T17:07:24.883 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-12-15T17:07:24.903 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:24 +0000 (0:00:00.218) 0:03:06.612 **** 2021-12-15T17:07:24.931 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-12-15T17:07:24.933 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-12-15T17:07:24.954 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:24 +0000 (0:00:00.050) 0:03:06.663 **** 2021-12-15T17:07:24.992 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-12-15T17:07:24.993 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-12-15T17:07:25.014 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.060) 0:03:06.723 **** 2021-12-15T17:07:25.052 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-12-15T17:07:25.054 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-12-15T17:07:25.055 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-12-15T17:07:25.076 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.062) 0:03:06.786 **** 2021-12-15T17:07:25.114 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-12-15T17:07:25.116 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-12-15T17:07:25.118 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-12-15T17:07:25.140 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.063) 0:03:06.849 **** 2021-12-15T17:07:25.323 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-12-15T17:07:25.325 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-12-15T17:07:25.345 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.205) 0:03:07.055 **** 2021-12-15T17:07:25.539 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-12-15T17:07:25.540 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2021-12-15T17:07:25.561 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.215) 0:03:07.271 **** 2021-12-15T17:07:25.589 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:25.612 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.050) 0:03:07.321 **** 2021-12-15T17:07:25.641 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:25.663 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.050) 0:03:07.372 **** 2021-12-15T17:07:25.893 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-12-15T17:07:25.895 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-12-15T17:07:25.917 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:25 +0000 (0:00:00.254) 0:03:07.626 **** 2021-12-15T17:07:26.491 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-12-15T17:07:26.493 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-12-15T17:07:26.513 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:26 +0000 (0:00:00.596) 0:03:08.222 **** 2021-12-15T17:07:26.540 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-12-15T17:07:26.542 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi122"}, "changed": false} 2021-12-15T17:07:26.562 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:26 +0000 (0:00:00.049) 0:03:08.271 **** 2021-12-15T17:07:27.372 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-12-15T17:07:27.374 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi122", "ansible_hostname": "smithi122", "ansible_nodename": "smithi122"}, "changed": false, "name": "smithi122"} 2021-12-15T17:07:27.394 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:27 +0000 (0:00:00.831) 0:03:09.103 **** 2021-12-15T17:07:27.847 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-12-15T17:07:27.849 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-12-15T17:07:27.869 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:27 +0000 (0:00:00.474) 0:03:09.578 **** 2021-12-15T17:07:28.335 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-12-15T17:07:28.337 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-12-15T17:07:28.358 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:28 +0000 (0:00:00.488) 0:03:10.067 **** 2021-12-15T17:07:28.919 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-12-15T17:07:28.921 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-12-15T17:07:28.944 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:28 +0000 (0:00:00.585) 0:03:10.653 **** 2021-12-15T17:07:29.417 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-12-15T17:07:29.419 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-12-15T17:07:29.444 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:29 +0000 (0:00:00.499) 0:03:11.153 **** 2021-12-15T17:07:29.665 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-12-15T17:07:29.667 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-12-15T17:07:29.694 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:29 +0000 (0:00:00.249) 0:03:11.403 **** 2021-12-15T17:07:29.901 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-12-15T17:07:29.922 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:29 +0000 (0:00:00.228) 0:03:11.631 **** 2021-12-15T17:07:30.151 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-12-15T17:07:30.154 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-12-15T17:07:30.184 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:30 +0000 (0:00:00.261) 0:03:11.893 **** 2021-12-15T17:07:30.672 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-12-15T17:07:30.674 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-12-15T17:07:30.696 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:30 +0000 (0:00:00.512) 0:03:12.406 **** 2021-12-15T17:07:31.058 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-12-15T17:07:31.059 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.443944", "end": "2021-12-15 17:07:31.037565", "rc": 0, "start": "2021-12-15 17:07:14.593621", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:31.081 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.384) 0:03:12.790 **** 2021-12-15T17:07:31.171 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-12-15T17:07:31.173 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-12-15T17:07:31.195 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.114) 0:03:12.905 **** 2021-12-15T17:07:31.285 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-12-15T17:07:31.287 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-12-15T17:07:31.312 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.116) 0:03:13.021 **** 2021-12-15T17:07:31.341 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-12-15T17:07:31.343 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-12-15T17:07:31.364 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.051) 0:03:13.073 **** 2021-12-15T17:07:31.403 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-12-15T17:07:31.405 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-12-15T17:07:31.426 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.061) 0:03:13.135 **** 2021-12-15T17:07:31.475 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-12-15T17:07:31.477 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-12-15T17:07:31.478 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-12-15T17:07:31.499 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.073) 0:03:13.209 **** 2021-12-15T17:07:31.538 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-12-15T17:07:31.540 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-12-15T17:07:31.541 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-12-15T17:07:31.562 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.062) 0:03:13.271 **** 2021-12-15T17:07:31.756 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-12-15T17:07:31.758 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-12-15T17:07:31.778 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.216) 0:03:13.487 **** 2021-12-15T17:07:31.982 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-12-15T17:07:31.984 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2021-12-15T17:07:32.004 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:31 +0000 (0:00:00.226) 0:03:13.713 **** 2021-12-15T17:07:32.032 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-12-15T17:07:32.034 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:32.055 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:32 +0000 (0:00:00.050) 0:03:13.764 **** 2021-12-15T17:07:32.084 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-12-15T17:07:32.085 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:32.106 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:32 +0000 (0:00:00.051) 0:03:13.815 **** 2021-12-15T17:07:32.334 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-12-15T17:07:32.336 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-12-15T17:07:32.357 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:32 +0000 (0:00:00.250) 0:03:14.066 **** 2021-12-15T17:07:32.914 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-12-15T17:07:32.915 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-12-15T17:07:32.936 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:32 +0000 (0:00:00.579) 0:03:14.645 **** 2021-12-15T17:07:32.974 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-12-15T17:07:32.976 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi022"}, "changed": false} 2021-12-15T17:07:33.000 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:32 +0000 (0:00:00.063) 0:03:14.709 **** 2021-12-15T17:07:33.469 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-12-15T17:07:33.470 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi022", "ansible_hostname": "smithi022", "ansible_nodename": "smithi022"}, "changed": false, "name": "smithi022"} 2021-12-15T17:07:33.491 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:33 +0000 (0:00:00.491) 0:03:15.200 **** 2021-12-15T17:07:33.643 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-12-15T17:07:33.645 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.255073", "end": "2021-12-15 17:07:33.622414", "rc": 0, "start": "2021-12-15 17:07:31.367341", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:33.667 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:33 +0000 (0:00:00.176) 0:03:15.377 **** 2021-12-15T17:07:33.882 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-12-15T17:07:33.884 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002873", "end": "2021-12-15 17:07:33.863883", "rc": 0, "start": "2021-12-15 17:07:33.861010", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:33.905 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:33 +0000 (0:00:00.237) 0:03:15.615 **** 2021-12-15T17:07:33.944 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:07:33.946 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-12-15T17:07:33.966 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:33 +0000 (0:00:00.060) 0:03:15.676 **** 2021-12-15T17:07:33.973 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-12-15T17:07:33.975 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-12-15T17:07:33.998 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:33 +0000 (0:00:00.032) 0:03:15.708 **** 2021-12-15T17:07:34.017 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-12-15T17:07:34.042 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:34 +0000 (0:00:00.043) 0:03:15.751 **** 2021-12-15T17:07:34.469 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-12-15T17:07:34.500 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:34 +0000 (0:00:00.457) 0:03:16.209 **** 2021-12-15T17:07:34.517 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-12-15T17:07:34.519 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-12-15T17:07:34.984 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-12-15T17:07:35.079 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-12-15T17:07:35.081 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-12-15T17:07:35.102 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:35 +0000 (0:00:00.602) 0:03:16.811 **** 2021-12-15T17:07:35.426 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-12-15T17:07:35.427 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-12-15T17:07:35.594 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-12-15T17:07:35.596 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-12-15T17:07:35.619 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:35 +0000 (0:00:00.517) 0:03:17.328 **** 2021-12-15T17:07:35.825 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-12-15T17:07:35.827 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-12-15T17:07:35.848 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:35 +0000 (0:00:00.228) 0:03:17.557 **** 2021-12-15T17:07:35.866 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-12-15T17:07:35.868 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-12-15T17:07:35.893 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:35 +0000 (0:00:00.045) 0:03:17.602 **** 2021-12-15T17:07:36.049 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-12-15T17:07:36.051 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-12-15T17:07:36.073 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:36 +0000 (0:00:00.179) 0:03:17.782 **** 2021-12-15T17:07:36.320 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-12-15T17:07:36.322 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-12-15T17:07:36.344 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:36 +0000 (0:00:00.270) 0:03:18.053 **** 2021-12-15T17:07:36.381 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-12-15T17:07:36.383 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-12-15T17:07:36.825 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/apache-arrow.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "state": "file", "uid": 0} 2021-12-15T17:07:36.850 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:36 +0000 (0:00:00.505) 0:03:18.559 **** 2021-12-15T17:07:36.858 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-12-15T17:07:36.859 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-12-15T17:07:36.893 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:36 +0000 (0:00:00.043) 0:03:18.603 **** 2021-12-15T17:07:37.330 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-12-15T17:07:37.332 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-12-15T17:07:37.384 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-12-15T17:07:37.386 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-12-15T17:07:37.407 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:37 +0000 (0:00:00.513) 0:03:19.116 **** 2021-12-15T17:07:37.786 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-12-15T17:07:38.236 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-12-15T17:07:38.673 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2021-12-15T17:07:38.705 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:38 +0000 (0:00:01.297) 0:03:20.414 **** 2021-12-15T17:07:39.130 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-12-15T17:07:39.132 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.224307", "end": "2021-12-15 17:07:39.112609", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-12-15 17:07:38.888302", "stderr": "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist\nInvalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist\nEnabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nInvalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist\nInvalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "stderr_lines": ["Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The 2021-12-15T17:07:39.133 INFO:teuthology.task.ansible.out: Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository.", "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist"], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-12-15T17:07:39.154 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:39 +0000 (0:00:00.449) 0:03:20.863 **** 2021-12-15T17:07:39.808 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-12-15T17:07:39.810 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.203083", "end": "2021-12-15 17:07:39.785101", "rc": 0, "start": "2021-12-15 17:07:37.582018", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:39.833 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:39 +0000 (0:00:00.678) 0:03:21.542 **** 2021-12-15T17:07:40.059 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-12-15T17:07:40.061 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002823", "end": "2021-12-15 17:07:40.036873", "rc": 0, "start": "2021-12-15 17:07:40.034050", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:07:40.081 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:40 +0000 (0:00:00.248) 0:03:21.790 **** 2021-12-15T17:07:40.130 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:07:40.132 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-12-15T17:07:40.156 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:40 +0000 (0:00:00.074) 0:03:21.865 **** 2021-12-15T17:07:40.214 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-12-15T17:07:40.216 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-12-15T17:07:40.239 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:40 +0000 (0:00:00.083) 0:03:21.948 **** 2021-12-15T17:07:40.734 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-12-15T17:07:40.736 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-12-15T17:07:41.192 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-12-15T17:07:41.649 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-12-15T17:07:41.661 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-12-15T17:07:41.662 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:07:41.684 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:41 +0000 (0:00:01.444) 0:03:23.393 **** 2021-12-15T17:07:42.101 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-12-15T17:07:42.103 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-12-15T17:07:42.127 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:42 +0000 (0:00:00.442) 0:03:23.836 **** 2021-12-15T17:07:42.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-12-15T17:07:42.428 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc"} 2021-12-15T17:07:42.625 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-12-15T17:07:42.627 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-12-15T17:07:42.823 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* ok: [smithi122.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-12-15T17:07:43.084 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-12-15T17:07:43.086 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/apache-arrow.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "state": "file", "uid": 0} 2021-12-15T17:07:43.108 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.981) 0:03:24.818 **** 2021-12-15T17:07:43.136 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-12-15T17:07:43.137 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": false, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-12-15T17:07:43.160 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.051) 0:03:24.869 **** 2021-12-15T17:07:43.198 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-12-15T17:07:43.200 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:43.228 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.067) 0:03:24.937 **** 2021-12-15T17:07:43.268 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:07:43.270 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-12-15T17:07:43.301 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.074) 0:03:25.011 **** 2021-12-15T17:07:43.339 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-12-15T17:07:43.340 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:43.361 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.060) 0:03:25.071 **** 2021-12-15T17:07:43.399 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:43.421 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.059) 0:03:25.131 **** 2021-12-15T17:07:43.605 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-12-15T17:07:43.607 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-12-15T17:07:43.628 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-12-15T17:07:43.630 INFO:teuthology.task.ansible.out:fatal: [smithi122.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007784", "end": "2021-12-15 17:07:43.609422", "msg": "non-zero return code", "rc": 1, "start": "2021-12-15 17:07:43.601638", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-12-15T17:07:43.650 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.228) 0:03:25.360 **** 2021-12-15T17:07:43.687 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-12-15T17:07:43.689 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:43.712 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:43 +0000 (0:00:00.061) 0:03:25.421 **** 2021-12-15T17:07:44.053 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-12-15T17:07:44.055 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-12-15T17:07:44.521 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-12-15T17:07:44.977 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2021-12-15T17:07:45.003 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:45 +0000 (0:00:01.290) 0:03:26.712 **** 2021-12-15T17:07:45.457 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-12-15T17:07:45.459 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.238935", "end": "2021-12-15 17:07:45.431321", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-12-15 17:07:45.192386", "stderr": "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist\nInvalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist\nEnabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nInvalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist\nInvalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "stderr_lines": ["Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The 2021-12-15T17:07:45.460 INFO:teuthology.task.ansible.out: Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository.", "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist", "Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id \"failovermethod\" does not exist"], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-12-15T17:07:45.480 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:45 +0000 (0:00:00.477) 0:03:27.189 **** 2021-12-15T17:07:46.533 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-12-15T17:07:46.560 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:46 +0000 (0:00:01.079) 0:03:28.269 **** 2021-12-15T17:07:48.055 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-12-15T17:07:48.057 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:07:48.078 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:48 +0000 (0:00:01.517) 0:03:29.787 **** 2021-12-15T17:07:48.556 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-12-15T17:07:48.558 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc"} 2021-12-15T17:07:48.964 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-12-15T17:07:49.037 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-12-15T17:07:49.039 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-12-15T17:07:49.061 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.983) 0:03:30.770 **** 2021-12-15T17:07:49.289 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-12-15T17:07:49.291 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": false, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-12-15T17:07:49.325 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.263) 0:03:31.033 **** 2021-12-15T17:07:49.356 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:49.379 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.054) 0:03:31.088 **** 2021-12-15T17:07:49.417 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:07:49.419 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-12-15T17:07:49.440 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.061) 0:03:31.149 **** 2021-12-15T17:07:49.478 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:49.500 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.059) 0:03:31.209 **** 2021-12-15T17:07:49.547 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-12-15T17:07:49.549 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:49.569 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.069) 0:03:31.278 **** 2021-12-15T17:07:49.774 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-12-15T17:07:49.776 INFO:teuthology.task.ansible.out:fatal: [smithi022.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007902", "end": "2021-12-15 17:07:49.757822", "msg": "non-zero return code", "rc": 1, "start": "2021-12-15 17:07:49.749920", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-12-15T17:07:49.799 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.229) 0:03:31.508 **** 2021-12-15T17:07:49.827 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-12-15T17:07:49.829 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:07:49.850 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:49 +0000 (0:00:00.051) 0:03:31.560 **** 2021-12-15T17:07:52.629 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-12-15T17:07:52.630 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-12-15T17:07:52.656 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:07:52 +0000 (0:00:02.805) 0:03:34.365 **** 2021-12-15T17:07:55.232 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-12-15T17:07:55.265 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} Wednesday 15 December 2021 17:07:55 +0000 (0:00:02.599) 0:03:36.964 **** 2021-12-15T17:08:00.015 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-12-15T17:08:00.017 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: yum-4.7.0-5.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: dnf-4.7.0-5.el8.noarch", "Installed: python3-dnf-plugins-core-4.0.21-7.el8.noarch", "Installed: yum-utils-4.0.21-7.el8.noarch", "Installed: qemu-kvm-block-rbd-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: dnf-data-4.7.0-5.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: python3-dnf-4.7.0-5.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-core-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: dnf-plugins-core-4.0.21-7.el8.noarch", "Removed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: dnf-4.7.0-4.el8.noarch", "Removed: dnf-data-4.7.0-4.el8.noarch", "Removed: dnf-plugins-core-4.0.21-3.el8.noarch", "Removed: python3-dnf-4.7.0-4.el8.noarch", "Removed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Removed: yum-4.7.0-4.el8.noarch", "Removed: yum-utils-4.0.21-3.el8.noarch"]} 2021-12-15T17:08:00.039 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:00 +0000 (0:00:04.783) 0:03:41.748 **** 2021-12-15T17:08:00.077 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-12-15T17:08:00.079 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:00.099 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:00 +0000 (0:00:00.059) 0:03:41.808 **** 2021-12-15T17:08:01.593 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-12-15T17:08:01.595 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:01.615 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:01 +0000 (0:00:01.516) 0:03:43.324 **** 2021-12-15T17:08:03.075 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-12-15T17:08:03.077 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:03.097 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:03 +0000 (0:00:01.482) 0:03:44.807 **** 2021-12-15T17:08:03.396 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-12-15T17:08:03.397 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket polkit.service sysinit.target basic.target dbus.service system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nftables.service ip6tables.service ebtables.service iptables.service ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-12-15T17:08:03.399 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-12-15T17:08:03.400 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target dbus.socket system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-12-15T17:08:03.402 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-12-15T17:08:03.420 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:03 +0000 (0:00:00.322) 0:03:45.129 **** 2021-12-15T17:08:03.707 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-12-15T17:08:03.709 INFO:teuthology.task.ansible.out:fatal: [smithi122.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-12-15T17:08:03.730 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:03 +0000 (0:00:00.309) 0:03:45.439 **** 2021-12-15T17:08:04.362 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-12-15T17:08:04.363 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-12-15T17:08:04.384 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.654) 0:03:46.094 **** 2021-12-15T17:08:04.413 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-12-15T17:08:04.415 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.436 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.051) 0:03:46.145 **** 2021-12-15T17:08:04.464 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-12-15T17:08:04.465 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.488 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.051) 0:03:46.197 **** 2021-12-15T17:08:04.515 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-12-15T17:08:04.517 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.538 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.049) 0:03:46.246 **** 2021-12-15T17:08:04.564 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-12-15T17:08:04.566 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.586 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.049) 0:03:46.295 **** 2021-12-15T17:08:04.613 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-12-15T17:08:04.615 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.635 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.049) 0:03:46.344 **** 2021-12-15T17:08:04.662 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-12-15T17:08:04.664 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.684 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.049) 0:03:46.393 **** 2021-12-15T17:08:04.711 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-12-15T17:08:04.733 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.048) 0:03:46.442 **** 2021-12-15T17:08:04.760 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-12-15T17:08:04.762 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.782 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.048) 0:03:46.491 **** 2021-12-15T17:08:04.808 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.830 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.048) 0:03:46.539 **** 2021-12-15T17:08:04.868 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-12-15T17:08:04.870 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.890 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.060) 0:03:46.600 **** 2021-12-15T17:08:04.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-12-15T17:08:04.919 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:04.939 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.048) 0:03:46.648 **** 2021-12-15T17:08:04.966 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-12-15T17:08:04.987 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:04 +0000 (0:00:00.048) 0:03:46.697 **** 2021-12-15T17:08:05.014 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-12-15T17:08:05.015 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.037 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:46.746 **** 2021-12-15T17:08:05.064 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-12-15T17:08:05.066 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.087 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:46.796 **** 2021-12-15T17:08:05.114 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-12-15T17:08:05.116 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.139 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.052) 0:03:46.849 **** 2021-12-15T17:08:05.166 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-12-15T17:08:05.168 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.189 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:46.898 **** 2021-12-15T17:08:05.216 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.239 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:46.948 **** 2021-12-15T17:08:05.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-12-15T17:08:05.268 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.289 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.050) 0:03:46.998 **** 2021-12-15T17:08:05.316 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-12-15T17:08:05.317 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.338 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:47.047 **** 2021-12-15T17:08:05.365 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-12-15T17:08:05.366 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.387 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.048) 0:03:47.096 **** 2021-12-15T17:08:05.414 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-12-15T17:08:05.416 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.436 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:47.145 **** 2021-12-15T17:08:05.463 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-12-15T17:08:05.465 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.486 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:47.195 **** 2021-12-15T17:08:05.514 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-12-15T17:08:05.515 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.536 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.050) 0:03:47.245 **** 2021-12-15T17:08:05.563 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-12-15T17:08:05.565 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.585 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.048) 0:03:47.294 **** 2021-12-15T17:08:05.611 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-12-15T17:08:05.613 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.634 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.048) 0:03:47.343 **** 2021-12-15T17:08:05.660 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-12-15T17:08:05.662 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.682 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.048) 0:03:47.391 **** 2021-12-15T17:08:05.709 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.733 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.050) 0:03:47.442 **** 2021-12-15T17:08:05.761 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-12-15T17:08:05.762 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.785 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.052) 0:03:47.494 **** 2021-12-15T17:08:05.813 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-12-15T17:08:05.817 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.835 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.049) 0:03:47.544 **** 2021-12-15T17:08:05.862 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.883 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.048) 0:03:47.593 **** 2021-12-15T17:08:05.909 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-12-15T17:08:05.911 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.931 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.047) 0:03:47.641 **** 2021-12-15T17:08:05.938 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-12-15T17:08:05.939 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: yum-4.7.0-5.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: dnf-4.7.0-5.el8.noarch", "Installed: python3-dnf-plugins-core-4.0.21-7.el8.noarch", "Installed: yum-utils-4.0.21-7.el8.noarch", "Installed: qemu-kvm-block-rbd-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: dnf-data-4.7.0-5.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: python3-dnf-4.7.0-5.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-core-15:6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64", "Installed: dnf-plugins-core-4.0.21-7.el8.noarch", "Removed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Removed: dnf-4.7.0-4.el8.noarch", "Removed: dnf-data-4.7.0-4.el8.noarch", "Removed: dnf-plugins-core-4.0.21-3.el8.noarch", "Removed: python3-dnf-4.7.0-4.el8.noarch", "Removed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Removed: yum-4.7.0-4.el8.noarch", "Removed: yum-utils-4.0.21-3.el8.noarch"]} 2021-12-15T17:08:05.962 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.030) 0:03:47.671 **** 2021-12-15T17:08:05.969 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-12-15T17:08:05.971 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:05.993 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:05 +0000 (0:00:00.031) 0:03:47.703 **** 2021-12-15T17:08:06.010 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-12-15T17:08:06.012 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.032 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.038) 0:03:47.741 **** 2021-12-15T17:08:06.038 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-12-15T17:08:06.040 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.062 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.030) 0:03:47.772 **** 2021-12-15T17:08:06.090 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-12-15T17:08:06.092 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.111 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.048) 0:03:47.821 **** 2021-12-15T17:08:06.138 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-12-15T17:08:06.140 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.160 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.048) 0:03:47.869 **** 2021-12-15T17:08:06.186 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.207 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.047) 0:03:47.917 **** 2021-12-15T17:08:06.234 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-12-15T17:08:06.236 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.256 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.048) 0:03:47.965 **** 2021-12-15T17:08:06.282 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-12-15T17:08:06.284 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.304 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.048) 0:03:48.013 **** 2021-12-15T17:08:06.330 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-12-15T17:08:06.332 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.351 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.047) 0:03:48.061 **** 2021-12-15T17:08:06.377 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-12-15T17:08:06.379 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:06.399 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.047) 0:03:48.108 **** 2021-12-15T17:08:06.611 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-12-15T17:08:06.613 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-12-15T17:08:06.637 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.238) 0:03:48.347 **** 2021-12-15T17:08:06.850 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-12-15T17:08:06.852 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:06.873 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:06 +0000 (0:00:00.235) 0:03:48.582 **** 2021-12-15T17:08:07.077 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-12-15T17:08:07.079 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-12-15T17:08:07.099 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.226) 0:03:48.809 **** 2021-12-15T17:08:07.127 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-12-15T17:08:07.128 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.149 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.049) 0:03:48.858 **** 2021-12-15T17:08:07.176 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-12-15T17:08:07.178 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.199 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.049) 0:03:48.908 **** 2021-12-15T17:08:07.226 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.248 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.049) 0:03:48.958 **** 2021-12-15T17:08:07.276 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-12-15T17:08:07.278 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.298 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.049) 0:03:49.008 **** 2021-12-15T17:08:07.326 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-12-15T17:08:07.328 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.348 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.050) 0:03:49.058 **** 2021-12-15T17:08:07.377 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.399 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.050) 0:03:49.109 **** 2021-12-15T17:08:07.428 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.450 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.050) 0:03:49.159 **** 2021-12-15T17:08:07.477 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-12-15T17:08:07.479 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.499 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.049) 0:03:49.208 **** 2021-12-15T17:08:07.527 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.548 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.048) 0:03:49.257 **** 2021-12-15T17:08:07.576 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-12-15T17:08:07.578 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} TASK [testnode : Set grub config.] ********************************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.602 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.053) 0:03:49.311 **** 2021-12-15T17:08:07.622 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.020) 0:03:49.331 **** 2021-12-15T17:08:07.650 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.672 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.050) 0:03:49.382 **** 2021-12-15T17:08:07.710 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.732 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.059) 0:03:49.441 **** 2021-12-15T17:08:07.759 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-12-15T17:08:07.761 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.781 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.048) 0:03:49.490 **** 2021-12-15T17:08:07.808 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.830 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.049) 0:03:49.540 **** 2021-12-15T17:08:07.868 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-12-15T17:08:07.870 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.891 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.060) 0:03:49.600 **** 2021-12-15T17:08:07.919 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-12-15T17:08:07.921 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:07.949 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:07 +0000 (0:00:00.058) 0:03:49.658 **** 2021-12-15T17:08:07.981 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-12-15T17:08:07.982 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.012 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.063) 0:03:49.722 **** 2021-12-15T17:08:08.043 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-12-15T17:08:08.045 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.065 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.052) 0:03:49.774 **** 2021-12-15T17:08:08.092 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-12-15T17:08:08.094 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.114 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.049) 0:03:49.824 **** 2021-12-15T17:08:08.142 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-12-15T17:08:08.143 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.164 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.049) 0:03:49.873 **** 2021-12-15T17:08:08.191 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-12-15T17:08:08.193 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.213 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.048) 0:03:49.922 **** 2021-12-15T17:08:08.250 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-12-15T17:08:08.252 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.273 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.060) 0:03:49.982 **** 2021-12-15T17:08:08.300 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:08.301 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-12-15T17:08:08.321 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.048) 0:03:50.031 **** 2021-12-15T17:08:08.369 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-12-15T17:08:08.371 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.380 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-12-15T17:08:08.382 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.392 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.394 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.403 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.414 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.443 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.121) 0:03:50.152 **** 2021-12-15T17:08:08.667 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003281", "end": "2021-12-15 17:08:08.641717", "rc": 0, "start": "2021-12-15 17:08:08.638436", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology-lock"]} 2021-12-15T17:08:08.691 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.248) 0:03:50.401 **** 2021-12-15T17:08:08.728 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-12-15T17:08:08.730 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.753 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.061) 0:03:50.462 **** 2021-12-15T17:08:08.779 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-12-15T17:08:08.781 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.801 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.048) 0:03:50.511 **** 2021-12-15T17:08:08.829 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-12-15T17:08:08.831 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.850 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.048) 0:03:50.559 **** 2021-12-15T17:08:08.876 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-12-15T17:08:08.878 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:08.897 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:08 +0000 (0:00:00.047) 0:03:50.606 **** 2021-12-15T17:08:09.152 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-12-15T17:08:09.154 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:09.176 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:09 +0000 (0:00:00.278) 0:03:50.885 **** 2021-12-15T17:08:09.505 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-12-15T17:08:09.506 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service sysinit.target dbus.socket polkit.service system.slice basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ipset.service ip6tables.service iptables.service ebtables.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-12-15T17:08:09.508 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-12-15T17:08:09.509 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127268", "LimitNPROCSoft": "127268", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127268", "LimitSIGPENDINGSoft": "127268", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-12-15T17:08:09.511 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203628", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-12-15T17:08:09.528 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:09 +0000 (0:00:00.351) 0:03:51.237 **** 2021-12-15T17:08:09.824 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-12-15T17:08:09.827 INFO:teuthology.task.ansible.out:fatal: [smithi022.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-12-15T17:08:09.847 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:09 +0000 (0:00:00.318) 0:03:51.556 **** 2021-12-15T17:08:10.133 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-12-15T17:08:10.135 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-12-15T17:08:10.157 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.309) 0:03:51.866 **** 2021-12-15T17:08:10.185 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-12-15T17:08:10.187 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.207 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.050) 0:03:51.916 **** 2021-12-15T17:08:10.235 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-12-15T17:08:10.237 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.261 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.053) 0:03:51.970 **** 2021-12-15T17:08:10.288 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-12-15T17:08:10.290 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.310 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.049) 0:03:52.019 **** 2021-12-15T17:08:10.338 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-12-15T17:08:10.340 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.360 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.049) 0:03:52.069 **** 2021-12-15T17:08:10.367 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-12-15T17:08:10.369 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:10.392 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.031) 0:03:52.101 **** 2021-12-15T17:08:10.398 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-12-15T17:08:10.400 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.427 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.034) 0:03:52.136 **** 2021-12-15T17:08:10.445 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-12-15T17:08:10.447 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930209841152, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227102012, 'block_used': 12993096, 'inode_total': 61054976, 'inode_available': 60989172, 'inode_used': 65804, 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227102012, "block_size": 4096, "block_total": 240095108, "block_used": 12993096, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989172, "inode_total": 61054976, "inode_used": 65804, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930209841152, "size_total": 983429562368, "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}} 2021-12-15T17:08:10.468 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.041) 0:03:52.178 **** 2021-12-15T17:08:10.474 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-12-15T17:08:10.476 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.498 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.029) 0:03:52.207 **** 2021-12-15T17:08:10.525 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-12-15T17:08:10.548 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.049) 0:03:52.257 **** 2021-12-15T17:08:10.576 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-12-15T17:08:10.577 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.597 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.049) 0:03:52.307 **** 2021-12-15T17:08:10.624 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-12-15T17:08:10.626 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.646 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.048) 0:03:52.356 **** 2021-12-15T17:08:10.664 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-12-15T17:08:10.665 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005271", "end": "2021-12-15 17:08:10.642888", "rc": 0, "start": "2021-12-15 17:08:10.637617", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:10.687 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.040) 0:03:52.396 **** 2021-12-15T17:08:10.693 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-12-15T17:08:10.695 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.718 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.031) 0:03:52.428 **** 2021-12-15T17:08:10.726 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-12-15T17:08:10.728 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930209841152, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227102012, 'block_used': 12993096, 'inode_total': 61054976, 'inode_available': 60989172, 'inode_used': 65804, 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227102012, "block_size": 4096, "block_total": 240095108, "block_used": 12993096, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989172, "inode_total": 61054976, "inode_used": 65804, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930209841152, "size_total": 983429562368, "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.750 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.032) 0:03:52.460 **** 2021-12-15T17:08:10.757 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-12-15T17:08:10.758 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.780 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.029) 0:03:52.489 **** 2021-12-15T17:08:10.787 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-12-15T17:08:10.810 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.029) 0:03:52.519 **** 2021-12-15T17:08:10.818 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-12-15T17:08:10.853 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.041) 0:03:52.561 **** 2021-12-15T17:08:10.884 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-12-15T17:08:10.886 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.917 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.065) 0:03:52.626 **** 2021-12-15T17:08:10.947 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-12-15T17:08:10.948 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:10.973 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:10 +0000 (0:00:00.055) 0:03:52.682 **** 2021-12-15T17:08:11.000 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-12-15T17:08:11.002 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.023 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:52.732 **** 2021-12-15T17:08:11.051 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.072 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:52.782 **** 2021-12-15T17:08:11.100 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-12-15T17:08:11.102 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.122 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:52.831 **** 2021-12-15T17:08:11.149 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-12-15T17:08:11.151 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.171 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:52.880 **** 2021-12-15T17:08:11.199 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-12-15T17:08:11.201 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.220 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:52.930 **** 2021-12-15T17:08:11.247 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-12-15T17:08:11.249 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.269 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:52.979 **** 2021-12-15T17:08:11.297 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-12-15T17:08:11.298 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.319 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:53.028 **** 2021-12-15T17:08:11.346 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.367 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.048) 0:03:53.077 **** 2021-12-15T17:08:11.395 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-12-15T17:08:11.396 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.416 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.048) 0:03:53.126 **** 2021-12-15T17:08:11.444 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.465 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.048) 0:03:53.174 **** 2021-12-15T17:08:11.492 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-12-15T17:08:11.494 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.513 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.048) 0:03:53.223 **** 2021-12-15T17:08:11.541 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.565 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.051) 0:03:53.274 **** 2021-12-15T17:08:11.593 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.614 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:53.323 **** 2021-12-15T17:08:11.641 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-12-15T17:08:11.642 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.663 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.048) 0:03:53.372 **** 2021-12-15T17:08:11.690 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-12-15T17:08:11.692 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.712 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:53.421 **** 2021-12-15T17:08:11.739 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-12-15T17:08:11.741 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.761 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:53.470 **** 2021-12-15T17:08:11.788 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-12-15T17:08:11.790 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.810 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.048) 0:03:53.519 **** 2021-12-15T17:08:11.837 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.860 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:53.569 **** 2021-12-15T17:08:11.886 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-12-15T17:08:11.888 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.908 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.048) 0:03:53.617 **** 2021-12-15T17:08:11.936 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-12-15T17:08:11.937 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:11.958 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:11 +0000 (0:00:00.049) 0:03:53.667 **** 2021-12-15T17:08:11.985 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-12-15T17:08:11.987 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:12.007 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.049) 0:03:53.717 **** 2021-12-15T17:08:12.035 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:12.037 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:12.057 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.049) 0:03:53.766 **** 2021-12-15T17:08:12.085 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-12-15T17:08:12.087 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:12.107 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.049) 0:03:53.816 **** 2021-12-15T17:08:12.134 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-12-15T17:08:12.136 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:12.157 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.050) 0:03:53.866 **** 2021-12-15T17:08:12.185 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:12.208 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.050) 0:03:53.917 **** 2021-12-15T17:08:12.235 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-12-15T17:08:12.237 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:12.260 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.052) 0:03:53.969 **** 2021-12-15T17:08:12.445 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:12.447 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVLZ3F0LW3nmRfXwRg7Qrn4XT3UTgkx08E'], 'uuids': ['940d6250-5bd8-47d4-8e98-a75761bb4cab'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.411221", "end": "2021-12-15 17:08:12.417844", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVLZ3F0LW3nmRfXwRg7Qrn4XT3UTgkx08E"], "labels": [], "masters": [], "uuids": ["940d6250-5bd8-47d4-8e98-a75761bb4cab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:11.006623", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:12.666 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-12-15T17:08:12.688 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.427) 0:03:54.397 **** 2021-12-15T17:08:12.913 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:12.935 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:12 +0000 (0:00:00.247) 0:03:54.645 **** 2021-12-15T17:08:13.139 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-12-15T17:08:13.141 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-12-15T17:08:13.162 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.226) 0:03:54.871 **** 2021-12-15T17:08:13.188 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-12-15T17:08:13.190 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.211 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:54.920 **** 2021-12-15T17:08:13.238 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-12-15T17:08:13.240 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.260 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:54.969 **** 2021-12-15T17:08:13.287 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-12-15T17:08:13.289 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.309 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:55.019 **** 2021-12-15T17:08:13.348 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.369 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.059) 0:03:55.078 **** 2021-12-15T17:08:13.397 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-12-15T17:08:13.399 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.419 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:55.128 **** 2021-12-15T17:08:13.446 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-12-15T17:08:13.448 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.468 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:55.177 **** 2021-12-15T17:08:13.495 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-12-15T17:08:13.497 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.517 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:55.226 **** 2021-12-15T17:08:13.544 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-12-15T17:08:13.546 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.566 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.048) 0:03:55.275 **** 2021-12-15T17:08:13.593 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-12-15T17:08:13.594 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.617 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.051) 0:03:55.326 **** 2021-12-15T17:08:13.645 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-12-15T17:08:13.647 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.668 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.050) 0:03:55.377 **** 2021-12-15T17:08:13.685 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:13.686 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-HYTG6x-l3nX-Ge5i-DyBQ-otvM-A23G-QcAiKH', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN', 'nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.049379", "end": "2021-12-15 17:08:13.654072", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-HYTG6x-l3nX-Ge5i-DyBQ-otvM-A23G-QcAiKH", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN", "nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:12.604693", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-12-15T17:08:13.687 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:13.697 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.720 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.051) 0:03:55.429 **** 2021-12-15T17:08:13.747 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-12-15T17:08:13.749 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.769 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:55.478 **** 2021-12-15T17:08:13.796 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-12-15T17:08:13.798 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.818 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:55.527 **** 2021-12-15T17:08:13.845 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-12-15T17:08:13.847 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.867 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.049) 0:03:55.577 **** 2021-12-15T17:08:13.894 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-12-15T17:08:13.896 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.916 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.048) 0:03:55.625 **** 2021-12-15T17:08:13.943 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-12-15T17:08:13.945 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:13.971 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:13 +0000 (0:00:00.055) 0:03:55.681 **** 2021-12-15T17:08:14.003 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-12-15T17:08:14.005 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.025 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.054) 0:03:55.735 **** 2021-12-15T17:08:14.052 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-12-15T17:08:14.054 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.074 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.048) 0:03:55.783 **** 2021-12-15T17:08:14.100 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-12-15T17:08:14.102 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.122 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.047) 0:03:55.831 **** 2021-12-15T17:08:14.149 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-12-15T17:08:14.151 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.170 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.048) 0:03:55.880 **** 2021-12-15T17:08:14.197 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.219 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.048) 0:03:55.928 **** 2021-12-15T17:08:14.246 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-12-15T17:08:14.248 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.268 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.049) 0:03:55.978 **** 2021-12-15T17:08:14.296 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:14.297 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-12-15T17:08:14.321 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.052) 0:03:56.030 **** 2021-12-15T17:08:14.369 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-12-15T17:08:14.371 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi022.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-12-15T17:08:14.380 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.381 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.391 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.393 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.402 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.428 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.106) 0:03:56.137 **** 2021-12-15T17:08:14.635 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-12-15T17:08:14.637 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003205", "end": "2021-12-15 17:08:14.615605", "rc": 0, "start": "2021-12-15 17:08:14.612400", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology-lock"]} 2021-12-15T17:08:14.662 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.233) 0:03:56.371 **** 2021-12-15T17:08:14.689 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-12-15T17:08:14.691 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.714 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.052) 0:03:56.423 **** 2021-12-15T17:08:14.740 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-12-15T17:08:14.742 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.762 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.048) 0:03:56.471 **** 2021-12-15T17:08:14.789 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-12-15T17:08:14.791 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.811 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.048) 0:03:56.520 **** 2021-12-15T17:08:14.837 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-12-15T17:08:14.839 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:14.860 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:14 +0000 (0:00:00.049) 0:03:56.569 **** 2021-12-15T17:08:14.930 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:14.932 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVDbGoAPItqNkWSmG4VBGpcvQqjl2lc295'], 'uuids': ['30a6b7cc-a9f7-41e4-b597-67c59fa90554'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.045175", "end": "2021-12-15 17:08:14.902175", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVDbGoAPItqNkWSmG4VBGpcvQqjl2lc295"], "labels": [], "masters": [], "uuids": ["30a6b7cc-a9f7-41e4-b597-67c59fa90554"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:13.857000", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:16.197 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVCiNa2DBVoWaADT9GvenCgPXquf28NCHg'], 'uuids': ['ffcefb45-8430-470c-a434-9298b8368839'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.069626", "end": "2021-12-15 17:08:16.170103", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVCiNa2DBVoWaADT9GvenCgPXquf28NCHg"], "labels": [], "masters": [], "uuids": ["ffcefb45-8430-470c-a434-9298b8368839"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:15.100477", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:16.437 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-12-15T17:08:16.438 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:16.462 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:16 +0000 (0:00:01.598) 0:03:58.168 **** 2021-12-15T17:08:16.521 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-12-15T17:08:16.523 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930209787904, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227101999, 'block_used': 12993109, 'inode_total': 61054976, 'inode_available': 60989168, 'inode_used': 65808, 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227101999, "block_size": 4096, "block_total": 240095108, "block_used": 12993109, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989168, "inode_total": 61054976, "inode_used": 65808, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930209787904, "size_total": 983429562368, "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}} 2021-12-15T17:08:16.546 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:16 +0000 (0:00:00.087) 0:03:58.255 **** 2021-12-15T17:08:16.750 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-12-15T17:08:16.752 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005531", "end": "2021-12-15 17:08:16.731104", "rc": 0, "start": "2021-12-15 17:08:16.725573", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:16.772 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:16 +0000 (0:00:00.225) 0:03:58.481 **** 2021-12-15T17:08:16.808 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-12-15T17:08:16.810 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930209787904, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227101999, 'block_used': 12993109, 'inode_total': 61054976, 'inode_available': 60989168, 'inode_used': 65808, 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227101999, "block_size": 4096, "block_total": 240095108, "block_used": 12993109, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989168, "inode_total": 61054976, "inode_used": 65808, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930209787904, "size_total": 983429562368, "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:16.830 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:16 +0000 (0:00:00.057) 0:03:58.539 **** 2021-12-15T17:08:16.856 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-12-15T17:08:16.877 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:16 +0000 (0:00:00.047) 0:03:58.587 **** 2021-12-15T17:08:17.433 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:17.435 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVR3JLi9K2UhwfGpYCbOTeYntsoQKQdaQo'], 'uuids': ['0406f99b-d08c-426b-9b80-48618d83f867'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.041291", "end": "2021-12-15 17:08:17.402524", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVR3JLi9K2UhwfGpYCbOTeYntsoQKQdaQo"], "labels": [], "masters": [], "uuids": ["0406f99b-d08c-426b-9b80-48618d83f867"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:16.361233", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi122.front.sepia.ceph.com] => 2021-12-15T17:08:17.436 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5J2', 'scsi-35000c500918e77a4', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2', 'wwn-0x5000c500918e77a4'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5L5J2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918e77a4', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5J2-part1', 'scsi-35000c500918e77a4-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2-part1', 'wwn-0x5000c500918e77a4-part1'], 'uuids': ['ccecca0b-fa27-451f-b628-c766cfeb470c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5J2", "scsi-35000c500918e77a4", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2", "wwn-0x5000c500918e77a4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5J2-part1", "scsi-35000c500918e77a4-part1 2021-12-15T17:08:17.438 INFO:teuthology.task.ansible.out:", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2-part1", "wwn-0x5000c500918e77a4-part1"], "labels": [], "masters": [], "uuids": ["ccecca0b-fa27-451f-b628-c766cfeb470c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L5J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e77a4"}}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:18.379 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rIGUDolLiyZNrs0Cr2T8gZqqcAnkfenAF'], 'uuids': ['eced2a54-7662-45c4-a6b8-31388f39953e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.278293", "end": "2021-12-15 17:08:18.348869", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rIGUDolLiyZNrs0Cr2T8gZqqcAnkfenAF"], "labels": [], "masters": [], "uuids": ["eced2a54-7662-45c4-a6b8-31388f39953e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:17.070576", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:18.712 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVQuLH0FTP2UZ2YyifBQvJI2QOc7qhbK1h'], 'uuids': ['a9ba3994-0f54-41f3-96ee-3904cf7ac834'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.089809", "end": "2021-12-15 17:08:18.686316", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVQuLH0FTP2UZ2YyifBQvJI2QOc7qhbK1h"], "labels": [], "masters": [], "uuids": ["a9ba3994-0f54-41f3-96ee-3904cf7ac834"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:17.596507", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:18.741 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:18 +0000 (0:00:01.863) 0:04:00.451 **** 2021-12-15T17:08:19.112 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-12-15T17:08:19.114 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVLZ3F0LW3nmRfXwRg7Qrn4XT3UTgkx08E'], 'uuids': ['940d6250-5bd8-47d4-8e98-a75761bb4cab'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.140003", "end": "2021-12-15 17:08:19.088165", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVLZ3F0LW3nmRfXwRg7Qrn4XT3UTgkx08E"], "labels": [], "masters": [], "uuids": ["940d6250-5bd8-47d4-8e98-a75761bb4cab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:18.948162", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137024 s, 842 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137024 s, 842 MB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:19.332 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-HYTG6x-l3nX-Ge5i-DyBQ-otvM-A23G-QcAiKH', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN', 'nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039863", "end": "2021-12-15 17:08:19.304106", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-HYTG6x-l3nX-Ge5i-DyBQ-otvM-A23G-QcAiKH", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN", "nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:19.264243", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0369465 s, 3.1 GB/s", "std 2021-12-15T17:08:19.333 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0369465 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:19.635 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:19.637 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-92v0up-410f-qtpQ-b708-Pn2V-z9Ko-xjYFKH', 'nvme-INTEL_SSDPED1D480GA_PHMB747402EC480DGN', 'nvme-nvme.8086-50484d42373437343032454334383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.058656", "end": "2021-12-15 17:08:19.608386", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-92v0up-410f-qtpQ-b708-Pn2V-z9Ko-xjYFKH", "nvme-INTEL_SSDPED1D480GA_PHMB747402EC480DGN", "nvme-nvme.8086-50484d42373437343032454334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:18.549730", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-12-15T17:08:19.638 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:19.647 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-12-15T17:08:19.649 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVDbGoAPItqNkWSmG4VBGpcvQqjl2lc295'], 'uuids': ['30a6b7cc-a9f7-41e4-b597-67c59fa90554'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140090", "end": "2021-12-15 17:08:19.621649", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVDbGoAPItqNkWSmG4VBGpcvQqjl2lc295"], "labels": [], "masters": [], "uuids": ["30a6b7cc-a9f7-41e4-b597-67c59fa90554"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:19.481559", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137263 s, 840 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137263 s, 840 MB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:19.959 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVCiNa2DBVoWaADT9GvenCgPXquf28NCHg'], 'uuids': ['ffcefb45-8430-470c-a434-9298b8368839'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.138405", "end": "2021-12-15 17:08:19.931837", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVCiNa2DBVoWaADT9GvenCgPXquf28NCHg"], "labels": [], "masters": [], "uuids": ["ffcefb45-8430-470c-a434-9298b8368839"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:19.793432", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135692 s, 850 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135692 s, 850 MB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:20.271 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVR3JLi9K2UhwfGpYCbOTeYntsoQKQdaQo'], 'uuids': ['0406f99b-d08c-426b-9b80-48618d83f867'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.141671", "end": "2021-12-15 17:08:20.248807", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVR3JLi9K2UhwfGpYCbOTeYntsoQKQdaQo"], "labels": [], "masters": [], "uuids": ["0406f99b-d08c-426b-9b80-48618d83f867"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:20.107136", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138945 s, 830 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138945 s, 830 MB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:20.282 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5J2', 'scsi-35000c500918e77a4', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2', 'wwn-0x5000c500918e77a4'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5L5J2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918e77a4', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5J2-part1', 'scsi-35000c500918e77a4-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2-part1', 'wwn-0x5000c500918e77a4-part1'], 'uuids': ['ccecca0b-fa27-451f-b628-c766cfeb470c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5J2", "scsi-35000c500918e77a4", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2", "wwn-0x5000c500918e77a4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5J2-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-12-15T17:08:20.284 INFO:teuthology.task.ansible.out:Z1W5L5J2-part1", "scsi-35000c500918e77a4-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5J2-part1", "wwn-0x5000c500918e77a4-part1"], "labels": [], "masters": [], "uuids": ["ccecca0b-fa27-451f-b628-c766cfeb470c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L5J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e77a4"}}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:20.594 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVQuLH0FTP2UZ2YyifBQvJI2QOc7qhbK1h'], 'uuids': ['a9ba3994-0f54-41f3-96ee-3904cf7ac834'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.140520", "end": "2021-12-15 17:08:20.571302", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ZXF3Bt31LkinGrLz3XwQJdEAJ62vHhIVQuLH0FTP2UZ2YyifBQvJI2QOc7qhbK1h"], "labels": [], "masters": [], "uuids": ["a9ba3994-0f54-41f3-96ee-3904cf7ac834"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:20.430782", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13777 s, 837 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13777 s, 837 MB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:20.622 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:20 +0000 (0:00:01.880) 0:04:02.331 **** 2021-12-15T17:08:20.878 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-12-15T17:08:20.880 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.065995", "end": "2021-12-15 17:08:20.850939", "rc": 0, "start": "2021-12-15 17:08:20.784944", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:20.902 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:20 +0000 (0:00:00.279) 0:04:02.611 **** 2021-12-15T17:08:20.908 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:20.909 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rhwfXe1wVyBMFGfnT737kx6hl0ViuVnK0'], 'uuids': ['62673dbe-635e-4b9d-be5f-e31d92faf6f6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.049209", "end": "2021-12-15 17:08:20.858053", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rhwfXe1wVyBMFGfnT737kx6hl0ViuVnK0"], "labels": [], "masters": [], "uuids": ["62673dbe-635e-4b9d-be5f-e31d92faf6f6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:19.808844", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:21.130 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-12-15T17:08:21.131 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048925", "end": "2021-12-15 17:08:21.113510", "rc": 0, "start": "2021-12-15 17:08:21.064585", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:21.153 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.251) 0:04:02.862 **** 2021-12-15T17:08:21.181 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-12-15T17:08:21.206 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.053) 0:04:02.915 **** 2021-12-15T17:08:21.234 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-12-15T17:08:21.236 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.262 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.056) 0:04:02.972 **** 2021-12-15T17:08:21.291 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-12-15T17:08:21.292 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.314 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.051) 0:04:03.023 **** 2021-12-15T17:08:21.342 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-12-15T17:08:21.343 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.368 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.053) 0:04:03.077 **** 2021-12-15T17:08:21.396 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-12-15T17:08:21.398 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.419 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.051) 0:04:03.129 **** 2021-12-15T17:08:21.605 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-12-15T17:08:21.606 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:08:21.630 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.210) 0:04:03.339 **** 2021-12-15T17:08:21.680 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi122.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930209841152, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227102012, 'block_used': 12993096, 'inode_total': 61054976, 'inode_available': 60989172, 'inode_used': 65804, 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227102012, "block_size": 4096, "block_total": 240095108, "block_used": 12993096, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989172, "inode_total": 61054976, "inode_used": 65804, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930209841152, "size_total": 983429562368, "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.703 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.073) 0:04:03.412 **** 2021-12-15T17:08:21.730 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-12-15T17:08:21.732 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.752 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.048) 0:04:03.461 **** 2021-12-15T17:08:21.789 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:21.790 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.810 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.058) 0:04:03.520 **** 2021-12-15T17:08:21.837 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:21.839 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.859 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.048) 0:04:03.568 **** 2021-12-15T17:08:21.886 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-12-15T17:08:21.888 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.909 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.049) 0:04:03.618 **** 2021-12-15T17:08:21.937 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-12-15T17:08:21.938 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:21.959 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:21 +0000 (0:00:00.049) 0:04:03.668 **** 2021-12-15T17:08:21.997 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-12-15T17:08:21.999 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:22.019 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:22 +0000 (0:00:00.059) 0:04:03.728 **** 2021-12-15T17:08:22.132 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:22.134 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2romRb6J91GlMyUUFr1pn0QfDkkSUAm43P'], 'uuids': ['2b9cdaf2-bcaf-4b31-bfcb-5f7145a8e51d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.044831", "end": "2021-12-15 17:08:22.100677", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2romRb6J91GlMyUUFr1pn0QfDkkSUAm43P"], "labels": [], "masters": [], "uuids": ["2b9cdaf2-bcaf-4b31-bfcb-5f7145a8e51d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:21.055846", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:22.531 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-12-15T17:08:22.533 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-12-15T17:08:22.554 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:22 +0000 (0:00:00.534) 0:04:04.263 **** 2021-12-15T17:08:23.021 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-12-15T17:08:23.023 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:23.388 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:23.482 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:23.484 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rDbFFl4ahhWPJlkvf4YGzXJcLvinMarNG'], 'uuids': ['64c5b507-0d9b-4b84-98c1-bc6de824d4b5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.146124", "end": "2021-12-15 17:08:23.449903", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rDbFFl4ahhWPJlkvf4YGzXJcLvinMarNG"], "labels": [], "masters": [], "uuids": ["64c5b507-0d9b-4b84-98c1-bc6de824d4b5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:22.303779", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi022.front.sepia.ceph.com] => 2021-12-15T17:08:23.486 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQJ7', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQJ7', 'scsi-35000c500871a6c97', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7', 'wwn-0x5000c500871a6c97'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HQJ7', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a6c97', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQJ7-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQJ7-part1', 'scsi-35000c500871a6c97-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1', 'wwn-0x5000c500871a6c97-part1'], 'uuids': ['ccecca0b-fa27-451f-b628-c766cfeb470c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQJ7", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQJ7", "scsi-35000c500871a6c97", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7", "wwn-0x5000c500871a6c97"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQJ7-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQJ7-part1", "scsi-35000c500871a6c97-part1 2021-12-15T17:08:23.488 INFO:teuthology.task.ansible.out:", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1", "wwn-0x5000c500871a6c97-part1"], "labels": [], "masters": [], "uuids": ["ccecca0b-fa27-451f-b628-c766cfeb470c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQJ7", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a6c97"}}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:23.715 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-12-15T17:08:23.717 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:24.074 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:24.439 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:24.462 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:24 +0000 (0:00:01.907) 0:04:06.171 **** 2021-12-15T17:08:24.658 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-12-15T17:08:24.681 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:24 +0000 (0:00:00.218) 0:04:06.390 **** 2021-12-15T17:08:24.730 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-12-15T17:08:24.732 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rYlMLsEjWCsATHi7DwczX4hJ1wMqrrNbX'], 'uuids': ['67d65591-cc83-4ae6-b129-e3463ae82f1b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.046739", "end": "2021-12-15 17:08:24.701538", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rYlMLsEjWCsATHi7DwczX4hJ1wMqrrNbX"], "labels": [], "masters": [], "uuids": ["67d65591-cc83-4ae6-b129-e3463ae82f1b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:23.654799", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-12-15T17:08:24.761 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:24 +0000 (0:00:00.079) 0:04:06.470 **** 2021-12-15T17:08:24.894 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-12-15T17:08:24.896 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:25.073 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:25.106 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-12-15T17:08:25.108 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rIGUDolLiyZNrs0Cr2T8gZqqcAnkfenAF'], 'uuids': ['eced2a54-7662-45c4-a6b8-31388f39953e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.099702", "end": "2021-12-15 17:08:25.068987", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rIGUDolLiyZNrs0Cr2T8gZqqcAnkfenAF"], "labels": [], "masters": [], "uuids": ["eced2a54-7662-45c4-a6b8-31388f39953e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:24.969285", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.096628 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.096628 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:25.244 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-12-15T17:08:25.246 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:25.318 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-12-15T17:08:25.320 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-92v0up-410f-qtpQ-b708-Pn2V-z9Ko-xjYFKH', 'nvme-INTEL_SSDPED1D480GA_PHMB747402EC480DGN', 'nvme-nvme.8086-50484d42373437343032454334383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039210", "end": "2021-12-15 17:08:25.293116", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-92v0up-410f-qtpQ-b708-Pn2V-z9Ko-xjYFKH", "nvme-INTEL_SSDPED1D480GA_PHMB747402EC480DGN", "nvme-nvme.8086-50484d42373437343032454334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:25.253906", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0362768 s, 3.2 GB/s", "stderr_lines": [" 2021-12-15T17:08:25.321 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0362768 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:25.423 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-12-15T17:08:25.425 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:25.435 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:25.464 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:25 +0000 (0:00:00.703) 0:04:07.173 **** 2021-12-15T17:08:25.492 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-12-15T17:08:25.493 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:25.515 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:25 +0000 (0:00:00.050) 0:04:07.224 **** 2021-12-15T17:08:25.552 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-12-15T17:08:25.554 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:25.575 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:25 +0000 (0:00:00.060) 0:04:07.284 **** 2021-12-15T17:08:25.603 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-12-15T17:08:25.605 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rhwfXe1wVyBMFGfnT737kx6hl0ViuVnK0'], 'uuids': ['62673dbe-635e-4b9d-be5f-e31d92faf6f6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.099464", "end": "2021-12-15 17:08:25.574748", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rhwfXe1wVyBMFGfnT737kx6hl0ViuVnK0"], "labels": [], "masters": [], "uuids": ["62673dbe-635e-4b9d-be5f-e31d92faf6f6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:25.475284", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0966333 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0966333 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:25.772 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-12-15T17:08:25.774 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:25.795 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:25 +0000 (0:00:00.219) 0:04:07.504 **** 2021-12-15T17:08:25.832 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-12-15T17:08:25.834 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-12-15T17:08:25.865 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:25 +0000 (0:00:00.070) 0:04:07.574 **** 2021-12-15T17:08:25.885 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-12-15T17:08:25.886 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2romRb6J91GlMyUUFr1pn0QfDkkSUAm43P'], 'uuids': ['2b9cdaf2-bcaf-4b31-bfcb-5f7145a8e51d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.099150", "end": "2021-12-15 17:08:25.856101", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2romRb6J91GlMyUUFr1pn0QfDkkSUAm43P"], "labels": [], "masters": [], "uuids": ["2b9cdaf2-bcaf-4b31-bfcb-5f7145a8e51d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:25.756951", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0963329 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0963329 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:26.159 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rDbFFl4ahhWPJlkvf4YGzXJcLvinMarNG'], 'uuids': ['64c5b507-0d9b-4b84-98c1-bc6de824d4b5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.100070", "end": "2021-12-15 17:08:26.139001", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rDbFFl4ahhWPJlkvf4YGzXJcLvinMarNG"], "labels": [], "masters": [], "uuids": ["64c5b507-0d9b-4b84-98c1-bc6de824d4b5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:26.038931", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0972435 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0972435 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:26.171 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQJ7', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQJ7', 'scsi-35000c500871a6c97', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7', 'wwn-0x5000c500871a6c97'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HQJ7', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a6c97', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQJ7-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQJ7-part1', 'scsi-35000c500871a6c97-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1', 'wwn-0x5000c500871a6c97-part1'], 'uuids': ['ccecca0b-fa27-451f-b628-c766cfeb470c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQJ7", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQJ7", "scsi-35000c500871a6c97", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7", "wwn-0x5000c500871a6c97"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQJ7-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-12-15T17:08:26.172 INFO:teuthology.task.ansible.out:Z1W4HQJ7-part1", "scsi-35000c500871a6c97-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQJ7-part1", "wwn-0x5000c500871a6c97-part1"], "labels": [], "masters": [], "uuids": ["ccecca0b-fa27-451f-b628-c766cfeb470c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQJ7", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a6c97"}}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:26.235 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-12-15T17:08:26.254 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true} 2021-12-15T17:08:26.261 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.396) 0:04:07.970 **** 2021-12-15T17:08:26.446 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-12-15T17:08:26.448 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rYlMLsEjWCsATHi7DwczX4hJ1wMqrrNbX'], 'uuids': ['67d65591-cc83-4ae6-b129-e3463ae82f1b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.098605", "end": "2021-12-15 17:08:26.425474", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lVWXXkzM8fYxvyQZzlHJ2l7ybAt9Ok2rYlMLsEjWCsATHi7DwczX4hJ1wMqrrNbX"], "labels": [], "masters": [], "uuids": ["67d65591-cc83-4ae6-b129-e3463ae82f1b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-12-15 17:08:26.326869", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0957477 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0957477 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:26.486 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.224) 0:04:08.195 **** 2021-12-15T17:08:26.661 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-12-15T17:08:26.663 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-12-15T17:08:26.685 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.198) 0:04:08.394 **** 2021-12-15T17:08:26.722 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-12-15T17:08:26.724 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:26.745 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.060) 0:04:08.454 **** 2021-12-15T17:08:26.751 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-12-15T17:08:26.753 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.057137", "end": "2021-12-15 17:08:26.714369", "rc": 0, "start": "2021-12-15 17:08:26.657232", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:26.776 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.030) 0:04:08.485 **** 2021-12-15T17:08:26.782 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-12-15T17:08:26.784 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:26.807 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.030) 0:04:08.516 **** 2021-12-15T17:08:26.834 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-12-15T17:08:26.836 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:26.857 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.050) 0:04:08.566 **** 2021-12-15T17:08:26.884 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:26.907 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.049) 0:04:08.616 **** 2021-12-15T17:08:26.934 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-12-15T17:08:26.936 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:26.957 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:26 +0000 (0:00:00.049) 0:04:08.666 **** 2021-12-15T17:08:26.984 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.007 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.050) 0:04:08.717 **** 2021-12-15T17:08:27.025 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-12-15T17:08:27.027 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.052119", "end": "2021-12-15 17:08:26.997552", "rc": 0, "start": "2021-12-15 17:08:26.945433", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:27.048 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.040) 0:04:08.757 **** 2021-12-15T17:08:27.055 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-12-15T17:08:27.057 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.081 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.032) 0:04:08.790 **** 2021-12-15T17:08:27.087 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-12-15T17:08:27.114 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.033) 0:04:08.823 **** 2021-12-15T17:08:27.121 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-12-15T17:08:27.123 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.145 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.031) 0:04:08.854 **** 2021-12-15T17:08:27.151 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-12-15T17:08:27.152 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.177 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.031) 0:04:08.886 **** 2021-12-15T17:08:27.183 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-12-15T17:08:27.185 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.207 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.030) 0:04:08.917 **** 2021-12-15T17:08:27.213 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-12-15T17:08:27.215 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.239 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.031) 0:04:08.948 **** 2021-12-15T17:08:27.245 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-12-15T17:08:27.247 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.268 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.029) 0:04:08.978 **** 2021-12-15T17:08:27.275 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-12-15T17:08:27.277 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.306 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.037) 0:04:09.016 **** 2021-12-15T17:08:27.314 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.337 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.031) 0:04:09.047 **** 2021-12-15T17:08:27.343 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-12-15T17:08:27.346 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.368 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.030) 0:04:09.077 **** 2021-12-15T17:08:27.374 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-12-15T17:08:27.376 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.399 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.031) 0:04:09.108 **** 2021-12-15T17:08:27.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-12-15T17:08:27.427 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.448 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.048) 0:04:09.157 **** 2021-12-15T17:08:27.475 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-12-15T17:08:27.476 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.498 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.049) 0:04:09.206 **** 2021-12-15T17:08:27.524 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.547 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.049) 0:04:09.256 **** 2021-12-15T17:08:27.574 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-12-15T17:08:27.576 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-12-15T17:08:27.598 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.051) 0:04:09.307 **** 2021-12-15T17:08:27.636 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-12-15T17:08:27.638 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930209787904, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227101999, 'block_used': 12993109, 'inode_total': 61054976, 'inode_available': 60989168, 'inode_used': 65808, 'uuid': 'ccecca0b-fa27-451f-b628-c766cfeb470c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227101999, "block_size": 4096, "block_total": 240095108, "block_used": 12993109, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989168, "inode_total": 61054976, "inode_used": 65808, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930209787904, "size_total": 983429562368, "uuid": "ccecca0b-fa27-451f-b628-c766cfeb470c"}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.662 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.064) 0:04:09.371 **** 2021-12-15T17:08:27.688 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-12-15T17:08:27.690 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.711 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.048) 0:04:09.420 **** 2021-12-15T17:08:27.738 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.761 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.049) 0:04:09.470 **** 2021-12-15T17:08:27.788 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:27.790 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.811 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.050) 0:04:09.520 **** 2021-12-15T17:08:27.839 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-12-15T17:08:27.841 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.861 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.050) 0:04:09.570 **** 2021-12-15T17:08:27.898 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-12-15T17:08:27.900 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.921 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.059) 0:04:09.630 **** 2021-12-15T17:08:27.958 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-12-15T17:08:27.960 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:27.980 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:27 +0000 (0:00:00.059) 0:04:09.689 **** 2021-12-15T17:08:28.039 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-12-15T17:08:28.041 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-12-15T17:08:28.066 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.085) 0:04:09.775 **** 2021-12-15T17:08:28.369 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-12-15T17:08:28.370 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-12-15 17:03:12 UTC", "ActiveEnterTimestampMonotonic": "14581451", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sntp.service systemd-journald.socket tmp.mount systemd-tmpfiles-setup.service basic.target ntpd.service system.slice ntpdate.service sysinit.target -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-12-15 17:03:10 UTC", "AssertTimestampMonotonic": "13239351", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-12-15 17:03:10 UTC", "ConditionTimestampMonotonic": "13239315", "ConfigurationDirectoryMode": "0755", "Conflicts": "syst 2021-12-15T17:08:28.371 INFO:teuthology.task.ansible.out:emd-timesyncd.service shutdown.target ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "927", "ExecMainStartTimestamp": "Wed 2021-12-15 17:03:12 UTC", "ExecMainStartTimestampMonotonic": "14445384", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper remove-daemon-state ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-12-15 17:03:10 UTC", "InactiveExitTimestampMonotonic": "13249705", "InvocationID": "f5d6262e75ca4c4285e0586a80addfd2", "JobRu 2021-12-15T17:08:28.373 INFO:teuthology.task.ansible.out:nningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "927", "MemoryAccounting": "yes", "MemoryCurrent": "4071424", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules" 2021-12-15T17:08:28.375 INFO:teuthology.task.ansible.out:: "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-12-15 17:03:12 UTC", "StateChangeTimestampMonotonic": "14581451", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-12-15 17:03:12 UTC", "WatchdogTimestampMonotonic": "14445400", "WatchdogUSec": "0"}} 2021-12-15T17:08:28.393 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.326) 0:04:10.102 **** 2021-12-15T17:08:28.409 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-12-15T17:08:28.411 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-12-15T17:08:28.433 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.039) 0:04:10.142 **** 2021-12-15T17:08:28.439 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:28.441 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-12-15T17:08:28.463 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.030) 0:04:10.172 **** 2021-12-15T17:08:28.501 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-12-15T17:08:28.523 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.060) 0:04:10.232 **** 2021-12-15T17:08:28.549 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-12-15T17:08:28.550 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:28.571 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.048) 0:04:10.281 **** 2021-12-15T17:08:28.597 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-12-15T17:08:28.599 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:28.619 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.047) 0:04:10.329 **** 2021-12-15T17:08:28.645 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-12-15T17:08:28.647 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:28.667 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.047) 0:04:10.376 **** 2021-12-15T17:08:28.818 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-12-15T17:08:28.820 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:28.871 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-12-15T17:08:28.873 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-12-15T17:08:28.893 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:28 +0000 (0:00:00.225) 0:04:10.602 **** 2021-12-15T17:08:29.149 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-12-15T17:08:29.150 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:29.368 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-12-15T17:08:29.370 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-12-15T17:08:29.391 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:29 +0000 (0:00:00.498) 0:04:11.100 **** 2021-12-15T17:08:29.482 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-12-15T17:08:29.484 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:29.826 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:30.165 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-12-15T17:08:30.187 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.795) 0:04:11.896 **** 2021-12-15T17:08:30.392 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-12-15T17:08:30.393 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-12-15T17:08:30.415 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.228) 0:04:12.125 **** 2021-12-15T17:08:30.537 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-12-15T17:08:30.539 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-12-15T17:08:30.561 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.145) 0:04:12.270 **** 2021-12-15T17:08:30.590 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-12-15T17:08:30.592 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:30.612 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.050) 0:04:12.321 **** 2021-12-15T17:08:30.639 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-12-15T17:08:30.641 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:30.662 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.050) 0:04:12.372 **** 2021-12-15T17:08:30.815 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:30.847 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-12-15T17:08:30.849 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-12-15T17:08:30.869 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.206) 0:04:12.578 **** 2021-12-15T17:08:30.896 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-12-15T17:08:30.898 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:30.919 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.049) 0:04:12.628 **** 2021-12-15T17:08:30.956 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-12-15T17:08:30.958 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-12-15T17:08:30.979 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:30 +0000 (0:00:00.059) 0:04:12.688 **** 2021-12-15T17:08:30.995 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-12-15T17:08:30.998 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:31.007 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.030 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.051) 0:04:12.739 **** 2021-12-15T17:08:31.183 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-12-15T17:08:31.184 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-12-15T17:08:31.194 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.221 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.190) 0:04:12.929 **** 2021-12-15T17:08:31.226 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-12-15T17:08:31.228 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-12-15T17:08:31.251 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.031) 0:04:12.960 **** 2021-12-15T17:08:31.258 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-12-15T17:08:31.259 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.283 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.030) 0:04:12.991 **** 2021-12-15T17:08:31.310 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-12-15T17:08:31.312 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.335 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.052) 0:04:13.044 **** 2021-12-15T17:08:31.461 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-12-15T17:08:31.463 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-12-15T17:08:31.498 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.163) 0:04:13.207 **** 2021-12-15T17:08:31.529 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-12-15T17:08:31.531 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.561 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.062) 0:04:13.270 **** 2021-12-15T17:08:31.581 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-12-15T17:08:31.583 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:31.606 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.044) 0:04:13.315 **** 2021-12-15T17:08:31.616 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-12-15T17:08:31.618 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.620 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.620 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.653 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.047) 0:04:13.362 **** 2021-12-15T17:08:31.662 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-12-15T17:08:31.663 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-12-15T17:08:31.697 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.043) 0:04:13.406 **** 2021-12-15T17:08:31.709 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.721 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.723 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.725 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.738 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.762 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.064) 0:04:13.471 **** 2021-12-15T17:08:31.800 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:31.801 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.824 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.062) 0:04:13.533 **** 2021-12-15T17:08:31.862 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.885 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.060) 0:04:13.594 **** 2021-12-15T17:08:31.912 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-12-15T17:08:31.914 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:31.944 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.059) 0:04:13.653 **** 2021-12-15T17:08:31.950 INFO:teuthology.task.ansible.out: 2021-12-15T17:08:31.952 INFO:teuthology.task.ansible.out:TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi022.front.sepia.ceph.com] => {"changed": true} 2021-12-15T17:08:31.975 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:31 +0000 (0:00:00.031) 0:04:13.685 **** 2021-12-15T17:08:31.982 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-12-15T17:08:31.984 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.007 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.031) 0:04:13.716 **** 2021-12-15T17:08:32.035 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-12-15T17:08:32.037 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.060 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.052) 0:04:13.769 **** 2021-12-15T17:08:32.087 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-12-15T17:08:32.089 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.113 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.052) 0:04:13.822 **** 2021-12-15T17:08:32.140 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.166 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.053) 0:04:13.875 **** 2021-12-15T17:08:32.193 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.223 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.057) 0:04:13.932 **** 2021-12-15T17:08:32.302 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-12-15T17:08:32.304 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-12-15T17:08:32.327 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.103) 0:04:14.036 **** 2021-12-15T17:08:32.354 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-12-15T17:08:32.356 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.388 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.060) 0:04:14.097 **** 2021-12-15T17:08:32.418 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-12-15T17:08:32.421 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.447 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.059) 0:04:14.156 **** 2021-12-15T17:08:32.454 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-12-15T17:08:32.456 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005228", "end": "2021-12-15 17:08:32.410771", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-12-15 17:08:32.405543", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:32.476 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-12-15T17:08:32.478 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.498 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.050) 0:04:14.207 **** 2021-12-15T17:08:32.535 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-12-15T17:08:32.537 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.564 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.066) 0:04:14.274 **** 2021-12-15T17:08:32.602 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-12-15T17:08:32.604 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.625 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.060) 0:04:14.334 **** 2021-12-15T17:08:32.633 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-12-15T17:08:32.635 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003131", "end": "2021-12-15 17:08:32.586300", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-12-15 17:08:32.583169", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:32.655 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.677 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.052) 0:04:14.386 **** 2021-12-15T17:08:32.704 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-12-15T17:08:32.705 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.726 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.049) 0:04:14.436 **** 2021-12-15T17:08:32.754 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-12-15T17:08:32.756 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.777 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.050) 0:04:14.486 **** 2021-12-15T17:08:32.795 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-12-15T17:08:32.796 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002877", "end": "2021-12-15 17:08:32.764225", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-12-15 17:08:32.761348", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:32.806 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.828 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.051) 0:04:14.538 **** 2021-12-15T17:08:32.855 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-12-15T17:08:32.857 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.878 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.049) 0:04:14.587 **** 2021-12-15T17:08:32.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-12-15T17:08:32.919 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.940 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.062) 0:04:14.649 **** 2021-12-15T17:08:32.968 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-12-15T17:08:32.970 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003289", "end": "2021-12-15 17:08:32.939458", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-12-15 17:08:32.936169", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:32.972 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:32.995 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:32 +0000 (0:00:00.055) 0:04:14.705 **** 2021-12-15T17:08:33.020 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:33 +0000 (0:00:00.024) 0:04:14.729 **** 2021-12-15T17:08:33.026 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-12-15T17:08:33.029 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:33.051 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:33 +0000 (0:00:00.031) 0:04:14.760 **** 2021-12-15T17:08:33.057 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-12-15T17:08:33.081 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:33 +0000 (0:00:00.030) 0:04:14.791 **** 2021-12-15T17:08:33.088 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-12-15T17:08:33.090 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:33.112 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:33 +0000 (0:00:00.030) 0:04:14.821 **** 2021-12-15T17:08:33.119 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-12-15T17:08:33.120 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:33.145 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-12-15T17:08:33.147 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:33.168 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:33 +0000 (0:00:00.055) 0:04:14.877 **** 2021-12-15T17:08:33.661 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-12-15T17:08:33.663 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-12-15T17:08:33.683 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:33 +0000 (0:00:00.515) 0:04:15.393 **** 2021-12-15T17:08:34.007 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-12-15T17:08:34.009 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-12-15 17:02:46 UTC", "ActiveEnterTimestampMonotonic": "14416096", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target sysinit.target system.slice ntpd.service sntp.service systemd-journald.socket ntpdate.service tmp.mount -.mount systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-12-15 17:02:45 UTC", "AssertTimestampMonotonic": "12703929", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-12-15 17:02:45 UTC", "ConditionTimestampMonotonic": "12703866", "ConfigurationDirectoryMode": "0755", "Conflicts": "shut 2021-12-15T17:08:34.010 INFO:teuthology.task.ansible.out:down.target ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "839", "ExecMainStartTimestamp": "Wed 2021-12-15 17:02:46 UTC", "ExecMainStartTimestampMonotonic": "14142502", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper remove-daemon-state ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-12-15 17:02:45 UTC", "InactiveExitTimestampMonotonic": "12742623", "InvocationID": "c881981d90af4bffaad06d9177bf1be4", "JobRu 2021-12-15T17:08:34.012 INFO:teuthology.task.ansible.out:nningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127268", "LimitNPROCSoft": "127268", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127268", "LimitSIGPENDINGSoft": "127268", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "839", "MemoryAccounting": "yes", "MemoryCurrent": "2646016", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules" 2021-12-15T17:08:34.014 INFO:teuthology.task.ansible.out:: "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-12-15 17:02:46 UTC", "StateChangeTimestampMonotonic": "14416096", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203628", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-12-15 17:02:46 UTC", "WatchdogTimestampMonotonic": "14142550", "WatchdogUSec": "0"}} 2021-12-15T17:08:34.032 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:34 +0000 (0:00:00.348) 0:04:15.741 **** 2021-12-15T17:08:34.060 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:34.062 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-12-15T17:08:34.086 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:34 +0000 (0:00:00.053) 0:04:15.795 **** 2021-12-15T17:08:34.132 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:34.134 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-12-15T17:08:34.154 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:34 +0000 (0:00:00.068) 0:04:15.863 **** 2021-12-15T17:08:34.182 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:34.204 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:34 +0000 (0:00:00.049) 0:04:15.913 **** 2021-12-15T17:08:34.230 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-12-15T17:08:34.232 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:34.252 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:34 +0000 (0:00:00.048) 0:04:15.961 **** 2021-12-15T17:08:34.280 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-12-15T17:08:34.281 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:34.312 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:34 +0000 (0:00:00.059) 0:04:16.021 **** 2021-12-15T17:08:34.529 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-12-15T17:08:34.530 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-12-15T17:08:34.552 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:34 +0000 (0:00:00.239) 0:04:16.261 **** 2021-12-15T17:08:35.049 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-12-15T17:08:35.051 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-12-15T17:08:35.079 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:35 +0000 (0:00:00.527) 0:04:16.788 **** 2021-12-15T17:08:36.159 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-12-15T17:08:36.160 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-12-15T17:08:36.180 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:01.101) 0:04:17.890 **** 2021-12-15T17:08:36.218 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-12-15T17:08:36.220 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:36.242 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:00.061) 0:04:17.951 **** 2021-12-15T17:08:36.269 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-12-15T17:08:36.271 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-12-15T17:08:36.291 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:00.049) 0:04:18.000 **** 2021-12-15T17:08:36.495 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-12-15T17:08:36.518 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:00.225) 0:04:18.226 **** 2021-12-15T17:08:36.544 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-12-15T17:08:36.546 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:36.566 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:00.049) 0:04:18.275 **** 2021-12-15T17:08:36.594 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-12-15T17:08:36.596 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-12-15T17:08:36.623 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:00.056) 0:04:18.332 **** 2021-12-15T17:08:36.650 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-12-15T17:08:36.652 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:36.672 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:00.049) 0:04:18.381 **** 2021-12-15T17:08:36.876 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-12-15T17:08:36.877 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-12-15T17:08:36.898 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:36 +0000 (0:00:00.225) 0:04:18.607 **** 2021-12-15T17:08:37.104 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-12-15T17:08:37.105 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-12-15T17:08:37.126 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.228) 0:04:18.835 **** 2021-12-15T17:08:37.153 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-12-15T17:08:37.155 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.174 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.048) 0:04:18.884 **** 2021-12-15T17:08:37.211 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-12-15T17:08:37.213 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.223 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.224 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.246 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.071) 0:04:18.956 **** 2021-12-15T17:08:37.285 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-12-15T17:08:37.287 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.296 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.298 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi022.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.308 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.333 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.085) 0:04:19.041 **** 2021-12-15T17:08:37.359 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-12-15T17:08:37.361 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.383 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.051) 0:04:19.092 **** 2021-12-15T17:08:37.410 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-12-15T17:08:37.412 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.433 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.050) 0:04:19.142 **** 2021-12-15T17:08:37.461 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-12-15T17:08:37.463 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.486 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.052) 0:04:19.195 **** 2021-12-15T17:08:37.513 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-12-15T17:08:37.515 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.536 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.050) 0:04:19.245 **** 2021-12-15T17:08:37.565 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-12-15T17:08:37.567 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.589 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.052) 0:04:19.298 **** 2021-12-15T17:08:37.617 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-12-15T17:08:37.619 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.641 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.052) 0:04:19.351 **** 2021-12-15T17:08:37.679 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-12-15T17:08:37.681 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.706 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.064) 0:04:19.415 **** 2021-12-15T17:08:37.734 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-12-15T17:08:37.735 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:37.756 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:37 +0000 (0:00:00.049) 0:04:19.465 **** 2021-12-15T17:08:37.982 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-12-15T17:08:37.984 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005544", "end": "2021-12-15 17:08:37.945686", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-12-15 17:08:37.940142", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:38.161 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003316", "end": "2021-12-15 17:08:38.132521", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-12-15 17:08:38.129205", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:38.338 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003081", "end": "2021-12-15 17:08:38.316733", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-12-15 17:08:38.313652", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:38.524 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003398", "end": "2021-12-15 17:08:38.495435", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-12-15 17:08:38.492037", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:38.548 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:38 +0000 (0:00:00.792) 0:04:20.257 **** 2021-12-15T17:08:38.576 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-12-15T17:08:38.599 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:38 +0000 (0:00:00.050) 0:04:20.308 **** 2021-12-15T17:08:38.626 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-12-15T17:08:38.627 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:38.693 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-12-15T17:08:38.710 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:38 +0000 (0:00:00.111) 0:04:20.420 **** 2021-12-15T17:08:38.730 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:38 +0000 (0:00:00.019) 0:04:20.439 **** 2021-12-15T17:08:39.376 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi022.front.sepia.ceph.com] 2021-12-15T17:08:39.405 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] 2021-12-15T17:08:39.430 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.700) 0:04:21.140 **** 2021-12-15T17:08:39.474 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.042) 0:04:21.183 **** 2021-12-15T17:08:39.482 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-12-15T17:08:39.484 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:39.506 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.032) 0:04:21.215 **** 2021-12-15T17:08:39.514 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:39.540 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.034) 0:04:21.249 **** 2021-12-15T17:08:39.547 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-12-15T17:08:39.548 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-12-15T17:08:39.572 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.031) 0:04:21.281 **** 2021-12-15T17:08:39.578 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-12-15T17:08:39.602 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.030) 0:04:21.312 **** 2021-12-15T17:08:39.621 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-12-15T17:08:39.623 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-12-15T17:08:39.644 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.041) 0:04:21.354 **** 2021-12-15T17:08:39.652 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-12-15T17:08:39.677 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:39 +0000 (0:00:00.032) 0:04:21.386 **** 2021-12-15T17:08:41.146 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-12-15T17:08:41.148 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:41.170 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:01.493) 0:04:22.879 **** 2021-12-15T17:08:41.177 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:41.201 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:00.030) 0:04:22.910 **** 2021-12-15T17:08:41.207 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-12-15T17:08:41.209 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-12-15T17:08:41.232 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:00.030) 0:04:22.941 **** 2021-12-15T17:08:41.239 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-12-15T17:08:41.262 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:00.030) 0:04:22.971 **** 2021-12-15T17:08:41.269 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-12-15T17:08:41.270 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:41.292 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:00.030) 0:04:23.002 **** 2021-12-15T17:08:41.299 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:41.325 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:00.031) 0:04:23.034 **** 2021-12-15T17:08:41.523 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-12-15T17:08:41.525 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:41.557 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:41.704 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:41.731 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:00.406) 0:04:23.440 **** 2021-12-15T17:08:41.750 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-12-15T17:08:41.775 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:41 +0000 (0:00:00.044) 0:04:23.484 **** 2021-12-15T17:08:42.211 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-12-15T17:08:42.213 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-12-15T17:08:42.276 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-12-15T17:08:42.652 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-12-15T17:08:42.681 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:42 +0000 (0:00:00.905) 0:04:24.390 **** 2021-12-15T17:08:42.720 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-12-15T17:08:42.747 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:42 +0000 (0:00:00.066) 0:04:24.456 **** 2021-12-15T17:08:44.114 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-12-15T17:08:44.116 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:44.137 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:44 +0000 (0:00:01.389) 0:04:25.846 **** 2021-12-15T17:08:44.260 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-12-15T17:08:44.284 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:44 +0000 (0:00:00.147) 0:04:25.994 **** 2021-12-15T17:08:46.663 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-12-15T17:08:46.665 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-gy47xh05-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-gy47xh05-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-12-15T17:08:46.667 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-12-15T17:08:46.687 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:46 +0000 (0:00:02.402) 0:04:28.396 **** 2021-12-15T17:08:46.725 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-12-15T17:08:46.727 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:46.750 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:46 +0000 (0:00:00.062) 0:04:28.459 **** 2021-12-15T17:08:46.787 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-12-15T17:08:46.788 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-2zkts_8g-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-2zkts_8g-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-12-15T17:08:46.791 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:46.810 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:46 +0000 (0:00:00.060) 0:04:28.520 **** 2021-12-15T17:08:46.831 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:46 +0000 (0:00:00.020) 0:04:28.540 **** 2021-12-15T17:08:46.849 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-12-15T17:08:46.851 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:46.874 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:46 +0000 (0:00:00.043) 0:04:28.584 **** 2021-12-15T17:08:46.902 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-12-15T17:08:46.904 INFO:teuthology.task.ansible.out:skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:46.925 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:46 +0000 (0:00:00.049) 0:04:28.633 **** 2021-12-15T17:08:47.173 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-12-15T17:08:47.175 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.148376", "end": "2021-12-15 17:08:47.157661", "rc": 0, "start": "2021-12-15 17:08:47.009285", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:47.198 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:47 +0000 (0:00:00.274) 0:04:28.907 **** 2021-12-15T17:08:47.237 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-12-15T17:08:47.239 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:47.302 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.169104", "end": "2021-12-15 17:08:47.282478", "rc": 0, "start": "2021-12-15 17:08:47.113374", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-12-15T17:08:47.324 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:47 +0000 (0:00:00.125) 0:04:29.033 **** 2021-12-15T17:08:47.351 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi022.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-12-15T17:08:47.418 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-12-15T17:08:47.421 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-12-15T17:08:47.424 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-12-15T17:08:47.434 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-12-15T17:08:47.436 INFO:teuthology.task.ansible.out:Wednesday 15 December 2021 17:08:47 +0000 (0:00:00.110) 0:04:29.144 **** 2021-12-15T17:08:48.094 INFO:teuthology.task.ansible.out:ok: [smithi022.front.sepia.ceph.com] 2021-12-15T17:08:48.114 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] 2021-12-15T17:08:48.132 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Wednesday 15 December 2021 17:08:48 +0000 (0:00:00.697) 0:04:29.841 **** 2021-12-15T17:08:48.337 INFO:teuthology.task.ansible.out:changed: [smithi022.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-12-15T17:08:48.359 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-12-15T17:08:48.361 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-12-15T17:08:48.363 INFO:teuthology.task.ansible.out:smithi022.front.sepia.ceph.com : ok=137 changed=38 unreachable=0 failed=0 skipped=182 rescued=0 ignored=3 smithi122.front.sepia.ceph.com : ok=137 changed=38 unreachable=0 failed=0 skipped=182 rescued=0 ignored=3 Wednesday 15 December 2021 17:08:48 +0000 (0:00:00.229) 0:04:30.071 **** =============================================================================== common : nrpe - Ensure SELinux policy is up to date -------------------- 35.40s users : Update authorized_keys using the keys repo --------------------- 34.98s Create all admin users with sudo access. ------------------------------- 33.45s common : nrpe - Remove SELinux policy package -------------------------- 10.89s common : nrpe - Load SELinux policy package ---------------------------- 10.27s Remove revoked users ---------------------------------------------------- 7.48s testnode : Install packages --------------------------------------------- 4.78s testnode : Ensure ceph packages are not present. ------------------------ 2.81s testnode : Ensure ceph dependency packages are not present. ------------- 2.60s container-host : Install registries-conf-ctl ---------------------------- 2.40s users : Update authorized_keys for each user with literal keys ---------- 2.01s testnode : Zap all non-root disks --------------------------------------- 1.86s testnode : Make sure rpm dependencies are installed --------------------- 1.60s common : nrpe - Install SELinux dependencies ---------------------------- 1.58s container-host : Install container packages ----------------------------- 1.49s Gathering Facts --------------------------------------------------------- 1.45s testnode : Configure local mirrorlists ---------------------------------- 1.44s container-host : Ensure git is installed -------------------------------- 1.39s 2021-12-15T17:08:48.363 INFO:teuthology.task.ansible.out:users : Clone the keys repo --------------------------------------------- 1.31s testnode : Rebuild rpmdb ------------------------ 2021-12-15T17:08:48.365 INFO:teuthology.task.ansible.out:------------------------ 1.30s 2021-12-15T17:08:48.758 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2021-12-15T17:08:48.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2021-12-15T17:08:48.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-12-15T17:08:48.836 DEBUG:teuthology.orchestra.run.smithi022:> true 2021-12-15T17:08:48.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2021-12-15T17:08:48.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-12-15T17:08:48.985 DEBUG:teuthology.orchestra.run.smithi122:> true 2021-12-15T17:08:49.061 INFO:teuthology.run_tasks:Running task clock... 2021-12-15T17:08:49.075 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-12-15T17:08:49.076 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-12-15T17:08:49.078 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-12-15T17:08:49.081 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-12-15T17:08:49.081 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-12-15T17:08:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-12-15T17:08:49.129 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-12-15T17:08:49.129 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-12-15T17:08:49.143 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-12-15T17:08:49.195 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2021-12-15T17:08:49.206 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2021-12-15T17:08:49.208 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2021-12-15T17:08:49.218 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2021-12-15T17:08:49.223 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-12-15T17:08:49.232 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-12-15T17:08:49.237 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-12-15T17:08:49.245 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-12-15T17:08:49.317 INFO:teuthology.orchestra.run.smithi022.stderr:bash: ntpq: command not found 2021-12-15T17:08:49.320 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-12-15T17:08:49.320 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2021-12-15T17:08:49.320 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.321 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.321 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.321 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.324 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2021-12-15T17:08:49.328 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-12-15T17:08:49.329 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2021-12-15T17:08:49.329 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.329 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.330 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.330 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-12-15T17:08:49.331 INFO:teuthology.run_tasks:Running task pexec... 2021-12-15T17:08:49.344 INFO:teuthology.task.pexec:Executing custom commands... 2021-12-15T17:08:49.345 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi022.front.sepia.ceph.com 2021-12-15T17:08:49.346 DEBUG:teuthology.orchestra.run.smithi022:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-12-15T17:08:49.346 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi122.front.sepia.ceph.com 2021-12-15T17:08:49.346 DEBUG:teuthology.orchestra.run.smithi122:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-12-15T17:08:49.587 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:49.587 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:49.589 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:49.590 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:49.921 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:09 ago on Wed 15 Dec 2021 05:07:40 PM UTC. 2021-12-15T17:08:49.942 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:02 ago on Wed 15 Dec 2021 05:07:47 PM UTC. 2021-12-15T17:08:50.825 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:08:50.826 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:08:50.826 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2021-12-15T17:08:50.826 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:08:50.827 INFO:teuthology.orchestra.run.smithi122.stdout:Resetting modules: 2021-12-15T17:08:50.827 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools 2021-12-15T17:08:50.827 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:08:50.827 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:08:50.828 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:08:50.828 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:08:50.919 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:08:50.920 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:08:50.920 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2021-12-15T17:08:50.921 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:08:50.921 INFO:teuthology.orchestra.run.smithi022.stdout:Resetting modules: 2021-12-15T17:08:50.921 INFO:teuthology.orchestra.run.smithi022.stdout: container-tools 2021-12-15T17:08:50.921 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:08:50.922 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:08:50.922 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:08:50.922 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:08:51.014 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:08:51.120 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:08:51.258 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:51.259 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:51.373 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:51.374 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:51.592 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:11 ago on Wed 15 Dec 2021 05:07:40 PM UTC. 2021-12-15T17:08:51.727 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:04 ago on Wed 15 Dec 2021 05:07:47 PM UTC. 2021-12-15T17:08:52.515 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:08:52.516 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2021-12-15T17:08:52.517 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:08:52.517 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2021-12-15T17:08:52.517 INFO:teuthology.orchestra.run.smithi122.stdout:Installing group/module packages: 2021-12-15T17:08:52.517 INFO:teuthology.orchestra.run.smithi122.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-12-15T17:08:52.518 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-12-15T17:08:52.518 INFO:teuthology.orchestra.run.smithi122.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-12-15T17:08:52.518 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-12-15T17:08:52.518 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-12-15T17:08:52.518 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-12-15T17:08:52.519 INFO:teuthology.orchestra.run.smithi122.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-12-15T17:08:52.519 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:08:52.519 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-12-15T17:08:52.519 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-12-15T17:08:52.520 INFO:teuthology.orchestra.run.smithi122.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-12-15T17:08:52.520 INFO:teuthology.orchestra.run.smithi122.stdout:Installing module profiles: 2021-12-15T17:08:52.520 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools/common 2021-12-15T17:08:52.520 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:08:52.521 INFO:teuthology.orchestra.run.smithi122.stdout:Enabling module streams: 2021-12-15T17:08:52.521 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools rhel8 2021-12-15T17:08:52.521 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:08:52.521 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:08:52.521 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2021-12-15T17:08:52.522 INFO:teuthology.orchestra.run.smithi122.stdout:Install 10 Packages 2021-12-15T17:08:52.522 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:08:52.523 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 18 M 2021-12-15T17:08:52.523 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 64 M 2021-12-15T17:08:52.523 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:08:52.638 INFO:teuthology.orchestra.run.smithi122.stdout:(1/10): crun-1.2-1.module_el8.6.0+954+963caf36. 1.6 MB/s | 193 kB 00:00 2021-12-15T17:08:52.663 INFO:teuthology.orchestra.run.smithi122.stdout:(2/10): cockpit-podman-35-1.module_el8.6.0+944+ 3.1 MB/s | 457 kB 00:00 2021-12-15T17:08:52.689 INFO:teuthology.orchestra.run.smithi122.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 991 kB/s | 25 kB 00:00 2021-12-15T17:08:52.714 INFO:teuthology.orchestra.run.smithi122.stdout:(4/10): python3-podman-3.2.1-4.module_el8.6.0+9 1.9 MB/s | 148 kB 00:00 2021-12-15T17:08:52.715 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:08:52.716 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2021-12-15T17:08:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:08:52.716 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2021-12-15T17:08:52.716 INFO:teuthology.orchestra.run.smithi022.stdout:Installing group/module packages: 2021-12-15T17:08:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-12-15T17:08:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-12-15T17:08:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-12-15T17:08:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-12-15T17:08:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-12-15T17:08:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-12-15T17:08:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-12-15T17:08:52.718 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:08:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-12-15T17:08:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-12-15T17:08:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-12-15T17:08:52.719 INFO:teuthology.orchestra.run.smithi022.stdout:Installing module profiles: 2021-12-15T17:08:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: container-tools/common 2021-12-15T17:08:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:08:52.720 INFO:teuthology.orchestra.run.smithi022.stdout:Enabling module streams: 2021-12-15T17:08:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: container-tools rhel8 2021-12-15T17:08:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:08:52.720 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:08:52.721 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2021-12-15T17:08:52.721 INFO:teuthology.orchestra.run.smithi022.stdout:Install 10 Packages 2021-12-15T17:08:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:08:52.722 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 18 M 2021-12-15T17:08:52.722 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 64 M 2021-12-15T17:08:52.722 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:08:52.740 INFO:teuthology.orchestra.run.smithi122.stdout:(5/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2021-12-15T17:08:52.826 INFO:teuthology.orchestra.run.smithi022.stdout:(1/10): crun-1.2-1.module_el8.6.0+954+963caf36. 1.8 MB/s | 193 kB 00:00 2021-12-15T17:08:52.852 INFO:teuthology.orchestra.run.smithi022.stdout:(2/10): python3-podman-3.2.1-4.module_el8.6.0+9 5.8 MB/s | 148 kB 00:00 2021-12-15T17:08:52.885 INFO:teuthology.orchestra.run.smithi022.stdout:(3/10): cockpit-podman-35-1.module_el8.6.0+944+ 2.7 MB/s | 457 kB 00:00 2021-12-15T17:08:52.906 INFO:teuthology.orchestra.run.smithi122.stdout:(6/10): buildah-1.23.1-2.module_el8.6.0+954+963 20 MB/s | 7.9 MB 00:00 2021-12-15T17:08:52.910 INFO:teuthology.orchestra.run.smithi022.stdout:(4/10): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2021-12-15T17:08:52.936 INFO:teuthology.orchestra.run.smithi022.stdout:(5/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2021-12-15T17:08:52.957 INFO:teuthology.orchestra.run.smithi122.stdout:(7/10): udica-0.2.6-2.module_el8.6.0+944+d413f9 972 kB/s | 48 kB 00:00 2021-12-15T17:08:53.007 INFO:teuthology.orchestra.run.smithi122.stdout:(8/10): yajl-2.1.0-10.el8.x86_64.rpm 819 kB/s | 41 kB 00:00 2021-12-15T17:08:53.019 INFO:teuthology.orchestra.run.smithi022.stdout:(6/10): toolbox-0.0.99.3-1.module_el8.5.0+877+1 27 MB/s | 2.3 MB 00:00 2021-12-15T17:08:53.044 INFO:teuthology.orchestra.run.smithi022.stdout:(7/10): udica-0.2.6-2.module_el8.6.0+944+d413f9 1.9 MB/s | 48 kB 00:00 2021-12-15T17:08:53.065 INFO:teuthology.orchestra.run.smithi122.stdout:(9/10): toolbox-0.0.99.3-1.module_el8.5.0+877+1 6.9 MB/s | 2.3 MB 00:00 2021-12-15T17:08:53.070 INFO:teuthology.orchestra.run.smithi022.stdout:(8/10): yajl-2.1.0-10.el8.x86_64.rpm 1.6 MB/s | 41 kB 00:00 2021-12-15T17:08:53.291 INFO:teuthology.orchestra.run.smithi122.stdout:(10/10): skopeo-1.5.0-2.module_el8.6.0+954+963c 12 MB/s | 6.7 MB 00:00 2021-12-15T17:08:53.293 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:08:53.293 INFO:teuthology.orchestra.run.smithi122.stdout:Total 23 MB/s | 18 MB 00:00 2021-12-15T17:08:53.412 INFO:teuthology.orchestra.run.smithi022.stdout:(9/10): skopeo-1.5.0-2.module_el8.6.0+954+963ca 13 MB/s | 6.7 MB 00:00 2021-12-15T17:08:53.488 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:08:53.506 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:08:53.506 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:08:53.520 INFO:teuthology.orchestra.run.smithi022.stdout:(10/10): buildah-1.23.1-2.module_el8.6.0+954+96 9.8 MB/s | 7.9 MB 00:00 2021-12-15T17:08:53.523 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:08:53.523 INFO:teuthology.orchestra.run.smithi022.stdout:Total 22 MB/s | 18 MB 00:00 2021-12-15T17:08:53.726 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:08:53.746 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:08:53.746 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:08:53.771 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:08:53.774 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:08:54.032 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:08:54.035 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:08:54.263 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:08:54.415 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/10 2021-12-15T17:08:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:08:54.560 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/10 2021-12-15T17:08:54.711 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/10 2021-12-15T17:08:54.726 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2021-12-15T17:08:54.855 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/10 2021-12-15T17:08:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2021-12-15T17:08:54.998 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 5/10 2021-12-15T17:08:55.065 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2021-12-15T17:08:55.197 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2021-12-15T17:08:55.317 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 5/10 2021-12-15T17:08:55.361 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 6/10 2021-12-15T17:08:55.671 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 6/10 2021-12-15T17:08:56.077 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 7/10 2021-12-15T17:08:56.257 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 8/10 2021-12-15T17:08:56.424 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 7/10 2021-12-15T17:08:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 8/10 2021-12-15T17:08:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 9/10 2021-12-15T17:08:57.262 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2021-12-15T17:08:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 9/10 2021-12-15T17:08:57.508 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2021-12-15T17:08:57.508 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/10 2021-12-15T17:08:57.508 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/10 2021-12-15T17:08:57.509 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/10 2021-12-15T17:08:57.509 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2021-12-15T17:08:57.509 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/10 2021-12-15T17:08:57.509 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/10 2021-12-15T17:08:57.509 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/10 2021-12-15T17:08:57.510 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/10 2021-12-15T17:08:57.510 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/10 2021-12-15T17:08:57.659 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2021-12-15T17:08:57.884 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/10 2021-12-15T17:08:57.884 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:08:57.885 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:08:57.885 INFO:teuthology.orchestra.run.smithi122.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-12-15T17:08:57.885 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-12-15T17:08:57.885 INFO:teuthology.orchestra.run.smithi122.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-12-15T17:08:57.886 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-12-15T17:08:57.886 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-12-15T17:08:57.886 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-12-15T17:08:57.886 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-12-15T17:08:57.887 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-12-15T17:08:57.887 INFO:teuthology.orchestra.run.smithi122.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-12-15T17:08:57.887 INFO:teuthology.orchestra.run.smithi122.stdout: yajl-2.1.0-10.el8.x86_64 2021-12-15T17:08:57.887 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:08:57.890 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:08:57.919 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2021-12-15T17:08:57.920 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/10 2021-12-15T17:08:57.920 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/10 2021-12-15T17:08:57.920 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/10 2021-12-15T17:08:57.920 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2021-12-15T17:08:57.921 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/10 2021-12-15T17:08:57.921 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/10 2021-12-15T17:08:57.921 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/10 2021-12-15T17:08:57.921 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/10 2021-12-15T17:08:57.921 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/10 2021-12-15T17:08:58.146 DEBUG:teuthology.parallel:result is None 2021-12-15T17:08:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/10 2021-12-15T17:08:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:08:58.337 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:08:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-12-15T17:08:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-12-15T17:08:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-12-15T17:08:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-12-15T17:08:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-12-15T17:08:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-12-15T17:08:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-12-15T17:08:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-12-15T17:08:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-12-15T17:08:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: yajl-2.1.0-10.el8.x86_64 2021-12-15T17:08:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:08:58.340 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:08:58.539 DEBUG:teuthology.parallel:result is None 2021-12-15T17:08:58.540 INFO:teuthology.run_tasks:Running task install... 2021-12-15T17:08:58.551 DEBUG:teuthology.task.install:project ceph 2021-12-15T17:08:58.552 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2'}, 'extra_packages': ['cephadm']} 2021-12-15T17:08:58.552 DEBUG:teuthology.task.install:config {'sha1': 'aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2'} 2021-12-15T17:08:58.552 INFO:teuthology.task.install:Using flavor: default 2021-12-15T17:08:58.559 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-12-15T17:08:58.560 INFO:teuthology.task.install:extra packages: [] 2021-12-15T17:08:58.560 DEBUG:teuthology.orchestra.run.smithi022:> sudo lsb_release -is 2021-12-15T17:08:58.595 INFO:teuthology.orchestra.run.smithi022.stdout:CentOSStream 2021-12-15T17:08:58.597 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-12-15T17:08:58.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo lsb_release -is 2021-12-15T17:08:58.598 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2', 'tag': None, 'wait_for_package': False} 2021-12-15T17:08:58.598 DEBUG:teuthology.orchestra.run.smithi022:> sudo lsb_release -is 2021-12-15T17:08:58.633 INFO:teuthology.orchestra.run.smithi122.stdout:CentOSStream 2021-12-15T17:08:58.634 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-12-15T17:08:58.635 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2', 'tag': None, 'wait_for_package': False} 2021-12-15T17:08:58.635 DEBUG:teuthology.orchestra.run.smithi122:> sudo lsb_release -is 2021-12-15T17:08:58.671 INFO:teuthology.orchestra.run.smithi022.stdout:CentOSStream 2021-12-15T17:08:58.672 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-12-15T17:08:58.673 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 2021-12-15T17:08:58.709 INFO:teuthology.orchestra.run.smithi122.stdout:CentOSStream 2021-12-15T17:08:58.711 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-12-15T17:08:58.712 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 2021-12-15T17:08:58.945 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/ 2021-12-15T17:08:58.945 INFO:teuthology.task.install.rpm:Package version is 17.0.0-9665.gaa33f58d 2021-12-15T17:08:58.958 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/ 2021-12-15T17:08:58.958 INFO:teuthology.task.install.rpm:Package version is 17.0.0-9665.gaa33f58d 2021-12-15T17:08:59.116 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-12-15T17:08:59.116 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:08:59.116 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-12-15T17:08:59.126 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-12-14-1714/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-12-15T17:08:59.127 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:08:59.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-12-15T17:08:59.145 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-12-15T17:08:59.145 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-12-15T17:08:59.154 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-12-15T17:08:59.154 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-12-15T17:08:59.214 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-12-15T17:08:59.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-12-15T17:08:59.292 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-12-15T17:08:59.294 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-12-15T17:08:59.358 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2021-12-15T17:08:59.359 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2021-12-15T17:08:59.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2021-12-15T17:08:59.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2021-12-15T17:08:59.577 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:59.578 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:59.585 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:59.585 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:59.599 INFO:teuthology.orchestra.run.smithi122.stdout:72 files removed 2021-12-15T17:08:59.608 INFO:teuthology.orchestra.run.smithi022.stdout:72 files removed 2021-12-15T17:08:59.620 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2021-12-15T17:08:59.629 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw 2021-12-15T17:08:59.794 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:59.795 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:59.814 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:08:59.815 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:09:00.334 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 187 kB/s | 75 kB 00:00 2021-12-15T17:09:00.355 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 192 kB/s | 75 kB 00:00 2021-12-15T17:09:00.731 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 27 kB/s | 9.1 kB 00:00 2021-12-15T17:09:00.743 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2021-12-15T17:09:01.098 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 4.1 kB/s | 1.4 kB 00:00 2021-12-15T17:09:01.111 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 4.1 kB/s | 1.4 kB 00:00 2021-12-15T17:09:01.658 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - AppStream 34 MB/s | 18 MB 00:00 2021-12-15T17:09:01.746 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - AppStream 29 MB/s | 18 MB 00:00 2021-12-15T17:09:05.380 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - Base 31 MB/s | 16 MB 00:00 2021-12-15T17:09:05.626 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - Base 26 MB/s | 16 MB 00:00 2021-12-15T17:09:07.729 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - Extras 133 kB/s | 16 kB 00:00 2021-12-15T17:09:08.073 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-8 - PowerTools 16 MB/s | 3.8 MB 00:00 2021-12-15T17:09:08.125 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - Extras 136 kB/s | 16 kB 00:00 2021-12-15T17:09:08.460 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-8 - PowerTools 17 MB/s | 3.8 MB 00:00 2021-12-15T17:09:11.327 INFO:teuthology.orchestra.run.smithi022.stdout:Copr repo for python3-asyncssh owned by ceph 1.7 kB/s | 3.5 kB 00:02 2021-12-15T17:09:11.348 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2021-12-15T17:09:12.757 INFO:teuthology.orchestra.run.smithi122.stdout:Apache Arrow 141 kB/s | 184 kB 00:01 2021-12-15T17:09:12.777 INFO:teuthology.orchestra.run.smithi022.stdout:Apache Arrow 137 kB/s | 184 kB 00:01 2021-12-15T17:09:14.073 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 11 MB 00:01 2021-12-15T17:09:14.135 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 11 MB 00:01 2021-12-15T17:09:16.866 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 311 kB/s | 24 kB 00:00 2021-12-15T17:09:16.894 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 524 kB/s | 24 kB 00:00 2021-12-15T17:09:19.324 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:09:19.327 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:09:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:09:19.327 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:09:19.327 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:09:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 9.0 M 2021-12-15T17:09:19.328 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2021-12-15T17:09:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.4 M 2021-12-15T17:09:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.9 M 2021-12-15T17:09:19.329 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:09:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 5.2 M 2021-12-15T17:09:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 20 M 2021-12-15T17:09:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 24 k 2021-12-15T17:09:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-12-15T17:09:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 785 k 2021-12-15T17:09:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-12-15T17:09:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-12-15T17:09:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 482 k 2021-12-15T17:09:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-12-15T17:09:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.9 M 2021-12-15T17:09:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-12-15T17:09:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-12-15T17:09:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse 2021-12-15T17:09:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 45 k 2021-12-15T17:09:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 98 k 2021-12-15T17:09:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 178 k 2021-12-15T17:09:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-12-15T17:09:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 367 k 2021-12-15T17:09:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 345 k 2021-12-15T17:09:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 110 k 2021-12-15T17:09:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:09:19.333 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:09:19.334 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:09:19.334 INFO:teuthology.orchestra.run.smithi122.stdout:Install 20 Packages 2021-12-15T17:09:19.334 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2021-12-15T17:09:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:09:19.335 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 49 M 2021-12-15T17:09:19.335 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:09:19.488 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:09:19.491 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:09:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:09:19.491 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:09:19.492 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:09:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 9.0 M 2021-12-15T17:09:19.492 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2021-12-15T17:09:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.4 M 2021-12-15T17:09:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.9 M 2021-12-15T17:09:19.493 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:09:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 5.2 M 2021-12-15T17:09:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 20 M 2021-12-15T17:09:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 24 k 2021-12-15T17:09:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-12-15T17:09:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 785 k 2021-12-15T17:09:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-12-15T17:09:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-12-15T17:09:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 482 k 2021-12-15T17:09:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-12-15T17:09:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.9 M 2021-12-15T17:09:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-12-15T17:09:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-12-15T17:09:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse 2021-12-15T17:09:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 45 k 2021-12-15T17:09:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 98 k 2021-12-15T17:09:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 178 k 2021-12-15T17:09:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-12-15T17:09:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 367 k 2021-12-15T17:09:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 345 k 2021-12-15T17:09:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 110 k 2021-12-15T17:09:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:09:19.497 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:09:19.498 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:09:19.498 INFO:teuthology.orchestra.run.smithi022.stdout:Install 20 Packages 2021-12-15T17:09:19.498 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 2 Packages 2021-12-15T17:09:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:09:19.499 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 49 M 2021-12-15T17:09:19.499 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:09:20.142 INFO:teuthology.orchestra.run.smithi122.stdout:(1/22): ceph-base-17.0.0-9665.gaa33f58d.el8.x86 6.5 MB/s | 5.2 MB 00:00 2021-12-15T17:09:20.242 INFO:teuthology.orchestra.run.smithi122.stdout:(2/22): ceph-selinux-17.0.0-9665.gaa33f58d.el8. 244 kB/s | 24 kB 00:00 2021-12-15T17:09:21.017 INFO:teuthology.orchestra.run.smithi122.stdout:(3/22): ceph-radosgw-17.0.0-9665.gaa33f58d.el8. 5.3 MB/s | 9.0 MB 00:01 2021-12-15T17:09:21.051 INFO:teuthology.orchestra.run.smithi122.stdout:(4/22): libcephfs2-17.0.0-9665.gaa33f58d.el8.x8 971 kB/s | 785 kB 00:00 2021-12-15T17:09:21.101 INFO:teuthology.orchestra.run.smithi122.stdout:(5/22): libradosstriper1-17.0.0-9665.gaa33f58d. 5.6 MB/s | 482 kB 00:00 2021-12-15T17:09:21.160 INFO:teuthology.orchestra.run.smithi122.stdout:(6/22): python3-ceph-argparse-17.0.0-9665.gaa33 766 kB/s | 45 kB 00:00 2021-12-15T17:09:21.235 INFO:teuthology.orchestra.run.smithi122.stdout:(7/22): python3-ceph-common-17.0.0-9665.gaa33f5 1.3 MB/s | 98 kB 00:00 2021-12-15T17:09:21.344 INFO:teuthology.orchestra.run.smithi122.stdout:(8/22): python3-cephfs-17.0.0-9665.gaa33f58d.el 1.6 MB/s | 178 kB 00:00 2021-12-15T17:09:21.444 INFO:teuthology.orchestra.run.smithi122.stdout:(9/22): python3-rados-17.0.0-9665.gaa33f58d.el8 3.6 MB/s | 367 kB 00:00 2021-12-15T17:09:21.528 INFO:teuthology.orchestra.run.smithi122.stdout:(10/22): python3-rbd-17.0.0-9665.gaa33f58d.el8. 4.1 MB/s | 345 kB 00:00 2021-12-15T17:09:21.528 INFO:teuthology.orchestra.run.smithi022.stdout:(1/22): ceph-base-17.0.0-9665.gaa33f58d.el8.x86 2.6 MB/s | 5.2 MB 00:02 2021-12-15T17:09:21.611 INFO:teuthology.orchestra.run.smithi022.stdout:(2/22): ceph-selinux-17.0.0-9665.gaa33f58d.el8. 293 kB/s | 24 kB 00:00 2021-12-15T17:09:21.623 INFO:teuthology.orchestra.run.smithi122.stdout:(11/22): python3-rgw-17.0.0-9665.gaa33f58d.el8. 1.1 MB/s | 110 kB 00:00 2021-12-15T17:09:21.687 INFO:teuthology.orchestra.run.smithi122.stdout:(12/22): librdkafka-0.11.4-3.el8.x86_64.rpm 5.5 MB/s | 354 kB 00:00 2021-12-15T17:09:21.704 INFO:teuthology.orchestra.run.smithi122.stdout:(13/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2021-12-15T17:09:21.729 INFO:teuthology.orchestra.run.smithi122.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2021-12-15T17:09:21.754 INFO:teuthology.orchestra.run.smithi122.stdout:(15/22): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2021-12-15T17:09:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:(3/22): libcephfs2-17.0.0-9665.gaa33f58d.el8.x8 3.5 MB/s | 785 kB 00:00 2021-12-15T17:09:21.838 INFO:teuthology.orchestra.run.smithi122.stdout:(16/22): librgw2-17.0.0-9665.gaa33f58d.el8.x86_ 5.0 MB/s | 3.9 MB 00:00 2021-12-15T17:09:21.962 INFO:teuthology.orchestra.run.smithi022.stdout:(4/22): libradosstriper1-17.0.0-9665.gaa33f58d. 3.5 MB/s | 482 kB 00:00 2021-12-15T17:09:21.980 INFO:teuthology.orchestra.run.smithi122.stdout:(17/22): liboath-2.6.2-3.el8.x86_64.rpm 415 kB/s | 59 kB 00:00 2021-12-15T17:09:22.071 INFO:teuthology.orchestra.run.smithi022.stdout:(5/22): ceph-radosgw-17.0.0-9665.gaa33f58d.el8. 3.5 MB/s | 9.0 MB 00:02 2021-12-15T17:09:22.138 INFO:teuthology.orchestra.run.smithi022.stdout:(6/22): python3-ceph-argparse-17.0.0-9665.gaa33 668 kB/s | 45 kB 00:00 2021-12-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi022.stdout:(7/22): python3-ceph-common-17.0.0-9665.gaa33f5 1.1 MB/s | 98 kB 00:00 2021-12-15T17:09:22.229 INFO:teuthology.orchestra.run.smithi122.stdout:(18/22): ceph-common-17.0.0-9665.gaa33f58d.el8. 6.9 MB/s | 20 MB 00:02 2021-12-15T17:09:22.255 INFO:teuthology.orchestra.run.smithi122.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 272 kB/s | 75 kB 00:00 2021-12-15T17:09:22.281 INFO:teuthology.orchestra.run.smithi122.stdout:(20/22): gperftools-libs-2.7-9.el8.x86_64.rpm 582 kB/s | 306 kB 00:00 2021-12-15T17:09:22.288 INFO:teuthology.orchestra.run.smithi022.stdout:(8/22): python3-cephfs-17.0.0-9665.gaa33f58d.el 2.6 MB/s | 178 kB 00:00 2021-12-15T17:09:22.380 INFO:teuthology.orchestra.run.smithi022.stdout:(9/22): python3-rados-17.0.0-9665.gaa33f58d.el8 3.9 MB/s | 367 kB 00:00 2021-12-15T17:09:22.472 INFO:teuthology.orchestra.run.smithi022.stdout:(10/22): librgw2-17.0.0-9665.gaa33f58d.el8.x86_ 7.7 MB/s | 3.9 MB 00:00 2021-12-15T17:09:22.497 INFO:teuthology.orchestra.run.smithi022.stdout:(11/22): python3-rbd-17.0.0-9665.gaa33f58d.el8. 2.9 MB/s | 345 kB 00:00 2021-12-15T17:09:22.548 INFO:teuthology.orchestra.run.smithi022.stdout:(12/22): python3-rgw-17.0.0-9665.gaa33f58d.el8. 1.4 MB/s | 110 kB 00:00 2021-12-15T17:09:22.581 INFO:teuthology.orchestra.run.smithi022.stdout:(13/22): librdkafka-0.11.4-3.el8.x86_64.rpm 4.1 MB/s | 354 kB 00:00 2021-12-15T17:09:22.598 INFO:teuthology.orchestra.run.smithi022.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 2.7 MB/s | 44 kB 00:00 2021-12-15T17:09:22.615 INFO:teuthology.orchestra.run.smithi022.stdout:(15/22): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.8 MB/s | 47 kB 00:00 2021-12-15T17:09:22.641 INFO:teuthology.orchestra.run.smithi022.stdout:(16/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.7 MB/s | 259 kB 00:00 2021-12-15T17:09:22.731 INFO:teuthology.orchestra.run.smithi122.stdout:(21/22): librados2-17.0.0-9665.gaa33f58d.el8.x8 6.8 MB/s | 3.4 MB 00:00 2021-12-15T17:09:22.774 INFO:teuthology.orchestra.run.smithi022.stdout:(17/22): liboath-2.6.2-3.el8.x86_64.rpm 442 kB/s | 59 kB 00:00 2021-12-15T17:09:22.798 INFO:teuthology.orchestra.run.smithi122.stdout:(22/22): librbd1-17.0.0-9665.gaa33f58d.el8.x86_ 7.1 MB/s | 3.9 MB 00:00 2021-12-15T17:09:22.800 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:09:22.801 INFO:teuthology.orchestra.run.smithi122.stdout:Total 14 MB/s | 49 MB 00:03 2021-12-15T17:09:22.841 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:09:22.858 INFO:teuthology.orchestra.run.smithi022.stdout:(18/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2021-12-15T17:09:22.883 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:09:22.883 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:09:22.883 INFO:teuthology.orchestra.run.smithi022.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 692 kB/s | 75 kB 00:00 2021-12-15T17:09:23.164 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:09:23.167 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:09:23.525 INFO:teuthology.orchestra.run.smithi022.stdout:(20/22): ceph-common-17.0.0-9665.gaa33f58d.el8. 5.0 MB/s | 20 MB 00:04 2021-12-15T17:09:23.575 INFO:teuthology.orchestra.run.smithi022.stdout:(21/22): librados2-17.0.0-9665.gaa33f58d.el8.x8 4.8 MB/s | 3.4 MB 00:00 2021-12-15T17:09:23.642 INFO:teuthology.orchestra.run.smithi022.stdout:(22/22): librbd1-17.0.0-9665.gaa33f58d.el8.x86_ 5.1 MB/s | 3.9 MB 00:00 2021-12-15T17:09:23.645 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:09:23.646 INFO:teuthology.orchestra.run.smithi022.stdout:Total 12 MB/s | 49 MB 00:04 2021-12-15T17:09:23.691 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:09:23.739 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:09:23.739 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:09:24.049 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:09:24.052 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:09:24.200 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:09:24.416 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2021-12-15T17:09:24.557 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2021-12-15T17:09:24.925 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2021-12-15T17:09:25.046 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:09:25.098 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/24 2021-12-15T17:09:25.179 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/24 2021-12-15T17:09:25.405 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2021-12-15T17:09:25.440 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 4/24 2021-12-15T17:09:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2021-12-15T17:09:25.649 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/24 2021-12-15T17:09:25.949 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2021-12-15T17:09:26.079 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/24 2021-12-15T17:09:26.091 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/24 2021-12-15T17:09:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/24 2021-12-15T17:09:26.248 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/24 2021-12-15T17:09:26.310 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/24 2021-12-15T17:09:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 4/24 2021-12-15T17:09:26.481 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/24 2021-12-15T17:09:26.607 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/24 2021-12-15T17:09:26.657 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/24 2021-12-15T17:09:27.041 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/24 2021-12-15T17:09:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/24 2021-12-15T17:09:27.199 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/24 2021-12-15T17:09:27.263 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/24 2021-12-15T17:09:27.273 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 9/24 2021-12-15T17:09:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 9/24 2021-12-15T17:09:27.433 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/24 2021-12-15T17:09:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:17.0.0-9665.gaa33f58d.el8. 10/24 2021-12-15T17:09:27.591 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/24 2021-12-15T17:09:27.635 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 11/24 2021-12-15T17:09:27.840 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 12/24 2021-12-15T17:09:28.021 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/24 2021-12-15T17:09:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 13/24 2021-12-15T17:09:28.199 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 9/24 2021-12-15T17:09:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_6 14/24 2021-12-15T17:09:28.267 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 9/24 2021-12-15T17:09:28.277 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_6 14/24 2021-12-15T17:09:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:17.0.0-9665.gaa33f58d.el8. 10/24 2021-12-15T17:09:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2021-12-15T17:09:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 11/24 2021-12-15T17:09:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2021-12-15T17:09:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 12/24 2021-12-15T17:09:28.761 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2021-12-15T17:09:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:17.0.0-9665.gaa33f58d.el8.x8 18/24 2021-12-15T17:09:28.946 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 13/24 2021-12-15T17:09:29.107 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_6 14/24 2021-12-15T17:09:29.180 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_6 14/24 2021-12-15T17:09:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2021-12-15T17:09:29.491 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2021-12-15T17:09:29.651 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2021-12-15T17:09:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:17.0.0-9665.gaa33f58d.el8.x8 18/24 2021-12-15T17:09:32.979 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 19/24 2021-12-15T17:09:33.198 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 19/24 2021-12-15T17:09:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 19/24 2021-12-15T17:09:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 19/24 2021-12-15T17:09:34.052 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 20/24 2021-12-15T17:09:34.099 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 19/24 2021-12-15T17:09:34.170 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 20/24 2021-12-15T17:09:34.290 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 21/24 2021-12-15T17:09:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 19/24 2021-12-15T17:09:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 20/24 2021-12-15T17:09:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 20/24 2021-12-15T17:09:35.198 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 21/24 2021-12-15T17:09:52.694 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 21/24 2021-12-15T17:09:52.694 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2021-12-15T17:09:52.694 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2021-12-15T17:09:52.695 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2021-12-15T17:09:52.695 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /run 2021-12-15T17:09:52.695 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2021-12-15T17:09:52.695 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2021-12-15T17:09:52.695 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2021-12-15T17:09:52.696 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2021-12-15T17:09:52.696 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2021-12-15T17:09:52.696 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:09:53.778 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 21/24 2021-12-15T17:09:53.778 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2021-12-15T17:09:53.779 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2021-12-15T17:09:53.779 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /dev 2021-12-15T17:09:53.779 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /run 2021-12-15T17:09:53.779 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2021-12-15T17:09:53.780 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2021-12-15T17:09:53.780 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2021-12-15T17:09:53.780 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2021-12-15T17:09:53.780 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /dev 2021-12-15T17:09:53.780 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:09:53.800 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 22/24 2021-12-15T17:09:53.874 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 22/24 2021-12-15T17:09:53.874 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2021-12-15T17:09:54.081 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2021-12-15T17:09:54.081 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2021-12-15T17:09:54.806 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2021-12-15T17:09:54.806 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/24 2021-12-15T17:09:54.806 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/24 2021-12-15T17:09:54.807 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/24 2021-12-15T17:09:54.807 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 4/24 2021-12-15T17:09:54.807 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/24 2021-12-15T17:09:54.807 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_6 6/24 2021-12-15T17:09:54.808 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 7/24 2021-12-15T17:09:54.808 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:17.0.0-9665.gaa33f58d.el8. 8/24 2021-12-15T17:09:54.808 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:17.0.0-9665.gaa33f58d.el8.x8 9/24 2021-12-15T17:09:54.808 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 10/24 2021-12-15T17:09:54.808 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 11/24 2021-12-15T17:09:54.809 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 12/24 2021-12-15T17:09:54.809 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 13/24 2021-12-15T17:09:54.809 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/24 2021-12-15T17:09:54.809 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/24 2021-12-15T17:09:54.810 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/24 2021-12-15T17:09:54.810 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/24 2021-12-15T17:09:54.810 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/24 2021-12-15T17:09:54.810 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2021-12-15T17:09:54.810 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2021-12-15T17:09:54.811 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 21/24 2021-12-15T17:09:54.811 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2021-12-15T17:09:54.811 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 23/24 2021-12-15T17:09:54.935 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 22/24 2021-12-15T17:09:55.007 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 22/24 2021-12-15T17:09:55.008 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2021-12-15T17:09:55.181 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2021-12-15T17:09:55.181 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2021-12-15T17:09:55.583 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2021-12-15T17:09:55.584 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:09:55.584 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2021-12-15T17:09:55.584 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.584 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.585 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:09:55.585 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.585 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.585 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.586 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.586 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-12-15T17:09:55.586 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.586 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.2-3.el8.x86_64 2021-12-15T17:09:55.587 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-12-15T17:09:55.587 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.587 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-12-15T17:09:55.587 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.587 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.3.1-3.el8.x86_64 2021-12-15T17:09:55.588 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-12-15T17:09:55.588 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.588 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.588 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.589 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-12-15T17:09:55.589 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.589 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.589 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:55.590 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:09:55.590 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:09:55.821 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2021-12-15T17:09:55.846 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2021-12-15T17:09:55.846 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/24 2021-12-15T17:09:55.847 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/24 2021-12-15T17:09:55.847 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/24 2021-12-15T17:09:55.847 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 4/24 2021-12-15T17:09:55.848 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/24 2021-12-15T17:09:55.848 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_6 6/24 2021-12-15T17:09:55.849 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 7/24 2021-12-15T17:09:55.849 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:17.0.0-9665.gaa33f58d.el8. 8/24 2021-12-15T17:09:55.849 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:17.0.0-9665.gaa33f58d.el8.x8 9/24 2021-12-15T17:09:55.849 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 10/24 2021-12-15T17:09:55.849 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 11/24 2021-12-15T17:09:55.850 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 12/24 2021-12-15T17:09:55.850 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 13/24 2021-12-15T17:09:55.850 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/24 2021-12-15T17:09:55.850 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/24 2021-12-15T17:09:55.851 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/24 2021-12-15T17:09:55.851 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/24 2021-12-15T17:09:55.851 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/24 2021-12-15T17:09:55.851 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2021-12-15T17:09:55.851 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2021-12-15T17:09:55.852 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 21/24 2021-12-15T17:09:55.852 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2021-12-15T17:09:55.852 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 23/24 2021-12-15T17:09:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:09:56.013 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:09:56.358 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:40 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:09:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2021-12-15T17:09:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:09:56.629 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2021-12-15T17:09:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.629 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:09:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-12-15T17:09:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.2-3.el8.x86_64 2021-12-15T17:09:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-12-15T17:09:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-12-15T17:09:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.3.1-3.el8.x86_64 2021-12-15T17:09:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-12-15T17:09:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-12-15T17:09:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:09:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:09:56.634 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:09:56.859 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-test 2021-12-15T17:09:57.059 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:09:57.060 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:09:57.076 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:09:57.077 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:09:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:09:57.078 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:09:57.078 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:09:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 45 M 2021-12-15T17:09:57.078 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:09:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-12-15T17:09:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 170 k 2021-12-15T17:09:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-12-15T17:09:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-12-15T17:09:57.080 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-12-15T17:09:57.080 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:09:57.080 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:09:57.080 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:09:57.081 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2021-12-15T17:09:57.081 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:09:57.082 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 46 M 2021-12-15T17:09:57.082 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 213 M 2021-12-15T17:09:57.082 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:09:57.149 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.9 MB/s | 202 kB 00:00 2021-12-15T17:09:57.182 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2021-12-15T17:09:57.208 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2021-12-15T17:09:57.350 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 491 kB/s | 69 kB 00:00 2021-12-15T17:09:57.416 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:41 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:09:57.433 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): libcephsqlite-17.0.0-9665.gaa33f58d.el8. 481 kB/s | 170 kB 00:00 2021-12-15T17:09:58.216 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:09:58.217 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:09:58.217 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:09:58.218 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:09:58.218 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:09:58.218 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 45 M 2021-12-15T17:09:58.218 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:09:58.219 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-12-15T17:09:58.219 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 170 k 2021-12-15T17:09:58.219 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-12-15T17:09:58.219 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-12-15T17:09:58.219 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-12-15T17:09:58.220 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:09:58.220 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:09:58.220 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:09:58.220 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2021-12-15T17:09:58.220 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:09:58.221 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 46 M 2021-12-15T17:09:58.221 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 213 M 2021-12-15T17:09:58.222 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:09:58.289 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.9 MB/s | 202 kB 00:00 2021-12-15T17:09:58.323 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.5 MB/s | 187 kB 00:00 2021-12-15T17:09:58.348 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2021-12-15T17:09:58.482 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 522 kB/s | 69 kB 00:00 2021-12-15T17:09:58.566 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): libcephsqlite-17.0.0-9665.gaa33f58d.el8. 492 kB/s | 170 kB 00:00 2021-12-15T17:10:01.409 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-test-17.0.0-9665.gaa33f58d.el8.x86_ 10 MB/s | 45 MB 00:04 2021-12-15T17:10:01.411 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:01.412 INFO:teuthology.orchestra.run.smithi122.stdout:Total 11 MB/s | 46 MB 00:04 2021-12-15T17:10:01.447 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:01.468 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:01.469 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:01.733 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:01.736 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:02.357 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-test-17.0.0-9665.gaa33f58d.el8.x86_ 11 MB/s | 45 MB 00:04 2021-12-15T17:10:02.359 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:02.360 INFO:teuthology.orchestra.run.smithi022.stdout:Total 11 MB/s | 46 MB 00:04 2021-12-15T17:10:02.398 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:10:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-12-15T17:10:02.421 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:10:02.421 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:10:02.605 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-12-15T17:10:02.697 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:10:02.700 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:10:02.727 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-12-15T17:10:02.815 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-12-15T17:10:03.017 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-12-15T17:10:03.144 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:10:03.160 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/6 2021-12-15T17:10:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-12-15T17:10:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-12-15T17:10:03.627 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-12-15T17:10:03.695 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-12-15T17:10:03.926 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-12-15T17:10:04.060 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/6 2021-12-15T17:10:07.544 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/6 2021-12-15T17:10:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/6 2021-12-15T17:10:08.012 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/6 2021-12-15T17:10:08.012 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/6 2021-12-15T17:10:08.012 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/6 2021-12-15T17:10:08.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-12-15T17:10:08.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-12-15T17:10:08.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-12-15T17:10:08.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-12-15T17:10:08.392 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:08.392 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:10:08.392 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:08.392 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-3.el8.x86_64 2021-12-15T17:10:08.393 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:08.393 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-12-15T17:10:08.393 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-1.el8.x86_64 2021-12-15T17:10:08.393 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-12-15T17:10:08.394 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:08.394 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:08.584 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2021-12-15T17:10:08.763 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:08.764 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:08.776 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/6 2021-12-15T17:10:08.982 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/6 2021-12-15T17:10:09.111 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:53 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:09.226 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 6/6 2021-12-15T17:10:09.227 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/6 2021-12-15T17:10:09.227 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/6 2021-12-15T17:10:09.227 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-12-15T17:10:09.227 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-12-15T17:10:09.228 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-12-15T17:10:09.632 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-12-15T17:10:09.632 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:09.633 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:10:09.633 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:09.633 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.6-3.el8.x86_64 2021-12-15T17:10:09.634 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:09.634 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-12-15T17:10:09.634 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.4.1-1.el8.x86_64 2021-12-15T17:10:09.635 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-12-15T17:10:09.635 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:09.635 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:09.803 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph 2021-12-15T17:10:09.829 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:09.833 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2021-12-15T17:10:09.833 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:10:09.833 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2021-12-15T17:10:09.833 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:10:09.834 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 6.4 k 2021-12-15T17:10:09.834 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:10:09.834 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 2.3 M 2021-12-15T17:10:09.834 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 1.5 M 2021-12-15T17:10:09.835 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 216 k 2021-12-15T17:10:09.835 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.9 M 2021-12-15T17:10:09.835 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 17 M 2021-12-15T17:10:09.835 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-12-15T17:10:09.835 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-12-15T17:10:09.836 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-12-15T17:10:09.836 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-12-15T17:10:09.836 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-12-15T17:10:09.836 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-12-15T17:10:09.837 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-12-15T17:10:09.837 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-12-15T17:10:09.837 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-12-15T17:10:09.837 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-12-15T17:10:09.838 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-12-15T17:10:09.838 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-12-15T17:10:09.838 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-12-15T17:10:09.838 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-12-15T17:10:09.838 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-12-15T17:10:09.839 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-12-15T17:10:09.839 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-12-15T17:10:09.839 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-12-15T17:10:09.839 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-12-15T17:10:09.840 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-12-15T17:10:09.840 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-12-15T17:10:09.840 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-12-15T17:10:09.840 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-12-15T17:10:09.840 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-12-15T17:10:09.841 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-12-15T17:10:09.841 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-12-15T17:10:09.841 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:09.841 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:10:09.842 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2021-12-15T17:10:09.842 INFO:teuthology.orchestra.run.smithi122.stdout:Install 32 Packages 2021-12-15T17:10:09.842 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:09.843 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 28 M 2021-12-15T17:10:09.843 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 112 M 2021-12-15T17:10:09.843 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:10:09.996 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:09.997 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:10.033 INFO:teuthology.orchestra.run.smithi122.stdout:(1/32): ceph-17.0.0-9665.gaa33f58d.el8.x86_64.r 33 kB/s | 6.4 kB 00:00 2021-12-15T17:10:10.333 INFO:teuthology.orchestra.run.smithi122.stdout:(2/32): ceph-mgr-17.0.0-9665.gaa33f58d.el8.x86_ 3.1 MB/s | 1.5 MB 00:00 2021-12-15T17:10:10.357 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:54 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:10.442 INFO:teuthology.orchestra.run.smithi122.stdout:(3/32): ceph-mds-17.0.0-9665.gaa33f58d.el8.x86_ 3.8 MB/s | 2.3 MB 00:00 2021-12-15T17:10:10.517 INFO:teuthology.orchestra.run.smithi122.stdout:(4/32): ceph-mgr-modules-core-17.0.0-9665.gaa33 2.8 MB/s | 216 kB 00:00 2021-12-15T17:10:10.576 INFO:teuthology.orchestra.run.smithi122.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2021-12-15T17:10:10.626 INFO:teuthology.orchestra.run.smithi122.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 777 kB/s | 39 kB 00:00 2021-12-15T17:10:10.651 INFO:teuthology.orchestra.run.smithi122.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.1 MB/s | 103 kB 00:00 2021-12-15T17:10:10.677 INFO:teuthology.orchestra.run.smithi122.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2021-12-15T17:10:10.710 INFO:teuthology.orchestra.run.smithi122.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2021-12-15T17:10:10.735 INFO:teuthology.orchestra.run.smithi122.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2021-12-15T17:10:10.777 INFO:teuthology.orchestra.run.smithi122.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2021-12-15T17:10:10.828 INFO:teuthology.orchestra.run.smithi122.stdout:(12/32): ceph-mon-17.0.0-9665.gaa33f58d.el8.x86 4.9 MB/s | 3.9 MB 00:00 2021-12-15T17:10:10.853 INFO:teuthology.orchestra.run.smithi122.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.4 MB/s | 109 kB 00:00 2021-12-15T17:10:10.962 INFO:teuthology.orchestra.run.smithi122.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 326 kB/s | 44 kB 00:00 2021-12-15T17:10:11.045 INFO:teuthology.orchestra.run.smithi122.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 965 kB/s | 185 kB 00:00 2021-12-15T17:10:11.071 INFO:teuthology.orchestra.run.smithi122.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2021-12-15T17:10:11.121 INFO:teuthology.orchestra.run.smithi122.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2021-12-15T17:10:11.138 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:11.142 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2021-12-15T17:10:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:10:11.143 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2021-12-15T17:10:11.143 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:10:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 6.4 k 2021-12-15T17:10:11.143 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:10:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 2.3 M 2021-12-15T17:10:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 1.5 M 2021-12-15T17:10:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 216 k 2021-12-15T17:10:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.9 M 2021-12-15T17:10:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 17 M 2021-12-15T17:10:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-12-15T17:10:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-12-15T17:10:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-12-15T17:10:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-12-15T17:10:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-12-15T17:10:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-12-15T17:10:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-12-15T17:10:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-12-15T17:10:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-12-15T17:10:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-12-15T17:10:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-12-15T17:10:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-12-15T17:10:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-12-15T17:10:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-12-15T17:10:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-12-15T17:10:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-12-15T17:10:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-12-15T17:10:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-12-15T17:10:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-12-15T17:10:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-12-15T17:10:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-12-15T17:10:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-12-15T17:10:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-12-15T17:10:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-12-15T17:10:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-12-15T17:10:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-12-15T17:10:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:11.150 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:10:11.151 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2021-12-15T17:10:11.151 INFO:teuthology.orchestra.run.smithi022.stdout:Install 32 Packages 2021-12-15T17:10:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:11.152 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 28 M 2021-12-15T17:10:11.152 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 112 M 2021-12-15T17:10:11.152 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:10:11.163 INFO:teuthology.orchestra.run.smithi122.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 422 kB/s | 18 kB 00:00 2021-12-15T17:10:11.222 INFO:teuthology.orchestra.run.smithi122.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 833 kB/s | 49 kB 00:00 2021-12-15T17:10:11.247 INFO:teuthology.orchestra.run.smithi122.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2021-12-15T17:10:11.272 INFO:teuthology.orchestra.run.smithi122.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2021-12-15T17:10:11.323 INFO:teuthology.orchestra.run.smithi122.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2021-12-15T17:10:11.364 INFO:teuthology.orchestra.run.smithi022.stdout:(1/32): ceph-17.0.0-9665.gaa33f58d.el8.x86_64.r 30 kB/s | 6.4 kB 00:00 2021-12-15T17:10:11.365 INFO:teuthology.orchestra.run.smithi122.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 449 kB/s | 19 kB 00:00 2021-12-15T17:10:11.415 INFO:teuthology.orchestra.run.smithi122.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 573 kB/s | 24 kB 00:00 2021-12-15T17:10:11.432 INFO:teuthology.orchestra.run.smithi122.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2021-12-15T17:10:11.449 INFO:teuthology.orchestra.run.smithi122.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 701 kB/s | 29 kB 00:00 2021-12-15T17:10:11.495 INFO:teuthology.orchestra.run.smithi122.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 440 kB/s | 27 kB 00:00 2021-12-15T17:10:11.608 INFO:teuthology.orchestra.run.smithi122.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.5 MB/s | 241 kB 00:00 2021-12-15T17:10:11.633 INFO:teuthology.orchestra.run.smithi122.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2021-12-15T17:10:11.684 INFO:teuthology.orchestra.run.smithi122.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2021-12-15T17:10:11.710 INFO:teuthology.orchestra.run.smithi122.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 847 kB/s | 85 kB 00:00 2021-12-15T17:10:11.831 INFO:teuthology.orchestra.run.smithi022.stdout:(2/32): ceph-mgr-17.0.0-9665.gaa33f58d.el8.x86_ 2.2 MB/s | 1.5 MB 00:00 2021-12-15T17:10:11.901 INFO:teuthology.orchestra.run.smithi122.stdout:(32/32): ceph-osd-17.0.0-9665.gaa33f58d.el8.x86 11 MB/s | 17 MB 00:01 2021-12-15T17:10:11.903 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:11.904 INFO:teuthology.orchestra.run.smithi122.stdout:Total 14 MB/s | 28 MB 00:02 2021-12-15T17:10:11.948 INFO:teuthology.orchestra.run.smithi022.stdout:(3/32): ceph-mds-17.0.0-9665.gaa33f58d.el8.x86_ 2.8 MB/s | 2.3 MB 00:00 2021-12-15T17:10:11.987 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:12.015 INFO:teuthology.orchestra.run.smithi022.stdout:(4/32): ceph-mgr-modules-core-17.0.0-9665.gaa33 3.2 MB/s | 216 kB 00:00 2021-12-15T17:10:12.023 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:12.023 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:12.082 INFO:teuthology.orchestra.run.smithi022.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2021-12-15T17:10:12.099 INFO:teuthology.orchestra.run.smithi022.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 2.3 MB/s | 39 kB 00:00 2021-12-15T17:10:12.124 INFO:teuthology.orchestra.run.smithi022.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2021-12-15T17:10:12.141 INFO:teuthology.orchestra.run.smithi022.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2021-12-15T17:10:12.200 INFO:teuthology.orchestra.run.smithi022.stdout:(9/32): ceph-mon-17.0.0-9665.gaa33f58d.el8.x86_ 4.6 MB/s | 3.9 MB 00:00 2021-12-15T17:10:12.234 INFO:teuthology.orchestra.run.smithi022.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.9 MB/s | 457 kB 00:00 2021-12-15T17:10:12.267 INFO:teuthology.orchestra.run.smithi022.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 17 MB/s | 559 kB 00:00 2021-12-15T17:10:12.292 INFO:teuthology.orchestra.run.smithi022.stdout:(12/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 237 kB 00:00 2021-12-15T17:10:12.294 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:12.297 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:12.317 INFO:teuthology.orchestra.run.smithi022.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2021-12-15T17:10:12.460 INFO:teuthology.orchestra.run.smithi022.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2021-12-15T17:10:12.510 INFO:teuthology.orchestra.run.smithi022.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 966 kB/s | 185 kB 00:00 2021-12-15T17:10:12.560 INFO:teuthology.orchestra.run.smithi022.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2021-12-15T17:10:12.610 INFO:teuthology.orchestra.run.smithi022.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2021-12-15T17:10:12.661 INFO:teuthology.orchestra.run.smithi022.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 353 kB/s | 18 kB 00:00 2021-12-15T17:10:12.702 INFO:teuthology.orchestra.run.smithi022.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.0 MB/s | 384 kB 00:00 2021-12-15T17:10:12.727 INFO:teuthology.orchestra.run.smithi022.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 728 kB/s | 49 kB 00:00 2021-12-15T17:10:12.753 INFO:teuthology.orchestra.run.smithi022.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2021-12-15T17:10:12.803 INFO:teuthology.orchestra.run.smithi022.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2021-12-15T17:10:12.854 INFO:teuthology.orchestra.run.smithi022.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 374 kB/s | 19 kB 00:00 2021-12-15T17:10:12.879 INFO:teuthology.orchestra.run.smithi022.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2021-12-15T17:10:12.904 INFO:teuthology.orchestra.run.smithi022.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2021-12-15T17:10:12.929 INFO:teuthology.orchestra.run.smithi022.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 583 kB/s | 29 kB 00:00 2021-12-15T17:10:12.955 INFO:teuthology.orchestra.run.smithi022.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2021-12-15T17:10:13.064 INFO:teuthology.orchestra.run.smithi022.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.8 MB/s | 241 kB 00:00 2021-12-15T17:10:13.105 INFO:teuthology.orchestra.run.smithi022.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.6 MB/s | 251 kB 00:00 2021-12-15T17:10:13.156 INFO:teuthology.orchestra.run.smithi022.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 932 kB/s | 85 kB 00:00 2021-12-15T17:10:13.181 INFO:teuthology.orchestra.run.smithi022.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2021-12-15T17:10:13.478 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:13.523 INFO:teuthology.orchestra.run.smithi022.stdout:(32/32): ceph-osd-17.0.0-9665.gaa33f58d.el8.x86 9.8 MB/s | 17 MB 00:01 2021-12-15T17:10:13.527 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:13.527 INFO:teuthology.orchestra.run.smithi022.stdout:Total 12 MB/s | 28 MB 00:02 2021-12-15T17:10:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:10:13.646 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-12-15T17:10:13.662 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:10:13.662 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:10:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-12-15T17:10:13.970 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:10:13.973 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:10:14.003 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-12-15T17:10:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-12-15T17:10:14.330 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-12-15T17:10:14.451 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-12-15T17:10:14.620 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-12-15T17:10:14.731 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-12-15T17:10:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-12-15T17:10:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:10:15.110 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-12-15T17:10:15.250 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-12-15T17:10:15.306 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-12-15T17:10:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-12-15T17:10:15.471 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-12-15T17:10:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-12-15T17:10:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-12-15T17:10:15.839 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-12-15T17:10:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-12-15T17:10:16.070 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-12-15T17:10:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-12-15T17:10:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-12-15T17:10:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-12-15T17:10:16.372 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-12-15T17:10:16.390 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-12-15T17:10:16.551 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-12-15T17:10:16.635 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-12-15T17:10:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-12-15T17:10:16.900 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-12-15T17:10:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-12-15T17:10:17.110 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-12-15T17:10:17.114 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-12-15T17:10:17.261 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-12-15T17:10:17.439 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-12-15T17:10:17.443 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-12-15T17:10:17.655 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-12-15T17:10:17.679 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-12-15T17:10:17.833 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-12-15T17:10:17.881 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-12-15T17:10:18.027 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-12-15T17:10:18.118 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-12-15T17:10:18.260 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-12-15T17:10:18.427 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-12-15T17:10:18.523 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-12-15T17:10:18.689 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-12-15T17:10:18.734 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-12-15T17:10:18.852 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-12-15T17:10:18.988 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-12-15T17:10:19.106 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-9665.gaa33f58d.el8. 27/32 2021-12-15T17:10:19.188 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-12-15T17:10:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 28/32 2021-12-15T17:10:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-12-15T17:10:19.616 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-12-15T17:10:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-12-15T17:10:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-12-15T17:10:20.575 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-9665.gaa33f58d.el8. 27/32 2021-12-15T17:10:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 28/32 2021-12-15T17:10:20.745 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 28/32 2021-12-15T17:10:20.888 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 29/32 2021-12-15T17:10:21.275 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 29/32 2021-12-15T17:10:21.400 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 30/32 2021-12-15T17:10:21.616 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 30/32 2021-12-15T17:10:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 31/32 2021-12-15T17:10:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 31/32 2021-12-15T17:10:21.932 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 32/32 2021-12-15T17:10:22.278 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 32/32 2021-12-15T17:10:22.278 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/32 2021-12-15T17:10:22.278 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/32 2021-12-15T17:10:22.279 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/32 2021-12-15T17:10:22.279 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 4/32 2021-12-15T17:10:22.279 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/32 2021-12-15T17:10:22.279 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-9665.gaa33f58d.el8. 6/32 2021-12-15T17:10:22.279 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-12-15T17:10:22.280 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-12-15T17:10:22.280 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-12-15T17:10:22.280 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-12-15T17:10:22.280 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-12-15T17:10:22.281 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-12-15T17:10:22.281 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-12-15T17:10:22.281 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-12-15T17:10:22.281 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-12-15T17:10:22.281 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-12-15T17:10:22.282 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-12-15T17:10:22.282 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-12-15T17:10:22.282 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-12-15T17:10:22.282 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-12-15T17:10:22.282 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-12-15T17:10:22.283 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-12-15T17:10:22.283 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-12-15T17:10:22.283 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-12-15T17:10:22.283 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-12-15T17:10:22.284 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-12-15T17:10:22.284 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-12-15T17:10:22.284 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-12-15T17:10:22.284 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-12-15T17:10:22.285 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-12-15T17:10:22.285 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-12-15T17:10:22.306 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 28/32 2021-12-15T17:10:22.464 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 29/32 2021-12-15T17:10:22.875 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 29/32 2021-12-15T17:10:22.999 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 30/32 2021-12-15T17:10:23.226 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 30/32 2021-12-15T17:10:23.227 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-12-15T17:10:23.227 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:23.227 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:10:23.227 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:23.228 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:23.228 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:23.228 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:23.228 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:23.229 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:23.229 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-12-15T17:10:23.229 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-12-15T17:10:23.229 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-12-15T17:10:23.229 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-12-15T17:10:23.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-12-15T17:10:23.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-12-15T17:10:23.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-6.2-6.el8.noarch 2021-12-15T17:10:23.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-12-15T17:10:23.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-12-15T17:10:23.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.0.6-13.el8.noarch 2021-12-15T17:10:23.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-12-15T17:10:23.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-12-15T17:10:23.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-12-15T17:10:23.232 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-2.6-1.el8.noarch 2021-12-15T17:10:23.232 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-12-15T17:10:23.232 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.14-14.el8.noarch 2021-12-15T17:10:23.232 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2017.2-9.el8.noarch 2021-12-15T17:10:23.233 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-12-15T17:10:23.233 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-12-15T17:10:23.233 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-12-15T17:10:23.233 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-12-15T17:10:23.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-12-15T17:10:23.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-12-15T17:10:23.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-12-15T17:10:23.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-12-15T17:10:23.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-12-15T17:10:23.235 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:23.235 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:23.374 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 31/32 2021-12-15T17:10:23.391 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 31/32 2021-12-15T17:10:23.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2021-12-15T17:10:23.556 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 32/32 2021-12-15T17:10:23.705 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:23.705 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:23.902 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 32/32 2021-12-15T17:10:23.902 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/32 2021-12-15T17:10:23.903 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/32 2021-12-15T17:10:23.903 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 3/32 2021-12-15T17:10:23.903 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 4/32 2021-12-15T17:10:23.903 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 5/32 2021-12-15T17:10:23.904 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-9665.gaa33f58d.el8. 6/32 2021-12-15T17:10:23.904 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-12-15T17:10:23.904 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-12-15T17:10:23.904 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-12-15T17:10:23.905 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-12-15T17:10:23.905 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-12-15T17:10:23.905 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-12-15T17:10:23.905 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-12-15T17:10:23.905 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-12-15T17:10:23.906 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-12-15T17:10:23.906 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-12-15T17:10:23.906 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-12-15T17:10:23.906 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-12-15T17:10:23.907 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-12-15T17:10:23.907 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-12-15T17:10:23.907 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-12-15T17:10:23.907 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-12-15T17:10:23.907 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-12-15T17:10:23.908 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-12-15T17:10:23.908 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-12-15T17:10:23.909 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-12-15T17:10:23.909 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-12-15T17:10:23.910 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-12-15T17:10:23.910 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-12-15T17:10:23.910 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-12-15T17:10:23.910 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-12-15T17:10:24.055 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:08 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-12-15T17:10:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:24.699 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:10:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-12-15T17:10:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-12-15T17:10:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-12-15T17:10:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-12-15T17:10:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-12-15T17:10:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-12-15T17:10:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-6.2-6.el8.noarch 2021-12-15T17:10:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-12-15T17:10:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-12-15T17:10:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.0.6-13.el8.noarch 2021-12-15T17:10:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-12-15T17:10:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-12-15T17:10:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-12-15T17:10:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-2.6-1.el8.noarch 2021-12-15T17:10:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-12-15T17:10:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser-2.14-14.el8.noarch 2021-12-15T17:10:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2017.2-9.el8.noarch 2021-12-15T17:10:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-12-15T17:10:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-12-15T17:10:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-12-15T17:10:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-12-15T17:10:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-12-15T17:10:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-12-15T17:10:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-12-15T17:10:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-12-15T17:10:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-12-15T17:10:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:24.707 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:24.737 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:10:24.773 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:24.774 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:10:24.775 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:24.856 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2021-12-15T17:10:24.960 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base 2021-12-15T17:10:25.032 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:25.033 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:25.151 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:25.152 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:25.380 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:09 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:25.521 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:09 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:26.115 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:26.117 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:26.118 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:10:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:10:26.118 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 68 k 2021-12-15T17:10:26.118 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:26.119 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:10:26.119 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:26.119 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:10:26.119 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:26.120 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 68 k 2021-12-15T17:10:26.120 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 305 k 2021-12-15T17:10:26.120 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:10:26.241 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-base-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:10:26.280 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:26.281 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:10:26.281 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:26.356 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install cephadm 2021-12-15T17:10:26.385 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-17.0.0-9665.gaa33f58d.el8.noarch.rpm 255 kB/s | 68 kB 00:00 2021-12-15T17:10:26.386 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:26.386 INFO:teuthology.orchestra.run.smithi122.stdout:Total 254 kB/s | 68 kB 00:00 2021-12-15T17:10:26.387 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:26.394 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:26.394 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:26.395 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:26.398 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:26.441 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:26.543 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:26.544 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:26.905 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:10 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:27.690 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:27.691 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:27.691 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:10:27.691 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:27.692 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:10:27.692 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 68 k 2021-12-15T17:10:27.692 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:27.692 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:10:27.693 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:27.693 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:10:27.693 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:27.693 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 68 k 2021-12-15T17:10:27.693 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 305 k 2021-12-15T17:10:27.694 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:10:27.950 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm-17.0.0-9665.gaa33f58d.el8.noarch.rpm 265 kB/s | 68 kB 00:00 2021-12-15T17:10:27.950 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:27.951 INFO:teuthology.orchestra.run.smithi022.stdout:Total 264 kB/s | 68 kB 00:00 2021-12-15T17:10:27.951 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:10:27.959 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:10:27.960 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:10:27.961 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:10:27.964 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:10:28.018 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:10:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:28.649 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:28.822 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:29.102 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:29.102 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:29.103 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:10:29.103 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:29.103 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:29.103 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:29.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2021-12-15T17:10:29.483 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:29.483 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:29.829 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:13 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:29.973 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:30.285 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:30.568 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:30.569 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:30.569 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2021-12-15T17:10:30.570 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:30.570 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:10:30.570 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 169 k 2021-12-15T17:10:30.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:30.571 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:10:30.571 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:30.571 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:10:30.571 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:30.571 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 169 k 2021-12-15T17:10:30.572 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 567 k 2021-12-15T17:10:30.572 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:10:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:10:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:30.600 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:10:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:30.601 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:30.788 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-immutable-object-cache 2021-12-15T17:10:30.889 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-17.0.0-9665.gaa33f5 530 kB/s | 169 kB 00:00 2021-12-15T17:10:30.890 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:30.890 INFO:teuthology.orchestra.run.smithi122.stdout:Total 527 kB/s | 169 kB 00:00 2021-12-15T17:10:30.890 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:30.901 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:30.901 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:30.947 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:30.950 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:30.999 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:30.999 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:31.055 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:31.224 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d. 1/1 2021-12-15T17:10:31.364 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:15 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:31.576 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d. 1/1 2021-12-15T17:10:31.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d. 1/1 2021-12-15T17:10:31.857 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:31.858 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:10:31.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:31.858 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:31.858 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:32.055 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2021-12-15T17:10:32.146 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:32.147 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:32.148 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2021-12-15T17:10:32.148 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:32.148 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:10:32.148 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 169 k 2021-12-15T17:10:32.149 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:32.149 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:10:32.149 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:32.149 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:10:32.150 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:32.150 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 169 k 2021-12-15T17:10:32.150 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 567 k 2021-12-15T17:10:32.150 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:10:32.237 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:32.237 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:32.422 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-immutable-object-cache-17.0.0-9665.gaa33f5 618 kB/s | 169 kB 00:00 2021-12-15T17:10:32.423 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:32.423 INFO:teuthology.orchestra.run.smithi022.stdout:Total 615 kB/s | 169 kB 00:00 2021-12-15T17:10:32.423 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:10:32.435 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:10:32.435 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:10:32.485 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:10:32.488 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:10:32.585 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:16 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:32.602 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:10:32.781 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d. 1/1 2021-12-15T17:10:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d. 1/1 2021-12-15T17:10:33.265 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:10:33.301 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:33.302 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:10:33.302 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:33.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2021-12-15T17:10:33.479 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d. 1/1 2021-12-15T17:10:33.479 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:33.479 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:10:33.480 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:10:33.480 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:33.480 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:33.593 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:33.594 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:33.681 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr 2021-12-15T17:10:33.881 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:33.883 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:33.939 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:17 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:34.247 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:18 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:34.648 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:34.649 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:34.650 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:10:34.650 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:34.650 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:10:34.650 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 1.5 M 2021-12-15T17:10:34.650 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:10:34.651 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 24 k 2021-12-15T17:10:34.651 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 15 k 2021-12-15T17:10:34.652 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-12-15T17:10:34.652 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-12-15T17:10:34.653 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-12-15T17:10:34.653 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:34.654 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:10:34.654 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:34.655 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2021-12-15T17:10:34.655 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:34.656 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 1.8 M 2021-12-15T17:10:34.657 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 7.4 M 2021-12-15T17:10:34.658 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:10:34.859 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): ceph-grafana-dashboards-17.0.0-9665.gaa3 115 kB/s | 24 kB 00:00 2021-12-15T17:10:34.876 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): ceph-prometheus-alerts-17.0.0-9665.gaa33 65 kB/s | 15 kB 00:00 2021-12-15T17:10:34.927 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 637 kB/s | 43 kB 00:00 2021-12-15T17:10:34.985 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 309 kB/s | 34 kB 00:00 2021-12-15T17:10:34.998 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-mgr-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:10:35.036 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:35.037 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:10:35.037 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:35.111 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-dashboard 2021-12-15T17:10:35.118 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): ceph-mgr-dashboard-17.0.0-9665.gaa33f58d 3.3 MB/s | 1.5 MB 00:00 2021-12-15T17:10:35.144 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 905 kB/s | 196 kB 00:00 2021-12-15T17:10:35.146 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:35.146 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.7 MB/s | 1.8 MB 00:00 2021-12-15T17:10:35.157 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:35.166 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:35.166 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:35.221 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:35.368 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:35.369 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:35.369 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:35.490 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:35.667 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:35.720 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-12-15T17:10:35.966 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-12-15T17:10:36.151 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-12-15T17:10:36.313 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-9665.gaa33f58d.el8.n 4/6 2021-12-15T17:10:36.450 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:36.451 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:36.451 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:10:36.451 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:36.451 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:10:36.452 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 1.5 M 2021-12-15T17:10:36.452 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:10:36.452 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 24 k 2021-12-15T17:10:36.452 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 15 k 2021-12-15T17:10:36.453 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-12-15T17:10:36.453 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-12-15T17:10:36.453 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-12-15T17:10:36.453 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:36.453 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:10:36.454 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:36.454 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2021-12-15T17:10:36.454 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:36.455 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 1.8 M 2021-12-15T17:10:36.455 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 7.4 M 2021-12-15T17:10:36.456 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:10:36.651 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-9665.gaa33f58d.el8. 5/6 2021-12-15T17:10:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): ceph-grafana-dashboards-17.0.0-9665.gaa3 116 kB/s | 24 kB 00:00 2021-12-15T17:10:36.683 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): ceph-prometheus-alerts-17.0.0-9665.gaa33 64 kB/s | 15 kB 00:00 2021-12-15T17:10:36.733 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 567 kB/s | 43 kB 00:00 2021-12-15T17:10:36.826 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2021-12-15T17:10:36.893 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarc 6/6 2021-12-15T17:10:37.034 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 655 kB/s | 196 kB 00:00 2021-12-15T17:10:37.076 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-mgr-dashboard-17.0.0-9665.gaa33f58d 2.5 MB/s | 1.5 MB 00:00 2021-12-15T17:10:37.078 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:37.078 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.9 MB/s | 1.8 MB 00:00 2021-12-15T17:10:37.090 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:10:37.100 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:10:37.100 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:10:37.117 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarc 6/6 2021-12-15T17:10:37.117 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-9665.gaa33f58d.el8. 1/6 2021-12-15T17:10:37.117 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarc 2/6 2021-12-15T17:10:37.118 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-9665.gaa33f58d.el8.n 3/6 2021-12-15T17:10:37.118 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-12-15T17:10:37.118 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-12-15T17:10:37.161 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:10:37.164 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:10:37.405 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:10:37.435 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-12-15T17:10:37.436 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:37.436 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:10:37.436 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:37.436 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:37.437 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:37.437 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-12-15T17:10:37.437 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-12-15T17:10:37.437 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.4.1-12.el8.noarch 2021-12-15T17:10:37.437 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:37.438 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:37.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2021-12-15T17:10:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:37.737 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-12-15T17:10:37.941 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-12-15T17:10:38.062 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:38.116 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-12-15T17:10:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-9665.gaa33f58d.el8.n 4/6 2021-12-15T17:10:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-9665.gaa33f58d.el8. 5/6 2021-12-15T17:10:38.782 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:38.783 INFO:teuthology.orchestra.run.smithi122.stdout:===================================================================================================== 2021-12-15T17:10:38.784 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:10:38.784 INFO:teuthology.orchestra.run.smithi122.stdout:===================================================================================================== 2021-12-15T17:10:38.784 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:10:38.784 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 7.4 M 2021-12-15T17:10:38.785 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:10:38.785 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 8.5.0-6.el8 CentOS-Base 643 k 2021-12-15T17:10:38.785 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 8.5.0-6.el8 CentOS-Base 171 k 2021-12-15T17:10:38.785 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-12-15T17:10:38.785 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-12-15T17:10:38.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2021-12-15T17:10:38.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2021-12-15T17:10:38.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-12-15T17:10:38.786 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:38.787 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:10:38.787 INFO:teuthology.orchestra.run.smithi122.stdout:===================================================================================================== 2021-12-15T17:10:38.787 INFO:teuthology.orchestra.run.smithi122.stdout:Install 8 Packages 2021-12-15T17:10:38.787 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:38.788 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 35 M 2021-12-15T17:10:38.788 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 210 M 2021-12-15T17:10:38.789 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:10:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarc 6/6 2021-12-15T17:10:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarc 6/6 2021-12-15T17:10:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-9665.gaa33f58d.el8. 1/6 2021-12-15T17:10:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarc 2/6 2021-12-15T17:10:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-9665.gaa33f58d.el8.n 3/6 2021-12-15T17:10:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-12-15T17:10:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-12-15T17:10:39.393 INFO:teuthology.orchestra.run.smithi122.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 7.8 MB/s | 4.7 MB 00:00 2021-12-15T17:10:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-12-15T17:10:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:39.403 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:10:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-12-15T17:10:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-12-15T17:10:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.4.1-12.el8.noarch 2021-12-15T17:10:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:39.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-diskprediction-local 2021-12-15T17:10:39.684 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:39.685 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:39.737 INFO:teuthology.orchestra.run.smithi122.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-12-15T17:10:40.044 INFO:teuthology.orchestra.run.smithi122.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 734 kB/s | 225 kB 00:00 2021-12-15T17:10:40.053 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:40.335 INFO:teuthology.orchestra.run.smithi122.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 3.1 MB/s | 4.9 MB 00:01 2021-12-15T17:10:40.470 INFO:teuthology.orchestra.run.smithi122.stdout:(5/8): libgfortran-8.5.0-6.el8.x86_64.rpm 4.6 MB/s | 643 kB 00:00 2021-12-15T17:10:40.645 INFO:teuthology.orchestra.run.smithi122.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 23 MB/s | 14 MB 00:00 2021-12-15T17:10:40.672 INFO:teuthology.orchestra.run.smithi122.stdout:(7/8): libquadmath-8.5.0-6.el8.x86_64.rpm 856 kB/s | 171 kB 00:00 2021-12-15T17:10:40.814 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:40.815 INFO:teuthology.orchestra.run.smithi022.stdout:===================================================================================================== 2021-12-15T17:10:40.815 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:10:40.815 INFO:teuthology.orchestra.run.smithi022.stdout:===================================================================================================== 2021-12-15T17:10:40.816 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:10:40.816 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 7.4 M 2021-12-15T17:10:40.816 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:10:40.816 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 8.5.0-6.el8 CentOS-Base 643 k 2021-12-15T17:10:40.817 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 8.5.0-6.el8 CentOS-Base 171 k 2021-12-15T17:10:40.817 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-12-15T17:10:40.817 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-12-15T17:10:40.817 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2021-12-15T17:10:40.817 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2021-12-15T17:10:40.818 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-12-15T17:10:40.818 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:40.818 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:10:40.818 INFO:teuthology.orchestra.run.smithi022.stdout:===================================================================================================== 2021-12-15T17:10:40.819 INFO:teuthology.orchestra.run.smithi022.stdout:Install 8 Packages 2021-12-15T17:10:40.819 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:40.820 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 35 M 2021-12-15T17:10:40.820 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 210 M 2021-12-15T17:10:40.820 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:10:41.028 INFO:teuthology.orchestra.run.smithi122.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-966 3.3 MB/s | 7.4 MB 00:02 2021-12-15T17:10:41.031 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:41.031 INFO:teuthology.orchestra.run.smithi122.stdout:Total 16 MB/s | 35 MB 00:02 2021-12-15T17:10:41.269 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:41.289 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:41.289 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:41.332 INFO:teuthology.orchestra.run.smithi022.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 9.2 MB/s | 4.7 MB 00:00 2021-12-15T17:10:41.441 INFO:teuthology.orchestra.run.smithi022.stdout:(2/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 7.8 MB/s | 4.9 MB 00:00 2021-12-15T17:10:41.474 INFO:teuthology.orchestra.run.smithi022.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.7 MB/s | 225 kB 00:00 2021-12-15T17:10:41.541 INFO:teuthology.orchestra.run.smithi022.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 18 MB/s | 3.7 MB 00:00 2021-12-15T17:10:41.583 INFO:teuthology.orchestra.run.smithi022.stdout:(5/8): libgfortran-8.5.0-6.el8.x86_64.rpm 15 MB/s | 643 kB 00:00 2021-12-15T17:10:41.601 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:41.604 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:41.616 INFO:teuthology.orchestra.run.smithi022.stdout:(6/8): libquadmath-8.5.0-6.el8.x86_64.rpm 5.0 MB/s | 171 kB 00:00 2021-12-15T17:10:41.850 INFO:teuthology.orchestra.run.smithi022.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 36 MB/s | 14 MB 00:00 2021-12-15T17:10:42.167 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-8.5.0-6.el8.x86_64 1/8 2021-12-15T17:10:42.375 INFO:teuthology.orchestra.run.smithi022.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-966 4.7 MB/s | 7.4 MB 00:01 2021-12-15T17:10:42.377 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:42.378 INFO:teuthology.orchestra.run.smithi022.stdout:Total 23 MB/s | 35 MB 00:01 2021-12-15T17:10:42.465 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libquadmath-8.5.0-6.el8.x86_64 1/8 2021-12-15T17:10:42.621 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:10:42.639 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-8.5.0-6.el8.x86_64 2/8 2021-12-15T17:10:42.641 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:10:42.641 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:10:42.961 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:10:42.964 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:10:43.234 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libgfortran-8.5.0-6.el8.x86_64 2/8 2021-12-15T17:10:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-12-15T17:10:43.525 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:10:43.645 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-8.5.0-6.el8.x86_64 1/8 2021-12-15T17:10:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libquadmath-8.5.0-6.el8.x86_64 1/8 2021-12-15T17:10:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-8.5.0-6.el8.x86_64 2/8 2021-12-15T17:10:44.020 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-12-15T17:10:44.222 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-12-15T17:10:44.593 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libgfortran-8.5.0-6.el8.x86_64 2/8 2021-12-15T17:10:44.770 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-12-15T17:10:44.891 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-12-15T17:10:45.242 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2021-12-15T17:10:45.422 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-12-15T17:10:45.447 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2021-12-15T17:10:45.595 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-12-15T17:10:46.328 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-12-15T17:10:46.735 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2021-12-15T17:10:46.928 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2021-12-15T17:10:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2021-12-15T17:10:48.469 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-12-15T17:10:48.680 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58 8/8 2021-12-15T17:10:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2021-12-15T17:10:48.930 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58 8/8 2021-12-15T17:10:48.930 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58 1/8 2021-12-15T17:10:48.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-12-15T17:10:48.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-12-15T17:10:48.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2021-12-15T17:10:48.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2021-12-15T17:10:48.932 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-12-15T17:10:48.932 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-8.5.0-6.el8.x86_64 7/8 2021-12-15T17:10:49.267 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-8.5.0-6.el8.x86_64 8/8 2021-12-15T17:10:49.268 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:49.268 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:10:49.268 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:49.269 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-8.5.0-6.el8.x86_64 2021-12-15T17:10:49.269 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-8.5.0-6.el8.x86_64 2021-12-15T17:10:49.269 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.15-3.el8.x86_64 2021-12-15T17:10:49.269 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-12-15T17:10:49.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2021-12-15T17:10:49.270 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2021-12-15T17:10:49.270 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-12-15T17:10:49.270 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:49.271 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:49.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2021-12-15T17:10:49.538 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:49.539 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:49.887 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:50.118 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-12-15T17:10:50.310 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58 8/8 2021-12-15T17:10:50.567 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58 8/8 2021-12-15T17:10:50.567 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58 1/8 2021-12-15T17:10:50.567 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-12-15T17:10:50.568 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-12-15T17:10:50.568 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2021-12-15T17:10:50.568 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2021-12-15T17:10:50.568 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-12-15T17:10:50.568 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-8.5.0-6.el8.x86_64 7/8 2021-12-15T17:10:50.598 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:50.600 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2021-12-15T17:10:50.601 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:10:50.601 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2021-12-15T17:10:50.601 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:10:50.601 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 47 k 2021-12-15T17:10:50.602 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:10:50.602 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-12-15T17:10:50.602 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-12-15T17:10:50.602 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-12-15T17:10:50.603 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-12-15T17:10:50.603 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-12-15T17:10:50.603 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-12-15T17:10:50.603 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-12-15T17:10:50.604 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-12-15T17:10:50.604 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-12-15T17:10:50.604 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-12-15T17:10:50.604 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2021-12-15T17:10:50.605 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-12-15T17:10:50.605 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:50.605 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:10:50.605 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2021-12-15T17:10:50.606 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2021-12-15T17:10:50.606 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:50.607 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 1.9 M 2021-12-15T17:10:50.607 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 25 M 2021-12-15T17:10:50.607 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:10:50.786 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 109 kB/s | 20 kB 00:00 2021-12-15T17:10:50.909 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-8.5.0-6.el8.x86_64 8/8 2021-12-15T17:10:50.910 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:50.910 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:10:50.910 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:50.910 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-8.5.0-6.el8.x86_64 2021-12-15T17:10:50.911 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-8.5.0-6.el8.x86_64 2021-12-15T17:10:50.911 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.15-3.el8.x86_64 2021-12-15T17:10:50.911 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-12-15T17:10:50.911 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2021-12-15T17:10:50.912 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2021-12-15T17:10:50.912 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-12-15T17:10:50.912 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:50.913 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:50.921 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 972 kB/s | 126 kB 00:00 2021-12-15T17:10:51.011 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-rook 2021-12-15T17:10:51.065 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 57 kB/s | 27 kB 00:00 2021-12-15T17:10:51.203 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:51.204 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:51.381 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 240 kB/s | 110 kB 00:00 2021-12-15T17:10:51.564 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:35 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:51.735 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): ceph-mgr-rook-17.0.0-9665.gaa33f58d.el8 41 kB/s | 47 kB 00:01 2021-12-15T17:10:51.981 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 71 kB/s | 43 kB 00:00 2021-12-15T17:10:52.032 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 160 kB/s | 155 kB 00:00 2021-12-15T17:10:52.099 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2021-12-15T17:10:52.116 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 87 kB/s | 33 kB 00:00 2021-12-15T17:10:52.166 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 607 kB/s | 81 kB 00:00 2021-12-15T17:10:52.183 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 919 kB/s | 61 kB 00:00 2021-12-15T17:10:52.216 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2021-12-15T17:10:52.306 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:52.308 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2021-12-15T17:10:52.308 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:10:52.308 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2021-12-15T17:10:52.309 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:10:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 47 k 2021-12-15T17:10:52.309 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:10:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-12-15T17:10:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-12-15T17:10:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-12-15T17:10:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-12-15T17:10:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-12-15T17:10:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-12-15T17:10:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-12-15T17:10:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-12-15T17:10:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-12-15T17:10:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-12-15T17:10:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2021-12-15T17:10:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-12-15T17:10:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:52.312 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:10:52.312 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2021-12-15T17:10:52.313 INFO:teuthology.orchestra.run.smithi022.stdout:Install 13 Packages 2021-12-15T17:10:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:52.314 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 1.9 M 2021-12-15T17:10:52.314 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 25 M 2021-12-15T17:10:52.314 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:10:52.342 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.8 MB/s | 1.2 MB 00:00 2021-12-15T17:10:52.344 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:52.344 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2021-12-15T17:10:52.368 INFO:teuthology.orchestra.run.smithi022.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 471 kB/s | 27 kB 00:00 2021-12-15T17:10:52.393 INFO:teuthology.orchestra.run.smithi022.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 244 kB/s | 20 kB 00:00 2021-12-15T17:10:52.405 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:52.419 INFO:teuthology.orchestra.run.smithi022.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.7 MB/s | 126 kB 00:00 2021-12-15T17:10:52.421 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:52.422 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:52.444 INFO:teuthology.orchestra.run.smithi022.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2021-12-15T17:10:52.469 INFO:teuthology.orchestra.run.smithi022.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2021-12-15T17:10:52.486 INFO:teuthology.orchestra.run.smithi022.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2021-12-15T17:10:52.494 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:52.498 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:52.587 INFO:teuthology.orchestra.run.smithi022.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 331 kB/s | 33 kB 00:00 2021-12-15T17:10:52.603 INFO:teuthology.orchestra.run.smithi022.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 140 kB/s | 16 kB 00:00 2021-12-15T17:10:52.620 INFO:teuthology.orchestra.run.smithi022.stdout:(9/13): ceph-mgr-rook-17.0.0-9665.gaa33f58d.el8 152 kB/s | 47 kB 00:00 2021-12-15T17:10:52.671 INFO:teuthology.orchestra.run.smithi022.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 968 kB/s | 81 kB 00:00 2021-12-15T17:10:52.713 INFO:teuthology.orchestra.run.smithi022.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.4 MB/s | 61 kB 00:00 2021-12-15T17:10:52.771 INFO:teuthology.orchestra.run.smithi022.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 409 kB/s | 61 kB 00:00 2021-12-15T17:10:52.855 INFO:teuthology.orchestra.run.smithi022.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.6 MB/s | 1.2 MB 00:00 2021-12-15T17:10:52.856 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:52.857 INFO:teuthology.orchestra.run.smithi022.stdout:Total 3.5 MB/s | 1.9 MB 00:00 2021-12-15T17:10:52.917 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:10:52.934 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:10:52.934 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:10:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:53.011 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:10:53.014 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:10:53.215 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-12-15T17:10:53.405 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-12-15T17:10:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:10:53.614 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2021-12-15T17:10:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-12-15T17:10:53.861 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-12-15T17:10:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-12-15T17:10:54.026 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-12-15T17:10:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2021-12-15T17:10:54.274 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-12-15T17:10:54.350 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-12-15T17:10:54.583 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-12-15T17:10:54.602 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-12-15T17:10:54.833 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-12-15T17:10:54.853 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-12-15T17:10:55.148 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-12-15T17:10:55.358 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-12-15T17:10:55.907 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-12-15T17:10:56.219 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-12-15T17:10:56.435 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-12-15T17:10:56.463 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-12-15T17:10:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-12-15T17:10:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-12-15T17:10:56.812 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 13/13 2021-12-15T17:10:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-12-15T17:10:57.082 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 13/13 2021-12-15T17:10:57.082 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 1/13 2021-12-15T17:10:57.083 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-12-15T17:10:57.083 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-12-15T17:10:57.083 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-12-15T17:10:57.083 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-12-15T17:10:57.083 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-12-15T17:10:57.084 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-12-15T17:10:57.084 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-12-15T17:10:57.084 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-12-15T17:10:57.084 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-12-15T17:10:57.085 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-12-15T17:10:57.085 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2021-12-15T17:10:57.095 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-12-15T17:10:57.243 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 13/13 2021-12-15T17:10:57.517 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-12-15T17:10:57.517 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:57.518 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:10:57.518 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:57.518 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-12-15T17:10:57.519 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-12-15T17:10:57.519 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-12-15T17:10:57.519 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-12-15T17:10:57.519 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-12-15T17:10:57.520 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-12-15T17:10:57.520 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-12-15T17:10:57.520 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-12-15T17:10:57.520 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-12-15T17:10:57.521 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-12-15T17:10:57.521 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.8-1.el8.noarch 2021-12-15T17:10:57.521 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-12-15T17:10:57.521 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:57.522 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:10:57.619 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2021-12-15T17:10:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 13/13 2021-12-15T17:10:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 1/13 2021-12-15T17:10:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-12-15T17:10:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-12-15T17:10:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-12-15T17:10:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-12-15T17:10:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-12-15T17:10:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-12-15T17:10:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-12-15T17:10:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-12-15T17:10:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-12-15T17:10:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-12-15T17:10:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2021-12-15T17:10:57.795 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:57.796 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:58.143 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-12-15T17:10:58.143 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:58.144 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:10:58.144 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:10:58.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-12-15T17:10:58.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-12-15T17:10:58.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-12-15T17:10:58.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-12-15T17:10:58.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-12-15T17:10:58.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-12-15T17:10:58.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-12-15T17:10:58.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-12-15T17:10:58.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-12-15T17:10:58.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-12-15T17:10:58.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.8-1.el8.noarch 2021-12-15T17:10:58.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-12-15T17:10:58.147 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:58.147 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:10:58.147 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:42 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:58.237 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-cephadm 2021-12-15T17:10:58.425 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:58.425 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:10:58.792 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:42 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:10:58.885 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:10:58.886 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:58.886 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:10:58.886 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:58.887 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:10:58.887 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 101 k 2021-12-15T17:10:58.887 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:10:58.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2021-12-15T17:10:58.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-12-15T17:10:58.888 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-12-15T17:10:58.888 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:58.888 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:10:58.888 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:10:58.889 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2021-12-15T17:10:58.889 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:10:58.890 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 5.9 M 2021-12-15T17:10:58.890 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 25 M 2021-12-15T17:10:58.890 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:10:59.017 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): python3-jinja2-2.10.1-3.el8.noarch.rpm 4.1 MB/s | 538 kB 00:00 2021-12-15T17:10:59.125 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2021-12-15T17:10:59.234 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.1 MB/s | 476 kB 00:00 2021-12-15T17:10:59.306 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): ceph-mgr-cephadm-17.0.0-9665.gaa33f58d.e 242 kB/s | 101 kB 00:00 2021-12-15T17:10:59.309 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:59.309 INFO:teuthology.orchestra.run.smithi122.stdout:Total 14 MB/s | 5.9 MB 00:00 2021-12-15T17:10:59.362 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:10:59.372 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:10:59.373 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:10:59.464 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:10:59.467 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:10:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:10:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:59.593 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:10:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:10:59.594 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 101 k 2021-12-15T17:10:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:10:59.595 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2021-12-15T17:10:59.595 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-12-15T17:10:59.595 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-12-15T17:10:59.595 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:10:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:10:59.597 INFO:teuthology.orchestra.run.smithi022.stdout:Install 4 Packages 2021-12-15T17:10:59.597 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:10:59.597 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 5.9 M 2021-12-15T17:10:59.597 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 25 M 2021-12-15T17:10:59.598 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:10:59.704 INFO:teuthology.orchestra.run.smithi022.stdout:(1/4): python3-jinja2-2.10.1-3.el8.noarch.rpm 4.8 MB/s | 538 kB 00:00 2021-12-15T17:10:59.778 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:10:59.805 INFO:teuthology.orchestra.run.smithi022.stdout:(2/4): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2021-12-15T17:10:59.954 INFO:teuthology.orchestra.run.smithi022.stdout:(3/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.9 MB/s | 476 kB 00:00 2021-12-15T17:10:59.980 INFO:teuthology.orchestra.run.smithi022.stdout:(4/4): ceph-mgr-cephadm-17.0.0-9665.gaa33f58d.e 263 kB/s | 101 kB 00:00 2021-12-15T17:10:59.982 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:10:59.982 INFO:teuthology.orchestra.run.smithi022.stdout:Total 15 MB/s | 5.9 MB 00:00 2021-12-15T17:11:00.037 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:00.049 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:00.049 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:00.147 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:00.151 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:00.484 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:00.749 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 1/4 2021-12-15T17:11:01.130 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 2/4 2021-12-15T17:11:01.335 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 3/4 2021-12-15T17:11:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 1/4 2021-12-15T17:11:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 4/4 2021-12-15T17:11:01.750 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 4/4 2021-12-15T17:11:01.750 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/4 2021-12-15T17:11:01.750 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/4 2021-12-15T17:11:01.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/4 2021-12-15T17:11:01.904 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 2/4 2021-12-15T17:11:02.088 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/4 2021-12-15T17:11:02.088 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:02.089 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:02.089 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:11:02.089 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2021-12-15T17:11:02.089 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.5.1-7.el8.noarch 2021-12-15T17:11:02.089 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.10.1-3.el8.noarch 2021-12-15T17:11:02.090 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:02.090 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 3/4 2021-12-15T17:11:02.180 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2021-12-15T17:11:02.250 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 4/4 2021-12-15T17:11:02.358 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:02.358 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 4/4 2021-12-15T17:11:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 1/4 2021-12-15T17:11:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/4 2021-12-15T17:11:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/4 2021-12-15T17:11:02.709 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:46 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/4 2021-12-15T17:11:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:02.799 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:11:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2021-12-15T17:11:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.5.1-7.el8.noarch 2021-12-15T17:11:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.10.1-3.el8.noarch 2021-12-15T17:11:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:02.800 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:02.902 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-fuse 2021-12-15T17:11:03.090 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:03.090 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:03.436 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:03.437 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:03.437 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:11:03.437 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:03.437 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:11:03.438 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 871 k 2021-12-15T17:11:03.438 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2021-12-15T17:11:03.438 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-12-15T17:11:03.438 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:03.439 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:11:03.439 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:03.439 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2021-12-15T17:11:03.439 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:03.440 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 954 k 2021-12-15T17:11:03.440 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 3.1 M 2021-12-15T17:11:03.440 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:11:03.452 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:47 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:03.500 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.3 MB/s | 82 kB 00:00 2021-12-15T17:11:03.886 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-17.0.0-9665.gaa33f58d.el8.x86_ 1.9 MB/s | 871 kB 00:00 2021-12-15T17:11:03.887 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:03.887 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.1 MB/s | 954 kB 00:00 2021-12-15T17:11:03.896 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:11:03.917 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:11:03.917 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:11:03.989 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:11:03.992 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:11:04.145 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:11:04.243 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:04.243 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:04.244 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:11:04.244 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:04.244 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:11:04.244 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 871 k 2021-12-15T17:11:04.245 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2021-12-15T17:11:04.245 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-12-15T17:11:04.245 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:04.245 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:11:04.245 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:04.246 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2021-12-15T17:11:04.246 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:04.247 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 954 k 2021-12-15T17:11:04.247 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 3.1 M 2021-12-15T17:11:04.247 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:11:04.312 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.2 MB/s | 82 kB 00:00 2021-12-15T17:11:04.395 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-12-15T17:11:04.585 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/2 2021-12-15T17:11:04.687 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-fuse-17.0.0-9665.gaa33f58d.el8.x86_ 1.9 MB/s | 871 kB 00:00 2021-12-15T17:11:04.688 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:04.689 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.1 MB/s | 954 kB 00:00 2021-12-15T17:11:04.701 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:04.723 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:04.724 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:04.802 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:04.805 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:04.924 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/2 2021-12-15T17:11:04.924 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/2 2021-12-15T17:11:04.968 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-12-15T17:11:05.260 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-12-15T17:11:05.260 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:05.261 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:05.261 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-12-15T17:11:05.261 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:05.261 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:05.424 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/2 2021-12-15T17:11:05.475 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2021-12-15T17:11:05.658 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:05.658 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2/2 2021-12-15T17:11:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/2 2021-12-15T17:11:06.016 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:50 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-12-15T17:11:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:06.083 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-12-15T17:11:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:06.085 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:06.284 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-volume 2021-12-15T17:11:06.476 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:06.476 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:06.735 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:06.736 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2021-12-15T17:11:06.737 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:06.737 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:11:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 256 k 2021-12-15T17:11:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:06.738 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:11:06.738 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:06.738 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:11:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:06.739 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 256 k 2021-12-15T17:11:06.739 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.2 M 2021-12-15T17:11:06.739 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:11:06.850 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:50 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:07.046 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-volume-17.0.0-9665.gaa33f58d.el8.noarch.rp 829 kB/s | 256 kB 00:00 2021-12-15T17:11:07.047 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:07.047 INFO:teuthology.orchestra.run.smithi122.stdout:Total 826 kB/s | 256 kB 00:00 2021-12-15T17:11:07.048 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:11:07.057 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:11:07.058 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:11:07.081 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:11:07.084 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:11:07.249 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:11:07.545 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:11:07.640 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:07.642 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:07.642 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2021-12-15T17:11:07.642 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:07.642 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:11:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume noarch 2:17.0.0-9665.gaa33f58d.el8 ceph-noarch 256 k 2021-12-15T17:11:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:07.643 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:11:07.643 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:07.643 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:11:07.644 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:07.644 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 256 k 2021-12-15T17:11:07.644 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.2 M 2021-12-15T17:11:07.645 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:11:07.903 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:11:07.970 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-volume-17.0.0-9665.gaa33f58d.el8.noarch.rp 781 kB/s | 256 kB 00:00 2021-12-15T17:11:07.971 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:07.973 INFO:teuthology.orchestra.run.smithi022.stdout:Total 778 kB/s | 256 kB 00:00 2021-12-15T17:11:07.973 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:07.987 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:07.987 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:08.007 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:08.010 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:08.184 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:08.266 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:11:08.267 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:08.267 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:08.267 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:11:08.267 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:08.268 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:08.391 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:11:08.440 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2021-12-15T17:11:08.623 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:08.624 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:08.747 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:11:08.975 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:52 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:09.156 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 1/1 2021-12-15T17:11:09.156 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:09.156 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:09.157 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume-2:17.0.0-9665.gaa33f58d.el8.noarch 2021-12-15T17:11:09.157 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:09.157 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:09.354 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados-devel 2021-12-15T17:11:09.547 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:09.547 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:09.698 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:09.699 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:09.700 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2021-12-15T17:11:09.700 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:09.700 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:11:09.700 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 125 k 2021-12-15T17:11:09.700 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:09.701 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:11:09.701 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:09.701 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:11:09.701 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:09.702 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 125 k 2021-12-15T17:11:09.702 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 432 k 2021-12-15T17:11:09.702 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:11:09.915 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:53 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:10.002 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-17.0.0-9665.gaa33f58d.el8.x86_64 413 kB/s | 125 kB 00:00 2021-12-15T17:11:10.003 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:10.003 INFO:teuthology.orchestra.run.smithi122.stdout:Total 411 kB/s | 125 kB 00:00 2021-12-15T17:11:10.003 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:11:10.009 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:11:10.009 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:11:10.057 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:11:10.060 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:11:10.165 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:11:10.350 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:10.584 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:10.711 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:10.712 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:10.712 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2021-12-15T17:11:10.712 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:10.712 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:11:10.713 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 125 k 2021-12-15T17:11:10.713 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:10.713 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:11:10.713 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:10.714 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:11:10.714 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:10.715 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 125 k 2021-12-15T17:11:10.715 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 432 k 2021-12-15T17:11:10.715 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:11:10.881 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:10.881 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:10.881 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:10.882 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:10.882 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:10.882 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:10.995 INFO:teuthology.orchestra.run.smithi022.stdout:librados-devel-17.0.0-9665.gaa33f58d.el8.x86_64 442 kB/s | 125 kB 00:00 2021-12-15T17:11:10.996 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:10.996 INFO:teuthology.orchestra.run.smithi022.stdout:Total 440 kB/s | 125 kB 00:00 2021-12-15T17:11:10.996 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:11.002 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:11.003 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:11.055 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:11.058 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:11.068 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2021-12-15T17:11:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:11.247 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:11.248 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:11.342 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:11.600 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:55 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:11.898 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:11.901 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:12.075 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs2 2021-12-15T17:11:12.265 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:12.266 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:12.298 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:12.334 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:12.335 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:11:12.335 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:12.418 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2021-12-15T17:11:12.595 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:12.596 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:12.638 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:56 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:12.947 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:56 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:13.395 INFO:teuthology.orchestra.run.smithi022.stdout:Package libcephfs2-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:13.435 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:13.436 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:11:13.436 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:13.517 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs-devel 2021-12-15T17:11:13.663 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:13.665 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:13.665 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2021-12-15T17:11:13.666 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:13.666 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:11:13.666 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 24 k 2021-12-15T17:11:13.666 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:13.667 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:11:13.667 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:13.667 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:11:13.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:13.667 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 24 k 2021-12-15T17:11:13.668 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 106 k 2021-12-15T17:11:13.668 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:11:13.705 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:13.705 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:13.872 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-17.0.0-9665.gaa33f58d.el8.x86_6 118 kB/s | 24 kB 00:00 2021-12-15T17:11:13.873 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:13.873 INFO:teuthology.orchestra.run.smithi122.stdout:Total 118 kB/s | 24 kB 00:00 2021-12-15T17:11:13.873 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:11:13.877 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:11:13.877 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:11:13.879 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:11:13.882 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:11:14.000 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:11:14.075 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:58 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:14.158 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:14.408 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:14.695 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:14.695 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:14.696 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:14.696 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:14.696 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:14.696 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:14.762 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2021-12-15T17:11:14.902 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:14.903 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:14.904 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2021-12-15T17:11:14.904 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:14.904 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:11:14.904 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 24 k 2021-12-15T17:11:14.905 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:14.905 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:11:14.905 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:14.905 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:11:14.905 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:14.906 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 24 k 2021-12-15T17:11:14.906 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 106 k 2021-12-15T17:11:14.906 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:11:14.940 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:14.940 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:15.106 INFO:teuthology.orchestra.run.smithi022.stdout:libcephfs-devel-17.0.0-9665.gaa33f58d.el8.x86_6 121 kB/s | 24 kB 00:00 2021-12-15T17:11:15.107 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:15.108 INFO:teuthology.orchestra.run.smithi022.stdout:Total 121 kB/s | 24 kB 00:00 2021-12-15T17:11:15.108 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:15.112 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:15.113 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:15.114 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:15.118 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:15.290 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:59 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:15.581 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:15.880 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:15.881 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:15.954 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados2 2021-12-15T17:11:15.967 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:16.003 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:16.004 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:11:16.004 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:16.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2021-12-15T17:11:16.143 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:16.144 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:16.243 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:16.244 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:16.510 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:00 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:16.594 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:00 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:17.258 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:17.286 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:17.296 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:17.297 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:11:17.298 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:17.321 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:17.322 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:11:17.322 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:17.358 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librbd1 2021-12-15T17:11:17.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2021-12-15T17:11:17.545 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:17.546 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:17.555 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:17.555 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:17.905 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:01 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:01 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:18.608 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:18.643 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:18.644 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:11:18.644 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:18.670 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:18.700 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2021-12-15T17:11:18.708 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:18.709 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:11:18.710 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:18.760 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rados 2021-12-15T17:11:18.877 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:18.877 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:18.949 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:18.950 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:19.227 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:19.314 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:19.922 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:19.956 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:19.958 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:11:19.958 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:20.076 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rados-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:20.113 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:20.115 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:11:20.115 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:20.380 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2021-12-15T17:11:20.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rgw 2021-12-15T17:11:20.557 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:20.557 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:20.610 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:20.611 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:20.907 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:04 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:20.978 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:04 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:21.596 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:21.631 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:21.632 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:11:21.632 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:21.691 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2021-12-15T17:11:21.723 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rgw-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:21.761 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:21.763 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:11:21.763 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:21.824 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-cephfs 2021-12-15T17:11:21.868 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:21.869 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:22.011 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:22.011 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:22.215 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:22.378 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:22.878 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:22.913 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:22.914 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2021-12-15T17:11:22.914 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:22.972 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2021-12-15T17:11:23.132 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-cephfs-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:23.170 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:23.171 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:11:23.171 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:23.237 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rbd 2021-12-15T17:11:23.424 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:23.424 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:23.499 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:07 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:23.791 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:07 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:24.235 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:24.236 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:24.236 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2021-12-15T17:11:24.237 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:24.237 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:11:24.237 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 93 k 2021-12-15T17:11:24.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:24.238 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:11:24.238 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:24.238 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:11:24.239 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:24.239 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 93 k 2021-12-15T17:11:24.239 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 281 k 2021-12-15T17:11:24.240 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:11:24.536 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-17.0.0-9665.gaa33f58d.el8.x86_64.rpm 312 kB/s | 93 kB 00:00 2021-12-15T17:11:24.537 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:24.538 INFO:teuthology.orchestra.run.smithi122.stdout:Total 310 kB/s | 93 kB 00:00 2021-12-15T17:11:24.538 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:11:24.541 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 is already installed. 2021-12-15T17:11:24.547 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:11:24.547 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:11:24.580 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:24.581 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2021-12-15T17:11:24.581 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:24.595 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:11:24.598 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:11:24.637 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-fuse 2021-12-15T17:11:24.699 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:11:24.823 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:24.824 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:24.931 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:25.192 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:09 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:25.195 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:25.543 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:25.543 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:25.543 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:25.544 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:25.544 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:25.544 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:25.703 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2021-12-15T17:11:25.881 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:25.882 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:25.980 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:25.981 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:25.981 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2021-12-15T17:11:25.982 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:25.982 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:11:25.982 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 93 k 2021-12-15T17:11:25.983 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:25.983 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:11:25.983 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:25.983 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:11:25.984 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:25.984 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 93 k 2021-12-15T17:11:25.984 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 281 k 2021-12-15T17:11:25.984 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:11:26.234 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:10 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:26.246 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-fuse-17.0.0-9665.gaa33f58d.el8.x86_64.rpm 354 kB/s | 93 kB 00:00 2021-12-15T17:11:26.246 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:26.247 INFO:teuthology.orchestra.run.smithi022.stdout:Total 352 kB/s | 93 kB 00:00 2021-12-15T17:11:26.247 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:26.258 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:26.258 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:26.310 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:26.313 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:26.595 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:26.833 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:26.976 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:26.977 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:26.977 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2021-12-15T17:11:26.977 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:26.977 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:11:26.978 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.8 M 2021-12-15T17:11:26.978 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:26.978 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:11:26.978 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:26.978 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:11:26.979 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:26.979 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.8 M 2021-12-15T17:11:26.979 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 19 M 2021-12-15T17:11:26.979 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:11:27.169 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:27.170 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:27.170 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:27.170 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:27.170 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:27.171 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:27.329 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-mirror 2021-12-15T17:11:27.516 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:27.517 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:27.888 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:28.208 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-17.0.0-9665.gaa33f58d.el8.x86_64.rpm 3.1 MB/s | 3.8 MB 00:01 2021-12-15T17:11:28.208 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:28.209 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.1 MB/s | 3.8 MB 00:01 2021-12-15T17:11:28.209 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:11:28.222 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:11:28.222 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:11:28.287 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:11:28.290 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:11:28.674 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:28.675 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2021-12-15T17:11:28.676 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:28.676 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:11:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 3.8 M 2021-12-15T17:11:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:28.677 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:11:28.677 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:28.677 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:11:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:28.677 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 3.8 M 2021-12-15T17:11:28.678 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 19 M 2021-12-15T17:11:28.678 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:11:28.772 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:11:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:29.291 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:29.598 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-mirror-17.0.0-9665.gaa33f58d.el8.x86_64.rpm 4.1 MB/s | 3.8 MB 00:00 2021-12-15T17:11:29.599 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:29.600 INFO:teuthology.orchestra.run.smithi022.stdout:Total 4.1 MB/s | 3.8 MB 00:00 2021-12-15T17:11:29.600 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:29.613 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:29.613 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:29.640 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:29.640 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:29.640 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:29.641 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:29.641 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:29.641 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:29.681 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:29.684 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:29.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2021-12-15T17:11:30.038 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:30.038 INFO:teuthology.orchestra.run.smithi122.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:30.363 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:30.393 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:14 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:31.114 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:31.115 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:31.115 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:31.116 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:31.116 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:31.116 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:31.118 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2021-12-15T17:11:31.119 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:31.120 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2021-12-15T17:11:31.121 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:31.122 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2021-12-15T17:11:31.122 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 175 k 2021-12-15T17:11:31.122 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:31.123 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2021-12-15T17:11:31.123 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2021-12-15T17:11:31.123 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2021-12-15T17:11:31.123 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:31.123 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 175 k 2021-12-15T17:11:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 568 k 2021-12-15T17:11:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2021-12-15T17:11:31.265 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-nbd 2021-12-15T17:11:31.415 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-17.0.0-9665.gaa33f58d.el8.x86_64.rpm 587 kB/s | 175 kB 00:00 2021-12-15T17:11:31.415 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:31.416 INFO:teuthology.orchestra.run.smithi122.stdout:Total 584 kB/s | 175 kB 00:00 2021-12-15T17:11:31.416 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2021-12-15T17:11:31.427 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2021-12-15T17:11:31.427 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2021-12-15T17:11:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2021-12-15T17:11:31.475 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2021-12-15T17:11:31.478 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2021-12-15T17:11:31.595 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2021-12-15T17:11:31.828 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 Dec 2021 05:09:16 PM UTC. 2021-12-15T17:11:31.831 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:32.073 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:32.395 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:32.395 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:32.395 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2021-12-15T17:11:32.396 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:32.396 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:11:32.396 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2021-12-15T17:11:32.588 DEBUG:teuthology.parallel:result is None 2021-12-15T17:11:32.613 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2021-12-15T17:11:32.614 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:32.614 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2021-12-15T17:11:32.615 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:32.615 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2021-12-15T17:11:32.615 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:17.0.0-9665.gaa33f58d.el8 ceph 175 k 2021-12-15T17:11:32.615 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:32.615 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2021-12-15T17:11:32.616 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2021-12-15T17:11:32.616 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2021-12-15T17:11:32.616 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:32.616 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 175 k 2021-12-15T17:11:32.617 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 568 k 2021-12-15T17:11:32.617 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2021-12-15T17:11:32.920 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-nbd-17.0.0-9665.gaa33f58d.el8.x86_64.rpm 573 kB/s | 175 kB 00:00 2021-12-15T17:11:32.921 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2021-12-15T17:11:32.922 INFO:teuthology.orchestra.run.smithi022.stdout:Total 571 kB/s | 175 kB 00:00 2021-12-15T17:11:32.922 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2021-12-15T17:11:32.934 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2021-12-15T17:11:32.934 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2021-12-15T17:11:32.987 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2021-12-15T17:11:32.989 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2021-12-15T17:11:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2021-12-15T17:11:33.293 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:33.519 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 1/1 2021-12-15T17:11:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:33.886 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2021-12-15T17:11:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:17.0.0-9665.gaa33f58d.el8.x86_64 2021-12-15T17:11:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:11:33.886 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2021-12-15T17:11:34.084 DEBUG:teuthology.parallel:result is None 2021-12-15T17:11:34.085 DEBUG:teuthology.orchestra.run.smithi022:> sudo lsb_release -is 2021-12-15T17:11:34.120 INFO:teuthology.orchestra.run.smithi022.stdout:CentOSStream 2021-12-15T17:11:34.122 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-12-15T17:11:34.123 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 2021-12-15T17:11:34.345 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-12-15T17:11:34.367 INFO:teuthology.orchestra.run.smithi022.stdout:17.0.0-9665.gaa33f58d.el8 2021-12-15T17:11:34.368 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-9665.gaa33f58d.el8 2021-12-15T17:11:34.369 INFO:teuthology.task.install:The correct ceph version 17.0.0-9665.gaa33f58d is installed. 2021-12-15T17:11:34.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo lsb_release -is 2021-12-15T17:11:34.412 INFO:teuthology.orchestra.run.smithi122.stdout:CentOSStream 2021-12-15T17:11:34.413 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-12-15T17:11:34.414 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 2021-12-15T17:11:34.620 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-12-15T17:11:34.647 INFO:teuthology.orchestra.run.smithi122.stdout:17.0.0-9665.gaa33f58d.el8 2021-12-15T17:11:34.647 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-9665.gaa33f58d.el8 2021-12-15T17:11:34.648 INFO:teuthology.task.install:The correct ceph version 17.0.0-9665.gaa33f58d is installed. 2021-12-15T17:11:34.649 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-12-15T17:11:34.650 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:11:34.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-12-15T17:11:34.679 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:11:34.680 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-12-15T17:11:34.718 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-12-15T17:11:34.719 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:11:34.719 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2021-12-15T17:11:34.747 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-12-15T17:11:34.815 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:11:34.815 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2021-12-15T17:11:34.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-12-15T17:11:34.919 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-12-15T17:11:34.919 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:11:34.920 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2021-12-15T17:11:34.948 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-12-15T17:11:35.016 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:11:35.016 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2021-12-15T17:11:35.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-12-15T17:11:35.115 INFO:teuthology.run_tasks:Running task cephadm... 2021-12-15T17:11:35.188 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': 'aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2', 'cephadm_mode': 'cephadm-package'} 2021-12-15T17:11:35.189 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 2021-12-15T17:11:35.190 INFO:tasks.cephadm:Cluster fsid is 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:11:35.190 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-12-15T17:11:35.191 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.22:6789]', 'mon.c': '[v1:172.21.15.22:6790]', 'mon.b': '[v1:172.21.15.122:6789]'} 2021-12-15T17:11:35.191 INFO:tasks.cephadm:First mon is mon.a on smithi022 2021-12-15T17:11:35.191 INFO:tasks.cephadm:First mgr is y 2021-12-15T17:11:35.192 INFO:tasks.cephadm:Normalizing hostnames... 2021-12-15T17:11:35.192 DEBUG:teuthology.orchestra.run.smithi022:> sudo hostname $(hostname -s) 2021-12-15T17:11:35.221 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2021-12-15T17:11:35.262 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 on all hosts... 2021-12-15T17:11:35.263 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 pull 2021-12-15T17:11:35.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 pull 2021-12-15T17:11:37.251 INFO:teuthology.orchestra.run.smithi022.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2... 2021-12-15T17:11:37.344 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2... 2021-12-15T17:12:05.403 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2021-12-15T17:12:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 17.0.0-9665-gaa33f58d (aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2) quincy (dev)", 2021-12-15T17:12:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "33a7cf5d8f62fb786fd446e6299f29cfb9f84871c954151e9745658e354ae99e", 2021-12-15T17:12:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2021-12-15T17:12:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c5c5f1dba7e3feb75377df5dd1b6e22ae9bf1b0e6827ad0511f3295524d50e0b" 2021-12-15T17:12:05.405 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2021-12-15T17:12:05.405 INFO:teuthology.orchestra.run.smithi122.stdout:} 2021-12-15T17:12:06.414 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2021-12-15T17:12:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_version": "ceph version 17.0.0-9665-gaa33f58d (aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2) quincy (dev)", 2021-12-15T17:12:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "image_id": "33a7cf5d8f62fb786fd446e6299f29cfb9f84871c954151e9745658e354ae99e", 2021-12-15T17:12:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "repo_digests": [ 2021-12-15T17:12:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c5c5f1dba7e3feb75377df5dd1b6e22ae9bf1b0e6827ad0511f3295524d50e0b" 2021-12-15T17:12:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2021-12-15T17:12:06.417 INFO:teuthology.orchestra.run.smithi022.stdout:} 2021-12-15T17:12:06.437 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph 2021-12-15T17:12:06.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2021-12-15T17:12:06.508 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /etc/ceph 2021-12-15T17:12:06.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2021-12-15T17:12:06.579 INFO:tasks.cephadm:Writing seed config... 2021-12-15T17:12:06.581 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-12-15T17:12:06.581 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-12-15T17:12:06.581 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2021-12-15T17:12:06.582 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2021-12-15T17:12:06.582 INFO:tasks.cephadm: override: [global] ms type = async 2021-12-15T17:12:06.582 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-12-15T17:12:06.582 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-12-15T17:12:06.583 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-12-15T17:12:06.583 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-12-15T17:12:06.583 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-12-15T17:12:06.584 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2021-12-15T17:12:06.584 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:12:06.585 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-12-15T17:12:06.603 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0ea1c4a2-5dca-11ec-8c31-001a4aab830c mon election default strategy = 1 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2021-12-15T17:12:06.604 DEBUG:teuthology.orchestra.run.smithi022:mon.a> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mon.a.service 2021-12-15T17:12:06.650 DEBUG:teuthology.orchestra.run.smithi022:mgr.y> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mgr.y.service 2021-12-15T17:12:06.675 INFO:journalctl@ceph.mon.a.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:12:06.676 INFO:tasks.cephadm:Bootstrapping... 2021-12-15T17:12:06.677 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 -v bootstrap --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.22:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-12-15T17:12:06.700 INFO:journalctl@ceph.mgr.y.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:12:06.871 INFO:teuthology.orchestra.run.smithi022.stderr:-------------------------------------------------------------------------------- 2021-12-15T17:12:06.871 INFO:teuthology.orchestra.run.smithi022.stderr:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2', '-v', 'bootstrap', '--fsid', '0ea1c4a2-5dca-11ec-8c31-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.22:6789]', '--skip-admin-label'] 2021-12-15T17:12:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: 3.4.1-dev 2021-12-15T17:12:07.010 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying podman|docker is present... 2021-12-15T17:12:07.048 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: 3.4.1-dev 2021-12-15T17:12:07.150 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying lvm2 is present... 2021-12-15T17:12:07.151 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying time synchronization is in place... 2021-12-15T17:12:07.157 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-12-15T17:12:07.162 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2021-12-15T17:12:07.168 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: enabled 2021-12-15T17:12:07.174 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: active 2021-12-15T17:12:07.175 INFO:teuthology.orchestra.run.smithi022.stderr:Unit chronyd.service is enabled and running 2021-12-15T17:12:07.175 INFO:teuthology.orchestra.run.smithi022.stderr:Repeating the final host check... 2021-12-15T17:12:07.212 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: 3.4.1-dev 2021-12-15T17:12:07.320 INFO:teuthology.orchestra.run.smithi022.stderr:podman (/bin/podman) version 3.4.1 is present 2021-12-15T17:12:07.320 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl is present 2021-12-15T17:12:07.321 INFO:teuthology.orchestra.run.smithi022.stderr:lvcreate is present 2021-12-15T17:12:07.325 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-12-15T17:12:07.331 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2021-12-15T17:12:07.337 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: enabled 2021-12-15T17:12:07.342 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: active 2021-12-15T17:12:07.344 INFO:teuthology.orchestra.run.smithi022.stderr:Unit chronyd.service is enabled and running 2021-12-15T17:12:07.345 INFO:teuthology.orchestra.run.smithi022.stderr:Host looks OK 2021-12-15T17:12:07.345 INFO:teuthology.orchestra.run.smithi022.stderr:Cluster fsid: 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:07.346 INFO:teuthology.orchestra.run.smithi022.stderr:Acquiring lock 140396726897128 on /run/cephadm/0ea1c4a2-5dca-11ec-8c31-001a4aab830c.lock 2021-12-15T17:12:07.347 INFO:teuthology.orchestra.run.smithi022.stderr:Lock 140396726897128 acquired on /run/cephadm/0ea1c4a2-5dca-11ec-8c31-001a4aab830c.lock 2021-12-15T17:12:07.347 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying IP 172.21.15.22 port 6789 ... 2021-12-15T17:12:07.347 INFO:teuthology.orchestra.run.smithi022.stderr:Base mon IP is 172.21.15.22, final addrv is [v1:172.21.15.22:6789] 2021-12-15T17:12:07.353 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2021-12-15T17:12:07.353 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.22 metric 100 2021-12-15T17:12:07.354 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-12-15T17:12:07.354 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2021-12-15T17:12:07.354 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2021-12-15T17:12:07.354 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-12-15T17:12:07.355 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-12-15T17:12:07.355 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-12-15T17:12:07.355 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2021-12-15T17:12:07.355 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe59:5de4/64 scope link 2021-12-15T17:12:07.355 INFO:teuthology.orchestra.run.smithi022.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-12-15T17:12:07.356 INFO:teuthology.orchestra.run.smithi022.stderr:Mon IP `172.21.15.22` is in CIDR network `172.21.0.0/20` 2021-12-15T17:12:07.356 INFO:teuthology.orchestra.run.smithi022.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-12-15T17:12:07.356 INFO:teuthology.orchestra.run.smithi022.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2... 2021-12-15T17:12:07.425 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Trying to pull quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2... 2021-12-15T17:12:07.515 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Getting image source signatures 2021-12-15T17:12:07.515 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying blob sha256:daf93ff923e169c2c226a86602ca3e5971ec999d248a7d763d136af662a1d227 2021-12-15T17:12:07.516 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying blob sha256:ab4052451ca36406e833b1fc81e1891fe75e551aff2bb9b04321fef9b0328a3a 2021-12-15T17:12:07.516 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Copying config sha256:33a7cf5d8f62fb786fd446e6299f29cfb9f84871c954151e9745658e354ae99e 2021-12-15T17:12:07.563 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Writing manifest to image destination 2021-12-15T17:12:07.564 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: Storing signatures 2021-12-15T17:12:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:/bin/podman: 33a7cf5d8f62fb786fd446e6299f29cfb9f84871c954151e9745658e354ae99e 2021-12-15T17:12:08.100 INFO:teuthology.orchestra.run.smithi022.stderr:ceph: ceph version 17.0.0-9665-gaa33f58d (aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2) quincy (dev) 2021-12-15T17:12:08.551 INFO:teuthology.orchestra.run.smithi022.stderr:Ceph version: ceph version 17.0.0-9665-gaa33f58d (aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2) quincy (dev) 2021-12-15T17:12:08.551 INFO:teuthology.orchestra.run.smithi022.stderr:Extracting ceph user uid/gid from container image... 2021-12-15T17:12:08.909 INFO:teuthology.orchestra.run.smithi022.stderr:stat: 167 167 2021-12-15T17:12:09.196 INFO:teuthology.orchestra.run.smithi022.stderr:Creating initial keys... 2021-12-15T17:12:09.575 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph-authtool: AQDpIbph07IFIhAAEJKTLJ0U5V8m2u61UGu1iQ== 2021-12-15T17:12:10.185 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph-authtool: AQDqIbph4HSdChAATeYOPIRnqDLY59p6FWMjaQ== 2021-12-15T17:12:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph-authtool: AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:12:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:Creating initial monmap... 2021-12-15T17:12:11.475 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-12-15T17:12:11.476 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2021-12-15T17:12:11.476 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:11.476 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-12-15T17:12:11.764 INFO:teuthology.orchestra.run.smithi022.stderr:monmaptool for a [v1:172.21.15.22:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-12-15T17:12:11.765 INFO:teuthology.orchestra.run.smithi022.stderr:setting min_mon_release = octopus 2021-12-15T17:12:11.765 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: set fsid to 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:11.765 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-12-15T17:12:11.765 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:12:11.766 INFO:teuthology.orchestra.run.smithi022.stderr:Creating mon... 2021-12-15T17:12:12.509 INFO:teuthology.orchestra.run.smithi022.stderr:create mon.a on 2021-12-15T17:12:12.522 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-12-15T17:12:12.621 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-12-15T17:12:12.718 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c.target → /etc/systemd/system/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c.target. 2021-12-15T17:12:12.719 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c.target → /etc/systemd/system/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c.target. 2021-12-15T17:12:12.934 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to reset failed state of unit ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mon.a.service: Unit ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mon.a.service not loaded. 2021-12-15T17:12:12.943 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c.target.wants/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@.service. 2021-12-15T17:12:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:13 smithi022 systemd[1]: Starting Ceph mon.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:12:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:13 smithi022 podman[30456]: 2021-12-15T17:12:15.056 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2021-12-15T17:12:15.065 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2021-12-15T17:12:15.065 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2021-12-15T17:12:15.066 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to enable service . firewalld.service is not available 2021-12-15T17:12:15.066 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mon to start... 2021-12-15T17:12:15.067 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mon... 2021-12-15T17:12:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:14 smithi022 ceph-mon[30520]: mkfs 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:14 smithi022 ceph-mon[30520]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:12:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:15 smithi022 bash[30304]: d42fe795ceddfcb7bbc30b1db76df7ebde86a7e5399a317b70909cea4516cde3 2021-12-15T17:12:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:15 smithi022 systemd[1]: Started Ceph mon.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:12:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:15 smithi022 ceph-mon[30520]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:12:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:15 smithi022 ceph-mon[30520]: monmap e1: 1 mons at {a=v1:172.21.15.22:6789/0} 2021-12-15T17:12:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:15 smithi022 ceph-mon[30520]: fsmap 2021-12-15T17:12:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:15 smithi022 ceph-mon[30520]: osdmap e1: 0 total, 0 up, 0 in 2021-12-15T17:12:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:15 smithi022 ceph-mon[30520]: mgrmap e1: no daemons active 2021-12-15T17:12:16.592 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: cluster: 2021-12-15T17:12:16.592 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: id: 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:16.592 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-12-15T17:12:16.593 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:16.593 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: services: 2021-12-15T17:12:16.594 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 1.70781s) 2021-12-15T17:12:16.594 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mgr: no daemons active 2021-12-15T17:12:16.594 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-12-15T17:12:16.595 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:16.595 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: data: 2021-12-15T17:12:16.595 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-12-15T17:12:16.595 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-12-15T17:12:16.596 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-12-15T17:12:16.596 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: pgs: 2021-12-15T17:12:16.596 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:16.917 INFO:teuthology.orchestra.run.smithi022.stderr:mon is available 2021-12-15T17:12:16.918 INFO:teuthology.orchestra.run.smithi022.stderr:Assimilating anything we can from ceph.conf... 2021-12-15T17:12:17.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:16 smithi022 ceph-mon[30520]: from='client.? v1:172.21.15.22:0/3266039324' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-12-15T17:12:17.470 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:17.472 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [global] 2021-12-15T17:12:17.472 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: fsid = 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:17.472 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_host = [v1:172.21.15.22:6789] 2021-12-15T17:12:17.473 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-12-15T17:12:17.473 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-12-15T17:12:17.473 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-12-15T17:12:17.473 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-12-15T17:12:17.473 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:17.474 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [mgr] 2021-12-15T17:12:17.474 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-12-15T17:12:17.474 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:17.474 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [osd] 2021-12-15T17:12:17.475 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-12-15T17:12:17.475 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-12-15T17:12:17.755 INFO:teuthology.orchestra.run.smithi022.stderr:Generating new minimal ceph.conf... 2021-12-15T17:12:18.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:17 smithi022 ceph-mon[30520]: from='client.? v1:172.21.15.22:0/2931260039' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-12-15T17:12:18.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:17 smithi022 ceph-mon[30520]: from='client.? v1:172.21.15.22:0/2931260039' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-12-15T17:12:18.591 INFO:teuthology.orchestra.run.smithi022.stderr:Restarting the monitor... 2021-12-15T17:12:18.976 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:18 smithi022 systemd[1]: Stopping Ceph mon.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:12:18.976 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:18 smithi022 bash[30973]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-mon.a" found: no such container 2021-12-15T17:12:18.976 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:18 smithi022 conmon[30496]: 2021-12-15T17:12:18.740+0000 7f9cb160e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:12:18.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:18 smithi022 conmon[30496]: 2021-12-15T17:12:18.740+0000 7f9cb160e700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-12-15T17:12:19.227 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 bash[30973]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-mon-a 2021-12-15T17:12:19.227 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 bash[30973]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-mon.a" found: no such container 2021-12-15T17:12:19.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mon.a.service: Succeeded. 2021-12-15T17:12:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 systemd[1]: Stopped Ceph mon.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:12:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 systemd[1]: Starting Ceph mon.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:12:19.729 INFO:teuthology.orchestra.run.smithi022.stderr:Setting mon public_network to 172.21.0.0/20 2021-12-15T17:12:19.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 podman[31296]: 2021-12-15T17:12:19.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: set uid:gid to 167:167 (ceph:ceph) 2021-12-15T17:12:19.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: ceph version 17.0.0-9665-gaa33f58d (aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2) quincy (dev), process ceph-mon, pid 7 2021-12-15T17:12:19.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: pidfile_write: ignore empty --pid-file 2021-12-15T17:12:19.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: load: jerasure load: lrc 2021-12-15T17:12:19.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: RocksDB version: 6.15.5 2021-12-15T17:12:19.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-12-15T17:12:19.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Compile date Dec 14 2021 2021-12-15T17:12:19.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: DB SUMMARY 2021-12-15T17:12:19.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: DB Session ID: ECSRTGXF09QL25HTU768 2021-12-15T17:12:19.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: CURRENT file: CURRENT 2021-12-15T17:12:19.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: IDENTITY file: IDENTITY 2021-12-15T17:12:19.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2021-12-15T17:12:19.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2021-12-15T17:12:19.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 80862 ; 2021-12-15T17:12:19.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.error_if_exists: 0 2021-12-15T17:12:19.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.create_if_missing: 0 2021-12-15T17:12:19.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.paranoid_checks: 1 2021-12-15T17:12:19.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2021-12-15T17:12:19.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.env: 0x55b382fe7da0 2021-12-15T17:12:19.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.fs: Posix File System 2021-12-15T17:12:19.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.info_log: 0x55b3855221c0 2021-12-15T17:12:19.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_file_opening_threads: 16 2021-12-15T17:12:19.788 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.statistics: (nil) 2021-12-15T17:12:19.788 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.use_fsync: 0 2021-12-15T17:12:19.788 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_log_file_size: 0 2021-12-15T17:12:19.788 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_manifest_file_size: 1073741824 2021-12-15T17:12:19.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.log_file_time_to_roll: 0 2021-12-15T17:12:19.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.keep_log_file_num: 1000 2021-12-15T17:12:19.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.recycle_log_file_num: 0 2021-12-15T17:12:19.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.allow_fallocate: 1 2021-12-15T17:12:19.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.allow_mmap_reads: 0 2021-12-15T17:12:19.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.allow_mmap_writes: 0 2021-12-15T17:12:19.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.use_direct_reads: 0 2021-12-15T17:12:19.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-12-15T17:12:19.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.create_missing_column_families: 0 2021-12-15T17:12:19.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.db_log_dir: 2021-12-15T17:12:19.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-12-15T17:12:19.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.table_cache_numshardbits: 6 2021-12-15T17:12:19.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.WAL_ttl_seconds: 0 2021-12-15T17:12:19.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.WAL_size_limit_MB: 0 2021-12-15T17:12:19.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-12-15T17:12:19.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.manifest_preallocation_size: 4194304 2021-12-15T17:12:19.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.is_fd_close_on_exec: 1 2021-12-15T17:12:19.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.advise_random_on_open: 1 2021-12-15T17:12:19.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.db_write_buffer_size: 0 2021-12-15T17:12:19.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.write_buffer_manager: 0x55b385600060 2021-12-15T17:12:19.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.access_hint_on_compaction_start: 1 2021-12-15T17:12:19.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-12-15T17:12:19.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.random_access_max_buffer_size: 1048576 2021-12-15T17:12:19.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.use_adaptive_mutex: 0 2021-12-15T17:12:19.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.rate_limiter: (nil) 2021-12-15T17:12:19.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-12-15T17:12:19.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.wal_recovery_mode: 2 2021-12-15T17:12:19.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.enable_thread_tracking: 0 2021-12-15T17:12:19.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.enable_pipelined_write: 0 2021-12-15T17:12:19.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.unordered_write: 0 2021-12-15T17:12:19.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.allow_concurrent_memtable_write: 1 2021-12-15T17:12:19.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-12-15T17:12:19.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.write_thread_max_yield_usec: 100 2021-12-15T17:12:19.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.write_thread_slow_yield_usec: 3 2021-12-15T17:12:19.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.row_cache: None 2021-12-15T17:12:19.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.wal_filter: None 2021-12-15T17:12:19.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.avoid_flush_during_recovery: 0 2021-12-15T17:12:19.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.allow_ingest_behind: 0 2021-12-15T17:12:19.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 bash[31144]: 2e26a30b05438f293c8a7e7e595d9c45793b7e78cab387edce1ced90a5a42165 2021-12-15T17:12:19.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.preserve_deletes: 0 2021-12-15T17:12:19.798 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.two_write_queues: 0 2021-12-15T17:12:19.798 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.manual_wal_flush: 0 2021-12-15T17:12:19.798 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.atomic_flush: 0 2021-12-15T17:12:19.798 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-12-15T17:12:19.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.persist_stats_to_disk: 0 2021-12-15T17:12:19.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.write_dbid_to_manifest: 0 2021-12-15T17:12:19.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.log_readahead_size: 0 2021-12-15T17:12:19.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.file_checksum_gen_factory: Unknown 2021-12-15T17:12:19.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.best_efforts_recovery: 0 2021-12-15T17:12:19.800 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2021-12-15T17:12:19.800 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2021-12-15T17:12:19.800 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.allow_data_in_errors: 0 2021-12-15T17:12:19.800 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.db_host_id: __hostname__ 2021-12-15T17:12:19.801 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_background_jobs: 2 2021-12-15T17:12:19.801 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_background_compactions: -1 2021-12-15T17:12:19.801 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_subcompactions: 1 2021-12-15T17:12:19.802 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.avoid_flush_during_shutdown: 0 2021-12-15T17:12:19.802 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-12-15T17:12:19.802 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.delayed_write_rate : 16777216 2021-12-15T17:12:19.802 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_total_wal_size: 0 2021-12-15T17:12:19.802 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-12-15T17:12:19.803 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.stats_dump_period_sec: 600 2021-12-15T17:12:19.803 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.stats_persist_period_sec: 600 2021-12-15T17:12:19.803 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.stats_history_buffer_size: 1048576 2021-12-15T17:12:19.803 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_open_files: -1 2021-12-15T17:12:19.804 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bytes_per_sync: 0 2021-12-15T17:12:19.804 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.wal_bytes_per_sync: 0 2021-12-15T17:12:19.804 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.strict_bytes_per_sync: 0 2021-12-15T17:12:19.804 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_readahead_size: 0 2021-12-15T17:12:19.804 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_background_flushes: -1 2021-12-15T17:12:19.805 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Compression algorithms supported: 2021-12-15T17:12:19.805 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kZSTDNotFinalCompression supported: 0 2021-12-15T17:12:19.805 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kZSTD supported: 0 2021-12-15T17:12:19.805 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kXpressCompression supported: 0 2021-12-15T17:12:19.805 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kLZ4HCCompression supported: 1 2021-12-15T17:12:19.806 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kLZ4Compression supported: 1 2021-12-15T17:12:19.806 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kBZip2Compression supported: 0 2021-12-15T17:12:19.806 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kZlibCompression supported: 1 2021-12-15T17:12:19.806 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: kSnappyCompression supported: 1 2021-12-15T17:12:19.807 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Fast CRC32 supported: Supported on x86 2021-12-15T17:12:19.807 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2021-12-15T17:12:19.807 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2021-12-15T17:12:19.807 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-12-15T17:12:19.807 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.merge_operator: 2021-12-15T17:12:19.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_filter: None 2021-12-15T17:12:19.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_filter_factory: None 2021-12-15T17:12:19.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.sst_partitioner_factory: None 2021-12-15T17:12:19.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.memtable_factory: SkipListFactory 2021-12-15T17:12:19.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.table_factory: BlockBasedTable 2021-12-15T17:12:19.809 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b3854edbb0) 2021-12-15T17:12:19.809 INFO:journalctl@ceph.mon.a.smithi022.stdout: cache_index_and_filter_blocks: 1 2021-12-15T17:12:19.809 INFO:journalctl@ceph.mon.a.smithi022.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2021-12-15T17:12:19.809 INFO:journalctl@ceph.mon.a.smithi022.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2021-12-15T17:12:19.809 INFO:journalctl@ceph.mon.a.smithi022.stdout: pin_top_level_index_and_filter: 1 2021-12-15T17:12:19.810 INFO:journalctl@ceph.mon.a.smithi022.stdout: index_type: 0 2021-12-15T17:12:19.810 INFO:journalctl@ceph.mon.a.smithi022.stdout: data_block_index_type: 0 2021-12-15T17:12:19.810 INFO:journalctl@ceph.mon.a.smithi022.stdout: index_shortening: 1 2021-12-15T17:12:19.810 INFO:journalctl@ceph.mon.a.smithi022.stdout: data_block_hash_table_util_ratio: 0.750000 2021-12-15T17:12:19.811 INFO:journalctl@ceph.mon.a.smithi022.stdout: hash_index_allow_collision: 1 2021-12-15T17:12:19.811 INFO:journalctl@ceph.mon.a.smithi022.stdout: checksum: 1 2021-12-15T17:12:19.811 INFO:journalctl@ceph.mon.a.smithi022.stdout: no_block_cache: 0 2021-12-15T17:12:19.811 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache: 0x55b38551ed10 2021-12-15T17:12:19.811 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache_name: BinnedLRUCache 2021-12-15T17:12:19.812 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache_options: 2021-12-15T17:12:19.812 INFO:journalctl@ceph.mon.a.smithi022.stdout: capacity : 536870912 2021-12-15T17:12:19.812 INFO:journalctl@ceph.mon.a.smithi022.stdout: num_shard_bits : 4 2021-12-15T17:12:19.812 INFO:journalctl@ceph.mon.a.smithi022.stdout: strict_capacity_limit : 0 2021-12-15T17:12:19.813 INFO:journalctl@ceph.mon.a.smithi022.stdout: high_pri_pool_ratio: 0.000 2021-12-15T17:12:19.813 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache_compressed: (nil) 2021-12-15T17:12:19.813 INFO:journalctl@ceph.mon.a.smithi022.stdout: persistent_cache: (nil) 2021-12-15T17:12:19.813 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_size: 4096 2021-12-15T17:12:19.813 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_size_deviation: 10 2021-12-15T17:12:19.814 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_restart_interval: 16 2021-12-15T17:12:19.814 INFO:journalctl@ceph.mon.a.smithi022.stdout: index_block_restart_interval: 1 2021-12-15T17:12:19.814 INFO:journalctl@ceph.mon.a.smithi022.stdout: metadata_block_size: 4096 2021-12-15T17:12:19.814 INFO:journalctl@ceph.mon.a.smithi022.stdout: partition_filters: 0 2021-12-15T17:12:19.815 INFO:journalctl@ceph.mon.a.smithi022.stdout: use_delta_encoding: 1 2021-12-15T17:12:19.815 INFO:journalctl@ceph.mon.a.smithi022.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2021-12-15T17:12:19.815 INFO:journalctl@ceph.mon.a.smithi022.stdout: whole_key_filtering: 1 2021-12-15T17:12:19.815 INFO:journalctl@ceph.mon.a.smithi022.stdout: verify_compression: 0 2021-12-15T17:12:19.815 INFO:journalctl@ceph.mon.a.smithi022.stdout: read_amp_bytes_per_bit: 0 2021-12-15T17:12:19.816 INFO:journalctl@ceph.mon.a.smithi022.stdout: format_version: 4 2021-12-15T17:12:19.816 INFO:journalctl@ceph.mon.a.smithi022.stdout: enable_index_compression: 1 2021-12-15T17:12:19.816 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_align: 0 2021-12-15T17:12:19.816 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.write_buffer_size: 33554432 2021-12-15T17:12:19.816 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_write_buffer_number: 2 2021-12-15T17:12:19.817 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression: NoCompression 2021-12-15T17:12:19.817 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression: Disabled 2021-12-15T17:12:19.817 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.prefix_extractor: nullptr 2021-12-15T17:12:19.817 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-12-15T17:12:19.817 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.num_levels: 7 2021-12-15T17:12:19.818 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-12-15T17:12:19.818 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-12-15T17:12:19.818 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-12-15T17:12:19.818 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-12-15T17:12:19.819 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression_opts.level: 32767 2021-12-15T17:12:19.819 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-12-15T17:12:19.819 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-12-15T17:12:19.819 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-12-15T17:12:19.819 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2021-12-15T17:12:19.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bottommost_compression_opts.enabled: false 2021-12-15T17:12:19.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression_opts.window_bits: -14 2021-12-15T17:12:19.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression_opts.level: 32767 2021-12-15T17:12:19.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression_opts.strategy: 0 2021-12-15T17:12:19.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-12-15T17:12:19.821 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-12-15T17:12:19.821 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression_opts.parallel_threads: 1 2021-12-15T17:12:19.821 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compression_opts.enabled: false 2021-12-15T17:12:19.821 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-12-15T17:12:19.822 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-12-15T17:12:19.822 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.level0_stop_writes_trigger: 36 2021-12-15T17:12:19.822 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.target_file_size_base: 67108864 2021-12-15T17:12:19.822 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.target_file_size_multiplier: 1 2021-12-15T17:12:19.822 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_base: 268435456 2021-12-15T17:12:19.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-12-15T17:12:19.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-12-15T17:12:19.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-12-15T17:12:19.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-12-15T17:12:19.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-12-15T17:12:19.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-12-15T17:12:19.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-12-15T17:12:19.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-12-15T17:12:19.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-12-15T17:12:19.825 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-12-15T17:12:19.825 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_compaction_bytes: 1677721600 2021-12-15T17:12:19.825 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.arena_block_size: 4194304 2021-12-15T17:12:19.825 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-12-15T17:12:19.825 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-12-15T17:12:19.826 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-12-15T17:12:19.826 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.disable_auto_compactions: 0 2021-12-15T17:12:19.826 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-12-15T17:12:19.826 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-12-15T17:12:19.826 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-12-15T17:12:19.827 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-12-15T17:12:19.827 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-12-15T17:12:19.827 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-12-15T17:12:19.827 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-12-15T17:12:19.827 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-12-15T17:12:19.828 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-12-15T17:12:19.828 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-12-15T17:12:19.828 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.table_properties_collectors: 2021-12-15T17:12:19.828 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.inplace_update_support: 0 2021-12-15T17:12:19.829 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.inplace_update_num_locks: 10000 2021-12-15T17:12:19.829 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-12-15T17:12:19.829 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.memtable_whole_key_filtering: 0 2021-12-15T17:12:19.829 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.memtable_huge_page_size: 0 2021-12-15T17:12:19.829 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.bloom_locality: 0 2021-12-15T17:12:19.830 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.max_successive_merges: 0 2021-12-15T17:12:19.830 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.optimize_filters_for_hits: 0 2021-12-15T17:12:19.830 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.paranoid_file_checks: 0 2021-12-15T17:12:19.830 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.force_consistency_checks: 1 2021-12-15T17:12:19.830 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.report_bg_io_stats: 0 2021-12-15T17:12:19.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.ttl: 2592000 2021-12-15T17:12:19.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.periodic_compaction_seconds: 0 2021-12-15T17:12:19.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.enable_blob_files: false 2021-12-15T17:12:19.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.min_blob_size: 0 2021-12-15T17:12:19.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.blob_file_size: 268435456 2021-12-15T17:12:19.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.blob_compression_type: NoCompression 2021-12-15T17:12:19.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.enable_blob_garbage_collection: false 2021-12-15T17:12:19.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2021-12-15T17:12:19.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-12-15T17:12:19.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2021-12-15T17:12:19.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2021-12-15T17:12:19.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588339707926, "job": 1, "event": "recovery_started", "wal_files": [10]} 2021-12-15T17:12:19.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2021-12-15T17:12:19.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-12-15T17:12:19.834 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588339708661, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 77475, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 75755, "index_size": 208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10033, "raw_average_key_size": 47, "raw_value_size": 69981, "raw_average_value_size": 331, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1639588339, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "951a105d-7d3b-44a7-a159-dfb43fb5bb3c", "db_session_id": "ECSRTGXF09QL25HTU768"}} 2021-12-15T17:12:19.834 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2021-12-15T17:12:19.834 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588339709026, "job": 1, "event": "recovery_finished"} 2021-12-15T17:12:19.834 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2021-12-15T17:12:19.835 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55b385538a80 2021-12-15T17:12:19.835 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: DB pointer 0x55b385549800 2021-12-15T17:12:19.835 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2021-12-15T17:12:19.835 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: rocksdb: [db/db_impl/db_impl.cc:903] 2021-12-15T17:12:19.835 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** DB Stats ** 2021-12-15T17:12:19.836 INFO:journalctl@ceph.mon.a.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:12:19.836 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-12-15T17:12:19.836 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-12-15T17:12:19.836 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-12-15T17:12:19.837 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-12-15T17:12:19.837 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-12-15T17:12:19.837 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-12-15T17:12:19.837 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:12:19.837 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:12:19.838 INFO:journalctl@ceph.mon.a.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:12:19.838 INFO:journalctl@ceph.mon.a.smithi022.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:12:19.838 INFO:journalctl@ceph.mon.a.smithi022.stdout: L0 2/0 77.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2021-12-15T17:12:19.838 INFO:journalctl@ceph.mon.a.smithi022.stdout: Sum 2/0 77.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2021-12-15T17:12:19.839 INFO:journalctl@ceph.mon.a.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2021-12-15T17:12:19.839 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:12:19.839 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:12:19.839 INFO:journalctl@ceph.mon.a.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:12:19.839 INFO:journalctl@ceph.mon.a.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:12:19.840 INFO:journalctl@ceph.mon.a.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2021-12-15T17:12:19.840 INFO:journalctl@ceph.mon.a.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:12:19.840 INFO:journalctl@ceph.mon.a.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2021-12-15T17:12:19.840 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2021-12-15T17:12:19.841 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2021-12-15T17:12:19.841 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2021-12-15T17:12:19.841 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2021-12-15T17:12:19.841 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative compaction: 0.00 GB write, 24.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:12:19.841 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval compaction: 0.00 GB write, 24.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:12:19.842 INFO:journalctl@ceph.mon.a.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-12-15T17:12:19.842 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:12:19.842 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2021-12-15T17:12:19.842 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:12:19.843 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:12:19.843 INFO:journalctl@ceph.mon.a.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:12:19.843 INFO:journalctl@ceph.mon.a.smithi022.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:12:19.843 INFO:journalctl@ceph.mon.a.smithi022.stdout: L0 2/0 77.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2021-12-15T17:12:19.843 INFO:journalctl@ceph.mon.a.smithi022.stdout: Sum 2/0 77.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2021-12-15T17:12:19.844 INFO:journalctl@ceph.mon.a.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-12-15T17:12:19.844 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:12:19.844 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:12:19.844 INFO:journalctl@ceph.mon.a.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:12:19.844 INFO:journalctl@ceph.mon.a.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:12:19.845 INFO:journalctl@ceph.mon.a.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2021-12-15T17:12:19.845 INFO:journalctl@ceph.mon.a.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:12:19.845 INFO:journalctl@ceph.mon.a.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2021-12-15T17:12:19.845 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2021-12-15T17:12:19.846 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2021-12-15T17:12:19.846 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2021-12-15T17:12:19.846 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2021-12-15T17:12:19.846 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative compaction: 0.00 GB write, 24.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:12:19.846 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:12:19.847 INFO:journalctl@ceph.mon.a.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-12-15T17:12:19.847 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:12:19.847 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2021-12-15T17:12:19.847 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: starting mon.a rank 0 at public addrs v1:172.21.15.22:6789/0 at bind addrs v1:172.21.15.22:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:19.848 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???) e1 preinit fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:19.848 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???).mds e1 new map 2021-12-15T17:12:19.848 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???).mds e1 print_map 2021-12-15T17:12:19.848 INFO:journalctl@ceph.mon.a.smithi022.stdout: e1 2021-12-15T17:12:19.848 INFO:journalctl@ceph.mon.a.smithi022.stdout: enable_multiple, ever_enabled_multiple: 1,1 2021-12-15T17:12:19.849 INFO:journalctl@ceph.mon.a.smithi022.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-12-15T17:12:19.849 INFO:journalctl@ceph.mon.a.smithi022.stdout: legacy client fscid: -1 2021-12-15T17:12:19.849 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:12:19.849 INFO:journalctl@ceph.mon.a.smithi022.stdout: No filesystems configured 2021-12-15T17:12:19.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-12-15T17:12:19.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:12:19.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:12:19.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:12:19.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-12-15T17:12:19.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expand map: {default=false} 2021-12-15T17:12:19.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta from 'false' to 'false' 2021-12-15T17:12:19.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expanded map: {default=false} 2021-12-15T17:12:19.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expand map: {default=info} 2021-12-15T17:12:19.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta from 'info' to 'info' 2021-12-15T17:12:19.852 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expanded map: {default=info} 2021-12-15T17:12:19.852 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expand map: {default=daemon} 2021-12-15T17:12:19.852 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta from 'daemon' to 'daemon' 2021-12-15T17:12:19.852 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expanded map: {default=daemon} 2021-12-15T17:12:19.852 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expand map: {default=debug} 2021-12-15T17:12:19.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta from 'debug' to 'debug' 2021-12-15T17:12:19.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: expand_channel_meta expanded map: {default=debug} 2021-12-15T17:12:19.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 systemd[1]: Started Ceph mon.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:12:19.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:12:19.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: monmap e1: 1 mons at {a=v1:172.21.15.22:6789/0} 2021-12-15T17:12:19.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: fsmap 2021-12-15T17:12:19.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: osdmap e1: 0 total, 0 up, 0 in 2021-12-15T17:12:19.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:19 smithi022 ceph-mon[31360]: mgrmap e1: no daemons active 2021-12-15T17:12:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:Wrote config to /etc/ceph/ceph.conf 2021-12-15T17:12:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-12-15T17:12:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:Creating mgr... 2021-12-15T17:12:20.570 INFO:teuthology.orchestra.run.smithi022.stderr:Verifying port 9283 ... 2021-12-15T17:12:20.698 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Failed to reset failed state of unit ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mgr.y.service: Unit ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mgr.y.service not loaded. 2021-12-15T17:12:20.707 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c.target.wants/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mgr.y.service → /etc/systemd/system/ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@.service. 2021-12-15T17:12:21.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:20 smithi022 systemd[1]: Starting Ceph mgr.y for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:12:21.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3644780694' entity='client.admin' 2021-12-15T17:12:21.400 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:21 smithi022 podman[31740]: 2021-12-15T17:12:21.400 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:21 smithi022 bash[31589]: 1c2b643760aa45b2c9d79f434a434a96e4377ab97e1d284e25bdccd31fa29e43 2021-12-15T17:12:21.413 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2021-12-15T17:12:21.421 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2021-12-15T17:12:21.422 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2021-12-15T17:12:21.422 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to enable service . firewalld.service is not available 2021-12-15T17:12:21.431 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2021-12-15T17:12:21.438 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2021-12-15T17:12:21.439 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2021-12-15T17:12:21.440 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-12-15T17:12:21.440 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr to start... 2021-12-15T17:12:21.440 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr... 2021-12-15T17:12:21.676 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:21 smithi022 systemd[1]: Started Ceph mgr.y for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:12:21.676 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:21 smithi022 conmon[31781]: 2021-12-15T17:12:21.451+0000 7f252b1df180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2021-12-15T17:12:21.993 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:12:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsid": "0ea1c4a2-5dca-11ec-8c31-001a4aab830c", 2021-12-15T17:12:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "health": { 2021-12-15T17:12:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-12-15T17:12:21.995 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "checks": {}, 2021-12-15T17:12:21.995 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mutes": [] 2021-12-15T17:12:21.995 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:21.996 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-12-15T17:12:21.996 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum": [ 2021-12-15T17:12:21.996 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 0 2021-12-15T17:12:21.996 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:21.997 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_names": [ 2021-12-15T17:12:21.997 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "a" 2021-12-15T17:12:21.997 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:21.997 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-12-15T17:12:21.997 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "monmap": { 2021-12-15T17:12:21.998 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:21.998 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-12-15T17:12:21.998 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_mons": 1 2021-12-15T17:12:21.998 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:21.999 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osdmap": { 2021-12-15T17:12:21.999 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:21.999 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_osds": 0, 2021-12-15T17:12:21.999 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-12-15T17:12:22.000 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-12-15T17:12:22.000 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-12-15T17:12:22.004 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-12-15T17:12:22.004 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-12-15T17:12:22.005 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:22.006 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgmap": { 2021-12-15T17:12:22.006 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-12-15T17:12:22.006 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-12-15T17:12:22.006 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pools": 0, 2021-12-15T17:12:22.007 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_objects": 0, 2021-12-15T17:12:22.007 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-12-15T17:12:22.007 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-12-15T17:12:22.007 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-12-15T17:12:22.008 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_total": 0 2021-12-15T17:12:22.008 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:22.008 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsmap": { 2021-12-15T17:12:22.008 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:22.009 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "by_rank": [], 2021-12-15T17:12:22.009 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "up:standby": 0 2021-12-15T17:12:22.009 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:22.009 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap": { 2021-12-15T17:12:22.009 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": false, 2021-12-15T17:12:22.010 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-12-15T17:12:22.010 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modules": [ 2021-12-15T17:12:22.010 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "iostat", 2021-12-15T17:12:22.011 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "nfs", 2021-12-15T17:12:22.011 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "restful" 2021-12-15T17:12:22.011 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:22.011 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:22.012 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:22.012 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "servicemap": { 2021-12-15T17:12:22.012 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:22.012 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modified": "2021-12-15T17:12:14.883766+0000", 2021-12-15T17:12:22.013 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:22.013 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:22.013 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "progress_events": {} 2021-12-15T17:12:22.013 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:12:22.241 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:22 smithi022 conmon[31781]: 2021-12-15T17:12:22.040+0000 7f252b1df180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2021-12-15T17:12:22.241 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:22 smithi022 conmon[31781]: 2021-12-15T17:12:22.162+0000 7f252b1df180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2021-12-15T17:12:22.241 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:22 smithi022 conmon[31781]: 2021-12-15T17:12:22.241+0000 7f252b1df180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2021-12-15T17:12:22.290 INFO:teuthology.orchestra.run.smithi022.stderr:mgr not available, waiting (1/15)... 2021-12-15T17:12:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2416579838' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:12:22.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:22 smithi022 conmon[31781]: 2021-12-15T17:12:22.334+0000 7f252b1df180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2021-12-15T17:12:23.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:23 smithi022 conmon[31781]: 2021-12-15T17:12:23.073+0000 7f252b1df180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:12:23.920 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:23 smithi022 conmon[31781]: 2021-12-15T17:12:23.659+0000 7f252b1df180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2021-12-15T17:12:23.920 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:23 smithi022 conmon[31781]: 2021-12-15T17:12:23.748+0000 7f252b1df180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2021-12-15T17:12:24.275 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:23 smithi022 conmon[31781]: 2021-12-15T17:12:23.920+0000 7f252b1df180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2021-12-15T17:12:24.275 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:24 smithi022 conmon[31781]: 2021-12-15T17:12:24.010+0000 7f252b1df180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2021-12-15T17:12:24.277 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:24 smithi022 conmon[31781]: 2021-12-15T17:12:24.101+0000 7f252b1df180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2021-12-15T17:12:24.277 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:24 smithi022 conmon[31781]: 2021-12-15T17:12:24.275+0000 7f252b1df180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2021-12-15T17:12:24.799 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:24 smithi022 conmon[31781]: 2021-12-15T17:12:24.566+0000 7f252b1df180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2021-12-15T17:12:24.800 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:24 smithi022 conmon[31781]: 2021-12-15T17:12:24.701+0000 7f252b1df180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2021-12-15T17:12:24.828 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:24.828 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:12:24.828 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsid": "0ea1c4a2-5dca-11ec-8c31-001a4aab830c", 2021-12-15T17:12:24.829 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "health": { 2021-12-15T17:12:24.829 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-12-15T17:12:24.829 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "checks": {}, 2021-12-15T17:12:24.829 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mutes": [] 2021-12-15T17:12:24.829 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-12-15T17:12:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum": [ 2021-12-15T17:12:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 0 2021-12-15T17:12:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_names": [ 2021-12-15T17:12:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "a" 2021-12-15T17:12:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_age": 5, 2021-12-15T17:12:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "monmap": { 2021-12-15T17:12:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-12-15T17:12:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_mons": 1 2021-12-15T17:12:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osdmap": { 2021-12-15T17:12:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_osds": 0, 2021-12-15T17:12:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-12-15T17:12:24.834 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-12-15T17:12:24.834 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-12-15T17:12:24.834 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-12-15T17:12:24.835 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-12-15T17:12:24.835 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:24.835 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgmap": { 2021-12-15T17:12:24.835 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-12-15T17:12:24.835 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-12-15T17:12:24.836 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pools": 0, 2021-12-15T17:12:24.836 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_objects": 0, 2021-12-15T17:12:24.836 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-12-15T17:12:24.836 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-12-15T17:12:24.837 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-12-15T17:12:24.837 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_total": 0 2021-12-15T17:12:24.837 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:24.837 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsmap": { 2021-12-15T17:12:24.837 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:24.838 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "by_rank": [], 2021-12-15T17:12:24.838 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "up:standby": 0 2021-12-15T17:12:24.838 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:24.838 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap": { 2021-12-15T17:12:24.838 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": false, 2021-12-15T17:12:24.839 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-12-15T17:12:24.839 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modules": [ 2021-12-15T17:12:24.839 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "iostat", 2021-12-15T17:12:24.839 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "nfs", 2021-12-15T17:12:24.839 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "restful" 2021-12-15T17:12:24.840 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:24.840 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:24.840 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:24.840 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "servicemap": { 2021-12-15T17:12:24.841 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:24.841 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modified": "2021-12-15T17:12:14.883766+0000", 2021-12-15T17:12:24.841 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:24.841 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:24.841 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "progress_events": {} 2021-12-15T17:12:24.842 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:12:25.130 INFO:teuthology.orchestra.run.smithi022.stderr:mgr not available, waiting (2/15)... 2021-12-15T17:12:25.154 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:24 smithi022 conmon[31781]: 2021-12-15T17:12:24.799+0000 7f252b1df180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2021-12-15T17:12:25.154 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:24 smithi022 conmon[31781]: 2021-12-15T17:12:24.890+0000 7f252b1df180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2021-12-15T17:12:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3594010203' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:12:26.029 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:25 smithi022 conmon[31781]: 2021-12-15T17:12:25.685+0000 7f252b1df180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2021-12-15T17:12:26.352 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:26 smithi022 conmon[31781]: 2021-12-15T17:12:26.042+0000 7f252b1df180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:12:26.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:26 smithi022 conmon[31781]: 2021-12-15T17:12:26.353+0000 7f252b1df180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2021-12-15T17:12:26.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:26 smithi022 conmon[31781]: 2021-12-15T17:12:26.501+0000 7f252b1df180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2021-12-15T17:12:27.228 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:26 smithi022 conmon[31781]: 2021-12-15T17:12:26.783+0000 7f252b1df180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2021-12-15T17:12:27.229 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:26 smithi022 conmon[31781]: 2021-12-15T17:12:26.872+0000 7f252b1df180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2021-12-15T17:12:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:27 smithi022 conmon[31781]: 2021-12-15T17:12:27.229+0000 7f252b1df180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2021-12-15T17:12:27.705 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:27.705 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:12:27.706 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsid": "0ea1c4a2-5dca-11ec-8c31-001a4aab830c", 2021-12-15T17:12:27.706 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "health": { 2021-12-15T17:12:27.707 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-12-15T17:12:27.707 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "checks": {}, 2021-12-15T17:12:27.707 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mutes": [] 2021-12-15T17:12:27.708 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:27.708 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-12-15T17:12:27.709 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum": [ 2021-12-15T17:12:27.709 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 0 2021-12-15T17:12:27.710 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:27.710 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_names": [ 2021-12-15T17:12:27.711 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "a" 2021-12-15T17:12:27.711 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:27.712 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_age": 7, 2021-12-15T17:12:27.712 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "monmap": { 2021-12-15T17:12:27.713 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:27.713 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-12-15T17:12:27.714 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_mons": 1 2021-12-15T17:12:27.714 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:27.714 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osdmap": { 2021-12-15T17:12:27.716 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:27.716 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_osds": 0, 2021-12-15T17:12:27.716 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-12-15T17:12:27.717 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-12-15T17:12:27.717 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-12-15T17:12:27.718 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-12-15T17:12:27.718 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-12-15T17:12:27.719 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:27.719 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgmap": { 2021-12-15T17:12:27.719 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-12-15T17:12:27.720 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-12-15T17:12:27.720 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pools": 0, 2021-12-15T17:12:27.721 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_objects": 0, 2021-12-15T17:12:27.721 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-12-15T17:12:27.721 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-12-15T17:12:27.722 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-12-15T17:12:27.722 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_total": 0 2021-12-15T17:12:27.723 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:27.723 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsmap": { 2021-12-15T17:12:27.724 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:27.725 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "by_rank": [], 2021-12-15T17:12:27.726 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "up:standby": 0 2021-12-15T17:12:27.727 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:27.727 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap": { 2021-12-15T17:12:27.728 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": false, 2021-12-15T17:12:27.728 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-12-15T17:12:27.729 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modules": [ 2021-12-15T17:12:27.729 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "iostat", 2021-12-15T17:12:27.729 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "nfs", 2021-12-15T17:12:27.730 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "restful" 2021-12-15T17:12:27.730 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:27.731 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:27.731 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:27.732 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "servicemap": { 2021-12-15T17:12:27.732 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:27.733 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modified": "2021-12-15T17:12:14.883766+0000", 2021-12-15T17:12:27.733 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:27.733 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:27.734 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "progress_events": {} 2021-12-15T17:12:27.735 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:12:27.848 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:27 smithi022 conmon[31781]: 2021-12-15T17:12:27.543+0000 7f252b1df180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2021-12-15T17:12:27.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: Activating manager daemon y 2021-12-15T17:12:27.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: mgrmap e2: y(active, starting, since 0.00279219s) 2021-12-15T17:12:27.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:12:27.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:12:27.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:12:27.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:12:27.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:12:27.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:12:27.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:12:27.851 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:12:27.852 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: Manager daemon y is now available 2021-12-15T17:12:27.852 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:12:27.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:12:27.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' 2021-12-15T17:12:27.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:27 smithi022 ceph-mon[31360]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' 2021-12-15T17:12:28.070 INFO:teuthology.orchestra.run.smithi022.stderr:mgr not available, waiting (3/15)... 2021-12-15T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2766790301' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:28 smithi022 ceph-mon[31360]: mgrmap e3: y(active, since 1.00676s) 2021-12-15T17:12:30.769 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:30.769 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:12:30.769 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsid": "0ea1c4a2-5dca-11ec-8c31-001a4aab830c", 2021-12-15T17:12:30.769 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "health": { 2021-12-15T17:12:30.770 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-12-15T17:12:30.770 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "checks": {}, 2021-12-15T17:12:30.770 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mutes": [] 2021-12-15T17:12:30.770 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:30.771 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-12-15T17:12:30.771 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum": [ 2021-12-15T17:12:30.771 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 0 2021-12-15T17:12:30.771 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:30.772 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_names": [ 2021-12-15T17:12:30.772 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "a" 2021-12-15T17:12:30.772 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:30.772 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "quorum_age": 11, 2021-12-15T17:12:30.773 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "monmap": { 2021-12-15T17:12:30.773 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:30.773 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-12-15T17:12:30.773 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_mons": 1 2021-12-15T17:12:30.773 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:30.774 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osdmap": { 2021-12-15T17:12:30.774 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:30.774 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_osds": 0, 2021-12-15T17:12:30.775 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-12-15T17:12:30.775 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-12-15T17:12:30.775 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-12-15T17:12:30.775 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-12-15T17:12:30.775 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-12-15T17:12:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgmap": { 2021-12-15T17:12:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-12-15T17:12:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-12-15T17:12:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_pools": 0, 2021-12-15T17:12:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_objects": 0, 2021-12-15T17:12:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-12-15T17:12:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-12-15T17:12:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-12-15T17:12:30.778 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "bytes_total": 0 2021-12-15T17:12:30.778 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:30.778 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "fsmap": { 2021-12-15T17:12:30.778 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:30.778 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "by_rank": [], 2021-12-15T17:12:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "up:standby": 0 2021-12-15T17:12:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap": { 2021-12-15T17:12:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": true, 2021-12-15T17:12:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-12-15T17:12:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modules": [ 2021-12-15T17:12:30.782 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "iostat", 2021-12-15T17:12:30.782 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "nfs", 2021-12-15T17:12:30.782 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "restful" 2021-12-15T17:12:30.782 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ], 2021-12-15T17:12:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "servicemap": { 2021-12-15T17:12:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 1, 2021-12-15T17:12:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "modified": "2021-12-15T17:12:14.883766+0000", 2021-12-15T17:12:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "services": {} 2021-12-15T17:12:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: }, 2021-12-15T17:12:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "progress_events": {} 2021-12-15T17:12:30.786 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:12:30.926 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:30 smithi022 ceph-mon[31360]: mgrmap e4: y(active, since 2s) 2021-12-15T17:12:31.114 INFO:teuthology.orchestra.run.smithi022.stderr:mgr is available 2021-12-15T17:12:31.721 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:31.722 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [global] 2021-12-15T17:12:31.723 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: fsid = 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:31.723 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-12-15T17:12:31.723 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-12-15T17:12:31.723 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-12-15T17:12:31.723 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-12-15T17:12:31.724 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:31.724 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [mgr] 2021-12-15T17:12:31.724 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-12-15T17:12:31.724 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 2021-12-15T17:12:31.725 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: [osd] 2021-12-15T17:12:31.725 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-12-15T17:12:31.725 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-12-15T17:12:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/886693411' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:12:32.931 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/291934710' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-12-15T17:12:33.555 INFO:teuthology.orchestra.run.smithi022.stderr:Enabling cephadm module... 2021-12-15T17:12:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/335991765' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-12-15T17:12:36.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:35 smithi022 conmon[31781]: ignoring --setuser ceph since I am not root 2021-12-15T17:12:36.029 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:35 smithi022 conmon[31781]: ignoring --setgroup ceph since I am not root 2021-12-15T17:12:36.029 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:35 smithi022 conmon[31781]: 2021-12-15T17:12:35.883+0000 7f846c577180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2021-12-15T17:12:36.718 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:36 smithi022 conmon[31781]: 2021-12-15T17:12:36.435+0000 7f846c577180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2021-12-15T17:12:36.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:36 smithi022 conmon[31781]: 2021-12-15T17:12:36.556+0000 7f846c577180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2021-12-15T17:12:36.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:36 smithi022 conmon[31781]: 2021-12-15T17:12:36.638+0000 7f846c577180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2021-12-15T17:12:37.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:36 smithi022 conmon[31781]: 2021-12-15T17:12:36.730+0000 7f846c577180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2021-12-15T17:12:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/335991765' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-12-15T17:12:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:36 smithi022 ceph-mon[31360]: mgrmap e5: y(active, since 8s) 2021-12-15T17:12:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:37 smithi022 conmon[31781]: 2021-12-15T17:12:37.475+0000 7f846c577180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:12:38.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:38 smithi022 conmon[31781]: 2021-12-15T17:12:38.042+0000 7f846c577180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2021-12-15T17:12:38.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:38 smithi022 conmon[31781]: 2021-12-15T17:12:38.131+0000 7f846c577180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2021-12-15T17:12:38.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:38 smithi022 conmon[31781]: 2021-12-15T17:12:38.306+0000 7f846c577180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2021-12-15T17:12:38.674 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:38 smithi022 conmon[31781]: 2021-12-15T17:12:38.401+0000 7f846c577180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2021-12-15T17:12:38.674 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:38 smithi022 conmon[31781]: 2021-12-15T17:12:38.494+0000 7f846c577180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2021-12-15T17:12:38.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:38 smithi022 conmon[31781]: 2021-12-15T17:12:38.674+0000 7f846c577180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2021-12-15T17:12:39.261 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:38 smithi022 conmon[31781]: 2021-12-15T17:12:38.951+0000 7f846c577180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2021-12-15T17:12:39.261 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:39 smithi022 conmon[31781]: 2021-12-15T17:12:39.083+0000 7f846c577180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2021-12-15T17:12:39.261 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:39 smithi022 conmon[31781]: 2021-12-15T17:12:39.172+0000 7f846c577180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2021-12-15T17:12:39.458 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:12:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 5, 2021-12-15T17:12:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": true, 2021-12-15T17:12:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "active_name": "y", 2021-12-15T17:12:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standby": 0 2021-12-15T17:12:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:12:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/838481017' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-12-15T17:12:39.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:39 smithi022 conmon[31781]: 2021-12-15T17:12:39.261+0000 7f846c577180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2021-12-15T17:12:40.309 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:40 smithi022 conmon[31781]: 2021-12-15T17:12:40.059+0000 7f846c577180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2021-12-15T17:12:40.344 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for the mgr to restart... 2021-12-15T17:12:40.344 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr epoch 5... 2021-12-15T17:12:40.581 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:40 smithi022 conmon[31781]: 2021-12-15T17:12:40.431+0000 7f846c577180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:12:40.876 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:40 smithi022 conmon[31781]: 2021-12-15T17:12:40.725+0000 7f846c577180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2021-12-15T17:12:41.156 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:40 smithi022 conmon[31781]: 2021-12-15T17:12:40.876+0000 7f846c577180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2021-12-15T17:12:41.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:41 smithi022 conmon[31781]: 2021-12-15T17:12:41.157+0000 7f846c577180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2021-12-15T17:12:41.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:41 smithi022 conmon[31781]: 2021-12-15T17:12:41.246+0000 7f846c577180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2021-12-15T17:12:41.903 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:41 smithi022 conmon[31781]: 2021-12-15T17:12:41.595+0000 7f846c577180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2021-12-15T17:12:42.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:41 smithi022 conmon[31781]: 2021-12-15T17:12:41.904+0000 7f846c577180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2021-12-15T17:12:42.978 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:12:42.978 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-12-15T17:12:42.978 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "initialized": true 2021-12-15T17:12:42.979 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:12:43.252 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:43 smithi022 conmon[31781]: [15/Dec/2021:17:12:43] ENGINE Bus STARTING 2021-12-15T17:12:43.252 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: Active manager daemon y restarted 2021-12-15T17:12:43.252 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: Activating manager daemon y 2021-12-15T17:12:43.253 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: osdmap e2: 0 total, 0 up, 0 in 2021-12-15T17:12:43.253 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: mgrmap e6: y(active, starting, since 0.052623s) 2021-12-15T17:12:43.253 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:12:43.253 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:12:43.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:12:43.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:12:43.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:12:43.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: Manager daemon y is now available 2021-12-15T17:12:43.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:43.255 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:43.255 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:12:43.255 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:12:43.255 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:12:43.256 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:12:43.256 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:42 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi022.stderr:mgr epoch 5 is available 2021-12-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi022.stderr:Setting orchestrator backend to cephadm... 2021-12-15T17:12:43.505 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:43 smithi022 conmon[31781]: [15/Dec/2021:17:12:43] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:12:43.506 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:43 smithi022 conmon[31781]: [15/Dec/2021:17:12:43] ENGINE Bus STARTED 2021-12-15T17:12:44.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:43 smithi022 ceph-mon[31360]: mgrmap e7: y(active, since 1.05487s) 2021-12-15T17:12:44.265 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:43 smithi022 ceph-mon[31360]: [15/Dec/2021:17:12:43] ENGINE Bus STARTING 2021-12-15T17:12:44.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:43 smithi022 ceph-mon[31360]: [15/Dec/2021:17:12:43] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:12:44.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:43 smithi022 ceph-mon[31360]: [15/Dec/2021:17:12:43] ENGINE Bus STARTED 2021-12-15T17:12:44.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:43 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:44.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:43 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:12:44.924 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: value unchanged 2021-12-15T17:12:45.257 INFO:teuthology.orchestra.run.smithi022.stderr:Generating ssh key... 2021-12-15T17:12:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:45 smithi022 ceph-mon[31360]: from='client.14132 v1:172.21.15.22:0/1266900416' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:12:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:45 smithi022 ceph-mon[31360]: mgrmap e8: y(active, since 2s) 2021-12-15T17:12:47.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:46 smithi022 ceph-mon[31360]: from='client.14134 v1:172.21.15.22:0/899131883' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:12:47.013 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:46 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:47.013 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:46 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:47.013 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: Generating public/private rsa key pair. 2021-12-15T17:12:47.014 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: Your identification has been saved in /tmp/tmp14grqihi/key. 2021-12-15T17:12:47.014 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: Your public key has been saved in /tmp/tmp14grqihi/key.pub. 2021-12-15T17:12:47.014 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: The key fingerprint is: 2021-12-15T17:12:47.014 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: SHA256:UG8KV+MMG+dijcH7OFYtP6k51JqDOAeveX6CAOx/imA ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:47.015 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: The key's randomart image is: 2021-12-15T17:12:47.015 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: +---[RSA 3072]----+ 2021-12-15T17:12:47.015 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: | .= + | 2021-12-15T17:12:47.015 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: | ..# . | 2021-12-15T17:12:47.016 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: | . o *.B. | 2021-12-15T17:12:47.016 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: | o =.+o . | 2021-12-15T17:12:47.016 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: | . . S+ + . | 2021-12-15T17:12:47.016 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: | . . . + o = | 2021-12-15T17:12:47.017 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: |.E . . * + = . | 2021-12-15T17:12:47.017 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: |.. .. =.* O | 2021-12-15T17:12:47.017 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: | . .oo*.o o | 2021-12-15T17:12:47.017 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:46 smithi022 conmon[31781]: +----[SHA256]-----+ 2021-12-15T17:12:47.599 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwr6ChKHltZMrIDcr6E3Kn0JppWJZ2CeftfpwhB0fxF3HiTX1VEVfBj7TPfMWCnUu9Zpn0s5dIbsFS1ba1YO7qTd4yqEx5A1KGJfrkd6hYVO1sKJU0XPlDzpEvHY9YMM5VtytwLKSetUOOsKF4IUi/xMh0o3hzFpV9QkpGnb2s9kT3eRTwB1dVtlSCUFc5zwcyVS47PPjZOWHM/sQ6ySk1cLUhVvEDqogNZ7vw9mJSUjXw+3Ss/crjau7voQmputKRgqrBKRJlSTrYr7WZ9LFfYnIF0l86KJVxin1pBiGJwBx31sM14Bo1ypzCOVZjDchYi7AOj3+99XIQ1VyQNoFTpiBT5QXrHrU32R0Dj0NiVlst4qRygb+PzhBsA6OOm9NX03D1A5Aa8DqV32kgLUkPtDmE1Vc8H1VWuEGL6j4Jo2IxZum9O9Xtg/E4lHSQ0XmAe6jv5h2NhGG9ax48Uhypq8DwWj3BfwYQO19E1T6xuVfGmqspA0fbd8W4WOV3IRU= ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:12:47.920 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:47 smithi022 ceph-mon[31360]: from='client.14136 v1:172.21.15.22:0/2997050750' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:12:47.921 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:47 smithi022 ceph-mon[31360]: Generating ssh key... 2021-12-15T17:12:48.135 INFO:teuthology.orchestra.run.smithi022.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-12-15T17:12:48.136 INFO:teuthology.orchestra.run.smithi022.stderr:Adding key to root@localhost authorized_keys... 2021-12-15T17:12:48.137 INFO:teuthology.orchestra.run.smithi022.stderr:Adding host smithi022... 2021-12-15T17:12:48.984 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:48 smithi022 ceph-mon[31360]: from='client.14138 v1:172.21.15.22:0/2853550753' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:12:49.683 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Added host 'smithi022' with addr '172.21.15.22' 2021-12-15T17:12:49.976 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying unmanaged mon service... 2021-12-15T17:12:50.607 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled mon update... 2021-12-15T17:12:50.893 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:50 smithi022 ceph-mon[31360]: from='client.14140 v1:172.21.15.22:0/2303086545' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi022", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:12:50.893 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:50 smithi022 ceph-mon[31360]: Deploying cephadm binary to smithi022 2021-12-15T17:12:50.894 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:50 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:50.894 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:50 smithi022 ceph-mon[31360]: Added host smithi022 2021-12-15T17:12:50.894 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:50 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:12:50.894 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:50 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:Deploying unmanaged mgr service... 2021-12-15T17:12:51.743 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-12-15T17:12:51.979 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:51 smithi022 ceph-mon[31360]: from='client.14142 v1:172.21.15.22:0/2607737481' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:12:51.980 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:51 smithi022 ceph-mon[31360]: Saving service mon spec with placement count:5 2021-12-15T17:12:51.980 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:51 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:52 smithi022 ceph-mon[31360]: from='client.14144 v1:172.21.15.22:0/2306284900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:12:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:52 smithi022 ceph-mon[31360]: Saving service mgr spec with placement count:2 2021-12-15T17:12:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:52 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:52 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:53.973 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3988911567' entity='client.admin' 2021-12-15T17:12:53.973 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:53 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:54.225 INFO:teuthology.orchestra.run.smithi022.stderr:Enabling the dashboard module... 2021-12-15T17:12:55.200 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3677929078' entity='client.admin' 2021-12-15T17:12:55.201 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:54 smithi022 ceph-mon[31360]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:12:55.201 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1632747641' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-12-15T17:12:56.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:55 smithi022 conmon[31781]: ignoring --setuser ceph since I am not root 2021-12-15T17:12:56.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:55 smithi022 conmon[31781]: ignoring --setgroup ceph since I am not root 2021-12-15T17:12:56.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:55 smithi022 conmon[31781]: 2021-12-15T17:12:55.871+0000 7f8d6fb33180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2021-12-15T17:12:56.448 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:56 smithi022 conmon[31781]: 2021-12-15T17:12:56.447+0000 7f8d6fb33180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2021-12-15T17:12:56.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:56 smithi022 conmon[31781]: 2021-12-15T17:12:56.572+0000 7f8d6fb33180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2021-12-15T17:12:56.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:56 smithi022 conmon[31781]: 2021-12-15T17:12:56.653+0000 7f8d6fb33180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2021-12-15T17:12:56.766 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:12:56.766 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "epoch": 9, 2021-12-15T17:12:56.766 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "available": true, 2021-12-15T17:12:56.767 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "active_name": "y", 2021-12-15T17:12:56.767 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "num_standby": 0 2021-12-15T17:12:56.767 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:12:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1632747641' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-12-15T17:12:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:56 smithi022 ceph-mon[31360]: mgrmap e9: y(active, since 13s) 2021-12-15T17:12:57.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:56 smithi022 conmon[31781]: 2021-12-15T17:12:56.744+0000 7f8d6fb33180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2021-12-15T17:12:57.112 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for the mgr to restart... 2021-12-15T17:12:57.112 INFO:teuthology.orchestra.run.smithi022.stderr:Waiting for mgr epoch 9... 2021-12-15T17:12:57.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:57 smithi022 conmon[31781]: 2021-12-15T17:12:57.513+0000 7f8d6fb33180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:12:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:12:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2707799736' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-12-15T17:12:58.344 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:58 smithi022 conmon[31781]: 2021-12-15T17:12:58.081+0000 7f8d6fb33180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2021-12-15T17:12:58.344 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:58 smithi022 conmon[31781]: 2021-12-15T17:12:58.171+0000 7f8d6fb33180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2021-12-15T17:12:58.703 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:58 smithi022 conmon[31781]: 2021-12-15T17:12:58.343+0000 7f8d6fb33180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2021-12-15T17:12:58.703 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:58 smithi022 conmon[31781]: 2021-12-15T17:12:58.434+0000 7f8d6fb33180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2021-12-15T17:12:58.703 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:58 smithi022 conmon[31781]: 2021-12-15T17:12:58.525+0000 7f8d6fb33180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2021-12-15T17:12:58.980 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:58 smithi022 conmon[31781]: 2021-12-15T17:12:58.702+0000 7f8d6fb33180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2021-12-15T17:12:59.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:58 smithi022 conmon[31781]: 2021-12-15T17:12:58.980+0000 7f8d6fb33180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2021-12-15T17:12:59.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:59 smithi022 conmon[31781]: 2021-12-15T17:12:59.112+0000 7f8d6fb33180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2021-12-15T17:12:59.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:59 smithi022 conmon[31781]: 2021-12-15T17:12:59.200+0000 7f8d6fb33180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2021-12-15T17:12:59.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:12:59 smithi022 conmon[31781]: 2021-12-15T17:12:59.286+0000 7f8d6fb33180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2021-12-15T17:13:00.438 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:00 smithi022 conmon[31781]: 2021-12-15T17:13:00.081+0000 7f8d6fb33180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2021-12-15T17:13:00.724 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:00 smithi022 conmon[31781]: 2021-12-15T17:13:00.438+0000 7f8d6fb33180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:13:01.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:00 smithi022 conmon[31781]: 2021-12-15T17:13:00.725+0000 7f8d6fb33180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2021-12-15T17:13:01.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:00 smithi022 conmon[31781]: 2021-12-15T17:13:00.873+0000 7f8d6fb33180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2021-12-15T17:13:01.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:01 smithi022 conmon[31781]: 2021-12-15T17:13:01.156+0000 7f8d6fb33180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2021-12-15T17:13:01.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:01 smithi022 conmon[31781]: 2021-12-15T17:13:01.245+0000 7f8d6fb33180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2021-12-15T17:13:01.905 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:01 smithi022 conmon[31781]: 2021-12-15T17:13:01.597+0000 7f8d6fb33180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2021-12-15T17:13:02.160 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:01 smithi022 conmon[31781]: 2021-12-15T17:13:01.904+0000 7f8d6fb33180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2021-12-15T17:13:02.980 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: { 2021-12-15T17:13:02.981 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2021-12-15T17:13:02.981 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: "initialized": true 2021-12-15T17:13:02.981 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: } 2021-12-15T17:13:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: Active manager daemon y restarted 2021-12-15T17:13:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: Activating manager daemon y 2021-12-15T17:13:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: osdmap e3: 0 total, 0 up, 0 in 2021-12-15T17:13:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: mgrmap e10: y(active, starting, since 0.0525412s) 2021-12-15T17:13:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:13:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:03.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: Manager daemon y is now available 2021-12-15T17:13:03.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:03.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:03.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:13:03.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:13:03.456 INFO:teuthology.orchestra.run.smithi022.stderr:mgr epoch 9 is available 2021-12-15T17:13:03.456 INFO:teuthology.orchestra.run.smithi022.stderr:Generating a dashboard self-signed certificate... 2021-12-15T17:13:03.961 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:03 smithi022 conmon[31781]: [15/Dec/2021:17:13:03] ENGINE Bus STARTING 2021-12-15T17:13:03.961 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:03 smithi022 conmon[31781]: [15/Dec/2021:17:13:03] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:13:03.962 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:13:03 smithi022 conmon[31781]: [15/Dec/2021:17:13:03] ENGINE Bus STARTED 2021-12-15T17:13:04.209 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: Self-signed certificate created 2021-12-15T17:13:04.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:03 smithi022 ceph-mon[31360]: mgrmap e11: y(active, since 1.05512s) 2021-12-15T17:13:04.238 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:03 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:04.238 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:03 smithi022 ceph-mon[31360]: [15/Dec/2021:17:13:03] ENGINE Bus STARTING 2021-12-15T17:13:04.692 INFO:teuthology.orchestra.run.smithi022.stderr:Creating initial admin user... 2021-12-15T17:13:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:05 smithi022 ceph-mon[31360]: [15/Dec/2021:17:13:03] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:13:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:05 smithi022 ceph-mon[31360]: [15/Dec/2021:17:13:03] ENGINE Bus STARTED 2021-12-15T17:13:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:05 smithi022 ceph-mon[31360]: from='client.14164 v1:172.21.15.22:0/1947165790' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:05 smithi022 ceph-mon[31360]: mgrmap e12: y(active, since 2s) 2021-12-15T17:13:05.613 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$pXirupL0rGgUNOQrXGnD2.LY8betZB3FlZrgUVjyjDeYJIk8fdpwO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1639588385, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-12-15T17:13:06.129 INFO:teuthology.orchestra.run.smithi022.stderr:Fetching dashboard port number... 2021-12-15T17:13:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:06 smithi022 ceph-mon[31360]: from='client.14166 v1:172.21.15.22:0/1590284040' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:06 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:07.276 INFO:teuthology.orchestra.run.smithi022.stderr:/usr/bin/ceph: 8443 2021-12-15T17:13:07.581 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: disabled 2021-12-15T17:13:07.585 INFO:teuthology.orchestra.run.smithi022.stderr:systemctl: inactive 2021-12-15T17:13:07.585 INFO:teuthology.orchestra.run.smithi022.stderr:firewalld.service is not enabled 2021-12-15T17:13:07.586 INFO:teuthology.orchestra.run.smithi022.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-12-15T17:13:07.586 INFO:teuthology.orchestra.run.smithi022.stderr:Ceph Dashboard is now available at: 2021-12-15T17:13:07.587 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.587 INFO:teuthology.orchestra.run.smithi022.stderr: URL: https://smithi022:8443/ 2021-12-15T17:13:07.587 INFO:teuthology.orchestra.run.smithi022.stderr: User: admin 2021-12-15T17:13:07.588 INFO:teuthology.orchestra.run.smithi022.stderr: Password: jn9msmql2q 2021-12-15T17:13:07.588 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.589 INFO:teuthology.orchestra.run.smithi022.stderr:You can access the Ceph CLI with: 2021-12-15T17:13:07.589 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.589 INFO:teuthology.orchestra.run.smithi022.stderr: sudo /sbin/cephadm shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:07.589 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.590 INFO:teuthology.orchestra.run.smithi022.stderr:Please consider enabling telemetry to help improve Ceph: 2021-12-15T17:13:07.590 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.590 INFO:teuthology.orchestra.run.smithi022.stderr: ceph telemetry on 2021-12-15T17:13:07.590 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.591 INFO:teuthology.orchestra.run.smithi022.stderr:For more information see: 2021-12-15T17:13:07.591 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.591 INFO:teuthology.orchestra.run.smithi022.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2021-12-15T17:13:07.591 INFO:teuthology.orchestra.run.smithi022.stderr: 2021-12-15T17:13:07.591 INFO:teuthology.orchestra.run.smithi022.stderr:Bootstrap complete. 2021-12-15T17:13:07.623 INFO:tasks.cephadm:Fetching config... 2021-12-15T17:13:07.623 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:13:07.624 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-12-15T17:13:07.641 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-12-15T17:13:07.642 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:13:07.642 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-12-15T17:13:07.698 INFO:tasks.cephadm:Fetching mon keyring... 2021-12-15T17:13:07.698 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:13:07.699 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/keyring of=/dev/stdout 2021-12-15T17:13:07.764 INFO:tasks.cephadm:Fetching pub ssh key... 2021-12-15T17:13:07.765 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:13:07.765 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-12-15T17:13:07.823 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-12-15T17:13:07.823 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwr6ChKHltZMrIDcr6E3Kn0JppWJZ2CeftfpwhB0fxF3HiTX1VEVfBj7TPfMWCnUu9Zpn0s5dIbsFS1ba1YO7qTd4yqEx5A1KGJfrkd6hYVO1sKJU0XPlDzpEvHY9YMM5VtytwLKSetUOOsKF4IUi/xMh0o3hzFpV9QkpGnb2s9kT3eRTwB1dVtlSCUFc5zwcyVS47PPjZOWHM/sQ6ySk1cLUhVvEDqogNZ7vw9mJSUjXw+3Ss/crjau7voQmputKRgqrBKRJlSTrYr7WZ9LFfYnIF0l86KJVxin1pBiGJwBx31sM14Bo1ypzCOVZjDchYi7AOj3+99XIQ1VyQNoFTpiBT5QXrHrU32R0Dj0NiVlst4qRygb+PzhBsA6OOm9NX03D1A5Aa8DqV32kgLUkPtDmE1Vc8H1VWuEGL6j4Jo2IxZum9O9Xtg/E4lHSQ0XmAe6jv5h2NhGG9ax48Uhypq8DwWj3BfwYQO19E1T6xuVfGmqspA0fbd8W4WOV3IRU= ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-12-15T17:13:07.843 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2330505381' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-12-15T17:13:07.880 INFO:teuthology.orchestra.run.smithi022.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwr6ChKHltZMrIDcr6E3Kn0JppWJZ2CeftfpwhB0fxF3HiTX1VEVfBj7TPfMWCnUu9Zpn0s5dIbsFS1ba1YO7qTd4yqEx5A1KGJfrkd6hYVO1sKJU0XPlDzpEvHY9YMM5VtytwLKSetUOOsKF4IUi/xMh0o3hzFpV9QkpGnb2s9kT3eRTwB1dVtlSCUFc5zwcyVS47PPjZOWHM/sQ6ySk1cLUhVvEDqogNZ7vw9mJSUjXw+3Ss/crjau7voQmputKRgqrBKRJlSTrYr7WZ9LFfYnIF0l86KJVxin1pBiGJwBx31sM14Bo1ypzCOVZjDchYi7AOj3+99XIQ1VyQNoFTpiBT5QXrHrU32R0Dj0NiVlst4qRygb+PzhBsA6OOm9NX03D1A5Aa8DqV32kgLUkPtDmE1Vc8H1VWuEGL6j4Jo2IxZum9O9Xtg/E4lHSQ0XmAe6jv5h2NhGG9ax48Uhypq8DwWj3BfwYQO19E1T6xuVfGmqspA0fbd8W4WOV3IRU= ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:07.892 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwr6ChKHltZMrIDcr6E3Kn0JppWJZ2CeftfpwhB0fxF3HiTX1VEVfBj7TPfMWCnUu9Zpn0s5dIbsFS1ba1YO7qTd4yqEx5A1KGJfrkd6hYVO1sKJU0XPlDzpEvHY9YMM5VtytwLKSetUOOsKF4IUi/xMh0o3hzFpV9QkpGnb2s9kT3eRTwB1dVtlSCUFc5zwcyVS47PPjZOWHM/sQ6ySk1cLUhVvEDqogNZ7vw9mJSUjXw+3Ss/crjau7voQmputKRgqrBKRJlSTrYr7WZ9LFfYnIF0l86KJVxin1pBiGJwBx31sM14Bo1ypzCOVZjDchYi7AOj3+99XIQ1VyQNoFTpiBT5QXrHrU32R0Dj0NiVlst4qRygb+PzhBsA6OOm9NX03D1A5Aa8DqV32kgLUkPtDmE1Vc8H1VWuEGL6j4Jo2IxZum9O9Xtg/E4lHSQ0XmAe6jv5h2NhGG9ax48Uhypq8DwWj3BfwYQO19E1T6xuVfGmqspA0fbd8W4WOV3IRU= ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-12-15T17:13:07.936 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwr6ChKHltZMrIDcr6E3Kn0JppWJZ2CeftfpwhB0fxF3HiTX1VEVfBj7TPfMWCnUu9Zpn0s5dIbsFS1ba1YO7qTd4yqEx5A1KGJfrkd6hYVO1sKJU0XPlDzpEvHY9YMM5VtytwLKSetUOOsKF4IUi/xMh0o3hzFpV9QkpGnb2s9kT3eRTwB1dVtlSCUFc5zwcyVS47PPjZOWHM/sQ6ySk1cLUhVvEDqogNZ7vw9mJSUjXw+3Ss/crjau7voQmputKRgqrBKRJlSTrYr7WZ9LFfYnIF0l86KJVxin1pBiGJwBx31sM14Bo1ypzCOVZjDchYi7AOj3+99XIQ1VyQNoFTpiBT5QXrHrU32R0Dj0NiVlst4qRygb+PzhBsA6OOm9NX03D1A5Aa8DqV32kgLUkPtDmE1Vc8H1VWuEGL6j4Jo2IxZum9O9Xtg/E4lHSQ0XmAe6jv5h2NhGG9ax48Uhypq8DwWj3BfwYQO19E1T6xuVfGmqspA0fbd8W4WOV3IRU= ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:07.949 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-12-15T17:13:09.873 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:09 smithi022 ceph-mon[31360]: mgrmap e13: y(active, since 6s) 2021-12-15T17:13:10.198 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2021-12-15T17:13:10.198 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2021-12-15T17:13:11.041 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3254258478' entity='client.admin' 2021-12-15T17:13:11.042 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:10 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:11.042 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:10 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:13:11.042 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:10 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:11.042 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:10 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:11.043 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:10 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:12.674 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2021-12-15T17:13:12.674 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:13:12.675 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2021-12-15T17:13:12.695 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:13:12.695 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:12.754 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2021-12-15T17:13:12.754 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch host add smithi122 2021-12-15T17:13:13.185 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: from='client.14172 v1:172.21.15.22:0/72517496' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:13.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:13.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:13.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:13.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:13.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:13:13.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:13.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:13.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:15.304 INFO:teuthology.orchestra.run.smithi022.stdout:Added host 'smithi122' with addr '172.21.15.122' 2021-12-15T17:13:15.681 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch host ls --format=json 2021-12-15T17:13:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:15 smithi022 ceph-mon[31360]: from='client.14174 v1:172.21.15.22:0/2409521881' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:15 smithi022 ceph-mon[31360]: Deploying cephadm binary to smithi122 2021-12-15T17:13:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:16 smithi022 ceph-mon[31360]: Added host smithi122 2021-12-15T17:13:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:16 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:17.300 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:13:17.300 INFO:teuthology.orchestra.run.smithi022.stdout:[{"addr": "172.21.15.22", "hostname": "smithi022", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2021-12-15T17:13:17.909 INFO:tasks.cephadm:Setting crush tunables to default 2021-12-15T17:13:17.910 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd crush tunables default 2021-12-15T17:13:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:17 smithi022 ceph-mon[31360]: from='client.14176 v1:172.21.15.22:0/566346912' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:13:19.516 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2021-12-15T17:13:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3738424088' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-12-15T17:13:19.834 INFO:tasks.cephadm:Adding mon.a on smithi022 2021-12-15T17:13:19.835 INFO:tasks.cephadm:Adding mon.c on smithi022 2021-12-15T17:13:19.836 INFO:tasks.cephadm:Adding mon.b on smithi122 2021-12-15T17:13:19.836 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply mon '3;smithi022:[v1:172.21.15.22:6789]=a;smithi022:[v1:172.21.15.22:6790]=c;smithi122:[v1:172.21.15.122:6789]=b' 2021-12-15T17:13:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3738424088' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-12-15T17:13:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:20 smithi022 ceph-mon[31360]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:13:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:21.589 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2021-12-15T17:13:22.069 DEBUG:teuthology.orchestra.run.smithi022:mon.c> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mon.c.service 2021-12-15T17:13:22.072 DEBUG:teuthology.orchestra.run.smithi122:mon.b> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mon.b.service 2021-12-15T17:13:22.074 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-12-15T17:13:22.075 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph mon dump -f json 2021-12-15T17:13:22.099 INFO:journalctl@ceph.mon.c.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:13:22.100 INFO:journalctl@ceph.mon.b.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:13:22.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='client.14180 v1:172.21.15.122:0/1643773664' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi022:[v1:172.21.15.22:6789]=a;smithi022:[v1:172.21.15.22:6790]=c;smithi122:[v1:172.21.15.122:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:22.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: Saving service mon spec with placement smithi022:[v1:172.21.15.22:6789]=a;smithi022:[v1:172.21.15.22:6790]=c;smithi122:[v1:172.21.15.122:6789]=b;count:3 2021-12-15T17:13:22.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:22.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:22.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:22.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:22.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:13:22.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:22.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:22.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:22.953 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:22.953 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:23.628 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:23 smithi022 ceph-mon[31360]: Deploying daemon mon.c on smithi022 2021-12-15T17:13:23.628 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:23 smithi022 ceph-mon[31360]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:23.935 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:13:23.936 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"0ea1c4a2-5dca-11ec-8c31-001a4aab830c","modified":"2021-12-15T17:12:11.469714Z","created":"2021-12-15T17:12:11.469714Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-12-15T17:13:23.937 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2021-12-15T17:13:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 systemd[1]: Starting Ceph mon.c for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:13:24.644 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/1594523919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.645 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 podman[40185]: 2021-12-15T17:13:24.646 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 bash[40033]: 3f1565a319ea48310ca431b47d5851d7fec9fed2985e5d4fb35af3bce0d58e44 2021-12-15T17:13:24.647 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: set uid:gid to 167:167 (ceph:ceph) 2021-12-15T17:13:24.648 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: ceph version 17.0.0-9665-gaa33f58d (aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2) quincy (dev), process ceph-mon, pid 7 2021-12-15T17:13:24.648 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: pidfile_write: ignore empty --pid-file 2021-12-15T17:13:24.649 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: load: jerasure load: lrc 2021-12-15T17:13:24.649 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: RocksDB version: 6.15.5 2021-12-15T17:13:24.650 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-12-15T17:13:24.650 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Compile date Dec 14 2021 2021-12-15T17:13:24.650 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: DB SUMMARY 2021-12-15T17:13:24.651 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: DB Session ID: FXVTIOFFSZ9GRCSBHOOL 2021-12-15T17:13:24.651 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: CURRENT file: CURRENT 2021-12-15T17:13:24.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: IDENTITY file: IDENTITY 2021-12-15T17:13:24.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2021-12-15T17:13:24.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2021-12-15T17:13:24.653 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 476 ; 2021-12-15T17:13:24.653 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.error_if_exists: 0 2021-12-15T17:13:24.654 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.create_if_missing: 0 2021-12-15T17:13:24.654 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.paranoid_checks: 1 2021-12-15T17:13:24.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2021-12-15T17:13:24.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.env: 0x55c0e7a0ada0 2021-12-15T17:13:24.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.fs: Posix File System 2021-12-15T17:13:24.656 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.info_log: 0x55c0e92601a0 2021-12-15T17:13:24.656 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_file_opening_threads: 16 2021-12-15T17:13:24.657 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.statistics: (nil) 2021-12-15T17:13:24.657 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.use_fsync: 0 2021-12-15T17:13:24.658 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_log_file_size: 0 2021-12-15T17:13:24.658 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_manifest_file_size: 1073741824 2021-12-15T17:13:24.658 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.log_file_time_to_roll: 0 2021-12-15T17:13:24.659 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.keep_log_file_num: 1000 2021-12-15T17:13:24.659 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.recycle_log_file_num: 0 2021-12-15T17:13:24.660 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.allow_fallocate: 1 2021-12-15T17:13:24.660 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.allow_mmap_reads: 0 2021-12-15T17:13:24.661 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.allow_mmap_writes: 0 2021-12-15T17:13:24.661 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.use_direct_reads: 0 2021-12-15T17:13:24.661 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-12-15T17:13:24.662 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.create_missing_column_families: 0 2021-12-15T17:13:24.662 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.db_log_dir: 2021-12-15T17:13:24.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-12-15T17:13:24.664 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.table_cache_numshardbits: 6 2021-12-15T17:13:24.664 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.WAL_ttl_seconds: 0 2021-12-15T17:13:24.666 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.WAL_size_limit_MB: 0 2021-12-15T17:13:24.666 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-12-15T17:13:24.666 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.manifest_preallocation_size: 4194304 2021-12-15T17:13:24.666 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.is_fd_close_on_exec: 1 2021-12-15T17:13:24.666 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.advise_random_on_open: 1 2021-12-15T17:13:24.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.db_write_buffer_size: 0 2021-12-15T17:13:24.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.write_buffer_manager: 0x55c0e933e0c0 2021-12-15T17:13:24.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.access_hint_on_compaction_start: 1 2021-12-15T17:13:24.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-12-15T17:13:24.668 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.random_access_max_buffer_size: 1048576 2021-12-15T17:13:24.668 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.use_adaptive_mutex: 0 2021-12-15T17:13:24.669 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.rate_limiter: (nil) 2021-12-15T17:13:24.669 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-12-15T17:13:24.669 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.wal_recovery_mode: 2 2021-12-15T17:13:24.669 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.enable_thread_tracking: 0 2021-12-15T17:13:24.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.enable_pipelined_write: 0 2021-12-15T17:13:24.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.unordered_write: 0 2021-12-15T17:13:24.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.allow_concurrent_memtable_write: 1 2021-12-15T17:13:24.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-12-15T17:13:24.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.write_thread_max_yield_usec: 100 2021-12-15T17:13:24.671 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.write_thread_slow_yield_usec: 3 2021-12-15T17:13:24.671 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.row_cache: None 2021-12-15T17:13:24.671 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.wal_filter: None 2021-12-15T17:13:24.671 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.avoid_flush_during_recovery: 0 2021-12-15T17:13:24.671 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.allow_ingest_behind: 0 2021-12-15T17:13:24.672 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.preserve_deletes: 0 2021-12-15T17:13:24.672 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.two_write_queues: 0 2021-12-15T17:13:24.672 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.manual_wal_flush: 0 2021-12-15T17:13:24.672 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.atomic_flush: 0 2021-12-15T17:13:24.672 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-12-15T17:13:24.673 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.persist_stats_to_disk: 0 2021-12-15T17:13:24.673 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.write_dbid_to_manifest: 0 2021-12-15T17:13:24.673 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.log_readahead_size: 0 2021-12-15T17:13:24.674 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.file_checksum_gen_factory: Unknown 2021-12-15T17:13:24.674 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.best_efforts_recovery: 0 2021-12-15T17:13:24.674 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2021-12-15T17:13:24.674 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2021-12-15T17:13:24.674 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.allow_data_in_errors: 0 2021-12-15T17:13:24.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.db_host_id: __hostname__ 2021-12-15T17:13:24.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_background_jobs: 2 2021-12-15T17:13:24.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_background_compactions: -1 2021-12-15T17:13:24.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_subcompactions: 1 2021-12-15T17:13:24.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.avoid_flush_during_shutdown: 0 2021-12-15T17:13:24.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-12-15T17:13:24.677 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.delayed_write_rate : 16777216 2021-12-15T17:13:24.677 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_total_wal_size: 0 2021-12-15T17:13:24.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-12-15T17:13:24.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.stats_dump_period_sec: 600 2021-12-15T17:13:24.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.stats_persist_period_sec: 600 2021-12-15T17:13:24.679 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.stats_history_buffer_size: 1048576 2021-12-15T17:13:24.679 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_open_files: -1 2021-12-15T17:13:24.679 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bytes_per_sync: 0 2021-12-15T17:13:24.679 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.wal_bytes_per_sync: 0 2021-12-15T17:13:24.679 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.strict_bytes_per_sync: 0 2021-12-15T17:13:24.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_readahead_size: 0 2021-12-15T17:13:24.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_background_flushes: -1 2021-12-15T17:13:24.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Compression algorithms supported: 2021-12-15T17:13:24.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kZSTDNotFinalCompression supported: 0 2021-12-15T17:13:24.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kZSTD supported: 0 2021-12-15T17:13:24.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kXpressCompression supported: 0 2021-12-15T17:13:24.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kLZ4HCCompression supported: 1 2021-12-15T17:13:24.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kLZ4Compression supported: 1 2021-12-15T17:13:24.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kBZip2Compression supported: 0 2021-12-15T17:13:24.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kZlibCompression supported: 1 2021-12-15T17:13:24.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: kSnappyCompression supported: 1 2021-12-15T17:13:24.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Fast CRC32 supported: Supported on x86 2021-12-15T17:13:24.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2021-12-15T17:13:24.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2021-12-15T17:13:24.683 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-12-15T17:13:24.683 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.merge_operator: 2021-12-15T17:13:24.683 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_filter: None 2021-12-15T17:13:24.683 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_filter_factory: None 2021-12-15T17:13:24.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.sst_partitioner_factory: None 2021-12-15T17:13:24.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.memtable_factory: SkipListFactory 2021-12-15T17:13:24.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.table_factory: BlockBasedTable 2021-12-15T17:13:24.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c0e922bbb0) 2021-12-15T17:13:24.685 INFO:journalctl@ceph.mon.c.smithi022.stdout: cache_index_and_filter_blocks: 1 2021-12-15T17:13:24.685 INFO:journalctl@ceph.mon.c.smithi022.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2021-12-15T17:13:24.685 INFO:journalctl@ceph.mon.c.smithi022.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2021-12-15T17:13:24.685 INFO:journalctl@ceph.mon.c.smithi022.stdout: pin_top_level_index_and_filter: 1 2021-12-15T17:13:24.686 INFO:journalctl@ceph.mon.c.smithi022.stdout: index_type: 0 2021-12-15T17:13:24.686 INFO:journalctl@ceph.mon.c.smithi022.stdout: data_block_index_type: 0 2021-12-15T17:13:24.686 INFO:journalctl@ceph.mon.c.smithi022.stdout: index_shortening: 1 2021-12-15T17:13:24.686 INFO:journalctl@ceph.mon.c.smithi022.stdout: data_block_hash_table_util_ratio: 0.750000 2021-12-15T17:13:24.687 INFO:journalctl@ceph.mon.c.smithi022.stdout: hash_index_allow_collision: 1 2021-12-15T17:13:24.687 INFO:journalctl@ceph.mon.c.smithi022.stdout: checksum: 1 2021-12-15T17:13:24.687 INFO:journalctl@ceph.mon.c.smithi022.stdout: no_block_cache: 0 2021-12-15T17:13:24.687 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_cache: 0x55c0e925cd10 2021-12-15T17:13:24.688 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_cache_name: BinnedLRUCache 2021-12-15T17:13:24.688 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_cache_options: 2021-12-15T17:13:24.688 INFO:journalctl@ceph.mon.c.smithi022.stdout: capacity : 536870912 2021-12-15T17:13:24.688 INFO:journalctl@ceph.mon.c.smithi022.stdout: num_shard_bits : 4 2021-12-15T17:13:24.688 INFO:journalctl@ceph.mon.c.smithi022.stdout: strict_capacity_limit : 0 2021-12-15T17:13:24.689 INFO:journalctl@ceph.mon.c.smithi022.stdout: high_pri_pool_ratio: 0.000 2021-12-15T17:13:24.689 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_cache_compressed: (nil) 2021-12-15T17:13:24.689 INFO:journalctl@ceph.mon.c.smithi022.stdout: persistent_cache: (nil) 2021-12-15T17:13:24.689 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_size: 4096 2021-12-15T17:13:24.689 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_size_deviation: 10 2021-12-15T17:13:24.690 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_restart_interval: 16 2021-12-15T17:13:24.690 INFO:journalctl@ceph.mon.c.smithi022.stdout: index_block_restart_interval: 1 2021-12-15T17:13:24.690 INFO:journalctl@ceph.mon.c.smithi022.stdout: metadata_block_size: 4096 2021-12-15T17:13:24.690 INFO:journalctl@ceph.mon.c.smithi022.stdout: partition_filters: 0 2021-12-15T17:13:24.690 INFO:journalctl@ceph.mon.c.smithi022.stdout: use_delta_encoding: 1 2021-12-15T17:13:24.691 INFO:journalctl@ceph.mon.c.smithi022.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2021-12-15T17:13:24.691 INFO:journalctl@ceph.mon.c.smithi022.stdout: whole_key_filtering: 1 2021-12-15T17:13:24.691 INFO:journalctl@ceph.mon.c.smithi022.stdout: verify_compression: 0 2021-12-15T17:13:24.691 INFO:journalctl@ceph.mon.c.smithi022.stdout: read_amp_bytes_per_bit: 0 2021-12-15T17:13:24.691 INFO:journalctl@ceph.mon.c.smithi022.stdout: format_version: 4 2021-12-15T17:13:24.692 INFO:journalctl@ceph.mon.c.smithi022.stdout: enable_index_compression: 1 2021-12-15T17:13:24.692 INFO:journalctl@ceph.mon.c.smithi022.stdout: block_align: 0 2021-12-15T17:13:24.692 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.write_buffer_size: 33554432 2021-12-15T17:13:24.692 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_write_buffer_number: 2 2021-12-15T17:13:24.692 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression: NoCompression 2021-12-15T17:13:24.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression: Disabled 2021-12-15T17:13:24.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.prefix_extractor: nullptr 2021-12-15T17:13:24.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-12-15T17:13:24.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.num_levels: 7 2021-12-15T17:13:24.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-12-15T17:13:24.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-12-15T17:13:24.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-12-15T17:13:24.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-12-15T17:13:24.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression_opts.level: 32767 2021-12-15T17:13:24.695 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-12-15T17:13:24.695 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-12-15T17:13:24.695 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-12-15T17:13:24.695 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2021-12-15T17:13:24.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bottommost_compression_opts.enabled: false 2021-12-15T17:13:24.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression_opts.window_bits: -14 2021-12-15T17:13:24.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression_opts.level: 32767 2021-12-15T17:13:24.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression_opts.strategy: 0 2021-12-15T17:13:24.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-12-15T17:13:24.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-12-15T17:13:24.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression_opts.parallel_threads: 1 2021-12-15T17:13:24.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compression_opts.enabled: false 2021-12-15T17:13:24.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-12-15T17:13:24.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-12-15T17:13:24.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.level0_stop_writes_trigger: 36 2021-12-15T17:13:24.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.target_file_size_base: 67108864 2021-12-15T17:13:24.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.target_file_size_multiplier: 1 2021-12-15T17:13:24.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_base: 268435456 2021-12-15T17:13:24.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-12-15T17:13:24.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-12-15T17:13:24.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-12-15T17:13:24.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-12-15T17:13:24.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-12-15T17:13:24.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-12-15T17:13:24.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-12-15T17:13:24.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-12-15T17:13:24.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-12-15T17:13:24.701 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-12-15T17:13:24.701 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_compaction_bytes: 1677721600 2021-12-15T17:13:24.701 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.arena_block_size: 4194304 2021-12-15T17:13:24.701 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-12-15T17:13:24.701 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-12-15T17:13:24.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-12-15T17:13:24.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.disable_auto_compactions: 0 2021-12-15T17:13:24.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-12-15T17:13:24.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-12-15T17:13:24.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-12-15T17:13:24.703 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-12-15T17:13:24.703 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-12-15T17:13:24.703 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-12-15T17:13:24.703 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-12-15T17:13:24.703 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-12-15T17:13:24.704 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-12-15T17:13:24.704 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-12-15T17:13:24.704 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.table_properties_collectors: 2021-12-15T17:13:24.704 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.inplace_update_support: 0 2021-12-15T17:13:24.705 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.inplace_update_num_locks: 10000 2021-12-15T17:13:24.705 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-12-15T17:13:24.705 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.memtable_whole_key_filtering: 0 2021-12-15T17:13:24.705 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.memtable_huge_page_size: 0 2021-12-15T17:13:24.706 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.bloom_locality: 0 2021-12-15T17:13:24.706 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.max_successive_merges: 0 2021-12-15T17:13:24.706 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.optimize_filters_for_hits: 0 2021-12-15T17:13:24.706 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.paranoid_file_checks: 0 2021-12-15T17:13:24.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.force_consistency_checks: 1 2021-12-15T17:13:24.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.report_bg_io_stats: 0 2021-12-15T17:13:24.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.ttl: 2592000 2021-12-15T17:13:24.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.periodic_compaction_seconds: 0 2021-12-15T17:13:24.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.enable_blob_files: false 2021-12-15T17:13:24.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.min_blob_size: 0 2021-12-15T17:13:24.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.blob_file_size: 268435456 2021-12-15T17:13:24.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.blob_compression_type: NoCompression 2021-12-15T17:13:24.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.enable_blob_garbage_collection: false 2021-12-15T17:13:24.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2021-12-15T17:13:24.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-12-15T17:13:24.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2021-12-15T17:13:24.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2021-12-15T17:13:24.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588404589344, "job": 1, "event": "recovery_started", "wal_files": [4]} 2021-12-15T17:13:24.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2021-12-15T17:13:24.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-12-15T17:13:24.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588404589612, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1510, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1639588404, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e3e1a770-c7fd-40d4-b6d2-90fdaa8e052e", "db_session_id": "FXVTIOFFSZ9GRCSBHOOL"}} 2021-12-15T17:13:24.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2021-12-15T17:13:24.711 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588404589953, "job": 1, "event": "recovery_finished"} 2021-12-15T17:13:24.711 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2021-12-15T17:13:24.711 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55c0e9276a80 2021-12-15T17:13:24.711 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: DB pointer 0x55c0e9287800 2021-12-15T17:13:24.711 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2021-12-15T17:13:24.712 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: rocksdb: [db/db_impl/db_impl.cc:903] 2021-12-15T17:13:24.712 INFO:journalctl@ceph.mon.c.smithi022.stdout: ** DB Stats ** 2021-12-15T17:13:24.712 INFO:journalctl@ceph.mon.c.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:13:24.712 INFO:journalctl@ceph.mon.c.smithi022.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-12-15T17:13:24.712 INFO:journalctl@ceph.mon.c.smithi022.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-12-15T17:13:24.713 INFO:journalctl@ceph.mon.c.smithi022.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-12-15T17:13:24.713 INFO:journalctl@ceph.mon.c.smithi022.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-12-15T17:13:24.713 INFO:journalctl@ceph.mon.c.smithi022.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-12-15T17:13:24.713 INFO:journalctl@ceph.mon.c.smithi022.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-12-15T17:13:24.714 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:13:24.714 INFO:journalctl@ceph.mon.c.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:24.714 INFO:journalctl@ceph.mon.c.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:24.715 INFO:journalctl@ceph.mon.c.smithi022.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:24.715 INFO:journalctl@ceph.mon.c.smithi022.stdout: L0 1/0 1.47 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:24.715 INFO:journalctl@ceph.mon.c.smithi022.stdout: Sum 1/0 1.47 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:24.715 INFO:journalctl@ceph.mon.c.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:24.715 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:13:24.716 INFO:journalctl@ceph.mon.c.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:24.716 INFO:journalctl@ceph.mon.c.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:24.716 INFO:journalctl@ceph.mon.c.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:24.716 INFO:journalctl@ceph.mon.c.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:24.717 INFO:journalctl@ceph.mon.c.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:13:24.717 INFO:journalctl@ceph.mon.c.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:24.717 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:24.717 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2021-12-15T17:13:24.717 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2021-12-15T17:13:24.718 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2021-12-15T17:13:24.718 INFO:journalctl@ceph.mon.c.smithi022.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:24.718 INFO:journalctl@ceph.mon.c.smithi022.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:24.718 INFO:journalctl@ceph.mon.c.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-12-15T17:13:24.718 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:13:24.719 INFO:journalctl@ceph.mon.c.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2021-12-15T17:13:24.719 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:13:24.719 INFO:journalctl@ceph.mon.c.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:24.719 INFO:journalctl@ceph.mon.c.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:24.719 INFO:journalctl@ceph.mon.c.smithi022.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:24.720 INFO:journalctl@ceph.mon.c.smithi022.stdout: L0 1/0 1.47 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:24.720 INFO:journalctl@ceph.mon.c.smithi022.stdout: Sum 1/0 1.47 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:24.720 INFO:journalctl@ceph.mon.c.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-12-15T17:13:24.720 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:13:24.720 INFO:journalctl@ceph.mon.c.smithi022.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:24.721 INFO:journalctl@ceph.mon.c.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:24.721 INFO:journalctl@ceph.mon.c.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:24.721 INFO:journalctl@ceph.mon.c.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:24.721 INFO:journalctl@ceph.mon.c.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:13:24.722 INFO:journalctl@ceph.mon.c.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:24.722 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:24.722 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2021-12-15T17:13:24.722 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2021-12-15T17:13:24.723 INFO:journalctl@ceph.mon.c.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2021-12-15T17:13:24.723 INFO:journalctl@ceph.mon.c.smithi022.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:24.723 INFO:journalctl@ceph.mon.c.smithi022.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:24.725 INFO:journalctl@ceph.mon.c.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-12-15T17:13:24.725 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:13:24.725 INFO:journalctl@ceph.mon.c.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2021-12-15T17:13:24.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c does not exist in monmap, will attempt to join an existing cluster 2021-12-15T17:13:24.726 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: using public_addrv v1:172.21.15.22:6790/0 2021-12-15T17:13:24.726 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: starting mon.c rank -1 at public addrs v1:172.21.15.22:6790/0 at bind addrs v1:172.21.15.22:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:24.726 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(???) e0 preinit fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:24.726 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).mds e1 new map 2021-12-15T17:13:24.726 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).mds e1 print_map 2021-12-15T17:13:24.727 INFO:journalctl@ceph.mon.c.smithi022.stdout: e1 2021-12-15T17:13:24.727 INFO:journalctl@ceph.mon.c.smithi022.stdout: enable_multiple, ever_enabled_multiple: 1,1 2021-12-15T17:13:24.727 INFO:journalctl@ceph.mon.c.smithi022.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-12-15T17:13:24.727 INFO:journalctl@ceph.mon.c.smithi022.stdout: legacy client fscid: -1 2021-12-15T17:13:24.727 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:13:24.728 INFO:journalctl@ceph.mon.c.smithi022.stdout: No filesystems configured 2021-12-15T17:13:24.728 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-12-15T17:13:24.728 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-12-15T17:13:24.728 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-12-15T17:13:24.728 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-12-15T17:13:24.729 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-12-15T17:13:24.729 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-12-15T17:13:24.729 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-12-15T17:13:24.729 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:13:24.730 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:13:24.730 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:13:24.730 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mkfs 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:24.730 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:13:24.730 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:13:24.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: monmap e1: 1 mons at {a=v1:172.21.15.22:6789/0} 2021-12-15T17:13:24.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: fsmap 2021-12-15T17:13:24.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: osdmap e1: 0 total, 0 up, 0 in 2021-12-15T17:13:24.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e1: no daemons active 2021-12-15T17:13:24.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3266039324' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-12-15T17:13:24.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2931260039' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-12-15T17:13:24.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2931260039' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-12-15T17:13:24.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:13:24.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: monmap e1: 1 mons at {a=v1:172.21.15.22:6789/0} 2021-12-15T17:13:24.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: fsmap 2021-12-15T17:13:24.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: osdmap e1: 0 total, 0 up, 0 in 2021-12-15T17:13:24.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e1: no daemons active 2021-12-15T17:13:24.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3644780694' entity='client.admin' 2021-12-15T17:13:24.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2416579838' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:24.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3594010203' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:24.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Activating manager daemon y 2021-12-15T17:13:24.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e2: y(active, starting, since 0.00279219s) 2021-12-15T17:13:24.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:24.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:24.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:24.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:24.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:24.736 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:24.736 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:24.736 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:13:24.736 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Manager daemon y is now available 2021-12-15T17:13:24.736 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:13:24.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:13:24.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' 2021-12-15T17:13:24.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' 2021-12-15T17:13:24.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2766790301' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:24.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e3: y(active, since 1.00676s) 2021-12-15T17:13:24.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e4: y(active, since 2s) 2021-12-15T17:13:24.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/886693411' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:24.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/291934710' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-12-15T17:13:24.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/335991765' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-12-15T17:13:24.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/335991765' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-12-15T17:13:24.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e5: y(active, since 8s) 2021-12-15T17:13:24.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/838481017' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-12-15T17:13:24.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Active manager daemon y restarted 2021-12-15T17:13:24.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Activating manager daemon y 2021-12-15T17:13:24.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: osdmap e2: 0 total, 0 up, 0 in 2021-12-15T17:13:24.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e6: y(active, starting, since 0.052623s) 2021-12-15T17:13:24.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:24.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:13:24.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:24.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:24.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:24.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Manager daemon y is now available 2021-12-15T17:13:24.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:13:24.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:13:24.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e7: y(active, since 1.05487s) 2021-12-15T17:13:24.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: [15/Dec/2021:17:12:43] ENGINE Bus STARTING 2021-12-15T17:13:24.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: [15/Dec/2021:17:12:43] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:13:24.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: [15/Dec/2021:17:12:43] ENGINE Bus STARTED 2021-12-15T17:13:24.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14132 v1:172.21.15.22:0/1266900416' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e8: y(active, since 2s) 2021-12-15T17:13:24.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14134 v1:172.21.15.22:0/899131883' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.746 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.746 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14136 v1:172.21.15.22:0/2997050750' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.746 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Generating ssh key... 2021-12-15T17:13:24.746 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14138 v1:172.21.15.22:0/2853550753' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.746 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14140 v1:172.21.15.22:0/2303086545' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi022", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Deploying cephadm binary to smithi022 2021-12-15T17:13:24.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Added host smithi022 2021-12-15T17:13:24.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.748 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14142 v1:172.21.15.22:0/2607737481' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.748 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Saving service mon spec with placement count:5 2021-12-15T17:13:24.748 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.748 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14144 v1:172.21.15.22:0/2306284900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.748 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Saving service mgr spec with placement count:2 2021-12-15T17:13:24.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3988911567' entity='client.admin' 2021-12-15T17:13:24.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3677929078' entity='client.admin' 2021-12-15T17:13:24.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:24.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1632747641' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-12-15T17:13:24.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1632747641' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-12-15T17:13:24.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e9: y(active, since 13s) 2021-12-15T17:13:24.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2707799736' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-12-15T17:13:24.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Active manager daemon y restarted 2021-12-15T17:13:24.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Activating manager daemon y 2021-12-15T17:13:24.752 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: osdmap e3: 0 total, 0 up, 0 in 2021-12-15T17:13:24.752 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e10: y(active, starting, since 0.0525412s) 2021-12-15T17:13:24.752 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:24.752 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:13:24.752 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:24.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:24.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:24.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Manager daemon y is now available 2021-12-15T17:13:24.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:13:24.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:13:24.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e11: y(active, since 1.05512s) 2021-12-15T17:13:24.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: [15/Dec/2021:17:13:03] ENGINE Bus STARTING 2021-12-15T17:13:24.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: [15/Dec/2021:17:13:03] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:13:24.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: [15/Dec/2021:17:13:03] ENGINE Bus STARTED 2021-12-15T17:13:24.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14164 v1:172.21.15.22:0/1947165790' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e12: y(active, since 2s) 2021-12-15T17:13:24.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14166 v1:172.21.15.22:0/1590284040' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2330505381' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-12-15T17:13:24.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mgrmap e13: y(active, since 6s) 2021-12-15T17:13:24.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3254258478' entity='client.admin' 2021-12-15T17:13:24.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:13:24.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14172 v1:172.21.15.22:0/72517496' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.761 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.761 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:24.761 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:24.761 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:13:24.762 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:24.762 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.762 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.762 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14174 v1:172.21.15.22:0/2409521881' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.762 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Deploying cephadm binary to smithi122 2021-12-15T17:13:24.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:24.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:24.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Added host smithi122 2021-12-15T17:13:24.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14176 v1:172.21.15.22:0/566346912' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:13:24.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3738424088' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-12-15T17:13:24.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3738424088' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-12-15T17:13:24.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:24.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:13:24.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.14180 v1:172.21.15.122:0/1643773664' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi022:[v1:172.21.15.22:6789]=a;smithi022:[v1:172.21.15.22:6790]=c;smithi122:[v1:172.21.15.122:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:24.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Saving service mon spec with placement smithi022:[v1:172.21.15.22:6789]=a;smithi022:[v1:172.21.15.22:6790]=c;smithi122:[v1:172.21.15.122:6789]=b;count:3 2021-12-15T17:13:24.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:24.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:24.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:13:24.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:24.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.768 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:24.768 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:24.768 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:24.771 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: Deploying daemon mon.c on smithi022 2021-12-15T17:13:24.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:24.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/1594523919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:24.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2021-12-15T17:13:24.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 systemd[1]: Started Ceph mon.c for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:13:24.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:24 smithi022 ceph-mon[40250]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2021-12-15T17:13:25.441 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-12-15T17:13:25.442 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph mon dump -f json 2021-12-15T17:13:29.911 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:13:29.912 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"0ea1c4a2-5dca-11ec-8c31-001a4aab830c","modified":"2021-12-15T17:13:24.611771Z","created":"2021-12-15T17:12:11.469714Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6790","nonce":0}]},"addr":"172.21.15.22:6790/0","public_addr":"172.21.15.22:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-12-15T17:13:29.913 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2021-12-15T17:13:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: mon.a calling monitor election 2021-12-15T17:13:30.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:30.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: mon.c calling monitor election 2021-12-15T17:13:30.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:30.038 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.038 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.039 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-12-15T17:13:30.039 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: monmap e2: 2 mons at {a=v1:172.21.15.22:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:30.040 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: fsmap 2021-12-15T17:13:30.040 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:30.041 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: mgrmap e13: y(active, since 27s) 2021-12-15T17:13:30.041 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T17:13:30.042 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:30.043 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:30.046 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:30.046 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:30.047 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:30.047 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: mon.a calling monitor election 2021-12-15T17:13:30.048 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.048 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.048 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:30.049 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.049 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: mon.c calling monitor election 2021-12-15T17:13:30.050 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.050 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:30.051 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.051 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:30.052 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-12-15T17:13:30.052 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: monmap e2: 2 mons at {a=v1:172.21.15.22:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:30.053 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: fsmap 2021-12-15T17:13:30.053 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:30.054 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: mgrmap e13: y(active, since 27s) 2021-12-15T17:13:30.054 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T17:13:30.055 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:30.055 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:30.055 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:30 smithi022 ceph-mon[31360]: Deploying daemon mon.b on smithi122 2021-12-15T17:13:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3530198743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:30 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:30 smithi022 ceph-mon[40250]: Deploying daemon mon.b on smithi122 2021-12-15T17:13:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3530198743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:30 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:31.450 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-12-15T17:13:31.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph mon dump -f json 2021-12-15T17:13:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:31 smithi022 ceph-mon[31360]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:31 smithi022 ceph-mon[40250]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:32.671 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:32 smithi122 systemd[1]: Starting Ceph mon.b for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:13:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 podman[32144]: 2021-12-15T17:13:33.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: set uid:gid to 167:167 (ceph:ceph) 2021-12-15T17:13:33.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: ceph version 17.0.0-9665-gaa33f58d (aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2) quincy (dev), process ceph-mon, pid 7 2021-12-15T17:13:33.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: pidfile_write: ignore empty --pid-file 2021-12-15T17:13:33.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: load: jerasure load: lrc 2021-12-15T17:13:33.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: RocksDB version: 6.15.5 2021-12-15T17:13:33.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-12-15T17:13:33.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Compile date Dec 14 2021 2021-12-15T17:13:33.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: DB SUMMARY 2021-12-15T17:13:33.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: DB Session ID: 419WO8MWV8C1XZQQH8QN 2021-12-15T17:13:33.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: CURRENT file: CURRENT 2021-12-15T17:13:33.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: IDENTITY file: IDENTITY 2021-12-15T17:13:33.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2021-12-15T17:13:33.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2021-12-15T17:13:33.945 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 566 ; 2021-12-15T17:13:33.945 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.error_if_exists: 0 2021-12-15T17:13:33.945 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.create_if_missing: 0 2021-12-15T17:13:33.946 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.paranoid_checks: 1 2021-12-15T17:13:33.946 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2021-12-15T17:13:33.947 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.env: 0x560c39e0cda0 2021-12-15T17:13:33.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.fs: Posix File System 2021-12-15T17:13:33.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.info_log: 0x560c3aa90220 2021-12-15T17:13:33.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_file_opening_threads: 16 2021-12-15T17:13:33.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.statistics: (nil) 2021-12-15T17:13:33.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.use_fsync: 0 2021-12-15T17:13:33.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_log_file_size: 0 2021-12-15T17:13:33.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_manifest_file_size: 1073741824 2021-12-15T17:13:33.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.log_file_time_to_roll: 0 2021-12-15T17:13:33.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.keep_log_file_num: 1000 2021-12-15T17:13:33.951 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.recycle_log_file_num: 0 2021-12-15T17:13:33.951 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.allow_fallocate: 1 2021-12-15T17:13:33.952 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.allow_mmap_reads: 0 2021-12-15T17:13:33.952 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.allow_mmap_writes: 0 2021-12-15T17:13:33.952 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.use_direct_reads: 0 2021-12-15T17:13:33.952 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-12-15T17:13:33.953 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.create_missing_column_families: 0 2021-12-15T17:13:33.953 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.db_log_dir: 2021-12-15T17:13:33.953 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2021-12-15T17:13:33.953 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.table_cache_numshardbits: 6 2021-12-15T17:13:33.953 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.WAL_ttl_seconds: 0 2021-12-15T17:13:33.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.WAL_size_limit_MB: 0 2021-12-15T17:13:33.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-12-15T17:13:33.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.manifest_preallocation_size: 4194304 2021-12-15T17:13:33.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.is_fd_close_on_exec: 1 2021-12-15T17:13:33.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.advise_random_on_open: 1 2021-12-15T17:13:33.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.db_write_buffer_size: 0 2021-12-15T17:13:33.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.write_buffer_manager: 0x560c3ab6a120 2021-12-15T17:13:33.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.access_hint_on_compaction_start: 1 2021-12-15T17:13:33.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-12-15T17:13:33.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.random_access_max_buffer_size: 1048576 2021-12-15T17:13:33.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.use_adaptive_mutex: 0 2021-12-15T17:13:33.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.rate_limiter: (nil) 2021-12-15T17:13:33.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-12-15T17:13:33.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.wal_recovery_mode: 2 2021-12-15T17:13:33.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.enable_thread_tracking: 0 2021-12-15T17:13:33.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.enable_pipelined_write: 0 2021-12-15T17:13:33.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.unordered_write: 0 2021-12-15T17:13:33.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.allow_concurrent_memtable_write: 1 2021-12-15T17:13:33.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-12-15T17:13:33.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.write_thread_max_yield_usec: 100 2021-12-15T17:13:33.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.write_thread_slow_yield_usec: 3 2021-12-15T17:13:33.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.row_cache: None 2021-12-15T17:13:33.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.wal_filter: None 2021-12-15T17:13:33.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.avoid_flush_during_recovery: 0 2021-12-15T17:13:33.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.allow_ingest_behind: 0 2021-12-15T17:13:33.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.preserve_deletes: 0 2021-12-15T17:13:33.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.two_write_queues: 0 2021-12-15T17:13:33.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.manual_wal_flush: 0 2021-12-15T17:13:33.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.atomic_flush: 0 2021-12-15T17:13:33.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-12-15T17:13:33.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.persist_stats_to_disk: 0 2021-12-15T17:13:33.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.write_dbid_to_manifest: 0 2021-12-15T17:13:33.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.log_readahead_size: 0 2021-12-15T17:13:33.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.file_checksum_gen_factory: Unknown 2021-12-15T17:13:33.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.best_efforts_recovery: 0 2021-12-15T17:13:33.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2021-12-15T17:13:33.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2021-12-15T17:13:33.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.allow_data_in_errors: 0 2021-12-15T17:13:33.963 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.db_host_id: __hostname__ 2021-12-15T17:13:33.963 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_background_jobs: 2 2021-12-15T17:13:33.963 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_background_compactions: -1 2021-12-15T17:13:33.963 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_subcompactions: 1 2021-12-15T17:13:33.963 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.avoid_flush_during_shutdown: 0 2021-12-15T17:13:33.964 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-12-15T17:13:33.964 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.delayed_write_rate : 16777216 2021-12-15T17:13:33.964 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_total_wal_size: 0 2021-12-15T17:13:33.964 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-12-15T17:13:33.965 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.stats_dump_period_sec: 600 2021-12-15T17:13:33.965 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.stats_persist_period_sec: 600 2021-12-15T17:13:33.965 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.stats_history_buffer_size: 1048576 2021-12-15T17:13:33.965 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_open_files: -1 2021-12-15T17:13:33.965 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bytes_per_sync: 0 2021-12-15T17:13:33.966 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.wal_bytes_per_sync: 0 2021-12-15T17:13:33.966 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.strict_bytes_per_sync: 0 2021-12-15T17:13:33.966 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_readahead_size: 0 2021-12-15T17:13:33.966 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_background_flushes: -1 2021-12-15T17:13:33.966 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Compression algorithms supported: 2021-12-15T17:13:33.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kZSTDNotFinalCompression supported: 0 2021-12-15T17:13:33.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kZSTD supported: 0 2021-12-15T17:13:33.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kXpressCompression supported: 0 2021-12-15T17:13:33.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kLZ4HCCompression supported: 1 2021-12-15T17:13:33.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kLZ4Compression supported: 1 2021-12-15T17:13:33.968 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kBZip2Compression supported: 0 2021-12-15T17:13:33.968 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kZlibCompression supported: 1 2021-12-15T17:13:33.968 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: kSnappyCompression supported: 1 2021-12-15T17:13:33.968 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Fast CRC32 supported: Supported on x86 2021-12-15T17:13:33.969 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2021-12-15T17:13:33.969 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2021-12-15T17:13:33.969 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-12-15T17:13:33.969 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.merge_operator: 2021-12-15T17:13:33.969 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_filter: None 2021-12-15T17:13:33.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_filter_factory: None 2021-12-15T17:13:33.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.sst_partitioner_factory: None 2021-12-15T17:13:33.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.memtable_factory: SkipListFactory 2021-12-15T17:13:33.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.table_factory: BlockBasedTable 2021-12-15T17:13:33.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560c3aa5bbb0) 2021-12-15T17:13:33.971 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks: 1 2021-12-15T17:13:33.971 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2021-12-15T17:13:33.971 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2021-12-15T17:13:33.971 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_top_level_index_and_filter: 1 2021-12-15T17:13:33.971 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_type: 0 2021-12-15T17:13:33.972 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_index_type: 0 2021-12-15T17:13:33.972 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_shortening: 1 2021-12-15T17:13:33.972 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2021-12-15T17:13:33.972 INFO:journalctl@ceph.mon.b.smithi122.stdout: hash_index_allow_collision: 1 2021-12-15T17:13:33.973 INFO:journalctl@ceph.mon.b.smithi122.stdout: checksum: 1 2021-12-15T17:13:33.973 INFO:journalctl@ceph.mon.b.smithi122.stdout: no_block_cache: 0 2021-12-15T17:13:33.973 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache: 0x560c3aa8cd10 2021-12-15T17:13:33.973 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_name: BinnedLRUCache 2021-12-15T17:13:33.973 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_options: 2021-12-15T17:13:33.974 INFO:journalctl@ceph.mon.b.smithi122.stdout: capacity : 536870912 2021-12-15T17:13:33.974 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_shard_bits : 4 2021-12-15T17:13:33.974 INFO:journalctl@ceph.mon.b.smithi122.stdout: strict_capacity_limit : 0 2021-12-15T17:13:33.974 INFO:journalctl@ceph.mon.b.smithi122.stdout: high_pri_pool_ratio: 0.000 2021-12-15T17:13:33.974 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_compressed: (nil) 2021-12-15T17:13:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout: persistent_cache: (nil) 2021-12-15T17:13:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size: 4096 2021-12-15T17:13:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size_deviation: 10 2021-12-15T17:13:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_restart_interval: 16 2021-12-15T17:13:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_block_restart_interval: 1 2021-12-15T17:13:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout: metadata_block_size: 4096 2021-12-15T17:13:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout: partition_filters: 0 2021-12-15T17:13:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout: use_delta_encoding: 1 2021-12-15T17:13:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2021-12-15T17:13:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout: whole_key_filtering: 1 2021-12-15T17:13:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout: verify_compression: 0 2021-12-15T17:13:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout: read_amp_bytes_per_bit: 0 2021-12-15T17:13:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout: format_version: 4 2021-12-15T17:13:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_index_compression: 1 2021-12-15T17:13:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_align: 0 2021-12-15T17:13:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.write_buffer_size: 33554432 2021-12-15T17:13:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_write_buffer_number: 2 2021-12-15T17:13:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression: NoCompression 2021-12-15T17:13:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression: Disabled 2021-12-15T17:13:33.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.prefix_extractor: nullptr 2021-12-15T17:13:33.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-12-15T17:13:33.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.num_levels: 7 2021-12-15T17:13:33.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-12-15T17:13:33.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-12-15T17:13:33.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-12-15T17:13:33.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-12-15T17:13:33.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression_opts.level: 32767 2021-12-15T17:13:33.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-12-15T17:13:33.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-12-15T17:13:33.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-12-15T17:13:33.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2021-12-15T17:13:33.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bottommost_compression_opts.enabled: false 2021-12-15T17:13:33.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression_opts.window_bits: -14 2021-12-15T17:13:33.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression_opts.level: 32767 2021-12-15T17:13:33.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression_opts.strategy: 0 2021-12-15T17:13:33.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-12-15T17:13:33.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-12-15T17:13:33.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression_opts.parallel_threads: 1 2021-12-15T17:13:33.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compression_opts.enabled: false 2021-12-15T17:13:33.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-12-15T17:13:33.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-12-15T17:13:33.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.level0_stop_writes_trigger: 36 2021-12-15T17:13:33.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.target_file_size_base: 67108864 2021-12-15T17:13:33.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.target_file_size_multiplier: 1 2021-12-15T17:13:33.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_base: 268435456 2021-12-15T17:13:33.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-12-15T17:13:33.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-12-15T17:13:33.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-12-15T17:13:33.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-12-15T17:13:33.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-12-15T17:13:33.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-12-15T17:13:33.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-12-15T17:13:33.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-12-15T17:13:33.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-12-15T17:13:33.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-12-15T17:13:33.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_compaction_bytes: 1677721600 2021-12-15T17:13:33.987 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.arena_block_size: 4194304 2021-12-15T17:13:33.987 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-12-15T17:13:33.987 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-12-15T17:13:33.987 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-12-15T17:13:33.987 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.disable_auto_compactions: 0 2021-12-15T17:13:33.988 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-12-15T17:13:33.988 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-12-15T17:13:33.988 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-12-15T17:13:33.988 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-12-15T17:13:33.988 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-12-15T17:13:33.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-12-15T17:13:33.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-12-15T17:13:33.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-12-15T17:13:33.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-12-15T17:13:33.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-12-15T17:13:33.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.table_properties_collectors: 2021-12-15T17:13:33.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.inplace_update_support: 0 2021-12-15T17:13:33.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.inplace_update_num_locks: 10000 2021-12-15T17:13:33.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-12-15T17:13:33.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.memtable_whole_key_filtering: 0 2021-12-15T17:13:33.991 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.memtable_huge_page_size: 0 2021-12-15T17:13:33.991 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.bloom_locality: 0 2021-12-15T17:13:33.991 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.max_successive_merges: 0 2021-12-15T17:13:33.991 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.optimize_filters_for_hits: 0 2021-12-15T17:13:33.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.paranoid_file_checks: 0 2021-12-15T17:13:33.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.force_consistency_checks: 1 2021-12-15T17:13:33.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.report_bg_io_stats: 0 2021-12-15T17:13:33.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.ttl: 2592000 2021-12-15T17:13:33.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.periodic_compaction_seconds: 0 2021-12-15T17:13:33.993 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.enable_blob_files: false 2021-12-15T17:13:33.993 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.min_blob_size: 0 2021-12-15T17:13:33.993 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.blob_file_size: 268435456 2021-12-15T17:13:33.993 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.blob_compression_type: NoCompression 2021-12-15T17:13:33.994 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.enable_blob_garbage_collection: false 2021-12-15T17:13:33.994 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2021-12-15T17:13:33.994 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-12-15T17:13:33.994 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2021-12-15T17:13:33.994 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2021-12-15T17:13:33.995 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588413504292, "job": 1, "event": "recovery_started", "wal_files": [4]} 2021-12-15T17:13:33.995 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2021-12-15T17:13:33.995 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-12-15T17:13:33.995 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588413504558, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1600, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1639588413, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "29048376-1122-4ec3-b0c4-02f560f84580", "db_session_id": "419WO8MWV8C1XZQQH8QN"}} 2021-12-15T17:13:33.995 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2021-12-15T17:13:33.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: EVENT_LOG_v1 {"time_micros": 1639588413507211, "job": 1, "event": "recovery_finished"} 2021-12-15T17:13:33.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2021-12-15T17:13:33.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x560c3aaa6a80 2021-12-15T17:13:33.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: DB pointer 0x560c3aab7800 2021-12-15T17:13:33.997 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2021-12-15T17:13:33.997 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: rocksdb: [db/db_impl/db_impl.cc:903] 2021-12-15T17:13:33.997 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** DB Stats ** 2021-12-15T17:13:33.997 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:13:33.997 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-12-15T17:13:33.998 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-12-15T17:13:33.998 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-12-15T17:13:33.998 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-12-15T17:13:33.998 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-12-15T17:13:33.998 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-12-15T17:13:33.999 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:13:33.999 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:33.999 INFO:journalctl@ceph.mon.b.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:34.000 INFO:journalctl@ceph.mon.b.smithi122.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:34.000 INFO:journalctl@ceph.mon.b.smithi122.stdout: L0 1/0 1.56 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.1 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:34.000 INFO:journalctl@ceph.mon.b.smithi122.stdout: Sum 1/0 1.56 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.1 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:34.000 INFO:journalctl@ceph.mon.b.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.1 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:34.000 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:13:34.001 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:34.001 INFO:journalctl@ceph.mon.b.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:34.001 INFO:journalctl@ceph.mon.b.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:34.001 INFO:journalctl@ceph.mon.b.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.1 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:34.002 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:13:34.002 INFO:journalctl@ceph.mon.b.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:34.002 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:34.002 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2021-12-15T17:13:34.002 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2021-12-15T17:13:34.003 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2021-12-15T17:13:34.003 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:34.003 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval compaction: 0.00 GB write, 0.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:34.003 INFO:journalctl@ceph.mon.b.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-12-15T17:13:34.003 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:13:34.004 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2021-12-15T17:13:34.004 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:13:34.004 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:34.005 INFO:journalctl@ceph.mon.b.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:34.005 INFO:journalctl@ceph.mon.b.smithi122.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:34.005 INFO:journalctl@ceph.mon.b.smithi122.stdout: L0 1/0 1.56 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.1 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:34.005 INFO:journalctl@ceph.mon.b.smithi122.stdout: Sum 1/0 1.56 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.1 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:34.005 INFO:journalctl@ceph.mon.b.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-12-15T17:13:34.006 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:13:34.006 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2021-12-15T17:13:34.006 INFO:journalctl@ceph.mon.b.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-12-15T17:13:34.006 INFO:journalctl@ceph.mon.b.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-12-15T17:13:34.006 INFO:journalctl@ceph.mon.b.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.1 0.00 0.00 1 0.000 0 0 2021-12-15T17:13:34.007 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2021-12-15T17:13:34.007 INFO:journalctl@ceph.mon.b.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:34.007 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2021-12-15T17:13:34.007 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2021-12-15T17:13:34.007 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2021-12-15T17:13:34.008 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2021-12-15T17:13:34.008 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:34.008 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-12-15T17:13:34.008 INFO:journalctl@ceph.mon.b.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-12-15T17:13:34.009 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:13:34.009 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2021-12-15T17:13:34.009 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b does not exist in monmap, will attempt to join an existing cluster 2021-12-15T17:13:34.009 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: using public_addrv v1:172.21.15.122:6789/0 2021-12-15T17:13:34.009 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: starting mon.b rank -1 at public addrs v1:172.21.15.122:6789/0 at bind addrs v1:172.21.15.122:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:34.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(???) e0 preinit fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:34.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 bash[31915]: e373293e1e57eaec2e4f339e8f91e869585c8e4c9f536caf6c6ad306616b7180 2021-12-15T17:13:34.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).mds e1 new map 2021-12-15T17:13:34.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).mds e1 print_map 2021-12-15T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi122.stdout: e1 2021-12-15T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2021-12-15T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-12-15T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi122.stdout: legacy client fscid: -1 2021-12-15T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:13:34.012 INFO:journalctl@ceph.mon.b.smithi122.stdout: No filesystems configured 2021-12-15T17:13:34.012 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-12-15T17:13:34.012 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-12-15T17:13:34.012 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-12-15T17:13:34.013 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-12-15T17:13:34.013 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-12-15T17:13:34.013 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-12-15T17:13:34.013 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-12-15T17:13:34.013 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:13:34.014 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:13:34.014 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-12-15T17:13:34.014 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mkfs 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:34.014 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:13:34.014 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:13:34.015 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: monmap e1: 1 mons at {a=v1:172.21.15.22:6789/0} 2021-12-15T17:13:34.015 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: fsmap 2021-12-15T17:13:34.015 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: osdmap e1: 0 total, 0 up, 0 in 2021-12-15T17:13:34.015 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e1: no daemons active 2021-12-15T17:13:34.015 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3266039324' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-12-15T17:13:34.016 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2931260039' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-12-15T17:13:34.016 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2931260039' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-12-15T17:13:34.016 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.a is new leader, mons a in quorum (ranks 0) 2021-12-15T17:13:34.016 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: monmap e1: 1 mons at {a=v1:172.21.15.22:6789/0} 2021-12-15T17:13:34.017 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: fsmap 2021-12-15T17:13:34.017 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: osdmap e1: 0 total, 0 up, 0 in 2021-12-15T17:13:34.017 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e1: no daemons active 2021-12-15T17:13:34.017 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3644780694' entity='client.admin' 2021-12-15T17:13:34.017 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2416579838' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:34.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3594010203' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:34.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Activating manager daemon y 2021-12-15T17:13:34.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e2: y(active, starting, since 0.00279219s) 2021-12-15T17:13:34.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:34.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:34.019 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:34.019 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:34.019 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:34.019 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:34.020 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:34.020 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:13:34.020 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Manager daemon y is now available 2021-12-15T17:13:34.020 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:13:34.020 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:13:34.021 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' 2021-12-15T17:13:34.021 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14100 v1:172.21.15.22:0/2964215838' entity='mgr.y' 2021-12-15T17:13:34.021 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2766790301' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:34.021 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e3: y(active, since 1.00676s) 2021-12-15T17:13:34.022 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e4: y(active, since 2s) 2021-12-15T17:13:34.022 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/886693411' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-12-15T17:13:34.022 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/291934710' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-12-15T17:13:34.022 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/335991765' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-12-15T17:13:34.022 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/335991765' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-12-15T17:13:34.023 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e5: y(active, since 8s) 2021-12-15T17:13:34.023 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/838481017' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-12-15T17:13:34.023 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Active manager daemon y restarted 2021-12-15T17:13:34.023 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Activating manager daemon y 2021-12-15T17:13:34.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: osdmap e2: 0 total, 0 up, 0 in 2021-12-15T17:13:34.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e6: y(active, starting, since 0.052623s) 2021-12-15T17:13:34.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:34.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:13:34.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:34.025 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:34.025 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:34.025 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Manager daemon y is now available 2021-12-15T17:13:34.025 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.025 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.026 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.026 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.026 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.026 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:13:34.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:13:34.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e7: y(active, since 1.05487s) 2021-12-15T17:13:34.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: [15/Dec/2021:17:12:43] ENGINE Bus STARTING 2021-12-15T17:13:34.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: [15/Dec/2021:17:12:43] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:13:34.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: [15/Dec/2021:17:12:43] ENGINE Bus STARTED 2021-12-15T17:13:34.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14132 v1:172.21.15.22:0/1266900416' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e8: y(active, since 2s) 2021-12-15T17:13:34.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14134 v1:172.21.15.22:0/899131883' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14136 v1:172.21.15.22:0/2997050750' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Generating ssh key... 2021-12-15T17:13:34.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14138 v1:172.21.15.22:0/2853550753' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.030 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14140 v1:172.21.15.22:0/2303086545' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi022", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.030 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Deploying cephadm binary to smithi022 2021-12-15T17:13:34.030 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.030 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Added host smithi022 2021-12-15T17:13:34.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14142 v1:172.21.15.22:0/2607737481' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Saving service mon spec with placement count:5 2021-12-15T17:13:34.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14144 v1:172.21.15.22:0/2306284900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Saving service mgr spec with placement count:2 2021-12-15T17:13:34.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3988911567' entity='client.admin' 2021-12-15T17:13:34.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3677929078' entity='client.admin' 2021-12-15T17:13:34.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14120 v1:172.21.15.22:0/2843748951' entity='mgr.y' 2021-12-15T17:13:34.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1632747641' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-12-15T17:13:34.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1632747641' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-12-15T17:13:34.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e9: y(active, since 13s) 2021-12-15T17:13:34.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2707799736' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-12-15T17:13:34.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Active manager daemon y restarted 2021-12-15T17:13:34.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Activating manager daemon y 2021-12-15T17:13:34.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: osdmap e3: 0 total, 0 up, 0 in 2021-12-15T17:13:34.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e10: y(active, starting, since 0.0525412s) 2021-12-15T17:13:34.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:34.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:13:34.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:13:34.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:13:34.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:13:34.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Manager daemon y is now available 2021-12-15T17:13:34.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.037 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:13:34.037 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:13:34.037 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e11: y(active, since 1.05512s) 2021-12-15T17:13:34.037 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.038 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: [15/Dec/2021:17:13:03] ENGINE Bus STARTING 2021-12-15T17:13:34.038 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: [15/Dec/2021:17:13:03] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:13:34.038 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: [15/Dec/2021:17:13:03] ENGINE Bus STARTED 2021-12-15T17:13:34.038 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14164 v1:172.21.15.22:0/1947165790' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.038 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e12: y(active, since 2s) 2021-12-15T17:13:34.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14166 v1:172.21.15.22:0/1590284040' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2330505381' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-12-15T17:13:34.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e13: y(active, since 6s) 2021-12-15T17:13:34.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3254258478' entity='client.admin' 2021-12-15T17:13:34.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:13:34.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14172 v1:172.21.15.22:0/72517496' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:34.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:34.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:13:34.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:34.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14174 v1:172.21.15.22:0/2409521881' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Deploying cephadm binary to smithi122 2021-12-15T17:13:34.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:34.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:34.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Added host smithi122 2021-12-15T17:13:34.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14176 v1:172.21.15.22:0/566346912' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:13:34.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3738424088' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-12-15T17:13:34.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3738424088' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-12-15T17:13:34.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:34.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:13:34.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.14180 v1:172.21.15.122:0/1643773664' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi022:[v1:172.21.15.22:6789]=a;smithi022:[v1:172.21.15.22:6790]=c;smithi122:[v1:172.21.15.122:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:34.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Saving service mon spec with placement smithi022:[v1:172.21.15.22:6789]=a;smithi022:[v1:172.21.15.22:6790]=c;smithi122:[v1:172.21.15.122:6789]=b;count:3 2021-12-15T17:13:34.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:34.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:34.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:13:34.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:34.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:34.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:34.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Deploying daemon mon.c on smithi022 2021-12-15T17:13:34.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:34.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/1594523919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:34.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:34.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.a calling monitor election 2021-12-15T17:13:34.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:34.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:34.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:34.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:34.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.c calling monitor election 2021-12-15T17:13:34.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:34.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:34.053 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:34.053 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:34.053 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-12-15T17:13:34.053 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: monmap e2: 2 mons at {a=v1:172.21.15.22:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:34.053 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: fsmap 2021-12-15T17:13:34.054 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:34.054 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mgrmap e13: y(active, since 27s) 2021-12-15T17:13:34.054 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T17:13:34.054 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:34.055 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:34.055 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:34.055 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: Deploying daemon mon.b on smithi122 2021-12-15T17:13:34.055 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3530198743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:34.055 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:34.056 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:34.056 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2021-12-15T17:13:34.056 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 ceph-mon[32255]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2021-12-15T17:13:34.056 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:33 smithi122 systemd[1]: Started Ceph mon.b for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:13:38.551 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-12-15T17:13:38.551 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":3,"fsid":"0ea1c4a2-5dca-11ec-8c31-001a4aab830c","modified":"2021-12-15T17:13:33.529316Z","created":"2021-12-15T17:12:11.469714Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6790","nonce":0}]},"addr":"172.21.15.22:6790/0","public_addr":"172.21.15.22:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-12-15T17:13:38.552 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 3 2021-12-15T17:13:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: mon.a calling monitor election 2021-12-15T17:13:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: mon.c calling monitor election 2021-12-15T17:13:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/2715739514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:39.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-12-15T17:13:39.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: monmap e3: 3 mons at {a=v1:172.21.15.22:6789/0,b=v1:172.21.15.122:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:39.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: fsmap 2021-12-15T17:13:39.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: mgrmap e13: y(active, since 36s) 2021-12-15T17:13:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-12-15T17:13:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-12-15T17:13:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-12-15T17:13:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: mon.b (rank 2) addr v1:172.21.15.122:6789/0 is down (out of quorum) 2021-12-15T17:13:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: mon.a calling monitor election 2021-12-15T17:13:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: mon.c calling monitor election 2021-12-15T17:13:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/2715739514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:39.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-12-15T17:13:39.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: monmap e3: 3 mons at {a=v1:172.21.15.22:6789/0,b=v1:172.21.15.122:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:39.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: fsmap 2021-12-15T17:13:39.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:39.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: mgrmap e13: y(active, since 36s) 2021-12-15T17:13:39.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-12-15T17:13:39.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-12-15T17:13:39.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-12-15T17:13:39.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: mon.b (rank 2) addr v1:172.21.15.122:6789/0 is down (out of quorum) 2021-12-15T17:13:39.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:39.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:39.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:39.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:39.188 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-12-15T17:13:39.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph config generate-minimal-conf 2021-12-15T17:13:39.868 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:39 smithi022 ceph-mon[31360]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:39.869 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:39 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:39.870 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:39 smithi022 ceph-mon[40250]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:39.870 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:39 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:13:40.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: mon.a calling monitor election 2021-12-15T17:13:40.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: mon.c calling monitor election 2021-12-15T17:13:40.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:40.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:13:40.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/2715739514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-12-15T17:13:40.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:40.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:40.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:40.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:40.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:40.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:40.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:40.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-12-15T17:13:40.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: monmap e3: 3 mons at {a=v1:172.21.15.22:6789/0,b=v1:172.21.15.122:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:40.957 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: fsmap 2021-12-15T17:13:40.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:40.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: mgrmap e13: y(active, since 36s) 2021-12-15T17:13:40.958 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-12-15T17:13:40.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-12-15T17:13:40.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-12-15T17:13:40.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: mon.b (rank 2) addr v1:172.21.15.122:6789/0 is down (out of quorum) 2021-12-15T17:13:40.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:40.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:40.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:40.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:40.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:40.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:41.044 INFO:teuthology.orchestra.run.smithi022.stdout:# minimal ceph.conf for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:41.044 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2021-12-15T17:13:41.045 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 0ea1c4a2-5dca-11ec-8c31-001a4aab830c 2021-12-15T17:13:41.045 INFO:teuthology.orchestra.run.smithi022.stdout: mon_host = 172.21.15.22:6789/0 172.21.15.122:6789/0 v1:172.21.15.22:6790/0 2021-12-15T17:13:41.482 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: mon.b calling monitor election 2021-12-15T17:13:41.482 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:13:41.482 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:41.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:13:41.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: Reconfiguring mon.a (unknown last config time)... 2021-12-15T17:13:41.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: Reconfiguring daemon mon.a on smithi022 2021-12-15T17:13:41.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: mon.b calling monitor election 2021-12-15T17:13:41.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: mon.c calling monitor election 2021-12-15T17:13:41.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: mon.a calling monitor election 2021-12-15T17:13:41.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-12-15T17:13:41.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: monmap e3: 3 mons at {a=v1:172.21.15.22:6789/0,b=v1:172.21.15.122:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:41.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: fsmap 2021-12-15T17:13:41.485 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:41.485 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: mgrmap e13: y(active, since 38s) 2021-12-15T17:13:41.485 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-12-15T17:13:41.485 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:13:41.485 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T17:13:41.486 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1863372798' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:41.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: mon.b calling monitor election 2021-12-15T17:13:41.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:13:41.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:41.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:13:41.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: Reconfiguring mon.a (unknown last config time)... 2021-12-15T17:13:41.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: Reconfiguring daemon mon.a on smithi022 2021-12-15T17:13:41.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: mon.b calling monitor election 2021-12-15T17:13:41.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: mon.c calling monitor election 2021-12-15T17:13:41.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: mon.a calling monitor election 2021-12-15T17:13:41.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-12-15T17:13:41.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: monmap e3: 3 mons at {a=v1:172.21.15.22:6789/0,b=v1:172.21.15.122:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:41.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: fsmap 2021-12-15T17:13:41.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:41.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: mgrmap e13: y(active, since 38s) 2021-12-15T17:13:41.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-12-15T17:13:41.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:13:41.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T17:13:41.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1863372798' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:41.491 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-12-15T17:13:41.491 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:13:41.492 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.conf 2021-12-15T17:13:41.523 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:13:41.524 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:41.592 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:13:41.593 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2021-12-15T17:13:41.626 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:13:41.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:13:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: mon.b calling monitor election 2021-12-15T17:13:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:13:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:13:41.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: Reconfiguring mon.a (unknown last config time)... 2021-12-15T17:13:41.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: Reconfiguring daemon mon.a on smithi022 2021-12-15T17:13:41.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: mon.b calling monitor election 2021-12-15T17:13:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: mon.c calling monitor election 2021-12-15T17:13:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: mon.a calling monitor election 2021-12-15T17:13:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-12-15T17:13:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: monmap e3: 3 mons at {a=v1:172.21.15.22:6789/0,b=v1:172.21.15.122:6789/0,c=v1:172.21.15.22:6790/0} 2021-12-15T17:13:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: fsmap 2021-12-15T17:13:41.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: osdmap e4: 0 total, 0 up, 0 in 2021-12-15T17:13:41.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: mgrmap e13: y(active, since 38s) 2021-12-15T17:13:41.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-12-15T17:13:41.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:13:41.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T17:13:41.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1863372798' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:41.698 INFO:tasks.cephadm:Adding mgr.y on smithi022 2021-12-15T17:13:41.698 INFO:tasks.cephadm:Adding mgr.x on smithi122 2021-12-15T17:13:41.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply mgr '2;smithi022=y;smithi122=x' 2021-12-15T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:42.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:42.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-12-15T17:13:42.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:42.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:42.514 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:42.514 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:42.515 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:42.515 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-12-15T17:13:42.515 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:42.515 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:42.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:13:42.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:42.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:42.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-12-15T17:13:42.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:42.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mgr update... 2021-12-15T17:13:43.608 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:43 smithi122 ceph-mon[32255]: Reconfiguring mon.c (monmap changed)... 2021-12-15T17:13:43.609 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:43 smithi122 ceph-mon[32255]: Reconfiguring daemon mon.c on smithi022 2021-12-15T17:13:43.609 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:43 smithi122 ceph-mon[32255]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:43.609 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:43.609 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:43.609 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-12-15T17:13:43.610 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[40250]: Reconfiguring mon.c (monmap changed)... 2021-12-15T17:13:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[40250]: Reconfiguring daemon mon.c on smithi022 2021-12-15T17:13:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[40250]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-12-15T17:13:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[31360]: Reconfiguring mon.c (monmap changed)... 2021-12-15T17:13:43.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[31360]: Reconfiguring daemon mon.c on smithi022 2021-12-15T17:13:43.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[31360]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:43.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:43.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-12-15T17:13:43.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-12-15T17:13:43.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:44.378 DEBUG:teuthology.orchestra.run.smithi122:mgr.x> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@mgr.x.service 2021-12-15T17:13:44.380 INFO:tasks.cephadm:Deploying OSDs... 2021-12-15T17:13:44.381 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:13:44.381 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2021-12-15T17:13:44.402 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-12-15T17:13:44.403 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-12-15T17:13:44.405 INFO:journalctl@ceph.mgr.x.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:13:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-12-15T17:13:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.465 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 154052 Links: 1 2021-12-15T17:13:44.465 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.465 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.465 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2021-12-15 17:13:24.063416808 +0000 2021-12-15T17:13:44.465 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2021-12-15 17:08:28.733177911 +0000 2021-12-15T17:13:44.466 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2021-12-15 17:08:28.733177911 +0000 2021-12-15T17:13:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2021-12-15T17:13:44.478 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2021-12-15T17:13:44.478 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2021-12-15T17:13:44.479 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000125304 s, 4.1 MB/s 2021-12-15T17:13:44.482 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-12-15T17:13:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-12-15T17:13:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.546 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 151319 Links: 1 2021-12-15T17:13:44.546 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.547 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.547 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2021-12-15 17:13:24.063416808 +0000 2021-12-15T17:13:44.548 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2021-12-15 17:08:29.075173516 +0000 2021-12-15T17:13:44.548 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2021-12-15 17:08:29.075173516 +0000 2021-12-15T17:13:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2021-12-15T17:13:44.555 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2021-12-15T17:13:44.555 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2021-12-15T17:13:44.555 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000102462 s, 5.0 MB/s 2021-12-15T17:13:44.556 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-12-15T17:13:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-12-15T17:13:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.615 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 155143 Links: 1 2021-12-15T17:13:44.616 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.616 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.616 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2021-12-15 17:13:24.063416808 +0000 2021-12-15T17:13:44.616 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2021-12-15 17:08:29.405169276 +0000 2021-12-15T17:13:44.617 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2021-12-15 17:08:29.405169276 +0000 2021-12-15T17:13:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2021-12-15T17:13:44.626 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2021-12-15T17:13:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2021-12-15T17:13:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000113652 s, 4.5 MB/s 2021-12-15T17:13:44.629 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-12-15T17:13:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: Reconfiguring mon.b (monmap changed)... 2021-12-15T17:13:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: Reconfiguring daemon mon.b on smithi122 2021-12-15T17:13:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='client.24112 v1:172.21.15.122:0/3365301255' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi022=y;smithi122=x", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: Saving service mgr spec with placement smithi022=y;smithi122=x;count:2 2021-12-15T17:13:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:44.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:44.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:44.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:13:44.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-12-15T17:13:44.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:13:44.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:44 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-12-15T17:13:44.692 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.692 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 158037 Links: 1 2021-12-15T17:13:44.694 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.694 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.694 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2021-12-15 17:13:24.063416808 +0000 2021-12-15T17:13:44.694 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2021-12-15 17:08:29.736165023 +0000 2021-12-15T17:13:44.695 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2021-12-15 17:08:29.736165023 +0000 2021-12-15T17:13:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2021-12-15T17:13:44.698 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2021-12-15T17:13:44.699 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2021-12-15T17:13:44.699 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.8394e-05 s, 5.2 MB/s 2021-12-15T17:13:44.702 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:13:44.702 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2021-12-15T17:13:44.720 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-12-15T17:13:44.721 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-12-15T17:13:44.776 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-12-15T17:13:44.776 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.776 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 152425 Links: 1 2021-12-15T17:13:44.777 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.777 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.777 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-12-15 17:13:32.549136494 +0000 2021-12-15T17:13:44.777 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-12-15 17:08:22.929845456 +0000 2021-12-15T17:13:44.778 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-12-15 17:08:22.929845456 +0000 2021-12-15T17:13:44.778 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-12-15T17:13:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: Reconfiguring mon.b (monmap changed)... 2021-12-15T17:13:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: Reconfiguring daemon mon.b on smithi122 2021-12-15T17:13:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='client.24112 v1:172.21.15.122:0/3365301255' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi022=y;smithi122=x", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: Saving service mgr spec with placement smithi022=y;smithi122=x;count:2 2021-12-15T17:13:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:13:44.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-12-15T17:13:44.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:13:44.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:44.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: Reconfiguring mon.b (monmap changed)... 2021-12-15T17:13:44.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: Reconfiguring daemon mon.b on smithi122 2021-12-15T17:13:44.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='client.24112 v1:172.21.15.122:0/3365301255' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi022=y;smithi122=x", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:44.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: Saving service mgr spec with placement smithi022=y;smithi122=x;count:2 2021-12-15T17:13:44.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:44.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:44.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:44.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:44.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:13:44.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-12-15T17:13:44.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:13:44.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:44 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:44.786 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-12-15T17:13:44.786 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-12-15T17:13:44.787 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000212338 s, 2.4 MB/s 2021-12-15T17:13:44.791 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-12-15T17:13:44.847 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-12-15T17:13:44.847 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.847 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 153895 Links: 1 2021-12-15T17:13:44.848 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.848 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.848 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-12-15 17:13:32.549136494 +0000 2021-12-15T17:13:44.848 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-12-15 17:08:23.299836778 +0000 2021-12-15T17:13:44.848 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-12-15 17:08:23.299836778 +0000 2021-12-15T17:13:44.849 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-12-15T17:13:44.856 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-12-15T17:13:44.857 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-12-15T17:13:44.857 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000209069 s, 2.4 MB/s 2021-12-15T17:13:44.859 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-12-15T17:13:44.916 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-12-15T17:13:44.917 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.917 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 149389 Links: 1 2021-12-15T17:13:44.917 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.917 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.918 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-12-15 17:13:32.549136494 +0000 2021-12-15T17:13:44.918 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-12-15 17:08:23.642828734 +0000 2021-12-15T17:13:44.918 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-12-15 17:08:23.642828734 +0000 2021-12-15T17:13:44.919 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-12-15T17:13:44.926 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-12-15T17:13:44.927 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-12-15T17:13:44.927 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000204072 s, 2.5 MB/s 2021-12-15T17:13:44.929 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-12-15T17:13:44.986 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-12-15T17:13:44.987 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-12-15T17:13:44.987 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 156694 Links: 1 2021-12-15T17:13:44.987 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-12-15T17:13:44.988 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2021-12-15T17:13:44.988 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-12-15 17:13:32.550136470 +0000 2021-12-15T17:13:44.988 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-12-15 17:08:23.973820971 +0000 2021-12-15T17:13:44.988 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-12-15 17:08:23.973820971 +0000 2021-12-15T17:13:44.988 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-12-15T17:13:44.999 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-12-15T17:13:44.999 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-12-15T17:13:45.000 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000167247 s, 3.1 MB/s 2021-12-15T17:13:45.003 INFO:tasks.cephadm:Deploying osd.0 on smithi022 with /dev/vg_nvme/lv_4... 2021-12-15T17:13:45.003 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-12-15T17:13:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:45 smithi122 ceph-mon[32255]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:45 smithi122 ceph-mon[32255]: Deploying daemon mgr.x on smithi122 2021-12-15T17:13:45.936 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:45 smithi122 systemd[1]: Starting Ceph mgr.x for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:13:46.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:45 smithi022 ceph-mon[31360]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:46.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:45 smithi022 ceph-mon[31360]: Deploying daemon mgr.x on smithi122 2021-12-15T17:13:46.025 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:45 smithi022 ceph-mon[40250]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:46.025 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:45 smithi022 ceph-mon[40250]: Deploying daemon mgr.x on smithi122 2021-12-15T17:13:46.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:46 smithi122 podman[33887]: 2021-12-15T17:13:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-12-15T17:13:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-12-15T17:13:46.736 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2021-12-15T17:13:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2021-12-15T17:13:46.737 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00872551 s, 1.2 GB/s 2021-12-15T17:13:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2021-12-15T17:13:46.936 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:46 smithi122 bash[33736]: 4d0b8bee87365ef9ee602450c8ce9addbaca66d08c2e286601a43eba9a8aee35 2021-12-15T17:13:46.936 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:46 smithi122 systemd[1]: Started Ceph mgr.x for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:13:46.936 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:46 smithi122 conmon[33927]: 2021-12-15T17:13:46.572+0000 7f8115ea8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2021-12-15T17:13:47.131 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi022:vg_nvme/lv_4 2021-12-15T17:13:47.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:47 smithi122 conmon[33927]: 2021-12-15T17:13:47.148+0000 7f8115ea8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2021-12-15T17:13:47.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:47 smithi122 conmon[33927]: 2021-12-15T17:13:47.270+0000 7f8115ea8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2021-12-15T17:13:47.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:47 smithi122 conmon[33927]: 2021-12-15T17:13:47.351+0000 7f8115ea8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2021-12-15T17:13:47.814 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[31360]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:47.814 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:47.814 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:47.815 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:47.815 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:47.815 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:47.815 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[40250]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:47.816 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:47.816 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:47.816 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:47.816 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:47.816 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:47.915 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:47 smithi122 conmon[33927]: 2021-12-15T17:13:47.440+0000 7f8115ea8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2021-12-15T17:13:47.915 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:47 smithi122 ceph-mon[32255]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:47.916 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:47.916 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:47.916 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:47.916 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:47.917 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:48.184 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:48 smithi122 conmon[33927]: 2021-12-15T17:13:48.163+0000 7f8115ea8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:13:48.946 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:48 smithi122 conmon[33927]: 2021-12-15T17:13:48.696+0000 7f8115ea8180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2021-12-15T17:13:48.946 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:48 smithi122 conmon[33927]: 2021-12-15T17:13:48.781+0000 7f8115ea8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2021-12-15T17:13:48.947 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:48.947 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:48.947 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: Reconfiguring mgr.y (unknown last config time)... 2021-12-15T17:13:48.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:13:48.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:13:48.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:48.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: Reconfiguring daemon mgr.y on smithi022 2021-12-15T17:13:48.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:48 smithi122 ceph-mon[32255]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: Reconfiguring mgr.y (unknown last config time)... 2021-12-15T17:13:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:13:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:13:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: Reconfiguring daemon mgr.y on smithi022 2021-12-15T17:13:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[31360]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: Reconfiguring mgr.y (unknown last config time)... 2021-12-15T17:13:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:13:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:13:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: Reconfiguring daemon mgr.y on smithi022 2021-12-15T17:13:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:48 smithi022 ceph-mon[40250]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:49.284 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:48 smithi122 conmon[33927]: 2021-12-15T17:13:48.946+0000 7f8115ea8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2021-12-15T17:13:49.284 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:49 smithi122 conmon[33927]: 2021-12-15T17:13:49.031+0000 7f8115ea8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2021-12-15T17:13:49.284 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:49 smithi122 conmon[33927]: 2021-12-15T17:13:49.118+0000 7f8115ea8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2021-12-15T17:13:49.547 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:49 smithi122 conmon[33927]: 2021-12-15T17:13:49.284+0000 7f8115ea8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2021-12-15T17:13:49.835 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:49 smithi122 conmon[33927]: 2021-12-15T17:13:49.548+0000 7f8115ea8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2021-12-15T17:13:49.835 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:49 smithi122 conmon[33927]: 2021-12-15T17:13:49.672+0000 7f8115ea8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2021-12-15T17:13:49.835 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:49 smithi122 conmon[33927]: 2021-12-15T17:13:49.753+0000 7f8115ea8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2021-12-15T17:13:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='client.14214 v1:172.21.15.22:0/2852701242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:13:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:13:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='client.14214 v1:172.21.15.22:0/2852701242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:13:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:13:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:50.184 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:49 smithi122 conmon[33927]: 2021-12-15T17:13:49.835+0000 7f8115ea8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2021-12-15T17:13:50.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='client.14214 v1:172.21.15.22:0/2852701242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:13:50.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:13:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:13:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:13:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:50.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:13:50.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:13:50.913 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:50 smithi122 conmon[33927]: 2021-12-15T17:13:50.579+0000 7f8115ea8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2021-12-15T17:13:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:50 smithi122 ceph-mon[32255]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:51.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:50 smithi122 conmon[33927]: 2021-12-15T17:13:50.913+0000 7f8115ea8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:13:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:50 smithi022 ceph-mon[31360]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:50 smithi022 ceph-mon[40250]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:51.592 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:51 smithi122 conmon[33927]: 2021-12-15T17:13:51.187+0000 7f8115ea8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2021-12-15T17:13:51.593 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:51 smithi122 conmon[33927]: 2021-12-15T17:13:51.327+0000 7f8115ea8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2021-12-15T17:13:51.932 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:51 smithi122 conmon[33927]: 2021-12-15T17:13:51.593+0000 7f8115ea8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2021-12-15T17:13:51.932 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:51 smithi122 conmon[33927]: 2021-12-15T17:13:51.677+0000 7f8115ea8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2021-12-15T17:13:52.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:52 smithi122 conmon[33927]: 2021-12-15T17:13:52.006+0000 7f8115ea8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2021-12-15T17:13:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3734938584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]: dispatch 2021-12-15T17:13:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:51 smithi122 ceph-mon[32255]: from='client.24130 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]: dispatch 2021-12-15T17:13:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:51 smithi122 ceph-mon[32255]: from='client.24130 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]': finished 2021-12-15T17:13:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:51 smithi122 ceph-mon[32255]: osdmap e5: 1 total, 0 up, 1 in 2021-12-15T17:13:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:13:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3543376676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:13:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3734938584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]: dispatch 2021-12-15T17:13:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[31360]: from='client.24130 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]: dispatch 2021-12-15T17:13:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[31360]: from='client.24130 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]': finished 2021-12-15T17:13:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[31360]: osdmap e5: 1 total, 0 up, 1 in 2021-12-15T17:13:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:13:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3543376676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:13:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3734938584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]: dispatch 2021-12-15T17:13:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[40250]: from='client.24130 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]: dispatch 2021-12-15T17:13:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[40250]: from='client.24130 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1d97a7c-370f-4a35-9216-1a6dc0e6cc09"}]': finished 2021-12-15T17:13:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[40250]: osdmap e5: 1 total, 0 up, 1 in 2021-12-15T17:13:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:13:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3543376676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:13:52.684 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:13:52 smithi122 conmon[33927]: 2021-12-15T17:13:52.299+0000 7f8115ea8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2021-12-15T17:13:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[31360]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[31360]: Standby manager daemon x started 2021-12-15T17:13:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-12-15T17:13:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:13:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-12-15T17:13:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:13:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[40250]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[40250]: Standby manager daemon x started 2021-12-15T17:13:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-12-15T17:13:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:13:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-12-15T17:13:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:52 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:13:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:52 smithi122 ceph-mon[32255]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:52 smithi122 ceph-mon[32255]: Standby manager daemon x started 2021-12-15T17:13:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:52 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-12-15T17:13:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:52 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:13:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:52 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-12-15T17:13:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:52 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/631693672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:13:54.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:53 smithi022 ceph-mon[31360]: mgrmap e14: y(active, since 51s), standbys: x 2021-12-15T17:13:54.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:53 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:13:54.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:53 smithi022 ceph-mon[40250]: mgrmap e14: y(active, since 51s), standbys: x 2021-12-15T17:13:54.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:53 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:13:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:53 smithi122 ceph-mon[32255]: mgrmap e14: y(active, since 51s), standbys: x 2021-12-15T17:13:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:53 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:13:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:55 smithi022 ceph-mon[31360]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:55 smithi022 ceph-mon[40250]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:55 smithi122 ceph-mon[32255]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:56.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:56 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:13:56.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:56 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:56 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:13:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:56 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:56 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:13:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:56 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:13:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:57 smithi022 ceph-mon[31360]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:57 smithi022 ceph-mon[31360]: Deploying daemon osd.0 on smithi022 2021-12-15T17:13:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:57 smithi022 ceph-mon[40250]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:57 smithi022 ceph-mon[40250]: Deploying daemon osd.0 on smithi022 2021-12-15T17:13:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:57 smithi122 ceph-mon[32255]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:13:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:57 smithi122 ceph-mon[32255]: Deploying daemon osd.0 on smithi022 2021-12-15T17:14:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:13:59 smithi022 ceph-mon[31360]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:13:59 smithi022 ceph-mon[40250]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:00.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:13:59 smithi122 ceph-mon[32255]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:00.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:00.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:00.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:00.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:00.998 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:00.998 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:00.999 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:00.999 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:01 smithi022 ceph-mon[31360]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:01 smithi022 ceph-mon[40250]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:01 smithi122 ceph-mon[32255]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:02.636 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 0 on host 'smithi022' 2021-12-15T17:14:02.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:14:02.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:14:02.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:02.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:02.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:02.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:02.948 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:02.948 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:02.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:14:02.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:14:02.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:02.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:02.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:02.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:02.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:02.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:02.992 DEBUG:teuthology.orchestra.run.smithi022:osd.0> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.0.service 2021-12-15T17:14:02.995 INFO:tasks.cephadm:Deploying osd.1 on smithi022 with /dev/vg_nvme/lv_3... 2021-12-15T17:14:02.996 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-12-15T17:14:03.027 INFO:journalctl@ceph.osd.0.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:14:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:14:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:14:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:03.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:03 smithi022 ceph-mon[31360]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:03.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:03 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:03.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:03 smithi022 ceph-mon[40250]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:03.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:03 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:03.905 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:03 smithi022 conmon[44880]: 2021-12-15T17:14:03.533+0000 7fc4ae6c03c0 -1 osd.0 0 log_to_monitors true 2021-12-15T17:14:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:03 smithi122 ceph-mon[32255]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:03 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[31360]: osdmap e6: 1 total, 0 up, 1 in 2021-12-15T17:14:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[40250]: osdmap e6: 1 total, 0 up, 1 in 2021-12-15T17:14:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:04 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:04 smithi022 conmon[44880]: 2021-12-15T17:14:04.755+0000 7fc4a50c7700 -1 osd.0 0 waiting for initial osdmap 2021-12-15T17:14:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:04 smithi022 conmon[44880]: 2021-12-15T17:14:04.757+0000 7fc49da59700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:14:05.144 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-12-15T17:14:05.145 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-12-15T17:14:05.145 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2021-12-15T17:14:05.146 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2021-12-15T17:14:05.146 INFO:teuthology.orchestra.run.smithi022.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00905401 s, 1.2 GB/s 2021-12-15T17:14:05.146 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2021-12-15T17:14:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:04 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:04 smithi122 ceph-mon[32255]: osdmap e6: 1 total, 0 up, 1 in 2021-12-15T17:14:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:04 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:04 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:05.640 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi022:vg_nvme/lv_3 2021-12-15T17:14:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[31360]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[31360]: osdmap e7: 1 total, 0 up, 1 in 2021-12-15T17:14:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[40250]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[40250]: osdmap e7: 1 total, 0 up, 1 in 2021-12-15T17:14:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:05 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:05 smithi122 ceph-mon[32255]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:05 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/3851520186' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:05 smithi122 ceph-mon[32255]: osdmap e7: 1 total, 0 up, 1 in 2021-12-15T17:14:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:05 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:05 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:14:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:14:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: osd.0 v1:172.21.15.22:6801/3851520186 boot 2021-12-15T17:14:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: osdmap e8: 1 total, 1 up, 1 in 2021-12-15T17:14:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:14:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:14:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: osd.0 v1:172.21.15.22:6801/3851520186 boot 2021-12-15T17:14:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: osdmap e8: 1 total, 1 up, 1 in 2021-12-15T17:14:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:06 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:14:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:14:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: osd.0 v1:172.21.15.22:6801/3851520186 boot 2021-12-15T17:14:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: osdmap e8: 1 total, 1 up, 1 in 2021-12-15T17:14:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:06 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:08.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi022 2021-12-15T17:14:08.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 16152M 2021-12-15T17:14:08.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[31360]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:08.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[31360]: osdmap e9: 1 total, 1 up, 1 in 2021-12-15T17:14:08.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:14:08.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:14:08.021 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:08.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi022 2021-12-15T17:14:08.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 16152M 2021-12-15T17:14:08.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[40250]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:08.022 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[40250]: osdmap e9: 1 total, 1 up, 1 in 2021-12-15T17:14:08.022 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:14:08.022 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:14:08.022 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:07 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi022 2021-12-15T17:14:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:07 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 16152M 2021-12-15T17:14:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:07 smithi122 ceph-mon[32255]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-12-15T17:14:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:07 smithi122 ceph-mon[32255]: osdmap e9: 1 total, 1 up, 1 in 2021-12-15T17:14:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:14:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:14:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:08 smithi022 ceph-mon[31360]: from='client.24151 v1:172.21.15.22:0/1049694343' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:14:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:08 smithi022 ceph-mon[40250]: from='client.24151 v1:172.21.15.22:0/1049694343' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:14:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:08 smithi122 ceph-mon[32255]: from='client.24151 v1:172.21.15.22:0/1049694343' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:14:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[31360]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1610641067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]: dispatch 2021-12-15T17:14:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[31360]: from='client.24157 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]: dispatch 2021-12-15T17:14:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[31360]: from='client.24157 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]': finished 2021-12-15T17:14:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[31360]: osdmap e10: 2 total, 1 up, 2 in 2021-12-15T17:14:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[40250]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1610641067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]: dispatch 2021-12-15T17:14:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[40250]: from='client.24157 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]: dispatch 2021-12-15T17:14:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[40250]: from='client.24157 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]': finished 2021-12-15T17:14:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[40250]: osdmap e10: 2 total, 1 up, 2 in 2021-12-15T17:14:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:09 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:09 smithi122 ceph-mon[32255]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1610641067' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]: dispatch 2021-12-15T17:14:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:09 smithi122 ceph-mon[32255]: from='client.24157 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]: dispatch 2021-12-15T17:14:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:09 smithi122 ceph-mon[32255]: from='client.24157 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da26a6f5-3ae4-46a9-b878-10f47104ea9d"}]': finished 2021-12-15T17:14:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:09 smithi122 ceph-mon[32255]: osdmap e10: 2 total, 1 up, 2 in 2021-12-15T17:14:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:09 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2204936771' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:14:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2204936771' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:14:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2204936771' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:14:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:11 smithi022 ceph-mon[31360]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:11 smithi022 ceph-mon[40250]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:11 smithi122 ceph-mon[32255]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:13.773 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:13 smithi022 ceph-mon[31360]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:13 smithi022 ceph-mon[40250]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:14.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:13 smithi122 ceph-mon[32255]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:15.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[31360]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:15.987 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:14:15.987 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:15.987 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[31360]: Deploying daemon osd.1 on smithi022 2021-12-15T17:14:15.988 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[40250]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:15.988 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:14:15.988 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:15.988 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:15 smithi022 ceph-mon[40250]: Deploying daemon osd.1 on smithi022 2021-12-15T17:14:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:15 smithi122 ceph-mon[32255]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:15 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:14:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:15 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:15 smithi122 ceph-mon[32255]: Deploying daemon osd.1 on smithi022 2021-12-15T17:14:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:17 smithi022 ceph-mon[31360]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:17 smithi022 ceph-mon[40250]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:17 smithi122 ceph-mon[32255]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:18.931 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:18.932 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:18.932 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:18.932 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:18.933 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:18.933 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:18.933 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:18.934 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:19 smithi022 ceph-mon[31360]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:19 smithi022 ceph-mon[40250]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:20.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:19 smithi122 ceph-mon[32255]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[31360]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:14:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[40250]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:14:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:21 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:21 smithi122 ceph-mon[32255]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:21 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:21 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:21 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:14:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:21 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:22 smithi022 ceph-mon[31360]: Deploying daemon osd.0 on smithi022 2021-12-15T17:14:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:22 smithi022 ceph-mon[31360]: from='osd.1 v1:172.21.15.22:6805/1716084643' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:22 smithi022 ceph-mon[31360]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:22 smithi022 ceph-mon[40250]: Deploying daemon osd.0 on smithi022 2021-12-15T17:14:22.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:22 smithi022 ceph-mon[40250]: from='osd.1 v1:172.21.15.22:6805/1716084643' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:22.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:22 smithi022 ceph-mon[40250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:22 smithi122 ceph-mon[32255]: Deploying daemon osd.0 on smithi022 2021-12-15T17:14:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:22 smithi122 ceph-mon[32255]: from='osd.1 v1:172.21.15.22:6805/1716084643' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:22 smithi122 ceph-mon[32255]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:23.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 systemd[1]: Stopping Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:14:23.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 bash[51482]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.0" found: no such container 2021-12-15T17:14:23.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 conmon[44880]: 2021-12-15T17:14:23.311+0000 7fc4a60c9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:14:23.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 conmon[44880]: 2021-12-15T17:14:23.311+0000 7fc4a60c9700 -1 osd.0 11 *** Got signal Terminated *** 2021-12-15T17:14:23.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 conmon[44880]: 2021-12-15T17:14:23.311+0000 7fc4a60c9700 -1 osd.0 11 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:14:23.894 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 bash[51482]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-0 2021-12-15T17:14:23.894 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 bash[51482]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.0" found: no such container 2021-12-15T17:14:23.894 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[40250]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:23.895 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[40250]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:14:23.895 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[40250]: osdmap e11: 2 total, 1 up, 2 in 2021-12-15T17:14:23.895 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:23.895 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[40250]: from='osd.1 v1:172.21.15.22:6805/1716084643' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:23.896 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[40250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:23.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[31360]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:23.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[31360]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:14:23.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[31360]: osdmap e11: 2 total, 1 up, 2 in 2021-12-15T17:14:23.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:23.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[31360]: from='osd.1 v1:172.21.15.22:6805/1716084643' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:23.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:23 smithi022 ceph-mon[31360]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:23 smithi122 ceph-mon[32255]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:23 smithi122 ceph-mon[32255]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:14:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:23 smithi122 ceph-mon[32255]: osdmap e11: 2 total, 1 up, 2 in 2021-12-15T17:14:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:23 smithi122 ceph-mon[32255]: from='osd.1 v1:172.21.15.22:6805/1716084643' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:23 smithi122 ceph-mon[32255]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:24.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:23 smithi022 podman[51800]: 2021-12-15T17:14:24.601 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[40250]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:24.601 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[40250]: osdmap e12: 2 total, 1 up, 2 in 2021-12-15T17:14:24.601 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:24.602 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:24.602 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[31360]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:24.603 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[31360]: osdmap e12: 2 total, 1 up, 2 in 2021-12-15T17:14:24.603 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:24.603 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:24 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:24 smithi122 ceph-mon[32255]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:24 smithi122 ceph-mon[32255]: osdmap e12: 2 total, 1 up, 2 in 2021-12-15T17:14:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:24 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:24 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:25.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:24 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.0.service: Succeeded. 2021-12-15T17:14:25.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:24 smithi022 systemd[1]: Stopped Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:14:25.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:24 smithi022 systemd[1]: Starting Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:14:25.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 podman[52100]: 2021-12-15T17:14:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:14:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:14:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.1 v1:172.21.15.22:6805/1716084643 boot 2021-12-15T17:14:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osdmap e13: 2 total, 2 up, 2 in 2021-12-15T17:14:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:14:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:14:25.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:14:25.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:25.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:25.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.1 v1:172.21.15.22:6805/1716084643 boot 2021-12-15T17:14:25.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osdmap e13: 2 total, 2 up, 2 in 2021-12-15T17:14:25.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:25.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:14:25.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:25 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.786 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:14:25.786 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:25.787 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:14:25.787 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:25.787 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-12-15T17:14:25.787 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:25.787 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:25.788 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:14:25.788 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:25.788 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 bash[51946]: --> ceph-volume lvm activate successful for osd ID: 0 2021-12-15T17:14:25.788 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-12-15T17:14:25.788 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:25.789 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:25.789 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:14:25.789 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:25.789 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:25 smithi022 conmon[52140]: --> ceph-volume lvm activate successful for osd ID: 0 2021-12-15T17:14:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:14:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:14:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2021-12-15T17:14:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.1 v1:172.21.15.22:6805/1716084643 boot 2021-12-15T17:14:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osdmap e13: 2 total, 2 up, 2 in 2021-12-15T17:14:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:14:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:14:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:25 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:26.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:26 smithi022 podman[52373]: 2021-12-15T17:14:26.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:26 smithi022 bash[51946]: 1bb4dad0ba318e40d3b6c619e0df4889a2fd18bae8b543caa5648a021f25caa7 2021-12-15T17:14:26.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:26 smithi022 systemd[1]: Started Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:14:26.367 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 1,0 on host 'smithi022' 2021-12-15T17:14:26.679 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[31360]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:14:26.679 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[31360]: osdmap e14: 2 total, 1 up, 2 in 2021-12-15T17:14:26.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:26.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:26.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:26.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:26.681 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:26.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[40250]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:14:26.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[40250]: osdmap e14: 2 total, 1 up, 2 in 2021-12-15T17:14:26.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:26.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:26.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:26.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:26.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:26.716 DEBUG:teuthology.orchestra.run.smithi022:osd.1> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.1.service 2021-12-15T17:14:26.719 INFO:tasks.cephadm:Deploying osd.2 on smithi022 with /dev/vg_nvme/lv_2... 2021-12-15T17:14:26.719 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-12-15T17:14:26.746 INFO:journalctl@ceph.osd.1.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:14:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:26 smithi122 ceph-mon[32255]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:14:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:26 smithi122 ceph-mon[32255]: osdmap e14: 2 total, 1 up, 2 in 2021-12-15T17:14:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:27.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:26 smithi022 conmon[52413]: 2021-12-15T17:14:26.774+0000 7f23b942d3c0 -1 Falling back to public interface 2021-12-15T17:14:27.848 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:27 smithi022 ceph-mon[31360]: pgmap v46: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:27.848 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:27 smithi022 ceph-mon[40250]: pgmap v46: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:27 smithi122 ceph-mon[32255]: pgmap v46: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:28.602 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-12-15T17:14:28.603 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-12-15T17:14:28.603 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2021-12-15T17:14:28.603 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2021-12-15T17:14:28.603 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010388 s, 1.0 GB/s 2021-12-15T17:14:28.604 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2021-12-15T17:14:28.873 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi022:vg_nvme/lv_2 2021-12-15T17:14:29.449 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:29 smithi022 conmon[52413]: 2021-12-15T17:14:29.244+0000 7f23b942d3c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:14:29.768 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:29 smithi022 conmon[52413]: 2021-12-15T17:14:29.529+0000 7f23b942d3c0 -1 osd.0 11 log_to_monitors true 2021-12-15T17:14:29.769 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:29 smithi022 ceph-mon[31360]: pgmap v47: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:29.769 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:29.769 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:29 smithi022 ceph-mon[40250]: pgmap v47: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:29.769 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:29 smithi122 ceph-mon[32255]: pgmap v47: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:29 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:30 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:30.779 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:30 smithi022 conmon[52413]: 2021-12-15T17:14:30.465+0000 7f23aafcb700 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:14:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:30 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:30.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:30 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: pgmap v48: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: osdmap e15: 2 total, 1 up, 2 in 2021-12-15T17:14:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:14:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:14:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:31.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: pgmap v48: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:31.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:31.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: osdmap e15: 2 total, 1 up, 2 in 2021-12-15T17:14:31.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:31.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:14:31.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:14:31.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:31.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:31.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:31.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:31.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:31.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:31.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: pgmap v48: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: osdmap e15: 2 total, 1 up, 2 in 2021-12-15T17:14:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/1676476291' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:14:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:14:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:14:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:32.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: from='client.14274 v1:172.21.15.22:0/4145814952' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:14:32.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi022 2021-12-15T17:14:32.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 8076M 2021-12-15T17:14:32.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:14:32.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:14:32.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: osd.0 v1:172.21.15.22:6801/1676476291 boot 2021-12-15T17:14:32.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: osdmap e16: 2 total, 2 up, 2 in 2021-12-15T17:14:32.470 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: from='client.14274 v1:172.21.15.22:0/4145814952' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi022 2021-12-15T17:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 8076M 2021-12-15T17:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:14:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:14:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: osd.0 v1:172.21.15.22:6801/1676476291 boot 2021-12-15T17:14:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: osdmap e16: 2 total, 2 up, 2 in 2021-12-15T17:14:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:32 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: from='client.14274 v1:172.21.15.22:0/4145814952' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:14:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi022 2021-12-15T17:14:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 8076M 2021-12-15T17:14:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:14:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:14:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: osd.0 v1:172.21.15.22:6801/1676476291 boot 2021-12-15T17:14:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: osdmap e16: 2 total, 2 up, 2 in 2021-12-15T17:14:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:32 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[31360]: pgmap v51: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2748652618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603393c6-9925-483a-a06a-383b3afc520b"}]: dispatch 2021-12-15T17:14:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2748652618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "603393c6-9925-483a-a06a-383b3afc520b"}]': finished 2021-12-15T17:14:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[31360]: osdmap e17: 3 total, 2 up, 3 in 2021-12-15T17:14:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[40250]: pgmap v51: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2748652618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603393c6-9925-483a-a06a-383b3afc520b"}]: dispatch 2021-12-15T17:14:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2748652618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "603393c6-9925-483a-a06a-383b3afc520b"}]': finished 2021-12-15T17:14:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[40250]: osdmap e17: 3 total, 2 up, 3 in 2021-12-15T17:14:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:33 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:33 smithi122 ceph-mon[32255]: pgmap v51: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2748652618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603393c6-9925-483a-a06a-383b3afc520b"}]: dispatch 2021-12-15T17:14:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2748652618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "603393c6-9925-483a-a06a-383b3afc520b"}]': finished 2021-12-15T17:14:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:33 smithi122 ceph-mon[32255]: osdmap e17: 3 total, 2 up, 3 in 2021-12-15T17:14:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:33 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/76732644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:14:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/76732644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:14:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/76732644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:14:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:35 smithi022 ceph-mon[31360]: pgmap v53: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:35 smithi022 ceph-mon[40250]: pgmap v53: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:35 smithi122 ceph-mon[32255]: pgmap v53: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:37.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:37 smithi022 ceph-mon[31360]: pgmap v54: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:37.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:37 smithi022 ceph-mon[40250]: pgmap v54: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:37 smithi122 ceph-mon[32255]: pgmap v54: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:39.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-12-15T17:14:39.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:39.020 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-12-15T17:14:39.020 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:38 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-12-15T17:14:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:38 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:39.915 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:39 smithi022 ceph-mon[31360]: pgmap v55: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:39.915 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:39 smithi022 ceph-mon[31360]: Deploying daemon osd.2 on smithi022 2021-12-15T17:14:39.915 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:39 smithi022 ceph-mon[40250]: pgmap v55: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:39.916 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:39 smithi022 ceph-mon[40250]: Deploying daemon osd.2 on smithi022 2021-12-15T17:14:40.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:39 smithi122 ceph-mon[32255]: pgmap v55: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:40.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:39 smithi122 ceph-mon[32255]: Deploying daemon osd.2 on smithi022 2021-12-15T17:14:42.026 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:41 smithi022 ceph-mon[31360]: pgmap v56: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:42.026 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:41 smithi022 ceph-mon[40250]: pgmap v56: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:41 smithi122 ceph-mon[32255]: pgmap v56: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:42.817 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:42.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:42.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:42.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:42.903 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:42.903 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:42.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:42.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:42 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:42 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:43 smithi022 ceph-mon[31360]: pgmap v57: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:43 smithi022 ceph-mon[40250]: pgmap v57: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:44.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:43 smithi122 ceph-mon[32255]: pgmap v57: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:45 smithi122 ceph-mon[32255]: pgmap v58: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:45 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:45 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:45 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:14:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:45 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:45 smithi122 ceph-mon[32255]: Deploying daemon osd.0 on smithi022 2021-12-15T17:14:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:45 smithi122 ceph-mon[32255]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-12-15T17:14:45.688 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[31360]: pgmap v58: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:45.688 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:45.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:45.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:14:45.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:45.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[31360]: Deploying daemon osd.0 on smithi022 2021-12-15T17:14:45.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[31360]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-12-15T17:14:45.690 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[40250]: pgmap v58: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:45.690 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:45.690 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:45.691 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:14:45.691 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:45.691 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[40250]: Deploying daemon osd.0 on smithi022 2021-12-15T17:14:45.691 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:45 smithi022 ceph-mon[40250]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-12-15T17:14:46.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:46 smithi022 systemd[1]: Stopping Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:14:46.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:46 smithi022 bash[58629]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.0" found: no such container 2021-12-15T17:14:46.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:46 smithi022 conmon[52413]: 2021-12-15T17:14:46.657+0000 7f23b0e36700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:14:46.758 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:46 smithi022 conmon[52413]: 2021-12-15T17:14:46.657+0000 7f23b0e36700 -1 osd.0 18 *** Got signal Terminated *** 2021-12-15T17:14:46.758 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:46 smithi022 conmon[52413]: 2021-12-15T17:14:46.657+0000 7f23b0e36700 -1 osd.0 18 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:14:46.758 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[31360]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-12-15T17:14:47.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:46 smithi022 bash[58629]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-0 2021-12-15T17:14:47.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:46 smithi022 bash[58629]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.0" found: no such container 2021-12-15T17:14:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[31360]: osdmap e18: 3 total, 2 up, 3 in 2021-12-15T17:14:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[31360]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[31360]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:14:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[40250]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-12-15T17:14:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[40250]: osdmap e18: 3 total, 2 up, 3 in 2021-12-15T17:14:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[40250]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:47.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:47.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[40250]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:14:47.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:46 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:46 smithi122 ceph-mon[32255]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-12-15T17:14:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:46 smithi122 ceph-mon[32255]: osdmap e18: 3 total, 2 up, 3 in 2021-12-15T17:14:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:46 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:46 smithi122 ceph-mon[32255]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:46 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:46 smithi122 ceph-mon[32255]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:14:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:46 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:14:47.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:47 smithi022 podman[58948]: 2021-12-15T17:14:47.943 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:47 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.0.service: Succeeded. 2021-12-15T17:14:47.944 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:47 smithi022 systemd[1]: Stopped Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:14:47.944 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[31360]: pgmap v60: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:47.944 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[31360]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:14:47.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[31360]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:47.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[31360]: osdmap e19: 3 total, 1 up, 3 in 2021-12-15T17:14:47.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:47.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:47.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[40250]: pgmap v60: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:47.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[40250]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:14:47.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[40250]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:47.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[40250]: osdmap e19: 3 total, 1 up, 3 in 2021-12-15T17:14:47.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:47.948 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:47 smithi122 ceph-mon[32255]: pgmap v60: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:47 smithi122 ceph-mon[32255]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:14:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:47 smithi122 ceph-mon[32255]: from='osd.2 v1:172.21.15.22:6809/2271831797' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:14:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:47 smithi122 ceph-mon[32255]: osdmap e19: 3 total, 1 up, 3 in 2021-12-15T17:14:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:48.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 systemd[1]: Starting Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:14:48.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 podman[59248]: 2021-12-15T17:14:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:14:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:14:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[31360]: osd.2 v1:172.21.15.22:6809/2271831797 boot 2021-12-15T17:14:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[31360]: osdmap e20: 3 total, 2 up, 3 in 2021-12-15T17:14:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:14:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:14:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[40250]: osd.2 v1:172.21.15.22:6809/2271831797 boot 2021-12-15T17:14:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[40250]: osdmap e20: 3 total, 2 up, 3 in 2021-12-15T17:14:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:14:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:14:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-12-15T17:14:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:14:49.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:49.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 bash[59094]: --> ceph-volume lvm activate successful for osd ID: 0 2021-12-15T17:14:49.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-12-15T17:14:49.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:49.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:14:49.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:14:49.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:14:49.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:48 smithi022 conmon[59288]: --> ceph-volume lvm activate successful for osd ID: 0 2021-12-15T17:14:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:48 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:14:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:48 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:14:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:48 smithi122 ceph-mon[32255]: osd.2 v1:172.21.15.22:6809/2271831797 boot 2021-12-15T17:14:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:48 smithi122 ceph-mon[32255]: osdmap e20: 3 total, 2 up, 3 in 2021-12-15T17:14:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:14:49.691 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:49 smithi022 podman[59521]: 2021-12-15T17:14:49.691 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:49 smithi022 bash[59094]: e7a87c7e143cf7d154b7eb84f537f8f862e8e214946f8677c8e6344fe0964790 2021-12-15T17:14:49.692 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:49 smithi022 systemd[1]: Started Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:14:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: osdmap e21: 3 total, 2 up, 3 in 2021-12-15T17:14:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:14:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: osdmap e21: 3 total, 2 up, 3 in 2021-12-15T17:14:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:14:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:50.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2021-12-15T17:14:50.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: osdmap e21: 3 total, 2 up, 3 in 2021-12-15T17:14:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:50.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:14:50.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:50.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:50.392 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:50 smithi022 conmon[59561]: 2021-12-15T17:14:50.126+0000 7f54258593c0 -1 Falling back to public interface 2021-12-15T17:14:51.007 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:50 smithi022 ceph-mon[31360]: Deploying daemon osd.1 on smithi022 2021-12-15T17:14:51.008 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:50 smithi022 ceph-mon[40250]: Deploying daemon osd.1 on smithi022 2021-12-15T17:14:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:50 smithi122 ceph-mon[32255]: Deploying daemon osd.1 on smithi022 2021-12-15T17:14:51.915 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:51 smithi022 ceph-mon[31360]: pgmap v65: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:51.916 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:51 smithi022 ceph-mon[40250]: pgmap v65: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:52.167 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:51 smithi022 systemd[1]: Stopping Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:14:52.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:51 smithi022 bash[60413]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.1" found: no such container 2021-12-15T17:14:52.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:52 smithi022 conmon[49696]: 2021-12-15T17:14:52.060+0000 7ff9d042c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:14:52.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:52 smithi022 conmon[49696]: 2021-12-15T17:14:52.060+0000 7ff9d042c700 -1 osd.1 21 *** Got signal Terminated *** 2021-12-15T17:14:52.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:52 smithi022 conmon[49696]: 2021-12-15T17:14:52.060+0000 7ff9d042c700 -1 osd.1 21 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:14:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:51 smithi122 ceph-mon[32255]: pgmap v65: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:52.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:52 smithi022 bash[60413]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-1 2021-12-15T17:14:52.734 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:52 smithi022 conmon[59561]: 2021-12-15T17:14:52.582+0000 7f54258593c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:14:52.734 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:52 smithi022 bash[60413]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.1" found: no such container 2021-12-15T17:14:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: osd.1 failed (root=default,host=smithi022) (connection refused reported by osd.2) 2021-12-15T17:14:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:53.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: osd.1 failed (root=default,host=smithi022) (connection refused reported by osd.2) 2021-12-15T17:14:53.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:53.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:53.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:52 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.044 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:52 smithi022 conmon[59561]: 2021-12-15T17:14:52.867+0000 7f54258593c0 -1 osd.0 18 log_to_monitors true 2021-12-15T17:14:53.044 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:52 smithi022 podman[61015]: 2021-12-15T17:14:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: osd.1 failed (root=default,host=smithi022) (connection refused reported by osd.2) 2021-12-15T17:14:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:52 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:14:53.795 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:53 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.1.service: Succeeded. 2021-12-15T17:14:53.795 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:53 smithi022 systemd[1]: Stopped Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:14:53.795 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:53 smithi022 systemd[1]: Starting Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:14:53.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[31360]: pgmap v66: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:53.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[31360]: pgmap v67: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:53.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[31360]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2021-12-15T17:14:53.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[31360]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:14:53.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[31360]: osdmap e22: 3 total, 1 up, 3 in 2021-12-15T17:14:53.797 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:53.797 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[40250]: pgmap v66: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:53.798 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[40250]: pgmap v67: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:53.798 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[40250]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2021-12-15T17:14:53.798 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[40250]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:14:53.798 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[40250]: osdmap e22: 3 total, 1 up, 3 in 2021-12-15T17:14:53.798 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:53 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:54.074 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:14:53 smithi022 conmon[59561]: 2021-12-15T17:14:53.794+0000 7f54173f7700 -1 osd.0 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:14:54.075 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:53 smithi022 podman[61543]: 2021-12-15T17:14:54.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:53 smithi122 ceph-mon[32255]: pgmap v66: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:53 smithi122 ceph-mon[32255]: pgmap v67: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:53 smithi122 ceph-mon[32255]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2021-12-15T17:14:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:53 smithi122 ceph-mon[32255]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:14:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:53 smithi122 ceph-mon[32255]: osdmap e22: 3 total, 1 up, 3 in 2021-12-15T17:14:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:53 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:14:54.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:14:54.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:14:54.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:14:54.779 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:14:54.779 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2021-12-15T17:14:54.779 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:14:54.779 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:14:54.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:14:54.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:14:54.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 bash[61390]: --> ceph-volume lvm activate successful for osd ID: 1 2021-12-15T17:14:54.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2021-12-15T17:14:54.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:14:54.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:14:54.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:14:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:14:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:54 smithi022 conmon[61583]: --> ceph-volume lvm activate successful for osd ID: 1 2021-12-15T17:14:55.098 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:55.099 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[31360]: osdmap e23: 3 total, 1 up, 3 in 2021-12-15T17:14:55.099 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:55.099 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[31360]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:55.100 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:55.100 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[40250]: osdmap e23: 3 total, 1 up, 3 in 2021-12-15T17:14:55.100 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:55.100 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:54 smithi022 ceph-mon[40250]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:54 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:14:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:54 smithi122 ceph-mon[32255]: osdmap e23: 3 total, 1 up, 3 in 2021-12-15T17:14:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:54 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/2295733756' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:54 smithi122 ceph-mon[32255]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:55.360 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:55 smithi022 podman[61815]: 2021-12-15T17:14:55.360 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:55 smithi022 bash[61390]: 4c008b18ed15aac2cc395d89ebcc65e66cbb9089b3e501c421629b77f3709f71 2021-12-15T17:14:55.413 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 2,0,1 on host 'smithi022' 2021-12-15T17:14:55.711 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:55 smithi022 systemd[1]: Started Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:14:55.712 DEBUG:teuthology.orchestra.run.smithi022:osd.2> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.2.service 2021-12-15T17:14:55.715 INFO:tasks.cephadm:Deploying osd.3 on smithi022 with /dev/vg_nvme/lv_1... 2021-12-15T17:14:55.715 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-12-15T17:14:55.742 INFO:journalctl@ceph.osd.2.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:14:56.028 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:55 smithi022 conmon[61855]: 2021-12-15T17:14:55.801+0000 7f6dbdc053c0 -1 Falling back to public interface 2021-12-15T17:14:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: osd.0 v1:172.21.15.22:6801/2295733756 boot 2021-12-15T17:14:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: osdmap e24: 3 total, 2 up, 3 in 2021-12-15T17:14:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: osd.0 v1:172.21.15.22:6801/2295733756 boot 2021-12-15T17:14:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: osdmap e24: 3 total, 2 up, 3 in 2021-12-15T17:14:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: osd.0 v1:172.21.15.22:6801/2295733756 boot 2021-12-15T17:14:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: osdmap e24: 3 total, 2 up, 3 in 2021-12-15T17:14:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:14:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:14:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:14:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:14:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:57.173 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:56 smithi022 ceph-mon[31360]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:57.174 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:56 smithi022 ceph-mon[40250]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:57.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:56 smithi122 ceph-mon[32255]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:57.718 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-12-15T17:14:57.719 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-12-15T17:14:57.719 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2021-12-15T17:14:57.720 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2021-12-15T17:14:57.720 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0104124 s, 1.0 GB/s 2021-12-15T17:14:57.721 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2021-12-15T17:14:58.096 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi022:vg_nvme/lv_1 2021-12-15T17:14:58.123 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:58 smithi022 conmon[61855]: 2021-12-15T17:14:58.008+0000 7f6dbdc053c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:14:58.377 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:58 smithi022 conmon[61855]: 2021-12-15T17:14:58.288+0000 7f6dbdc053c0 -1 osd.1 21 log_to_monitors true 2021-12-15T17:14:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:58 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:58 smithi022 ceph-mon[31360]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:58.779 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:14:58 smithi022 conmon[61855]: 2021-12-15T17:14:58.533+0000 7f6db0fa6700 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:14:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:58 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:58 smithi022 ceph-mon[40250]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:58 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:14:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:58 smithi122 ceph-mon[32255]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:14:59.798 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[31360]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:59.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[31360]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:14:59.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[31360]: osdmap e25: 3 total, 2 up, 3 in 2021-12-15T17:14:59.800 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[31360]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:59.800 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[40250]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:59.801 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[40250]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:14:59.802 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[40250]: osdmap e25: 3 total, 2 up, 3 in 2021-12-15T17:14:59.802 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:14:59 smithi022 ceph-mon[40250]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:14:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:59 smithi122 ceph-mon[32255]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:14:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:59 smithi122 ceph-mon[32255]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:14:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:59 smithi122 ceph-mon[32255]: osdmap e25: 3 total, 2 up, 3 in 2021-12-15T17:14:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:14:59 smithi122 ceph-mon[32255]: from='osd.1 v1:172.21.15.22:6805/179918980' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:15:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: osd.1 v1:172.21.15.22:6805/179918980 boot 2021-12-15T17:15:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: osdmap e26: 3 total, 3 up, 3 in 2021-12-15T17:15:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:15:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-12-15T17:15:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:15:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: osd.1 v1:172.21.15.22:6805/179918980 boot 2021-12-15T17:15:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: osdmap e26: 3 total, 3 up, 3 in 2021-12-15T17:15:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:15:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-12-15T17:15:00.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:00.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:00.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:00 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:15:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: osd.1 v1:172.21.15.22:6805/179918980 boot 2021-12-15T17:15:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: osdmap e26: 3 total, 3 up, 3 in 2021-12-15T17:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-12-15T17:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:00 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:01.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='client.14313 v1:172.21.15.22:0/2338802352' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:01.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: pgmap v76: 0 pgs: ; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:01.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi022 2021-12-15T17:15:01.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 5384M 2021-12-15T17:15:01.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: pgmap v77: 0 pgs: ; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:01.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.905 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:01.905 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:01.905 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:01.905 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.906 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2021-12-15T17:15:01.906 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:15:01.906 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1}]': finished 2021-12-15T17:15:01.906 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: osdmap e27: 3 total, 3 up, 3 in 2021-12-15T17:15:01.907 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-12-15T17:15:01.907 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='client.14313 v1:172.21.15.22:0/2338802352' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:01.907 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: pgmap v76: 0 pgs: ; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:01.907 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi022 2021-12-15T17:15:01.908 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 5384M 2021-12-15T17:15:01.910 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.910 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: pgmap v77: 0 pgs: ; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:01.910 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:01.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:01.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:01.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2021-12-15T17:15:01.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:15:01.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1}]': finished 2021-12-15T17:15:01.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: osdmap e27: 3 total, 3 up, 3 in 2021-12-15T17:15:01.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:01 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-12-15T17:15:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='client.14313 v1:172.21.15.22:0/2338802352' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: pgmap v76: 0 pgs: ; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi022 2021-12-15T17:15:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 5384M 2021-12-15T17:15:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: pgmap v77: 0 pgs: ; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:01.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:01.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:01.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2021-12-15T17:15:01.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:15:01.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1}]': finished 2021-12-15T17:15:01.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: osdmap e27: 3 total, 3 up, 3 in 2021-12-15T17:15:01.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:01 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2021-12-15T17:15:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2021-12-15T17:15:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: osdmap e28: 3 total, 3 up, 3 in 2021-12-15T17:15:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:15:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:15:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/215628187' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]: dispatch 2021-12-15T17:15:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]: dispatch 2021-12-15T17:15:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]': finished 2021-12-15T17:15:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: osdmap e29: 4 total, 3 up, 4 in 2021-12-15T17:15:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2021-12-15T17:15:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: osdmap e28: 3 total, 3 up, 3 in 2021-12-15T17:15:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:15:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:15:03.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/215628187' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]: dispatch 2021-12-15T17:15:03.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]: dispatch 2021-12-15T17:15:03.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]': finished 2021-12-15T17:15:03.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: osdmap e29: 4 total, 3 up, 4 in 2021-12-15T17:15:03.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:03.038 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:02 smithi022 sudo[64218]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-15T17:15:03.038 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:02 smithi022 sudo[64218]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:03.038 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:02 smithi022 sudo[64218]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:03.038 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:02 smithi022 sudo[64218]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2021-12-15T17:15:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: osdmap e28: 3 total, 3 up, 3 in 2021-12-15T17:15:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:15:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:15:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/215628187' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]: dispatch 2021-12-15T17:15:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]: dispatch 2021-12-15T17:15:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0140e354-7030-40ef-8564-a25c37a35d1e"}]': finished 2021-12-15T17:15:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: osdmap e29: 4 total, 3 up, 4 in 2021-12-15T17:15:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:03.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64250]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-15T17:15:03.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64250]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:03.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64250]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:03.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64250]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[31360]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[31360]: mgrmap e15: y(active, since 2m), standbys: x 2021-12-15T17:15:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/286271555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:15:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[40250]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[40250]: mgrmap e15: y(active, since 2m), standbys: x 2021-12-15T17:15:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/286271555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:15:04.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64305]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-15T17:15:04.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64305]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:04.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64305]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:04.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:03 smithi022 sudo[64305]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:03 smithi122 ceph-mon[32255]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:03 smithi122 ceph-mon[32255]: mgrmap e15: y(active, since 2m), standbys: x 2021-12-15T17:15:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:03 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/286271555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:15:04.523 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64494]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-15T17:15:04.523 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64494]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:04.523 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64494]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:04.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64494]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:04.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64692]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-15T17:15:04.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64692]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:04.779 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64692]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:04.779 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[64692]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:05.278 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[65075]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-15T17:15:05.278 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[65075]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:05.278 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[65075]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:05.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:04 smithi022 sudo[65075]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65083]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-15T17:15:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65083]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65083]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65083]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:05 smithi122 ceph-mon[32255]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:05 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:05 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:05 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65140]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-15T17:15:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65140]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65140]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 sudo[65140]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[31360]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[40250]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:05 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:06.579 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65252]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-15T17:15:06.579 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65252]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:06.579 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65252]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:06.580 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65252]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65362]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-15T17:15:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65362]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65362]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:06 smithi022 sudo[65362]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:07.282 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34482]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-15T17:15:07.282 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34482]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:07.283 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34482]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:07.283 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34482]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:07.284 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34485]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-15T17:15:07.284 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34485]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-15T17:15:07.284 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34485]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-15T17:15:07.285 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 sudo[34485]: pam_unix(sudo:session): session closed for user root 2021-12-15T17:15:07.335 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:07.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:07.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:07.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:07.337 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:07.337 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:07.338 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:07.338 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:07.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:07.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:07.340 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:07.341 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:07.341 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:07.341 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:07.342 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:07.342 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:07.343 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:07.343 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:07.344 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:07.344 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:07.344 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:07.345 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:07 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2021-12-15T17:15:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:15:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:15:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:07 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:15:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-12-15T17:15:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-12-15T17:15:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:08 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:08 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-15T17:15:08.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:08 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-15T17:15:08.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:08 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:08.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:08 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-12-15T17:15:08.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:08 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:09.413 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:09 smithi022 ceph-mon[31360]: Deploying daemon osd.3 on smithi022 2021-12-15T17:15:09.413 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:09 smithi022 ceph-mon[31360]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2021-12-15T17:15:09.414 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:09 smithi022 ceph-mon[40250]: Deploying daemon osd.3 on smithi022 2021-12-15T17:15:09.415 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:09 smithi022 ceph-mon[40250]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2021-12-15T17:15:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:09 smithi122 ceph-mon[32255]: Deploying daemon osd.3 on smithi022 2021-12-15T17:15:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:09 smithi122 ceph-mon[32255]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2021-12-15T17:15:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:11 smithi022 ceph-mon[31360]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2021-12-15T17:15:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:11 smithi022 ceph-mon[40250]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2021-12-15T17:15:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:11 smithi122 ceph-mon[32255]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2021-12-15T17:15:13.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:13.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:13.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:13.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:12 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:12 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:12 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:12 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:13.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:13 smithi022 ceph-mon[31360]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2021-12-15T17:15:13.876 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:13 smithi022 ceph-mon[40250]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2021-12-15T17:15:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:13 smithi122 ceph-mon[32255]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2021-12-15T17:15:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-12-15T17:15:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: Deploying daemon osd.2 on smithi022 2021-12-15T17:15:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2021-12-15T17:15:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: from='osd.3 v1:172.21.15.22:6813/3933129392' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-12-15T17:15:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[31360]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-12-15T17:15:15.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-12-15T17:15:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:15.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: Deploying daemon osd.2 on smithi022 2021-12-15T17:15:15.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2021-12-15T17:15:15.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: from='osd.3 v1:172.21.15.22:6813/3933129392' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-12-15T17:15:15.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:15 smithi022 ceph-mon[40250]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-12-15T17:15:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-12-15T17:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: Deploying daemon osd.2 on smithi022 2021-12-15T17:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2021-12-15T17:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: from='osd.3 v1:172.21.15.22:6813/3933129392' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-12-15T17:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:15 smithi122 ceph-mon[32255]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-12-15T17:15:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[31360]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-12-15T17:15:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[31360]: osdmap e30: 4 total, 3 up, 4 in 2021-12-15T17:15:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[31360]: from='osd.3 v1:172.21.15.22:6813/3933129392' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[31360]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[40250]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-12-15T17:15:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[40250]: osdmap e30: 4 total, 3 up, 4 in 2021-12-15T17:15:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[40250]: from='osd.3 v1:172.21.15.22:6813/3933129392' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:16 smithi022 ceph-mon[40250]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:17.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:16 smithi022 systemd[1]: Stopping Ceph osd.2 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:15:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:16 smithi022 bash[68222]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.2" found: no such container 2021-12-15T17:15:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:16 smithi022 conmon[56885]: 2021-12-15T17:15:16.951+0000 7fa56081e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:15:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:16 smithi022 conmon[56885]: 2021-12-15T17:15:16.951+0000 7fa56081e700 -1 osd.2 31 *** Got signal Terminated *** 2021-12-15T17:15:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:16 smithi022 conmon[56885]: 2021-12-15T17:15:16.951+0000 7fa56081e700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:15:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:16 smithi122 ceph-mon[32255]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-12-15T17:15:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:16 smithi122 ceph-mon[32255]: osdmap e30: 4 total, 3 up, 4 in 2021-12-15T17:15:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:16 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:16 smithi122 ceph-mon[32255]: from='osd.3 v1:172.21.15.22:6813/3933129392' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:16 smithi122 ceph-mon[32255]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:17.346 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:17 smithi022 bash[68222]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-2 2021-12-15T17:15:17.635 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:17 smithi022 bash[68222]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.2" found: no such container 2021-12-15T17:15:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2021-12-15T17:15:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:15:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osdmap e31: 4 total, 3 up, 4 in 2021-12-15T17:15:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 failed (root=default,host=smithi022) (connection refused reported by osd.0) 2021-12-15T17:15:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[31360]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2021-12-15T17:15:18.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:15:18.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osdmap e31: 4 total, 3 up, 4 in 2021-12-15T17:15:18.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:18.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:18.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 failed (root=default,host=smithi022) (connection refused reported by osd.0) 2021-12-15T17:15:18.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:17 smithi022 ceph-mon[40250]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.038 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:17 smithi022 podman[68541]: 2021-12-15T17:15:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2021-12-15T17:15:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2021-12-15T17:15:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osdmap e31: 4 total, 3 up, 4 in 2021-12-15T17:15:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 failed (root=default,host=smithi022) (connection refused reported by osd.0) 2021-12-15T17:15:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.0 2021-12-15T17:15:18.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:17 smithi122 ceph-mon[32255]: osd.2 reported immediately failed by osd.1 2021-12-15T17:15:18.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:18 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.2.service: Succeeded. 2021-12-15T17:15:18.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:18 smithi022 systemd[1]: Stopped Ceph osd.2 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:15:18.755 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:18 smithi022 systemd[1]: Starting Ceph osd.2 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:15:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:15:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:15:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[31360]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:15:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[31360]: osd.3 v1:172.21.15.22:6813/3933129392 boot 2021-12-15T17:15:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[31360]: osdmap e32: 4 total, 3 up, 4 in 2021-12-15T17:15:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:15:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:15:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[40250]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:15:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[40250]: osd.3 v1:172.21.15.22:6813/3933129392 boot 2021-12-15T17:15:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[40250]: osdmap e32: 4 total, 3 up, 4 in 2021-12-15T17:15:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:18 smithi022 podman[68840]: 2021-12-15T17:15:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:18 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:15:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:18 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:15:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:18 smithi122 ceph-mon[32255]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:15:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:18 smithi122 ceph-mon[32255]: osd.3 v1:172.21.15.22:6813/3933129392 boot 2021-12-15T17:15:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:18 smithi122 ceph-mon[32255]: osdmap e32: 4 total, 3 up, 4 in 2021-12-15T17:15:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:15:19.755 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:15:19.755 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-12-15T17:15:19.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:15:19.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-12-15T17:15:19.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2021-12-15T17:15:19.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2021-12-15T17:15:19.757 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2021-12-15T17:15:19.757 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2021-12-15T17:15:19.757 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-12-15T17:15:19.757 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 bash[68686]: --> ceph-volume lvm activate successful for osd ID: 2 2021-12-15T17:15:19.757 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2021-12-15T17:15:19.758 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2021-12-15T17:15:19.758 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2021-12-15T17:15:19.758 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2021-12-15T17:15:19.758 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-12-15T17:15:19.758 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 conmon[68880]: --> ceph-volume lvm activate successful for osd ID: 2 2021-12-15T17:15:19.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:19 smithi022 ceph-mon[40250]: pgmap v92: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:19.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:19 smithi022 ceph-mon[40250]: osdmap e33: 4 total, 3 up, 4 in 2021-12-15T17:15:20.028 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:19 smithi022 podman[69114]: 2021-12-15T17:15:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:19 smithi022 ceph-mon[31360]: pgmap v92: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:19 smithi022 ceph-mon[31360]: osdmap e33: 4 total, 3 up, 4 in 2021-12-15T17:15:20.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:19 smithi122 ceph-mon[32255]: pgmap v92: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:20.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:19 smithi122 ceph-mon[32255]: osdmap e33: 4 total, 3 up, 4 in 2021-12-15T17:15:20.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:20 smithi022 bash[68686]: 955dfe5aaa79540b1384b147b812df852c0e41164c387e7d7cd83e0d8d1137fc 2021-12-15T17:15:20.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:20 smithi022 systemd[1]: Started Ceph osd.2 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:15:20.929 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:20 smithi022 conmon[69154]: 2021-12-15T17:15:20.623+0000 7f4f496213c0 -1 Falling back to public interface 2021-12-15T17:15:20.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:20.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:20.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:20.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:15:20.931 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:20.931 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:20.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:20.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:20.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:20.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:15:20.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:20.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:20 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:20 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:20 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:20 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:20 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-12-15T17:15:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:20 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:20 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:21.866 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:21 smithi022 ceph-mon[31360]: Deploying daemon osd.0 on smithi022 2021-12-15T17:15:21.867 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:21 smithi022 ceph-mon[31360]: pgmap v94: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:21.867 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:21 smithi022 ceph-mon[40250]: Deploying daemon osd.0 on smithi022 2021-12-15T17:15:21.868 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:21 smithi022 ceph-mon[40250]: pgmap v94: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:21 smithi122 ceph-mon[32255]: Deploying daemon osd.0 on smithi022 2021-12-15T17:15:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:21 smithi122 ceph-mon[32255]: pgmap v94: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:22.460 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:22 smithi022 systemd[1]: Stopping Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:15:22.460 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:22 smithi022 bash[69960]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.0" found: no such container 2021-12-15T17:15:22.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:22.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:15:22.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:22.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:22 smithi022 conmon[59561]: 2021-12-15T17:15:22.459+0000 7f541d262700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:15:22.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:22 smithi022 conmon[59561]: 2021-12-15T17:15:22.459+0000 7f541d262700 -1 osd.0 33 *** Got signal Terminated *** 2021-12-15T17:15:22.763 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:22 smithi022 conmon[59561]: 2021-12-15T17:15:22.459+0000 7f541d262700 -1 osd.0 33 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:15:22.763 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:22 smithi022 bash[69960]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-0 2021-12-15T17:15:22.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:22.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:15:22.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:22.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:22.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:22.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:22.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:22.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:22.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:22.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:22.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[40250]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:22 smithi022 bash[69960]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.0" found: no such container 2021-12-15T17:15:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:23.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:22 smithi022 ceph-mon[31360]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 failed (root=default,host=smithi022) (connection refused reported by osd.1) 2021-12-15T17:15:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:23.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.1 2021-12-15T17:15:23.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:22 smithi122 ceph-mon[32255]: osd.0 reported immediately failed by osd.3 2021-12-15T17:15:23.324 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:23 smithi022 podman[70682]: 2021-12-15T17:15:23.325 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:23 smithi022 conmon[69154]: 2021-12-15T17:15:23.088+0000 7f4f496213c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:15:23.699 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:23 smithi022 conmon[69154]: 2021-12-15T17:15:23.369+0000 7f4f496213c0 -1 osd.2 31 log_to_monitors true 2021-12-15T17:15:24.029 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 15 17:15:23 smithi022 conmon[69154]: 2021-12-15T17:15:23.769+0000 7f4f3c1c1700 -1 osd.2 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:15:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[40250]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[40250]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:15:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[40250]: osdmap e34: 4 total, 2 up, 4 in 2021-12-15T17:15:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[40250]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-12-15T17:15:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:24.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[31360]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:24.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[31360]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:15:24.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[31360]: osdmap e34: 4 total, 2 up, 4 in 2021-12-15T17:15:24.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:24.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[31360]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-12-15T17:15:24.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:23 smithi122 ceph-mon[32255]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:23 smithi122 ceph-mon[32255]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:15:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:23 smithi122 ceph-mon[32255]: osdmap e34: 4 total, 2 up, 4 in 2021-12-15T17:15:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:23 smithi122 ceph-mon[32255]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-12-15T17:15:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:24.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.0.service: Succeeded. 2021-12-15T17:15:24.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 systemd[1]: Stopped Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:15:24.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 systemd[1]: Starting Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:15:24.770 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 podman[71270]: 2021-12-15T17:15:24.770 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:24 smithi022 ceph-mon[31360]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-12-15T17:15:24.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:24 smithi022 ceph-mon[31360]: osdmap e35: 4 total, 2 up, 4 in 2021-12-15T17:15:24.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:24 smithi022 ceph-mon[31360]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:24.771 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:24 smithi022 ceph-mon[40250]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-12-15T17:15:24.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:24 smithi022 ceph-mon[40250]: osdmap e35: 4 total, 2 up, 4 in 2021-12-15T17:15:24.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:24 smithi022 ceph-mon[40250]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:25.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:15:25.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:15:25.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:15:25.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:15:25.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-12-15T17:15:25.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:15:25.030 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:15:25.030 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:15:25.030 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:15:25.030 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 bash[71116]: --> ceph-volume lvm activate successful for osd ID: 0 2021-12-15T17:15:25.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-12-15T17:15:25.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:15:25.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-12-15T17:15:25.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:15:25.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-12-15T17:15:25.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:24 smithi022 conmon[71310]: --> ceph-volume lvm activate successful for osd ID: 0 2021-12-15T17:15:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:24 smithi122 ceph-mon[32255]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-12-15T17:15:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:24 smithi122 ceph-mon[32255]: osdmap e35: 4 total, 2 up, 4 in 2021-12-15T17:15:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:24 smithi122 ceph-mon[32255]: from='osd.2 v1:172.21.15.22:6809/3252411997' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:25.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:25 smithi022 podman[71544]: 2021-12-15T17:15:25.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:25 smithi022 bash[71116]: eafaf7ea30dee66716bc2b3f62296acbf597160d80ca1aef5ab72b29c40bc530 2021-12-15T17:15:25.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[31360]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:25.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[31360]: osd.2 v1:172.21.15.22:6809/3252411997 boot 2021-12-15T17:15:25.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[31360]: osdmap e36: 4 total, 3 up, 4 in 2021-12-15T17:15:25.854 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:15:25.855 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[40250]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:25.855 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[40250]: osd.2 v1:172.21.15.22:6809/3252411997 boot 2021-12-15T17:15:25.855 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[40250]: osdmap e36: 4 total, 3 up, 4 in 2021-12-15T17:15:25.857 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:25 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:15:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:25 smithi122 ceph-mon[32255]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:25 smithi122 ceph-mon[32255]: osd.2 v1:172.21.15.22:6809/3252411997 boot 2021-12-15T17:15:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:25 smithi122 ceph-mon[32255]: osdmap e36: 4 total, 3 up, 4 in 2021-12-15T17:15:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:25 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:15:26.267 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:25 smithi022 systemd[1]: Started Ceph osd.0 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:15:26.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:26 smithi022 conmon[71584]: 2021-12-15T17:15:26.266+0000 7f25ae8803c0 -1 Falling back to public interface 2021-12-15T17:15:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[31360]: osdmap e37: 4 total, 3 up, 4 in 2021-12-15T17:15:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:15:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[40250]: osdmap e37: 4 total, 3 up, 4 in 2021-12-15T17:15:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:15:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:27.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:26 smithi122 ceph-mon[32255]: osdmap e37: 4 total, 3 up, 4 in 2021-12-15T17:15:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-12-15T17:15:27.197 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:27.201 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:27 smithi022 ceph-mon[31360]: Deploying daemon osd.1 on smithi022 2021-12-15T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:27 smithi022 ceph-mon[31360]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:27.952 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:27 smithi022 ceph-mon[40250]: Deploying daemon osd.1 on smithi022 2021-12-15T17:15:27.952 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:27 smithi022 ceph-mon[40250]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:27 smithi122 ceph-mon[32255]: Deploying daemon osd.1 on smithi022 2021-12-15T17:15:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:27 smithi122 ceph-mon[32255]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:28.264 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 systemd[1]: Stopping Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:15:28.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 bash[72436]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.1" found: no such container 2021-12-15T17:15:28.528 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 conmon[61855]: 2021-12-15T17:15:28.264+0000 7f6db560e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:15:28.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 conmon[61855]: 2021-12-15T17:15:28.264+0000 7f6db560e700 -1 osd.1 37 *** Got signal Terminated *** 2021-12-15T17:15:28.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 conmon[61855]: 2021-12-15T17:15:28.264+0000 7f6db560e700 -1 osd.1 37 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:15:28.931 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:28 smithi022 conmon[71584]: 2021-12-15T17:15:28.737+0000 7f25ae8803c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:15:28.932 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 bash[72436]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-1 2021-12-15T17:15:28.932 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 bash[72436]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.1" found: no such container 2021-12-15T17:15:28.932 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.933 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 failed (root=default,host=smithi022) (connection refused reported by osd.2) 2021-12-15T17:15:28.933 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.933 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.933 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.933 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.934 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.934 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.934 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.935 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:28.935 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:28.935 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.935 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 failed (root=default,host=smithi022) (connection refused reported by osd.2) 2021-12-15T17:15:28.936 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.936 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.936 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.936 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.937 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:28.937 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.937 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:28.937 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:28.937 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:28 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 failed (root=default,host=smithi022) (connection refused reported by osd.2) 2021-12-15T17:15:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.2 2021-12-15T17:15:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: osd.1 reported immediately failed by osd.3 2021-12-15T17:15:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:28 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:29.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:29 smithi022 conmon[71584]: 2021-12-15T17:15:29.016+0000 7f25ae8803c0 -1 osd.0 33 log_to_monitors true 2021-12-15T17:15:29.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:28 smithi022 podman[73083]: 2021-12-15T17:15:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[31360]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[31360]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:15:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[31360]: osdmap e38: 4 total, 2 up, 4 in 2021-12-15T17:15:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:15:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[31360]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2021-12-15T17:15:30.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 15 17:15:29 smithi022 conmon[71584]: 2021-12-15T17:15:29.791+0000 7f25a1c21700 -1 osd.0 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:15:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:29 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.1.service: Succeeded. 2021-12-15T17:15:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:29 smithi022 systemd[1]: Stopped Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:15:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:29 smithi022 systemd[1]: Starting Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:15:30.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[40250]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:30.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[40250]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:15:30.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[40250]: osdmap e38: 4 total, 2 up, 4 in 2021-12-15T17:15:30.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:15:30.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:29 smithi022 ceph-mon[40250]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2021-12-15T17:15:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:29 smithi122 ceph-mon[32255]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:29 smithi122 ceph-mon[32255]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:15:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:29 smithi122 ceph-mon[32255]: osdmap e38: 4 total, 2 up, 4 in 2021-12-15T17:15:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:29 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-12-15T17:15:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:29 smithi122 ceph-mon[32255]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2021-12-15T17:15:30.528 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 podman[73613]: 2021-12-15T17:15:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:30 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:15:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:30 smithi022 ceph-mon[40250]: osdmap e39: 4 total, 2 up, 4 in 2021-12-15T17:15:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:30 smithi022 ceph-mon[40250]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:30 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:15:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:30 smithi022 ceph-mon[31360]: osdmap e39: 4 total, 2 up, 4 in 2021-12-15T17:15:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:30 smithi022 ceph-mon[31360]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:31.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:15:31.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:15:31.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:15:31.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:15:31.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2021-12-15T17:15:31.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:15:31.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:15:31.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:15:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:15:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 bash[73460]: --> ceph-volume lvm activate successful for osd ID: 1 2021-12-15T17:15:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2021-12-15T17:15:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:15:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2021-12-15T17:15:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:15:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-12-15T17:15:31.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:30 smithi022 conmon[73653]: --> ceph-volume lvm activate successful for osd ID: 1 2021-12-15T17:15:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:30 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-12-15T17:15:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:30 smithi122 ceph-mon[32255]: osdmap e39: 4 total, 2 up, 4 in 2021-12-15T17:15:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:30 smithi122 ceph-mon[32255]: from='osd.0 v1:172.21.15.22:6801/3812237890' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:31.509 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:31 smithi022 podman[73886]: 2021-12-15T17:15:31.510 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:31 smithi022 bash[73460]: ecd03ba67b79b38b7db03e39ab5058c5d1b732af51cb4a00eab5e6a0dbc358f2 2021-12-15T17:15:31.510 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:31 smithi022 systemd[1]: Started Ceph osd.1 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:15:31.510 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 3,2,0,1 on host 'smithi022' 2021-12-15T17:15:31.907 DEBUG:teuthology.orchestra.run.smithi022:osd.3> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.3.service 2021-12-15T17:15:31.951 INFO:tasks.cephadm:Deploying osd.4 on smithi122 with /dev/vg_nvme/lv_4... 2021-12-15T17:15:31.952 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-12-15T17:15:31.985 INFO:journalctl@ceph.osd.3.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:15:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: pgmap v105: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: osd.0 v1:172.21.15.22:6801/3812237890 boot 2021-12-15T17:15:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: osdmap e40: 4 total, 3 up, 4 in 2021-12-15T17:15:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:15:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:32.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:32.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:32.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:32.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:32.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:31 smithi022 conmon[73926]: 2021-12-15T17:15:31.951+0000 7f8e479de3c0 -1 Falling back to public interface 2021-12-15T17:15:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: pgmap v105: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:32.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: osd.0 v1:172.21.15.22:6801/3812237890 boot 2021-12-15T17:15:32.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: osdmap e40: 4 total, 3 up, 4 in 2021-12-15T17:15:32.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:15:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:32.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:32.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:32.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:32.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:32.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: pgmap v105: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:32.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: osd.0 v1:172.21.15.22:6801/3812237890 boot 2021-12-15T17:15:32.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: osdmap e40: 4 total, 3 up, 4 in 2021-12-15T17:15:32.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:15:32.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:32.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:32.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:32.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:32.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:32 smithi122 ceph-mon[32255]: osdmap e41: 4 total, 3 up, 4 in 2021-12-15T17:15:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:32 smithi122 ceph-mon[32255]: pgmap v108: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:32 smithi022 ceph-mon[31360]: osdmap e41: 4 total, 3 up, 4 in 2021-12-15T17:15:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:32 smithi022 ceph-mon[31360]: pgmap v108: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:32 smithi022 ceph-mon[40250]: osdmap e41: 4 total, 3 up, 4 in 2021-12-15T17:15:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:32 smithi022 ceph-mon[40250]: pgmap v108: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:33.918 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-12-15T17:15:33.919 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-12-15T17:15:33.919 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2021-12-15T17:15:33.919 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2021-12-15T17:15:33.919 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124495 s, 842 MB/s 2021-12-15T17:15:33.919 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2021-12-15T17:15:34.371 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2021-12-15T17:15:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:34.730 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:34.731 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:34 smithi022 conmon[73926]: 2021-12-15T17:15:34.454+0000 7f8e479de3c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:15:34.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:35.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:34 smithi022 conmon[73926]: 2021-12-15T17:15:34.730+0000 7f8e479de3c0 -1 osd.1 37 log_to_monitors true 2021-12-15T17:15:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:35 smithi122 ceph-mon[32255]: pgmap v109: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:35 smithi122 ceph-mon[32255]: from='osd.1 v1:172.21.15.22:6805/2823805689' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:15:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:35 smithi122 ceph-mon[32255]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:15:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:35 smithi122 ceph-mon[32255]: Health check update: 1 osds down (OSD_DOWN) 2021-12-15T17:15:35.792 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 15 17:15:35 smithi022 conmon[73926]: 2021-12-15T17:15:35.549+0000 7f8e37d79700 -1 osd.1 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:15:35.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[31360]: pgmap v109: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:35.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[31360]: from='osd.1 v1:172.21.15.22:6805/2823805689' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:15:35.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[31360]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:15:35.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[31360]: Health check update: 1 osds down (OSD_DOWN) 2021-12-15T17:15:35.795 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[40250]: pgmap v109: 1 pgs: 1 stale+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:35.795 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[40250]: from='osd.1 v1:172.21.15.22:6805/2823805689' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:15:35.796 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[40250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-12-15T17:15:35.796 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:35 smithi022 ceph-mon[40250]: Health check update: 1 osds down (OSD_DOWN) 2021-12-15T17:15:36.808 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:15:36.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: osdmap e42: 4 total, 3 up, 4 in 2021-12-15T17:15:36.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='osd.1 v1:172.21.15.22:6805/2823805689' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:36.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:36.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:36.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:36.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:36.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:36.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:36.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:36.813 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:36.813 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:36.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:36.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:36.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:36.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:36.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:36.815 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:36 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:15:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: osdmap e42: 4 total, 3 up, 4 in 2021-12-15T17:15:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='osd.1 v1:172.21.15.22:6805/2823805689' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:37.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:37.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:37.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:37.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-12-15T17:15:37.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: osdmap e42: 4 total, 3 up, 4 in 2021-12-15T17:15:37.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='osd.1 v1:172.21.15.22:6805/2823805689' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:37.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2021-12-15T17:15:37.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:37.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:37.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:37.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:37.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:37.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:37.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:37.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:36 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi022 2021-12-15T17:15:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 4038M 2021-12-15T17:15:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: pgmap v111: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: from='client.34268 v1:172.21.15.122:0/2808149099' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:15:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: osd.1 v1:172.21.15.22:6805/2823805689 boot 2021-12-15T17:15:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: osdmap e43: 4 total, 4 up, 4 in 2021-12-15T17:15:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:15:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2021-12-15T17:15:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:37 smithi122 ceph-mon[32255]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2021-12-15T17:15:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi022 2021-12-15T17:15:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 4038M 2021-12-15T17:15:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: pgmap v111: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: from='client.34268 v1:172.21.15.122:0/2808149099' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:15:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: osd.1 v1:172.21.15.22:6805/2823805689 boot 2021-12-15T17:15:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: osdmap e43: 4 total, 4 up, 4 in 2021-12-15T17:15:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:15:38.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2021-12-15T17:15:38.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[31360]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2021-12-15T17:15:38.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi022 2021-12-15T17:15:38.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 4038M 2021-12-15T17:15:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: pgmap v111: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: from='client.34268 v1:172.21.15.122:0/2808149099' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:15:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: osd.1 v1:172.21.15.22:6805/2823805689 boot 2021-12-15T17:15:38.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: osdmap e43: 4 total, 4 up, 4 in 2021-12-15T17:15:38.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:15:38.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2021-12-15T17:15:38.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:37 smithi022 ceph-mon[40250]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2021-12-15T17:15:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:38 smithi122 ceph-mon[32255]: osdmap e44: 4 total, 4 up, 4 in 2021-12-15T17:15:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/1308213521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]: dispatch 2021-12-15T17:15:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:38 smithi122 ceph-mon[32255]: from='client.34274 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]: dispatch 2021-12-15T17:15:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:38 smithi122 ceph-mon[32255]: from='client.34274 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]': finished 2021-12-15T17:15:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:38 smithi122 ceph-mon[32255]: osdmap e45: 5 total, 4 up, 5 in 2021-12-15T17:15:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:38 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[40250]: osdmap e44: 4 total, 4 up, 4 in 2021-12-15T17:15:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/1308213521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]: dispatch 2021-12-15T17:15:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[40250]: from='client.34274 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]: dispatch 2021-12-15T17:15:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[40250]: from='client.34274 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]': finished 2021-12-15T17:15:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[40250]: osdmap e45: 5 total, 4 up, 5 in 2021-12-15T17:15:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[31360]: osdmap e44: 4 total, 4 up, 4 in 2021-12-15T17:15:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/1308213521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]: dispatch 2021-12-15T17:15:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[31360]: from='client.34274 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]: dispatch 2021-12-15T17:15:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[31360]: from='client.34274 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6125b2a3-33b6-48dd-8e49-58ee4b3612bb"}]': finished 2021-12-15T17:15:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[31360]: osdmap e45: 5 total, 4 up, 5 in 2021-12-15T17:15:39.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:38 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:39 smithi122 ceph-mon[32255]: pgmap v114: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/1184451850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:15:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:39 smithi022 ceph-mon[31360]: pgmap v114: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/1184451850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:15:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:39 smithi022 ceph-mon[40250]: pgmap v114: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/1184451850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:15:41.819 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:41 smithi122 ceph-mon[32255]: pgmap v116: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:41 smithi022 ceph-mon[31360]: pgmap v116: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:41 smithi022 ceph-mon[40250]: pgmap v116: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2021-12-15T17:15:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:42 smithi122 ceph-mon[32255]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:42 smithi122 ceph-mon[32255]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2021-12-15T17:15:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:42 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:15:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:42 smithi022 ceph-mon[31360]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:42 smithi022 ceph-mon[31360]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2021-12-15T17:15:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:42 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:15:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:42 smithi022 ceph-mon[40250]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:42 smithi022 ceph-mon[40250]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2021-12-15T17:15:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:42 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:15:45.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:45 smithi122 ceph-mon[32255]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:45.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:45 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:15:45.330 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:45 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:45.330 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:45 smithi122 ceph-mon[32255]: Deploying daemon osd.4 on smithi122 2021-12-15T17:15:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[31360]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:15:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[31360]: Deploying daemon osd.4 on smithi122 2021-12-15T17:15:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[40250]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:15:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:45 smithi022 ceph-mon[40250]: Deploying daemon osd.4 on smithi122 2021-12-15T17:15:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:47 smithi122 ceph-mon[32255]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:47 smithi022 ceph-mon[31360]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:47 smithi022 ceph-mon[40250]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:49.373 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:49 smithi122 ceph-mon[32255]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:49.373 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:49.373 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:49.374 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:49.374 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[31360]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[40250]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:51.410 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 4 on host 'smithi122' 2021-12-15T17:15:52.022 DEBUG:teuthology.orchestra.run.smithi122:osd.4> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.4.service 2021-12-15T17:15:52.024 INFO:tasks.cephadm:Deploying osd.5 on smithi122 with /dev/vg_nvme/lv_3... 2021-12-15T17:15:52.024 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-12-15T17:15:52.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:51 smithi122 ceph-mon[32255]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:52.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:52.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:52.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:52.050 INFO:journalctl@ceph.osd.4.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:15:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[31360]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[40250]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:15:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:15:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:52 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/1025793931' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:15:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:52 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:15:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:52 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/1025793931' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:15:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:52 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:15:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:52 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/1025793931' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:15:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:52 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:15:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:53 smithi122 ceph-mon[32255]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:53 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:15:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:53 smithi122 ceph-mon[32255]: osdmap e46: 5 total, 4 up, 5 in 2021-12-15T17:15:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:53 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:53 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/1025793931' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:15:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:53 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:15:53.937 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:15:53 smithi122 conmon[37626]: 2021-12-15T17:15:53.797+0000 7fce493ae700 -1 osd.4 0 waiting for initial osdmap 2021-12-15T17:15:53.937 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:15:53 smithi122 conmon[37626]: 2021-12-15T17:15:53.802+0000 7fce45d48700 -1 osd.4 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:15:54.166 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-12-15T17:15:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-12-15T17:15:54.167 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2021-12-15T17:15:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2021-12-15T17:15:54.167 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131143 s, 800 MB/s 2021-12-15T17:15:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2021-12-15T17:15:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[31360]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:15:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[31360]: osdmap e46: 5 total, 4 up, 5 in 2021-12-15T17:15:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/1025793931' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:15:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:15:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[40250]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:15:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[40250]: osdmap e46: 5 total, 4 up, 5 in 2021-12-15T17:15:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/1025793931' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:15:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:53 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:15:54.773 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2021-12-15T17:15:54.972 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:15:54.973 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: osdmap e47: 5 total, 4 up, 5 in 2021-12-15T17:15:54.973 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:54.973 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:54.973 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:54.973 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: osd.4 v1:172.21.15.122:6800/1025793931 boot 2021-12-15T17:15:54.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: osdmap e48: 5 total, 5 up, 5 in 2021-12-15T17:15:54.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:54 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:15:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: osdmap e47: 5 total, 4 up, 5 in 2021-12-15T17:15:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: osd.4 v1:172.21.15.122:6800/1025793931 boot 2021-12-15T17:15:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: osdmap e48: 5 total, 5 up, 5 in 2021-12-15T17:15:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:15:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: osdmap e47: 5 total, 4 up, 5 in 2021-12-15T17:15:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2021-12-15T17:15:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: osd.4 v1:172.21.15.122:6800/1025793931 boot 2021-12-15T17:15:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: osdmap e48: 5 total, 5 up, 5 in 2021-12-15T17:15:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:54 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:15:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:15:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:15:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi122 2021-12-15T17:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 17172M 2021-12-15T17:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:55 smithi122 ceph-mon[32255]: osdmap e49: 5 total, 5 up, 5 in 2021-12-15T17:15:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:15:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:15:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi122 2021-12-15T17:15:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 17172M 2021-12-15T17:15:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[31360]: osdmap e49: 5 total, 5 up, 5 in 2021-12-15T17:15:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:15:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:15:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi122 2021-12-15T17:15:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:15:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 17172M 2021-12-15T17:15:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:15:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:55 smithi022 ceph-mon[40250]: osdmap e49: 5 total, 5 up, 5 in 2021-12-15T17:15:57.124 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:56 smithi122 ceph-mon[32255]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:15:57.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:56 smithi122 ceph-mon[32255]: from='client.34295 v1:172.21.15.122:0/93672226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:57.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:56 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:57.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:56 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:57.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:56 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:57.126 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:56 smithi122 ceph-mon[32255]: osdmap e50: 5 total, 5 up, 5 in 2021-12-15T17:15:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[31360]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:15:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[31360]: from='client.34295 v1:172.21.15.122:0/93672226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[31360]: osdmap e50: 5 total, 5 up, 5 in 2021-12-15T17:15:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[40250]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:15:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[40250]: from='client.34295 v1:172.21.15.122:0/93672226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:15:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:15:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:15:57.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:15:57.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:56 smithi022 ceph-mon[40250]: osdmap e50: 5 total, 5 up, 5 in 2021-12-15T17:15:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:58 smithi022 ceph-mon[31360]: osdmap e51: 5 total, 5 up, 5 in 2021-12-15T17:15:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:58 smithi022 ceph-mon[40250]: osdmap e51: 5 total, 5 up, 5 in 2021-12-15T17:15:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:58 smithi122 ceph-mon[32255]: osdmap e51: 5 total, 5 up, 5 in 2021-12-15T17:16:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: osdmap e52: 5 total, 5 up, 5 in 2021-12-15T17:16:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/1081476802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]: dispatch 2021-12-15T17:16:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: from='client.34301 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]: dispatch 2021-12-15T17:16:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: from='client.34301 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]': finished 2021-12-15T17:16:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: osdmap e53: 6 total, 5 up, 6 in 2021-12-15T17:16:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3671786507' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: osdmap e52: 5 total, 5 up, 5 in 2021-12-15T17:16:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/1081476802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]: dispatch 2021-12-15T17:16:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: from='client.34301 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]: dispatch 2021-12-15T17:16:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: from='client.34301 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]': finished 2021-12-15T17:16:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: osdmap e53: 6 total, 5 up, 6 in 2021-12-15T17:16:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:15:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3671786507' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:00.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:00.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: osdmap e52: 5 total, 5 up, 5 in 2021-12-15T17:16:00.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/1081476802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]: dispatch 2021-12-15T17:16:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: from='client.34301 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]: dispatch 2021-12-15T17:16:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: from='client.34301 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d723e884-3aaa-4bec-990b-c70d6c3b6b4e"}]': finished 2021-12-15T17:16:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: osdmap e53: 6 total, 5 up, 6 in 2021-12-15T17:16:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:00.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:15:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3671786507' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:01 smithi122 ceph-mon[32255]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:01 smithi022 ceph-mon[31360]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:01 smithi022 ceph-mon[40250]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:16:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:16:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:16:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:16:03.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:16:03.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:16:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:03 smithi022 ceph-mon[31360]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2021-12-15T17:16:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:03 smithi022 ceph-mon[40250]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2021-12-15T17:16:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:03 smithi122 ceph-mon[32255]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2021-12-15T17:16:04.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:04 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:16:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:04 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:04 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:16:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:04 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:04 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:16:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:04 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:05 smithi022 ceph-mon[31360]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2021-12-15T17:16:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:05 smithi022 ceph-mon[31360]: Deploying daemon osd.5 on smithi122 2021-12-15T17:16:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:05 smithi022 ceph-mon[40250]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2021-12-15T17:16:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:05 smithi022 ceph-mon[40250]: Deploying daemon osd.5 on smithi122 2021-12-15T17:16:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:05 smithi122 ceph-mon[32255]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2021-12-15T17:16:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:05 smithi122 ceph-mon[32255]: Deploying daemon osd.5 on smithi122 2021-12-15T17:16:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:07 smithi122 ceph-mon[32255]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2021-12-15T17:16:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:07 smithi022 ceph-mon[31360]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2021-12-15T17:16:08.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:07 smithi022 ceph-mon[40250]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2021-12-15T17:16:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[31360]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2021-12-15T17:16:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[40250]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2021-12-15T17:16:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:09 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:09 smithi122 ceph-mon[32255]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2021-12-15T17:16:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:09 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:09 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:09 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:09 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:11 smithi122 ceph-mon[32255]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 51 KiB/s, 0 objects/s recovering 2021-12-15T17:16:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:11 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:11 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:11 smithi022 ceph-mon[31360]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 51 KiB/s, 0 objects/s recovering 2021-12-15T17:16:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:11 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:11 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:11 smithi022 ceph-mon[40250]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 51 KiB/s, 0 objects/s recovering 2021-12-15T17:16:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:11 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:11 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:13.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:12 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:16:13.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:12 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:13.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:12 smithi122 ceph-mon[32255]: from='osd.5 v1:172.21.15.122:6804/40917492' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:13.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:12 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:16:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[31360]: from='osd.5 v1:172.21.15.122:6804/40917492' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:16:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[40250]: from='osd.5 v1:172.21.15.122:6804/40917492' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:12 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:14.062 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:13 smithi122 ceph-mon[32255]: Deploying daemon osd.4 on smithi122 2021-12-15T17:16:14.063 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:13 smithi122 ceph-mon[32255]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 48 KiB/s, 0 objects/s recovering 2021-12-15T17:16:14.063 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:13 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:16:14.063 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:13 smithi122 ceph-mon[32255]: osdmap e54: 6 total, 5 up, 6 in 2021-12-15T17:16:14.063 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:13 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:14.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:13 smithi122 ceph-mon[32255]: from='osd.5 v1:172.21.15.122:6804/40917492' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:14.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:13 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[40250]: Deploying daemon osd.4 on smithi122 2021-12-15T17:16:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[40250]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 48 KiB/s, 0 objects/s recovering 2021-12-15T17:16:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:16:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[40250]: osdmap e54: 6 total, 5 up, 6 in 2021-12-15T17:16:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[40250]: from='osd.5 v1:172.21.15.122:6804/40917492' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[31360]: Deploying daemon osd.4 on smithi122 2021-12-15T17:16:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[31360]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail; 48 KiB/s, 0 objects/s recovering 2021-12-15T17:16:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:16:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[31360]: osdmap e54: 6 total, 5 up, 6 in 2021-12-15T17:16:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[31360]: from='osd.5 v1:172.21.15.122:6804/40917492' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:13 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:14.381 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:14 smithi122 systemd[1]: Stopping Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:16:14.381 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:14 smithi122 bash[44044]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.4" found: no such container 2021-12-15T17:16:14.660 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:14 smithi122 conmon[37626]: 2021-12-15T17:16:14.380+0000 7fce4a3b0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:16:14.661 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:14 smithi122 conmon[37626]: 2021-12-15T17:16:14.380+0000 7fce4a3b0700 -1 osd.4 55 *** Got signal Terminated *** 2021-12-15T17:16:14.661 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:14 smithi122 conmon[37626]: 2021-12-15T17:16:14.380+0000 7fce4a3b0700 -1 osd.4 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:16:14.661 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:14 smithi122 bash[44044]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-4 2021-12-15T17:16:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:16:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osdmap e55: 6 total, 5 up, 6 in 2021-12-15T17:16:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:16:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:14.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:14.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:14.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:14.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:14.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:14.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:14.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:14.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:16:14.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osd.5 v1:172.21.15.122:6804/40917492 boot 2021-12-15T17:16:14.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: osdmap e56: 6 total, 5 up, 6 in 2021-12-15T17:16:14.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:14 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:14.941 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:14 smithi122 bash[44044]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.4" found: no such container 2021-12-15T17:16:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:16:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osdmap e55: 6 total, 5 up, 6 in 2021-12-15T17:16:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:16:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:16:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osd.5 v1:172.21.15.122:6804/40917492 boot 2021-12-15T17:16:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: osdmap e56: 6 total, 5 up, 6 in 2021-12-15T17:16:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:16:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osdmap e55: 6 total, 5 up, 6 in 2021-12-15T17:16:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:15.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:16:15.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:15.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:15.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:15.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:15.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:16:15.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osd.5 v1:172.21.15.122:6804/40917492 boot 2021-12-15T17:16:15.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: osdmap e56: 6 total, 5 up, 6 in 2021-12-15T17:16:15.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:14 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:15.434 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:15 smithi122 podman[44357]: 2021-12-15T17:16:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:15 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:16:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:15 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:16:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:15 smithi122 ceph-mon[32255]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:15 smithi122 ceph-mon[32255]: osdmap e57: 6 total, 5 up, 6 in 2021-12-15T17:16:16.186 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:15 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.4.service: Succeeded. 2021-12-15T17:16:16.187 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:15 smithi122 systemd[1]: Stopped Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:16:16.187 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:15 smithi122 systemd[1]: Starting Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:16:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:16:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:16:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[31360]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[31360]: osdmap e57: 6 total, 5 up, 6 in 2021-12-15T17:16:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:16:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:16:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[40250]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2021-12-15T17:16:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:15 smithi022 ceph-mon[40250]: osdmap e57: 6 total, 5 up, 6 in 2021-12-15T17:16:16.684 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 podman[44653]: 2021-12-15T17:16:17.185 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:16:17.185 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:17.186 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:16:17.186 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:17.186 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2021-12-15T17:16:17.187 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:17.187 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:17.187 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:16:17.187 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:17.188 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 bash[44501]: --> ceph-volume lvm activate successful for osd ID: 4 2021-12-15T17:16:17.188 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2021-12-15T17:16:17.188 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:17.188 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:17.189 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:16:17.189 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:17.189 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:16 smithi122 conmon[44693]: --> ceph-volume lvm activate successful for osd ID: 4 2021-12-15T17:16:17.685 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:17 smithi122 podman[44927]: 2021-12-15T17:16:17.685 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:17 smithi122 bash[44501]: 3ca8d36c99a0a8fa995fd054255976bcfbdda6cf957d6cd41c3dd978e2849bcf 2021-12-15T17:16:17.784 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 5,4 on host 'smithi122' 2021-12-15T17:16:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:17 smithi022 ceph-mon[31360]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:17 smithi022 ceph-mon[31360]: osdmap e58: 6 total, 5 up, 6 in 2021-12-15T17:16:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:17 smithi022 ceph-mon[40250]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:17 smithi022 ceph-mon[40250]: osdmap e58: 6 total, 5 up, 6 in 2021-12-15T17:16:18.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:17 smithi122 ceph-mon[32255]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:18.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:17 smithi122 ceph-mon[32255]: osdmap e58: 6 total, 5 up, 6 in 2021-12-15T17:16:18.148 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:17 smithi122 systemd[1]: Started Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:16:18.177 DEBUG:teuthology.orchestra.run.smithi122:osd.5> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.5.service 2021-12-15T17:16:18.180 INFO:tasks.cephadm:Deploying osd.6 on smithi122 with /dev/vg_nvme/lv_2... 2021-12-15T17:16:18.180 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-12-15T17:16:18.207 INFO:journalctl@ceph.osd.5.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:16:18.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:18 smithi122 conmon[44967]: 2021-12-15T17:16:18.147+0000 7fbb837cf3c0 -1 Falling back to public interface 2021-12-15T17:16:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:18 smithi122 ceph-mon[32255]: osdmap e59: 6 total, 5 up, 6 in 2021-12-15T17:16:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:18 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[31360]: osdmap e59: 6 total, 5 up, 6 in 2021-12-15T17:16:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[40250]: osdmap e59: 6 total, 5 up, 6 in 2021-12-15T17:16:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:18 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:19.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:19 smithi122 ceph-mon[32255]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-12-15T17:16:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-12-15T17:16:20.203 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2021-12-15T17:16:20.204 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2021-12-15T17:16:20.204 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119724 s, 876 MB/s 2021-12-15T17:16:20.204 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 2021-12-15T17:16:20.204 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2021-12-15T17:16:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:19 smithi022 ceph-mon[31360]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:19 smithi022 ceph-mon[40250]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:20.619 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2021-12-15T17:16:20.879 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:20 smithi122 conmon[44967]: 2021-12-15T17:16:20.628+0000 7fbb837cf3c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:16:21.182 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:20 smithi122 conmon[44967]: 2021-12-15T17:16:20.904+0000 7fbb837cf3c0 -1 osd.4 55 log_to_monitors true 2021-12-15T17:16:21.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:20 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:21.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:20 smithi122 ceph-mon[32255]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:20 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:20 smithi022 ceph-mon[31360]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:20 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:20 smithi022 ceph-mon[40250]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:22.185 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:21 smithi122 conmon[44967]: 2021-12-15T17:16:21.895+0000 7fbb7536d700 -1 osd.4 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:16:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:21 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/763480588' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:21 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:21 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/763480588' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:21 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:21 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/763480588' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:21 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:23.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:22 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:16:23.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:22 smithi122 ceph-mon[32255]: osdmap e60: 6 total, 5 up, 6 in 2021-12-15T17:16:23.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:22 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/763480588' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:23.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:22 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:23.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:22 smithi122 ceph-mon[32255]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2021-12-15T17:16:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:16:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[31360]: osdmap e60: 6 total, 5 up, 6 in 2021-12-15T17:16:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/763480588' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[31360]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2021-12-15T17:16:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:16:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[40250]: osdmap e60: 6 total, 5 up, 6 in 2021-12-15T17:16:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/763480588' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:22 smithi022 ceph-mon[40250]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2021-12-15T17:16:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:16:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:16:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: osd.4 v1:172.21.15.122:6800/763480588 boot 2021-12-15T17:16:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: osdmap e61: 6 total, 6 up, 6 in 2021-12-15T17:16:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:16:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='client.34328 v1:172.21.15.122:0/4114697404' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:16:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:16:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:16:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi122 2021-12-15T17:16:24.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:24.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:24.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 8586M 2021-12-15T17:16:24.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:24.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:24.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:24.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:16:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:16:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: osd.4 v1:172.21.15.122:6800/763480588 boot 2021-12-15T17:16:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: osdmap e61: 6 total, 6 up, 6 in 2021-12-15T17:16:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:16:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='client.34328 v1:172.21.15.122:0/4114697404' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:16:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:16:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:16:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi122 2021-12-15T17:16:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 8586M 2021-12-15T17:16:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:24.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:24.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:24.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:16:24.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:16:24.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: osd.4 v1:172.21.15.122:6800/763480588 boot 2021-12-15T17:16:24.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: osdmap e61: 6 total, 6 up, 6 in 2021-12-15T17:16:24.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:16:24.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='client.34328 v1:172.21.15.122:0/4114697404' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:16:24.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:16:24.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:16:24.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:24.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi122 2021-12-15T17:16:24.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:24.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:24.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 8586M 2021-12-15T17:16:24.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:24.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:24.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:24.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:24.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[31360]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2021-12-15T17:16:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3499841258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]: dispatch 2021-12-15T17:16:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[31360]: from='client.34334 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]: dispatch 2021-12-15T17:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[31360]: from='client.34334 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]': finished 2021-12-15T17:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[31360]: osdmap e62: 7 total, 6 up, 7 in 2021-12-15T17:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3593772944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[40250]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2021-12-15T17:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3499841258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]: dispatch 2021-12-15T17:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[40250]: from='client.34334 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]: dispatch 2021-12-15T17:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[40250]: from='client.34334 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]': finished 2021-12-15T17:16:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[40250]: osdmap e62: 7 total, 6 up, 7 in 2021-12-15T17:16:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3593772944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:25 smithi122 ceph-mon[32255]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2021-12-15T17:16:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3499841258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]: dispatch 2021-12-15T17:16:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:25 smithi122 ceph-mon[32255]: from='client.34334 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]: dispatch 2021-12-15T17:16:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:25 smithi122 ceph-mon[32255]: from='client.34334 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49064534-bd3e-4197-a0e1-ed10f6fbcf1d"}]': finished 2021-12-15T17:16:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:25 smithi122 ceph-mon[32255]: osdmap e62: 7 total, 6 up, 7 in 2021-12-15T17:16:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:25 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3593772944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:27 smithi022 ceph-mon[31360]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:27 smithi022 ceph-mon[40250]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:28.165 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:27 smithi122 ceph-mon[32255]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:30.014 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:29 smithi122 ceph-mon[32255]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:29 smithi022 ceph-mon[31360]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:29 smithi022 ceph-mon[40250]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:31 smithi122 ceph-mon[32255]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-12-15T17:16:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:31 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:31 smithi122 ceph-mon[32255]: Deploying daemon osd.6 on smithi122 2021-12-15T17:16:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[31360]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-12-15T17:16:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[31360]: Deploying daemon osd.6 on smithi122 2021-12-15T17:16:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[40250]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-12-15T17:16:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:31 smithi022 ceph-mon[40250]: Deploying daemon osd.6 on smithi122 2021-12-15T17:16:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:33 smithi122 ceph-mon[32255]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:33 smithi022 ceph-mon[31360]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:33 smithi022 ceph-mon[40250]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:35.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:35 smithi122 ceph-mon[32255]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:35.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:35 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:35.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:35 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:35.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:35 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:35.990 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:35 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[31360]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[40250]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:35 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:37 smithi122 ceph-mon[32255]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:37 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:37 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:37 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:16:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:37 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[31360]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:16:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[40250]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:38.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:16:38.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:37 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:38 smithi122 ceph-mon[32255]: Deploying daemon osd.4 on smithi122 2021-12-15T17:16:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:38 smithi122 ceph-mon[32255]: from='osd.6 v1:172.21.15.122:6808/291548913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:16:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:38 smithi122 ceph-mon[32255]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:16:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:38 smithi022 ceph-mon[31360]: Deploying daemon osd.4 on smithi122 2021-12-15T17:16:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:38 smithi022 ceph-mon[31360]: from='osd.6 v1:172.21.15.122:6808/291548913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:16:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:38 smithi022 ceph-mon[31360]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:16:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:38 smithi022 ceph-mon[40250]: Deploying daemon osd.4 on smithi122 2021-12-15T17:16:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:38 smithi022 ceph-mon[40250]: from='osd.6 v1:172.21.15.122:6808/291548913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:16:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:38 smithi022 ceph-mon[40250]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:16:40.069 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:39 smithi122 systemd[1]: Stopping Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:16:40.070 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:39 smithi122 bash[51180]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.4" found: no such container 2021-12-15T17:16:40.070 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:39 smithi122 ceph-mon[32255]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:40.070 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:39 smithi122 ceph-mon[32255]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-12-15T17:16:40.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:39 smithi122 ceph-mon[32255]: osdmap e63: 7 total, 6 up, 7 in 2021-12-15T17:16:40.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:39 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:40.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:39 smithi122 ceph-mon[32255]: from='osd.6 v1:172.21.15.122:6808/291548913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:40.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:39 smithi122 ceph-mon[32255]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[31360]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[31360]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-12-15T17:16:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[31360]: osdmap e63: 7 total, 6 up, 7 in 2021-12-15T17:16:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[31360]: from='osd.6 v1:172.21.15.122:6808/291548913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[31360]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[40250]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[40250]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-12-15T17:16:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[40250]: osdmap e63: 7 total, 6 up, 7 in 2021-12-15T17:16:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[40250]: from='osd.6 v1:172.21.15.122:6808/291548913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:39 smithi022 ceph-mon[40250]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:40.390 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:40 smithi122 conmon[44967]: 2021-12-15T17:16:40.069+0000 7fbb7b1d8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:16:40.390 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:40 smithi122 conmon[44967]: 2021-12-15T17:16:40.069+0000 7fbb7b1d8700 -1 osd.4 64 *** Got signal Terminated *** 2021-12-15T17:16:40.391 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:40 smithi122 conmon[44967]: 2021-12-15T17:16:40.069+0000 7fbb7b1d8700 -1 osd.4 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:16:40.685 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:40 smithi122 bash[51180]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-4 2021-12-15T17:16:40.685 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:40 smithi122 bash[51180]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.4" found: no such container 2021-12-15T17:16:41.135 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:40 smithi122 podman[51493]: 2021-12-15T17:16:41.135 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:16:41.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osdmap e64: 7 total, 6 up, 7 in 2021-12-15T17:16:41.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:41.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:41.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.0) 2021-12-15T17:16:41.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.139 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.139 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.139 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.139 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.139 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.140 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.140 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.141 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.141 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.141 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.141 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.141 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.142 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.142 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.143 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.143 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.143 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.143 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.144 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.144 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.144 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.144 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:40 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:16:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osdmap e64: 7 total, 6 up, 7 in 2021-12-15T17:16:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.0) 2021-12-15T17:16:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:16:41.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osdmap e64: 7 total, 6 up, 7 in 2021-12-15T17:16:41.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:41.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:41.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.0) 2021-12-15T17:16:41.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.291 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.291 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.292 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.292 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.292 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.292 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.293 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.293 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:16:41.293 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.293 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.294 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:16:41.294 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:16:41.294 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.294 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:16:41.295 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.295 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:40 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:16:41.792 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:41 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.4.service: Succeeded. 2021-12-15T17:16:41.793 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:41 smithi122 systemd[1]: Stopped Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:16:41.793 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:41 smithi122 systemd[1]: Starting Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:16:42.059 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:41 smithi122 podman[51792]: 2021-12-15T17:16:42.060 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:16:42.060 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:16:42.060 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:42.060 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:16:42.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:42.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: osd.6 v1:172.21.15.122:6808/291548913 boot 2021-12-15T17:16:42.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: osdmap e65: 7 total, 6 up, 7 in 2021-12-15T17:16:42.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:41 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:16:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:16:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:16:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: osd.6 v1:172.21.15.122:6808/291548913 boot 2021-12-15T17:16:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: osdmap e65: 7 total, 6 up, 7 in 2021-12-15T17:16:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:16:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:16:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2021-12-15T17:16:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:16:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: osd.6 v1:172.21.15.122:6808/291548913 boot 2021-12-15T17:16:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: osdmap e65: 7 total, 6 up, 7 in 2021-12-15T17:16:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:41 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:16:42.685 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:16:42.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:42.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:16:42.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:42.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2021-12-15T17:16:42.687 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:42.687 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:42.687 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:16:42.687 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:42.687 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 bash[51638]: --> ceph-volume lvm activate successful for osd ID: 4 2021-12-15T17:16:42.688 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2021-12-15T17:16:42.688 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:42.688 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:16:42.688 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:16:42.689 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:16:42.689 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:42 smithi122 conmon[51832]: --> ceph-volume lvm activate successful for osd ID: 4 2021-12-15T17:16:43.059 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:42 smithi122 ceph-mon[32255]: osdmap e66: 7 total, 6 up, 7 in 2021-12-15T17:16:43.060 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:42 smithi122 ceph-mon[32255]: osdmap e67: 7 total, 6 up, 7 in 2021-12-15T17:16:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:42 smithi022 ceph-mon[31360]: osdmap e66: 7 total, 6 up, 7 in 2021-12-15T17:16:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:42 smithi022 ceph-mon[31360]: osdmap e67: 7 total, 6 up, 7 in 2021-12-15T17:16:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:42 smithi022 ceph-mon[40250]: osdmap e66: 7 total, 6 up, 7 in 2021-12-15T17:16:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:42 smithi022 ceph-mon[40250]: osdmap e67: 7 total, 6 up, 7 in 2021-12-15T17:16:43.321 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:43 smithi122 podman[52066]: 2021-12-15T17:16:43.321 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-osd[52131]: -- 172.21.15.122:0/2818234687 <== mon.2 v1:172.21.15.122:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 616469758 0 0) 0x556b8b4cab60 con 0x556b8b491000 2021-12-15T17:16:43.322 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:43 smithi122 bash[51638]: b18f9ce458b2fbfb35bf41f81285bbe6a93197b05246eb3e1ec65b0a0a002ac3 2021-12-15T17:16:43.685 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:43 smithi122 systemd[1]: Started Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:16:44.071 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:43 smithi122 conmon[52106]: 2021-12-15T17:16:43.772+0000 7f12d52073c0 -1 Falling back to public interface 2021-12-15T17:16:44.072 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-mon[32255]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:44.072 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:44.072 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:44.072 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:44.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:16:44.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:44.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[31360]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:16:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[40250]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:16:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:44 smithi122 ceph-mon[32255]: Deploying daemon osd.5 on smithi122 2021-12-15T17:16:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:44 smithi122 ceph-mon[32255]: osdmap e68: 7 total, 6 up, 7 in 2021-12-15T17:16:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:44 smithi122 ceph-mon[32255]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:44 smithi122 ceph-mon[32255]: osdmap e69: 7 total, 6 up, 7 in 2021-12-15T17:16:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[31360]: Deploying daemon osd.5 on smithi122 2021-12-15T17:16:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[31360]: osdmap e68: 7 total, 6 up, 7 in 2021-12-15T17:16:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[31360]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[31360]: osdmap e69: 7 total, 6 up, 7 in 2021-12-15T17:16:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[40250]: Deploying daemon osd.5 on smithi122 2021-12-15T17:16:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[40250]: osdmap e68: 7 total, 6 up, 7 in 2021-12-15T17:16:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[40250]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:44 smithi022 ceph-mon[40250]: osdmap e69: 7 total, 6 up, 7 in 2021-12-15T17:16:46.104 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:45 smithi122 systemd[1]: Stopping Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:16:46.105 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:45 smithi122 bash[53002]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.5" found: no such container 2021-12-15T17:16:46.423 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:46 smithi122 conmon[52106]: 2021-12-15T17:16:46.248+0000 7f12d52073c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:16:46.423 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:46 smithi122 conmon[42384]: 2021-12-15T17:16:46.104+0000 7ff67d8a3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:16:46.423 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:46 smithi122 conmon[42384]: 2021-12-15T17:16:46.104+0000 7ff67d8a3700 -1 osd.5 69 *** Got signal Terminated *** 2021-12-15T17:16:46.424 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:46 smithi122 conmon[42384]: 2021-12-15T17:16:46.104+0000 7ff67d8a3700 -1 osd.5 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:16:46.685 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:46 smithi122 bash[53002]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-5 2021-12-15T17:16:46.685 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:46 smithi122 bash[53002]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.5" found: no such container 2021-12-15T17:16:46.685 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:46 smithi122 conmon[52106]: 2021-12-15T17:16:46.529+0000 7f12d52073c0 -1 osd.4 64 log_to_monitors true 2021-12-15T17:16:47.185 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:16:46 smithi122 conmon[52106]: 2021-12-15T17:16:46.940+0000 7f12c45a0700 -1 osd.4 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:16:47.185 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:46 smithi122 podman[53746]: 2021-12-15T17:16:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:16:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:47.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/3675847345' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.194 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:46 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:47.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:47.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:16:47.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:47.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.290 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.291 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.291 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.292 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.292 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/3675847345' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:47.293 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:47.293 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.293 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.294 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.294 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.295 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.295 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.296 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.296 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.297 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.298 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.298 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:47.299 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:47.299 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.300 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:16:47.301 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.301 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.302 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.302 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.303 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.303 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.304 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.304 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.304 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.305 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:47.305 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.306 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.306 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.307 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.307 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.308 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.308 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.308 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.309 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.309 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.310 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/3675847345' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:47.310 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:16:47.311 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.311 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:16:47.312 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.312 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.312 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:16:47.313 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.313 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.2 2021-12-15T17:16:47.314 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:16:47.314 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.315 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:46 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:16:47.933 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:47 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.5.service: Succeeded. 2021-12-15T17:16:47.933 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:47 smithi122 systemd[1]: Stopped Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:16:47.933 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:47 smithi122 systemd[1]: Starting Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:16:48.183 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 podman[54045]: 2021-12-15T17:16:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:47 smithi122 ceph-mon[32255]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:16:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:47 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:16:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:47 smithi122 ceph-mon[32255]: osdmap e70: 7 total, 5 up, 7 in 2021-12-15T17:16:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:47 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/3675847345' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:47 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[31360]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:16:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:16:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[31360]: osdmap e70: 7 total, 5 up, 7 in 2021-12-15T17:16:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/3675847345' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[40250]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:16:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:16:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[40250]: osdmap e70: 7 total, 5 up, 7 in 2021-12-15T17:16:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/3675847345' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:47 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:48.935 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:16:48.936 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:16:48.936 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:16:48.936 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:16:48.936 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2021-12-15T17:16:48.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:16:48.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:16:48.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:16:48.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:16:48.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 bash[53893]: --> ceph-volume lvm activate successful for osd ID: 5 2021-12-15T17:16:48.938 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2021-12-15T17:16:48.938 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:16:48.938 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:16:48.938 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:16:48.939 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:16:48.939 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:48 smithi122 conmon[54085]: --> ceph-volume lvm activate successful for osd ID: 5 2021-12-15T17:16:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:48 smithi122 ceph-mon[32255]: osd.4 v1:172.21.15.122:6800/3675847345 boot 2021-12-15T17:16:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:48 smithi122 ceph-mon[32255]: osdmap e71: 7 total, 6 up, 7 in 2021-12-15T17:16:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:48 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:16:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:48 smithi122 ceph-mon[32255]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 133 KiB/s, 0 objects/s recovering 2021-12-15T17:16:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[31360]: osd.4 v1:172.21.15.122:6800/3675847345 boot 2021-12-15T17:16:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[31360]: osdmap e71: 7 total, 6 up, 7 in 2021-12-15T17:16:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:16:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[31360]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 133 KiB/s, 0 objects/s recovering 2021-12-15T17:16:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[40250]: osd.4 v1:172.21.15.122:6800/3675847345 boot 2021-12-15T17:16:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[40250]: osdmap e71: 7 total, 6 up, 7 in 2021-12-15T17:16:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:16:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:48 smithi022 ceph-mon[40250]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 133 KiB/s, 0 objects/s recovering 2021-12-15T17:16:49.527 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:49 smithi122 podman[54318]: 2021-12-15T17:16:49.528 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:49 smithi122 bash[53893]: 518f077c6d9e2f896fec458d62dde5831e61216441b9f8820a05c80949dca2f4 2021-12-15T17:16:49.581 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 6,4,5 on host 'smithi122' 2021-12-15T17:16:49.812 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:49 smithi122 systemd[1]: Started Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:16:50.012 DEBUG:teuthology.orchestra.run.smithi122:osd.6> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.6.service 2021-12-15T17:16:50.015 INFO:tasks.cephadm:Deploying osd.7 on smithi122 with /dev/vg_nvme/lv_1... 2021-12-15T17:16:50.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-12-15T17:16:50.042 INFO:journalctl@ceph.osd.6.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:16:50.063 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:50.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:50.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:50.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:50.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:50.065 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:49 smithi122 conmon[54358]: 2021-12-15T17:16:49.981+0000 7f6c0a62b3c0 -1 Falling back to public interface 2021-12-15T17:16:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:50 smithi022 ceph-mon[31360]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2021-12-15T17:16:51.349 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:50 smithi022 ceph-mon[40250]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2021-12-15T17:16:51.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:50 smithi122 ceph-mon[32255]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2021-12-15T17:16:52.221 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-12-15T17:16:52.221 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-12-15T17:16:52.221 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2021-12-15T17:16:52.222 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2021-12-15T17:16:52.222 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012353 s, 849 MB/s 2021-12-15T17:16:52.222 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 2021-12-15T17:16:52.223 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2021-12-15T17:16:52.554 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:52 smithi122 conmon[54358]: 2021-12-15T17:16:52.439+0000 7f6c0a62b3c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:16:52.806 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:52 smithi122 conmon[54358]: 2021-12-15T17:16:52.719+0000 7f6c0a62b3c0 -1 osd.5 69 log_to_monitors true 2021-12-15T17:16:52.862 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_1 2021-12-15T17:16:53.069 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:52 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:53.069 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:52 smithi122 ceph-mon[32255]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2021-12-15T17:16:53.069 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:52 smithi122 ceph-mon[32255]: from='osd.5 v1:172.21.15.122:6804/2449686773' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:53.070 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:52 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:53.070 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:16:52 smithi122 conmon[54358]: 2021-12-15T17:16:52.891+0000 7f6bfb9c8700 -1 osd.5 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:16:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[31360]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2021-12-15T17:16:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[31360]: from='osd.5 v1:172.21.15.122:6804/2449686773' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[40250]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2021-12-15T17:16:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[40250]: from='osd.5 v1:172.21.15.122:6804/2449686773' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:52 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:16:54.156 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:53 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:16:54.157 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:53 smithi122 ceph-mon[32255]: osdmap e72: 7 total, 6 up, 7 in 2021-12-15T17:16:54.158 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:53 smithi122 ceph-mon[32255]: from='osd.5 v1:172.21.15.122:6804/2449686773' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:54.158 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:53 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:16:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[31360]: osdmap e72: 7 total, 6 up, 7 in 2021-12-15T17:16:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[31360]: from='osd.5 v1:172.21.15.122:6804/2449686773' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:16:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[40250]: osdmap e72: 7 total, 6 up, 7 in 2021-12-15T17:16:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[40250]: from='osd.5 v1:172.21.15.122:6804/2449686773' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:53 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:16:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:54 smithi122 ceph-mon[32255]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:16:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:54 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:16:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:54 smithi122 ceph-mon[32255]: osd.5 v1:172.21.15.122:6804/2449686773 boot 2021-12-15T17:16:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:54 smithi122 ceph-mon[32255]: osdmap e73: 7 total, 7 up, 7 in 2021-12-15T17:16:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:54 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:54 smithi122 ceph-mon[32255]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[31360]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:16:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:16:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[31360]: osd.5 v1:172.21.15.122:6804/2449686773 boot 2021-12-15T17:16:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[31360]: osdmap e73: 7 total, 7 up, 7 in 2021-12-15T17:16:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:55.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[31360]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[40250]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:16:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:16:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[40250]: osd.5 v1:172.21.15.122:6804/2449686773 boot 2021-12-15T17:16:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[40250]: osdmap e73: 7 total, 7 up, 7 in 2021-12-15T17:16:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:16:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:54 smithi022 ceph-mon[40250]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:55.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='client.34367 v1:172.21.15.122:0/2547759815' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:16:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:16:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:16:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi122 2021-12-15T17:16:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 5724M 2021-12-15T17:16:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='client.34367 v1:172.21.15.122:0/2547759815' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:16:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:16:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:16:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi122 2021-12-15T17:16:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 5724M 2021-12-15T17:16:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:56.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:56.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:56.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='client.34367 v1:172.21.15.122:0/2547759815' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:16:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-12-15T17:16:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-12-15T17:16:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi122 2021-12-15T17:16:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:56.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:16:56.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 5724M 2021-12-15T17:16:56.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:56.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:16:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:16:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:16:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:16:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:56 smithi122 ceph-mon[32255]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:56 smithi022 ceph-mon[31360]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:56 smithi022 ceph-mon[40250]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:57.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3355619065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]: dispatch 2021-12-15T17:16:57.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[31360]: from='client.34373 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]: dispatch 2021-12-15T17:16:57.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[31360]: from='client.34373 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]': finished 2021-12-15T17:16:57.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[31360]: osdmap e74: 8 total, 7 up, 8 in 2021-12-15T17:16:57.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:16:57.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3355619065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]: dispatch 2021-12-15T17:16:57.905 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[40250]: from='client.34373 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]: dispatch 2021-12-15T17:16:57.905 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[40250]: from='client.34373 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]': finished 2021-12-15T17:16:57.905 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[40250]: osdmap e74: 8 total, 7 up, 8 in 2021-12-15T17:16:57.905 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:57 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:16:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3355619065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]: dispatch 2021-12-15T17:16:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:57 smithi122 ceph-mon[32255]: from='client.34373 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]: dispatch 2021-12-15T17:16:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:57 smithi122 ceph-mon[32255]: from='client.34373 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c4fea5f-e883-4ceb-8dea-830393be48e3"}]': finished 2021-12-15T17:16:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:57 smithi122 ceph-mon[32255]: osdmap e74: 8 total, 7 up, 8 in 2021-12-15T17:16:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:57 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:16:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:58 smithi122 ceph-mon[32255]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:16:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3679403042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:58 smithi022 ceph-mon[31360]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:16:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3679403042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:16:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:58 smithi022 ceph-mon[40250]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:16:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:16:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3679403042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-12-15T17:17:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:01 smithi122 ceph-mon[32255]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:01 smithi022 ceph-mon[31360]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:01 smithi022 ceph-mon[40250]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:17:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:02 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:17:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:17:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:02 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:17:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:17:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:02 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:17:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:03 smithi122 ceph-mon[32255]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:03 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-12-15T17:17:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:03 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:03 smithi022 ceph-mon[31360]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:03 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-12-15T17:17:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:03 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:03 smithi022 ceph-mon[40250]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:03 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-12-15T17:17:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:03 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:04 smithi122 ceph-mon[32255]: Deploying daemon osd.7 on smithi122 2021-12-15T17:17:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:04 smithi022 ceph-mon[31360]: Deploying daemon osd.7 on smithi122 2021-12-15T17:17:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:04 smithi022 ceph-mon[40250]: Deploying daemon osd.7 on smithi122 2021-12-15T17:17:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:05 smithi122 ceph-mon[32255]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:05 smithi022 ceph-mon[31360]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:05 smithi022 ceph-mon[40250]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:07 smithi022 ceph-mon[31360]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:08.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:07 smithi022 ceph-mon[40250]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:08.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:07 smithi122 ceph-mon[32255]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:08.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:08 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:08.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:08 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:08.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:08 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:08.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:08 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:08 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:10.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:09 smithi122 ceph-mon[32255]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:09 smithi022 ceph-mon[31360]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:09 smithi022 ceph-mon[40250]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[31360]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-12-15T17:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[31360]: from='osd.7 v1:172.21.15.122:6812/1132880908' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-12-15T17:17:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[31360]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-12-15T17:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[40250]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-12-15T17:17:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[40250]: from='osd.7 v1:172.21.15.122:6812/1132880908' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-12-15T17:17:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:11 smithi022 ceph-mon[40250]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-12-15T17:17:12.101 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:11 smithi122 ceph-mon[32255]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:12.102 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:11 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:12.102 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:11 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:12.102 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:11 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-12-15T17:17:12.102 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:11 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:12.103 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:11 smithi122 ceph-mon[32255]: from='osd.7 v1:172.21.15.122:6812/1132880908' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-12-15T17:17:12.103 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:11 smithi122 ceph-mon[32255]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-12-15T17:17:13.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:12 smithi122 ceph-mon[32255]: Deploying daemon osd.6 on smithi122 2021-12-15T17:17:13.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:12 smithi122 ceph-mon[32255]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-12-15T17:17:13.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:12 smithi122 ceph-mon[32255]: osdmap e75: 8 total, 7 up, 8 in 2021-12-15T17:17:13.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:12 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:13.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:12 smithi122 ceph-mon[32255]: from='osd.7 v1:172.21.15.122:6812/1132880908' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:13.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:12 smithi122 ceph-mon[32255]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[31360]: Deploying daemon osd.6 on smithi122 2021-12-15T17:17:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[31360]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-12-15T17:17:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[31360]: osdmap e75: 8 total, 7 up, 8 in 2021-12-15T17:17:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[31360]: from='osd.7 v1:172.21.15.122:6812/1132880908' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[31360]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[40250]: Deploying daemon osd.6 on smithi122 2021-12-15T17:17:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[40250]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-12-15T17:17:13.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[40250]: osdmap e75: 8 total, 7 up, 8 in 2021-12-15T17:17:13.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:13.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[40250]: from='osd.7 v1:172.21.15.122:6812/1132880908' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:13.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:12 smithi022 ceph-mon[40250]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:13.685 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:13 smithi122 systemd[1]: Stopping Ceph osd.6 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:17:13.686 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:13 smithi122 bash[60563]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.6" found: no such container 2021-12-15T17:17:13.687 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:13 smithi122 conmon[49483]: 2021-12-15T17:17:13.565+0000 7f6e500b5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:17:13.687 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:13 smithi122 conmon[49483]: 2021-12-15T17:17:13.565+0000 7f6e500b5700 -1 osd.6 76 *** Got signal Terminated *** 2021-12-15T17:17:13.688 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:13 smithi122 conmon[49483]: 2021-12-15T17:17:13.565+0000 7f6e500b5700 -1 osd.6 76 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:17:14.185 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:13 smithi122 bash[60563]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-6 2021-12-15T17:17:14.185 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:13 smithi122 bash[60563]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.6" found: no such container 2021-12-15T17:17:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:17:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osdmap e76: 8 total, 7 up, 8 in 2021-12-15T17:17:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.0 2021-12-15T17:17:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 failed (root=default,host=smithi122) (connection refused reported by osd.0) 2021-12-15T17:17:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.2 2021-12-15T17:17:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.0 2021-12-15T17:17:14.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.3 2021-12-15T17:17:14.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.4 2021-12-15T17:17:14.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.2 2021-12-15T17:17:14.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.3 2021-12-15T17:17:14.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.5 2021-12-15T17:17:14.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.1 2021-12-15T17:17:14.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.4 2021-12-15T17:17:14.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.1 2021-12-15T17:17:14.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: osd.6 reported immediately failed by osd.5 2021-12-15T17:17:14.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:13 smithi122 ceph-mon[32255]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:17:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:17:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osdmap e76: 8 total, 7 up, 8 in 2021-12-15T17:17:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.0 2021-12-15T17:17:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 failed (root=default,host=smithi122) (connection refused reported by osd.0) 2021-12-15T17:17:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.2 2021-12-15T17:17:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.0 2021-12-15T17:17:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.3 2021-12-15T17:17:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.4 2021-12-15T17:17:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.2 2021-12-15T17:17:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.3 2021-12-15T17:17:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.5 2021-12-15T17:17:14.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.1 2021-12-15T17:17:14.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.4 2021-12-15T17:17:14.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.1 2021-12-15T17:17:14.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: osd.6 reported immediately failed by osd.5 2021-12-15T17:17:14.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[31360]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:17:14.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 43 MiB used, 626 GiB / 626 GiB avail 2021-12-15T17:17:14.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2021-12-15T17:17:14.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osdmap e76: 8 total, 7 up, 8 in 2021-12-15T17:17:14.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:14.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:14.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.0 2021-12-15T17:17:14.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 failed (root=default,host=smithi122) (connection refused reported by osd.0) 2021-12-15T17:17:14.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.2 2021-12-15T17:17:14.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.0 2021-12-15T17:17:14.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.3 2021-12-15T17:17:14.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.4 2021-12-15T17:17:14.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.2 2021-12-15T17:17:14.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.3 2021-12-15T17:17:14.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.5 2021-12-15T17:17:14.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.1 2021-12-15T17:17:14.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.4 2021-12-15T17:17:14.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.1 2021-12-15T17:17:14.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: osd.6 reported immediately failed by osd.5 2021-12-15T17:17:14.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:13 smithi022 ceph-mon[40250]: Health check failed: 1 osds down (OSD_DOWN) 2021-12-15T17:17:14.685 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:14 smithi122 podman[60883]: 2021-12-15T17:17:15.123 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:14 smithi122 ceph-mon[32255]: purged_snaps scrub starts 2021-12-15T17:17:15.124 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:14 smithi122 ceph-mon[32255]: purged_snaps scrub ok 2021-12-15T17:17:15.124 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:14 smithi122 ceph-mon[32255]: osd.7 v1:172.21.15.122:6812/1132880908 boot 2021-12-15T17:17:15.124 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:14 smithi122 ceph-mon[32255]: osdmap e77: 8 total, 7 up, 8 in 2021-12-15T17:17:15.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:14 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:15.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:14 smithi122 ceph-mon[32255]: osdmap e78: 8 total, 7 up, 8 in 2021-12-15T17:17:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[31360]: purged_snaps scrub starts 2021-12-15T17:17:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[31360]: purged_snaps scrub ok 2021-12-15T17:17:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[31360]: osd.7 v1:172.21.15.122:6812/1132880908 boot 2021-12-15T17:17:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[31360]: osdmap e77: 8 total, 7 up, 8 in 2021-12-15T17:17:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[31360]: osdmap e78: 8 total, 7 up, 8 in 2021-12-15T17:17:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[40250]: purged_snaps scrub starts 2021-12-15T17:17:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[40250]: purged_snaps scrub ok 2021-12-15T17:17:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[40250]: osd.7 v1:172.21.15.122:6812/1132880908 boot 2021-12-15T17:17:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[40250]: osdmap e77: 8 total, 7 up, 8 in 2021-12-15T17:17:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:17:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:14 smithi022 ceph-mon[40250]: osdmap e78: 8 total, 7 up, 8 in 2021-12-15T17:17:15.435 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:15 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.6.service: Succeeded. 2021-12-15T17:17:15.435 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:15 smithi122 systemd[1]: Stopped Ceph osd.6 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:15.435 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:15 smithi122 systemd[1]: Starting Ceph osd.6 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:17:15.798 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:15 smithi122 podman[61176]: 2021-12-15T17:17:16.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:15 smithi122 ceph-mon[32255]: pgmap v195: 1 pgs: 1 stale+active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:16.176 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:17:16.176 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-12-15T17:17:16.176 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:17:16.176 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-12-15T17:17:16.177 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-6 --no-mon-config 2021-12-15T17:17:16.177 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6/block 2021-12-15T17:17:16.177 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-6/block 2021-12-15T17:17:16.177 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2021-12-15T17:17:16.177 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-12-15T17:17:16.178 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: --> ceph-volume lvm activate successful for osd ID: 6 2021-12-15T17:17:16.178 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-6 --no-mon-config 2021-12-15T17:17:16.178 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6/block 2021-12-15T17:17:16.178 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-6/block 2021-12-15T17:17:16.178 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2021-12-15T17:17:16.179 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-12-15T17:17:16.179 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 conmon[61216]: --> ceph-volume lvm activate successful for osd ID: 6 2021-12-15T17:17:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:15 smithi022 ceph-mon[31360]: pgmap v195: 1 pgs: 1 stale+active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:15 smithi022 ceph-mon[40250]: pgmap v195: 1 pgs: 1 stale+active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:17.020 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:16 smithi122 ceph-mon[32255]: osdmap e79: 8 total, 7 up, 8 in 2021-12-15T17:17:17.020 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 podman[61450]: 2021-12-15T17:17:17.020 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 ceph-osd[61514]: -- 172.21.15.122:0/3875001904 <== mon.2 v1:172.21.15.122:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 1086738234 0 0) 0x560ace7bab60 con 0x560ace781000 2021-12-15T17:17:17.021 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:16 smithi122 bash[61026]: 3683700fd80f741c30d8acfb7f707bc5b1f1ccc901195dec5ea4e660b7b2ba88 2021-12-15T17:17:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:16 smithi022 ceph-mon[31360]: osdmap e79: 8 total, 7 up, 8 in 2021-12-15T17:17:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:16 smithi022 ceph-mon[40250]: osdmap e79: 8 total, 7 up, 8 in 2021-12-15T17:17:17.433 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:17 smithi122 systemd[1]: Started Ceph osd.6 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:17.684 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:17 smithi122 conmon[61490]: 2021-12-15T17:17:17.432+0000 7fc7703413c0 -1 Falling back to public interface 2021-12-15T17:17:18.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: pgmap v198: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%); 2/6 objects misplaced (33.333%) 2021-12-15T17:17:18.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2021-12-15T17:17:18.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: osdmap e80: 8 total, 7 up, 8 in 2021-12-15T17:17:18.125 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:18.126 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:18.126 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:18.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:17:18.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:18.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:18.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:17 smithi122 ceph-mon[32255]: Deploying daemon osd.4 on smithi122 2021-12-15T17:17:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: pgmap v198: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%); 2/6 objects misplaced (33.333%) 2021-12-15T17:17:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2021-12-15T17:17:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: osdmap e80: 8 total, 7 up, 8 in 2021-12-15T17:17:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:17:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[31360]: Deploying daemon osd.4 on smithi122 2021-12-15T17:17:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: pgmap v198: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%); 2/6 objects misplaced (33.333%) 2021-12-15T17:17:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2021-12-15T17:17:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: osdmap e80: 8 total, 7 up, 8 in 2021-12-15T17:17:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-12-15T17:17:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:17 smithi022 ceph-mon[40250]: Deploying daemon osd.4 on smithi122 2021-12-15T17:17:19.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:18 smithi122 ceph-mon[32255]: pgmap v200: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%); 2/6 objects misplaced (33.333%) 2021-12-15T17:17:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:18 smithi022 ceph-mon[31360]: pgmap v200: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%); 2/6 objects misplaced (33.333%) 2021-12-15T17:17:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:18 smithi022 ceph-mon[40250]: pgmap v200: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%); 2/6 objects misplaced (33.333%) 2021-12-15T17:17:19.727 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:19 smithi122 systemd[1]: Stopping Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:17:19.728 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:19 smithi122 bash[62388]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.4" found: no such container 2021-12-15T17:17:19.728 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:19 smithi122 conmon[52106]: 2021-12-15T17:17:19.710+0000 7f12ccc10700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:17:19.728 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:19 smithi122 conmon[52106]: 2021-12-15T17:17:19.710+0000 7f12ccc10700 -1 osd.4 80 *** Got signal Terminated *** 2021-12-15T17:17:19.728 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:19 smithi122 conmon[52106]: 2021-12-15T17:17:19.710+0000 7f12ccc10700 -1 osd.4 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:17:20.087 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:19 smithi122 bash[62388]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-4 2021-12-15T17:17:20.087 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:19 smithi122 conmon[61490]: 2021-12-15T17:17:19.899+0000 7fc7703413c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:17:20.088 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.088 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:17:20.088 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:17:20.089 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:17:20.089 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.089 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.0 2021-12-15T17:17:20.090 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.3 2021-12-15T17:17:20.090 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.5 2021-12-15T17:17:20.090 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.090 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:17:20.091 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.7 2021-12-15T17:17:20.091 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.1 2021-12-15T17:17:20.091 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: osd.4 reported immediately failed by osd.7 2021-12-15T17:17:20.091 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:20.091 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:19 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:17:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:17:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:17:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.0 2021-12-15T17:17:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.3 2021-12-15T17:17:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.5 2021-12-15T17:17:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:17:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.7 2021-12-15T17:17:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.1 2021-12-15T17:17:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: osd.4 reported immediately failed by osd.7 2021-12-15T17:17:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:17:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:17:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:17:20.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.0 2021-12-15T17:17:20.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.3 2021-12-15T17:17:20.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.5 2021-12-15T17:17:20.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.2 2021-12-15T17:17:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:17:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.7 2021-12-15T17:17:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.1 2021-12-15T17:17:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: osd.4 reported immediately failed by osd.7 2021-12-15T17:17:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:20.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:19 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:20.381 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:20 smithi122 bash[62388]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.4" found: no such container 2021-12-15T17:17:20.382 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:20 smithi122 conmon[61490]: 2021-12-15T17:17:20.182+0000 7fc7703413c0 -1 osd.6 76 log_to_monitors true 2021-12-15T17:17:20.684 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:20 smithi122 podman[63170]: 2021-12-15T17:17:21.180 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 15 17:17:20 smithi122 conmon[61490]: 2021-12-15T17:17:20.834+0000 7fc7616de700 -1 osd.6 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:17:21.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:20 smithi122 ceph-mon[32255]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:17:21.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:20 smithi122 ceph-mon[32255]: osdmap e81: 8 total, 6 up, 8 in 2021-12-15T17:17:21.181 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:20 smithi122 ceph-mon[32255]: pgmap v202: 1 pgs: 1 active+clean+remapped; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2/6 objects misplaced (33.333%); 108 KiB/s, 0 objects/s recovering 2021-12-15T17:17:21.181 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:20 smithi122 ceph-mon[32255]: from='osd.6 v1:172.21.15.122:6808/2434053838' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:17:21.181 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:20 smithi122 ceph-mon[32255]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:17:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[31360]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:17:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[31360]: osdmap e81: 8 total, 6 up, 8 in 2021-12-15T17:17:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[31360]: pgmap v202: 1 pgs: 1 active+clean+remapped; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2/6 objects misplaced (33.333%); 108 KiB/s, 0 objects/s recovering 2021-12-15T17:17:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[31360]: from='osd.6 v1:172.21.15.122:6808/2434053838' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:17:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[31360]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:17:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[40250]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:17:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[40250]: osdmap e81: 8 total, 6 up, 8 in 2021-12-15T17:17:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[40250]: pgmap v202: 1 pgs: 1 active+clean+remapped; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2/6 objects misplaced (33.333%); 108 KiB/s, 0 objects/s recovering 2021-12-15T17:17:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[40250]: from='osd.6 v1:172.21.15.122:6808/2434053838' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:17:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:20 smithi022 ceph-mon[40250]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-12-15T17:17:21.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:21 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.4.service: Succeeded. 2021-12-15T17:17:21.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:21 smithi122 systemd[1]: Stopped Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:21.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:21 smithi122 systemd[1]: Starting Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:17:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:21 smithi122 ceph-mon[32255]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2021-12-15T17:17:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:21 smithi122 ceph-mon[32255]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-12-15T17:17:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:21 smithi122 ceph-mon[32255]: osdmap e82: 8 total, 6 up, 8 in 2021-12-15T17:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:21 smithi122 ceph-mon[32255]: from='osd.6 v1:172.21.15.122:6808/2434053838' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:21 smithi122 ceph-mon[32255]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:21.936 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:21 smithi122 podman[63469]: 2021-12-15T17:17:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[31360]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2021-12-15T17:17:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[31360]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-12-15T17:17:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[31360]: osdmap e82: 8 total, 6 up, 8 in 2021-12-15T17:17:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[31360]: from='osd.6 v1:172.21.15.122:6808/2434053838' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[31360]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[40250]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2021-12-15T17:17:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[40250]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-12-15T17:17:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[40250]: osdmap e82: 8 total, 6 up, 8 in 2021-12-15T17:17:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[40250]: from='osd.6 v1:172.21.15.122:6808/2434053838' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:21 smithi022 ceph-mon[40250]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:22.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:17:22.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:17:22.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:17:22.436 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:17:22.436 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2021-12-15T17:17:22.436 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:17:22.436 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:17:22.437 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:17:22.437 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:17:22.437 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: --> ceph-volume lvm activate successful for osd ID: 4 2021-12-15T17:17:22.438 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2021-12-15T17:17:22.438 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:17:22.438 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2021-12-15T17:17:22.438 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-12-15T17:17:22.439 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-12-15T17:17:22.439 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 conmon[63509]: --> ceph-volume lvm activate successful for osd ID: 4 2021-12-15T17:17:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:22 smithi122 ceph-mon[32255]: osd.6 v1:172.21.15.122:6808/2434053838 boot 2021-12-15T17:17:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:22 smithi122 ceph-mon[32255]: osdmap e83: 8 total, 7 up, 8 in 2021-12-15T17:17:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:22 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:17:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:22 smithi122 ceph-mon[32255]: pgmap v205: 1 pgs: 1 active+clean+remapped; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 2/6 objects misplaced (33.333%); 108 KiB/s, 0 objects/s recovering 2021-12-15T17:17:22.976 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 podman[63742]: 2021-12-15T17:17:22.976 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 ceph-osd[63806]: -- 172.21.15.122:0/1398717835 <== mon.2 v1:172.21.15.122:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 3659811968 0 0) 0x55c5a0eb0d00 con 0x55c5a0e77400 2021-12-15T17:17:22.976 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 bash[63316]: f5f9e09fd082167634f0fa8e65e0fb117238c18c0e239973ba521f6b5766a909 2021-12-15T17:17:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[31360]: osd.6 v1:172.21.15.122:6808/2434053838 boot 2021-12-15T17:17:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[31360]: osdmap e83: 8 total, 7 up, 8 in 2021-12-15T17:17:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:17:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[31360]: pgmap v205: 1 pgs: 1 active+clean+remapped; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 2/6 objects misplaced (33.333%); 108 KiB/s, 0 objects/s recovering 2021-12-15T17:17:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[40250]: osd.6 v1:172.21.15.122:6808/2434053838 boot 2021-12-15T17:17:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[40250]: osdmap e83: 8 total, 7 up, 8 in 2021-12-15T17:17:23.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:17:23.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:22 smithi022 ceph-mon[40250]: pgmap v205: 1 pgs: 1 active+clean+remapped; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 2/6 objects misplaced (33.333%); 108 KiB/s, 0 objects/s recovering 2021-12-15T17:17:23.399 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:22 smithi122 systemd[1]: Started Ceph osd.4 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:23.665 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:23 smithi122 conmon[63782]: 2021-12-15T17:17:23.399+0000 7f7f38a723c0 -1 Falling back to public interface 2021-12-15T17:17:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: osdmap e84: 8 total, 7 up, 8 in 2021-12-15T17:17:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:17:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:23 smithi122 ceph-mon[32255]: Deploying daemon osd.5 on smithi122 2021-12-15T17:17:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: osdmap e84: 8 total, 7 up, 8 in 2021-12-15T17:17:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:17:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[31360]: Deploying daemon osd.5 on smithi122 2021-12-15T17:17:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: osdmap e84: 8 total, 7 up, 8 in 2021-12-15T17:17:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-12-15T17:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:24.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:24.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:23 smithi022 ceph-mon[40250]: Deploying daemon osd.5 on smithi122 2021-12-15T17:17:25.090 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:24 smithi122 ceph-mon[32255]: osdmap e85: 8 total, 7 up, 8 in 2021-12-15T17:17:25.090 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:24 smithi122 ceph-mon[32255]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:24 smithi022 ceph-mon[40250]: osdmap e85: 8 total, 7 up, 8 in 2021-12-15T17:17:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:24 smithi022 ceph-mon[40250]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:24 smithi022 ceph-mon[31360]: osdmap e85: 8 total, 7 up, 8 in 2021-12-15T17:17:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:24 smithi022 ceph-mon[31360]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:25.814 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:25 smithi122 systemd[1]: Stopping Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:17:25.814 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:25 smithi122 bash[64685]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.5" found: no such container 2021-12-15T17:17:26.143 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:25 smithi122 conmon[63782]: 2021-12-15T17:17:25.876+0000 7f7f38a723c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:17:26.144 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:25 smithi122 conmon[54358]: 2021-12-15T17:17:25.813+0000 7f6c02034700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2021-12-15T17:17:26.144 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:25 smithi122 conmon[54358]: 2021-12-15T17:17:25.813+0000 7f6c02034700 -1 osd.5 85 *** Got signal Terminated *** 2021-12-15T17:17:26.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:25 smithi122 conmon[54358]: 2021-12-15T17:17:25.813+0000 7f6c02034700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-12-15T17:17:26.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.2 2021-12-15T17:17:26.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:17:26.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:17:26.146 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:17:26.146 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:17:26.146 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.7 2021-12-15T17:17:26.146 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.0 2021-12-15T17:17:26.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:17:26.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.6 2021-12-15T17:17:26.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.1 2021-12-15T17:17:26.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.7 2021-12-15T17:17:26.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:25 smithi122 ceph-mon[32255]: osd.5 reported immediately failed by osd.3 2021-12-15T17:17:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.2 2021-12-15T17:17:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:17:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:17:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:17:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:17:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.7 2021-12-15T17:17:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.0 2021-12-15T17:17:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:17:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.6 2021-12-15T17:17:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.1 2021-12-15T17:17:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.7 2021-12-15T17:17:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[31360]: osd.5 reported immediately failed by osd.3 2021-12-15T17:17:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.2 2021-12-15T17:17:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2021-12-15T17:17:26.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:17:26.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:17:26.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:17:26.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.7 2021-12-15T17:17:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.0 2021-12-15T17:17:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:17:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.6 2021-12-15T17:17:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.1 2021-12-15T17:17:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.7 2021-12-15T17:17:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:25 smithi022 ceph-mon[40250]: osd.5 reported immediately failed by osd.3 2021-12-15T17:17:26.435 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:26 smithi122 conmon[63782]: 2021-12-15T17:17:26.166+0000 7f7f38a723c0 -1 osd.4 80 log_to_monitors true 2021-12-15T17:17:26.435 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:26 smithi122 bash[64685]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd-5 2021-12-15T17:17:26.436 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:26 smithi122 bash[64685]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-osd.5" found: no such container 2021-12-15T17:17:26.853 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:26 smithi122 podman[65547]: 2021-12-15T17:17:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:26 smithi122 ceph-mon[32255]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:17:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:26 smithi122 ceph-mon[32255]: osdmap e86: 8 total, 6 up, 8 in 2021-12-15T17:17:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:26 smithi122 ceph-mon[32255]: pgmap v210: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:26 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/3954553330' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:17:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:26 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:17:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:26 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:27.188 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 15 17:17:26 smithi122 conmon[63782]: 2021-12-15T17:17:26.935+0000 7f7f2be13700 -1 osd.4 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:17:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[31360]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:17:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[31360]: osdmap e86: 8 total, 6 up, 8 in 2021-12-15T17:17:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[31360]: pgmap v210: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/3954553330' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:17:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:17:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[40250]: Health check update: 2 osds down (OSD_DOWN) 2021-12-15T17:17:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[40250]: osdmap e86: 8 total, 6 up, 8 in 2021-12-15T17:17:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[40250]: pgmap v210: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/3954553330' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:17:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-12-15T17:17:27.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:26 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:27.491 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:27 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.5.service: Succeeded. 2021-12-15T17:17:27.492 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:27 smithi122 systemd[1]: Stopped Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:27.780 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:27 smithi122 systemd[1]: Starting Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:17:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:27 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:17:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:27 smithi122 ceph-mon[32255]: osdmap e87: 8 total, 6 up, 8 in 2021-12-15T17:17:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:27 smithi122 ceph-mon[32255]: from='osd.4 v1:172.21.15.122:6800/3954553330' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:27 smithi122 ceph-mon[32255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:28.186 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:27 smithi122 podman[65845]: 2021-12-15T17:17:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:17:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[31360]: osdmap e87: 8 total, 6 up, 8 in 2021-12-15T17:17:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[31360]: from='osd.4 v1:172.21.15.122:6800/3954553330' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[31360]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-12-15T17:17:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[40250]: osdmap e87: 8 total, 6 up, 8 in 2021-12-15T17:17:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[40250]: from='osd.4 v1:172.21.15.122:6800/3954553330' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:27 smithi022 ceph-mon[40250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:28.530 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:17:28.530 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:17:28.530 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: --> Failed to activate via raw: activate() got an unexpected keyword argument 'block_wal' 2021-12-15T17:17:28.531 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:17:28.531 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2021-12-15T17:17:28.531 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:17:28.531 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:17:28.531 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:17:28.532 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:17:28.532 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 bash[65693]: --> ceph-volume lvm activate successful for osd ID: 5 2021-12-15T17:17:28.532 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2021-12-15T17:17:28.532 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:17:28.533 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2021-12-15T17:17:28.533 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-12-15T17:17:28.533 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-12-15T17:17:28.533 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:28 smithi122 conmon[65885]: --> ceph-volume lvm activate successful for osd ID: 5 2021-12-15T17:17:29.239 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:29 smithi122 podman[66118]: 2021-12-15T17:17:29.239 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:28 smithi122 ceph-mon[32255]: osd.4 v1:172.21.15.122:6800/3954553330 boot 2021-12-15T17:17:29.240 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:28 smithi122 ceph-mon[32255]: osdmap e88: 8 total, 7 up, 8 in 2021-12-15T17:17:29.240 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:28 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:17:29.241 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:28 smithi122 ceph-mon[32255]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[31360]: osd.4 v1:172.21.15.122:6800/3954553330 boot 2021-12-15T17:17:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[31360]: osdmap e88: 8 total, 7 up, 8 in 2021-12-15T17:17:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:17:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[31360]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[40250]: osd.4 v1:172.21.15.122:6800/3954553330 boot 2021-12-15T17:17:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[40250]: osdmap e88: 8 total, 7 up, 8 in 2021-12-15T17:17:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:17:29.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:28 smithi022 ceph-mon[40250]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:29.427 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 7,6,4,5 on host 'smithi122' 2021-12-15T17:17:29.591 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:29 smithi122 bash[65693]: 6d0166350afcf3519f98ff7caa9366facec79346a4ac1b8f0ad3f02e89368f9f 2021-12-15T17:17:29.591 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:29 smithi122 systemd[1]: Started Ceph osd.5 for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:29.879 DEBUG:teuthology.orchestra.run.smithi122:osd.7> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@osd.7.service 2021-12-15T17:17:29.882 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2021-12-15T17:17:29.883 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd stat -f json 2021-12-15T17:17:29.909 INFO:journalctl@ceph.osd.7.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:17:30.185 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:29 smithi122 conmon[66158]: 2021-12-15T17:17:29.853+0000 7f2281f8a3c0 -1 Falling back to public interface 2021-12-15T17:17:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:29 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:29 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:29 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:29 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:29 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:29 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:30.944 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:30 smithi022 ceph-mon[40250]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:30 smithi022 ceph-mon[31360]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:31.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:30 smithi122 ceph-mon[32255]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:31.467 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:17:31.831 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":88,"num_osds":8,"num_up_osds":7,"osd_up_since":1639588647,"num_in_osds":8,"osd_in_since":1639588617,"num_remapped_pgs":0} 2021-12-15T17:17:32.582 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:32 smithi122 conmon[66158]: 2021-12-15T17:17:32.302+0000 7f2281f8a3c0 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2021-12-15T17:17:32.582 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:32 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:32.583 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2435840383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:32 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2435840383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:32 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2435840383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:32.832 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd stat -f json 2021-12-15T17:17:32.836 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:32 smithi122 conmon[66158]: 2021-12-15T17:17:32.587+0000 7f2281f8a3c0 -1 osd.5 85 log_to_monitors true 2021-12-15T17:17:33.640 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:33 smithi022 ceph-mon[31360]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:33.641 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:33 smithi022 ceph-mon[31360]: from='osd.5 v1:172.21.15.122:6804/443063154' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:17:33.641 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:33 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:17:33.642 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:33 smithi022 ceph-mon[40250]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:33.642 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:33 smithi022 ceph-mon[40250]: from='osd.5 v1:172.21.15.122:6804/443063154' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:17:33.642 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:33 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:17:33.715 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 15 17:17:33 smithi122 conmon[66158]: 2021-12-15T17:17:33.446+0000 7f2272b26700 -1 osd.5 85 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-12-15T17:17:33.715 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:33 smithi122 ceph-mon[32255]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:33.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:33 smithi122 ceph-mon[32255]: from='osd.5 v1:172.21.15.122:6804/443063154' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:17:33.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:33 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-12-15T17:17:34.377 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:17:34.734 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:17:34.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: osdmap e89: 8 total, 7 up, 8 in 2021-12-15T17:17:34.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='osd.5 v1:172.21.15.122:6804/443063154' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:34.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:34.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:34.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:34.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:34.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1272331789' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:34.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:17:34.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: osdmap e89: 8 total, 7 up, 8 in 2021-12-15T17:17:34.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='osd.5 v1:172.21.15.122:6804/443063154' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:34.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:34.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:34.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:34.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:34.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1272331789' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:34.769 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"num_osds":8,"num_up_osds":7,"osd_up_since":1639588647,"num_in_osds":8,"osd_in_since":1639588617,"num_remapped_pgs":0} 2021-12-15T17:17:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-12-15T17:17:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: osdmap e89: 8 total, 7 up, 8 in 2021-12-15T17:17:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='osd.5 v1:172.21.15.122:6804/443063154' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2021-12-15T17:17:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:17:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1272331789' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:35.770 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd stat -f json 2021-12-15T17:17:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: Detected new or changed devices on smithi122 2021-12-15T17:17:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 4293M 2021-12-15T17:17:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:17:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:17:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: osd.5 v1:172.21.15.122:6804/443063154 boot 2021-12-15T17:17:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: osdmap e90: 8 total, 8 up, 8 in 2021-12-15T17:17:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:17:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: Detected new or changed devices on smithi122 2021-12-15T17:17:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 4293M 2021-12-15T17:17:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:17:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:17:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: osd.5 v1:172.21.15.122:6804/443063154 boot 2021-12-15T17:17:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: osdmap e90: 8 total, 8 up, 8 in 2021-12-15T17:17:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:35 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:17:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: Detected new or changed devices on smithi122 2021-12-15T17:17:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 4293M 2021-12-15T17:17:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-12-15T17:17:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:17:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: osd.5 v1:172.21.15.122:6804/443063154 boot 2021-12-15T17:17:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: osdmap e90: 8 total, 8 up, 8 in 2021-12-15T17:17:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:35 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:17:37.328 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:17:37.686 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:37 smithi022 ceph-mon[31360]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:37.686 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2208557252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:37.687 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:37 smithi022 ceph-mon[40250]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:37.687 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2208557252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:37.714 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1639588654,"num_in_osds":8,"osd_in_since":1639588617,"num_remapped_pgs":0} 2021-12-15T17:17:37.714 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi022 2021-12-15T17:17:37.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply rgw foo.a --placement '1;smithi022=foo.a' 2021-12-15T17:17:37.746 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:37 smithi122 ceph-mon[32255]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:37.747 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2208557252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-12-15T17:17:39.359 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled rgw.foo.a update... 2021-12-15T17:17:39.712 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:39 smithi122 ceph-mon[32255]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:39.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:39 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:39.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:39 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:39.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:39 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:39.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:39 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:39.735 DEBUG:teuthology.orchestra.run.smithi022:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@rgw.foo.a.service 2021-12-15T17:17:39.738 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi122 2021-12-15T17:17:39.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd pool create datapool 3 3 replicated 2021-12-15T17:17:39.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[31360]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:39.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:39.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:39.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:39.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:39.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[40250]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:39.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:39.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:39.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:39.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:39 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:39.766 INFO:journalctl@ceph.rgw.foo.a.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:17:41.010 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: from='client.34430 v1:172.21.15.122:0/310765241' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi022=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:41.011 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: Saving service rgw.foo.a spec with placement smithi022=foo.a;count:1 2021-12-15T17:17:41.011 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.011 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.011 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-12-15T17:17:41.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-12-15T17:17:41.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:41.013 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: from='client.34430 v1:172.21.15.122:0/310765241' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi022=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:41.013 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: Saving service rgw.foo.a spec with placement smithi022=foo.a;count:1 2021-12-15T17:17:41.013 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.013 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.014 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-12-15T17:17:41.014 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-12-15T17:17:41.014 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.014 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:40 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:41.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: from='client.34430 v1:172.21.15.122:0/310765241' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi022=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:41.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: Saving service rgw.foo.a spec with placement smithi022=foo.a;count:1 2021-12-15T17:17:41.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.066 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.066 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-12-15T17:17:41.066 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-12-15T17:17:41.066 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:41.067 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:40 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:41.278 INFO:journalctl@ceph.rgw.foo.a.smithi022.stdout:Dec 15 17:17:41 smithi022 systemd[1]: Starting Ceph rgw.foo.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:17:41.752 INFO:journalctl@ceph.rgw.foo.a.smithi022.stdout:Dec 15 17:17:41 smithi022 podman[77049]: 2021-12-15T17:17:41.753 INFO:journalctl@ceph.rgw.foo.a.smithi022.stdout:Dec 15 17:17:41 smithi022 bash[76896]: 7b52768d7fd3d433f09d1f07e502066e3af7b957660b90ecd6733b7081d9eaf7 2021-12-15T17:17:42.028 INFO:journalctl@ceph.rgw.foo.a.smithi022.stdout:Dec 15 17:17:41 smithi022 systemd[1]: Started Ceph rgw.foo.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:42.403 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'datapool' created 2021-12-15T17:17:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:42 smithi122 ceph-mon[32255]: Saving service rgw.foo.a spec with placement smithi022=foo.a;count:1 2021-12-15T17:17:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:42 smithi122 ceph-mon[32255]: Deploying daemon rgw.foo.a on smithi022 2021-12-15T17:17:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:42 smithi122 ceph-mon[32255]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:42 smithi022 ceph-mon[31360]: Saving service rgw.foo.a spec with placement smithi022=foo.a;count:1 2021-12-15T17:17:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:42 smithi022 ceph-mon[31360]: Deploying daemon rgw.foo.a on smithi022 2021-12-15T17:17:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:42 smithi022 ceph-mon[31360]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:42 smithi022 ceph-mon[40250]: Saving service rgw.foo.a spec with placement smithi022=foo.a;count:1 2021-12-15T17:17:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:42 smithi022 ceph-mon[40250]: Deploying daemon rgw.foo.a on smithi022 2021-12-15T17:17:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:42 smithi022 ceph-mon[40250]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:42.802 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- rbd pool init datapool 2021-12-15T17:17:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3929235961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-12-15T17:17:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='client.34436 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-12-15T17:17:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='client.34436 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-12-15T17:17:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: osdmap e91: 8 total, 8 up, 8 in 2021-12-15T17:17:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-12-15T17:17:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-12-15T17:17:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:43.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:43 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3929235961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-12-15T17:17:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='client.34436 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-12-15T17:17:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='client.34436 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-12-15T17:17:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: osdmap e91: 8 total, 8 up, 8 in 2021-12-15T17:17:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-12-15T17:17:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-12-15T17:17:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:43.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3929235961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-12-15T17:17:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='client.34436 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-12-15T17:17:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:17:43.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:43.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='client.34436 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-12-15T17:17:43.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: osdmap e91: 8 total, 8 up, 8 in 2021-12-15T17:17:43.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:17:43.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-12-15T17:17:43.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-12-15T17:17:43.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:43.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:43 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:17:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:44 smithi122 ceph-mon[32255]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-12-15T17:17:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:44 smithi122 ceph-mon[32255]: osdmap e92: 8 total, 8 up, 8 in 2021-12-15T17:17:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/366031533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2021-12-15T17:17:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:44 smithi122 ceph-mon[32255]: from='client.34451 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2021-12-15T17:17:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[31360]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-12-15T17:17:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[31360]: osdmap e92: 8 total, 8 up, 8 in 2021-12-15T17:17:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/366031533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2021-12-15T17:17:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[31360]: from='client.34451 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2021-12-15T17:17:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[40250]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-12-15T17:17:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[40250]: osdmap e92: 8 total, 8 up, 8 in 2021-12-15T17:17:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/366031533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2021-12-15T17:17:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:44 smithi022 ceph-mon[40250]: from='client.34451 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2021-12-15T17:17:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:45 smithi122 ceph-mon[32255]: pgmap v225: 36 pgs: 15 active+clean, 6 creating+peering, 15 unknown; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-12-15T17:17:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:45 smithi122 ceph-mon[32255]: from='client.34451 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2021-12-15T17:17:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:45 smithi122 ceph-mon[32255]: osdmap e93: 8 total, 8 up, 8 in 2021-12-15T17:17:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-12-15T17:17:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:45 smithi122 ceph-mon[32255]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-12-15T17:17:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:45 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:45 smithi122 ceph-mon[32255]: Checking dashboard <-> RGW credentials 2021-12-15T17:17:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[31360]: pgmap v225: 36 pgs: 15 active+clean, 6 creating+peering, 15 unknown; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-12-15T17:17:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[31360]: from='client.34451 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2021-12-15T17:17:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[31360]: osdmap e93: 8 total, 8 up, 8 in 2021-12-15T17:17:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-12-15T17:17:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[31360]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-12-15T17:17:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[31360]: Checking dashboard <-> RGW credentials 2021-12-15T17:17:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[40250]: pgmap v225: 36 pgs: 15 active+clean, 6 creating+peering, 15 unknown; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-12-15T17:17:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[40250]: from='client.34451 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2021-12-15T17:17:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[40250]: osdmap e93: 8 total, 8 up, 8 in 2021-12-15T17:17:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-12-15T17:17:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[40250]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-12-15T17:17:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:45 smithi022 ceph-mon[40250]: Checking dashboard <-> RGW credentials 2021-12-15T17:17:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:46 smithi122 ceph-mon[32255]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-12-15T17:17:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:46 smithi122 ceph-mon[32255]: osdmap e94: 8 total, 8 up, 8 in 2021-12-15T17:17:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:46 smithi022 ceph-mon[31360]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-12-15T17:17:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:46 smithi022 ceph-mon[31360]: osdmap e94: 8 total, 8 up, 8 in 2021-12-15T17:17:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:46 smithi022 ceph-mon[40250]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-12-15T17:17:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:46 smithi022 ceph-mon[40250]: osdmap e94: 8 total, 8 up, 8 in 2021-12-15T17:17:46.816 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.122 --placement '1;smithi122=iscsi.a' 2021-12-15T17:17:47.515 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:47 smithi122 ceph-mon[32255]: pgmap v228: 68 pgs: 34 active+clean, 11 creating+peering, 23 unknown; 577 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2021-12-15T17:17:47.516 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:47 smithi122 ceph-mon[32255]: osdmap e95: 8 total, 8 up, 8 in 2021-12-15T17:17:47.516 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/423612080' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.516 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:47 smithi122 ceph-mon[32255]: from='client.24466 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.516 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.516 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:47 smithi122 ceph-mon[32255]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.517 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:47 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[31360]: pgmap v228: 68 pgs: 34 active+clean, 11 creating+peering, 23 unknown; 577 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2021-12-15T17:17:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[31360]: osdmap e95: 8 total, 8 up, 8 in 2021-12-15T17:17:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/423612080' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[31360]: from='client.24466 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[31360]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[40250]: pgmap v228: 68 pgs: 34 active+clean, 11 creating+peering, 23 unknown; 577 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2021-12-15T17:17:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[40250]: osdmap e95: 8 total, 8 up, 8 in 2021-12-15T17:17:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/423612080' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[40250]: from='client.24466 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2258093383' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[40250]: from='client.24439 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-12-15T17:17:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:47 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:48.466 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled iscsi.datapool update... 2021-12-15T17:17:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:48 smithi122 ceph-mon[32255]: from='client.24466 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-12-15T17:17:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:48 smithi122 ceph-mon[32255]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-12-15T17:17:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:48 smithi122 ceph-mon[32255]: osdmap e96: 8 total, 8 up, 8 in 2021-12-15T17:17:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:48 smithi022 ceph-mon[31360]: from='client.24466 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-12-15T17:17:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:48 smithi022 ceph-mon[31360]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-12-15T17:17:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:48 smithi022 ceph-mon[31360]: osdmap e96: 8 total, 8 up, 8 in 2021-12-15T17:17:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:48 smithi022 ceph-mon[40250]: from='client.24466 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-12-15T17:17:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:48 smithi022 ceph-mon[40250]: from='client.24439 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-12-15T17:17:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:48 smithi022 ceph-mon[40250]: osdmap e96: 8 total, 8 up, 8 in 2021-12-15T17:17:48.836 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2021-12-15T17:17:48.837 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:17:48.837 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2021-12-15T17:17:48.870 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:17:48.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2021-12-15T17:17:48.900 DEBUG:teuthology.orchestra.run.smithi122:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@iscsi.iscsi.a.service 2021-12-15T17:17:48.942 INFO:tasks.cephadm:Adding prometheus.a on smithi122 2021-12-15T17:17:48.943 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply prometheus '1;smithi122=a' 2021-12-15T17:17:48.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:17:49.639 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: pgmap v231: 100 pgs: 64 active+clean, 9 creating+peering, 27 unknown; 578 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 2 op/s 2021-12-15T17:17:49.639 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: osdmap e97: 8 total, 8 up, 8 in 2021-12-15T17:17:49.639 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.640 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.640 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.640 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.640 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: from='client.34493 v1:172.21.15.122:0/786758068' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.122", "placement": "1;smithi122=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:49.640 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: Saving service iscsi.datapool spec with placement smithi122=iscsi.a;count:1 2021-12-15T17:17:49.641 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:49 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: pgmap v231: 100 pgs: 64 active+clean, 9 creating+peering, 27 unknown; 578 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 2 op/s 2021-12-15T17:17:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: osdmap e97: 8 total, 8 up, 8 in 2021-12-15T17:17:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: from='client.34493 v1:172.21.15.122:0/786758068' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.122", "placement": "1;smithi122=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: Saving service iscsi.datapool spec with placement smithi122=iscsi.a;count:1 2021-12-15T17:17:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: pgmap v231: 100 pgs: 64 active+clean, 9 creating+peering, 27 unknown; 578 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 2 op/s 2021-12-15T17:17:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: osdmap e97: 8 total, 8 up, 8 in 2021-12-15T17:17:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-12-15T17:17:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: from='client.34493 v1:172.21.15.122:0/786758068' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.122", "placement": "1;smithi122=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: Saving service iscsi.datapool spec with placement smithi122=iscsi.a;count:1 2021-12-15T17:17:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:49 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:50.621 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled prometheus update... 2021-12-15T17:17:50.621 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-12-15T17:17:50.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-12-15T17:17:50.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: osdmap e98: 8 total, 8 up, 8 in 2021-12-15T17:17:50.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-12-15T17:17:50.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-12-15T17:17:50.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: osdmap e99: 8 total, 8 up, 8 in 2021-12-15T17:17:50.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:50 smithi122 ceph-mon[32255]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-12-15T17:17:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-12-15T17:17:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: osdmap e98: 8 total, 8 up, 8 in 2021-12-15T17:17:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-12-15T17:17:50.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-12-15T17:17:50.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: osdmap e99: 8 total, 8 up, 8 in 2021-12-15T17:17:50.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[31360]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-12-15T17:17:50.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-12-15T17:17:50.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: osdmap e98: 8 total, 8 up, 8 in 2021-12-15T17:17:50.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-12-15T17:17:50.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-12-15T17:17:50.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-12-15T17:17:50.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: osdmap e99: 8 total, 8 up, 8 in 2021-12-15T17:17:50.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1032445210' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.24475 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/623030770' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:50 smithi022 ceph-mon[40250]: from='client.34487 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-12-15T17:17:50.967 DEBUG:teuthology.orchestra.run.smithi122:prometheus.a> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@prometheus.a.service 2021-12-15T17:17:50.970 INFO:tasks.cephadm:Adding node-exporter.a on smithi022 2021-12-15T17:17:50.970 INFO:tasks.cephadm:Adding node-exporter.b on smithi122 2021-12-15T17:17:50.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply node-exporter '2;smithi022=a;smithi122=b' 2021-12-15T17:17:50.998 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:17:51.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: pgmap v234: 132 pgs: 94 active+clean, 16 creating+peering, 22 unknown; 578 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 3 op/s 2021-12-15T17:17:51.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: from='client.24487 v1:172.21.15.122:0/2756555944' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:51.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: Saving service prometheus spec with placement smithi122=a;count:1 2021-12-15T17:17:51.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:51.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-12-15T17:17:51.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-12-15T17:17:51.667 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: osdmap e100: 8 total, 8 up, 8 in 2021-12-15T17:17:51.667 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:51.667 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:51.667 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:51 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: pgmap v234: 132 pgs: 94 active+clean, 16 creating+peering, 22 unknown; 578 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 3 op/s 2021-12-15T17:17:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: from='client.24487 v1:172.21.15.122:0/2756555944' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: Saving service prometheus spec with placement smithi122=a;count:1 2021-12-15T17:17:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-12-15T17:17:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-12-15T17:17:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: osdmap e100: 8 total, 8 up, 8 in 2021-12-15T17:17:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: pgmap v234: 132 pgs: 94 active+clean, 16 creating+peering, 22 unknown; 578 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 3 op/s 2021-12-15T17:17:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: from='client.24487 v1:172.21.15.122:0/2756555944' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: Saving service prometheus spec with placement smithi122=a;count:1 2021-12-15T17:17:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: from='client.24475 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-12-15T17:17:52.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: from='client.34487 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-12-15T17:17:52.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: osdmap e100: 8 total, 8 up, 8 in 2021-12-15T17:17:52.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:51 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:52.670 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled node-exporter update... 2021-12-15T17:17:52.753 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:52 smithi122 ceph-mon[32255]: Checking pool "datapool" exists for service iscsi.datapool 2021-12-15T17:17:52.890 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:52 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:17:52.891 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:52 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-12-15T17:17:52.891 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:52 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:52.892 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:52 smithi122 ceph-mon[32255]: Deploying daemon iscsi.iscsi.a on smithi122 2021-12-15T17:17:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[31360]: Checking pool "datapool" exists for service iscsi.datapool 2021-12-15T17:17:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:17:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-12-15T17:17:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[31360]: Deploying daemon iscsi.iscsi.a on smithi122 2021-12-15T17:17:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[40250]: Checking pool "datapool" exists for service iscsi.datapool 2021-12-15T17:17:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:17:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-12-15T17:17:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:17:53.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:52 smithi022 ceph-mon[40250]: Deploying daemon iscsi.iscsi.a on smithi122 2021-12-15T17:17:53.202 DEBUG:teuthology.orchestra.run.smithi022:node-exporter.a> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@node-exporter.a.service 2021-12-15T17:17:53.205 DEBUG:teuthology.orchestra.run.smithi122:node-exporter.b> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@node-exporter.b.service 2021-12-15T17:17:53.208 INFO:tasks.cephadm:Adding alertmanager.a on smithi022 2021-12-15T17:17:53.209 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply alertmanager '1;smithi022=a' 2021-12-15T17:17:53.237 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:17:53.238 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:17:53.913 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:53 smithi122 ceph-mon[32255]: pgmap v237: 132 pgs: 113 active+clean, 12 creating+peering, 7 unknown; 579 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.2 KiB/s wr, 75 op/s 2021-12-15T17:17:53.913 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:53 smithi122 ceph-mon[32255]: from='client.34520 v1:172.21.15.122:0/1258499550' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi022=a;smithi122=b", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:53.913 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:53 smithi122 ceph-mon[32255]: Saving service node-exporter spec with placement smithi022=a;smithi122=b;count:2 2021-12-15T17:17:53.914 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:53 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[31360]: pgmap v237: 132 pgs: 113 active+clean, 12 creating+peering, 7 unknown; 579 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.2 KiB/s wr, 75 op/s 2021-12-15T17:17:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[31360]: from='client.34520 v1:172.21.15.122:0/1258499550' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi022=a;smithi122=b", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[31360]: Saving service node-exporter spec with placement smithi022=a;smithi122=b;count:2 2021-12-15T17:17:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[40250]: pgmap v237: 132 pgs: 113 active+clean, 12 creating+peering, 7 unknown; 579 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.2 KiB/s wr, 75 op/s 2021-12-15T17:17:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[40250]: from='client.34520 v1:172.21.15.122:0/1258499550' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi022=a;smithi122=b", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[40250]: Saving service node-exporter spec with placement smithi022=a;smithi122=b;count:2 2021-12-15T17:17:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:53 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:54.169 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:54 smithi122 podman[70123]: 2021-12-15T17:17:54.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:54 smithi122 conmon[70277]: log file path now is '/var/log/tcmu-runner.log' 2021-12-15T17:17:54.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:54 smithi122 bash[69936]: log file path now is '/var/log/tcmu-runner.log' 2021-12-15T17:17:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:54 smithi122 podman[70409]: 2021-12-15T17:17:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:54 smithi122 ceph-mon[32255]: mgrmap e16: y(active, since 4m), standbys: x 2021-12-15T17:17:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:54 smithi022 ceph-mon[31360]: mgrmap e16: y(active, since 4m), standbys: x 2021-12-15T17:17:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:54 smithi022 ceph-mon[40250]: mgrmap e16: y(active, since 4m), standbys: x 2021-12-15T17:17:55.393 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled alertmanager update... 2021-12-15T17:17:55.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 bash[69936]: a55ee4390bcecbc6941d93abf8723118aa65dd295b5747d65fa31a8eb1998af3 2021-12-15T17:17:55.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 systemd[1]: Started Ceph iscsi.iscsi.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:17:55.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 conmon[70494]: debug Started the configuration object watcher 2021-12-15T17:17:55.395 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 conmon[70494]: debug Checking for config object changes every 1s 2021-12-15T17:17:55.395 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 conmon[70494]: debug Processing osd blocklist entries for this node 2021-12-15T17:17:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 conmon[70494]: debug Reading the configuration object to update local LIO configuration 2021-12-15T17:17:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 conmon[70494]: debug Configuration does not have an entry for this host(smithi122) - nothing to define to LIO 2021-12-15T17:17:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:17:55 smithi122 conmon[70494]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2021-12-15T17:17:56.017 DEBUG:teuthology.orchestra.run.smithi022:alertmanager.a> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@alertmanager.a.service 2021-12-15T17:17:56.020 INFO:tasks.cephadm:Adding grafana.a on smithi122 2021-12-15T17:17:56.021 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph orch apply grafana '1;smithi122=a' 2021-12-15T17:17:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[40250]: pgmap v238: 132 pgs: 132 active+clean; 580 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 4.0 KiB/s wr, 111 op/s 2021-12-15T17:17:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[40250]: osdmap e101: 8 total, 8 up, 8 in 2021-12-15T17:17:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-12-15T17:17:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/2542306566' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-12-15T17:17:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[31360]: pgmap v238: 132 pgs: 132 active+clean; 580 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 4.0 KiB/s wr, 111 op/s 2021-12-15T17:17:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[31360]: osdmap e101: 8 total, 8 up, 8 in 2021-12-15T17:17:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-12-15T17:17:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:56.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/2542306566' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-12-15T17:17:56.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:55 smithi122 ceph-mon[32255]: pgmap v238: 132 pgs: 132 active+clean; 580 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 4.0 KiB/s wr, 111 op/s 2021-12-15T17:17:56.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:55 smithi122 ceph-mon[32255]: osdmap e101: 8 total, 8 up, 8 in 2021-12-15T17:17:56.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:56.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-12-15T17:17:56.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:55 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' 2021-12-15T17:17:56.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/2542306566' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-12-15T17:17:56.051 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:-- Logs begin at Wed 2021-12-15 17:02:36 UTC. -- 2021-12-15T17:17:56.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:56 smithi122 conmon[33927]: ignoring --setuser ceph since I am not root 2021-12-15T17:17:56.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:56 smithi122 conmon[33927]: ignoring --setgroup ceph since I am not root 2021-12-15T17:17:56.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:56 smithi122 conmon[33927]: 2021-12-15T17:17:56.275+0000 7f113149a180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2021-12-15T17:17:56.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:56 smithi022 conmon[31781]: ignoring --setuser ceph since I am not root 2021-12-15T17:17:56.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:56 smithi022 conmon[31781]: ignoring --setgroup ceph since I am not root 2021-12-15T17:17:56.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:56 smithi022 conmon[31781]: 2021-12-15T17:17:56.283+0000 7fa17baa8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2021-12-15T17:17:57.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:56 smithi022 conmon[31781]: 2021-12-15T17:17:56.830+0000 7fa17baa8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2021-12-15T17:17:57.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:56 smithi022 conmon[31781]: 2021-12-15T17:17:56.950+0000 7fa17baa8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2021-12-15T17:17:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[31360]: from='client.34544 v1:172.21.15.122:0/3066030047' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi022=a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[31360]: Saving service alertmanager spec with placement smithi022=a;count:1 2021-12-15T17:17:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[31360]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-12-15T17:17:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[31360]: mgrmap e17: y(active, since 4m), standbys: x 2021-12-15T17:17:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[40250]: from='client.34544 v1:172.21.15.122:0/3066030047' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi022=a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[40250]: Saving service alertmanager spec with placement smithi022=a;count:1 2021-12-15T17:17:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[40250]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-12-15T17:17:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:17:56 smithi022 ceph-mon[40250]: mgrmap e17: y(active, since 4m), standbys: x 2021-12-15T17:17:57.034 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:56 smithi122 conmon[33927]: 2021-12-15T17:17:56.837+0000 7f113149a180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2021-12-15T17:17:57.034 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:56 smithi122 conmon[33927]: 2021-12-15T17:17:56.955+0000 7f113149a180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2021-12-15T17:17:57.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:56 smithi122 ceph-mon[32255]: from='client.34544 v1:172.21.15.122:0/3066030047' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi022=a", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:17:57.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:56 smithi122 ceph-mon[32255]: Saving service alertmanager spec with placement smithi022=a;count:1 2021-12-15T17:17:57.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:56 smithi122 ceph-mon[32255]: from='mgr.14152 v1:172.21.15.22:0/1197381774' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-12-15T17:17:57.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:17:56 smithi122 ceph-mon[32255]: mgrmap e17: y(active, since 4m), standbys: x 2021-12-15T17:17:57.302 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:57 smithi122 conmon[33927]: 2021-12-15T17:17:57.034+0000 7f113149a180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2021-12-15T17:17:57.303 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:57 smithi122 conmon[33927]: 2021-12-15T17:17:57.133+0000 7f113149a180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2021-12-15T17:17:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:57 smithi022 conmon[31781]: 2021-12-15T17:17:57.029+0000 7fa17baa8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2021-12-15T17:17:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:57 smithi022 conmon[31781]: 2021-12-15T17:17:57.119+0000 7fa17baa8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2021-12-15T17:17:58.184 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:57 smithi122 conmon[33927]: 2021-12-15T17:17:57.868+0000 7f113149a180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:17:58.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:57 smithi022 conmon[31781]: 2021-12-15T17:17:57.850+0000 7fa17baa8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:17:58.641 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:58 smithi122 conmon[33927]: 2021-12-15T17:17:58.390+0000 7f113149a180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2021-12-15T17:17:58.642 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:58 smithi122 conmon[33927]: 2021-12-15T17:17:58.475+0000 7f113149a180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2021-12-15T17:17:58.642 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:58 smithi122 conmon[33927]: 2021-12-15T17:17:58.642+0000 7f113149a180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2021-12-15T17:17:58.677 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:58 smithi022 conmon[31781]: 2021-12-15T17:17:58.412+0000 7fa17baa8180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2021-12-15T17:17:58.678 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:58 smithi022 conmon[31781]: 2021-12-15T17:17:58.503+0000 7fa17baa8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2021-12-15T17:17:58.935 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:58 smithi122 conmon[33927]: 2021-12-15T17:17:58.728+0000 7f113149a180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2021-12-15T17:17:58.935 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:58 smithi122 conmon[33927]: 2021-12-15T17:17:58.815+0000 7f113149a180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2021-12-15T17:17:59.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:58 smithi022 conmon[31781]: 2021-12-15T17:17:58.675+0000 7fa17baa8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2021-12-15T17:17:59.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:58 smithi022 conmon[31781]: 2021-12-15T17:17:58.765+0000 7fa17baa8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2021-12-15T17:17:59.029 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:58 smithi022 conmon[31781]: 2021-12-15T17:17:58.858+0000 7fa17baa8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2021-12-15T17:17:59.248 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:58 smithi122 conmon[33927]: 2021-12-15T17:17:58.983+0000 7f113149a180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2021-12-15T17:17:59.314 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:59 smithi022 conmon[31781]: 2021-12-15T17:17:59.034+0000 7fa17baa8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2021-12-15T17:17:59.548 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:59 smithi122 conmon[33927]: 2021-12-15T17:17:59.248+0000 7f113149a180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2021-12-15T17:17:59.548 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:59 smithi122 conmon[33927]: 2021-12-15T17:17:59.374+0000 7f113149a180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2021-12-15T17:17:59.548 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:59 smithi122 conmon[33927]: 2021-12-15T17:17:59.462+0000 7f113149a180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2021-12-15T17:17:59.622 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:59 smithi022 conmon[31781]: 2021-12-15T17:17:59.314+0000 7fa17baa8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2021-12-15T17:17:59.622 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:59 smithi022 conmon[31781]: 2021-12-15T17:17:59.447+0000 7fa17baa8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2021-12-15T17:17:59.622 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:59 smithi022 conmon[31781]: 2021-12-15T17:17:59.535+0000 7fa17baa8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2021-12-15T17:18:00.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:17:59 smithi022 conmon[31781]: 2021-12-15T17:17:59.621+0000 7fa17baa8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2021-12-15T17:18:00.633 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:17:59 smithi122 conmon[33927]: 2021-12-15T17:17:59.547+0000 7f113149a180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2021-12-15T17:18:00.634 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:00 smithi122 conmon[33927]: 2021-12-15T17:18:00.297+0000 7f113149a180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2021-12-15T17:18:00.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:00 smithi022 conmon[31781]: 2021-12-15T17:18:00.415+0000 7fa17baa8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2021-12-15T17:18:00.907 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:00 smithi122 conmon[33927]: 2021-12-15T17:18:00.634+0000 7f113149a180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:18:01.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:00 smithi022 conmon[31781]: 2021-12-15T17:18:00.774+0000 7fa17baa8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:18:01.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:00 smithi122 conmon[33927]: 2021-12-15T17:18:00.908+0000 7f113149a180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2021-12-15T17:18:01.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:01 smithi122 conmon[33927]: 2021-12-15T17:18:01.048+0000 7f113149a180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2021-12-15T17:18:01.514 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:01 smithi022 conmon[31781]: 2021-12-15T17:18:01.082+0000 7fa17baa8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2021-12-15T17:18:01.514 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:01 smithi022 conmon[31781]: 2021-12-15T17:18:01.230+0000 7fa17baa8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2021-12-15T17:18:01.685 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:01 smithi122 conmon[33927]: 2021-12-15T17:18:01.316+0000 7f113149a180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2021-12-15T17:18:01.685 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:01 smithi122 conmon[33927]: 2021-12-15T17:18:01.401+0000 7f113149a180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2021-12-15T17:18:01.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:01 smithi022 conmon[31781]: 2021-12-15T17:18:01.514+0000 7fa17baa8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2021-12-15T17:18:01.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:01 smithi022 conmon[31781]: 2021-12-15T17:18:01.604+0000 7fa17baa8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2021-12-15T17:18:02.031 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:01 smithi122 conmon[33927]: 2021-12-15T17:18:01.736+0000 7f113149a180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2021-12-15T17:18:02.267 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:01 smithi022 conmon[31781]: 2021-12-15T17:18:01.956+0000 7fa17baa8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2021-12-15T17:18:02.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[31360]: Standby manager daemon x restarted 2021-12-15T17:18:02.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[31360]: Standby manager daemon x started 2021-12-15T17:18:02.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-12-15T17:18:02.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:18:02.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-12-15T17:18:02.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:18:02.269 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[40250]: Standby manager daemon x restarted 2021-12-15T17:18:02.270 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[40250]: Standby manager daemon x started 2021-12-15T17:18:02.271 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-12-15T17:18:02.272 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:18:02.272 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-12-15T17:18:02.272 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:02 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:18:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:02 smithi122 ceph-mon[32255]: Standby manager daemon x restarted 2021-12-15T17:18:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:02 smithi122 ceph-mon[32255]: Standby manager daemon x started 2021-12-15T17:18:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:02 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-12-15T17:18:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:02 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:18:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:02 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-12-15T17:18:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:02 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:18:02.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:02 smithi122 conmon[33927]: 2021-12-15T17:18:02.031+0000 7f113149a180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2021-12-15T17:18:02.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:02 smithi122 conmon[33927]: [15/Dec/2021:17:18:02] ENGINE Bus STARTING 2021-12-15T17:18:02.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:02 smithi122 conmon[33927]: CherryPy Checker: 2021-12-15T17:18:02.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:02 smithi122 conmon[33927]: The Application mounted at '' has an empty config. 2021-12-15T17:18:02.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:02 smithi122 conmon[33927]: 2021-12-15T17:18:02.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:02 smithi122 conmon[33927]: [15/Dec/2021:17:18:02] ENGINE Serving on http://:::9283 2021-12-15T17:18:02.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:18:02 smithi122 conmon[33927]: [15/Dec/2021:17:18:02] ENGINE Bus STARTED 2021-12-15T17:18:02.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:02 smithi022 conmon[31781]: 2021-12-15T17:18:02.268+0000 7fa17baa8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2021-12-15T17:18:03.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:03 smithi122 ceph-mon[32255]: mgrmap e18: y(active, since 5m), standbys: x 2021-12-15T17:18:03.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:03 smithi122 ceph-mon[32255]: Active manager daemon y restarted 2021-12-15T17:18:03.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:03 smithi122 ceph-mon[32255]: Activating manager daemon y 2021-12-15T17:18:03.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:03 smithi122 ceph-mon[32255]: osdmap e102: 8 total, 8 up, 8 in 2021-12-15T17:18:03.443 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:03 smithi022 conmon[31781]: [15/Dec/2021:17:18:03] ENGINE Bus STARTING 2021-12-15T17:18:03.443 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:03 smithi022 conmon[31781]: CherryPy Checker: 2021-12-15T17:18:03.444 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:03 smithi022 conmon[31781]: The Application mounted at '' has an empty config. 2021-12-15T17:18:03.444 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:03 smithi022 conmon[31781]: 2021-12-15T17:18:03.444 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:03 smithi022 conmon[31781]: [15/Dec/2021:17:18:03] ENGINE Serving on http://:::9283 2021-12-15T17:18:03.444 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:03 smithi022 conmon[31781]: [15/Dec/2021:17:18:03] ENGINE Bus STARTED 2021-12-15T17:18:03.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[31360]: mgrmap e18: y(active, since 5m), standbys: x 2021-12-15T17:18:03.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[31360]: Active manager daemon y restarted 2021-12-15T17:18:03.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[31360]: Activating manager daemon y 2021-12-15T17:18:03.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[31360]: osdmap e102: 8 total, 8 up, 8 in 2021-12-15T17:18:03.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[40250]: mgrmap e18: y(active, since 5m), standbys: x 2021-12-15T17:18:03.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[40250]: Active manager daemon y restarted 2021-12-15T17:18:03.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[40250]: Activating manager daemon y 2021-12-15T17:18:03.447 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:03 smithi022 ceph-mon[40250]: osdmap e102: 8 total, 8 up, 8 in 2021-12-15T17:18:04.134 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled grafana update... 2021-12-15T17:18:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: mgrmap e19: y(active, starting, since 0.828416s), standbys: x 2021-12-15T17:18:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:18:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:18:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:18:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:18:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:18:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:18:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:18:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:18:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:18:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:18:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:18:04.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:18:04.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:18:04.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:18:04.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:18:04.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:18:04.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: Manager daemon y is now available 2021-12-15T17:18:04.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:18:04.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:18:04.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:18:04.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:18:04.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:04.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:18:04.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:18:04.471 DEBUG:teuthology.orchestra.run.smithi122:grafana.a> sudo journalctl -f -n 0 -u ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@grafana.a.service 2021-12-15T17:18:04.513 INFO:tasks.cephadm:Setting up client nodes... 2021-12-15T17:18:04.514 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-12-15T17:18:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: mgrmap e19: y(active, starting, since 0.828416s), standbys: x 2021-12-15T17:18:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:18:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:18:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:18:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:18:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:18:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:18:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:18:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:18:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:18:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:18:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:18:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:18:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:18:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:18:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:18:04.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:18:04.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: Manager daemon y is now available 2021-12-15T17:18:04.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:18:04.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:18:04.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:18:04.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:18:04.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:04.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:18:04.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:18:04.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: mgrmap e19: y(active, starting, since 0.828416s), standbys: x 2021-12-15T17:18:04.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:18:04.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:18:04.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:18:04.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:18:04.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:18:04.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:18:04.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:18:04.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:18:04.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:18:04.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:18:04.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:18:04.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:18:04.537 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:18:04.537 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:18:04.537 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:18:04.537 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:18:04.538 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: Manager daemon y is now available 2021-12-15T17:18:04.538 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:18:04.538 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:18:04.538 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:18:04.538 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:18:04.539 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:04.539 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:18:04.539 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:18:04.540 INFO:journalctl@ceph.grafana.a.smithi122.stdout:-- Logs begin at Wed 2021-12-15 17:03:01 UTC. -- 2021-12-15T17:18:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[31360]: mgrmap e20: y(active, since 1.84094s), standbys: x 2021-12-15T17:18:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[31360]: Saving service grafana spec with placement smithi122=a;count:1 2021-12-15T17:18:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[31360]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[40250]: mgrmap e20: y(active, since 1.84094s), standbys: x 2021-12-15T17:18:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[40250]: Saving service grafana spec with placement smithi122=a;count:1 2021-12-15T17:18:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[40250]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:05 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:05.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:18:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:18:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:05 smithi122 ceph-mon[32255]: mgrmap e20: y(active, since 1.84094s), standbys: x 2021-12-15T17:18:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:05 smithi122 ceph-mon[32255]: Saving service grafana spec with placement smithi122=a;count:1 2021-12-15T17:18:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:05 smithi122 ceph-mon[32255]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:05 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:05.709 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:05 smithi022 conmon[31781]: [15/Dec/2021:17:18:05] ENGINE Bus STARTING 2021-12-15T17:18:05.710 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:05 smithi022 conmon[31781]: [15/Dec/2021:17:18:05] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:18:05.710 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:05 smithi022 conmon[31781]: [15/Dec/2021:17:18:05] ENGINE Bus STARTED 2021-12-15T17:18:06.246 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[31360]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:06.247 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[31360]: mgrmap e21: y(active, since 2s), standbys: x 2021-12-15T17:18:06.247 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:06.247 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:06.247 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:06.248 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2021-12-15T17:18:06.248 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBOI7phZNuEDhAAUcAR05Ve9ka7wJ7LHCCiFQ== 2021-12-15T17:18:06.249 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[40250]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:06.249 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[40250]: mgrmap e21: y(active, since 2s), standbys: x 2021-12-15T17:18:06.249 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:06.249 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:06.249 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:06 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:06 smithi122 ceph-mon[32255]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:06 smithi122 ceph-mon[32255]: mgrmap e21: y(active, since 2s), standbys: x 2021-12-15T17:18:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:06 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:06 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:06.543 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2021-12-15T17:18:06.543 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-12-15T17:18:06.544 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-12-15T17:18:06.624 DEBUG:teuthology.orchestra.run.smithi122:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-12-15T17:18:07.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: [15/Dec/2021:17:18:05] ENGINE Bus STARTING 2021-12-15T17:18:07.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: [15/Dec/2021:17:18:05] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:18:07.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: [15/Dec/2021:17:18:05] ENGINE Bus STARTED 2021-12-15T17:18:07.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4208600232' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:07.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: from='client.24583 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:07.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: from='client.24583 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-12-15T17:18:07.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:07.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:07 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:07.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: [15/Dec/2021:17:18:05] ENGINE Bus STARTING 2021-12-15T17:18:07.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: [15/Dec/2021:17:18:05] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:18:07.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: [15/Dec/2021:17:18:05] ENGINE Bus STARTED 2021-12-15T17:18:07.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4208600232' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:07.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: from='client.24583 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: from='client.24583 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-12-15T17:18:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:07.399 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: [15/Dec/2021:17:18:05] ENGINE Bus STARTING 2021-12-15T17:18:07.400 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: [15/Dec/2021:17:18:05] ENGINE Serving on https://172.21.15.22:7150 2021-12-15T17:18:07.400 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: [15/Dec/2021:17:18:05] ENGINE Bus STARTED 2021-12-15T17:18:07.400 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4208600232' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:07.400 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: from='client.24583 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:07.401 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: from='client.24583 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-12-15T17:18:07.401 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:07.401 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:07 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:08.241 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:08 smithi122 ceph-mon[32255]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:08 smithi022 ceph-mon[31360]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:08 smithi022 ceph-mon[40250]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:08.752 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2021-12-15T17:18:08.753 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBQI7phET6zLBAA8Lo27XhKfWmvJRspQpraRg== 2021-12-15T17:18:09.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3776322880' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:09.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:09 smithi022 ceph-mon[31360]: from='client.34604 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:09.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:09 smithi022 ceph-mon[31360]: from='client.34604 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-12-15T17:18:09.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3776322880' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:09.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:09 smithi022 ceph-mon[40250]: from='client.34604 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:09.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:09 smithi022 ceph-mon[40250]: from='client.34604 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-12-15T17:18:09.341 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-12-15T17:18:09.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-12-15T17:18:09.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-12-15T17:18:09.384 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd dump --format=json 2021-12-15T17:18:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3776322880' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:09 smithi122 ceph-mon[32255]: from='client.34604 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-12-15T17:18:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:09 smithi122 ceph-mon[32255]: from='client.34604 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-12-15T17:18:09.841 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:10.230 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:10.230 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.233 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.233 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.233 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.233 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.234 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:10.234 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.234 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.237 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.237 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:10 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:18:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:18:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:10 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:18:11.253 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":102,"fsid":"0ea1c4a2-5dca-11ec-8c31-001a4aab830c","created":"2021-12-15T17:12:14.883080+0000","modified":"2021-12-15T17:18:02.282427+0000","last_up_change":"2021-12-15T17:17:34.441288+0000","last_in_change":"2021-12-15T17:16:57.761205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-12-15T17:15:00.481237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2021-12-15T17:17:41.953027+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-12-15T17:17:42.386398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-12-15T17:17:43.889355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-12-15T17:17:45.863720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-12-15T17:17:47.933406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"c1d97a7c-370f-4a35-9216-1a6dc0e6cc09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":24,"last_clean_end":33,"up_from":40,"up_thru":97,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6801","nonce":3812237890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6802","nonce":3812237890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6804","nonce":3812237890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6803","nonce":3812237890}]},"public_addr":"172.21.15.22:6801/3812237890","cluster_addr":"172.21.15.22:6802/3812237890","heartbeat_back_addr":"172.21.15.22:6804/3812237890","heartbeat_front_addr":"172.21.15.22:6803/3812237890","state":["exists","up"]},{"osd":1,"uuid":"da26a6f5-3ae4-46a9-b878-10f47104ea9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":37,"up_from":43,"up_thru":97,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6805","nonce":2823805689}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6806","nonce":2823805689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6808","nonce":2823805689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6807","nonce":2823805689}]},"public_addr":"172.21.15.22:6805/2823805689","cluster_addr":"172.21.15.22:6806/2823805689","heartbeat_back_addr":"172.21.15.22:6808/2823805689","heartbeat_front_addr":"172.21.15.22:6807/2823805689","state":["exists","up"]},{"osd":2,"uuid":"603393c6-9925-483a-a06a-383b3afc520b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":31,"up_from":36,"up_thru":97,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6809","nonce":3252411997}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6810","nonce":3252411997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6812","nonce":3252411997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6811","nonce":3252411997}]},"public_addr":"172.21.15.22:6809/3252411997","cluster_addr":"172.21.15.22:6810/3252411997","heartbeat_back_addr":"172.21.15.22:6812/3252411997","heartbeat_front_addr":"172.21.15.22:6811/3252411997","state":["exists","up"]},{"osd":3,"uuid":"0140e354-7030-40ef-8564-a25c37a35d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6813","nonce":3933129392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6814","nonce":3933129392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6816","nonce":3933129392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6815","nonce":3933129392}]},"public_addr":"172.21.15.22:6813/3933129392","cluster_addr":"172.21.15.22:6814/3933129392","heartbeat_back_addr":"172.21.15.22:6816/3933129392","heartbeat_front_addr":"172.21.15.22:6815/3933129392","state":["exists","up"]},{"osd":4,"uuid":"6125b2a3-33b6-48dd-8e49-58ee4b3612bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":71,"last_clean_end":80,"up_from":88,"up_thru":97,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6800","nonce":3954553330}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6801","nonce":3954553330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6803","nonce":3954553330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6802","nonce":3954553330}]},"public_addr":"172.21.15.122:6800/3954553330","cluster_addr":"172.21.15.122:6801/3954553330","heartbeat_back_addr":"172.21.15.122:6803/3954553330","heartbeat_front_addr":"172.21.15.122:6802/3954553330","state":["exists","up"]},{"osd":5,"uuid":"d723e884-3aaa-4bec-990b-c70d6c3b6b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":73,"last_clean_end":85,"up_from":90,"up_thru":97,"down_at":86,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6804","nonce":443063154}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6805","nonce":443063154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6807","nonce":443063154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6806","nonce":443063154}]},"public_addr":"172.21.15.122:6804/443063154","cluster_addr":"172.21.15.122:6805/443063154","heartbeat_back_addr":"172.21.15.122:6807/443063154","heartbeat_front_addr":"172.21.15.122:6806/443063154","state":["exists","up"]},{"osd":6,"uuid":"49064534-bd3e-4197-a0e1-ed10f6fbcf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":65,"last_clean_end":76,"up_from":83,"up_thru":95,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6808","nonce":2434053838}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6809","nonce":2434053838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6811","nonce":2434053838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6810","nonce":2434053838}]},"public_addr":"172.21.15.122:6808/2434053838","cluster_addr":"172.21.15.122:6809/2434053838","heartbeat_back_addr":"172.21.15.122:6811/2434053838","heartbeat_front_addr":"172.21.15.122:6810/2434053838","state":["exists","up"]},{"osd":7,"uuid":"2c4fea5f-e883-4ceb-8dea-830393be48e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6812","nonce":1132880908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6813","nonce":1132880908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6815","nonce":1132880908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6814","nonce":1132880908}]},"public_addr":"172.21.15.122:6812/1132880908","cluster_addr":"172.21.15.122:6813/1132880908","heartbeat_back_addr":"172.21.15.122:6815/1132880908","heartbeat_front_addr":"172.21.15.122:6814/1132880908","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:04.508111+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:22.715551+0000","dead_epoch":38},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:46.281031+0000","dead_epoch":32},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:15:16.215277+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:15:52.827482+0000","dead_epoch":81},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:16:13.254106+0000","dead_epoch":86},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:16:39.212151+0000","dead_epoch":77},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:17:12.380814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/3227215263":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1986515556":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1619922885":"2021-12-16T17:13:01.918853+0000","172.21.15.22:6800/1740939671":"2021-12-16T17:13:01.918853+0000","172.21.15.22:0/1935887333":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/795717723":"2021-12-16T17:13:01.918853+0000","172.21.15.22:0/3996822845":"2021-12-16T17:13:01.918853+0000","172.21.15.22:6800/3267813977":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/670289207":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/3033545855":"2021-12-16T17:18:02.282393+0000","172.21.15.22:6800/4198419672":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/119111919":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/1042530731":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1597061619":"2021-12-16T17:12:41.917906+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-12-15T17:18:11.400 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3782M 2021-12-15T17:18:11.401 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:18:11.401 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:18:11.401 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 4037M 2021-12-15T17:18:11.401 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:18:11.402 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:18:11.402 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:11.402 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:11.403 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3782M 2021-12-15T17:18:11.403 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:18:11.403 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:18:11.404 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 4037M 2021-12-15T17:18:11.404 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:18:11.404 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:18:11.405 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:11.405 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:11 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3782M 2021-12-15T17:18:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:18:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:18:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 4037M 2021-12-15T17:18:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:18:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:18:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:11 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:11.556 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2021-12-15T17:15:00.481237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2021-12-15T17:17:41.953027+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-12-15T17:17:42.386398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-12-15T17:17:43.889355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-12-15T17:17:45.863720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-12-15T17:17:47.933406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}] 2021-12-15T17:18:11.557 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd pool get .mgr pg_num 2021-12-15T17:18:12.002 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:12 smithi122 ceph-mon[32255]: Deploying daemon alertmanager.a on smithi022 2021-12-15T17:18:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:12 smithi122 ceph-mon[32255]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2021-12-15T17:18:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3291494836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:12.500 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:12 smithi022 ceph-mon[31360]: Deploying daemon alertmanager.a on smithi022 2021-12-15T17:18:12.501 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:12 smithi022 ceph-mon[31360]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2021-12-15T17:18:12.501 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3291494836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:12.501 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:12 smithi022 ceph-mon[40250]: Deploying daemon alertmanager.a on smithi022 2021-12-15T17:18:12.502 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:12 smithi022 ceph-mon[40250]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2021-12-15T17:18:12.502 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3291494836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:13.369 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2021-12-15T17:18:13.719 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd pool get datapool pg_num 2021-12-15T17:18:14.170 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:14 smithi022 ceph-mon[31360]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-12-15T17:18:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1268162847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2021-12-15T17:18:14.399 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:14 smithi022 ceph-mon[40250]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-12-15T17:18:14.400 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1268162847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2021-12-15T17:18:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:14 smithi122 ceph-mon[32255]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-12-15T17:18:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1268162847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2021-12-15T17:18:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:18:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:18:15.868 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:15 smithi022 ceph-mon[31360]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2021-12-15T17:18:15.868 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:15 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:15.869 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:15 smithi022 ceph-mon[40250]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2021-12-15T17:18:15.869 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:15 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:15 smithi122 ceph-mon[32255]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2021-12-15T17:18:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:15 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:16.273 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 3 2021-12-15T17:18:16.682 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd pool get .rgw.root pg_num 2021-12-15T17:18:16.982 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3506613678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "datapool", "var": "pg_num"}]: dispatch 2021-12-15T17:18:16.982 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3506613678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "datapool", "var": "pg_num"}]: dispatch 2021-12-15T17:18:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3506613678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "datapool", "var": "pg_num"}]: dispatch 2021-12-15T17:18:17.318 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:18.066 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:17 smithi022 ceph-mon[31360]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:18.067 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:17 smithi022 ceph-mon[40250]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:17 smithi122 ceph-mon[32255]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:19.180 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2021-12-15T17:18:19.481 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd pool get default.rgw.log pg_num 2021-12-15T17:18:19.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:19 smithi022 podman[82575]: 2021-12-15T17:18:20.046 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:20.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:20 smithi022 ceph-mon[31360]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:20.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2516550471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-12-15T17:18:20.274 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:20 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:20.274 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:19 smithi022 bash[82358]: af6418612b1f978e150de417157cfa4bd4803d38c2d22642dd91b1217db4b546 2021-12-15T17:18:20.274 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:19 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:19.983Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-12-15T17:18:20.274 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:19 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:19.983Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-12-15T17:18:20.275 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:19 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:19.984Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.22 port=9094 2021-12-15T17:18:20.275 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:19 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:19.985Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-12-15T17:18:20.275 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:20 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:20.004Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-12-15T17:18:20.275 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:20 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:20.005Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-12-15T17:18:20.276 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:20 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:20.007Z caller=main.go:497 msg=Listening address=:9093 2021-12-15T17:18:20.276 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:20 smithi022 systemd[1]: Started Ceph alertmanager.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:18:20.276 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:20 smithi022 ceph-mon[40250]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:20.276 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2516550471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-12-15T17:18:20.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:20 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:20 smithi122 ceph-mon[32255]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2516550471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-12-15T17:18:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:20 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:21.018 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:21 smithi022 systemd[1]: Starting Ceph node-exporter.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:18:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:21 smithi022 ceph-mon[31360]: Deploying daemon node-exporter.a on smithi022 2021-12-15T17:18:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:21 smithi022 ceph-mon[40250]: Deploying daemon node-exporter.a on smithi022 2021-12-15T17:18:21.421 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2021-12-15T17:18:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:21 smithi122 ceph-mon[32255]: Deploying daemon node-exporter.a on smithi022 2021-12-15T17:18:21.696 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:21 smithi022 bash[83108]: Trying to pull quay.io/prometheus/node-exporter:v0.18.1... 2021-12-15T17:18:21.697 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd pool get default.rgw.control pg_num 2021-12-15T17:18:22.145 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:22.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:22 smithi022 ceph-mon[31360]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:22.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2553283339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-12-15T17:18:22.262 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:22 smithi022 ceph-mon[40250]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:22.262 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2553283339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-12-15T17:18:22.263 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:21 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:21.985Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000104425s 2021-12-15T17:18:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:22 smithi122 ceph-mon[32255]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2553283339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-12-15T17:18:22.512 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:22 smithi022 bash[83108]: Getting image source signatures 2021-12-15T17:18:22.513 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:22 smithi022 bash[83108]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-12-15T17:18:22.513 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:22 smithi022 bash[83108]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-12-15T17:18:22.513 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:22 smithi022 bash[83108]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-12-15T17:18:23.375 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2021-12-15T17:18:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:23 smithi122 ceph-mon[32255]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:23.443 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:23 smithi022 bash[83108]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-12-15T17:18:23.505 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:23 smithi022 bash[83108]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-12-15T17:18:23.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:23 smithi022 ceph-mon[40250]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:23.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:23 smithi022 ceph-mon[31360]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:23.698 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:23 smithi022 bash[83108]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-12-15T17:18:23.752 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd pool get default.rgw.meta pg_num 2021-12-15T17:18:24.198 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/217667434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-12-15T17:18:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/217667434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-12-15T17:18:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/217667434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-12-15T17:18:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:18:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:18:25.492 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2021-12-15T17:18:26.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:25 smithi022 ceph-mon[31360]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:26.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:25 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:26.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1409571249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-12-15T17:18:26.048 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:25 smithi022 ceph-mon[40250]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:26.048 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:25 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:26.048 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1409571249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-12-15T17:18:26.049 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:25 smithi022 bash[83108]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-12-15T17:18:26.049 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:25 smithi022 bash[83108]: Writing manifest to image destination 2021-12-15T17:18:26.049 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:25 smithi022 bash[83108]: Storing signatures 2021-12-15T17:18:26.105 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-12-15T17:18:26.106 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-12-15T17:18:26.106 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph mgr dump --format=json 2021-12-15T17:18:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:25 smithi122 ceph-mon[32255]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:25 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1409571249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-12-15T17:18:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:26.702 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 podman[83304]: 2021-12-15T17:18:26.703 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-12-15T17:18:26.703 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-12-15T17:18:26.703 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-12-15T17:18:26.703 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-12-15T17:18:26.704 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-12-15T17:18:26.704 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-12-15T17:18:26.704 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-12-15T17:18:26.704 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-12-15T17:18:26.705 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-12-15T17:18:26.705 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-12-15T17:18:26.705 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-12-15T17:18:26.705 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-12-15T17:18:26.706 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-12-15T17:18:26.706 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-12-15T17:18:26.706 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-12-15T17:18:26.706 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-12-15T17:18:26.707 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-12-15T17:18:26.707 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-12-15T17:18:26.707 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-12-15T17:18:26.707 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-12-15T17:18:26.707 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-12-15T17:18:26.708 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-12-15T17:18:26.708 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-12-15T17:18:26.708 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-12-15T17:18:26.708 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-12-15T17:18:26.708 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-12-15T17:18:26.709 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-12-15T17:18:26.709 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-12-15T17:18:26.709 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-12-15T17:18:26.709 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - time" source="node_exporter.go:104" 2021-12-15T17:18:26.710 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-12-15T17:18:26.710 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-12-15T17:18:26.710 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-12-15T17:18:26.711 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-12-15T17:18:26.711 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 conmon[84199]: time="2021-12-15T17:18:26Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-12-15T17:18:26.711 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 bash[83108]: 2f796310dd55d581ada22631ce2c357b99e03c1cab64d8121e4e560da415b180 2021-12-15T17:18:27.028 INFO:journalctl@ceph.node-exporter.a.smithi022.stdout:Dec 15 17:18:26 smithi022 systemd[1]: Started Ceph node-exporter.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:18:27.683 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:27 smithi122 systemd[1]: Starting Ceph node-exporter.b for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:18:27.775 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:27 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:27.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:27 smithi022 ceph-mon[31360]: Deploying daemon node-exporter.b on smithi122 2021-12-15T17:18:27.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:27 smithi022 ceph-mon[31360]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:27.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:27 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:27.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:27 smithi022 ceph-mon[40250]: Deploying daemon node-exporter.b on smithi122 2021-12-15T17:18:27.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:27 smithi022 ceph-mon[40250]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:27.935 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:27 smithi122 bash[73052]: Trying to pull quay.io/prometheus/node-exporter:v0.18.1... 2021-12-15T17:18:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:27 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:27 smithi122 ceph-mon[32255]: Deploying daemon node-exporter.b on smithi122 2021-12-15T17:18:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:27 smithi122 ceph-mon[32255]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:27.990 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:18:28.428 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":21,"active_gid":34562,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6800","nonce":3632387897}]},"active_addr":"172.21.15.22:6800/3632387897","active_change":"2021-12-15T17:18:02.282786+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34556,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm, `scale-up` means that it starts out with minmum pgs and scales up when there is pressure`scale-down means start out with full pgs and scalesdown when there is pressure","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm, `scale-up` means that it starts out with minmum pgs and scales up when there is pressure`scale-down means start out with full pgs and scalesdown when there is pressure","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.22:8443/","prometheus":"http://172.21.15.22:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":102,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":3730920400}]},{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":1303413583}]},{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":781043399}]},{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":142888001}]}]}} 2021-12-15T17:18:28.433 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-12-15T17:18:28.434 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-12-15T17:18:28.434 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd dump --format=json 2021-12-15T17:18:28.775 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1845866174' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-12-15T17:18:28.881 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1845866174' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-12-15T17:18:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1845866174' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-12-15T17:18:29.185 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:28 smithi122 bash[73052]: Getting image source signatures 2021-12-15T17:18:29.186 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:28 smithi122 bash[73052]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-12-15T17:18:29.186 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:28 smithi122 bash[73052]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-12-15T17:18:29.186 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:28 smithi122 bash[73052]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-12-15T17:18:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:29 smithi022 ceph-mon[31360]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:29 smithi022 ceph-mon[40250]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:30.030 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:18:29 smithi022 conmon[82628]: level=info ts=2021-12-15T17:18:29.986Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001135984s 2021-12-15T17:18:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:29 smithi122 ceph-mon[32255]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:30.185 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:29 smithi122 bash[73052]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-12-15T17:18:30.185 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:29 smithi122 bash[73052]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-12-15T17:18:30.186 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:29 smithi122 bash[73052]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-12-15T17:18:30.328 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:18:30.328 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":102,"fsid":"0ea1c4a2-5dca-11ec-8c31-001a4aab830c","created":"2021-12-15T17:12:14.883080+0000","modified":"2021-12-15T17:18:02.282427+0000","last_up_change":"2021-12-15T17:17:34.441288+0000","last_in_change":"2021-12-15T17:16:57.761205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-12-15T17:15:00.481237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2021-12-15T17:17:41.953027+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-12-15T17:17:42.386398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-12-15T17:17:43.889355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-12-15T17:17:45.863720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-12-15T17:17:47.933406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"c1d97a7c-370f-4a35-9216-1a6dc0e6cc09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":24,"last_clean_end":33,"up_from":40,"up_thru":97,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6801","nonce":3812237890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6802","nonce":3812237890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6804","nonce":3812237890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6803","nonce":3812237890}]},"public_addr":"172.21.15.22:6801/3812237890","cluster_addr":"172.21.15.22:6802/3812237890","heartbeat_back_addr":"172.21.15.22:6804/3812237890","heartbeat_front_addr":"172.21.15.22:6803/3812237890","state":["exists","up"]},{"osd":1,"uuid":"da26a6f5-3ae4-46a9-b878-10f47104ea9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":37,"up_from":43,"up_thru":97,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6805","nonce":2823805689}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6806","nonce":2823805689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6808","nonce":2823805689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6807","nonce":2823805689}]},"public_addr":"172.21.15.22:6805/2823805689","cluster_addr":"172.21.15.22:6806/2823805689","heartbeat_back_addr":"172.21.15.22:6808/2823805689","heartbeat_front_addr":"172.21.15.22:6807/2823805689","state":["exists","up"]},{"osd":2,"uuid":"603393c6-9925-483a-a06a-383b3afc520b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":31,"up_from":36,"up_thru":97,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6809","nonce":3252411997}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6810","nonce":3252411997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6812","nonce":3252411997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6811","nonce":3252411997}]},"public_addr":"172.21.15.22:6809/3252411997","cluster_addr":"172.21.15.22:6810/3252411997","heartbeat_back_addr":"172.21.15.22:6812/3252411997","heartbeat_front_addr":"172.21.15.22:6811/3252411997","state":["exists","up"]},{"osd":3,"uuid":"0140e354-7030-40ef-8564-a25c37a35d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6813","nonce":3933129392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6814","nonce":3933129392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6816","nonce":3933129392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6815","nonce":3933129392}]},"public_addr":"172.21.15.22:6813/3933129392","cluster_addr":"172.21.15.22:6814/3933129392","heartbeat_back_addr":"172.21.15.22:6816/3933129392","heartbeat_front_addr":"172.21.15.22:6815/3933129392","state":["exists","up"]},{"osd":4,"uuid":"6125b2a3-33b6-48dd-8e49-58ee4b3612bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":71,"last_clean_end":80,"up_from":88,"up_thru":97,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6800","nonce":3954553330}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6801","nonce":3954553330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6803","nonce":3954553330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6802","nonce":3954553330}]},"public_addr":"172.21.15.122:6800/3954553330","cluster_addr":"172.21.15.122:6801/3954553330","heartbeat_back_addr":"172.21.15.122:6803/3954553330","heartbeat_front_addr":"172.21.15.122:6802/3954553330","state":["exists","up"]},{"osd":5,"uuid":"d723e884-3aaa-4bec-990b-c70d6c3b6b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":73,"last_clean_end":85,"up_from":90,"up_thru":97,"down_at":86,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6804","nonce":443063154}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6805","nonce":443063154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6807","nonce":443063154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6806","nonce":443063154}]},"public_addr":"172.21.15.122:6804/443063154","cluster_addr":"172.21.15.122:6805/443063154","heartbeat_back_addr":"172.21.15.122:6807/443063154","heartbeat_front_addr":"172.21.15.122:6806/443063154","state":["exists","up"]},{"osd":6,"uuid":"49064534-bd3e-4197-a0e1-ed10f6fbcf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":65,"last_clean_end":76,"up_from":83,"up_thru":95,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6808","nonce":2434053838}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6809","nonce":2434053838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6811","nonce":2434053838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6810","nonce":2434053838}]},"public_addr":"172.21.15.122:6808/2434053838","cluster_addr":"172.21.15.122:6809/2434053838","heartbeat_back_addr":"172.21.15.122:6811/2434053838","heartbeat_front_addr":"172.21.15.122:6810/2434053838","state":["exists","up"]},{"osd":7,"uuid":"2c4fea5f-e883-4ceb-8dea-830393be48e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6812","nonce":1132880908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6813","nonce":1132880908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6815","nonce":1132880908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6814","nonce":1132880908}]},"public_addr":"172.21.15.122:6812/1132880908","cluster_addr":"172.21.15.122:6813/1132880908","heartbeat_back_addr":"172.21.15.122:6815/1132880908","heartbeat_front_addr":"172.21.15.122:6814/1132880908","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:04.508111+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:22.715551+0000","dead_epoch":38},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:46.281031+0000","dead_epoch":32},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:15:16.215277+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:15:52.827482+0000","dead_epoch":81},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:16:13.254106+0000","dead_epoch":86},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:16:39.212151+0000","dead_epoch":77},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:17:12.380814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/3227215263":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1986515556":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1619922885":"2021-12-16T17:13:01.918853+0000","172.21.15.22:6800/1740939671":"2021-12-16T17:13:01.918853+0000","172.21.15.22:0/1935887333":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/795717723":"2021-12-16T17:13:01.918853+0000","172.21.15.22:0/3996822845":"2021-12-16T17:13:01.918853+0000","172.21.15.22:6800/3267813977":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/670289207":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/3033545855":"2021-12-16T17:18:02.282393+0000","172.21.15.22:6800/4198419672":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/119111919":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/1042530731":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1597061619":"2021-12-16T17:12:41.917906+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-12-15T17:18:30.599 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-12-15T17:18:30.599 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd dump --format=json 2021-12-15T17:18:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4248941395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4248941395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:31.044 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4248941395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:32.046 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:31 smithi022 ceph-mon[31360]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:32.047 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:31 smithi022 ceph-mon[40250]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:31 smithi122 ceph-mon[32255]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:32.358 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:18:32.358 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":102,"fsid":"0ea1c4a2-5dca-11ec-8c31-001a4aab830c","created":"2021-12-15T17:12:14.883080+0000","modified":"2021-12-15T17:18:02.282427+0000","last_up_change":"2021-12-15T17:17:34.441288+0000","last_in_change":"2021-12-15T17:16:57.761205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-12-15T17:15:00.481237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2021-12-15T17:17:41.953027+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-12-15T17:17:42.386398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-12-15T17:17:43.889355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-12-15T17:17:45.863720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-12-15T17:17:47.933406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"c1d97a7c-370f-4a35-9216-1a6dc0e6cc09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":24,"last_clean_end":33,"up_from":40,"up_thru":97,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6801","nonce":3812237890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6802","nonce":3812237890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6804","nonce":3812237890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6803","nonce":3812237890}]},"public_addr":"172.21.15.22:6801/3812237890","cluster_addr":"172.21.15.22:6802/3812237890","heartbeat_back_addr":"172.21.15.22:6804/3812237890","heartbeat_front_addr":"172.21.15.22:6803/3812237890","state":["exists","up"]},{"osd":1,"uuid":"da26a6f5-3ae4-46a9-b878-10f47104ea9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":37,"up_from":43,"up_thru":97,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6805","nonce":2823805689}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6806","nonce":2823805689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6808","nonce":2823805689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6807","nonce":2823805689}]},"public_addr":"172.21.15.22:6805/2823805689","cluster_addr":"172.21.15.22:6806/2823805689","heartbeat_back_addr":"172.21.15.22:6808/2823805689","heartbeat_front_addr":"172.21.15.22:6807/2823805689","state":["exists","up"]},{"osd":2,"uuid":"603393c6-9925-483a-a06a-383b3afc520b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":31,"up_from":36,"up_thru":97,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6809","nonce":3252411997}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6810","nonce":3252411997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6812","nonce":3252411997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6811","nonce":3252411997}]},"public_addr":"172.21.15.22:6809/3252411997","cluster_addr":"172.21.15.22:6810/3252411997","heartbeat_back_addr":"172.21.15.22:6812/3252411997","heartbeat_front_addr":"172.21.15.22:6811/3252411997","state":["exists","up"]},{"osd":3,"uuid":"0140e354-7030-40ef-8564-a25c37a35d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6813","nonce":3933129392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6814","nonce":3933129392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6816","nonce":3933129392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.22:6815","nonce":3933129392}]},"public_addr":"172.21.15.22:6813/3933129392","cluster_addr":"172.21.15.22:6814/3933129392","heartbeat_back_addr":"172.21.15.22:6816/3933129392","heartbeat_front_addr":"172.21.15.22:6815/3933129392","state":["exists","up"]},{"osd":4,"uuid":"6125b2a3-33b6-48dd-8e49-58ee4b3612bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":71,"last_clean_end":80,"up_from":88,"up_thru":97,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6800","nonce":3954553330}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6801","nonce":3954553330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6803","nonce":3954553330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6802","nonce":3954553330}]},"public_addr":"172.21.15.122:6800/3954553330","cluster_addr":"172.21.15.122:6801/3954553330","heartbeat_back_addr":"172.21.15.122:6803/3954553330","heartbeat_front_addr":"172.21.15.122:6802/3954553330","state":["exists","up"]},{"osd":5,"uuid":"d723e884-3aaa-4bec-990b-c70d6c3b6b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":73,"last_clean_end":85,"up_from":90,"up_thru":97,"down_at":86,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6804","nonce":443063154}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6805","nonce":443063154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6807","nonce":443063154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6806","nonce":443063154}]},"public_addr":"172.21.15.122:6804/443063154","cluster_addr":"172.21.15.122:6805/443063154","heartbeat_back_addr":"172.21.15.122:6807/443063154","heartbeat_front_addr":"172.21.15.122:6806/443063154","state":["exists","up"]},{"osd":6,"uuid":"49064534-bd3e-4197-a0e1-ed10f6fbcf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":65,"last_clean_end":76,"up_from":83,"up_thru":95,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6808","nonce":2434053838}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6809","nonce":2434053838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6811","nonce":2434053838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6810","nonce":2434053838}]},"public_addr":"172.21.15.122:6808/2434053838","cluster_addr":"172.21.15.122:6809/2434053838","heartbeat_back_addr":"172.21.15.122:6811/2434053838","heartbeat_front_addr":"172.21.15.122:6810/2434053838","state":["exists","up"]},{"osd":7,"uuid":"2c4fea5f-e883-4ceb-8dea-830393be48e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6812","nonce":1132880908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6813","nonce":1132880908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6815","nonce":1132880908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6814","nonce":1132880908}]},"public_addr":"172.21.15.122:6812/1132880908","cluster_addr":"172.21.15.122:6813/1132880908","heartbeat_back_addr":"172.21.15.122:6815/1132880908","heartbeat_front_addr":"172.21.15.122:6814/1132880908","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:04.508111+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:22.715551+0000","dead_epoch":38},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:14:46.281031+0000","dead_epoch":32},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:15:16.215277+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:15:52.827482+0000","dead_epoch":81},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:16:13.254106+0000","dead_epoch":86},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:16:39.212151+0000","dead_epoch":77},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-12-15T17:17:12.380814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/3227215263":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1986515556":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1619922885":"2021-12-16T17:13:01.918853+0000","172.21.15.22:6800/1740939671":"2021-12-16T17:13:01.918853+0000","172.21.15.22:0/1935887333":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/795717723":"2021-12-16T17:13:01.918853+0000","172.21.15.22:0/3996822845":"2021-12-16T17:13:01.918853+0000","172.21.15.22:6800/3267813977":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/670289207":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/3033545855":"2021-12-16T17:18:02.282393+0000","172.21.15.22:6800/4198419672":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/119111919":"2021-12-16T17:12:41.917906+0000","172.21.15.22:0/1042530731":"2021-12-16T17:18:02.282393+0000","172.21.15.22:0/1597061619":"2021-12-16T17:12:41.917906+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-12-15T17:18:32.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-12-15T17:18:32.662 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-12-15T17:18:32.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-12-15T17:18:32.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-12-15T17:18:32.664 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-12-15T17:18:32.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-12-15T17:18:32.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2021-12-15T17:18:32.666 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2021-12-15T17:18:32.684 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:32 smithi122 bash[73052]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-12-15T17:18:32.685 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:32 smithi122 bash[73052]: Writing manifest to image destination 2021-12-15T17:18:32.685 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:32 smithi122 bash[73052]: Storing signatures 2021-12-15T17:18:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3147811230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3147811230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:33.124 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3147811230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-12-15T17:18:33.124 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:32 smithi122 podman[73205]: 2021-12-15T17:18:33.391 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:33.435 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-12-15T17:18:33.435 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-12-15T17:18:33.436 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-12-15T17:18:33.436 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-12-15T17:18:33.436 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-12-15T17:18:33.437 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-12-15T17:18:33.437 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-12-15T17:18:33.437 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-12-15T17:18:33.437 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-12-15T17:18:33.438 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-12-15T17:18:33.438 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-12-15T17:18:33.439 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-12-15T17:18:33.439 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-12-15T17:18:33.440 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-12-15T17:18:33.440 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-12-15T17:18:33.440 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-12-15T17:18:33.441 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-12-15T17:18:33.441 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-12-15T17:18:33.441 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-12-15T17:18:33.441 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-12-15T17:18:33.442 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-12-15T17:18:33.442 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-12-15T17:18:33.442 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-12-15T17:18:33.442 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-12-15T17:18:33.442 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-12-15T17:18:33.443 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-12-15T17:18:33.443 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-12-15T17:18:33.443 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-12-15T17:18:33.443 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-12-15T17:18:33.444 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - time" source="node_exporter.go:104" 2021-12-15T17:18:33.444 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-12-15T17:18:33.444 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-12-15T17:18:33.445 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-12-15T17:18:33.445 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-12-15T17:18:33.445 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 conmon[73281]: time="2021-12-15T17:18:33Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-12-15T17:18:33.446 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 bash[73052]: 616edf41444f0d651fa04bb7fc06d87b58b088dd23f83a408b42f29623b16c30 2021-12-15T17:18:33.446 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Dec 15 17:18:33 smithi122 systemd[1]: Started Ceph node-exporter.b for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:18:33.605 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:33.724 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:33.776 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:33.805 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:33.823 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:34.082 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:34.107 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:33 smithi022 ceph-mon[31360]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:34.107 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:34.108 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:33 smithi022 ceph-mon[40250]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:34.108 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:34.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:33 smithi122 ceph-mon[32255]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:34.219 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:34.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:34 smithi022 ceph-mon[40250]: Deploying daemon prometheus.a on smithi122 2021-12-15T17:18:34.909 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:34 smithi022 ceph-mon[31360]: Deploying daemon prometheus.a on smithi122 2021-12-15T17:18:35.160 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:34 smithi122 ceph-mon[32255]: Deploying daemon prometheus.a on smithi122 2021-12-15T17:18:35.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:18:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:18:35.983 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:35 smithi022 ceph-mon[31360]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:35.983 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:35 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:35.984 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:35 smithi022 ceph-mon[40250]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:35.984 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:35 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:35 smithi122 ceph-mon[32255]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:35 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:37.860 INFO:teuthology.orchestra.run.smithi022.stdout:171798691878 2021-12-15T17:18:37.861 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-12-15T17:18:37.946 INFO:teuthology.orchestra.run.smithi022.stdout:184683593765 2021-12-15T17:18:37.946 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-12-15T17:18:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:38 smithi122 ceph-mon[32255]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:38 smithi022 ceph-mon[31360]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:38 smithi022 ceph-mon[40250]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:38.548 INFO:teuthology.orchestra.run.smithi022.stdout:154618822695 2021-12-15T17:18:38.548 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-12-15T17:18:39.233 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:39 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:39.233 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:39 smithi022 ceph-mon[31360]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:39.234 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:39 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:39.234 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:39 smithi022 ceph-mon[40250]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:356482285584 2021-12-15T17:18:39.400 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.6 2021-12-15T17:18:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:39 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:39 smithi122 ceph-mon[32255]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:39.914 INFO:teuthology.orchestra.run.smithi022.stdout:137438953513 2021-12-15T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-12-15T17:18:40.026 INFO:teuthology.orchestra.run.smithi022.stdout:377957122063 2021-12-15T17:18:40.027 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-12-15T17:18:40.217 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:40.231 INFO:teuthology.orchestra.run.smithi022.stdout:330712481811 2021-12-15T17:18:40.232 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.7 2021-12-15T17:18:40.371 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:40.409 INFO:teuthology.orchestra.run.smithi022.stdout:386547056653 2021-12-15T17:18:40.409 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-12-15T17:18:40.474 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:40.487 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:40.936 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:42 smithi122 ceph-mon[32255]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-12-15T17:18:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:42 smithi022 ceph-mon[31360]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-12-15T17:18:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:42 smithi022 ceph-mon[40250]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-12-15T17:18:42.779 INFO:teuthology.orchestra.run.smithi022.stdout:154618822696 2021-12-15T17:18:42.892 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:42.916 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/860868621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-12-15T17:18:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:43 smithi022 ceph-mon[31360]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2021-12-15T17:18:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/860868621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-12-15T17:18:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:43 smithi022 ceph-mon[40250]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2021-12-15T17:18:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/860868621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-12-15T17:18:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:43 smithi122 ceph-mon[32255]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2021-12-15T17:18:43.562 INFO:teuthology.orchestra.run.smithi022.stdout:184683593766 2021-12-15T17:18:43.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822695 got 154618822696 for osd.2 2021-12-15T17:18:43.624 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2861896066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-12-15T17:18:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2861896066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-12-15T17:18:44.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2861896066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-12-15T17:18:44.601 INFO:teuthology.orchestra.run.smithi022.stdout:137438953514 2021-12-15T17:18:44.658 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593765 got 184683593766 for osd.1 2021-12-15T17:18:44.658 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:45.144 INFO:teuthology.orchestra.run.smithi022.stdout:171798691879 2021-12-15T17:18:45.154 INFO:teuthology.orchestra.run.smithi022.stdout:356482285585 2021-12-15T17:18:45.171 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 systemd[1]: Starting Ceph prometheus.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:18:45.319 INFO:teuthology.orchestra.run.smithi022.stdout:377957122063 2021-12-15T17:18:45.423 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:18:45.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2713145302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-12-15T17:18:45.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:45 smithi122 ceph-mon[32255]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:45.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1836477015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-12-15T17:18:45.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/404626584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-12-15T17:18:45.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:45 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:45.449 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2713145302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-12-15T17:18:45.449 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[31360]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:45.449 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1836477015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-12-15T17:18:45.449 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/404626584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-12-15T17:18:45.450 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:45.450 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2713145302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-12-15T17:18:45.450 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[40250]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:45.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1836477015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-12-15T17:18:45.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/404626584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-12-15T17:18:45.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:45 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:45.452 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953513 got 137438953514 for osd.3 2021-12-15T17:18:45.452 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:45.679 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 podman[74039]: 2021-12-15T17:18:45.935 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 bash[73885]: b2357526d93ad82496a7d7e1b365e403095b9396ecbedd472ea878f10690e478 2021-12-15T17:18:45.936 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.687Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-12-15T17:18:45.936 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.687Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-12-15T17:18:45.937 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.687Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-12-15T17:18:45.937 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.687Z caller=main.go:339 host_details="(Linux 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Tue Nov 16 14:42:35 UTC 2021 x86_64 smithi122 (none))" 2021-12-15T17:18:45.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.687Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-12-15T17:18:45.939 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.687Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-12-15T17:18:45.939 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.688Z caller=main.go:678 msg="Starting TSDB ..." 2021-12-15T17:18:45.939 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.688Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-12-15T17:18:45.939 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.690Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-12-15T17:18:45.940 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.690Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2021-12-15T17:18:45.940 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.690Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=129.956µs 2021-12-15T17:18:45.940 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.690Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-12-15T17:18:45.940 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.691Z caller=main.go:695 msg="TSDB started" 2021-12-15T17:18:45.941 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.691Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-12-15T17:18:45.941 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.704Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-12-15T17:18:45.941 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 conmon[74079]: level=info ts=2021-12-15T17:18:45.704Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-12-15T17:18:45.941 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:18:45 smithi122 systemd[1]: Started Ceph prometheus.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:18:46.333 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285584 got 356482285585 for osd.6 2021-12-15T17:18:46.333 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1127357654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-12-15T17:18:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-12-15T17:18:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-12-15T17:18:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:46 smithi122 ceph-mon[32255]: Deploying daemon grafana.a on smithi122 2021-12-15T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1127357654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-12-15T17:18:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-12-15T17:18:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-12-15T17:18:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[31360]: Deploying daemon grafana.a on smithi122 2021-12-15T17:18:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1127357654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-12-15T17:18:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-12-15T17:18:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-12-15T17:18:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:46 smithi022 ceph-mon[40250]: Deploying daemon grafana.a on smithi122 2021-12-15T17:18:46.637 INFO:tasks.cephadm.ceph_manager.ceph:need seq 377957122063 got 377957122063 for osd.4 2021-12-15T17:18:46.637 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:47.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691878 got 171798691879 for osd.0 2021-12-15T17:18:47.161 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:47.343 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:47 smithi022 ceph-mon[31360]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:47.344 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:47 smithi022 ceph-mon[40250]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:47.458 INFO:teuthology.orchestra.run.smithi022.stdout:330712481812 2021-12-15T17:18:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:47 smithi122 ceph-mon[32255]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:47.723 INFO:teuthology.orchestra.run.smithi022.stdout:386547056655 2021-12-15T17:18:47.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481811 got 330712481812 for osd.7 2021-12-15T17:18:47.878 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:48.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 386547056653 got 386547056655 for osd.5 2021-12-15T17:18:48.107 DEBUG:teuthology.parallel:result is None 2021-12-15T17:18:48.107 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-12-15T17:18:48.108 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph pg dump --format=json 2021-12-15T17:18:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3560029956' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-12-15T17:18:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4240158835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-12-15T17:18:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3560029956' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-12-15T17:18:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4240158835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-12-15T17:18:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3560029956' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-12-15T17:18:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4240158835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-12-15T17:18:49.387 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:49 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:49.388 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:49 smithi022 ceph-mon[31360]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:49.388 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:49 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:49.389 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:49 smithi022 ceph-mon[40250]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:49 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:18:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:49 smithi122 ceph-mon[32255]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:50.241 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:18:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2021-12-15T17:18:50.591 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2021-12-15T17:18:49.130158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596187,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":998,"num_read_kb":844,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596,"ondisk_log_size":596,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":61492,"kb_used_data":9204,"kb_used_omap":6,"kb_used_meta":52217,"kb_avail":749899724,"statfs":{"total":767960285184,"available":767897317376,"internally_reserved":0,"allocated":9424896,"data_stored":4874832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6480,"internal_metadata":53470896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003564"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776944+0000","last_change":"2021-12-15T17:17:49.441711+0000","last_active":"2021-12-15T17:17:54.776944+0000","last_peered":"2021-12-15T17:17:54.776944+0000","last_clean":"2021-12-15T17:17:54.776944+0000","last_became_active":"2021-12-15T17:17:49.441429+0000","last_became_peered":"2021-12-15T17:17:49.441429+0000","last_unstale":"2021-12-15T17:17:54.776944+0000","last_undegraded":"2021-12-15T17:17:54.776944+0000","last_fullsized":"2021-12-15T17:17:54.776944+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:39:00.644601+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829506+0000","last_change":"2021-12-15T17:17:47.436682+0000","last_active":"2021-12-15T17:17:54.829506+0000","last_peered":"2021-12-15T17:17:54.829506+0000","last_clean":"2021-12-15T17:17:54.829506+0000","last_became_active":"2021-12-15T17:17:47.436432+0000","last_became_peered":"2021-12-15T17:17:47.436432+0000","last_unstale":"2021-12-15T17:17:54.829506+0000","last_undegraded":"2021-12-15T17:17:54.829506+0000","last_fullsized":"2021-12-15T17:17:54.829506+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:07:42.742385+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778067+0000","last_change":"2021-12-15T17:17:43.412256+0000","last_active":"2021-12-15T17:17:54.778067+0000","last_peered":"2021-12-15T17:17:54.778067+0000","last_clean":"2021-12-15T17:17:54.778067+0000","last_became_active":"2021-12-15T17:17:43.411941+0000","last_became_peered":"2021-12-15T17:17:43.411941+0000","last_unstale":"2021-12-15T17:17:54.778067+0000","last_undegraded":"2021-12-15T17:17:54.778067+0000","last_fullsized":"2021-12-15T17:17:54.778067+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:09:19.798323+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776821+0000","last_change":"2021-12-15T17:17:45.419334+0000","last_active":"2021-12-15T17:17:54.776821+0000","last_peered":"2021-12-15T17:17:54.776821+0000","last_clean":"2021-12-15T17:17:54.776821+0000","last_became_active":"2021-12-15T17:17:45.418985+0000","last_became_peered":"2021-12-15T17:17:45.418985+0000","last_unstale":"2021-12-15T17:17:54.776821+0000","last_undegraded":"2021-12-15T17:17:54.776821+0000","last_fullsized":"2021-12-15T17:17:54.776821+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:30:49.171533+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777564+0000","last_change":"2021-12-15T17:17:49.731564+0000","last_active":"2021-12-15T17:17:54.777564+0000","last_peered":"2021-12-15T17:17:54.777564+0000","last_clean":"2021-12-15T17:17:54.777564+0000","last_became_active":"2021-12-15T17:17:49.731160+0000","last_became_peered":"2021-12-15T17:17:49.731160+0000","last_unstale":"2021-12-15T17:17:54.777564+0000","last_undegraded":"2021-12-15T17:17:54.777564+0000","last_fullsized":"2021-12-15T17:17:54.777564+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:03:28.483057+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776838+0000","last_change":"2021-12-15T17:17:47.436913+0000","last_active":"2021-12-15T17:17:54.776838+0000","last_peered":"2021-12-15T17:17:54.776838+0000","last_clean":"2021-12-15T17:17:54.776838+0000","last_became_active":"2021-12-15T17:17:47.436609+0000","last_became_peered":"2021-12-15T17:17:47.436609+0000","last_unstale":"2021-12-15T17:17:54.776838+0000","last_undegraded":"2021-12-15T17:17:54.776838+0000","last_fullsized":"2021-12-15T17:17:54.776838+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:59:13.782727+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778323+0000","last_change":"2021-12-15T17:17:43.414926+0000","last_active":"2021-12-15T17:17:54.778323+0000","last_peered":"2021-12-15T17:17:54.778323+0000","last_clean":"2021-12-15T17:17:54.778323+0000","last_became_active":"2021-12-15T17:17:43.414585+0000","last_became_peered":"2021-12-15T17:17:43.414585+0000","last_unstale":"2021-12-15T17:17:54.778323+0000","last_undegraded":"2021-12-15T17:17:54.778323+0000","last_fullsized":"2021-12-15T17:17:54.778323+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:48:33.892199+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779143+0000","last_change":"2021-12-15T17:17:45.426575+0000","last_active":"2021-12-15T17:17:54.779143+0000","last_peered":"2021-12-15T17:17:54.779143+0000","last_clean":"2021-12-15T17:17:54.779143+0000","last_became_active":"2021-12-15T17:17:45.425469+0000","last_became_peered":"2021-12-15T17:17:45.425469+0000","last_unstale":"2021-12-15T17:17:54.779143+0000","last_undegraded":"2021-12-15T17:17:54.779143+0000","last_fullsized":"2021-12-15T17:17:54.779143+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:20:21.028242+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776797+0000","last_change":"2021-12-15T17:17:49.442795+0000","last_active":"2021-12-15T17:17:54.776797+0000","last_peered":"2021-12-15T17:17:54.776797+0000","last_clean":"2021-12-15T17:17:54.776797+0000","last_became_active":"2021-12-15T17:17:49.442246+0000","last_became_peered":"2021-12-15T17:17:49.442246+0000","last_unstale":"2021-12-15T17:17:54.776797+0000","last_undegraded":"2021-12-15T17:17:54.776797+0000","last_fullsized":"2021-12-15T17:17:54.776797+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:47:58.984541+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830722+0000","last_change":"2021-12-15T17:17:47.431969+0000","last_active":"2021-12-15T17:17:54.830722+0000","last_peered":"2021-12-15T17:17:54.830722+0000","last_clean":"2021-12-15T17:17:54.830722+0000","last_became_active":"2021-12-15T17:17:47.431473+0000","last_became_peered":"2021-12-15T17:17:47.431473+0000","last_unstale":"2021-12-15T17:17:54.830722+0000","last_undegraded":"2021-12-15T17:17:54.830722+0000","last_fullsized":"2021-12-15T17:17:54.830722+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:17:28.771149+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830928+0000","last_change":"2021-12-15T17:17:43.418183+0000","last_active":"2021-12-15T17:17:54.830928+0000","last_peered":"2021-12-15T17:17:54.830928+0000","last_clean":"2021-12-15T17:17:54.830928+0000","last_became_active":"2021-12-15T17:17:43.417807+0000","last_became_peered":"2021-12-15T17:17:43.417807+0000","last_unstale":"2021-12-15T17:17:54.830928+0000","last_undegraded":"2021-12-15T17:17:54.830928+0000","last_fullsized":"2021-12-15T17:17:54.830928+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:57:40.445606+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777070+0000","last_change":"2021-12-15T17:17:45.423371+0000","last_active":"2021-12-15T17:17:54.777070+0000","last_peered":"2021-12-15T17:17:54.777070+0000","last_clean":"2021-12-15T17:17:54.777070+0000","last_became_active":"2021-12-15T17:17:45.423018+0000","last_became_peered":"2021-12-15T17:17:45.423018+0000","last_unstale":"2021-12-15T17:17:54.777070+0000","last_undegraded":"2021-12-15T17:17:54.777070+0000","last_fullsized":"2021-12-15T17:17:54.777070+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:10:13.101257+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829170+0000","last_change":"2021-12-15T17:17:49.442447+0000","last_active":"2021-12-15T17:17:54.829170+0000","last_peered":"2021-12-15T17:17:54.829170+0000","last_clean":"2021-12-15T17:17:54.829170+0000","last_became_active":"2021-12-15T17:17:49.442167+0000","last_became_peered":"2021-12-15T17:17:49.442167+0000","last_unstale":"2021-12-15T17:17:54.829170+0000","last_undegraded":"2021-12-15T17:17:54.829170+0000","last_fullsized":"2021-12-15T17:17:54.829170+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:22:15.562970+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830546+0000","last_change":"2021-12-15T17:17:47.438208+0000","last_active":"2021-12-15T17:17:54.830546+0000","last_peered":"2021-12-15T17:17:54.830546+0000","last_clean":"2021-12-15T17:17:54.830546+0000","last_became_active":"2021-12-15T17:17:47.437950+0000","last_became_peered":"2021-12-15T17:17:47.437950+0000","last_unstale":"2021-12-15T17:17:54.830546+0000","last_undegraded":"2021-12-15T17:17:54.830546+0000","last_fullsized":"2021-12-15T17:17:54.830546+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:10:36.042297+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829639+0000","last_change":"2021-12-15T17:17:43.416039+0000","last_active":"2021-12-15T17:17:54.829639+0000","last_peered":"2021-12-15T17:17:54.829639+0000","last_clean":"2021-12-15T17:17:54.829639+0000","last_became_active":"2021-12-15T17:17:43.415706+0000","last_became_peered":"2021-12-15T17:17:43.415706+0000","last_unstale":"2021-12-15T17:17:54.829639+0000","last_undegraded":"2021-12-15T17:17:54.829639+0000","last_fullsized":"2021-12-15T17:17:54.829639+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:37:54.602918+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"100'5","reported_seq":27,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777255+0000","last_change":"2021-12-15T17:17:45.419389+0000","last_active":"2021-12-15T17:17:54.777255+0000","last_peered":"2021-12-15T17:17:54.777255+0000","last_clean":"2021-12-15T17:17:54.777255+0000","last_became_active":"2021-12-15T17:17:45.419069+0000","last_became_peered":"2021-12-15T17:17:45.419069+0000","last_unstale":"2021-12-15T17:17:54.777255+0000","last_undegraded":"2021-12-15T17:17:54.777255+0000","last_fullsized":"2021-12-15T17:17:54.777255+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:09:28.698033+0000","stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777396+0000","last_change":"2021-12-15T17:17:49.731380+0000","last_active":"2021-12-15T17:17:54.777396+0000","last_peered":"2021-12-15T17:17:54.777396+0000","last_clean":"2021-12-15T17:17:54.777396+0000","last_became_active":"2021-12-15T17:17:49.731039+0000","last_became_peered":"2021-12-15T17:17:49.731039+0000","last_unstale":"2021-12-15T17:17:54.777396+0000","last_undegraded":"2021-12-15T17:17:54.777396+0000","last_fullsized":"2021-12-15T17:17:54.777396+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:37:10.446853+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830216+0000","last_change":"2021-12-15T17:17:47.439549+0000","last_active":"2021-12-15T17:17:54.830216+0000","last_peered":"2021-12-15T17:17:54.830216+0000","last_clean":"2021-12-15T17:17:54.830216+0000","last_became_active":"2021-12-15T17:17:47.439031+0000","last_became_peered":"2021-12-15T17:17:47.439031+0000","last_unstale":"2021-12-15T17:17:54.830216+0000","last_undegraded":"2021-12-15T17:17:54.830216+0000","last_fullsized":"2021-12-15T17:17:54.830216+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:11:19.927526+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778621+0000","last_change":"2021-12-15T17:17:43.416573+0000","last_active":"2021-12-15T17:17:54.778621+0000","last_peered":"2021-12-15T17:17:54.778621+0000","last_clean":"2021-12-15T17:17:54.778621+0000","last_became_active":"2021-12-15T17:17:43.416285+0000","last_became_peered":"2021-12-15T17:17:43.416285+0000","last_unstale":"2021-12-15T17:17:54.778621+0000","last_undegraded":"2021-12-15T17:17:54.778621+0000","last_fullsized":"2021-12-15T17:17:54.778621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:58:58.992557+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830551+0000","last_change":"2021-12-15T17:17:45.420350+0000","last_active":"2021-12-15T17:17:54.830551+0000","last_peered":"2021-12-15T17:17:54.830551+0000","last_clean":"2021-12-15T17:17:54.830551+0000","last_became_active":"2021-12-15T17:17:45.420013+0000","last_became_peered":"2021-12-15T17:17:45.420013+0000","last_unstale":"2021-12-15T17:17:54.830551+0000","last_undegraded":"2021-12-15T17:17:54.830551+0000","last_fullsized":"2021-12-15T17:17:54.830551+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:50:04.023775+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778437+0000","last_change":"2021-12-15T17:17:49.732358+0000","last_active":"2021-12-15T17:17:54.778437+0000","last_peered":"2021-12-15T17:17:54.778437+0000","last_clean":"2021-12-15T17:17:54.778437+0000","last_became_active":"2021-12-15T17:17:49.731857+0000","last_became_peered":"2021-12-15T17:17:49.731857+0000","last_unstale":"2021-12-15T17:17:54.778437+0000","last_undegraded":"2021-12-15T17:17:54.778437+0000","last_fullsized":"2021-12-15T17:17:54.778437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:05:10.519251+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777230+0000","last_change":"2021-12-15T17:17:47.432479+0000","last_active":"2021-12-15T17:17:54.777230+0000","last_peered":"2021-12-15T17:17:54.777230+0000","last_clean":"2021-12-15T17:17:54.777230+0000","last_became_active":"2021-12-15T17:17:47.431115+0000","last_became_peered":"2021-12-15T17:17:47.431115+0000","last_unstale":"2021-12-15T17:17:54.777230+0000","last_undegraded":"2021-12-15T17:17:54.777230+0000","last_fullsized":"2021-12-15T17:17:54.777230+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:01:39.879259+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777101+0000","last_change":"2021-12-15T17:17:43.413035+0000","last_active":"2021-12-15T17:17:54.777101+0000","last_peered":"2021-12-15T17:17:54.777101+0000","last_clean":"2021-12-15T17:17:54.777101+0000","last_became_active":"2021-12-15T17:17:43.412718+0000","last_became_peered":"2021-12-15T17:17:43.412718+0000","last_unstale":"2021-12-15T17:17:54.777101+0000","last_undegraded":"2021-12-15T17:17:54.777101+0000","last_fullsized":"2021-12-15T17:17:54.777101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:12:50.829486+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"100'12","reported_seq":40,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778328+0000","last_change":"2021-12-15T17:17:45.422277+0000","last_active":"2021-12-15T17:17:54.778328+0000","last_peered":"2021-12-15T17:17:54.778328+0000","last_clean":"2021-12-15T17:17:54.778328+0000","last_became_active":"2021-12-15T17:17:45.421957+0000","last_became_peered":"2021-12-15T17:17:45.421957+0000","last_unstale":"2021-12-15T17:17:54.778328+0000","last_undegraded":"2021-12-15T17:17:54.778328+0000","last_fullsized":"2021-12-15T17:17:54.778328+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:12:54.877225+0000","stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"100'1","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831365+0000","last_change":"2021-12-15T17:17:49.440790+0000","last_active":"2021-12-15T17:17:54.831365+0000","last_peered":"2021-12-15T17:17:54.831365+0000","last_clean":"2021-12-15T17:17:54.831365+0000","last_became_active":"2021-12-15T17:17:49.440496+0000","last_became_peered":"2021-12-15T17:17:49.440496+0000","last_unstale":"2021-12-15T17:17:54.831365+0000","last_undegraded":"2021-12-15T17:17:54.831365+0000","last_fullsized":"2021-12-15T17:17:54.831365+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:43:52.309837+0000","stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'8","reported_seq":38,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:36.720373+0000","last_change":"2021-12-15T17:17:47.436945+0000","last_active":"2021-12-15T17:18:36.720373+0000","last_peered":"2021-12-15T17:18:36.720373+0000","last_clean":"2021-12-15T17:18:36.720373+0000","last_became_active":"2021-12-15T17:17:47.436505+0000","last_became_peered":"2021-12-15T17:17:47.436505+0000","last_unstale":"2021-12-15T17:18:36.720373+0000","last_undegraded":"2021-12-15T17:18:36.720373+0000","last_fullsized":"2021-12-15T17:18:36.720373+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:41:06.698413+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829594+0000","last_change":"2021-12-15T17:17:43.406919+0000","last_active":"2021-12-15T17:17:54.829594+0000","last_peered":"2021-12-15T17:17:54.829594+0000","last_clean":"2021-12-15T17:17:54.829594+0000","last_became_active":"2021-12-15T17:17:43.406558+0000","last_became_peered":"2021-12-15T17:17:43.406558+0000","last_unstale":"2021-12-15T17:17:54.829594+0000","last_undegraded":"2021-12-15T17:17:54.829594+0000","last_fullsized":"2021-12-15T17:17:54.829594+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:16:48.133094+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777463+0000","last_change":"2021-12-15T17:17:45.573216+0000","last_active":"2021-12-15T17:17:54.777463+0000","last_peered":"2021-12-15T17:17:54.777463+0000","last_clean":"2021-12-15T17:17:54.777463+0000","last_became_active":"2021-12-15T17:17:45.572876+0000","last_became_peered":"2021-12-15T17:17:45.572876+0000","last_unstale":"2021-12-15T17:17:54.777463+0000","last_undegraded":"2021-12-15T17:17:54.777463+0000","last_fullsized":"2021-12-15T17:17:54.777463+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:32:00.102468+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830349+0000","last_change":"2021-12-15T17:17:49.441787+0000","last_active":"2021-12-15T17:17:54.830349+0000","last_peered":"2021-12-15T17:17:54.830349+0000","last_clean":"2021-12-15T17:17:54.830349+0000","last_became_active":"2021-12-15T17:17:49.441409+0000","last_became_peered":"2021-12-15T17:17:49.441409+0000","last_unstale":"2021-12-15T17:17:54.830349+0000","last_undegraded":"2021-12-15T17:17:54.830349+0000","last_fullsized":"2021-12-15T17:17:54.830349+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:29:43.861392+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777738+0000","last_change":"2021-12-15T17:17:47.431872+0000","last_active":"2021-12-15T17:17:54.777738+0000","last_peered":"2021-12-15T17:17:54.777738+0000","last_clean":"2021-12-15T17:17:54.777738+0000","last_became_active":"2021-12-15T17:17:47.431536+0000","last_became_peered":"2021-12-15T17:17:47.431536+0000","last_unstale":"2021-12-15T17:17:54.777738+0000","last_undegraded":"2021-12-15T17:17:54.777738+0000","last_fullsized":"2021-12-15T17:17:54.777738+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:49:15.183095+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776638+0000","last_change":"2021-12-15T17:17:43.405875+0000","last_active":"2021-12-15T17:17:54.776638+0000","last_peered":"2021-12-15T17:17:54.776638+0000","last_clean":"2021-12-15T17:17:54.776638+0000","last_became_active":"2021-12-15T17:17:43.405547+0000","last_became_peered":"2021-12-15T17:17:43.405547+0000","last_unstale":"2021-12-15T17:17:54.776638+0000","last_undegraded":"2021-12-15T17:17:54.776638+0000","last_fullsized":"2021-12-15T17:17:54.776638+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:30:18.838252+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829907+0000","last_change":"2021-12-15T17:17:45.425045+0000","last_active":"2021-12-15T17:17:54.829907+0000","last_peered":"2021-12-15T17:17:54.829907+0000","last_clean":"2021-12-15T17:17:54.829907+0000","last_became_active":"2021-12-15T17:17:45.424729+0000","last_became_peered":"2021-12-15T17:17:45.424729+0000","last_unstale":"2021-12-15T17:17:54.829907+0000","last_undegraded":"2021-12-15T17:17:54.829907+0000","last_fullsized":"2021-12-15T17:17:54.829907+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:20:38.839206+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830452+0000","last_change":"2021-12-15T17:17:49.439633+0000","last_active":"2021-12-15T17:17:54.830452+0000","last_peered":"2021-12-15T17:17:54.830452+0000","last_clean":"2021-12-15T17:17:54.830452+0000","last_became_active":"2021-12-15T17:17:49.439201+0000","last_became_peered":"2021-12-15T17:17:49.439201+0000","last_unstale":"2021-12-15T17:17:54.830452+0000","last_undegraded":"2021-12-15T17:17:54.830452+0000","last_fullsized":"2021-12-15T17:17:54.830452+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:26:39.843841+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830170+0000","last_change":"2021-12-15T17:17:47.442769+0000","last_active":"2021-12-15T17:17:54.830170+0000","last_peered":"2021-12-15T17:17:54.830170+0000","last_clean":"2021-12-15T17:17:54.830170+0000","last_became_active":"2021-12-15T17:17:47.442448+0000","last_became_peered":"2021-12-15T17:17:47.442448+0000","last_unstale":"2021-12-15T17:17:54.830170+0000","last_undegraded":"2021-12-15T17:17:54.830170+0000","last_fullsized":"2021-12-15T17:17:54.830170+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:05:35.027221+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777974+0000","last_change":"2021-12-15T17:17:43.412365+0000","last_active":"2021-12-15T17:17:54.777974+0000","last_peered":"2021-12-15T17:17:54.777974+0000","last_clean":"2021-12-15T17:17:54.777974+0000","last_became_active":"2021-12-15T17:17:43.412071+0000","last_became_peered":"2021-12-15T17:17:43.412071+0000","last_unstale":"2021-12-15T17:17:54.777974+0000","last_undegraded":"2021-12-15T17:17:54.777974+0000","last_fullsized":"2021-12-15T17:17:54.777974+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:38:22.706215+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'4","reported_seq":23,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778597+0000","last_change":"2021-12-15T17:17:45.419024+0000","last_active":"2021-12-15T17:17:54.778597+0000","last_peered":"2021-12-15T17:17:54.778597+0000","last_clean":"2021-12-15T17:17:54.778597+0000","last_became_active":"2021-12-15T17:17:45.418608+0000","last_became_peered":"2021-12-15T17:17:45.418608+0000","last_unstale":"2021-12-15T17:17:54.778597+0000","last_undegraded":"2021-12-15T17:17:54.778597+0000","last_fullsized":"2021-12-15T17:17:54.778597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:22:00.679507+0000","stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777332+0000","last_change":"2021-12-15T17:17:49.731793+0000","last_active":"2021-12-15T17:17:54.777332+0000","last_peered":"2021-12-15T17:17:54.777332+0000","last_clean":"2021-12-15T17:17:54.777332+0000","last_became_active":"2021-12-15T17:17:49.731393+0000","last_became_peered":"2021-12-15T17:17:49.731393+0000","last_unstale":"2021-12-15T17:17:54.777332+0000","last_undegraded":"2021-12-15T17:17:54.777332+0000","last_fullsized":"2021-12-15T17:17:54.777332+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:03:56.411886+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830598+0000","last_change":"2021-12-15T17:17:47.432912+0000","last_active":"2021-12-15T17:17:54.830598+0000","last_peered":"2021-12-15T17:17:54.830598+0000","last_clean":"2021-12-15T17:17:54.830598+0000","last_became_active":"2021-12-15T17:17:47.432604+0000","last_became_peered":"2021-12-15T17:17:47.432604+0000","last_unstale":"2021-12-15T17:17:54.830598+0000","last_undegraded":"2021-12-15T17:17:54.830598+0000","last_fullsized":"2021-12-15T17:17:54.830598+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:07:09.892394+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830116+0000","last_change":"2021-12-15T17:17:43.415192+0000","last_active":"2021-12-15T17:17:54.830116+0000","last_peered":"2021-12-15T17:17:54.830116+0000","last_clean":"2021-12-15T17:17:54.830116+0000","last_became_active":"2021-12-15T17:17:43.414904+0000","last_became_peered":"2021-12-15T17:17:43.414904+0000","last_unstale":"2021-12-15T17:17:54.830116+0000","last_undegraded":"2021-12-15T17:17:54.830116+0000","last_fullsized":"2021-12-15T17:17:54.830116+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:03:06.583030+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779090+0000","last_change":"2021-12-15T17:17:45.573138+0000","last_active":"2021-12-15T17:17:54.779090+0000","last_peered":"2021-12-15T17:17:54.779090+0000","last_clean":"2021-12-15T17:17:54.779090+0000","last_became_active":"2021-12-15T17:17:45.572449+0000","last_became_peered":"2021-12-15T17:17:45.572449+0000","last_unstale":"2021-12-15T17:17:54.779090+0000","last_undegraded":"2021-12-15T17:17:54.779090+0000","last_fullsized":"2021-12-15T17:17:54.779090+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T05:13:53.844783+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776501+0000","last_change":"2021-12-15T17:17:49.442586+0000","last_active":"2021-12-15T17:17:54.776501+0000","last_peered":"2021-12-15T17:17:54.776501+0000","last_clean":"2021-12-15T17:17:54.776501+0000","last_became_active":"2021-12-15T17:17:49.441784+0000","last_became_peered":"2021-12-15T17:17:49.441784+0000","last_unstale":"2021-12-15T17:17:54.776501+0000","last_undegraded":"2021-12-15T17:17:54.776501+0000","last_fullsized":"2021-12-15T17:17:54.776501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:39:29.966309+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777658+0000","last_change":"2021-12-15T17:17:47.435697+0000","last_active":"2021-12-15T17:17:54.777658+0000","last_peered":"2021-12-15T17:17:54.777658+0000","last_clean":"2021-12-15T17:17:54.777658+0000","last_became_active":"2021-12-15T17:17:47.435315+0000","last_became_peered":"2021-12-15T17:17:47.435315+0000","last_unstale":"2021-12-15T17:17:54.777658+0000","last_undegraded":"2021-12-15T17:17:54.777658+0000","last_fullsized":"2021-12-15T17:17:54.777658+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:16:04.371863+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829315+0000","last_change":"2021-12-15T17:17:43.415619+0000","last_active":"2021-12-15T17:17:54.829315+0000","last_peered":"2021-12-15T17:17:54.829315+0000","last_clean":"2021-12-15T17:17:54.829315+0000","last_became_active":"2021-12-15T17:17:43.414683+0000","last_became_peered":"2021-12-15T17:17:43.414683+0000","last_unstale":"2021-12-15T17:17:54.829315+0000","last_undegraded":"2021-12-15T17:17:54.829315+0000","last_fullsized":"2021-12-15T17:17:54.829315+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:42:47.257062+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829962+0000","last_change":"2021-12-15T17:17:45.424983+0000","last_active":"2021-12-15T17:17:54.829962+0000","last_peered":"2021-12-15T17:17:54.829962+0000","last_clean":"2021-12-15T17:17:54.829962+0000","last_became_active":"2021-12-15T17:17:45.424600+0000","last_became_peered":"2021-12-15T17:17:45.424600+0000","last_unstale":"2021-12-15T17:17:54.829962+0000","last_undegraded":"2021-12-15T17:17:54.829962+0000","last_fullsized":"2021-12-15T17:17:54.829962+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:24:51.543541+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778788+0000","last_change":"2021-12-15T17:17:49.435850+0000","last_active":"2021-12-15T17:17:54.778788+0000","last_peered":"2021-12-15T17:17:54.778788+0000","last_clean":"2021-12-15T17:17:54.778788+0000","last_became_active":"2021-12-15T17:17:49.435474+0000","last_became_peered":"2021-12-15T17:17:49.435474+0000","last_unstale":"2021-12-15T17:17:54.778788+0000","last_undegraded":"2021-12-15T17:17:54.778788+0000","last_fullsized":"2021-12-15T17:17:54.778788+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:24:49.786695+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829866+0000","last_change":"2021-12-15T17:17:47.437623+0000","last_active":"2021-12-15T17:17:54.829866+0000","last_peered":"2021-12-15T17:17:54.829866+0000","last_clean":"2021-12-15T17:17:54.829866+0000","last_became_active":"2021-12-15T17:17:47.437243+0000","last_became_peered":"2021-12-15T17:17:47.437243+0000","last_unstale":"2021-12-15T17:17:54.829866+0000","last_undegraded":"2021-12-15T17:17:54.829866+0000","last_fullsized":"2021-12-15T17:17:54.829866+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:43:35.789713+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777417+0000","last_change":"2021-12-15T17:17:43.418873+0000","last_active":"2021-12-15T17:17:54.777417+0000","last_peered":"2021-12-15T17:17:54.777417+0000","last_clean":"2021-12-15T17:17:54.777417+0000","last_became_active":"2021-12-15T17:17:43.418572+0000","last_became_peered":"2021-12-15T17:17:43.418572+0000","last_unstale":"2021-12-15T17:17:54.777417+0000","last_undegraded":"2021-12-15T17:17:54.777417+0000","last_fullsized":"2021-12-15T17:17:54.777417+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:21:49.546873+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777399+0000","last_change":"2021-12-15T17:17:45.419509+0000","last_active":"2021-12-15T17:17:54.777399+0000","last_peered":"2021-12-15T17:17:54.777399+0000","last_clean":"2021-12-15T17:17:54.777399+0000","last_became_active":"2021-12-15T17:17:45.419095+0000","last_became_peered":"2021-12-15T17:17:45.419095+0000","last_unstale":"2021-12-15T17:17:54.777399+0000","last_undegraded":"2021-12-15T17:17:54.777399+0000","last_fullsized":"2021-12-15T17:17:54.777399+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:48:51.809132+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777165+0000","last_change":"2021-12-15T17:17:49.441156+0000","last_active":"2021-12-15T17:17:54.777165+0000","last_peered":"2021-12-15T17:17:54.777165+0000","last_clean":"2021-12-15T17:17:54.777165+0000","last_became_active":"2021-12-15T17:17:49.440837+0000","last_became_peered":"2021-12-15T17:17:49.440837+0000","last_unstale":"2021-12-15T17:17:54.777165+0000","last_undegraded":"2021-12-15T17:17:54.777165+0000","last_fullsized":"2021-12-15T17:17:54.777165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:21:36.995185+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"100'8","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:36.720401+0000","last_change":"2021-12-15T17:17:47.438906+0000","last_active":"2021-12-15T17:18:36.720401+0000","last_peered":"2021-12-15T17:18:36.720401+0000","last_clean":"2021-12-15T17:18:36.720401+0000","last_became_active":"2021-12-15T17:17:47.438599+0000","last_became_peered":"2021-12-15T17:17:47.438599+0000","last_unstale":"2021-12-15T17:18:36.720401+0000","last_undegraded":"2021-12-15T17:18:36.720401+0000","last_fullsized":"2021-12-15T17:18:36.720401+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:53:07.715077+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830563+0000","last_change":"2021-12-15T17:17:43.413938+0000","last_active":"2021-12-15T17:17:54.830563+0000","last_peered":"2021-12-15T17:17:54.830563+0000","last_clean":"2021-12-15T17:17:54.830563+0000","last_became_active":"2021-12-15T17:17:43.413646+0000","last_became_peered":"2021-12-15T17:17:43.413646+0000","last_unstale":"2021-12-15T17:17:54.830563+0000","last_undegraded":"2021-12-15T17:17:54.830563+0000","last_fullsized":"2021-12-15T17:17:54.830563+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:58:39.548600+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779379+0000","last_change":"2021-12-15T17:17:45.572845+0000","last_active":"2021-12-15T17:17:54.779379+0000","last_peered":"2021-12-15T17:17:54.779379+0000","last_clean":"2021-12-15T17:17:54.779379+0000","last_became_active":"2021-12-15T17:17:45.572502+0000","last_became_peered":"2021-12-15T17:17:45.572502+0000","last_unstale":"2021-12-15T17:17:54.779379+0000","last_undegraded":"2021-12-15T17:17:54.779379+0000","last_fullsized":"2021-12-15T17:17:54.779379+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:32:48.287963+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777682+0000","last_change":"2021-12-15T17:17:49.438435+0000","last_active":"2021-12-15T17:17:54.777682+0000","last_peered":"2021-12-15T17:17:54.777682+0000","last_clean":"2021-12-15T17:17:54.777682+0000","last_became_active":"2021-12-15T17:17:49.437861+0000","last_became_peered":"2021-12-15T17:17:49.437861+0000","last_unstale":"2021-12-15T17:17:54.777682+0000","last_undegraded":"2021-12-15T17:17:54.777682+0000","last_fullsized":"2021-12-15T17:17:54.777682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:58:59.354570+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"100'8","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:36.720283+0000","last_change":"2021-12-15T17:17:47.427960+0000","last_active":"2021-12-15T17:18:36.720283+0000","last_peered":"2021-12-15T17:18:36.720283+0000","last_clean":"2021-12-15T17:18:36.720283+0000","last_became_active":"2021-12-15T17:17:47.427581+0000","last_became_peered":"2021-12-15T17:17:47.427581+0000","last_unstale":"2021-12-15T17:18:36.720283+0000","last_undegraded":"2021-12-15T17:18:36.720283+0000","last_fullsized":"2021-12-15T17:18:36.720283+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:49:44.704651+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776951+0000","last_change":"2021-12-15T17:17:43.417472+0000","last_active":"2021-12-15T17:17:54.776951+0000","last_peered":"2021-12-15T17:17:54.776951+0000","last_clean":"2021-12-15T17:17:54.776951+0000","last_became_active":"2021-12-15T17:17:43.417151+0000","last_became_peered":"2021-12-15T17:17:43.417151+0000","last_unstale":"2021-12-15T17:17:54.776951+0000","last_undegraded":"2021-12-15T17:17:54.776951+0000","last_fullsized":"2021-12-15T17:17:54.776951+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:24:16.283189+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831118+0000","last_change":"2021-12-15T17:17:45.573975+0000","last_active":"2021-12-15T17:17:54.831118+0000","last_peered":"2021-12-15T17:17:54.831118+0000","last_clean":"2021-12-15T17:17:54.831118+0000","last_became_active":"2021-12-15T17:17:45.573645+0000","last_became_peered":"2021-12-15T17:17:45.573645+0000","last_unstale":"2021-12-15T17:17:54.831118+0000","last_undegraded":"2021-12-15T17:17:54.831118+0000","last_fullsized":"2021-12-15T17:17:54.831118+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:06:50.053349+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830784+0000","last_change":"2021-12-15T17:17:49.437708+0000","last_active":"2021-12-15T17:17:54.830784+0000","last_peered":"2021-12-15T17:17:54.830784+0000","last_clean":"2021-12-15T17:17:54.830784+0000","last_became_active":"2021-12-15T17:17:49.437412+0000","last_became_peered":"2021-12-15T17:17:49.437412+0000","last_unstale":"2021-12-15T17:17:54.830784+0000","last_undegraded":"2021-12-15T17:17:54.830784+0000","last_fullsized":"2021-12-15T17:17:54.830784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:39:36.145667+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778341+0000","last_change":"2021-12-15T17:17:47.435731+0000","last_active":"2021-12-15T17:17:54.778341+0000","last_peered":"2021-12-15T17:17:54.778341+0000","last_clean":"2021-12-15T17:17:54.778341+0000","last_became_active":"2021-12-15T17:17:47.435345+0000","last_became_peered":"2021-12-15T17:17:47.435345+0000","last_unstale":"2021-12-15T17:17:54.778341+0000","last_undegraded":"2021-12-15T17:17:54.778341+0000","last_fullsized":"2021-12-15T17:17:54.778341+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:14:56.225151+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831692+0000","last_change":"2021-12-15T17:17:43.415236+0000","last_active":"2021-12-15T17:17:54.831692+0000","last_peered":"2021-12-15T17:17:54.831692+0000","last_clean":"2021-12-15T17:17:54.831692+0000","last_became_active":"2021-12-15T17:17:43.414935+0000","last_became_peered":"2021-12-15T17:17:43.414935+0000","last_unstale":"2021-12-15T17:17:54.831692+0000","last_undegraded":"2021-12-15T17:17:54.831692+0000","last_fullsized":"2021-12-15T17:17:54.831692+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:46:42.657026+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777566+0000","last_change":"2021-12-15T17:17:45.574038+0000","last_active":"2021-12-15T17:17:54.777566+0000","last_peered":"2021-12-15T17:17:54.777566+0000","last_clean":"2021-12-15T17:17:54.777566+0000","last_became_active":"2021-12-15T17:17:45.573651+0000","last_became_peered":"2021-12-15T17:17:45.573651+0000","last_unstale":"2021-12-15T17:17:54.777566+0000","last_undegraded":"2021-12-15T17:17:54.777566+0000","last_fullsized":"2021-12-15T17:17:54.777566+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:25:25.200521+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829227+0000","last_change":"2021-12-15T17:17:49.732632+0000","last_active":"2021-12-15T17:17:54.829227+0000","last_peered":"2021-12-15T17:17:54.829227+0000","last_clean":"2021-12-15T17:17:54.829227+0000","last_became_active":"2021-12-15T17:17:49.732209+0000","last_became_peered":"2021-12-15T17:17:49.732209+0000","last_unstale":"2021-12-15T17:17:54.829227+0000","last_undegraded":"2021-12-15T17:17:54.829227+0000","last_fullsized":"2021-12-15T17:17:54.829227+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:09:18.301336+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830040+0000","last_change":"2021-12-15T17:17:47.432909+0000","last_active":"2021-12-15T17:17:54.830040+0000","last_peered":"2021-12-15T17:17:54.830040+0000","last_clean":"2021-12-15T17:17:54.830040+0000","last_became_active":"2021-12-15T17:17:47.432495+0000","last_became_peered":"2021-12-15T17:17:47.432495+0000","last_unstale":"2021-12-15T17:17:54.830040+0000","last_undegraded":"2021-12-15T17:17:54.830040+0000","last_fullsized":"2021-12-15T17:17:54.830040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:32:55.501940+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829033+0000","last_change":"2021-12-15T17:17:43.415983+0000","last_active":"2021-12-15T17:17:54.829033+0000","last_peered":"2021-12-15T17:17:54.829033+0000","last_clean":"2021-12-15T17:17:54.829033+0000","last_became_active":"2021-12-15T17:17:43.415688+0000","last_became_peered":"2021-12-15T17:17:43.415688+0000","last_unstale":"2021-12-15T17:17:54.829033+0000","last_undegraded":"2021-12-15T17:17:54.829033+0000","last_fullsized":"2021-12-15T17:17:54.829033+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:21:45.430437+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"100'6","reported_seq":26,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778508+0000","last_change":"2021-12-15T17:17:45.574024+0000","last_active":"2021-12-15T17:17:54.778508+0000","last_peered":"2021-12-15T17:17:54.778508+0000","last_clean":"2021-12-15T17:17:54.778508+0000","last_became_active":"2021-12-15T17:17:45.573689+0000","last_became_peered":"2021-12-15T17:17:45.573689+0000","last_unstale":"2021-12-15T17:17:54.778508+0000","last_undegraded":"2021-12-15T17:17:54.778508+0000","last_fullsized":"2021-12-15T17:17:54.778508+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:52:45.792453+0000","stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830704+0000","last_change":"2021-12-15T17:17:49.731214+0000","last_active":"2021-12-15T17:17:54.830704+0000","last_peered":"2021-12-15T17:17:54.830704+0000","last_clean":"2021-12-15T17:17:54.830704+0000","last_became_active":"2021-12-15T17:17:49.730894+0000","last_became_peered":"2021-12-15T17:17:49.730894+0000","last_unstale":"2021-12-15T17:17:54.830704+0000","last_undegraded":"2021-12-15T17:17:54.830704+0000","last_fullsized":"2021-12-15T17:17:54.830704+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:26:07.086508+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"92'2","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831169+0000","last_change":"2021-12-15T17:17:43.414458+0000","last_active":"2021-12-15T17:17:54.831169+0000","last_peered":"2021-12-15T17:17:54.831169+0000","last_clean":"2021-12-15T17:17:54.831169+0000","last_became_active":"2021-12-15T17:17:43.413818+0000","last_became_peered":"2021-12-15T17:17:43.413818+0000","last_unstale":"2021-12-15T17:17:54.831169+0000","last_undegraded":"2021-12-15T17:17:54.831169+0000","last_fullsized":"2021-12-15T17:17:54.831169+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:24:24.460096+0000","stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.828972+0000","last_change":"2021-12-15T17:17:45.572866+0000","last_active":"2021-12-15T17:17:54.828972+0000","last_peered":"2021-12-15T17:17:54.828972+0000","last_clean":"2021-12-15T17:17:54.828972+0000","last_became_active":"2021-12-15T17:17:45.572582+0000","last_became_peered":"2021-12-15T17:17:45.572582+0000","last_unstale":"2021-12-15T17:17:54.828972+0000","last_undegraded":"2021-12-15T17:17:54.828972+0000","last_fullsized":"2021-12-15T17:17:54.828972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:40:34.911283+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'8","reported_seq":39,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721472+0000","last_change":"2021-12-15T17:17:47.442765+0000","last_active":"2021-12-15T17:18:41.721472+0000","last_peered":"2021-12-15T17:18:41.721472+0000","last_clean":"2021-12-15T17:18:41.721472+0000","last_became_active":"2021-12-15T17:17:47.442441+0000","last_became_peered":"2021-12-15T17:17:47.442441+0000","last_unstale":"2021-12-15T17:18:41.721472+0000","last_undegraded":"2021-12-15T17:18:41.721472+0000","last_fullsized":"2021-12-15T17:18:41.721472+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:40:30.648434+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777451+0000","last_change":"2021-12-15T17:17:49.443281+0000","last_active":"2021-12-15T17:17:54.777451+0000","last_peered":"2021-12-15T17:17:54.777451+0000","last_clean":"2021-12-15T17:17:54.777451+0000","last_became_active":"2021-12-15T17:17:49.442905+0000","last_became_peered":"2021-12-15T17:17:49.442905+0000","last_unstale":"2021-12-15T17:17:54.777451+0000","last_undegraded":"2021-12-15T17:17:54.777451+0000","last_fullsized":"2021-12-15T17:17:54.777451+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:12:38.744110+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829622+0000","last_change":"2021-12-15T17:17:43.415621+0000","last_active":"2021-12-15T17:17:54.829622+0000","last_peered":"2021-12-15T17:17:54.829622+0000","last_clean":"2021-12-15T17:17:54.829622+0000","last_became_active":"2021-12-15T17:17:43.415318+0000","last_became_peered":"2021-12-15T17:17:43.415318+0000","last_unstale":"2021-12-15T17:17:54.829622+0000","last_undegraded":"2021-12-15T17:17:54.829622+0000","last_fullsized":"2021-12-15T17:17:54.829622+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:48:23.145536+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"100'12","reported_seq":40,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777261+0000","last_change":"2021-12-15T17:17:45.422470+0000","last_active":"2021-12-15T17:17:54.777261+0000","last_peered":"2021-12-15T17:17:54.777261+0000","last_clean":"2021-12-15T17:17:54.777261+0000","last_became_active":"2021-12-15T17:17:45.422161+0000","last_became_peered":"2021-12-15T17:17:45.422161+0000","last_unstale":"2021-12-15T17:17:54.777261+0000","last_undegraded":"2021-12-15T17:17:54.777261+0000","last_fullsized":"2021-12-15T17:17:54.777261+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:46:33.717392+0000","stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829100+0000","last_change":"2021-12-15T17:17:47.439693+0000","last_active":"2021-12-15T17:17:54.829100+0000","last_peered":"2021-12-15T17:17:54.829100+0000","last_clean":"2021-12-15T17:17:54.829100+0000","last_became_active":"2021-12-15T17:17:47.439356+0000","last_became_peered":"2021-12-15T17:17:47.439356+0000","last_unstale":"2021-12-15T17:17:54.829100+0000","last_undegraded":"2021-12-15T17:17:54.829100+0000","last_fullsized":"2021-12-15T17:17:54.829100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:34:31.291815+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830038+0000","last_change":"2021-12-15T17:17:49.443784+0000","last_active":"2021-12-15T17:17:54.830038+0000","last_peered":"2021-12-15T17:17:54.830038+0000","last_clean":"2021-12-15T17:17:54.830038+0000","last_became_active":"2021-12-15T17:17:49.443473+0000","last_became_peered":"2021-12-15T17:17:49.443473+0000","last_unstale":"2021-12-15T17:17:54.830038+0000","last_undegraded":"2021-12-15T17:17:54.830038+0000","last_fullsized":"2021-12-15T17:17:54.830038+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:04:48.645023+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830360+0000","last_change":"2021-12-15T17:17:47.434182+0000","last_active":"2021-12-15T17:17:54.830360+0000","last_peered":"2021-12-15T17:17:54.830360+0000","last_clean":"2021-12-15T17:17:54.830360+0000","last_became_active":"2021-12-15T17:17:47.433835+0000","last_became_peered":"2021-12-15T17:17:47.433835+0000","last_unstale":"2021-12-15T17:17:54.830360+0000","last_undegraded":"2021-12-15T17:17:54.830360+0000","last_fullsized":"2021-12-15T17:17:54.830360+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:56:47.451092+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829525+0000","last_change":"2021-12-15T17:17:43.417488+0000","last_active":"2021-12-15T17:17:54.829525+0000","last_peered":"2021-12-15T17:17:54.829525+0000","last_clean":"2021-12-15T17:17:54.829525+0000","last_became_active":"2021-12-15T17:17:43.416657+0000","last_became_peered":"2021-12-15T17:17:43.416657+0000","last_unstale":"2021-12-15T17:17:54.829525+0000","last_undegraded":"2021-12-15T17:17:54.829525+0000","last_fullsized":"2021-12-15T17:17:54.829525+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:50:12.739994+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"100'19","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830250+0000","last_change":"2021-12-15T17:17:45.573209+0000","last_active":"2021-12-15T17:17:54.830250+0000","last_peered":"2021-12-15T17:17:54.830250+0000","last_clean":"2021-12-15T17:17:54.830250+0000","last_became_active":"2021-12-15T17:17:45.572890+0000","last_became_peered":"2021-12-15T17:17:45.572890+0000","last_unstale":"2021-12-15T17:17:54.830250+0000","last_undegraded":"2021-12-15T17:17:54.830250+0000","last_fullsized":"2021-12-15T17:17:54.830250+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:03:09.778593+0000","stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776815+0000","last_change":"2021-12-15T17:17:49.441166+0000","last_active":"2021-12-15T17:17:54.776815+0000","last_peered":"2021-12-15T17:17:54.776815+0000","last_clean":"2021-12-15T17:17:54.776815+0000","last_became_active":"2021-12-15T17:17:49.440803+0000","last_became_peered":"2021-12-15T17:17:49.440803+0000","last_unstale":"2021-12-15T17:17:54.776815+0000","last_undegraded":"2021-12-15T17:17:54.776815+0000","last_fullsized":"2021-12-15T17:17:54.776815+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:02:42.913488+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829966+0000","last_change":"2021-12-15T17:17:47.431130+0000","last_active":"2021-12-15T17:17:54.829966+0000","last_peered":"2021-12-15T17:17:54.829966+0000","last_clean":"2021-12-15T17:17:54.829966+0000","last_became_active":"2021-12-15T17:17:47.430798+0000","last_became_peered":"2021-12-15T17:17:47.430798+0000","last_unstale":"2021-12-15T17:17:54.829966+0000","last_undegraded":"2021-12-15T17:17:54.829966+0000","last_fullsized":"2021-12-15T17:17:54.829966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:26:06.673982+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829714+0000","last_change":"2021-12-15T17:17:43.419864+0000","last_active":"2021-12-15T17:17:54.829714+0000","last_peered":"2021-12-15T17:17:54.829714+0000","last_clean":"2021-12-15T17:17:54.829714+0000","last_became_active":"2021-12-15T17:17:43.419576+0000","last_became_peered":"2021-12-15T17:17:43.419576+0000","last_unstale":"2021-12-15T17:17:54.829714+0000","last_undegraded":"2021-12-15T17:17:54.829714+0000","last_fullsized":"2021-12-15T17:17:54.829714+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:31:24.882855+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778541+0000","last_change":"2021-12-15T17:17:45.425317+0000","last_active":"2021-12-15T17:17:54.778541+0000","last_peered":"2021-12-15T17:17:54.778541+0000","last_clean":"2021-12-15T17:17:54.778541+0000","last_became_active":"2021-12-15T17:17:45.424975+0000","last_became_peered":"2021-12-15T17:17:45.424975+0000","last_unstale":"2021-12-15T17:17:54.778541+0000","last_undegraded":"2021-12-15T17:17:54.778541+0000","last_fullsized":"2021-12-15T17:17:54.778541+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:17:35.795389+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778017+0000","last_change":"2021-12-15T17:17:49.438882+0000","last_active":"2021-12-15T17:17:54.778017+0000","last_peered":"2021-12-15T17:17:54.778017+0000","last_clean":"2021-12-15T17:17:54.778017+0000","last_became_active":"2021-12-15T17:17:49.437700+0000","last_became_peered":"2021-12-15T17:17:49.437700+0000","last_unstale":"2021-12-15T17:17:54.778017+0000","last_undegraded":"2021-12-15T17:17:54.778017+0000","last_fullsized":"2021-12-15T17:17:54.778017+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:46:14.072600+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"95'2","reported_seq":26,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830406+0000","last_change":"2021-12-15T17:17:46.424639+0000","last_active":"2021-12-15T17:17:54.830406+0000","last_peered":"2021-12-15T17:17:54.830406+0000","last_clean":"2021-12-15T17:17:54.830406+0000","last_became_active":"2021-12-15T17:17:43.416157+0000","last_became_peered":"2021-12-15T17:17:43.416157+0000","last_unstale":"2021-12-15T17:17:54.830406+0000","last_undegraded":"2021-12-15T17:17:54.830406+0000","last_fullsized":"2021-12-15T17:17:54.830406+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:44:21.915689+0000","stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777115+0000","last_change":"2021-12-15T17:17:43.419574+0000","last_active":"2021-12-15T17:17:54.777115+0000","last_peered":"2021-12-15T17:17:54.777115+0000","last_clean":"2021-12-15T17:17:54.777115+0000","last_became_active":"2021-12-15T17:17:43.419149+0000","last_became_peered":"2021-12-15T17:17:43.419149+0000","last_unstale":"2021-12-15T17:17:54.777115+0000","last_undegraded":"2021-12-15T17:17:54.777115+0000","last_fullsized":"2021-12-15T17:17:54.777115+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:53:37.171963+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"100'28","reported_seq":77,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:36.720216+0000","last_change":"2021-12-15T17:17:45.426120+0000","last_active":"2021-12-15T17:18:36.720216+0000","last_peered":"2021-12-15T17:18:36.720216+0000","last_clean":"2021-12-15T17:18:36.720216+0000","last_became_active":"2021-12-15T17:17:45.425783+0000","last_became_peered":"2021-12-15T17:17:45.425783+0000","last_unstale":"2021-12-15T17:18:36.720216+0000","last_undegraded":"2021-12-15T17:18:36.720216+0000","last_fullsized":"2021-12-15T17:18:36.720216+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:54:20.609566+0000","stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777556+0000","last_change":"2021-12-15T17:17:47.434873+0000","last_active":"2021-12-15T17:17:54.777556+0000","last_peered":"2021-12-15T17:17:54.777556+0000","last_clean":"2021-12-15T17:17:54.777556+0000","last_became_active":"2021-12-15T17:17:47.434524+0000","last_became_peered":"2021-12-15T17:17:47.434524+0000","last_unstale":"2021-12-15T17:17:54.777556+0000","last_undegraded":"2021-12-15T17:17:54.777556+0000","last_fullsized":"2021-12-15T17:17:54.777556+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:16:49.589922+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829670+0000","last_change":"2021-12-15T17:17:49.732497+0000","last_active":"2021-12-15T17:17:54.829670+0000","last_peered":"2021-12-15T17:17:54.829670+0000","last_clean":"2021-12-15T17:17:54.829670+0000","last_became_active":"2021-12-15T17:17:49.731820+0000","last_became_peered":"2021-12-15T17:17:49.731820+0000","last_unstale":"2021-12-15T17:17:54.829670+0000","last_undegraded":"2021-12-15T17:17:54.829670+0000","last_fullsized":"2021-12-15T17:17:54.829670+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:29:34.691847+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829377+0000","last_change":"2021-12-15T17:17:43.415232+0000","last_active":"2021-12-15T17:17:54.829377+0000","last_peered":"2021-12-15T17:17:54.829377+0000","last_clean":"2021-12-15T17:17:54.829377+0000","last_became_active":"2021-12-15T17:17:43.414921+0000","last_became_peered":"2021-12-15T17:17:43.414921+0000","last_unstale":"2021-12-15T17:17:54.829377+0000","last_undegraded":"2021-12-15T17:17:54.829377+0000","last_fullsized":"2021-12-15T17:17:54.829377+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:17:42.782361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829178+0000","last_change":"2021-12-15T17:17:46.414933+0000","last_active":"2021-12-15T17:17:54.829178+0000","last_peered":"2021-12-15T17:17:54.829178+0000","last_clean":"2021-12-15T17:17:54.829178+0000","last_became_active":"2021-12-15T17:17:43.404993+0000","last_became_peered":"2021-12-15T17:17:43.404993+0000","last_unstale":"2021-12-15T17:17:54.829178+0000","last_undegraded":"2021-12-15T17:17:54.829178+0000","last_fullsized":"2021-12-15T17:17:54.829178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:00:48.541924+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"100'13","reported_seq":44,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830618+0000","last_change":"2021-12-15T17:17:45.427376+0000","last_active":"2021-12-15T17:17:54.830618+0000","last_peered":"2021-12-15T17:17:54.830618+0000","last_clean":"2021-12-15T17:17:54.830618+0000","last_became_active":"2021-12-15T17:17:45.426543+0000","last_became_peered":"2021-12-15T17:17:45.426543+0000","last_unstale":"2021-12-15T17:17:54.830618+0000","last_undegraded":"2021-12-15T17:17:54.830618+0000","last_fullsized":"2021-12-15T17:17:54.830618+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:48:21.252080+0000","stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830989+0000","last_change":"2021-12-15T17:17:47.433850+0000","last_active":"2021-12-15T17:17:54.830989+0000","last_peered":"2021-12-15T17:17:54.830989+0000","last_clean":"2021-12-15T17:17:54.830989+0000","last_became_active":"2021-12-15T17:17:47.433485+0000","last_became_peered":"2021-12-15T17:17:47.433485+0000","last_unstale":"2021-12-15T17:17:54.830989+0000","last_undegraded":"2021-12-15T17:17:54.830989+0000","last_fullsized":"2021-12-15T17:17:54.830989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:50:16.243563+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829372+0000","last_change":"2021-12-15T17:17:49.438181+0000","last_active":"2021-12-15T17:17:54.829372+0000","last_peered":"2021-12-15T17:17:54.829372+0000","last_clean":"2021-12-15T17:17:54.829372+0000","last_became_active":"2021-12-15T17:17:49.437819+0000","last_became_peered":"2021-12-15T17:17:49.437819+0000","last_unstale":"2021-12-15T17:17:54.829372+0000","last_undegraded":"2021-12-15T17:17:54.829372+0000","last_fullsized":"2021-12-15T17:17:54.829372+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:29:31.064149+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"102'128","reported_seq":294,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:03.228053+0000","last_change":"2021-12-15T17:17:23.848102+0000","last_active":"2021-12-15T17:18:03.228053+0000","last_peered":"2021-12-15T17:18:03.228053+0000","last_clean":"2021-12-15T17:18:03.228053+0000","last_became_active":"2021-12-15T17:17:23.847618+0000","last_became_peered":"2021-12-15T17:17:23.847618+0000","last_unstale":"2021-12-15T17:18:03.228053+0000","last_undegraded":"2021-12-15T17:18:03.228053+0000","last_fullsized":"2021-12-15T17:18:03.228053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:15:00.738443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:15:00.738443+0000","last_clean_scrub_stamp":"2021-12-15T17:15:00.738443+0000","log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:04:14.432173+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'1","reported_seq":28,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777109+0000","last_change":"2021-12-15T17:17:43.412942+0000","last_active":"2021-12-15T17:17:54.777109+0000","last_peered":"2021-12-15T17:17:54.777109+0000","last_clean":"2021-12-15T17:17:54.777109+0000","last_became_active":"2021-12-15T17:17:43.412650+0000","last_became_peered":"2021-12-15T17:17:43.412650+0000","last_unstale":"2021-12-15T17:17:54.777109+0000","last_undegraded":"2021-12-15T17:17:54.777109+0000","last_fullsized":"2021-12-15T17:17:54.777109+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:33:05.763482+0000","stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'14","reported_seq":39,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829674+0000","last_change":"2021-12-15T17:17:45.574126+0000","last_active":"2021-12-15T17:17:54.829674+0000","last_peered":"2021-12-15T17:17:54.829674+0000","last_clean":"2021-12-15T17:17:54.829674+0000","last_became_active":"2021-12-15T17:17:45.573804+0000","last_became_peered":"2021-12-15T17:17:45.573804+0000","last_unstale":"2021-12-15T17:17:54.829674+0000","last_undegraded":"2021-12-15T17:17:54.829674+0000","last_fullsized":"2021-12-15T17:17:54.829674+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:39:49.338571+0000","stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831412+0000","last_change":"2021-12-15T17:17:47.434735+0000","last_active":"2021-12-15T17:17:54.831412+0000","last_peered":"2021-12-15T17:17:54.831412+0000","last_clean":"2021-12-15T17:17:54.831412+0000","last_became_active":"2021-12-15T17:17:47.434420+0000","last_became_peered":"2021-12-15T17:17:47.434420+0000","last_unstale":"2021-12-15T17:17:54.831412+0000","last_undegraded":"2021-12-15T17:17:54.831412+0000","last_fullsized":"2021-12-15T17:17:54.831412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:31:22.712195+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830030+0000","last_change":"2021-12-15T17:17:49.442324+0000","last_active":"2021-12-15T17:17:54.830030+0000","last_peered":"2021-12-15T17:17:54.830030+0000","last_clean":"2021-12-15T17:17:54.830030+0000","last_became_active":"2021-12-15T17:17:49.441989+0000","last_became_peered":"2021-12-15T17:17:49.441989+0000","last_unstale":"2021-12-15T17:17:54.830030+0000","last_undegraded":"2021-12-15T17:17:54.830030+0000","last_fullsized":"2021-12-15T17:17:54.830030+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:11:12.811754+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777063+0000","last_change":"2021-12-15T17:17:43.407169+0000","last_active":"2021-12-15T17:17:54.777063+0000","last_peered":"2021-12-15T17:17:54.777063+0000","last_clean":"2021-12-15T17:17:54.777063+0000","last_became_active":"2021-12-15T17:17:43.406880+0000","last_became_peered":"2021-12-15T17:17:43.406880+0000","last_unstale":"2021-12-15T17:17:54.777063+0000","last_undegraded":"2021-12-15T17:17:54.777063+0000","last_fullsized":"2021-12-15T17:17:54.777063+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:18:42.377136+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"101'5","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:47.202035+0000","last_change":"2021-12-15T17:17:46.424820+0000","last_active":"2021-12-15T17:18:47.202035+0000","last_peered":"2021-12-15T17:18:47.202035+0000","last_clean":"2021-12-15T17:18:47.202035+0000","last_became_active":"2021-12-15T17:17:43.418415+0000","last_became_peered":"2021-12-15T17:17:43.418415+0000","last_unstale":"2021-12-15T17:18:47.202035+0000","last_undegraded":"2021-12-15T17:18:47.202035+0000","last_fullsized":"2021-12-15T17:18:47.202035+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:49:53.495628+0000","stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"100'12","reported_seq":35,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777267+0000","last_change":"2021-12-15T17:17:45.422402+0000","last_active":"2021-12-15T17:17:54.777267+0000","last_peered":"2021-12-15T17:17:54.777267+0000","last_clean":"2021-12-15T17:17:54.777267+0000","last_became_active":"2021-12-15T17:17:45.421776+0000","last_became_peered":"2021-12-15T17:17:45.421776+0000","last_unstale":"2021-12-15T17:17:54.777267+0000","last_undegraded":"2021-12-15T17:17:54.777267+0000","last_fullsized":"2021-12-15T17:17:54.777267+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:43:56.890537+0000","stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829817+0000","last_change":"2021-12-15T17:17:47.439835+0000","last_active":"2021-12-15T17:17:54.829817+0000","last_peered":"2021-12-15T17:17:54.829817+0000","last_clean":"2021-12-15T17:17:54.829817+0000","last_became_active":"2021-12-15T17:17:47.439541+0000","last_became_peered":"2021-12-15T17:17:47.439541+0000","last_unstale":"2021-12-15T17:17:54.829817+0000","last_undegraded":"2021-12-15T17:17:54.829817+0000","last_fullsized":"2021-12-15T17:17:54.829817+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:19:37.238922+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829308+0000","last_change":"2021-12-15T17:17:49.731392+0000","last_active":"2021-12-15T17:17:54.829308+0000","last_peered":"2021-12-15T17:17:54.829308+0000","last_clean":"2021-12-15T17:17:54.829308+0000","last_became_active":"2021-12-15T17:17:49.730995+0000","last_became_peered":"2021-12-15T17:17:49.730995+0000","last_unstale":"2021-12-15T17:17:54.829308+0000","last_undegraded":"2021-12-15T17:17:54.829308+0000","last_fullsized":"2021-12-15T17:17:54.829308+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:11:34.042095+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830372+0000","last_change":"2021-12-15T17:17:43.413575+0000","last_active":"2021-12-15T17:17:54.830372+0000","last_peered":"2021-12-15T17:17:54.830372+0000","last_clean":"2021-12-15T17:17:54.830372+0000","last_became_active":"2021-12-15T17:17:43.413207+0000","last_became_peered":"2021-12-15T17:17:43.413207+0000","last_unstale":"2021-12-15T17:17:54.830372+0000","last_undegraded":"2021-12-15T17:17:54.830372+0000","last_fullsized":"2021-12-15T17:17:54.830372+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:53:54.821004+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"100'19","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778668+0000","last_change":"2021-12-15T17:17:45.573057+0000","last_active":"2021-12-15T17:17:54.778668+0000","last_peered":"2021-12-15T17:17:54.778668+0000","last_clean":"2021-12-15T17:17:54.778668+0000","last_became_active":"2021-12-15T17:17:45.572581+0000","last_became_peered":"2021-12-15T17:17:45.572581+0000","last_unstale":"2021-12-15T17:17:54.778668+0000","last_undegraded":"2021-12-15T17:17:54.778668+0000","last_fullsized":"2021-12-15T17:17:54.778668+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:53:13.658962+0000","stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830334+0000","last_change":"2021-12-15T17:17:47.429700+0000","last_active":"2021-12-15T17:17:54.830334+0000","last_peered":"2021-12-15T17:17:54.830334+0000","last_clean":"2021-12-15T17:17:54.830334+0000","last_became_active":"2021-12-15T17:17:47.429404+0000","last_became_peered":"2021-12-15T17:17:47.429404+0000","last_unstale":"2021-12-15T17:17:54.830334+0000","last_undegraded":"2021-12-15T17:17:54.830334+0000","last_fullsized":"2021-12-15T17:17:54.830334+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:33:53.733979+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778174+0000","last_change":"2021-12-15T17:17:49.440617+0000","last_active":"2021-12-15T17:17:54.778174+0000","last_peered":"2021-12-15T17:17:54.778174+0000","last_clean":"2021-12-15T17:17:54.778174+0000","last_became_active":"2021-12-15T17:17:49.440263+0000","last_became_peered":"2021-12-15T17:17:49.440263+0000","last_unstale":"2021-12-15T17:17:54.778174+0000","last_undegraded":"2021-12-15T17:17:54.778174+0000","last_fullsized":"2021-12-15T17:17:54.778174+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:47:10.769285+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831002+0000","last_change":"2021-12-15T17:17:43.420003+0000","last_active":"2021-12-15T17:17:54.831002+0000","last_peered":"2021-12-15T17:17:54.831002+0000","last_clean":"2021-12-15T17:17:54.831002+0000","last_became_active":"2021-12-15T17:17:43.419696+0000","last_became_peered":"2021-12-15T17:17:43.419696+0000","last_unstale":"2021-12-15T17:17:54.831002+0000","last_undegraded":"2021-12-15T17:17:54.831002+0000","last_fullsized":"2021-12-15T17:17:54.831002+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:30:08.130096+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829924+0000","last_change":"2021-12-15T17:17:45.423586+0000","last_active":"2021-12-15T17:17:54.829924+0000","last_peered":"2021-12-15T17:17:54.829924+0000","last_clean":"2021-12-15T17:17:54.829924+0000","last_became_active":"2021-12-15T17:17:45.422796+0000","last_became_peered":"2021-12-15T17:17:45.422796+0000","last_unstale":"2021-12-15T17:17:54.829924+0000","last_undegraded":"2021-12-15T17:17:54.829924+0000","last_fullsized":"2021-12-15T17:17:54.829924+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:29:04.901529+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'8","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:36.720270+0000","last_change":"2021-12-15T17:17:47.430921+0000","last_active":"2021-12-15T17:18:36.720270+0000","last_peered":"2021-12-15T17:18:36.720270+0000","last_clean":"2021-12-15T17:18:36.720270+0000","last_became_active":"2021-12-15T17:17:47.430561+0000","last_became_peered":"2021-12-15T17:17:47.430561+0000","last_unstale":"2021-12-15T17:18:36.720270+0000","last_undegraded":"2021-12-15T17:18:36.720270+0000","last_fullsized":"2021-12-15T17:18:36.720270+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:30:31.785850+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830512+0000","last_change":"2021-12-15T17:17:49.731518+0000","last_active":"2021-12-15T17:17:54.830512+0000","last_peered":"2021-12-15T17:17:54.830512+0000","last_clean":"2021-12-15T17:17:54.830512+0000","last_became_active":"2021-12-15T17:17:49.731210+0000","last_became_peered":"2021-12-15T17:17:49.731210+0000","last_unstale":"2021-12-15T17:17:54.830512+0000","last_undegraded":"2021-12-15T17:17:54.830512+0000","last_fullsized":"2021-12-15T17:17:54.830512+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:16:08.107381+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777367+0000","last_change":"2021-12-15T17:17:43.406077+0000","last_active":"2021-12-15T17:17:54.777367+0000","last_peered":"2021-12-15T17:17:54.777367+0000","last_clean":"2021-12-15T17:17:54.777367+0000","last_became_active":"2021-12-15T17:17:43.405892+0000","last_became_peered":"2021-12-15T17:17:43.405892+0000","last_unstale":"2021-12-15T17:17:54.777367+0000","last_undegraded":"2021-12-15T17:17:54.777367+0000","last_fullsized":"2021-12-15T17:17:54.777367+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:04:01.364809+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"100'14","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777116+0000","last_change":"2021-12-15T17:17:45.418354+0000","last_active":"2021-12-15T17:17:54.777116+0000","last_peered":"2021-12-15T17:17:54.777116+0000","last_clean":"2021-12-15T17:17:54.777116+0000","last_became_active":"2021-12-15T17:17:45.418078+0000","last_became_peered":"2021-12-15T17:17:45.418078+0000","last_unstale":"2021-12-15T17:17:54.777116+0000","last_undegraded":"2021-12-15T17:17:54.777116+0000","last_fullsized":"2021-12-15T17:17:54.777116+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:46:15.695736+0000","stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"100'8","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:36.720276+0000","last_change":"2021-12-15T17:17:47.435779+0000","last_active":"2021-12-15T17:18:36.720276+0000","last_peered":"2021-12-15T17:18:36.720276+0000","last_clean":"2021-12-15T17:18:36.720276+0000","last_became_active":"2021-12-15T17:17:47.435401+0000","last_became_peered":"2021-12-15T17:17:47.435401+0000","last_unstale":"2021-12-15T17:18:36.720276+0000","last_undegraded":"2021-12-15T17:18:36.720276+0000","last_fullsized":"2021-12-15T17:18:36.720276+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:23:58.681929+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777020+0000","last_change":"2021-12-15T17:17:49.437323+0000","last_active":"2021-12-15T17:17:54.777020+0000","last_peered":"2021-12-15T17:17:54.777020+0000","last_clean":"2021-12-15T17:17:54.777020+0000","last_became_active":"2021-12-15T17:17:49.437042+0000","last_became_peered":"2021-12-15T17:17:49.437042+0000","last_unstale":"2021-12-15T17:17:54.777020+0000","last_undegraded":"2021-12-15T17:17:54.777020+0000","last_fullsized":"2021-12-15T17:17:54.777020+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:21:19.573770+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778035+0000","last_change":"2021-12-15T17:17:43.414204+0000","last_active":"2021-12-15T17:17:54.778035+0000","last_peered":"2021-12-15T17:17:54.778035+0000","last_clean":"2021-12-15T17:17:54.778035+0000","last_became_active":"2021-12-15T17:17:43.413812+0000","last_became_peered":"2021-12-15T17:17:43.413812+0000","last_unstale":"2021-12-15T17:17:54.778035+0000","last_undegraded":"2021-12-15T17:17:54.778035+0000","last_fullsized":"2021-12-15T17:17:54.778035+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:27:03.708324+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"100'18","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778790+0000","last_change":"2021-12-15T17:17:45.573058+0000","last_active":"2021-12-15T17:17:54.778790+0000","last_peered":"2021-12-15T17:17:54.778790+0000","last_clean":"2021-12-15T17:17:54.778790+0000","last_became_active":"2021-12-15T17:17:45.572536+0000","last_became_peered":"2021-12-15T17:17:45.572536+0000","last_unstale":"2021-12-15T17:17:54.778790+0000","last_undegraded":"2021-12-15T17:17:54.778790+0000","last_fullsized":"2021-12-15T17:17:54.778790+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:02:08.628715+0000","stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776876+0000","last_change":"2021-12-15T17:17:47.439943+0000","last_active":"2021-12-15T17:17:54.776876+0000","last_peered":"2021-12-15T17:17:54.776876+0000","last_clean":"2021-12-15T17:17:54.776876+0000","last_became_active":"2021-12-15T17:17:47.439691+0000","last_became_peered":"2021-12-15T17:17:47.439691+0000","last_unstale":"2021-12-15T17:17:54.776876+0000","last_undegraded":"2021-12-15T17:17:54.776876+0000","last_fullsized":"2021-12-15T17:17:54.776876+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:35:16.226157+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830855+0000","last_change":"2021-12-15T17:17:49.444143+0000","last_active":"2021-12-15T17:17:54.830855+0000","last_peered":"2021-12-15T17:17:54.830855+0000","last_clean":"2021-12-15T17:17:54.830855+0000","last_became_active":"2021-12-15T17:17:49.443881+0000","last_became_peered":"2021-12-15T17:17:49.443881+0000","last_unstale":"2021-12-15T17:17:54.830855+0000","last_undegraded":"2021-12-15T17:17:54.830855+0000","last_fullsized":"2021-12-15T17:17:54.830855+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:47:11.561021+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"100'8","reported_seq":36,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721435+0000","last_change":"2021-12-15T17:17:47.431998+0000","last_active":"2021-12-15T17:18:41.721435+0000","last_peered":"2021-12-15T17:18:41.721435+0000","last_clean":"2021-12-15T17:18:41.721435+0000","last_became_active":"2021-12-15T17:17:47.431109+0000","last_became_peered":"2021-12-15T17:17:47.431109+0000","last_unstale":"2021-12-15T17:18:41.721435+0000","last_undegraded":"2021-12-15T17:18:41.721435+0000","last_fullsized":"2021-12-15T17:18:41.721435+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:48:06.202277+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778690+0000","last_change":"2021-12-15T17:17:43.414849+0000","last_active":"2021-12-15T17:17:54.778690+0000","last_peered":"2021-12-15T17:17:54.778690+0000","last_clean":"2021-12-15T17:17:54.778690+0000","last_became_active":"2021-12-15T17:17:43.414544+0000","last_became_peered":"2021-12-15T17:17:43.414544+0000","last_unstale":"2021-12-15T17:17:54.778690+0000","last_undegraded":"2021-12-15T17:17:54.778690+0000","last_fullsized":"2021-12-15T17:17:54.778690+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:31:28.998600+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830798+0000","last_change":"2021-12-15T17:17:45.427449+0000","last_active":"2021-12-15T17:17:54.830798+0000","last_peered":"2021-12-15T17:17:54.830798+0000","last_clean":"2021-12-15T17:17:54.830798+0000","last_became_active":"2021-12-15T17:17:45.427007+0000","last_became_peered":"2021-12-15T17:17:45.427007+0000","last_unstale":"2021-12-15T17:17:54.830798+0000","last_undegraded":"2021-12-15T17:17:54.830798+0000","last_fullsized":"2021-12-15T17:17:54.830798+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:50:20.208164+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779326+0000","last_change":"2021-12-15T17:17:49.731619+0000","last_active":"2021-12-15T17:17:54.779326+0000","last_peered":"2021-12-15T17:17:54.779326+0000","last_clean":"2021-12-15T17:17:54.779326+0000","last_became_active":"2021-12-15T17:17:49.731230+0000","last_became_peered":"2021-12-15T17:17:49.731230+0000","last_unstale":"2021-12-15T17:17:54.779326+0000","last_undegraded":"2021-12-15T17:17:54.779326+0000","last_fullsized":"2021-12-15T17:17:54.779326+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:33:22.796985+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829740+0000","last_change":"2021-12-15T17:17:47.433607+0000","last_active":"2021-12-15T17:17:54.829740+0000","last_peered":"2021-12-15T17:17:54.829740+0000","last_clean":"2021-12-15T17:17:54.829740+0000","last_became_active":"2021-12-15T17:17:47.433263+0000","last_became_peered":"2021-12-15T17:17:47.433263+0000","last_unstale":"2021-12-15T17:17:54.829740+0000","last_undegraded":"2021-12-15T17:17:54.829740+0000","last_fullsized":"2021-12-15T17:17:54.829740+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:44:51.917330+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776903+0000","last_change":"2021-12-15T17:17:43.413835+0000","last_active":"2021-12-15T17:17:54.776903+0000","last_peered":"2021-12-15T17:17:54.776903+0000","last_clean":"2021-12-15T17:17:54.776903+0000","last_became_active":"2021-12-15T17:17:43.413530+0000","last_became_peered":"2021-12-15T17:17:43.413530+0000","last_unstale":"2021-12-15T17:17:54.776903+0000","last_undegraded":"2021-12-15T17:17:54.776903+0000","last_fullsized":"2021-12-15T17:17:54.776903+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:01:57.843942+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777548+0000","last_change":"2021-12-15T17:17:45.421988+0000","last_active":"2021-12-15T17:17:54.777548+0000","last_peered":"2021-12-15T17:17:54.777548+0000","last_clean":"2021-12-15T17:17:54.777548+0000","last_became_active":"2021-12-15T17:17:45.421646+0000","last_became_peered":"2021-12-15T17:17:45.421646+0000","last_unstale":"2021-12-15T17:17:54.777548+0000","last_undegraded":"2021-12-15T17:17:54.777548+0000","last_fullsized":"2021-12-15T17:17:54.777548+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:05:21.540941+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830720+0000","last_change":"2021-12-15T17:17:49.438323+0000","last_active":"2021-12-15T17:17:54.830720+0000","last_peered":"2021-12-15T17:17:54.830720+0000","last_clean":"2021-12-15T17:17:54.830720+0000","last_became_active":"2021-12-15T17:17:49.437986+0000","last_became_peered":"2021-12-15T17:17:49.437986+0000","last_unstale":"2021-12-15T17:17:54.830720+0000","last_undegraded":"2021-12-15T17:17:54.830720+0000","last_fullsized":"2021-12-15T17:17:54.830720+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:35:28.244626+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830638+0000","last_change":"2021-12-15T17:17:47.438479+0000","last_active":"2021-12-15T17:17:54.830638+0000","last_peered":"2021-12-15T17:17:54.830638+0000","last_clean":"2021-12-15T17:17:54.830638+0000","last_became_active":"2021-12-15T17:17:47.438106+0000","last_became_peered":"2021-12-15T17:17:47.438106+0000","last_unstale":"2021-12-15T17:17:54.830638+0000","last_undegraded":"2021-12-15T17:17:54.830638+0000","last_fullsized":"2021-12-15T17:17:54.830638+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T05:01:49.895601+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"92'1","reported_seq":26,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830084+0000","last_change":"2021-12-15T17:17:43.412118+0000","last_active":"2021-12-15T17:17:54.830084+0000","last_peered":"2021-12-15T17:17:54.830084+0000","last_clean":"2021-12-15T17:17:54.830084+0000","last_became_active":"2021-12-15T17:17:43.411789+0000","last_became_peered":"2021-12-15T17:17:43.411789+0000","last_unstale":"2021-12-15T17:17:54.830084+0000","last_undegraded":"2021-12-15T17:17:54.830084+0000","last_fullsized":"2021-12-15T17:17:54.830084+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:27:22.907469+0000","stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"100'17","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777423+0000","last_change":"2021-12-15T17:17:45.419285+0000","last_active":"2021-12-15T17:17:54.777423+0000","last_peered":"2021-12-15T17:17:54.777423+0000","last_clean":"2021-12-15T17:17:54.777423+0000","last_became_active":"2021-12-15T17:17:45.418977+0000","last_became_peered":"2021-12-15T17:17:45.418977+0000","last_unstale":"2021-12-15T17:17:54.777423+0000","last_undegraded":"2021-12-15T17:17:54.777423+0000","last_fullsized":"2021-12-15T17:17:54.777423+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:13:27.066711+0000","stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"100'1","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777713+0000","last_change":"2021-12-15T17:17:49.441280+0000","last_active":"2021-12-15T17:17:54.777713+0000","last_peered":"2021-12-15T17:17:54.777713+0000","last_clean":"2021-12-15T17:17:54.777713+0000","last_became_active":"2021-12-15T17:17:49.440942+0000","last_became_peered":"2021-12-15T17:17:49.440942+0000","last_unstale":"2021-12-15T17:17:54.777713+0000","last_undegraded":"2021-12-15T17:17:54.777713+0000","last_fullsized":"2021-12-15T17:17:54.777713+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:12:04.128538+0000","stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'8","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:36.720264+0000","last_change":"2021-12-15T17:17:47.433808+0000","last_active":"2021-12-15T17:18:36.720264+0000","last_peered":"2021-12-15T17:18:36.720264+0000","last_clean":"2021-12-15T17:18:36.720264+0000","last_became_active":"2021-12-15T17:17:47.433435+0000","last_became_peered":"2021-12-15T17:17:47.433435+0000","last_unstale":"2021-12-15T17:18:36.720264+0000","last_undegraded":"2021-12-15T17:18:36.720264+0000","last_fullsized":"2021-12-15T17:18:36.720264+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:11:40.036969+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777744+0000","last_change":"2021-12-15T17:17:43.408339+0000","last_active":"2021-12-15T17:17:54.777744+0000","last_peered":"2021-12-15T17:17:54.777744+0000","last_clean":"2021-12-15T17:17:54.777744+0000","last_became_active":"2021-12-15T17:17:43.408022+0000","last_became_peered":"2021-12-15T17:17:43.408022+0000","last_unstale":"2021-12-15T17:17:54.777744+0000","last_undegraded":"2021-12-15T17:17:54.777744+0000","last_fullsized":"2021-12-15T17:17:54.777744+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:41:06.580984+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777290+0000","last_change":"2021-12-15T17:17:45.416524+0000","last_active":"2021-12-15T17:17:54.777290+0000","last_peered":"2021-12-15T17:17:54.777290+0000","last_clean":"2021-12-15T17:17:54.777290+0000","last_became_active":"2021-12-15T17:17:45.416222+0000","last_became_peered":"2021-12-15T17:17:45.416222+0000","last_unstale":"2021-12-15T17:17:54.777290+0000","last_undegraded":"2021-12-15T17:17:54.777290+0000","last_fullsized":"2021-12-15T17:17:54.777290+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:03:22.824992+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1327,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":77,"seq":330712481813,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7024,"kb_used_data":1512,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738128,"statfs":{"total":95995035648,"available":95987843072,"internally_reserved":0,"allocated":1548288,"data_stored":978393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.95399999999999996}]},{"osd":2,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.623},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.70999999999999996}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.505},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48599999999999999}]},{"osd":5,"last update":"Wed Dec 15 17:18:38 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Wed Dec 15 17:18:25 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.65000000000000002}]}]},{"osd":6,"up_from":83,"seq":356482285586,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7320,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":5822,"kb_avail":93737832,"statfs":{"total":95995035648,"available":95987539968,"internally_reserved":0,"allocated":1523712,"data_stored":977724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":5962696},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.432},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.54100000000000004}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63900000000000001}]},{"osd":7,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.67700000000000005}]}]},{"osd":1,"up_from":43,"seq":184683593767,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11544,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":10558,"kb_avail":93733608,"statfs":{"total":95995035648,"available":95983214592,"internally_reserved":0,"allocated":999424,"data_stored":389042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":10812360},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.42699999999999999,"15min":0.39300000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.585,"5min":4.585,"15min":4.585},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.42099999999999999,"15min":0.42699999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.498}]},{"osd":2,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.41999999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.6130000000000004,"5min":4.6130000000000004,"15min":4.6130000000000004},"last":0.51300000000000001}]},{"osd":3,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47399999999999998,"15min":0.46400000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.4630000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48399999999999999,"15min":0.48599999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48499999999999999}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55600000000000005}]},{"osd":5,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":4.4649999999999999,"5min":4.4649999999999999,"15min":4.4649999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.57399999999999995}]},{"osd":6,"last update":"Wed Dec 15 17:18:23 2021","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.60499999999999998}]},{"osd":7,"last update":"Wed Dec 15 17:18:17 2021","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.59799999999999998}]}]},{"osd":0,"up_from":40,"seq":171798691880,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7984,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":6462,"kb_avail":93737168,"statfs":{"total":95995035648,"available":95986860032,"internally_reserved":0,"allocated":1548288,"data_stored":978090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":6618056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 15 17:18:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.39100000000000001,"15min":0.40300000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.39500000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.52600000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.49299999999999999}]},{"osd":2,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.44,"15min":0.45700000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.625,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.42499999999999999,"15min":0.44500000000000001},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.48799999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44600000000000001}]},{"osd":3,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.43099999999999999,"15min":0.437},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54100000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47499999999999998,"15min":0.47999999999999998},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58499999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45900000000000002}]},{"osd":4,"last update":"Wed Dec 15 17:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57199999999999995}]},{"osd":5,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Dec 15 17:18:24 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.629}]},{"osd":7,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.61299999999999999}]}]},{"osd":2,"up_from":36,"seq":154618822697,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6996,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":6078,"kb_avail":93738156,"statfs":{"total":95995035648,"available":95987871744,"internally_reserved":0,"allocated":929792,"data_stored":386857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":6224840},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:39 2021","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.375,"15min":0.378},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.45200000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.57599999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.26000000000000001}]},{"osd":1,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41599999999999998,"15min":0.41899999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41599999999999998,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.45500000000000002,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.32300000000000001}]},{"osd":3,"last update":"Wed Dec 15 17:18:30 2021","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.44800000000000001,"15min":0.44900000000000001},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.434}]},{"osd":4,"last update":"Wed Dec 15 17:18:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.502},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47299999999999998}]},{"osd":5,"last update":"Wed Dec 15 17:18:39 2021","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.42299999999999999}]},{"osd":6,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.48299999999999998}]},{"osd":7,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.41499999999999998}]}]},{"osd":3,"up_from":32,"seq":137438953515,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6780,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738372,"statfs":{"total":95995035648,"available":95988092928,"internally_reserved":0,"allocated":970752,"data_stored":388692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:36 2021","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35899999999999999,"15min":0.36299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.36699999999999999,"15min":0.35999999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53900000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.39000000000000001}]},{"osd":1,"last update":"Wed Dec 15 17:17:39 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.31,"5min":0.309,"15min":0.309},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.37}]},{"osd":2,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40699999999999997,"15min":0.39800000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.40600000000000003}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.45000000000000001}]},{"osd":5,"last update":"Wed Dec 15 17:18:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.627},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47999999999999998}]},{"osd":6,"last update":"Wed Dec 15 17:18:23 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.51800000000000002}]},{"osd":7,"last update":"Wed Dec 15 17:18:15 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.66300000000000003}]}]},{"osd":4,"up_from":88,"seq":377957122065,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7092,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":6142,"kb_avail":93738060,"statfs":{"total":95995035648,"available":95987773440,"internally_reserved":0,"allocated":962560,"data_stored":387862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":6290376},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.438}]},{"osd":1,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.58299999999999996}]},{"osd":2,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.505},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.53200000000000003}]},{"osd":5,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.55900000000000005}]},{"osd":6,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.625},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.64900000000000002}]}]},{"osd":5,"up_from":90,"seq":386547056655,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6752,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":5822,"kb_avail":93738400,"statfs":{"total":95995035648,"available":95988121600,"internally_reserved":0,"allocated":942080,"data_stored":388172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":5962696},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.37}]},{"osd":1,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.35599999999999998}]},{"osd":2,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.38}]},{"osd":3,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.42099999999999999}]},{"osd":4,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34799999999999998}]},{"osd":6,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.30099999999999999}]},{"osd":7,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.311},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-12-15T17:18:50.602 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph pg dump --format=json 2021-12-15T17:18:51.049 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:51.464 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:51 smithi022 ceph-mon[31360]: from='client.14841 v1:172.21.15.22:0/3231710832' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:18:51.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:51 smithi022 ceph-mon[31360]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:51.465 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:51 smithi022 ceph-mon[40250]: from='client.14841 v1:172.21.15.22:0/3231710832' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:18:51.465 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:51 smithi022 ceph-mon[40250]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:51 smithi122 ceph-mon[32255]: from='client.14841 v1:172.21.15.22:0/3231710832' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:18:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:51 smithi122 ceph-mon[32255]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:52.391 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:18:52.394 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2021-12-15T17:18:52.771 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2021-12-15T17:18:51.130680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596187,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":998,"num_read_kb":844,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596,"ondisk_log_size":596,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":61492,"kb_used_data":9204,"kb_used_omap":6,"kb_used_meta":52217,"kb_avail":749899724,"statfs":{"total":767960285184,"available":767897317376,"internally_reserved":0,"allocated":9424896,"data_stored":4874832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6480,"internal_metadata":53470896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003628"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776944+0000","last_change":"2021-12-15T17:17:49.441711+0000","last_active":"2021-12-15T17:17:54.776944+0000","last_peered":"2021-12-15T17:17:54.776944+0000","last_clean":"2021-12-15T17:17:54.776944+0000","last_became_active":"2021-12-15T17:17:49.441429+0000","last_became_peered":"2021-12-15T17:17:49.441429+0000","last_unstale":"2021-12-15T17:17:54.776944+0000","last_undegraded":"2021-12-15T17:17:54.776944+0000","last_fullsized":"2021-12-15T17:17:54.776944+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:39:00.644601+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829506+0000","last_change":"2021-12-15T17:17:47.436682+0000","last_active":"2021-12-15T17:17:54.829506+0000","last_peered":"2021-12-15T17:17:54.829506+0000","last_clean":"2021-12-15T17:17:54.829506+0000","last_became_active":"2021-12-15T17:17:47.436432+0000","last_became_peered":"2021-12-15T17:17:47.436432+0000","last_unstale":"2021-12-15T17:17:54.829506+0000","last_undegraded":"2021-12-15T17:17:54.829506+0000","last_fullsized":"2021-12-15T17:17:54.829506+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:07:42.742385+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778067+0000","last_change":"2021-12-15T17:17:43.412256+0000","last_active":"2021-12-15T17:17:54.778067+0000","last_peered":"2021-12-15T17:17:54.778067+0000","last_clean":"2021-12-15T17:17:54.778067+0000","last_became_active":"2021-12-15T17:17:43.411941+0000","last_became_peered":"2021-12-15T17:17:43.411941+0000","last_unstale":"2021-12-15T17:17:54.778067+0000","last_undegraded":"2021-12-15T17:17:54.778067+0000","last_fullsized":"2021-12-15T17:17:54.778067+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:09:19.798323+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776821+0000","last_change":"2021-12-15T17:17:45.419334+0000","last_active":"2021-12-15T17:17:54.776821+0000","last_peered":"2021-12-15T17:17:54.776821+0000","last_clean":"2021-12-15T17:17:54.776821+0000","last_became_active":"2021-12-15T17:17:45.418985+0000","last_became_peered":"2021-12-15T17:17:45.418985+0000","last_unstale":"2021-12-15T17:17:54.776821+0000","last_undegraded":"2021-12-15T17:17:54.776821+0000","last_fullsized":"2021-12-15T17:17:54.776821+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:30:49.171533+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777564+0000","last_change":"2021-12-15T17:17:49.731564+0000","last_active":"2021-12-15T17:17:54.777564+0000","last_peered":"2021-12-15T17:17:54.777564+0000","last_clean":"2021-12-15T17:17:54.777564+0000","last_became_active":"2021-12-15T17:17:49.731160+0000","last_became_peered":"2021-12-15T17:17:49.731160+0000","last_unstale":"2021-12-15T17:17:54.777564+0000","last_undegraded":"2021-12-15T17:17:54.777564+0000","last_fullsized":"2021-12-15T17:17:54.777564+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:03:28.483057+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776838+0000","last_change":"2021-12-15T17:17:47.436913+0000","last_active":"2021-12-15T17:17:54.776838+0000","last_peered":"2021-12-15T17:17:54.776838+0000","last_clean":"2021-12-15T17:17:54.776838+0000","last_became_active":"2021-12-15T17:17:47.436609+0000","last_became_peered":"2021-12-15T17:17:47.436609+0000","last_unstale":"2021-12-15T17:17:54.776838+0000","last_undegraded":"2021-12-15T17:17:54.776838+0000","last_fullsized":"2021-12-15T17:17:54.776838+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:59:13.782727+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778323+0000","last_change":"2021-12-15T17:17:43.414926+0000","last_active":"2021-12-15T17:17:54.778323+0000","last_peered":"2021-12-15T17:17:54.778323+0000","last_clean":"2021-12-15T17:17:54.778323+0000","last_became_active":"2021-12-15T17:17:43.414585+0000","last_became_peered":"2021-12-15T17:17:43.414585+0000","last_unstale":"2021-12-15T17:17:54.778323+0000","last_undegraded":"2021-12-15T17:17:54.778323+0000","last_fullsized":"2021-12-15T17:17:54.778323+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:48:33.892199+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779143+0000","last_change":"2021-12-15T17:17:45.426575+0000","last_active":"2021-12-15T17:17:54.779143+0000","last_peered":"2021-12-15T17:17:54.779143+0000","last_clean":"2021-12-15T17:17:54.779143+0000","last_became_active":"2021-12-15T17:17:45.425469+0000","last_became_peered":"2021-12-15T17:17:45.425469+0000","last_unstale":"2021-12-15T17:17:54.779143+0000","last_undegraded":"2021-12-15T17:17:54.779143+0000","last_fullsized":"2021-12-15T17:17:54.779143+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:20:21.028242+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776797+0000","last_change":"2021-12-15T17:17:49.442795+0000","last_active":"2021-12-15T17:17:54.776797+0000","last_peered":"2021-12-15T17:17:54.776797+0000","last_clean":"2021-12-15T17:17:54.776797+0000","last_became_active":"2021-12-15T17:17:49.442246+0000","last_became_peered":"2021-12-15T17:17:49.442246+0000","last_unstale":"2021-12-15T17:17:54.776797+0000","last_undegraded":"2021-12-15T17:17:54.776797+0000","last_fullsized":"2021-12-15T17:17:54.776797+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:47:58.984541+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830722+0000","last_change":"2021-12-15T17:17:47.431969+0000","last_active":"2021-12-15T17:17:54.830722+0000","last_peered":"2021-12-15T17:17:54.830722+0000","last_clean":"2021-12-15T17:17:54.830722+0000","last_became_active":"2021-12-15T17:17:47.431473+0000","last_became_peered":"2021-12-15T17:17:47.431473+0000","last_unstale":"2021-12-15T17:17:54.830722+0000","last_undegraded":"2021-12-15T17:17:54.830722+0000","last_fullsized":"2021-12-15T17:17:54.830722+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:17:28.771149+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830928+0000","last_change":"2021-12-15T17:17:43.418183+0000","last_active":"2021-12-15T17:17:54.830928+0000","last_peered":"2021-12-15T17:17:54.830928+0000","last_clean":"2021-12-15T17:17:54.830928+0000","last_became_active":"2021-12-15T17:17:43.417807+0000","last_became_peered":"2021-12-15T17:17:43.417807+0000","last_unstale":"2021-12-15T17:17:54.830928+0000","last_undegraded":"2021-12-15T17:17:54.830928+0000","last_fullsized":"2021-12-15T17:17:54.830928+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:57:40.445606+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777070+0000","last_change":"2021-12-15T17:17:45.423371+0000","last_active":"2021-12-15T17:17:54.777070+0000","last_peered":"2021-12-15T17:17:54.777070+0000","last_clean":"2021-12-15T17:17:54.777070+0000","last_became_active":"2021-12-15T17:17:45.423018+0000","last_became_peered":"2021-12-15T17:17:45.423018+0000","last_unstale":"2021-12-15T17:17:54.777070+0000","last_undegraded":"2021-12-15T17:17:54.777070+0000","last_fullsized":"2021-12-15T17:17:54.777070+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:10:13.101257+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829170+0000","last_change":"2021-12-15T17:17:49.442447+0000","last_active":"2021-12-15T17:17:54.829170+0000","last_peered":"2021-12-15T17:17:54.829170+0000","last_clean":"2021-12-15T17:17:54.829170+0000","last_became_active":"2021-12-15T17:17:49.442167+0000","last_became_peered":"2021-12-15T17:17:49.442167+0000","last_unstale":"2021-12-15T17:17:54.829170+0000","last_undegraded":"2021-12-15T17:17:54.829170+0000","last_fullsized":"2021-12-15T17:17:54.829170+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:22:15.562970+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830546+0000","last_change":"2021-12-15T17:17:47.438208+0000","last_active":"2021-12-15T17:17:54.830546+0000","last_peered":"2021-12-15T17:17:54.830546+0000","last_clean":"2021-12-15T17:17:54.830546+0000","last_became_active":"2021-12-15T17:17:47.437950+0000","last_became_peered":"2021-12-15T17:17:47.437950+0000","last_unstale":"2021-12-15T17:17:54.830546+0000","last_undegraded":"2021-12-15T17:17:54.830546+0000","last_fullsized":"2021-12-15T17:17:54.830546+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:10:36.042297+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829639+0000","last_change":"2021-12-15T17:17:43.416039+0000","last_active":"2021-12-15T17:17:54.829639+0000","last_peered":"2021-12-15T17:17:54.829639+0000","last_clean":"2021-12-15T17:17:54.829639+0000","last_became_active":"2021-12-15T17:17:43.415706+0000","last_became_peered":"2021-12-15T17:17:43.415706+0000","last_unstale":"2021-12-15T17:17:54.829639+0000","last_undegraded":"2021-12-15T17:17:54.829639+0000","last_fullsized":"2021-12-15T17:17:54.829639+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:37:54.602918+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"100'5","reported_seq":27,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777255+0000","last_change":"2021-12-15T17:17:45.419389+0000","last_active":"2021-12-15T17:17:54.777255+0000","last_peered":"2021-12-15T17:17:54.777255+0000","last_clean":"2021-12-15T17:17:54.777255+0000","last_became_active":"2021-12-15T17:17:45.419069+0000","last_became_peered":"2021-12-15T17:17:45.419069+0000","last_unstale":"2021-12-15T17:17:54.777255+0000","last_undegraded":"2021-12-15T17:17:54.777255+0000","last_fullsized":"2021-12-15T17:17:54.777255+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:09:28.698033+0000","stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777396+0000","last_change":"2021-12-15T17:17:49.731380+0000","last_active":"2021-12-15T17:17:54.777396+0000","last_peered":"2021-12-15T17:17:54.777396+0000","last_clean":"2021-12-15T17:17:54.777396+0000","last_became_active":"2021-12-15T17:17:49.731039+0000","last_became_peered":"2021-12-15T17:17:49.731039+0000","last_unstale":"2021-12-15T17:17:54.777396+0000","last_undegraded":"2021-12-15T17:17:54.777396+0000","last_fullsized":"2021-12-15T17:17:54.777396+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:37:10.446853+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830216+0000","last_change":"2021-12-15T17:17:47.439549+0000","last_active":"2021-12-15T17:17:54.830216+0000","last_peered":"2021-12-15T17:17:54.830216+0000","last_clean":"2021-12-15T17:17:54.830216+0000","last_became_active":"2021-12-15T17:17:47.439031+0000","last_became_peered":"2021-12-15T17:17:47.439031+0000","last_unstale":"2021-12-15T17:17:54.830216+0000","last_undegraded":"2021-12-15T17:17:54.830216+0000","last_fullsized":"2021-12-15T17:17:54.830216+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:11:19.927526+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778621+0000","last_change":"2021-12-15T17:17:43.416573+0000","last_active":"2021-12-15T17:17:54.778621+0000","last_peered":"2021-12-15T17:17:54.778621+0000","last_clean":"2021-12-15T17:17:54.778621+0000","last_became_active":"2021-12-15T17:17:43.416285+0000","last_became_peered":"2021-12-15T17:17:43.416285+0000","last_unstale":"2021-12-15T17:17:54.778621+0000","last_undegraded":"2021-12-15T17:17:54.778621+0000","last_fullsized":"2021-12-15T17:17:54.778621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:58:58.992557+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830551+0000","last_change":"2021-12-15T17:17:45.420350+0000","last_active":"2021-12-15T17:17:54.830551+0000","last_peered":"2021-12-15T17:17:54.830551+0000","last_clean":"2021-12-15T17:17:54.830551+0000","last_became_active":"2021-12-15T17:17:45.420013+0000","last_became_peered":"2021-12-15T17:17:45.420013+0000","last_unstale":"2021-12-15T17:17:54.830551+0000","last_undegraded":"2021-12-15T17:17:54.830551+0000","last_fullsized":"2021-12-15T17:17:54.830551+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:50:04.023775+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778437+0000","last_change":"2021-12-15T17:17:49.732358+0000","last_active":"2021-12-15T17:17:54.778437+0000","last_peered":"2021-12-15T17:17:54.778437+0000","last_clean":"2021-12-15T17:17:54.778437+0000","last_became_active":"2021-12-15T17:17:49.731857+0000","last_became_peered":"2021-12-15T17:17:49.731857+0000","last_unstale":"2021-12-15T17:17:54.778437+0000","last_undegraded":"2021-12-15T17:17:54.778437+0000","last_fullsized":"2021-12-15T17:17:54.778437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:05:10.519251+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777230+0000","last_change":"2021-12-15T17:17:47.432479+0000","last_active":"2021-12-15T17:17:54.777230+0000","last_peered":"2021-12-15T17:17:54.777230+0000","last_clean":"2021-12-15T17:17:54.777230+0000","last_became_active":"2021-12-15T17:17:47.431115+0000","last_became_peered":"2021-12-15T17:17:47.431115+0000","last_unstale":"2021-12-15T17:17:54.777230+0000","last_undegraded":"2021-12-15T17:17:54.777230+0000","last_fullsized":"2021-12-15T17:17:54.777230+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:01:39.879259+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777101+0000","last_change":"2021-12-15T17:17:43.413035+0000","last_active":"2021-12-15T17:17:54.777101+0000","last_peered":"2021-12-15T17:17:54.777101+0000","last_clean":"2021-12-15T17:17:54.777101+0000","last_became_active":"2021-12-15T17:17:43.412718+0000","last_became_peered":"2021-12-15T17:17:43.412718+0000","last_unstale":"2021-12-15T17:17:54.777101+0000","last_undegraded":"2021-12-15T17:17:54.777101+0000","last_fullsized":"2021-12-15T17:17:54.777101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:12:50.829486+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"100'12","reported_seq":40,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778328+0000","last_change":"2021-12-15T17:17:45.422277+0000","last_active":"2021-12-15T17:17:54.778328+0000","last_peered":"2021-12-15T17:17:54.778328+0000","last_clean":"2021-12-15T17:17:54.778328+0000","last_became_active":"2021-12-15T17:17:45.421957+0000","last_became_peered":"2021-12-15T17:17:45.421957+0000","last_unstale":"2021-12-15T17:17:54.778328+0000","last_undegraded":"2021-12-15T17:17:54.778328+0000","last_fullsized":"2021-12-15T17:17:54.778328+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:12:54.877225+0000","stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"100'1","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831365+0000","last_change":"2021-12-15T17:17:49.440790+0000","last_active":"2021-12-15T17:17:54.831365+0000","last_peered":"2021-12-15T17:17:54.831365+0000","last_clean":"2021-12-15T17:17:54.831365+0000","last_became_active":"2021-12-15T17:17:49.440496+0000","last_became_peered":"2021-12-15T17:17:49.440496+0000","last_unstale":"2021-12-15T17:17:54.831365+0000","last_undegraded":"2021-12-15T17:17:54.831365+0000","last_fullsized":"2021-12-15T17:17:54.831365+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:43:52.309837+0000","stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'8","reported_seq":39,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721441+0000","last_change":"2021-12-15T17:17:47.436945+0000","last_active":"2021-12-15T17:18:41.721441+0000","last_peered":"2021-12-15T17:18:41.721441+0000","last_clean":"2021-12-15T17:18:41.721441+0000","last_became_active":"2021-12-15T17:17:47.436505+0000","last_became_peered":"2021-12-15T17:17:47.436505+0000","last_unstale":"2021-12-15T17:18:41.721441+0000","last_undegraded":"2021-12-15T17:18:41.721441+0000","last_fullsized":"2021-12-15T17:18:41.721441+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:41:06.698413+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829594+0000","last_change":"2021-12-15T17:17:43.406919+0000","last_active":"2021-12-15T17:17:54.829594+0000","last_peered":"2021-12-15T17:17:54.829594+0000","last_clean":"2021-12-15T17:17:54.829594+0000","last_became_active":"2021-12-15T17:17:43.406558+0000","last_became_peered":"2021-12-15T17:17:43.406558+0000","last_unstale":"2021-12-15T17:17:54.829594+0000","last_undegraded":"2021-12-15T17:17:54.829594+0000","last_fullsized":"2021-12-15T17:17:54.829594+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:16:48.133094+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777463+0000","last_change":"2021-12-15T17:17:45.573216+0000","last_active":"2021-12-15T17:17:54.777463+0000","last_peered":"2021-12-15T17:17:54.777463+0000","last_clean":"2021-12-15T17:17:54.777463+0000","last_became_active":"2021-12-15T17:17:45.572876+0000","last_became_peered":"2021-12-15T17:17:45.572876+0000","last_unstale":"2021-12-15T17:17:54.777463+0000","last_undegraded":"2021-12-15T17:17:54.777463+0000","last_fullsized":"2021-12-15T17:17:54.777463+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:32:00.102468+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830349+0000","last_change":"2021-12-15T17:17:49.441787+0000","last_active":"2021-12-15T17:17:54.830349+0000","last_peered":"2021-12-15T17:17:54.830349+0000","last_clean":"2021-12-15T17:17:54.830349+0000","last_became_active":"2021-12-15T17:17:49.441409+0000","last_became_peered":"2021-12-15T17:17:49.441409+0000","last_unstale":"2021-12-15T17:17:54.830349+0000","last_undegraded":"2021-12-15T17:17:54.830349+0000","last_fullsized":"2021-12-15T17:17:54.830349+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:29:43.861392+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777738+0000","last_change":"2021-12-15T17:17:47.431872+0000","last_active":"2021-12-15T17:17:54.777738+0000","last_peered":"2021-12-15T17:17:54.777738+0000","last_clean":"2021-12-15T17:17:54.777738+0000","last_became_active":"2021-12-15T17:17:47.431536+0000","last_became_peered":"2021-12-15T17:17:47.431536+0000","last_unstale":"2021-12-15T17:17:54.777738+0000","last_undegraded":"2021-12-15T17:17:54.777738+0000","last_fullsized":"2021-12-15T17:17:54.777738+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:49:15.183095+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776638+0000","last_change":"2021-12-15T17:17:43.405875+0000","last_active":"2021-12-15T17:17:54.776638+0000","last_peered":"2021-12-15T17:17:54.776638+0000","last_clean":"2021-12-15T17:17:54.776638+0000","last_became_active":"2021-12-15T17:17:43.405547+0000","last_became_peered":"2021-12-15T17:17:43.405547+0000","last_unstale":"2021-12-15T17:17:54.776638+0000","last_undegraded":"2021-12-15T17:17:54.776638+0000","last_fullsized":"2021-12-15T17:17:54.776638+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:30:18.838252+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829907+0000","last_change":"2021-12-15T17:17:45.425045+0000","last_active":"2021-12-15T17:17:54.829907+0000","last_peered":"2021-12-15T17:17:54.829907+0000","last_clean":"2021-12-15T17:17:54.829907+0000","last_became_active":"2021-12-15T17:17:45.424729+0000","last_became_peered":"2021-12-15T17:17:45.424729+0000","last_unstale":"2021-12-15T17:17:54.829907+0000","last_undegraded":"2021-12-15T17:17:54.829907+0000","last_fullsized":"2021-12-15T17:17:54.829907+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:20:38.839206+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830452+0000","last_change":"2021-12-15T17:17:49.439633+0000","last_active":"2021-12-15T17:17:54.830452+0000","last_peered":"2021-12-15T17:17:54.830452+0000","last_clean":"2021-12-15T17:17:54.830452+0000","last_became_active":"2021-12-15T17:17:49.439201+0000","last_became_peered":"2021-12-15T17:17:49.439201+0000","last_unstale":"2021-12-15T17:17:54.830452+0000","last_undegraded":"2021-12-15T17:17:54.830452+0000","last_fullsized":"2021-12-15T17:17:54.830452+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:26:39.843841+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830170+0000","last_change":"2021-12-15T17:17:47.442769+0000","last_active":"2021-12-15T17:17:54.830170+0000","last_peered":"2021-12-15T17:17:54.830170+0000","last_clean":"2021-12-15T17:17:54.830170+0000","last_became_active":"2021-12-15T17:17:47.442448+0000","last_became_peered":"2021-12-15T17:17:47.442448+0000","last_unstale":"2021-12-15T17:17:54.830170+0000","last_undegraded":"2021-12-15T17:17:54.830170+0000","last_fullsized":"2021-12-15T17:17:54.830170+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:05:35.027221+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777974+0000","last_change":"2021-12-15T17:17:43.412365+0000","last_active":"2021-12-15T17:17:54.777974+0000","last_peered":"2021-12-15T17:17:54.777974+0000","last_clean":"2021-12-15T17:17:54.777974+0000","last_became_active":"2021-12-15T17:17:43.412071+0000","last_became_peered":"2021-12-15T17:17:43.412071+0000","last_unstale":"2021-12-15T17:17:54.777974+0000","last_undegraded":"2021-12-15T17:17:54.777974+0000","last_fullsized":"2021-12-15T17:17:54.777974+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:38:22.706215+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"100'4","reported_seq":23,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778597+0000","last_change":"2021-12-15T17:17:45.419024+0000","last_active":"2021-12-15T17:17:54.778597+0000","last_peered":"2021-12-15T17:17:54.778597+0000","last_clean":"2021-12-15T17:17:54.778597+0000","last_became_active":"2021-12-15T17:17:45.418608+0000","last_became_peered":"2021-12-15T17:17:45.418608+0000","last_unstale":"2021-12-15T17:17:54.778597+0000","last_undegraded":"2021-12-15T17:17:54.778597+0000","last_fullsized":"2021-12-15T17:17:54.778597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:22:00.679507+0000","stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777332+0000","last_change":"2021-12-15T17:17:49.731793+0000","last_active":"2021-12-15T17:17:54.777332+0000","last_peered":"2021-12-15T17:17:54.777332+0000","last_clean":"2021-12-15T17:17:54.777332+0000","last_became_active":"2021-12-15T17:17:49.731393+0000","last_became_peered":"2021-12-15T17:17:49.731393+0000","last_unstale":"2021-12-15T17:17:54.777332+0000","last_undegraded":"2021-12-15T17:17:54.777332+0000","last_fullsized":"2021-12-15T17:17:54.777332+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:03:56.411886+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830598+0000","last_change":"2021-12-15T17:17:47.432912+0000","last_active":"2021-12-15T17:17:54.830598+0000","last_peered":"2021-12-15T17:17:54.830598+0000","last_clean":"2021-12-15T17:17:54.830598+0000","last_became_active":"2021-12-15T17:17:47.432604+0000","last_became_peered":"2021-12-15T17:17:47.432604+0000","last_unstale":"2021-12-15T17:17:54.830598+0000","last_undegraded":"2021-12-15T17:17:54.830598+0000","last_fullsized":"2021-12-15T17:17:54.830598+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:07:09.892394+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830116+0000","last_change":"2021-12-15T17:17:43.415192+0000","last_active":"2021-12-15T17:17:54.830116+0000","last_peered":"2021-12-15T17:17:54.830116+0000","last_clean":"2021-12-15T17:17:54.830116+0000","last_became_active":"2021-12-15T17:17:43.414904+0000","last_became_peered":"2021-12-15T17:17:43.414904+0000","last_unstale":"2021-12-15T17:17:54.830116+0000","last_undegraded":"2021-12-15T17:17:54.830116+0000","last_fullsized":"2021-12-15T17:17:54.830116+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:03:06.583030+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779090+0000","last_change":"2021-12-15T17:17:45.573138+0000","last_active":"2021-12-15T17:17:54.779090+0000","last_peered":"2021-12-15T17:17:54.779090+0000","last_clean":"2021-12-15T17:17:54.779090+0000","last_became_active":"2021-12-15T17:17:45.572449+0000","last_became_peered":"2021-12-15T17:17:45.572449+0000","last_unstale":"2021-12-15T17:17:54.779090+0000","last_undegraded":"2021-12-15T17:17:54.779090+0000","last_fullsized":"2021-12-15T17:17:54.779090+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T05:13:53.844783+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776501+0000","last_change":"2021-12-15T17:17:49.442586+0000","last_active":"2021-12-15T17:17:54.776501+0000","last_peered":"2021-12-15T17:17:54.776501+0000","last_clean":"2021-12-15T17:17:54.776501+0000","last_became_active":"2021-12-15T17:17:49.441784+0000","last_became_peered":"2021-12-15T17:17:49.441784+0000","last_unstale":"2021-12-15T17:17:54.776501+0000","last_undegraded":"2021-12-15T17:17:54.776501+0000","last_fullsized":"2021-12-15T17:17:54.776501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:39:29.966309+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777658+0000","last_change":"2021-12-15T17:17:47.435697+0000","last_active":"2021-12-15T17:17:54.777658+0000","last_peered":"2021-12-15T17:17:54.777658+0000","last_clean":"2021-12-15T17:17:54.777658+0000","last_became_active":"2021-12-15T17:17:47.435315+0000","last_became_peered":"2021-12-15T17:17:47.435315+0000","last_unstale":"2021-12-15T17:17:54.777658+0000","last_undegraded":"2021-12-15T17:17:54.777658+0000","last_fullsized":"2021-12-15T17:17:54.777658+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:16:04.371863+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829315+0000","last_change":"2021-12-15T17:17:43.415619+0000","last_active":"2021-12-15T17:17:54.829315+0000","last_peered":"2021-12-15T17:17:54.829315+0000","last_clean":"2021-12-15T17:17:54.829315+0000","last_became_active":"2021-12-15T17:17:43.414683+0000","last_became_peered":"2021-12-15T17:17:43.414683+0000","last_unstale":"2021-12-15T17:17:54.829315+0000","last_undegraded":"2021-12-15T17:17:54.829315+0000","last_fullsized":"2021-12-15T17:17:54.829315+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:42:47.257062+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829962+0000","last_change":"2021-12-15T17:17:45.424983+0000","last_active":"2021-12-15T17:17:54.829962+0000","last_peered":"2021-12-15T17:17:54.829962+0000","last_clean":"2021-12-15T17:17:54.829962+0000","last_became_active":"2021-12-15T17:17:45.424600+0000","last_became_peered":"2021-12-15T17:17:45.424600+0000","last_unstale":"2021-12-15T17:17:54.829962+0000","last_undegraded":"2021-12-15T17:17:54.829962+0000","last_fullsized":"2021-12-15T17:17:54.829962+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:24:51.543541+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778788+0000","last_change":"2021-12-15T17:17:49.435850+0000","last_active":"2021-12-15T17:17:54.778788+0000","last_peered":"2021-12-15T17:17:54.778788+0000","last_clean":"2021-12-15T17:17:54.778788+0000","last_became_active":"2021-12-15T17:17:49.435474+0000","last_became_peered":"2021-12-15T17:17:49.435474+0000","last_unstale":"2021-12-15T17:17:54.778788+0000","last_undegraded":"2021-12-15T17:17:54.778788+0000","last_fullsized":"2021-12-15T17:17:54.778788+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:24:49.786695+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829866+0000","last_change":"2021-12-15T17:17:47.437623+0000","last_active":"2021-12-15T17:17:54.829866+0000","last_peered":"2021-12-15T17:17:54.829866+0000","last_clean":"2021-12-15T17:17:54.829866+0000","last_became_active":"2021-12-15T17:17:47.437243+0000","last_became_peered":"2021-12-15T17:17:47.437243+0000","last_unstale":"2021-12-15T17:17:54.829866+0000","last_undegraded":"2021-12-15T17:17:54.829866+0000","last_fullsized":"2021-12-15T17:17:54.829866+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:43:35.789713+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777417+0000","last_change":"2021-12-15T17:17:43.418873+0000","last_active":"2021-12-15T17:17:54.777417+0000","last_peered":"2021-12-15T17:17:54.777417+0000","last_clean":"2021-12-15T17:17:54.777417+0000","last_became_active":"2021-12-15T17:17:43.418572+0000","last_became_peered":"2021-12-15T17:17:43.418572+0000","last_unstale":"2021-12-15T17:17:54.777417+0000","last_undegraded":"2021-12-15T17:17:54.777417+0000","last_fullsized":"2021-12-15T17:17:54.777417+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:21:49.546873+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777399+0000","last_change":"2021-12-15T17:17:45.419509+0000","last_active":"2021-12-15T17:17:54.777399+0000","last_peered":"2021-12-15T17:17:54.777399+0000","last_clean":"2021-12-15T17:17:54.777399+0000","last_became_active":"2021-12-15T17:17:45.419095+0000","last_became_peered":"2021-12-15T17:17:45.419095+0000","last_unstale":"2021-12-15T17:17:54.777399+0000","last_undegraded":"2021-12-15T17:17:54.777399+0000","last_fullsized":"2021-12-15T17:17:54.777399+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:48:51.809132+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777165+0000","last_change":"2021-12-15T17:17:49.441156+0000","last_active":"2021-12-15T17:17:54.777165+0000","last_peered":"2021-12-15T17:17:54.777165+0000","last_clean":"2021-12-15T17:17:54.777165+0000","last_became_active":"2021-12-15T17:17:49.440837+0000","last_became_peered":"2021-12-15T17:17:49.440837+0000","last_unstale":"2021-12-15T17:17:54.777165+0000","last_undegraded":"2021-12-15T17:17:54.777165+0000","last_fullsized":"2021-12-15T17:17:54.777165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:21:36.995185+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"100'8","reported_seq":36,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721436+0000","last_change":"2021-12-15T17:17:47.438906+0000","last_active":"2021-12-15T17:18:41.721436+0000","last_peered":"2021-12-15T17:18:41.721436+0000","last_clean":"2021-12-15T17:18:41.721436+0000","last_became_active":"2021-12-15T17:17:47.438599+0000","last_became_peered":"2021-12-15T17:17:47.438599+0000","last_unstale":"2021-12-15T17:18:41.721436+0000","last_undegraded":"2021-12-15T17:18:41.721436+0000","last_fullsized":"2021-12-15T17:18:41.721436+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:53:07.715077+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830563+0000","last_change":"2021-12-15T17:17:43.413938+0000","last_active":"2021-12-15T17:17:54.830563+0000","last_peered":"2021-12-15T17:17:54.830563+0000","last_clean":"2021-12-15T17:17:54.830563+0000","last_became_active":"2021-12-15T17:17:43.413646+0000","last_became_peered":"2021-12-15T17:17:43.413646+0000","last_unstale":"2021-12-15T17:17:54.830563+0000","last_undegraded":"2021-12-15T17:17:54.830563+0000","last_fullsized":"2021-12-15T17:17:54.830563+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:58:39.548600+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779379+0000","last_change":"2021-12-15T17:17:45.572845+0000","last_active":"2021-12-15T17:17:54.779379+0000","last_peered":"2021-12-15T17:17:54.779379+0000","last_clean":"2021-12-15T17:17:54.779379+0000","last_became_active":"2021-12-15T17:17:45.572502+0000","last_became_peered":"2021-12-15T17:17:45.572502+0000","last_unstale":"2021-12-15T17:17:54.779379+0000","last_undegraded":"2021-12-15T17:17:54.779379+0000","last_fullsized":"2021-12-15T17:17:54.779379+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:32:48.287963+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777682+0000","last_change":"2021-12-15T17:17:49.438435+0000","last_active":"2021-12-15T17:17:54.777682+0000","last_peered":"2021-12-15T17:17:54.777682+0000","last_clean":"2021-12-15T17:17:54.777682+0000","last_became_active":"2021-12-15T17:17:49.437861+0000","last_became_peered":"2021-12-15T17:17:49.437861+0000","last_unstale":"2021-12-15T17:17:54.777682+0000","last_undegraded":"2021-12-15T17:17:54.777682+0000","last_fullsized":"2021-12-15T17:17:54.777682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:58:59.354570+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"100'8","reported_seq":36,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721433+0000","last_change":"2021-12-15T17:17:47.427960+0000","last_active":"2021-12-15T17:18:41.721433+0000","last_peered":"2021-12-15T17:18:41.721433+0000","last_clean":"2021-12-15T17:18:41.721433+0000","last_became_active":"2021-12-15T17:17:47.427581+0000","last_became_peered":"2021-12-15T17:17:47.427581+0000","last_unstale":"2021-12-15T17:18:41.721433+0000","last_undegraded":"2021-12-15T17:18:41.721433+0000","last_fullsized":"2021-12-15T17:18:41.721433+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:49:44.704651+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776951+0000","last_change":"2021-12-15T17:17:43.417472+0000","last_active":"2021-12-15T17:17:54.776951+0000","last_peered":"2021-12-15T17:17:54.776951+0000","last_clean":"2021-12-15T17:17:54.776951+0000","last_became_active":"2021-12-15T17:17:43.417151+0000","last_became_peered":"2021-12-15T17:17:43.417151+0000","last_unstale":"2021-12-15T17:17:54.776951+0000","last_undegraded":"2021-12-15T17:17:54.776951+0000","last_fullsized":"2021-12-15T17:17:54.776951+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:24:16.283189+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831118+0000","last_change":"2021-12-15T17:17:45.573975+0000","last_active":"2021-12-15T17:17:54.831118+0000","last_peered":"2021-12-15T17:17:54.831118+0000","last_clean":"2021-12-15T17:17:54.831118+0000","last_became_active":"2021-12-15T17:17:45.573645+0000","last_became_peered":"2021-12-15T17:17:45.573645+0000","last_unstale":"2021-12-15T17:17:54.831118+0000","last_undegraded":"2021-12-15T17:17:54.831118+0000","last_fullsized":"2021-12-15T17:17:54.831118+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:06:50.053349+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830784+0000","last_change":"2021-12-15T17:17:49.437708+0000","last_active":"2021-12-15T17:17:54.830784+0000","last_peered":"2021-12-15T17:17:54.830784+0000","last_clean":"2021-12-15T17:17:54.830784+0000","last_became_active":"2021-12-15T17:17:49.437412+0000","last_became_peered":"2021-12-15T17:17:49.437412+0000","last_unstale":"2021-12-15T17:17:54.830784+0000","last_undegraded":"2021-12-15T17:17:54.830784+0000","last_fullsized":"2021-12-15T17:17:54.830784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:39:36.145667+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778341+0000","last_change":"2021-12-15T17:17:47.435731+0000","last_active":"2021-12-15T17:17:54.778341+0000","last_peered":"2021-12-15T17:17:54.778341+0000","last_clean":"2021-12-15T17:17:54.778341+0000","last_became_active":"2021-12-15T17:17:47.435345+0000","last_became_peered":"2021-12-15T17:17:47.435345+0000","last_unstale":"2021-12-15T17:17:54.778341+0000","last_undegraded":"2021-12-15T17:17:54.778341+0000","last_fullsized":"2021-12-15T17:17:54.778341+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:14:56.225151+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831692+0000","last_change":"2021-12-15T17:17:43.415236+0000","last_active":"2021-12-15T17:17:54.831692+0000","last_peered":"2021-12-15T17:17:54.831692+0000","last_clean":"2021-12-15T17:17:54.831692+0000","last_became_active":"2021-12-15T17:17:43.414935+0000","last_became_peered":"2021-12-15T17:17:43.414935+0000","last_unstale":"2021-12-15T17:17:54.831692+0000","last_undegraded":"2021-12-15T17:17:54.831692+0000","last_fullsized":"2021-12-15T17:17:54.831692+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:46:42.657026+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777566+0000","last_change":"2021-12-15T17:17:45.574038+0000","last_active":"2021-12-15T17:17:54.777566+0000","last_peered":"2021-12-15T17:17:54.777566+0000","last_clean":"2021-12-15T17:17:54.777566+0000","last_became_active":"2021-12-15T17:17:45.573651+0000","last_became_peered":"2021-12-15T17:17:45.573651+0000","last_unstale":"2021-12-15T17:17:54.777566+0000","last_undegraded":"2021-12-15T17:17:54.777566+0000","last_fullsized":"2021-12-15T17:17:54.777566+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:25:25.200521+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829227+0000","last_change":"2021-12-15T17:17:49.732632+0000","last_active":"2021-12-15T17:17:54.829227+0000","last_peered":"2021-12-15T17:17:54.829227+0000","last_clean":"2021-12-15T17:17:54.829227+0000","last_became_active":"2021-12-15T17:17:49.732209+0000","last_became_peered":"2021-12-15T17:17:49.732209+0000","last_unstale":"2021-12-15T17:17:54.829227+0000","last_undegraded":"2021-12-15T17:17:54.829227+0000","last_fullsized":"2021-12-15T17:17:54.829227+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:09:18.301336+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830040+0000","last_change":"2021-12-15T17:17:47.432909+0000","last_active":"2021-12-15T17:17:54.830040+0000","last_peered":"2021-12-15T17:17:54.830040+0000","last_clean":"2021-12-15T17:17:54.830040+0000","last_became_active":"2021-12-15T17:17:47.432495+0000","last_became_peered":"2021-12-15T17:17:47.432495+0000","last_unstale":"2021-12-15T17:17:54.830040+0000","last_undegraded":"2021-12-15T17:17:54.830040+0000","last_fullsized":"2021-12-15T17:17:54.830040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:32:55.501940+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829033+0000","last_change":"2021-12-15T17:17:43.415983+0000","last_active":"2021-12-15T17:17:54.829033+0000","last_peered":"2021-12-15T17:17:54.829033+0000","last_clean":"2021-12-15T17:17:54.829033+0000","last_became_active":"2021-12-15T17:17:43.415688+0000","last_became_peered":"2021-12-15T17:17:43.415688+0000","last_unstale":"2021-12-15T17:17:54.829033+0000","last_undegraded":"2021-12-15T17:17:54.829033+0000","last_fullsized":"2021-12-15T17:17:54.829033+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:21:45.430437+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"100'6","reported_seq":26,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778508+0000","last_change":"2021-12-15T17:17:45.574024+0000","last_active":"2021-12-15T17:17:54.778508+0000","last_peered":"2021-12-15T17:17:54.778508+0000","last_clean":"2021-12-15T17:17:54.778508+0000","last_became_active":"2021-12-15T17:17:45.573689+0000","last_became_peered":"2021-12-15T17:17:45.573689+0000","last_unstale":"2021-12-15T17:17:54.778508+0000","last_undegraded":"2021-12-15T17:17:54.778508+0000","last_fullsized":"2021-12-15T17:17:54.778508+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:52:45.792453+0000","stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830704+0000","last_change":"2021-12-15T17:17:49.731214+0000","last_active":"2021-12-15T17:17:54.830704+0000","last_peered":"2021-12-15T17:17:54.830704+0000","last_clean":"2021-12-15T17:17:54.830704+0000","last_became_active":"2021-12-15T17:17:49.730894+0000","last_became_peered":"2021-12-15T17:17:49.730894+0000","last_unstale":"2021-12-15T17:17:54.830704+0000","last_undegraded":"2021-12-15T17:17:54.830704+0000","last_fullsized":"2021-12-15T17:17:54.830704+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:26:07.086508+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"92'2","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831169+0000","last_change":"2021-12-15T17:17:43.414458+0000","last_active":"2021-12-15T17:17:54.831169+0000","last_peered":"2021-12-15T17:17:54.831169+0000","last_clean":"2021-12-15T17:17:54.831169+0000","last_became_active":"2021-12-15T17:17:43.413818+0000","last_became_peered":"2021-12-15T17:17:43.413818+0000","last_unstale":"2021-12-15T17:17:54.831169+0000","last_undegraded":"2021-12-15T17:17:54.831169+0000","last_fullsized":"2021-12-15T17:17:54.831169+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:24:24.460096+0000","stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.828972+0000","last_change":"2021-12-15T17:17:45.572866+0000","last_active":"2021-12-15T17:17:54.828972+0000","last_peered":"2021-12-15T17:17:54.828972+0000","last_clean":"2021-12-15T17:17:54.828972+0000","last_became_active":"2021-12-15T17:17:45.572582+0000","last_became_peered":"2021-12-15T17:17:45.572582+0000","last_unstale":"2021-12-15T17:17:54.828972+0000","last_undegraded":"2021-12-15T17:17:54.828972+0000","last_fullsized":"2021-12-15T17:17:54.828972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:40:34.911283+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'8","reported_seq":39,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721472+0000","last_change":"2021-12-15T17:17:47.442765+0000","last_active":"2021-12-15T17:18:41.721472+0000","last_peered":"2021-12-15T17:18:41.721472+0000","last_clean":"2021-12-15T17:18:41.721472+0000","last_became_active":"2021-12-15T17:17:47.442441+0000","last_became_peered":"2021-12-15T17:17:47.442441+0000","last_unstale":"2021-12-15T17:18:41.721472+0000","last_undegraded":"2021-12-15T17:18:41.721472+0000","last_fullsized":"2021-12-15T17:18:41.721472+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:40:30.648434+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777451+0000","last_change":"2021-12-15T17:17:49.443281+0000","last_active":"2021-12-15T17:17:54.777451+0000","last_peered":"2021-12-15T17:17:54.777451+0000","last_clean":"2021-12-15T17:17:54.777451+0000","last_became_active":"2021-12-15T17:17:49.442905+0000","last_became_peered":"2021-12-15T17:17:49.442905+0000","last_unstale":"2021-12-15T17:17:54.777451+0000","last_undegraded":"2021-12-15T17:17:54.777451+0000","last_fullsized":"2021-12-15T17:17:54.777451+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:12:38.744110+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829622+0000","last_change":"2021-12-15T17:17:43.415621+0000","last_active":"2021-12-15T17:17:54.829622+0000","last_peered":"2021-12-15T17:17:54.829622+0000","last_clean":"2021-12-15T17:17:54.829622+0000","last_became_active":"2021-12-15T17:17:43.415318+0000","last_became_peered":"2021-12-15T17:17:43.415318+0000","last_unstale":"2021-12-15T17:17:54.829622+0000","last_undegraded":"2021-12-15T17:17:54.829622+0000","last_fullsized":"2021-12-15T17:17:54.829622+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:48:23.145536+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"100'12","reported_seq":40,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777261+0000","last_change":"2021-12-15T17:17:45.422470+0000","last_active":"2021-12-15T17:17:54.777261+0000","last_peered":"2021-12-15T17:17:54.777261+0000","last_clean":"2021-12-15T17:17:54.777261+0000","last_became_active":"2021-12-15T17:17:45.422161+0000","last_became_peered":"2021-12-15T17:17:45.422161+0000","last_unstale":"2021-12-15T17:17:54.777261+0000","last_undegraded":"2021-12-15T17:17:54.777261+0000","last_fullsized":"2021-12-15T17:17:54.777261+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:46:33.717392+0000","stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829100+0000","last_change":"2021-12-15T17:17:47.439693+0000","last_active":"2021-12-15T17:17:54.829100+0000","last_peered":"2021-12-15T17:17:54.829100+0000","last_clean":"2021-12-15T17:17:54.829100+0000","last_became_active":"2021-12-15T17:17:47.439356+0000","last_became_peered":"2021-12-15T17:17:47.439356+0000","last_unstale":"2021-12-15T17:17:54.829100+0000","last_undegraded":"2021-12-15T17:17:54.829100+0000","last_fullsized":"2021-12-15T17:17:54.829100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:34:31.291815+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830038+0000","last_change":"2021-12-15T17:17:49.443784+0000","last_active":"2021-12-15T17:17:54.830038+0000","last_peered":"2021-12-15T17:17:54.830038+0000","last_clean":"2021-12-15T17:17:54.830038+0000","last_became_active":"2021-12-15T17:17:49.443473+0000","last_became_peered":"2021-12-15T17:17:49.443473+0000","last_unstale":"2021-12-15T17:17:54.830038+0000","last_undegraded":"2021-12-15T17:17:54.830038+0000","last_fullsized":"2021-12-15T17:17:54.830038+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:04:48.645023+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830360+0000","last_change":"2021-12-15T17:17:47.434182+0000","last_active":"2021-12-15T17:17:54.830360+0000","last_peered":"2021-12-15T17:17:54.830360+0000","last_clean":"2021-12-15T17:17:54.830360+0000","last_became_active":"2021-12-15T17:17:47.433835+0000","last_became_peered":"2021-12-15T17:17:47.433835+0000","last_unstale":"2021-12-15T17:17:54.830360+0000","last_undegraded":"2021-12-15T17:17:54.830360+0000","last_fullsized":"2021-12-15T17:17:54.830360+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:56:47.451092+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829525+0000","last_change":"2021-12-15T17:17:43.417488+0000","last_active":"2021-12-15T17:17:54.829525+0000","last_peered":"2021-12-15T17:17:54.829525+0000","last_clean":"2021-12-15T17:17:54.829525+0000","last_became_active":"2021-12-15T17:17:43.416657+0000","last_became_peered":"2021-12-15T17:17:43.416657+0000","last_unstale":"2021-12-15T17:17:54.829525+0000","last_undegraded":"2021-12-15T17:17:54.829525+0000","last_fullsized":"2021-12-15T17:17:54.829525+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:50:12.739994+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"100'19","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830250+0000","last_change":"2021-12-15T17:17:45.573209+0000","last_active":"2021-12-15T17:17:54.830250+0000","last_peered":"2021-12-15T17:17:54.830250+0000","last_clean":"2021-12-15T17:17:54.830250+0000","last_became_active":"2021-12-15T17:17:45.572890+0000","last_became_peered":"2021-12-15T17:17:45.572890+0000","last_unstale":"2021-12-15T17:17:54.830250+0000","last_undegraded":"2021-12-15T17:17:54.830250+0000","last_fullsized":"2021-12-15T17:17:54.830250+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:03:09.778593+0000","stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776815+0000","last_change":"2021-12-15T17:17:49.441166+0000","last_active":"2021-12-15T17:17:54.776815+0000","last_peered":"2021-12-15T17:17:54.776815+0000","last_clean":"2021-12-15T17:17:54.776815+0000","last_became_active":"2021-12-15T17:17:49.440803+0000","last_became_peered":"2021-12-15T17:17:49.440803+0000","last_unstale":"2021-12-15T17:17:54.776815+0000","last_undegraded":"2021-12-15T17:17:54.776815+0000","last_fullsized":"2021-12-15T17:17:54.776815+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:02:42.913488+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829966+0000","last_change":"2021-12-15T17:17:47.431130+0000","last_active":"2021-12-15T17:17:54.829966+0000","last_peered":"2021-12-15T17:17:54.829966+0000","last_clean":"2021-12-15T17:17:54.829966+0000","last_became_active":"2021-12-15T17:17:47.430798+0000","last_became_peered":"2021-12-15T17:17:47.430798+0000","last_unstale":"2021-12-15T17:17:54.829966+0000","last_undegraded":"2021-12-15T17:17:54.829966+0000","last_fullsized":"2021-12-15T17:17:54.829966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:26:06.673982+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829714+0000","last_change":"2021-12-15T17:17:43.419864+0000","last_active":"2021-12-15T17:17:54.829714+0000","last_peered":"2021-12-15T17:17:54.829714+0000","last_clean":"2021-12-15T17:17:54.829714+0000","last_became_active":"2021-12-15T17:17:43.419576+0000","last_became_peered":"2021-12-15T17:17:43.419576+0000","last_unstale":"2021-12-15T17:17:54.829714+0000","last_undegraded":"2021-12-15T17:17:54.829714+0000","last_fullsized":"2021-12-15T17:17:54.829714+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:31:24.882855+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"100'9","reported_seq":33,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778541+0000","last_change":"2021-12-15T17:17:45.425317+0000","last_active":"2021-12-15T17:17:54.778541+0000","last_peered":"2021-12-15T17:17:54.778541+0000","last_clean":"2021-12-15T17:17:54.778541+0000","last_became_active":"2021-12-15T17:17:45.424975+0000","last_became_peered":"2021-12-15T17:17:45.424975+0000","last_unstale":"2021-12-15T17:17:54.778541+0000","last_undegraded":"2021-12-15T17:17:54.778541+0000","last_fullsized":"2021-12-15T17:17:54.778541+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:17:35.795389+0000","stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778017+0000","last_change":"2021-12-15T17:17:49.438882+0000","last_active":"2021-12-15T17:17:54.778017+0000","last_peered":"2021-12-15T17:17:54.778017+0000","last_clean":"2021-12-15T17:17:54.778017+0000","last_became_active":"2021-12-15T17:17:49.437700+0000","last_became_peered":"2021-12-15T17:17:49.437700+0000","last_unstale":"2021-12-15T17:17:54.778017+0000","last_undegraded":"2021-12-15T17:17:54.778017+0000","last_fullsized":"2021-12-15T17:17:54.778017+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:46:14.072600+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"95'2","reported_seq":26,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830406+0000","last_change":"2021-12-15T17:17:46.424639+0000","last_active":"2021-12-15T17:17:54.830406+0000","last_peered":"2021-12-15T17:17:54.830406+0000","last_clean":"2021-12-15T17:17:54.830406+0000","last_became_active":"2021-12-15T17:17:43.416157+0000","last_became_peered":"2021-12-15T17:17:43.416157+0000","last_unstale":"2021-12-15T17:17:54.830406+0000","last_undegraded":"2021-12-15T17:17:54.830406+0000","last_fullsized":"2021-12-15T17:17:54.830406+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:44:21.915689+0000","stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777115+0000","last_change":"2021-12-15T17:17:43.419574+0000","last_active":"2021-12-15T17:17:54.777115+0000","last_peered":"2021-12-15T17:17:54.777115+0000","last_clean":"2021-12-15T17:17:54.777115+0000","last_became_active":"2021-12-15T17:17:43.419149+0000","last_became_peered":"2021-12-15T17:17:43.419149+0000","last_unstale":"2021-12-15T17:17:54.777115+0000","last_undegraded":"2021-12-15T17:17:54.777115+0000","last_fullsized":"2021-12-15T17:17:54.777115+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:53:37.171963+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"100'28","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721396+0000","last_change":"2021-12-15T17:17:45.426120+0000","last_active":"2021-12-15T17:18:41.721396+0000","last_peered":"2021-12-15T17:18:41.721396+0000","last_clean":"2021-12-15T17:18:41.721396+0000","last_became_active":"2021-12-15T17:17:45.425783+0000","last_became_peered":"2021-12-15T17:17:45.425783+0000","last_unstale":"2021-12-15T17:18:41.721396+0000","last_undegraded":"2021-12-15T17:18:41.721396+0000","last_fullsized":"2021-12-15T17:18:41.721396+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:54:20.609566+0000","stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777556+0000","last_change":"2021-12-15T17:17:47.434873+0000","last_active":"2021-12-15T17:17:54.777556+0000","last_peered":"2021-12-15T17:17:54.777556+0000","last_clean":"2021-12-15T17:17:54.777556+0000","last_became_active":"2021-12-15T17:17:47.434524+0000","last_became_peered":"2021-12-15T17:17:47.434524+0000","last_unstale":"2021-12-15T17:17:54.777556+0000","last_undegraded":"2021-12-15T17:17:54.777556+0000","last_fullsized":"2021-12-15T17:17:54.777556+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:16:49.589922+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829670+0000","last_change":"2021-12-15T17:17:49.732497+0000","last_active":"2021-12-15T17:17:54.829670+0000","last_peered":"2021-12-15T17:17:54.829670+0000","last_clean":"2021-12-15T17:17:54.829670+0000","last_became_active":"2021-12-15T17:17:49.731820+0000","last_became_peered":"2021-12-15T17:17:49.731820+0000","last_unstale":"2021-12-15T17:17:54.829670+0000","last_undegraded":"2021-12-15T17:17:54.829670+0000","last_fullsized":"2021-12-15T17:17:54.829670+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:29:34.691847+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829377+0000","last_change":"2021-12-15T17:17:43.415232+0000","last_active":"2021-12-15T17:17:54.829377+0000","last_peered":"2021-12-15T17:17:54.829377+0000","last_clean":"2021-12-15T17:17:54.829377+0000","last_became_active":"2021-12-15T17:17:43.414921+0000","last_became_peered":"2021-12-15T17:17:43.414921+0000","last_unstale":"2021-12-15T17:17:54.829377+0000","last_undegraded":"2021-12-15T17:17:54.829377+0000","last_fullsized":"2021-12-15T17:17:54.829377+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:17:42.782361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829178+0000","last_change":"2021-12-15T17:17:46.414933+0000","last_active":"2021-12-15T17:17:54.829178+0000","last_peered":"2021-12-15T17:17:54.829178+0000","last_clean":"2021-12-15T17:17:54.829178+0000","last_became_active":"2021-12-15T17:17:43.404993+0000","last_became_peered":"2021-12-15T17:17:43.404993+0000","last_unstale":"2021-12-15T17:17:54.829178+0000","last_undegraded":"2021-12-15T17:17:54.829178+0000","last_fullsized":"2021-12-15T17:17:54.829178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:00:48.541924+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"100'13","reported_seq":44,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830618+0000","last_change":"2021-12-15T17:17:45.427376+0000","last_active":"2021-12-15T17:17:54.830618+0000","last_peered":"2021-12-15T17:17:54.830618+0000","last_clean":"2021-12-15T17:17:54.830618+0000","last_became_active":"2021-12-15T17:17:45.426543+0000","last_became_peered":"2021-12-15T17:17:45.426543+0000","last_unstale":"2021-12-15T17:17:54.830618+0000","last_undegraded":"2021-12-15T17:17:54.830618+0000","last_fullsized":"2021-12-15T17:17:54.830618+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:48:21.252080+0000","stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830989+0000","last_change":"2021-12-15T17:17:47.433850+0000","last_active":"2021-12-15T17:17:54.830989+0000","last_peered":"2021-12-15T17:17:54.830989+0000","last_clean":"2021-12-15T17:17:54.830989+0000","last_became_active":"2021-12-15T17:17:47.433485+0000","last_became_peered":"2021-12-15T17:17:47.433485+0000","last_unstale":"2021-12-15T17:17:54.830989+0000","last_undegraded":"2021-12-15T17:17:54.830989+0000","last_fullsized":"2021-12-15T17:17:54.830989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:50:16.243563+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829372+0000","last_change":"2021-12-15T17:17:49.438181+0000","last_active":"2021-12-15T17:17:54.829372+0000","last_peered":"2021-12-15T17:17:54.829372+0000","last_clean":"2021-12-15T17:17:54.829372+0000","last_became_active":"2021-12-15T17:17:49.437819+0000","last_became_peered":"2021-12-15T17:17:49.437819+0000","last_unstale":"2021-12-15T17:17:54.829372+0000","last_undegraded":"2021-12-15T17:17:54.829372+0000","last_fullsized":"2021-12-15T17:17:54.829372+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:29:31.064149+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"102'128","reported_seq":294,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:03.228053+0000","last_change":"2021-12-15T17:17:23.848102+0000","last_active":"2021-12-15T17:18:03.228053+0000","last_peered":"2021-12-15T17:18:03.228053+0000","last_clean":"2021-12-15T17:18:03.228053+0000","last_became_active":"2021-12-15T17:17:23.847618+0000","last_became_peered":"2021-12-15T17:17:23.847618+0000","last_unstale":"2021-12-15T17:18:03.228053+0000","last_undegraded":"2021-12-15T17:18:03.228053+0000","last_fullsized":"2021-12-15T17:18:03.228053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:15:00.738443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:15:00.738443+0000","last_clean_scrub_stamp":"2021-12-15T17:15:00.738443+0000","log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:04:14.432173+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'1","reported_seq":28,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777109+0000","last_change":"2021-12-15T17:17:43.412942+0000","last_active":"2021-12-15T17:17:54.777109+0000","last_peered":"2021-12-15T17:17:54.777109+0000","last_clean":"2021-12-15T17:17:54.777109+0000","last_became_active":"2021-12-15T17:17:43.412650+0000","last_became_peered":"2021-12-15T17:17:43.412650+0000","last_unstale":"2021-12-15T17:17:54.777109+0000","last_undegraded":"2021-12-15T17:17:54.777109+0000","last_fullsized":"2021-12-15T17:17:54.777109+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:33:05.763482+0000","stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'14","reported_seq":39,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829674+0000","last_change":"2021-12-15T17:17:45.574126+0000","last_active":"2021-12-15T17:17:54.829674+0000","last_peered":"2021-12-15T17:17:54.829674+0000","last_clean":"2021-12-15T17:17:54.829674+0000","last_became_active":"2021-12-15T17:17:45.573804+0000","last_became_peered":"2021-12-15T17:17:45.573804+0000","last_unstale":"2021-12-15T17:17:54.829674+0000","last_undegraded":"2021-12-15T17:17:54.829674+0000","last_fullsized":"2021-12-15T17:17:54.829674+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:39:49.338571+0000","stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831412+0000","last_change":"2021-12-15T17:17:47.434735+0000","last_active":"2021-12-15T17:17:54.831412+0000","last_peered":"2021-12-15T17:17:54.831412+0000","last_clean":"2021-12-15T17:17:54.831412+0000","last_became_active":"2021-12-15T17:17:47.434420+0000","last_became_peered":"2021-12-15T17:17:47.434420+0000","last_unstale":"2021-12-15T17:17:54.831412+0000","last_undegraded":"2021-12-15T17:17:54.831412+0000","last_fullsized":"2021-12-15T17:17:54.831412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:31:22.712195+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830030+0000","last_change":"2021-12-15T17:17:49.442324+0000","last_active":"2021-12-15T17:17:54.830030+0000","last_peered":"2021-12-15T17:17:54.830030+0000","last_clean":"2021-12-15T17:17:54.830030+0000","last_became_active":"2021-12-15T17:17:49.441989+0000","last_became_peered":"2021-12-15T17:17:49.441989+0000","last_unstale":"2021-12-15T17:17:54.830030+0000","last_undegraded":"2021-12-15T17:17:54.830030+0000","last_fullsized":"2021-12-15T17:17:54.830030+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:11:12.811754+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777063+0000","last_change":"2021-12-15T17:17:43.407169+0000","last_active":"2021-12-15T17:17:54.777063+0000","last_peered":"2021-12-15T17:17:54.777063+0000","last_clean":"2021-12-15T17:17:54.777063+0000","last_became_active":"2021-12-15T17:17:43.406880+0000","last_became_peered":"2021-12-15T17:17:43.406880+0000","last_unstale":"2021-12-15T17:17:54.777063+0000","last_undegraded":"2021-12-15T17:17:54.777063+0000","last_fullsized":"2021-12-15T17:17:54.777063+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:18:42.377136+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"101'5","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:47.202035+0000","last_change":"2021-12-15T17:17:46.424820+0000","last_active":"2021-12-15T17:18:47.202035+0000","last_peered":"2021-12-15T17:18:47.202035+0000","last_clean":"2021-12-15T17:18:47.202035+0000","last_became_active":"2021-12-15T17:17:43.418415+0000","last_became_peered":"2021-12-15T17:17:43.418415+0000","last_unstale":"2021-12-15T17:18:47.202035+0000","last_undegraded":"2021-12-15T17:18:47.202035+0000","last_fullsized":"2021-12-15T17:18:47.202035+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:49:53.495628+0000","stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"100'12","reported_seq":35,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777267+0000","last_change":"2021-12-15T17:17:45.422402+0000","last_active":"2021-12-15T17:17:54.777267+0000","last_peered":"2021-12-15T17:17:54.777267+0000","last_clean":"2021-12-15T17:17:54.777267+0000","last_became_active":"2021-12-15T17:17:45.421776+0000","last_became_peered":"2021-12-15T17:17:45.421776+0000","last_unstale":"2021-12-15T17:17:54.777267+0000","last_undegraded":"2021-12-15T17:17:54.777267+0000","last_fullsized":"2021-12-15T17:17:54.777267+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:43:56.890537+0000","stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829817+0000","last_change":"2021-12-15T17:17:47.439835+0000","last_active":"2021-12-15T17:17:54.829817+0000","last_peered":"2021-12-15T17:17:54.829817+0000","last_clean":"2021-12-15T17:17:54.829817+0000","last_became_active":"2021-12-15T17:17:47.439541+0000","last_became_peered":"2021-12-15T17:17:47.439541+0000","last_unstale":"2021-12-15T17:17:54.829817+0000","last_undegraded":"2021-12-15T17:17:54.829817+0000","last_fullsized":"2021-12-15T17:17:54.829817+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:19:37.238922+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829308+0000","last_change":"2021-12-15T17:17:49.731392+0000","last_active":"2021-12-15T17:17:54.829308+0000","last_peered":"2021-12-15T17:17:54.829308+0000","last_clean":"2021-12-15T17:17:54.829308+0000","last_became_active":"2021-12-15T17:17:49.730995+0000","last_became_peered":"2021-12-15T17:17:49.730995+0000","last_unstale":"2021-12-15T17:17:54.829308+0000","last_undegraded":"2021-12-15T17:17:54.829308+0000","last_fullsized":"2021-12-15T17:17:54.829308+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:11:34.042095+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830372+0000","last_change":"2021-12-15T17:17:43.413575+0000","last_active":"2021-12-15T17:17:54.830372+0000","last_peered":"2021-12-15T17:17:54.830372+0000","last_clean":"2021-12-15T17:17:54.830372+0000","last_became_active":"2021-12-15T17:17:43.413207+0000","last_became_peered":"2021-12-15T17:17:43.413207+0000","last_unstale":"2021-12-15T17:17:54.830372+0000","last_undegraded":"2021-12-15T17:17:54.830372+0000","last_fullsized":"2021-12-15T17:17:54.830372+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:53:54.821004+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"100'19","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778668+0000","last_change":"2021-12-15T17:17:45.573057+0000","last_active":"2021-12-15T17:17:54.778668+0000","last_peered":"2021-12-15T17:17:54.778668+0000","last_clean":"2021-12-15T17:17:54.778668+0000","last_became_active":"2021-12-15T17:17:45.572581+0000","last_became_peered":"2021-12-15T17:17:45.572581+0000","last_unstale":"2021-12-15T17:17:54.778668+0000","last_undegraded":"2021-12-15T17:17:54.778668+0000","last_fullsized":"2021-12-15T17:17:54.778668+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:53:13.658962+0000","stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830334+0000","last_change":"2021-12-15T17:17:47.429700+0000","last_active":"2021-12-15T17:17:54.830334+0000","last_peered":"2021-12-15T17:17:54.830334+0000","last_clean":"2021-12-15T17:17:54.830334+0000","last_became_active":"2021-12-15T17:17:47.429404+0000","last_became_peered":"2021-12-15T17:17:47.429404+0000","last_unstale":"2021-12-15T17:17:54.830334+0000","last_undegraded":"2021-12-15T17:17:54.830334+0000","last_fullsized":"2021-12-15T17:17:54.830334+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:33:53.733979+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778174+0000","last_change":"2021-12-15T17:17:49.440617+0000","last_active":"2021-12-15T17:17:54.778174+0000","last_peered":"2021-12-15T17:17:54.778174+0000","last_clean":"2021-12-15T17:17:54.778174+0000","last_became_active":"2021-12-15T17:17:49.440263+0000","last_became_peered":"2021-12-15T17:17:49.440263+0000","last_unstale":"2021-12-15T17:17:54.778174+0000","last_undegraded":"2021-12-15T17:17:54.778174+0000","last_fullsized":"2021-12-15T17:17:54.778174+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:47:10.769285+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.831002+0000","last_change":"2021-12-15T17:17:43.420003+0000","last_active":"2021-12-15T17:17:54.831002+0000","last_peered":"2021-12-15T17:17:54.831002+0000","last_clean":"2021-12-15T17:17:54.831002+0000","last_became_active":"2021-12-15T17:17:43.419696+0000","last_became_peered":"2021-12-15T17:17:43.419696+0000","last_unstale":"2021-12-15T17:17:54.831002+0000","last_undegraded":"2021-12-15T17:17:54.831002+0000","last_fullsized":"2021-12-15T17:17:54.831002+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:30:08.130096+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829924+0000","last_change":"2021-12-15T17:17:45.423586+0000","last_active":"2021-12-15T17:17:54.829924+0000","last_peered":"2021-12-15T17:17:54.829924+0000","last_clean":"2021-12-15T17:17:54.829924+0000","last_became_active":"2021-12-15T17:17:45.422796+0000","last_became_peered":"2021-12-15T17:17:45.422796+0000","last_unstale":"2021-12-15T17:17:54.829924+0000","last_undegraded":"2021-12-15T17:17:54.829924+0000","last_fullsized":"2021-12-15T17:17:54.829924+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:29:04.901529+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'8","reported_seq":36,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721367+0000","last_change":"2021-12-15T17:17:47.430921+0000","last_active":"2021-12-15T17:18:41.721367+0000","last_peered":"2021-12-15T17:18:41.721367+0000","last_clean":"2021-12-15T17:18:41.721367+0000","last_became_active":"2021-12-15T17:17:47.430561+0000","last_became_peered":"2021-12-15T17:17:47.430561+0000","last_unstale":"2021-12-15T17:18:41.721367+0000","last_undegraded":"2021-12-15T17:18:41.721367+0000","last_fullsized":"2021-12-15T17:18:41.721367+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:30:31.785850+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830512+0000","last_change":"2021-12-15T17:17:49.731518+0000","last_active":"2021-12-15T17:17:54.830512+0000","last_peered":"2021-12-15T17:17:54.830512+0000","last_clean":"2021-12-15T17:17:54.830512+0000","last_became_active":"2021-12-15T17:17:49.731210+0000","last_became_peered":"2021-12-15T17:17:49.731210+0000","last_unstale":"2021-12-15T17:17:54.830512+0000","last_undegraded":"2021-12-15T17:17:54.830512+0000","last_fullsized":"2021-12-15T17:17:54.830512+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:16:08.107381+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777367+0000","last_change":"2021-12-15T17:17:43.406077+0000","last_active":"2021-12-15T17:17:54.777367+0000","last_peered":"2021-12-15T17:17:54.777367+0000","last_clean":"2021-12-15T17:17:54.777367+0000","last_became_active":"2021-12-15T17:17:43.405892+0000","last_became_peered":"2021-12-15T17:17:43.405892+0000","last_unstale":"2021-12-15T17:17:54.777367+0000","last_undegraded":"2021-12-15T17:17:54.777367+0000","last_fullsized":"2021-12-15T17:17:54.777367+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:04:01.364809+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"100'14","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777116+0000","last_change":"2021-12-15T17:17:45.418354+0000","last_active":"2021-12-15T17:17:54.777116+0000","last_peered":"2021-12-15T17:17:54.777116+0000","last_clean":"2021-12-15T17:17:54.777116+0000","last_became_active":"2021-12-15T17:17:45.418078+0000","last_became_peered":"2021-12-15T17:17:45.418078+0000","last_unstale":"2021-12-15T17:17:54.777116+0000","last_undegraded":"2021-12-15T17:17:54.777116+0000","last_fullsized":"2021-12-15T17:17:54.777116+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:46:15.695736+0000","stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"100'8","reported_seq":36,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721386+0000","last_change":"2021-12-15T17:17:47.435779+0000","last_active":"2021-12-15T17:18:41.721386+0000","last_peered":"2021-12-15T17:18:41.721386+0000","last_clean":"2021-12-15T17:18:41.721386+0000","last_became_active":"2021-12-15T17:17:47.435401+0000","last_became_peered":"2021-12-15T17:17:47.435401+0000","last_unstale":"2021-12-15T17:18:41.721386+0000","last_undegraded":"2021-12-15T17:18:41.721386+0000","last_fullsized":"2021-12-15T17:18:41.721386+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T04:23:58.681929+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777020+0000","last_change":"2021-12-15T17:17:49.437323+0000","last_active":"2021-12-15T17:17:54.777020+0000","last_peered":"2021-12-15T17:17:54.777020+0000","last_clean":"2021-12-15T17:17:54.777020+0000","last_became_active":"2021-12-15T17:17:49.437042+0000","last_became_peered":"2021-12-15T17:17:49.437042+0000","last_unstale":"2021-12-15T17:17:54.777020+0000","last_undegraded":"2021-12-15T17:17:54.777020+0000","last_fullsized":"2021-12-15T17:17:54.777020+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:21:19.573770+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778035+0000","last_change":"2021-12-15T17:17:43.414204+0000","last_active":"2021-12-15T17:17:54.778035+0000","last_peered":"2021-12-15T17:17:54.778035+0000","last_clean":"2021-12-15T17:17:54.778035+0000","last_became_active":"2021-12-15T17:17:43.413812+0000","last_became_peered":"2021-12-15T17:17:43.413812+0000","last_unstale":"2021-12-15T17:17:54.778035+0000","last_undegraded":"2021-12-15T17:17:54.778035+0000","last_fullsized":"2021-12-15T17:17:54.778035+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:27:03.708324+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"100'18","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778790+0000","last_change":"2021-12-15T17:17:45.573058+0000","last_active":"2021-12-15T17:17:54.778790+0000","last_peered":"2021-12-15T17:17:54.778790+0000","last_clean":"2021-12-15T17:17:54.778790+0000","last_became_active":"2021-12-15T17:17:45.572536+0000","last_became_peered":"2021-12-15T17:17:45.572536+0000","last_unstale":"2021-12-15T17:17:54.778790+0000","last_undegraded":"2021-12-15T17:17:54.778790+0000","last_fullsized":"2021-12-15T17:17:54.778790+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:02:08.628715+0000","stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776876+0000","last_change":"2021-12-15T17:17:47.439943+0000","last_active":"2021-12-15T17:17:54.776876+0000","last_peered":"2021-12-15T17:17:54.776876+0000","last_clean":"2021-12-15T17:17:54.776876+0000","last_became_active":"2021-12-15T17:17:47.439691+0000","last_became_peered":"2021-12-15T17:17:47.439691+0000","last_unstale":"2021-12-15T17:17:54.776876+0000","last_undegraded":"2021-12-15T17:17:54.776876+0000","last_fullsized":"2021-12-15T17:17:54.776876+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:35:16.226157+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830855+0000","last_change":"2021-12-15T17:17:49.444143+0000","last_active":"2021-12-15T17:17:54.830855+0000","last_peered":"2021-12-15T17:17:54.830855+0000","last_clean":"2021-12-15T17:17:54.830855+0000","last_became_active":"2021-12-15T17:17:49.443881+0000","last_became_peered":"2021-12-15T17:17:49.443881+0000","last_unstale":"2021-12-15T17:17:54.830855+0000","last_undegraded":"2021-12-15T17:17:54.830855+0000","last_fullsized":"2021-12-15T17:17:54.830855+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T17:47:11.561021+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"100'8","reported_seq":36,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721435+0000","last_change":"2021-12-15T17:17:47.431998+0000","last_active":"2021-12-15T17:18:41.721435+0000","last_peered":"2021-12-15T17:18:41.721435+0000","last_clean":"2021-12-15T17:18:41.721435+0000","last_became_active":"2021-12-15T17:17:47.431109+0000","last_became_peered":"2021-12-15T17:17:47.431109+0000","last_unstale":"2021-12-15T17:18:41.721435+0000","last_undegraded":"2021-12-15T17:18:41.721435+0000","last_fullsized":"2021-12-15T17:18:41.721435+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:48:06.202277+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.778690+0000","last_change":"2021-12-15T17:17:43.414849+0000","last_active":"2021-12-15T17:17:54.778690+0000","last_peered":"2021-12-15T17:17:54.778690+0000","last_clean":"2021-12-15T17:17:54.778690+0000","last_became_active":"2021-12-15T17:17:43.414544+0000","last_became_peered":"2021-12-15T17:17:43.414544+0000","last_unstale":"2021-12-15T17:17:54.778690+0000","last_undegraded":"2021-12-15T17:17:54.778690+0000","last_fullsized":"2021-12-15T17:17:54.778690+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:31:28.998600+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"100'15","reported_seq":42,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830798+0000","last_change":"2021-12-15T17:17:45.427449+0000","last_active":"2021-12-15T17:17:54.830798+0000","last_peered":"2021-12-15T17:17:54.830798+0000","last_clean":"2021-12-15T17:17:54.830798+0000","last_became_active":"2021-12-15T17:17:45.427007+0000","last_became_peered":"2021-12-15T17:17:45.427007+0000","last_unstale":"2021-12-15T17:17:54.830798+0000","last_undegraded":"2021-12-15T17:17:54.830798+0000","last_fullsized":"2021-12-15T17:17:54.830798+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T03:50:20.208164+0000","stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.779326+0000","last_change":"2021-12-15T17:17:49.731619+0000","last_active":"2021-12-15T17:17:54.779326+0000","last_peered":"2021-12-15T17:17:54.779326+0000","last_clean":"2021-12-15T17:17:54.779326+0000","last_became_active":"2021-12-15T17:17:49.731230+0000","last_became_peered":"2021-12-15T17:17:49.731230+0000","last_unstale":"2021-12-15T17:17:54.779326+0000","last_undegraded":"2021-12-15T17:17:54.779326+0000","last_fullsized":"2021-12-15T17:17:54.779326+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T01:33:22.796985+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.829740+0000","last_change":"2021-12-15T17:17:47.433607+0000","last_active":"2021-12-15T17:17:54.829740+0000","last_peered":"2021-12-15T17:17:54.829740+0000","last_clean":"2021-12-15T17:17:54.829740+0000","last_became_active":"2021-12-15T17:17:47.433263+0000","last_became_peered":"2021-12-15T17:17:47.433263+0000","last_unstale":"2021-12-15T17:17:54.829740+0000","last_undegraded":"2021-12-15T17:17:54.829740+0000","last_fullsized":"2021-12-15T17:17:54.829740+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:44:51.917330+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.776903+0000","last_change":"2021-12-15T17:17:43.413835+0000","last_active":"2021-12-15T17:17:54.776903+0000","last_peered":"2021-12-15T17:17:54.776903+0000","last_clean":"2021-12-15T17:17:54.776903+0000","last_became_active":"2021-12-15T17:17:43.413530+0000","last_became_peered":"2021-12-15T17:17:43.413530+0000","last_unstale":"2021-12-15T17:17:54.776903+0000","last_undegraded":"2021-12-15T17:17:54.776903+0000","last_fullsized":"2021-12-15T17:17:54.776903+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T21:01:57.843942+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"100'11","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777548+0000","last_change":"2021-12-15T17:17:45.421988+0000","last_active":"2021-12-15T17:17:54.777548+0000","last_peered":"2021-12-15T17:17:54.777548+0000","last_clean":"2021-12-15T17:17:54.777548+0000","last_became_active":"2021-12-15T17:17:45.421646+0000","last_became_peered":"2021-12-15T17:17:45.421646+0000","last_unstale":"2021-12-15T17:17:54.777548+0000","last_undegraded":"2021-12-15T17:17:54.777548+0000","last_fullsized":"2021-12-15T17:17:54.777548+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T00:05:21.540941+0000","stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":13,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830720+0000","last_change":"2021-12-15T17:17:49.438323+0000","last_active":"2021-12-15T17:17:54.830720+0000","last_peered":"2021-12-15T17:17:54.830720+0000","last_clean":"2021-12-15T17:17:54.830720+0000","last_became_active":"2021-12-15T17:17:49.437986+0000","last_became_peered":"2021-12-15T17:17:49.437986+0000","last_unstale":"2021-12-15T17:17:54.830720+0000","last_undegraded":"2021-12-15T17:17:54.830720+0000","last_fullsized":"2021-12-15T17:17:54.830720+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T18:35:28.244626+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830638+0000","last_change":"2021-12-15T17:17:47.438479+0000","last_active":"2021-12-15T17:17:54.830638+0000","last_peered":"2021-12-15T17:17:54.830638+0000","last_clean":"2021-12-15T17:17:54.830638+0000","last_became_active":"2021-12-15T17:17:47.438106+0000","last_became_peered":"2021-12-15T17:17:47.438106+0000","last_unstale":"2021-12-15T17:17:54.830638+0000","last_undegraded":"2021-12-15T17:17:54.830638+0000","last_fullsized":"2021-12-15T17:17:54.830638+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T05:01:49.895601+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"92'1","reported_seq":26,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.830084+0000","last_change":"2021-12-15T17:17:43.412118+0000","last_active":"2021-12-15T17:17:54.830084+0000","last_peered":"2021-12-15T17:17:54.830084+0000","last_clean":"2021-12-15T17:17:54.830084+0000","last_became_active":"2021-12-15T17:17:43.411789+0000","last_became_peered":"2021-12-15T17:17:43.411789+0000","last_unstale":"2021-12-15T17:17:54.830084+0000","last_undegraded":"2021-12-15T17:17:54.830084+0000","last_fullsized":"2021-12-15T17:17:54.830084+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-17T02:27:22.907469+0000","stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"100'17","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777423+0000","last_change":"2021-12-15T17:17:45.419285+0000","last_active":"2021-12-15T17:17:54.777423+0000","last_peered":"2021-12-15T17:17:54.777423+0000","last_clean":"2021-12-15T17:17:54.777423+0000","last_became_active":"2021-12-15T17:17:45.418977+0000","last_became_peered":"2021-12-15T17:17:45.418977+0000","last_unstale":"2021-12-15T17:17:54.777423+0000","last_undegraded":"2021-12-15T17:17:54.777423+0000","last_fullsized":"2021-12-15T17:17:54.777423+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T19:13:27.066711+0000","stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"100'1","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777713+0000","last_change":"2021-12-15T17:17:49.441280+0000","last_active":"2021-12-15T17:17:54.777713+0000","last_peered":"2021-12-15T17:17:54.777713+0000","last_clean":"2021-12-15T17:17:54.777713+0000","last_became_active":"2021-12-15T17:17:49.440942+0000","last_became_peered":"2021-12-15T17:17:49.440942+0000","last_unstale":"2021-12-15T17:17:54.777713+0000","last_undegraded":"2021-12-15T17:17:54.777713+0000","last_fullsized":"2021-12-15T17:17:54.777713+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:48.418562+0000","last_clean_scrub_stamp":"2021-12-15T17:17:48.418562+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:12:04.128538+0000","stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"100'8","reported_seq":36,"reported_epoch":102,"state":"active+clean","last_fresh":"2021-12-15T17:18:41.721386+0000","last_change":"2021-12-15T17:17:47.433808+0000","last_active":"2021-12-15T17:18:41.721386+0000","last_peered":"2021-12-15T17:18:41.721386+0000","last_clean":"2021-12-15T17:18:41.721386+0000","last_became_active":"2021-12-15T17:17:47.433435+0000","last_became_peered":"2021-12-15T17:17:47.433435+0000","last_unstale":"2021-12-15T17:18:41.721386+0000","last_undegraded":"2021-12-15T17:18:41.721386+0000","last_fullsized":"2021-12-15T17:18:41.721386+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:46.408036+0000","last_clean_scrub_stamp":"2021-12-15T17:17:46.408036+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T23:11:40.036969+0000","stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777744+0000","last_change":"2021-12-15T17:17:43.408339+0000","last_active":"2021-12-15T17:17:54.777744+0000","last_peered":"2021-12-15T17:17:54.777744+0000","last_clean":"2021-12-15T17:17:54.777744+0000","last_became_active":"2021-12-15T17:17:43.408022+0000","last_became_peered":"2021-12-15T17:17:43.408022+0000","last_unstale":"2021-12-15T17:17:54.777744+0000","last_undegraded":"2021-12-15T17:17:54.777744+0000","last_fullsized":"2021-12-15T17:17:54.777744+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:42.389802+0000","last_clean_scrub_stamp":"2021-12-15T17:17:42.389802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T20:41:06.580984+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"100'10","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2021-12-15T17:17:54.777290+0000","last_change":"2021-12-15T17:17:45.416524+0000","last_active":"2021-12-15T17:17:54.777290+0000","last_peered":"2021-12-15T17:17:54.777290+0000","last_clean":"2021-12-15T17:17:54.777290+0000","last_became_active":"2021-12-15T17:17:45.416222+0000","last_became_peered":"2021-12-15T17:17:45.416222+0000","last_unstale":"2021-12-15T17:17:54.777290+0000","last_undegraded":"2021-12-15T17:17:54.777290+0000","last_fullsized":"2021-12-15T17:17:54.777290+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-12-15T17:17:44.397545+0000","last_clean_scrub_stamp":"2021-12-15T17:17:44.397545+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2021-12-16T22:03:22.824992+0000","stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1327,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":77,"seq":330712481813,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7024,"kb_used_data":1512,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738128,"statfs":{"total":95995035648,"available":95987843072,"internally_reserved":0,"allocated":1548288,"data_stored":978393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.95399999999999996}]},{"osd":2,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.623},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.70999999999999996}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.505},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48599999999999999}]},{"osd":5,"last update":"Wed Dec 15 17:18:38 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Wed Dec 15 17:18:25 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.65000000000000002}]}]},{"osd":6,"up_from":83,"seq":356482285587,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7320,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":5822,"kb_avail":93737832,"statfs":{"total":95995035648,"available":95987539968,"internally_reserved":0,"allocated":1523712,"data_stored":977724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":5962696},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.436}]},{"osd":2,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54500000000000004}]},{"osd":3,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.443},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55300000000000005}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.33400000000000002}]},{"osd":5,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.317},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.29099999999999998}]},{"osd":7,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.35399999999999998}]}]},{"osd":1,"up_from":43,"seq":184683593768,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11544,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":10558,"kb_avail":93733608,"statfs":{"total":95995035648,"available":95983214592,"internally_reserved":0,"allocated":999424,"data_stored":389042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":10812360},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.42699999999999999,"15min":0.39300000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.585,"5min":4.585,"15min":4.585},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.42099999999999999,"15min":0.42699999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.41999999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.6130000000000004,"5min":4.6130000000000004,"15min":4.6130000000000004},"last":0.313}]},{"osd":3,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47399999999999998,"15min":0.46400000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.4630000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48399999999999999,"15min":0.48599999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.35299999999999998}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.40300000000000002}]},{"osd":5,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":4.4649999999999999,"5min":4.4649999999999999,"15min":4.4649999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.45100000000000001}]},{"osd":6,"last update":"Wed Dec 15 17:18:23 2021","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.378}]},{"osd":7,"last update":"Wed Dec 15 17:18:17 2021","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47099999999999997}]}]},{"osd":0,"up_from":40,"seq":171798691881,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7984,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":6462,"kb_avail":93737168,"statfs":{"total":95995035648,"available":95986860032,"internally_reserved":0,"allocated":1548288,"data_stored":978090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":6618056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 15 17:18:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.39100000000000001,"15min":0.40300000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.39500000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.52600000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.49299999999999999}]},{"osd":2,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.44,"15min":0.45700000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.625,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.42499999999999999,"15min":0.44500000000000001},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.48799999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44600000000000001}]},{"osd":3,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.43099999999999999,"15min":0.437},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54100000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47499999999999998,"15min":0.47999999999999998},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58499999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45900000000000002}]},{"osd":4,"last update":"Wed Dec 15 17:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57199999999999995}]},{"osd":5,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Dec 15 17:18:24 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.629}]},{"osd":7,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.61299999999999999}]}]},{"osd":2,"up_from":36,"seq":154618822698,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6996,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":6078,"kb_avail":93738156,"statfs":{"total":95995035648,"available":95987871744,"internally_reserved":0,"allocated":929792,"data_stored":386857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":6224840},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:39 2021","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.375,"15min":0.378},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.45200000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.57599999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.32400000000000001}]},{"osd":1,"last update":"Wed Dec 15 17:18:41 2021","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41599999999999998,"15min":0.41899999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41599999999999998,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.45500000000000002,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.42299999999999999}]},{"osd":3,"last update":"Wed Dec 15 17:18:30 2021","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.44800000000000001,"15min":0.44900000000000001},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.38300000000000001}]},{"osd":4,"last update":"Wed Dec 15 17:18:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.69399999999999995}]},{"osd":5,"last update":"Wed Dec 15 17:18:39 2021","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.55800000000000005}]},{"osd":6,"last update":"Wed Dec 15 17:18:22 2021","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.74299999999999999}]},{"osd":7,"last update":"Wed Dec 15 17:18:14 2021","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.77300000000000002}]}]},{"osd":3,"up_from":32,"seq":137438953516,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6780,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738372,"statfs":{"total":95995035648,"available":95988092928,"internally_reserved":0,"allocated":970752,"data_stored":388692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:36 2021","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35899999999999999,"15min":0.36299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.36699999999999999,"15min":0.35999999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53900000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.35399999999999998}]},{"osd":1,"last update":"Wed Dec 15 17:18:44 2021","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.45200000000000001,"15min":0.45700000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70699999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.437,"15min":0.44700000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.57499999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.26700000000000002}]},{"osd":2,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40699999999999997,"15min":0.39800000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.34499999999999997}]},{"osd":4,"last update":"Wed Dec 15 17:18:29 2021","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59299999999999997}]},{"osd":5,"last update":"Wed Dec 15 17:18:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55000000000000004}]},{"osd":6,"last update":"Wed Dec 15 17:18:23 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57699999999999996}]},{"osd":7,"last update":"Wed Dec 15 17:18:15 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73299999999999998}]}]},{"osd":4,"up_from":88,"seq":377957122065,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7092,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":6142,"kb_avail":93738060,"statfs":{"total":95995035648,"available":95987773440,"internally_reserved":0,"allocated":962560,"data_stored":387862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":6290376},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.438}]},{"osd":1,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.58299999999999996}]},{"osd":2,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.505},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.53200000000000003}]},{"osd":5,"last update":"Wed Dec 15 17:18:35 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.55900000000000005}]},{"osd":6,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.625},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Wed Dec 15 17:18:27 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.64900000000000002}]}]},{"osd":5,"up_from":90,"seq":386547056656,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6752,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":5822,"kb_avail":93738400,"statfs":{"total":95995035648,"available":95988121600,"internally_reserved":0,"allocated":942080,"data_stored":388172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":5962696},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41499999999999998}]},{"osd":1,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.373},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.496}]},{"osd":6,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.436}]},{"osd":7,"last update":"Wed Dec 15 17:18:37 2021","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-12-15T17:18:52.779 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-12-15T17:18:52.780 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-12-15T17:18:52.780 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-12-15T17:18:52.781 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- ceph health --format=json 2021-12-15T17:18:53.226 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/0ea1c4a2-5dca-11ec-8c31-001a4aab830c/mon.a/config 2021-12-15T17:18:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:53 smithi022 ceph-mon[31360]: from='client.24754 v1:172.21.15.22:0/2732576171' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:18:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:53 smithi022 ceph-mon[31360]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:18:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:53 smithi022 ceph-mon[40250]: from='client.24754 v1:172.21.15.22:0/2732576171' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:18:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:53 smithi022 ceph-mon[40250]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:18:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:53 smithi122 ceph-mon[32255]: from='client.24754 v1:172.21.15.22:0/2732576171' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-12-15T17:18:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:53 smithi122 ceph-mon[32255]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:18:54.475 INFO:teuthology.orchestra.run.smithi022.stdout: 2021-12-15T17:18:54.475 INFO:teuthology.orchestra.run.smithi022.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-12-15T17:18:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/757479191' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-12-15T17:18:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/757479191' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-12-15T17:18:54.933 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-12-15T17:18:54.934 INFO:tasks.cephadm:Setup complete, yielding 2021-12-15T17:18:54.934 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-12-15T17:18:54.946 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi022.front.sepia.ceph.com 2021-12-15T17:18:54.947 DEBUG:teuthology.orchestra.run.smithi022:> sudo cephadm --image quay.ceph.io/ceph-ci/ceph:aa33f58de13d9f2f8820a4f6d9e4438f3d64acb2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ea1c4a2-5dca-11ec-8c31-001a4aab830c -- bash -c 'set -ex 2021-12-15T17:18:54.947 DEBUG:teuthology.orchestra.run.smithi022:> for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:18:54.947 DEBUG:teuthology.orchestra.run.smithi022:> do 2021-12-15T17:18:54.948 DEBUG:teuthology.orchestra.run.smithi022:> echo "rotating key for $f" 2021-12-15T17:18:54.948 DEBUG:teuthology.orchestra.run.smithi022:> K=$(ceph auth get-key $f) 2021-12-15T17:18:54.948 DEBUG:teuthology.orchestra.run.smithi022:> NK="$K" 2021-12-15T17:18:54.949 DEBUG:teuthology.orchestra.run.smithi022:> ceph orch daemon rotate-key $f 2021-12-15T17:18:54.949 DEBUG:teuthology.orchestra.run.smithi022:> while [ "$K" == "$NK" ]; do 2021-12-15T17:18:54.949 DEBUG:teuthology.orchestra.run.smithi022:> sleep 5 2021-12-15T17:18:54.950 DEBUG:teuthology.orchestra.run.smithi022:> NK=$(ceph auth get-key $f) 2021-12-15T17:18:54.950 DEBUG:teuthology.orchestra.run.smithi022:> done 2021-12-15T17:18:54.950 DEBUG:teuthology.orchestra.run.smithi022:> done 2021-12-15T17:18:54.950 DEBUG:teuthology.orchestra.run.smithi022:> ' 2021-12-15T17:18:54.951 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/757479191' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-12-15T17:18:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:18:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:18:55.980 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:55 smithi022 ceph-mon[31360]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:55.980 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:55 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:55.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:55 smithi022 ceph-mon[40250]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:55.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:55 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:55 smithi122 ceph-mon[32255]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:55 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:18:56.325 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.0 2021-12-15T17:18:56.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:18:56.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.0' 2021-12-15T17:18:56.326 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:18:56.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:18:56.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:18:56.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.0 2021-12-15T17:18:56.969 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.0 on host 'smithi022' 2021-12-15T17:18:56.970 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1043058868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:18:56.970 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1043058868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:18:56.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:18:56.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:18:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1043058868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:18:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:18:57] "GET /metrics HTTP/1.1" 200 201863 "" "Prometheus/2.18.1" 2021-12-15T17:18:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:57 smithi122 ceph-mon[32255]: from='client.34787 v1:172.21.15.22:0/1438923202' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:18:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:57 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.0 2021-12-15T17:18:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:57 smithi122 ceph-mon[32255]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:57 smithi022 ceph-mon[31360]: from='client.34787 v1:172.21.15.22:0/1438923202' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:18:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:57 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.0 2021-12-15T17:18:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:57 smithi022 ceph-mon[31360]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:57 smithi022 ceph-mon[40250]: from='client.34787 v1:172.21.15.22:0/1438923202' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:18:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:57 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.0 2021-12-15T17:18:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:57 smithi022 ceph-mon[40250]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:18:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:18:59 smithi022 ceph-mon[31360]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:18:59 smithi022 ceph-mon[40250]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:18:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:18:59 smithi122 ceph-mon[32255]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:19:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:01 smithi022 ceph-mon[31360]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:01 smithi022 ceph-mon[40250]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:01 smithi122 ceph-mon[32255]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:01.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:02.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:02.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:02.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3645403901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3645403901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3645403901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:03 smithi122 ceph-mon[32255]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:19:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:19:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[31360]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:19:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:19:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[40250]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2021-12-15T17:19:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:19:03.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:19:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2021-12-15T17:19:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2021-12-15T17:19:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2021-12-15T17:19:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:04 smithi122 ceph-mon[32255]: osdmap e103: 8 total, 8 up, 8 in 2021-12-15T17:19:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2021-12-15T17:19:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2021-12-15T17:19:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2021-12-15T17:19:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[31360]: osdmap e103: 8 total, 8 up, 8 in 2021-12-15T17:19:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2021-12-15T17:19:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2021-12-15T17:19:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2021-12-15T17:19:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:04 smithi022 ceph-mon[40250]: osdmap e103: 8 total, 8 up, 8 in 2021-12-15T17:19:05.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:19:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:19:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:05 smithi122 ceph-mon[32255]: osdmap e104: 8 total, 8 up, 8 in 2021-12-15T17:19:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:05 smithi122 ceph-mon[32255]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:19:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:05 smithi022 ceph-mon[31360]: osdmap e104: 8 total, 8 up, 8 in 2021-12-15T17:19:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:05 smithi022 ceph-mon[31360]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:19:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:05 smithi022 ceph-mon[40250]: osdmap e104: 8 total, 8 up, 8 in 2021-12-15T17:19:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:05 smithi022 ceph-mon[40250]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:19:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:06 smithi122 ceph-mon[32255]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2021-12-15T17:19:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:06 smithi022 ceph-mon[31360]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2021-12-15T17:19:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:06 smithi022 ceph-mon[40250]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2021-12-15T17:19:07.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:07 smithi122 ceph-mon[32255]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:19:07.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:07.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:07.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:07.727 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:07] "GET /metrics HTTP/1.1" 200 201857 "" "Prometheus/2.18.1" 2021-12-15T17:19:07.727 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:07 smithi022 ceph-mon[31360]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:19:07.728 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:07 smithi022 ceph-mon[40250]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:19:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1480127440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1480127440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1480127440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:09 smithi122 ceph-mon[32255]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2021-12-15T17:19:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:09 smithi022 ceph-mon[31360]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2021-12-15T17:19:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:09 smithi022 ceph-mon[40250]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2021-12-15T17:19:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:19:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:10 smithi122 ceph-mon[32255]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2021-12-15T17:19:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:10 smithi122 ceph-mon[32255]: Cluster is now healthy 2021-12-15T17:19:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:10 smithi022 ceph-mon[31360]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2021-12-15T17:19:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:10 smithi022 ceph-mon[31360]: Cluster is now healthy 2021-12-15T17:19:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:10 smithi022 ceph-mon[40250]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2021-12-15T17:19:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:10 smithi022 ceph-mon[40250]: Cluster is now healthy 2021-12-15T17:19:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:11 smithi122 ceph-mon[32255]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2021-12-15T17:19:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:11 smithi022 ceph-mon[31360]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2021-12-15T17:19:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:11 smithi022 ceph-mon[40250]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2021-12-15T17:19:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:13.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:13.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:13.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1552480790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1552480790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1552480790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:14 smithi122 ceph-mon[32255]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2021-12-15T17:19:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:14 smithi022 ceph-mon[31360]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2021-12-15T17:19:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:14 smithi022 ceph-mon[40250]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2021-12-15T17:19:15.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:19:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:19:16.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:15 smithi122 ceph-mon[32255]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s; 20 B/s, 1 objects/s recovering 2021-12-15T17:19:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:15 smithi022 ceph-mon[31360]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s; 20 B/s, 1 objects/s recovering 2021-12-15T17:19:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:15 smithi022 ceph-mon[40250]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s; 20 B/s, 1 objects/s recovering 2021-12-15T17:19:16.685 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 systemd[1]: Starting Ceph grafana.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:19:16.992 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 podman[74742]: 2021-12-15T17:19:16.994 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:17.130 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:17.131 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:17.246 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2021-12-15T17:19:17.246 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2021-12-15T17:19:17.247 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2021-12-15T17:19:17.247 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2021-12-15T17:19:17.247 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2021-12-15T17:19:17.248 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2021-12-15T17:19:17.248 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2021-12-15T17:19:17.248 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2021-12-15T17:19:17.248 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2021-12-15T17:19:17.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2021-12-15T17:19:17.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2021-12-15T17:19:17.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2021-12-15T17:19:17.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="App mode production" logger=settings 2021-12-15T17:19:17.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Initializing SqlStore" logger=server 2021-12-15T17:19:17.250 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2021-12-15T17:19:17.250 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Starting DB migration" logger=migrator 2021-12-15T17:19:17.250 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2021-12-15T17:19:17.250 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2021-12-15T17:19:17.251 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2021-12-15T17:19:17.251 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:16 smithi122 conmon[74782]: t=2021-12-15T17:19:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2021-12-15T17:19:17.251 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2021-12-15T17:19:17.252 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 bash[74589]: 2d7951b512d19f44673f5c26aeac89d23891025ccdca34f3adcf05b0f0a2a748 2021-12-15T17:19:17.252 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2021-12-15T17:19:17.252 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2021-12-15T17:19:17.252 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2021-12-15T17:19:17.253 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2021-12-15T17:19:17.253 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2021-12-15T17:19:17.253 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-12-15T17:19:17.254 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2021-12-15T17:19:17.254 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2021-12-15T17:19:17.254 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2021-12-15T17:19:17.254 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2021-12-15T17:19:17.255 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2021-12-15T17:19:17.255 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2021-12-15T17:19:17.255 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2021-12-15T17:19:17.255 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2021-12-15T17:19:17.255 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2021-12-15T17:19:17.256 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2021-12-15T17:19:17.256 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2021-12-15T17:19:17.256 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2021-12-15T17:19:17.257 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2021-12-15T17:19:17.257 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2021-12-15T17:19:17.257 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2021-12-15T17:19:17.257 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2021-12-15T17:19:17.258 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2021-12-15T17:19:17.258 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2021-12-15T17:19:17.258 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2021-12-15T17:19:17.259 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2021-12-15T17:19:17.259 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2021-12-15T17:19:17.259 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2021-12-15T17:19:17.259 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2021-12-15T17:19:17.260 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2021-12-15T17:19:17.260 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2021-12-15T17:19:17.260 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2021-12-15T17:19:17.260 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2021-12-15T17:19:17.260 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2021-12-15T17:19:17.261 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2021-12-15T17:19:17.261 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2021-12-15T17:19:17.261 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2021-12-15T17:19:17.262 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2021-12-15T17:19:17.262 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2021-12-15T17:19:17.262 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 systemd[1]: Started Ceph grafana.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:19:17.262 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2021-12-15T17:19:17.263 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2021-12-15T17:19:17.263 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2021-12-15T17:19:17.263 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2021-12-15T17:19:17.263 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2021-12-15T17:19:17.263 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2021-12-15T17:19:17.264 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2021-12-15T17:19:17.264 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2021-12-15T17:19:17.264 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2021-12-15T17:19:17.264 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2021-12-15T17:19:17.265 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2021-12-15T17:19:17.265 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2021-12-15T17:19:17.265 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2021-12-15T17:19:17.265 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2021-12-15T17:19:17.265 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2021-12-15T17:19:17.266 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2021-12-15T17:19:17.266 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2021-12-15T17:19:17.266 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2021-12-15T17:19:17.266 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2021-12-15T17:19:17.266 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2021-12-15T17:19:17.267 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2021-12-15T17:19:17.267 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2021-12-15T17:19:17.267 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2021-12-15T17:19:17.267 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2021-12-15T17:19:17.268 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2021-12-15T17:19:17.268 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2021-12-15T17:19:17.268 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2021-12-15T17:19:17.269 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2021-12-15T17:19:17.269 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2021-12-15T17:19:17.269 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2021-12-15T17:19:17.269 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2021-12-15T17:19:17.269 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2021-12-15T17:19:17.270 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2021-12-15T17:19:17.270 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2021-12-15T17:19:17.270 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2021-12-15T17:19:17.271 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2021-12-15T17:19:17.271 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2021-12-15T17:19:17.271 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-12-15T17:19:17.272 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2021-12-15T17:19:17.272 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2021-12-15T17:19:17.272 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2021-12-15T17:19:17.272 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2021-12-15T17:19:17.273 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2021-12-15T17:19:17.273 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2021-12-15T17:19:17.273 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2021-12-15T17:19:17.273 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2021-12-15T17:19:17.273 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2021-12-15T17:19:17.274 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2021-12-15T17:19:17.274 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2021-12-15T17:19:17.274 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2021-12-15T17:19:17.274 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2021-12-15T17:19:17.275 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2021-12-15T17:19:17.275 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2021-12-15T17:19:17.275 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2021-12-15T17:19:17.275 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2021-12-15T17:19:17.275 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2021-12-15T17:19:17.276 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2021-12-15T17:19:17.276 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2021-12-15T17:19:17.276 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2021-12-15T17:19:17.276 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2021-12-15T17:19:17.276 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2021-12-15T17:19:17.277 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2021-12-15T17:19:17.277 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2021-12-15T17:19:17.277 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2021-12-15T17:19:17.277 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2021-12-15T17:19:17.277 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2021-12-15T17:19:17.278 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2021-12-15T17:19:17.278 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2021-12-15T17:19:17.278 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2021-12-15T17:19:17.278 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2021-12-15T17:19:17.279 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2021-12-15T17:19:17.279 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2021-12-15T17:19:17.279 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2021-12-15T17:19:17.279 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2021-12-15T17:19:17.279 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2021-12-15T17:19:17.280 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2021-12-15T17:19:17.280 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2021-12-15T17:19:17.280 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2021-12-15T17:19:17.280 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2021-12-15T17:19:17.280 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2021-12-15T17:19:17.281 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2021-12-15T17:19:17.281 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2021-12-15T17:19:17.281 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2021-12-15T17:19:17.281 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2021-12-15T17:19:17.281 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2021-12-15T17:19:17.282 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2021-12-15T17:19:17.282 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2021-12-15T17:19:17.282 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2021-12-15T17:19:17.282 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2021-12-15T17:19:17.283 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2021-12-15T17:19:17.283 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2021-12-15T17:19:17.283 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2021-12-15T17:19:17.283 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2021-12-15T17:19:17.283 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2021-12-15T17:19:17.284 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2021-12-15T17:19:17.284 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2021-12-15T17:19:17.284 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2021-12-15T17:19:17.284 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2021-12-15T17:19:17.285 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2021-12-15T17:19:17.285 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2021-12-15T17:19:17.285 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2021-12-15T17:19:17.285 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2021-12-15T17:19:17.285 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2021-12-15T17:19:17.286 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2021-12-15T17:19:17.286 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2021-12-15T17:19:17.286 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2021-12-15T17:19:17.286 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2021-12-15T17:19:17.286 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2021-12-15T17:19:17.287 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2021-12-15T17:19:17.287 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2021-12-15T17:19:17.287 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2021-12-15T17:19:17.287 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2021-12-15T17:19:17.288 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2021-12-15T17:19:17.288 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2021-12-15T17:19:17.288 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2021-12-15T17:19:17.288 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2021-12-15T17:19:17.288 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2021-12-15T17:19:17.289 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2021-12-15T17:19:17.289 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2021-12-15T17:19:17.289 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2021-12-15T17:19:17.289 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2021-12-15T17:19:17.289 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2021-12-15T17:19:17.290 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2021-12-15T17:19:17.290 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2021-12-15T17:19:17.290 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2021-12-15T17:19:17.290 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2021-12-15T17:19:17.291 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2021-12-15T17:19:17.291 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2021-12-15T17:19:17.291 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2021-12-15T17:19:17.291 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2021-12-15T17:19:17.291 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2021-12-15T17:19:17.292 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2021-12-15T17:19:17.292 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2021-12-15T17:19:17.292 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2021-12-15T17:19:17.292 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2021-12-15T17:19:17.292 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2021-12-15T17:19:17.293 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2021-12-15T17:19:17.293 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2021-12-15T17:19:17.293 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2021-12-15T17:19:17.293 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2021-12-15T17:19:17.293 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2021-12-15T17:19:17.294 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2021-12-15T17:19:17.294 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2021-12-15T17:19:17.294 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2021-12-15T17:19:17.294 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2021-12-15T17:19:17.295 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2021-12-15T17:19:17.295 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2021-12-15T17:19:17.295 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2021-12-15T17:19:17.296 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2021-12-15T17:19:17.296 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2021-12-15T17:19:17.296 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2021-12-15T17:19:17.296 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2021-12-15T17:19:17.297 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2021-12-15T17:19:17.297 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2021-12-15T17:19:17.297 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2021-12-15T17:19:17.297 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2021-12-15T17:19:17.297 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2021-12-15T17:19:17.298 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2021-12-15T17:19:17.298 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2021-12-15T17:19:17.298 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2021-12-15T17:19:17.298 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2021-12-15T17:19:17.299 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2021-12-15T17:19:17.299 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2021-12-15T17:19:17.299 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2021-12-15T17:19:17.299 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2021-12-15T17:19:17.299 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2021-12-15T17:19:17.300 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2021-12-15T17:19:17.300 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2021-12-15T17:19:17.300 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2021-12-15T17:19:17.685 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2021-12-15T17:19:17.686 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2021-12-15T17:19:17.686 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2021-12-15T17:19:17.687 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2021-12-15T17:19:17.687 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2021-12-15T17:19:17.688 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2021-12-15T17:19:17.688 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2021-12-15T17:19:17.688 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2021-12-15T17:19:17.689 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2021-12-15T17:19:17.689 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2021-12-15T17:19:17.690 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2021-12-15T17:19:17.690 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2021-12-15T17:19:17.691 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2021-12-15T17:19:17.691 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2021-12-15T17:19:17.691 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2021-12-15T17:19:17.692 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2021-12-15T17:19:17.692 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2021-12-15T17:19:17.693 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2021-12-15T17:19:17.693 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2021-12-15T17:19:17.694 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2021-12-15T17:19:17.694 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2021-12-15T17:19:17.695 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Created default organization" logger=sqlstore 2021-12-15T17:19:17.695 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing HTTPServer" logger=server 2021-12-15T17:19:17.695 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2021-12-15T17:19:17.696 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing PluginManager" logger=server 2021-12-15T17:19:17.696 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Starting plugin search" logger=plugins 2021-12-15T17:19:17.696 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2021-12-15T17:19:17.697 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2021-12-15T17:19:17.697 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing HooksService" logger=server 2021-12-15T17:19:17.697 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2021-12-15T17:19:17.698 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2021-12-15T17:19:17.698 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing RemoteCache" logger=server 2021-12-15T17:19:17.698 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing RenderingService" logger=server 2021-12-15T17:19:17.698 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing AlertEngine" logger=server 2021-12-15T17:19:17.699 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing QuotaService" logger=server 2021-12-15T17:19:17.699 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing ServerLockService" logger=server 2021-12-15T17:19:17.699 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2021-12-15T17:19:17.699 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2021-12-15T17:19:17.699 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing LoginService" logger=server 2021-12-15T17:19:17.700 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing SearchService" logger=server 2021-12-15T17:19:17.700 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing TracingService" logger=server 2021-12-15T17:19:17.700 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2021-12-15T17:19:17.700 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing CleanUpService" logger=server 2021-12-15T17:19:17.701 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing NotificationService" logger=server 2021-12-15T17:19:17.701 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2021-12-15T17:19:17.701 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2021-12-15T17:19:17.701 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2021-12-15T17:19:17.701 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2021-12-15T17:19:17.702 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2021-12-15T17:19:17.702 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="Initializing Stream Manager" 2021-12-15T17:19:17.702 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Dec 15 17:19:17 smithi122 conmon[74782]: t=2021-12-15T17:19:17+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2021-12-15T17:19:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:17] "GET /metrics HTTP/1.1" 200 201857 "" "Prometheus/2.18.1" 2021-12-15T17:19:18.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:18 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:18 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:18 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:18 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:18 smithi122 ceph-mon[32255]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2021-12-15T17:19:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:18.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:18.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:18.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:18.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:18.473 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[31360]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2021-12-15T17:19:18.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:18.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:18.474 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:18.474 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:18.474 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:18 smithi022 ceph-mon[40250]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2021-12-15T17:19:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:19 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3269523388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:19 smithi022 ceph-mon[31360]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 3 op/s; 18 B/s, 1 objects/s recovering 2021-12-15T17:19:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:19 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3269523388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:19 smithi022 ceph-mon[40250]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 3 op/s; 18 B/s, 1 objects/s recovering 2021-12-15T17:19:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:19 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3269523388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:19 smithi122 ceph-mon[32255]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 3 op/s; 18 B/s, 1 objects/s recovering 2021-12-15T17:19:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:19:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:21 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:21 smithi022 ceph-mon[31360]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:21 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:21 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:21 smithi022 ceph-mon[40250]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:21 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:21 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:21 smithi122 ceph-mon[32255]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:21 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[31360]: Rotating authentication key for osd.0 2021-12-15T17:19:22.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2021-12-15T17:19:22.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2021-12-15T17:19:22.973 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.973 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[31360]: Reconfiguring alertmanager.a (dependencies changed)... 2021-12-15T17:19:22.973 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[31360]: Reconfiguring daemon alertmanager.a on smithi022 2021-12-15T17:19:22.974 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.974 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[40250]: Rotating authentication key for osd.0 2021-12-15T17:19:22.974 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2021-12-15T17:19:22.974 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2021-12-15T17:19:22.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:22.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[40250]: Reconfiguring alertmanager.a (dependencies changed)... 2021-12-15T17:19:22.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:22 smithi022 ceph-mon[40250]: Reconfiguring daemon alertmanager.a on smithi022 2021-12-15T17:19:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:22 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:22 smithi122 ceph-mon[32255]: Rotating authentication key for osd.0 2021-12-15T17:19:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:22 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2021-12-15T17:19:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:22 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2021-12-15T17:19:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:22 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:22 smithi122 ceph-mon[32255]: Reconfiguring alertmanager.a (dependencies changed)... 2021-12-15T17:19:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:22 smithi122 ceph-mon[32255]: Reconfiguring daemon alertmanager.a on smithi022 2021-12-15T17:19:23.472 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:24.175 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:23 smithi022 ceph-mon[31360]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:24.175 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3350434516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:24.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:23 smithi022 ceph-mon[40250]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:24.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3350434516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:23 smithi122 ceph-mon[32255]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3350434516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:24.448 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 systemd[1]: Stopping Ceph alertmanager.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:19:24.449 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 bash[93680]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-alertmanager.a" found: no such container 2021-12-15T17:19:24.449 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 conmon[82628]: level=info ts=2021-12-15T17:19:24.348Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2021-12-15T17:19:24.728 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 bash[93680]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-alertmanager-a 2021-12-15T17:19:25.028 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 bash[93680]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-alertmanager.a" found: no such container 2021-12-15T17:19:25.028 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@alertmanager.a.service: Succeeded. 2021-12-15T17:19:25.028 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 systemd[1]: Stopped Ceph alertmanager.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:19:25.029 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:24 smithi022 systemd[1]: Starting Ceph alertmanager.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:19:25.320 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 podman[93998]: 2021-12-15T17:19:25.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:19:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:19:25.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 bash[93847]: 3537b2ea94296f9588c9d74629266da4d0544595c601b8f9696e93039ab955e2 2021-12-15T17:19:25.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:25.328Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-12-15T17:19:25.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:25.328Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-12-15T17:19:25.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:25.329Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.22 port=9094 2021-12-15T17:19:25.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:25.330Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-12-15T17:19:25.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 systemd[1]: Started Ceph alertmanager.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:19:25.780 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:25.350Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-12-15T17:19:25.780 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:25.350Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-12-15T17:19:25.780 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:25 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:25.353Z caller=main.go:497 msg=Listening address=:9093 2021-12-15T17:19:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-12-15T17:19:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2021-12-15T17:19:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-12-15T17:19:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2021-12-15T17:19:26.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-12-15T17:19:26.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2021-12-15T17:19:26.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-12-15T17:19:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2021-12-15T17:19:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-12-15T17:19:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2021-12-15T17:19:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-12-15T17:19:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2021-12-15T17:19:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:26.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:26.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:26.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-12-15T17:19:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2021-12-15T17:19:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-12-15T17:19:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2021-12-15T17:19:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-12-15T17:19:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2021-12-15T17:19:26.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:26.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:26.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:26.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:26 smithi122 ceph-mon[32255]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-12-15T17:19:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:26 smithi122 ceph-mon[32255]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2021-12-15T17:19:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:26 smithi122 ceph-mon[32255]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-12-15T17:19:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:26 smithi122 ceph-mon[32255]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2021-12-15T17:19:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:26 smithi122 ceph-mon[32255]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-12-15T17:19:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:26 smithi122 ceph-mon[32255]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2021-12-15T17:19:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[31360]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-12-15T17:19:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[31360]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2021-12-15T17:19:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[31360]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-12-15T17:19:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[31360]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2021-12-15T17:19:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[31360]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-12-15T17:19:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[31360]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2021-12-15T17:19:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[40250]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-12-15T17:19:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[40250]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.22:9093"}]: dispatch 2021-12-15T17:19:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[40250]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-12-15T17:19:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[40250]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2021-12-15T17:19:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[40250]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-12-15T17:19:27.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:26 smithi022 ceph-mon[40250]: from='mon.0 v1:172.21.15.22:0/2555796505' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2021-12-15T17:19:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:27] "GET /metrics HTTP/1.1" 200 201874 "" "Prometheus/2.18.1" 2021-12-15T17:19:27.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:27 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:27.330Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000140931s 2021-12-15T17:19:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:27 smithi122 ceph-mon[32255]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:27 smithi022 ceph-mon[31360]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:27 smithi022 ceph-mon[40250]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:28.848 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:29.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:29.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:29.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/984677193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:29 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/984677193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/984677193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:19:30.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:19:30.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:29 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:19:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:19:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:31 smithi022 ceph-mon[31360]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:31 smithi022 ceph-mon[40250]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:31 smithi122 ceph-mon[32255]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:33 smithi022 ceph-mon[31360]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:33 smithi022 ceph-mon[40250]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:33 smithi122 ceph-mon[32255]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:34.217 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:34.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:34.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:19:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:19:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:19:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1000701298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:35 smithi122 ceph-mon[32255]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1000701298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:35 smithi022 ceph-mon[31360]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:35.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:19:35 smithi022 conmon[94037]: level=info ts=2021-12-15T17:19:35.331Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001115932s 2021-12-15T17:19:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1000701298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:35 smithi022 ceph-mon[40250]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:37] "GET /metrics HTTP/1.1" 200 201875 "" "Prometheus/2.18.1" 2021-12-15T17:19:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:37 smithi122 ceph-mon[32255]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:37 smithi022 ceph-mon[31360]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:37 smithi022 ceph-mon[40250]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:39 smithi022 ceph-mon[31360]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:39 smithi022 ceph-mon[40250]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:39 smithi122 ceph-mon[32255]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:39.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:39.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:39.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:19:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1308445301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1308445301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1308445301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:41 smithi122 ceph-mon[32255]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:41 smithi022 ceph-mon[31360]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:41 smithi022 ceph-mon[40250]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:43 smithi022 ceph-mon[31360]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:43 smithi022 ceph-mon[40250]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:43 smithi122 ceph-mon[32255]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:44.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:45.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:45.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:45.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:19:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:19:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:45 smithi122 ceph-mon[32255]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3073970478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:45 smithi022 ceph-mon[31360]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3073970478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:45 smithi022 ceph-mon[40250]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3073970478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:47] "GET /metrics HTTP/1.1" 200 201875 "" "Prometheus/2.18.1" 2021-12-15T17:19:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:47 smithi122 ceph-mon[32255]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:47 smithi022 ceph-mon[31360]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:47 smithi022 ceph-mon[40250]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:49 smithi022 ceph-mon[31360]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:49 smithi022 ceph-mon[40250]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:49 smithi122 ceph-mon[32255]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:50.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:19:50.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:50.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:50.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2004832335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2004832335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2004832335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:51 smithi122 ceph-mon[32255]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:51 smithi022 ceph-mon[31360]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:51 smithi022 ceph-mon[40250]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:53 smithi022 ceph-mon[31360]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:53 smithi022 ceph-mon[40250]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:53 smithi122 ceph-mon[32255]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:19:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:19:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:19:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:55 smithi022 ceph-mon[31360]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:55 smithi022 ceph-mon[40250]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:56.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:19:56.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:19:56.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:19:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:55 smithi122 ceph-mon[32255]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2908991477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2908991477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:19:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2908991477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:19:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:19:57] "GET /metrics HTTP/1.1" 200 201876 "" "Prometheus/2.18.1" 2021-12-15T17:19:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:57 smithi122 ceph-mon[32255]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:57 smithi022 ceph-mon[31360]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:57 smithi022 ceph-mon[40250]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:19:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:19:59 smithi022 ceph-mon[31360]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:19:59 smithi022 ceph-mon[40250]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:19:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:19:59 smithi122 ceph-mon[32255]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:20:01.076 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:01.077 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T17:20:01.078 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T17:20:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T17:20:01.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:01.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:01.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:01 smithi122 ceph-mon[32255]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/711850809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:01 smithi022 ceph-mon[31360]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/711850809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:01 smithi022 ceph-mon[40250]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/711850809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:03 smithi022 ceph-mon[31360]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:20:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:20:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:03 smithi022 ceph-mon[40250]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:20:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:20:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:03 smithi122 ceph-mon[32255]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:20:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:20:05.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:20:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:20:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:05 smithi122 ceph-mon[32255]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:05 smithi022 ceph-mon[31360]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:05 smithi022 ceph-mon[40250]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:06.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:06.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:06.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:06.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:07] "GET /metrics HTTP/1.1" 200 201883 "" "Prometheus/2.18.1" 2021-12-15T17:20:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1964184372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:07 smithi122 ceph-mon[32255]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1964184372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:07 smithi022 ceph-mon[31360]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1964184372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:07 smithi022 ceph-mon[40250]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:09 smithi022 ceph-mon[31360]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:09 smithi022 ceph-mon[40250]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:09 smithi122 ceph-mon[32255]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:20:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:11 smithi022 ceph-mon[31360]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:11 smithi022 ceph-mon[40250]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:11 smithi122 ceph-mon[32255]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:11.821 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:12.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:12.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:12.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4070226667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4070226667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4070226667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:13 smithi022 ceph-mon[31360]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:13 smithi022 ceph-mon[40250]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:13 smithi122 ceph-mon[32255]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:15.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:20:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:20:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:15 smithi122 ceph-mon[32255]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:15 smithi022 ceph-mon[31360]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:15 smithi022 ceph-mon[40250]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:17.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:17.192 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:17.192 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:17] "GET /metrics HTTP/1.1" 200 201883 "" "Prometheus/2.18.1" 2021-12-15T17:20:17.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:17.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:17.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:17 smithi022 ceph-mon[31360]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3736007544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:17 smithi022 ceph-mon[40250]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3736007544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:17 smithi122 ceph-mon[32255]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3736007544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:19 smithi022 ceph-mon[31360]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:19 smithi022 ceph-mon[40250]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:19 smithi122 ceph-mon[32255]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:20:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:21 smithi022 ceph-mon[31360]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:21 smithi022 ceph-mon[40250]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:21 smithi122 ceph-mon[32255]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:22.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:22.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:22.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:22.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1928158506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1928158506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1928158506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:23 smithi022 ceph-mon[31360]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:23 smithi022 ceph-mon[40250]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:23 smithi122 ceph-mon[32255]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:20:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:20:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:25 smithi122 ceph-mon[32255]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:25 smithi022 ceph-mon[31360]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:25 smithi022 ceph-mon[40250]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:27] "GET /metrics HTTP/1.1" 200 201887 "" "Prometheus/2.18.1" 2021-12-15T17:20:27.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:27 smithi122 ceph-mon[32255]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:27 smithi022 ceph-mon[31360]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:27 smithi022 ceph-mon[40250]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2940130649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2940130649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2940130649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:29 smithi122 ceph-mon[32255]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:29 smithi022 ceph-mon[31360]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:29 smithi022 ceph-mon[40250]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:20:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:31 smithi022 ceph-mon[31360]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:31 smithi022 ceph-mon[40250]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:31 smithi122 ceph-mon[32255]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:33.307 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:33.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:33 smithi022 ceph-mon[31360]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:33.509 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:33 smithi022 ceph-mon[40250]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:33 smithi122 ceph-mon[32255]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:33.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:33 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:33.504Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:20:33.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:33 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:33.504Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:20:33.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:33 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:33.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs; Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2021-12-15T17:20:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3964951768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3964951768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3964951768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:20:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:20:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:20:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:20:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[31360]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:20:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:20:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:20:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:20:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:35 smithi022 ceph-mon[40250]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:20:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:20:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:20:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:20:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:20:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:20:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:35 smithi122 ceph-mon[32255]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:37] "GET /metrics HTTP/1.1" 200 201887 "" "Prometheus/2.18.1" 2021-12-15T17:20:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:37 smithi122 ceph-mon[32255]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:37 smithi022 ceph-mon[31360]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:37 smithi022 ceph-mon[40250]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:38.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:39.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:39.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:39.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3929085966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3929085966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3929085966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:40 smithi122 ceph-mon[32255]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:20:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:40 smithi022 ceph-mon[31360]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:40 smithi022 ceph-mon[40250]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:41 smithi022 ceph-mon[31360]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:41 smithi022 ceph-mon[40250]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:41 smithi122 ceph-mon[32255]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:43.504 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:43 smithi022 ceph-mon[40250]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:43.504 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:43 smithi022 ceph-mon[31360]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:43 smithi122 ceph-mon[32255]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:43.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:43 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:43.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:20:43.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:43 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:43.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:20:43.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:43 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:43.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs; Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2021-12-15T17:20:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:44.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:44.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:44.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/123701756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:44.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/123701756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/123701756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:20:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:20:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:45 smithi122 ceph-mon[32255]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:45 smithi022 ceph-mon[31360]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:45 smithi022 ceph-mon[40250]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:47] "GET /metrics HTTP/1.1" 200 201887 "" "Prometheus/2.18.1" 2021-12-15T17:20:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:47 smithi122 ceph-mon[32255]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:47 smithi022 ceph-mon[31360]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:47 smithi022 ceph-mon[40250]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:49.425 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:49 smithi022 ceph-mon[31360]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:49 smithi022 ceph-mon[40250]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:49 smithi122 ceph-mon[32255]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3422741687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:20:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3422741687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3422741687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:51 smithi022 ceph-mon[31360]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:51 smithi022 ceph-mon[40250]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:51 smithi122 ceph-mon[32255]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:53.504 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:53 smithi022 ceph-mon[31360]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:53.504 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:53 smithi022 ceph-mon[40250]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:53 smithi122 ceph-mon[32255]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:53.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:53 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:53.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:20:53.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:53 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:53.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:20:53.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:20:53 smithi022 conmon[94037]: level=error ts=2021-12-15T17:20:53.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs" 2021-12-15T17:20:54.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:20:55.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:20:55.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:20:55.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:20:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:20:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:20:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3837179188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:55 smithi122 ceph-mon[32255]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3837179188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:55 smithi022 ceph-mon[31360]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3837179188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:20:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:55 smithi022 ceph-mon[40250]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:20:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:20:57] "GET /metrics HTTP/1.1" 200 201902 "" "Prometheus/2.18.1" 2021-12-15T17:20:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:57 smithi122 ceph-mon[32255]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:57 smithi022 ceph-mon[31360]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:57 smithi022 ceph-mon[40250]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:20:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:20:59 smithi022 ceph-mon[31360]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:20:59 smithi022 ceph-mon[40250]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:20:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:20:59 smithi122 ceph-mon[32255]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:00.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:21:00.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:00.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:00.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1628560293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1628560293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1628560293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:01 smithi122 ceph-mon[32255]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:01 smithi022 ceph-mon[31360]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:01 smithi022 ceph-mon[40250]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:03.504 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:03 smithi022 ceph-mon[31360]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:03.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:21:03.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:21:03.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:03 smithi022 ceph-mon[40250]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:03.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:21:03.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:21:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:03 smithi122 ceph-mon[32255]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:21:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:21:03.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:21:03 smithi022 conmon[94037]: level=error ts=2021-12-15T17:21:03.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:21:03.778 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:21:03 smithi022 conmon[94037]: level=error ts=2021-12-15T17:21:03.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs" context_err="context deadline exceeded" 2021-12-15T17:21:03.779 INFO:journalctl@ceph.alertmanager.a.smithi022.stdout:Dec 15 17:21:03 smithi022 conmon[94037]: level=error ts=2021-12-15T17:21:03.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post https://172.21.15.122:8443/api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; Post https://172.21.15.22:8443//api/prometheus_receiver: x509: cannot validate certificate for 172.21.15.22 because it doesn't contain any IP SANs" 2021-12-15T17:21:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:05.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:21:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:21:05.818 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:05 smithi022 ceph-mon[31360]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:05.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:05.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:05.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:05 smithi122 ceph-mon[32255]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:05 smithi022 ceph-mon[40250]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/215066922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/215066922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/215066922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:07] "GET /metrics HTTP/1.1" 200 201900 "" "Prometheus/2.18.1" 2021-12-15T17:21:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:07 smithi122 ceph-mon[32255]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:07 smithi022 ceph-mon[31360]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:07 smithi022 ceph-mon[40250]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:09 smithi022 ceph-mon[31360]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:09 smithi022 ceph-mon[40250]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:09 smithi122 ceph-mon[32255]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:21:10.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:11.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:11.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:11.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:11 smithi022 ceph-mon[31360]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:11 smithi022 ceph-mon[40250]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:11 smithi122 ceph-mon[32255]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3628219778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3628219778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3628219778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:13 smithi022 ceph-mon[31360]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:13 smithi022 ceph-mon[40250]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:13 smithi122 ceph-mon[32255]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:15.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:21:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:21:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:15 smithi122 ceph-mon[32255]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:15 smithi022 ceph-mon[31360]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:15 smithi022 ceph-mon[40250]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:16.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:16.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:16.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2818845943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2818845943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2818845943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:17] "GET /metrics HTTP/1.1" 200 201900 "" "Prometheus/2.18.1" 2021-12-15T17:21:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:17 smithi122 ceph-mon[32255]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:17 smithi022 ceph-mon[31360]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:17 smithi022 ceph-mon[40250]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:19 smithi022 ceph-mon[31360]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:19 smithi022 ceph-mon[40250]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:19 smithi122 ceph-mon[32255]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:21:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:21 smithi022 ceph-mon[31360]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:21 smithi022 ceph-mon[40250]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:21.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:21 smithi122 ceph-mon[32255]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:22.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:22.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:22.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2224267746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2224267746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2224267746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:23 smithi022 ceph-mon[31360]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:23 smithi022 ceph-mon[40250]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:23 smithi122 ceph-mon[32255]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:21:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:21:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:25 smithi122 ceph-mon[32255]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:25 smithi022 ceph-mon[31360]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:25 smithi022 ceph-mon[40250]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:27.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:27.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:27.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:27.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:27] "GET /metrics HTTP/1.1" 200 201903 "" "Prometheus/2.18.1" 2021-12-15T17:21:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:27 smithi122 ceph-mon[32255]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1805863842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:27 smithi022 ceph-mon[31360]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1805863842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:27 smithi022 ceph-mon[40250]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1805863842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:29 smithi022 ceph-mon[31360]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:29 smithi022 ceph-mon[40250]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:29 smithi122 ceph-mon[32255]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:21:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:31 smithi022 ceph-mon[31360]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:31 smithi022 ceph-mon[40250]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:31 smithi122 ceph-mon[32255]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:32.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:32.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:32.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/34257471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/34257471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/34257471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:33 smithi122 ceph-mon[32255]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:33 smithi022 ceph-mon[31360]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:33 smithi022 ceph-mon[40250]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:21:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:21:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:21:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:21:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:21:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:35 smithi122 ceph-mon[32255]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:21:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:21:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:21:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:21:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[31360]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:21:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:21:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:21:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:21:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[40250]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:21:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:37.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:37.771 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:37] "GET /metrics HTTP/1.1" 200 201902 "" "Prometheus/2.18.1" 2021-12-15T17:21:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:37 smithi022 ceph-mon[31360]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:37 smithi022 ceph-mon[40250]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:38.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:38.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:38.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:37 smithi122 ceph-mon[32255]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2457756827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2457756827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2457756827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:39 smithi122 ceph-mon[32255]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:39 smithi022 ceph-mon[31360]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:39 smithi022 ceph-mon[40250]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:21:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:41 smithi022 ceph-mon[31360]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:41 smithi022 ceph-mon[40250]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:41 smithi122 ceph-mon[32255]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:43.142 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:43.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:43.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:43.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:43.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:43 smithi022 ceph-mon[40250]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:43 smithi022 ceph-mon[31360]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:43 smithi122 ceph-mon[32255]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1356077669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1356077669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1356077669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:21:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:21:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:45 smithi122 ceph-mon[32255]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:45 smithi022 ceph-mon[31360]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:45 smithi022 ceph-mon[40250]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:47] "GET /metrics HTTP/1.1" 200 201902 "" "Prometheus/2.18.1" 2021-12-15T17:21:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:47 smithi122 ceph-mon[32255]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:47 smithi022 ceph-mon[31360]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:47 smithi022 ceph-mon[40250]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:48.523 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:48.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:48.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:48.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/410500596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/410500596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/410500596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:49 smithi022 ceph-mon[31360]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:49 smithi022 ceph-mon[40250]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:21:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:49 smithi122 ceph-mon[32255]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:51 smithi022 ceph-mon[31360]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:51 smithi022 ceph-mon[40250]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:51 smithi122 ceph-mon[32255]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:53 smithi022 ceph-mon[31360]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:53.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:53 smithi022 ceph-mon[40250]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:53 smithi122 ceph-mon[32255]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:53.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:54.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:54.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:54.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2093844344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2093844344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2093844344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:21:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:21:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:21:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:55 smithi122 ceph-mon[32255]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:55 smithi022 ceph-mon[31360]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:55 smithi022 ceph-mon[40250]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:21:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:21:57] "GET /metrics HTTP/1.1" 200 201905 "" "Prometheus/2.18.1" 2021-12-15T17:21:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:57 smithi122 ceph-mon[32255]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:57 smithi022 ceph-mon[31360]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:57 smithi022 ceph-mon[40250]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:21:59.269 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:21:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:21:59 smithi022 ceph-mon[31360]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:21:59 smithi022 ceph-mon[40250]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:21:59.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:21:59.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:21:59.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:21:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:21:59 smithi122 ceph-mon[32255]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:22:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1590514763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1590514763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1590514763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:01 smithi022 ceph-mon[31360]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:01 smithi022 ceph-mon[40250]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:01 smithi122 ceph-mon[32255]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:03 smithi022 ceph-mon[31360]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:22:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:22:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:03 smithi022 ceph-mon[40250]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:22:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:22:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:03 smithi122 ceph-mon[32255]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:22:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:22:04.643 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:05.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:05.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:05.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:05.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:22:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:22:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3152013978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:05 smithi122 ceph-mon[32255]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3152013978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:05 smithi022 ceph-mon[40250]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3152013978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:05 smithi022 ceph-mon[31360]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:07] "GET /metrics HTTP/1.1" 200 201916 "" "Prometheus/2.18.1" 2021-12-15T17:22:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:07 smithi122 ceph-mon[32255]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:07 smithi022 ceph-mon[31360]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:07 smithi022 ceph-mon[40250]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:09 smithi022 ceph-mon[31360]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:09 smithi022 ceph-mon[40250]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:09 smithi122 ceph-mon[32255]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:10.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:10.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:10.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:10.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:22:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2603128880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2603128880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2603128880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:11 smithi122 ceph-mon[32255]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:11 smithi022 ceph-mon[31360]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:11 smithi022 ceph-mon[40250]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:13 smithi022 ceph-mon[31360]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:13 smithi022 ceph-mon[40250]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:13 smithi122 ceph-mon[32255]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:15.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:15.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:22:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:22:15.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:15.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:15.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:15 smithi022 ceph-mon[31360]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3622721005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:15 smithi022 ceph-mon[40250]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3622721005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:15 smithi122 ceph-mon[32255]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3622721005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:17] "GET /metrics HTTP/1.1" 200 201916 "" "Prometheus/2.18.1" 2021-12-15T17:22:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:17 smithi122 ceph-mon[32255]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:17 smithi022 ceph-mon[31360]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:17 smithi022 ceph-mon[40250]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:19 smithi022 ceph-mon[31360]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:19 smithi022 ceph-mon[40250]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:19 smithi122 ceph-mon[32255]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:22:20.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:21.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:21.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:21.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/452746145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/452746145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/452746145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:22 smithi122 ceph-mon[32255]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:22 smithi022 ceph-mon[31360]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:22 smithi022 ceph-mon[40250]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:23 smithi022 ceph-mon[40250]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:23 smithi022 ceph-mon[31360]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:23 smithi122 ceph-mon[32255]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:22:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:22:26.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:26.128 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:25 smithi022 ceph-mon[31360]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:26.129 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:25 smithi022 ceph-mon[40250]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:25 smithi122 ceph-mon[32255]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:26.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:26.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:26.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1086311790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1086311790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1086311790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:27] "GET /metrics HTTP/1.1" 200 201932 "" "Prometheus/2.18.1" 2021-12-15T17:22:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:27 smithi122 ceph-mon[32255]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:27 smithi022 ceph-mon[31360]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:27 smithi022 ceph-mon[40250]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:29 smithi022 ceph-mon[31360]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:29 smithi022 ceph-mon[40250]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:29 smithi122 ceph-mon[32255]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:22:31.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:31 smithi022 ceph-mon[31360]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:31 smithi022 ceph-mon[40250]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:31 smithi122 ceph-mon[32255]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:31.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:31.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:31.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3299473410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3299473410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3299473410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:33 smithi022 ceph-mon[31360]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T17:22:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:33 smithi022 ceph-mon[40250]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T17:22:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:33 smithi122 ceph-mon[32255]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T17:22:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:22:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:22:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[31360]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:22:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:22:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:22:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:22:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[40250]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:22:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:22:36.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:22:36.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:22:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:35 smithi122 ceph-mon[32255]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:22:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:22:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:22:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:22:36.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:37.240 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:37.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:37] "GET /metrics HTTP/1.1" 200 201925 "" "Prometheus/2.18.1" 2021-12-15T17:22:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:37 smithi122 ceph-mon[32255]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1653072764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:37 smithi022 ceph-mon[31360]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1653072764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:37 smithi022 ceph-mon[40250]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1653072764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:39 smithi022 ceph-mon[31360]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:39 smithi022 ceph-mon[40250]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:39 smithi122 ceph-mon[32255]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:22:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:41 smithi022 ceph-mon[31360]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:41 smithi022 ceph-mon[40250]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:41 smithi122 ceph-mon[32255]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:42.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:42.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:42.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:42.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2923493613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2923493613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2923493613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:43 smithi122 ceph-mon[32255]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:43 smithi022 ceph-mon[31360]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:43 smithi022 ceph-mon[40250]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:22:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:22:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:45 smithi122 ceph-mon[32255]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:45 smithi022 ceph-mon[31360]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:45 smithi022 ceph-mon[40250]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:47.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:47] "GET /metrics HTTP/1.1" 200 201925 "" "Prometheus/2.18.1" 2021-12-15T17:22:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:47.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:47.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:47 smithi122 ceph-mon[32255]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:47 smithi022 ceph-mon[31360]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:47 smithi022 ceph-mon[40250]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4291574009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4291574009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4291574009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:50.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:49 smithi122 ceph-mon[32255]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:49 smithi022 ceph-mon[31360]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:49 smithi022 ceph-mon[40250]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:22:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:51 smithi022 ceph-mon[31360]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:51 smithi022 ceph-mon[40250]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:51 smithi122 ceph-mon[32255]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:52.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:53.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:53 smithi022 ceph-mon[40250]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:53.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:53.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:53.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:53 smithi022 ceph-mon[31360]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:53 smithi122 ceph-mon[32255]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3062517379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3062517379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3062517379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:22:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:22:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:55 smithi122 ceph-mon[32255]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:55 smithi022 ceph-mon[31360]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:55 smithi022 ceph-mon[40250]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:22:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:22:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:22:57] "GET /metrics HTTP/1.1" 200 201931 "" "Prometheus/2.18.1" 2021-12-15T17:22:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:57 smithi122 ceph-mon[32255]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:57 smithi022 ceph-mon[31360]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:57 smithi022 ceph-mon[40250]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:22:58.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:22:58.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:22:58.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:22:58.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:22:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1088563337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1088563337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:22:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1088563337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:22:59 smithi122 ceph-mon[32255]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:22:59 smithi022 ceph-mon[31360]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:22:59 smithi022 ceph-mon[40250]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:23:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:01 smithi022 ceph-mon[31360]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:01 smithi022 ceph-mon[40250]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:01 smithi122 ceph-mon[32255]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:03 smithi022 ceph-mon[31360]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:23:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:23:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:03 smithi022 ceph-mon[40250]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:23:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:23:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:03 smithi122 ceph-mon[32255]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:23:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:23:03.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:04.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:04.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:04.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/763263634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/763263634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/763263634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:05.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:23:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:23:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:05 smithi122 ceph-mon[32255]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:05 smithi022 ceph-mon[31360]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:05 smithi022 ceph-mon[40250]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:07] "GET /metrics HTTP/1.1" 200 201934 "" "Prometheus/2.18.1" 2021-12-15T17:23:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:07 smithi122 ceph-mon[32255]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:07 smithi022 ceph-mon[31360]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:07 smithi022 ceph-mon[40250]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:09.107 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:09.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:09.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:09.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:09.480 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:09 smithi022 ceph-mon[31360]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:09.480 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:09 smithi022 ceph-mon[40250]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:09 smithi122 ceph-mon[32255]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2175661219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:23:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2175661219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2175661219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:11 smithi122 ceph-mon[32255]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:11 smithi022 ceph-mon[31360]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:11 smithi022 ceph-mon[40250]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:13 smithi022 ceph-mon[31360]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:13 smithi022 ceph-mon[40250]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:13 smithi122 ceph-mon[32255]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:14.481 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:14.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:14.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:14.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3956966679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3956966679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3956966679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:15.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:23:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:23:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:15 smithi122 ceph-mon[32255]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:15 smithi022 ceph-mon[31360]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:15 smithi022 ceph-mon[40250]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:17] "GET /metrics HTTP/1.1" 200 201934 "" "Prometheus/2.18.1" 2021-12-15T17:23:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:17 smithi122 ceph-mon[32255]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:17 smithi022 ceph-mon[31360]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:17 smithi022 ceph-mon[40250]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:19 smithi022 ceph-mon[31360]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:19 smithi022 ceph-mon[40250]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:19 smithi122 ceph-mon[32255]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:19.851 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:20.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:20.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:20.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:23:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1435450919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1435450919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1435450919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:21 smithi122 ceph-mon[32255]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:21 smithi022 ceph-mon[31360]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:21 smithi022 ceph-mon[40250]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:23 smithi022 ceph-mon[31360]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:23 smithi022 ceph-mon[40250]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:23 smithi122 ceph-mon[32255]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:25.223 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:23:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:23:25.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:25 smithi022 ceph-mon[31360]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:25.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3507380701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:25 smithi122 ceph-mon[32255]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3507380701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:25 smithi022 ceph-mon[40250]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3507380701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:27] "GET /metrics HTTP/1.1" 200 201934 "" "Prometheus/2.18.1" 2021-12-15T17:23:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:27 smithi122 ceph-mon[32255]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:27 smithi022 ceph-mon[31360]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:27 smithi022 ceph-mon[40250]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:29 smithi122 ceph-mon[32255]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:29.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:29 smithi022 ceph-mon[31360]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:29.796 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:29 smithi022 ceph-mon[40250]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:23:30.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:30.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:30.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:30.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/417796010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/417796010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/417796010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:32 smithi022 ceph-mon[31360]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:32 smithi022 ceph-mon[40250]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:32 smithi122 ceph-mon[32255]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:33 smithi122 ceph-mon[32255]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:33 smithi022 ceph-mon[31360]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:33 smithi022 ceph-mon[40250]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:23:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:23:35.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:36.163 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:35 smithi122 ceph-mon[32255]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:36.164 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:23:36.164 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:23:36.165 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:23:36.185 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[31360]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:36.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:23:36.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:23:36.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:23:36.187 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[40250]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:36.187 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:23:36.187 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:23:36.187 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:23:36.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:36.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:36.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:37 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:37 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3133468097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:37 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:37] "GET /metrics HTTP/1.1" 200 201936 "" "Prometheus/2.18.1" 2021-12-15T17:23:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3133468097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:37.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:37.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:37.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3133468097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:37.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:37 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:38 smithi122 ceph-mon[32255]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:38 smithi022 ceph-mon[31360]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:38 smithi022 ceph-mon[40250]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:39 smithi122 ceph-mon[32255]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:39 smithi022 ceph-mon[31360]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:39 smithi022 ceph-mon[40250]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:23:41.347 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:41 smithi122 ceph-mon[32255]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== 2021-12-15T17:23:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== ']' 2021-12-15T17:23:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:41.721 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:41 smithi022 ceph-mon[31360]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:41.721 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:41 smithi022 ceph-mon[40250]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/670622686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/670622686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:42.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/670622686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:43 smithi122 ceph-mon[32255]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:43 smithi022 ceph-mon[31360]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:43 smithi022 ceph-mon[40250]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:45.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:23:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:23:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:45 smithi122 ceph-mon[32255]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:45 smithi022 ceph-mon[31360]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:45 smithi022 ceph-mon[40250]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.0 2021-12-15T17:23:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:47.092 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.1 2021-12-15T17:23:47.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCZI7phwhAKMxAA5a1ZBhoLwHywN3KhlalhOQ== 2021-12-15T17:23:47.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBOIrph6waBORAAKCQKwnZKL0YVqmWJjM2jXQ== == AQCZI7phwhAKMxAA5a1ZBhoLwHywN3KhlalhOQ== ']' 2021-12-15T17:23:47.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:23:47.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.1' 2021-12-15T17:23:47.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.1 2021-12-15T17:23:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:47.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== 2021-12-15T17:23:47.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== 2021-12-15T17:23:47.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.1 2021-12-15T17:23:47.740 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.1 on host 'smithi022' 2021-12-15T17:23:47.741 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:47] "GET /metrics HTTP/1.1" 200 201936 "" "Prometheus/2.18.1" 2021-12-15T17:23:47.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== == AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== ']' 2021-12-15T17:23:47.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2038507708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4120341309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:23:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:23:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:23:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2021-12-15T17:23:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2021-12-15T17:23:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2038507708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4120341309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:23:48.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:23:48.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:23:48.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:48.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2021-12-15T17:23:48.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:47 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2021-12-15T17:23:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2038507708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2021-12-15T17:23:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4120341309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:23:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:23:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:23:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2021-12-15T17:23:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:47 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2021-12-15T17:23:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:49 smithi122 ceph-mon[32255]: from='client.25120 v1:172.21.15.22:0/2095297089' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:23:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:49 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.1 2021-12-15T17:23:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:49 smithi122 ceph-mon[32255]: Rotating authentication key for osd.1 2021-12-15T17:23:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:49 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[40250]: from='client.25120 v1:172.21.15.22:0/2095297089' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:23:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.1 2021-12-15T17:23:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[40250]: Rotating authentication key for osd.1 2021-12-15T17:23:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[31360]: from='client.25120 v1:172.21.15.22:0/2095297089' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:23:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.1 2021-12-15T17:23:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[31360]: Rotating authentication key for osd.1 2021-12-15T17:23:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:49 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:23:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:50 smithi122 ceph-mon[32255]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:23:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:50 smithi022 ceph-mon[31360]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:50 smithi022 ceph-mon[40250]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:51 smithi122 ceph-mon[32255]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:51 smithi022 ceph-mon[31360]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:51 smithi022 ceph-mon[40250]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:52.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.1 2021-12-15T17:23:53.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== 2021-12-15T17:23:53.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== == AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== ']' 2021-12-15T17:23:53.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3046940483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3046940483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3046940483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:54 smithi122 ceph-mon[32255]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:54 smithi022 ceph-mon[31360]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:54 smithi022 ceph-mon[40250]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:55.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:23:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:23:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:55 smithi122 ceph-mon[32255]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:55 smithi022 ceph-mon[31360]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:55 smithi022 ceph-mon[40250]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:23:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:23:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:23:57] "GET /metrics HTTP/1.1" 200 201932 "" "Prometheus/2.18.1" 2021-12-15T17:23:58.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.1 2021-12-15T17:23:58.126 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:57 smithi022 ceph-mon[40250]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:58.127 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:57 smithi022 ceph-mon[31360]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:57 smithi122 ceph-mon[32255]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:23:58.495 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.2 2021-12-15T17:23:58.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCjJLphhSbNLBAAz0073m1yLhRWmEHWj/lzlg== 2021-12-15T17:23:58.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQBgIrphh064NhAAHjQECqjPvhaTVDxqvrrZ+A== == AQCjJLphhSbNLBAAz0073m1yLhRWmEHWj/lzlg== ']' 2021-12-15T17:23:58.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:23:58.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.2' 2021-12-15T17:23:58.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:23:58.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:23:58.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:23:58.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.2 2021-12-15T17:23:59.141 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.2 on host 'smithi022' 2021-12-15T17:23:59.142 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1870027103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:59.142 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2734347251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:23:59.143 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1870027103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:59.143 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2734347251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:23:59.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:23:59.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:23:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1870027103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2021-12-15T17:23:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2734347251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='client.15249 v1:172.21.15.22:0/2875566780' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:24:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.2 2021-12-15T17:24:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:24:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:24:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:24:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:24:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: Rotating authentication key for osd.2 2021-12-15T17:24:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2021-12-15T17:24:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2021-12-15T17:24:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:23:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:24:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='client.15249 v1:172.21.15.22:0/2875566780' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:24:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.2 2021-12-15T17:24:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:24:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:24:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:24:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:24:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: Rotating authentication key for osd.2 2021-12-15T17:24:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2021-12-15T17:24:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2021-12-15T17:24:00.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:00.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:24:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='client.15249 v1:172.21.15.22:0/2875566780' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:24:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.2 2021-12-15T17:24:00.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:24:00.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:24:00.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:24:00.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:24:00.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: Rotating authentication key for osd.2 2021-12-15T17:24:00.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2021-12-15T17:24:00.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2021-12-15T17:24:00.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:00.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:23:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:24:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:24:01.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:01 smithi122 ceph-mon[32255]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:01 smithi022 ceph-mon[31360]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:01 smithi022 ceph-mon[40250]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:03 smithi122 ceph-mon[32255]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:24:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:24:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:03 smithi022 ceph-mon[31360]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:24:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:24:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:03 smithi022 ceph-mon[40250]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:24:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:24:04.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:04.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:04.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:04.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1037998947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1037998947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1037998947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:05.856 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:24:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:24:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:05 smithi122 ceph-mon[32255]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:05 smithi022 ceph-mon[31360]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:06.372 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:05 smithi022 ceph-mon[40250]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:07] "GET /metrics HTTP/1.1" 200 201940 "" "Prometheus/2.18.1" 2021-12-15T17:24:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:07 smithi122 ceph-mon[32255]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:07 smithi022 ceph-mon[31360]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:07 smithi022 ceph-mon[40250]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:09.526 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:09 smithi122 ceph-mon[32255]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:09 smithi022 ceph-mon[31360]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:09 smithi022 ceph-mon[40250]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:09.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:09.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:09.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:24:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3774652884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3774652884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3774652884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:11 smithi122 ceph-mon[32255]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:11 smithi022 ceph-mon[31360]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:11 smithi022 ceph-mon[40250]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:13 smithi122 ceph-mon[32255]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:13 smithi022 ceph-mon[31360]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:13 smithi022 ceph-mon[40250]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:14.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:15.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:15.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:15.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:15.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:24:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:24:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:15 smithi122 ceph-mon[32255]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/728090577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:15 smithi022 ceph-mon[31360]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/728090577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:15 smithi022 ceph-mon[40250]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/728090577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:17] "GET /metrics HTTP/1.1" 200 201940 "" "Prometheus/2.18.1" 2021-12-15T17:24:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:17 smithi122 ceph-mon[32255]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:17 smithi022 ceph-mon[31360]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:17 smithi022 ceph-mon[40250]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:19 smithi122 ceph-mon[32255]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:19 smithi022 ceph-mon[31360]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:19 smithi022 ceph-mon[40250]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:20.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:24:20.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:20.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:20.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1130209837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1130209837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1130209837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:21 smithi122 ceph-mon[32255]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:21 smithi022 ceph-mon[31360]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:21 smithi022 ceph-mon[40250]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:23 smithi122 ceph-mon[32255]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:23 smithi022 ceph-mon[31360]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:23 smithi022 ceph-mon[40250]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:25.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:25.861 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:24:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:24:26.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:26.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:26.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:26.009 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:25 smithi022 ceph-mon[31360]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:26.010 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:25 smithi022 ceph-mon[40250]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:25 smithi122 ceph-mon[32255]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4105017286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4105017286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4105017286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:27] "GET /metrics HTTP/1.1" 200 201940 "" "Prometheus/2.18.1" 2021-12-15T17:24:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:27 smithi122 ceph-mon[32255]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:27 smithi022 ceph-mon[31360]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:27 smithi022 ceph-mon[40250]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:29 smithi122 ceph-mon[32255]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:29 smithi022 ceph-mon[31360]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:29 smithi022 ceph-mon[40250]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:24:31.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:31.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:31.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:31.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:31 smithi122 ceph-mon[32255]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:31 smithi022 ceph-mon[31360]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:31 smithi022 ceph-mon[40250]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3136099352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3136099352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3136099352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:33.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:33 smithi122 ceph-mon[32255]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:33 smithi022 ceph-mon[31360]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:33 smithi022 ceph-mon[40250]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:35.864 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:24:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:24:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:35 smithi122 ceph-mon[32255]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:35 smithi022 ceph-mon[31360]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:35 smithi022 ceph-mon[40250]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:36.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:36.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:36.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1429871307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1429871307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1429871307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:37] "GET /metrics HTTP/1.1" 200 201938 "" "Prometheus/2.18.1" 2021-12-15T17:24:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:37 smithi122 ceph-mon[32255]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:37 smithi022 ceph-mon[31360]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:37 smithi022 ceph-mon[40250]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:39 smithi122 ceph-mon[32255]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:39 smithi022 ceph-mon[31360]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:39 smithi022 ceph-mon[40250]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:24:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:41 smithi122 ceph-mon[32255]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:41.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:41.750 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:41 smithi022 ceph-mon[31360]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:41.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:41 smithi022 ceph-mon[40250]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:42.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:42.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:42.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1309575930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1309575930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1309575930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:43.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:43 smithi122 ceph-mon[32255]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:43 smithi022 ceph-mon[31360]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:43 smithi022 ceph-mon[40250]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:45.866 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:24:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:24:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:45 smithi122 ceph-mon[32255]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:45 smithi022 ceph-mon[31360]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:45 smithi022 ceph-mon[40250]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:47.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:47.119 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:47.120 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:47.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:47.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:47.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:47] "GET /metrics HTTP/1.1" 200 201938 "" "Prometheus/2.18.1" 2021-12-15T17:24:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:47 smithi122 ceph-mon[32255]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/299128552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:47 smithi022 ceph-mon[31360]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/299128552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:47 smithi022 ceph-mon[40250]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/299128552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:49 smithi122 ceph-mon[32255]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:49 smithi022 ceph-mon[31360]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:49 smithi022 ceph-mon[40250]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:24:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:51 smithi122 ceph-mon[32255]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:51 smithi022 ceph-mon[31360]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:51 smithi022 ceph-mon[40250]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:52.493 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2656678504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:53.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2656678504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:53.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2656678504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:53 smithi122 ceph-mon[32255]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:53 smithi022 ceph-mon[31360]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:53 smithi022 ceph-mon[40250]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:55.868 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:24:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:24:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:55 smithi122 ceph-mon[32255]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:55 smithi022 ceph-mon[31360]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:55 smithi022 ceph-mon[40250]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:24:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:24:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:24:57] "GET /metrics HTTP/1.1" 200 201945 "" "Prometheus/2.18.1" 2021-12-15T17:24:57.866 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:24:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:57 smithi122 ceph-mon[32255]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:58.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:24:58.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:24:58.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:24:58.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:57 smithi022 ceph-mon[31360]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:58.239 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:57 smithi022 ceph-mon[40250]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:24:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3227607618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3227607618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:24:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3227607618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:59 smithi122 ceph-mon[32255]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:25:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:25:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:24:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:25:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[31360]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:25:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:25:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:25:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[40250]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:25:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:25:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:24:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:25:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:25:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:25:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:25:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:25:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:01 smithi122 ceph-mon[32255]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:01 smithi022 ceph-mon[31360]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:01 smithi022 ceph-mon[40250]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:03.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:25:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:25:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:03 smithi022 ceph-mon[40250]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:25:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:25:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:03 smithi022 ceph-mon[31360]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:03.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:03.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:03.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:25:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:25:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:03 smithi122 ceph-mon[32255]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1458829843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1458829843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1458829843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:05.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:25:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:25:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:05 smithi122 ceph-mon[32255]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:05 smithi022 ceph-mon[31360]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:05 smithi022 ceph-mon[40250]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:07] "GET /metrics HTTP/1.1" 200 201937 "" "Prometheus/2.18.1" 2021-12-15T17:25:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:07 smithi122 ceph-mon[32255]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:08.356 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:07 smithi022 ceph-mon[31360]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:08.356 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:07 smithi022 ceph-mon[40250]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:08.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:08.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:08.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:08.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3470223254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3470223254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3470223254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:10 smithi022 ceph-mon[31360]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:10 smithi022 ceph-mon[40250]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:25:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:10 smithi122 ceph-mon[32255]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:11 smithi122 ceph-mon[32255]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:11 smithi022 ceph-mon[31360]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:11 smithi022 ceph-mon[40250]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:13 smithi122 ceph-mon[32255]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:13 smithi022 ceph-mon[31360]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:13 smithi022 ceph-mon[40250]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:13.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2338487044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2338487044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2338487044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:15.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:25:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:25:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:15 smithi122 ceph-mon[32255]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:15 smithi022 ceph-mon[31360]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:15 smithi022 ceph-mon[40250]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:17] "GET /metrics HTTP/1.1" 200 201937 "" "Prometheus/2.18.1" 2021-12-15T17:25:18.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:17 smithi122 ceph-mon[32255]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:17 smithi022 ceph-mon[31360]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:17 smithi022 ceph-mon[40250]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:19.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:19 smithi122 ceph-mon[32255]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:19.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:19.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:19.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:19.726 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:19 smithi022 ceph-mon[40250]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:19.726 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:19 smithi022 ceph-mon[31360]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2339577374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:25:20.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2339577374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2339577374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:22 smithi122 ceph-mon[32255]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:22 smithi022 ceph-mon[40250]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:22 smithi022 ceph-mon[31360]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:24 smithi122 ceph-mon[32255]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:24.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:24.727 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:24 smithi022 ceph-mon[31360]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:24.727 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:24 smithi022 ceph-mon[40250]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:25.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:25.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:25.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:25.874 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:25:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:25:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/405382396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/405382396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/405382396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:26 smithi122 ceph-mon[32255]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:26 smithi022 ceph-mon[31360]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:26 smithi022 ceph-mon[40250]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:27] "GET /metrics HTTP/1.1" 200 201948 "" "Prometheus/2.18.1" 2021-12-15T17:25:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:28 smithi122 ceph-mon[32255]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:28 smithi022 ceph-mon[31360]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:28 smithi022 ceph-mon[40250]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:30.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:25:30.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:30.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:30.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:30 smithi122 ceph-mon[32255]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2348112322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:30 smithi022 ceph-mon[40250]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2348112322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:30 smithi022 ceph-mon[31360]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2348112322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:32 smithi122 ceph-mon[32255]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:32 smithi022 ceph-mon[31360]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:32 smithi022 ceph-mon[40250]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:34 smithi122 ceph-mon[32255]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:34 smithi022 ceph-mon[31360]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:34 smithi022 ceph-mon[40250]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:35.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:35.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:35.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:35.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:35.890 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:25:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:25:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3912004282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3912004282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3912004282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:36 smithi122 ceph-mon[32255]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:36 smithi022 ceph-mon[31360]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:36 smithi022 ceph-mon[40250]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:37] "GET /metrics HTTP/1.1" 200 201950 "" "Prometheus/2.18.1" 2021-12-15T17:25:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:38 smithi122 ceph-mon[32255]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:38 smithi022 ceph-mon[31360]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:38 smithi022 ceph-mon[40250]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:25:40.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:40 smithi122 ceph-mon[32255]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:41.206 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:40 smithi022 ceph-mon[31360]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:41.207 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:40 smithi022 ceph-mon[40250]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/285421115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/285421115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/285421115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:42 smithi122 ceph-mon[32255]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:42 smithi022 ceph-mon[31360]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:42 smithi022 ceph-mon[40250]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:44 smithi122 ceph-mon[32255]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:44 smithi022 ceph-mon[31360]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:44 smithi022 ceph-mon[40250]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:45.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:25:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:25:46.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:46.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:46.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:46.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:46 smithi122 ceph-mon[32255]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1875187067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:46 smithi022 ceph-mon[31360]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1875187067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:46 smithi022 ceph-mon[40250]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1875187067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:47] "GET /metrics HTTP/1.1" 200 201950 "" "Prometheus/2.18.1" 2021-12-15T17:25:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:48 smithi122 ceph-mon[32255]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:48 smithi022 ceph-mon[31360]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:48 smithi022 ceph-mon[40250]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:25:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:50 smithi122 ceph-mon[32255]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:50 smithi022 ceph-mon[31360]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:50 smithi022 ceph-mon[40250]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:51.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:51.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:51.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:51.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:52 smithi122 ceph-mon[32255]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3876555248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:52 smithi022 ceph-mon[31360]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3876555248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:52 smithi022 ceph-mon[40250]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3876555248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:54 smithi122 ceph-mon[32255]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:54 smithi022 ceph-mon[31360]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:54 smithi022 ceph-mon[40250]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:25:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:25:56.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:25:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:56 smithi122 ceph-mon[32255]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:56 smithi022 ceph-mon[31360]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:56 smithi022 ceph-mon[40250]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:25:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:25:57.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:25:57.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:25:57.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:25:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:25:57] "GET /metrics HTTP/1.1" 200 201948 "" "Prometheus/2.18.1" 2021-12-15T17:25:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4127975446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4127975446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4127975446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:25:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:25:58 smithi122 ceph-mon[32255]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:25:58 smithi022 ceph-mon[31360]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:25:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:25:58 smithi022 ceph-mon[40250]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:00.360 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:26:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:00 smithi122 ceph-mon[32255]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:26:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:26:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:26:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:26:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[31360]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:26:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:26:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:26:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:26:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[40250]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:26:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:26:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:26:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:26:02.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:02.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:02.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:02.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:02 smithi022 ceph-mon[31360]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/157311080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:02 smithi022 ceph-mon[40250]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/157311080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:02 smithi122 ceph-mon[32255]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/157311080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:26:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:26:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:26:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:26:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:26:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:26:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:04 smithi122 ceph-mon[32255]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:04 smithi022 ceph-mon[31360]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:04 smithi022 ceph-mon[40250]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:05.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:26:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:26:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:06 smithi122 ceph-mon[32255]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:06 smithi022 ceph-mon[31360]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:06 smithi022 ceph-mon[40250]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:07.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:07.692 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:07] "GET /metrics HTTP/1.1" 200 201943 "" "Prometheus/2.18.1" 2021-12-15T17:26:08.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:08.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:08.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:08 smithi122 ceph-mon[32255]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1979367724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:08 smithi022 ceph-mon[31360]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1979367724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:08 smithi022 ceph-mon[40250]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1979367724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:26:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:10 smithi122 ceph-mon[32255]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:10 smithi022 ceph-mon[31360]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:10 smithi022 ceph-mon[40250]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:13.062 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:12 smithi122 ceph-mon[32255]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:12 smithi022 ceph-mon[31360]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:12 smithi022 ceph-mon[40250]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:13.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:13.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:13.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3051595965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3051595965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3051595965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:14 smithi122 ceph-mon[32255]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:14 smithi022 ceph-mon[31360]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:14 smithi022 ceph-mon[40250]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:26:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:26:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:16 smithi122 ceph-mon[32255]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:16 smithi022 ceph-mon[31360]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:16 smithi022 ceph-mon[40250]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:17] "GET /metrics HTTP/1.1" 200 201943 "" "Prometheus/2.18.1" 2021-12-15T17:26:18.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:18 smithi122 ceph-mon[32255]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2747068323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:18 smithi022 ceph-mon[31360]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2747068323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:18 smithi022 ceph-mon[40250]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2747068323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:26:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:20 smithi122 ceph-mon[32255]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:20 smithi022 ceph-mon[31360]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:20 smithi022 ceph-mon[40250]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:22 smithi122 ceph-mon[32255]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:22 smithi022 ceph-mon[31360]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:22 smithi022 ceph-mon[40250]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:23.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:24.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:24.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:24.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:24 smithi122 ceph-mon[32255]: pgmap v255: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2193089012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:24 smithi022 ceph-mon[31360]: pgmap v255: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2193089012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:24 smithi022 ceph-mon[40250]: pgmap v255: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2193089012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:25.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:26:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:26:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:26 smithi122 ceph-mon[32255]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:26 smithi022 ceph-mon[31360]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:26 smithi022 ceph-mon[40250]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:27] "GET /metrics HTTP/1.1" 200 201942 "" "Prometheus/2.18.1" 2021-12-15T17:26:29.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:29.178 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:28 smithi022 ceph-mon[40250]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:29.178 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:28 smithi022 ceph-mon[31360]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:28 smithi122 ceph-mon[32255]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:29.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:29.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:29.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3418515369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:30.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:26:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3418515369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3418515369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:30 smithi122 ceph-mon[32255]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:30 smithi022 ceph-mon[31360]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:30 smithi022 ceph-mon[40250]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:32 smithi122 ceph-mon[32255]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:32 smithi022 ceph-mon[31360]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:32 smithi022 ceph-mon[40250]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:34.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:34.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:34.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:34 smithi122 ceph-mon[32255]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/973547588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:34 smithi022 ceph-mon[31360]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/973547588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:34 smithi022 ceph-mon[40250]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:35.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/973547588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:35.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:26:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:26:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:36 smithi122 ceph-mon[32255]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:36 smithi022 ceph-mon[31360]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:36 smithi022 ceph-mon[40250]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:37] "GET /metrics HTTP/1.1" 200 201950 "" "Prometheus/2.18.1" 2021-12-15T17:26:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:38 smithi022 ceph-mon[31360]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:38 smithi022 ceph-mon[40250]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:38 smithi122 ceph-mon[32255]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:39.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:40.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:40.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:40.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:26:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:40 smithi022 ceph-mon[31360]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4059805592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:40 smithi022 ceph-mon[40250]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4059805592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:40 smithi122 ceph-mon[32255]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4059805592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:42 smithi022 ceph-mon[31360]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:42 smithi022 ceph-mon[40250]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:42 smithi122 ceph-mon[32255]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:44 smithi022 ceph-mon[31360]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:44 smithi022 ceph-mon[40250]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:45.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:44 smithi122 ceph-mon[32255]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:45.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:45.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:45.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:45.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:26:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:26:45.949 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1707909403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1707909403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1707909403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:46 smithi022 ceph-mon[31360]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:46 smithi022 ceph-mon[40250]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:46 smithi122 ceph-mon[32255]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:47] "GET /metrics HTTP/1.1" 200 201950 "" "Prometheus/2.18.1" 2021-12-15T17:26:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:48 smithi022 ceph-mon[31360]: pgmap v267: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:48 smithi022 ceph-mon[40250]: pgmap v267: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:48 smithi122 ceph-mon[32255]: pgmap v267: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:26:50.658 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:51.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:51.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:51.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:50 smithi022 ceph-mon[31360]: pgmap v268: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:50 smithi022 ceph-mon[40250]: pgmap v268: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:50 smithi122 ceph-mon[32255]: pgmap v268: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/762154857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/762154857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/762154857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:52 smithi022 ceph-mon[31360]: pgmap v269: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:52 smithi022 ceph-mon[40250]: pgmap v269: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:52 smithi122 ceph-mon[32255]: pgmap v269: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:54 smithi022 ceph-mon[40250]: pgmap v270: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:54 smithi022 ceph-mon[31360]: pgmap v270: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:54 smithi122 ceph-mon[32255]: pgmap v270: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:26:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:26:56.031 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:26:56.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:26:56.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:26:56.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:26:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:56 smithi022 ceph-mon[31360]: pgmap v271: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1833053244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:56 smithi022 ceph-mon[40250]: pgmap v271: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1833053244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:56 smithi122 ceph-mon[32255]: pgmap v271: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:26:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:26:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1833053244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:26:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:26:57] "GET /metrics HTTP/1.1" 200 201947 "" "Prometheus/2.18.1" 2021-12-15T17:26:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:26:58 smithi022 ceph-mon[31360]: pgmap v272: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:26:58 smithi022 ceph-mon[40250]: pgmap v272: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:26:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:26:58 smithi122 ceph-mon[32255]: pgmap v272: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:27:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:00 smithi122 ceph-mon[32255]: pgmap v273: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:27:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:27:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:27:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:27:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[40250]: pgmap v273: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:27:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:27:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:27:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:27:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[31360]: pgmap v273: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:27:01.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:27:01.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:27:01.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:27:01.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3802164763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3802164763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:02.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3802164763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:02 smithi022 ceph-mon[31360]: pgmap v274: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:02 smithi022 ceph-mon[40250]: pgmap v274: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:02 smithi122 ceph-mon[32255]: pgmap v274: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:27:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:27:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:27:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:27:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:27:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:27:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:04 smithi022 ceph-mon[31360]: pgmap v275: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:04 smithi022 ceph-mon[40250]: pgmap v275: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:04 smithi122 ceph-mon[32255]: pgmap v275: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:05.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:27:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:27:06.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:06 smithi022 ceph-mon[31360]: pgmap v276: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:06 smithi022 ceph-mon[40250]: pgmap v276: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:07.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:07.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:07.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:06 smithi122 ceph-mon[32255]: pgmap v276: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:07] "GET /metrics HTTP/1.1" 200 201948 "" "Prometheus/2.18.1" 2021-12-15T17:27:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3335638504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3335638504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3335638504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:08 smithi022 ceph-mon[31360]: pgmap v277: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:08 smithi022 ceph-mon[40250]: pgmap v277: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:08 smithi122 ceph-mon[32255]: pgmap v277: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:27:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:10 smithi022 ceph-mon[31360]: pgmap v278: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:10 smithi022 ceph-mon[40250]: pgmap v278: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:10 smithi122 ceph-mon[32255]: pgmap v278: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:12.137 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:12.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:12.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:12.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:12 smithi022 ceph-mon[31360]: pgmap v279: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1096731469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:12 smithi022 ceph-mon[40250]: pgmap v279: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1096731469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:12 smithi122 ceph-mon[32255]: pgmap v279: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1096731469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:15 smithi022 ceph-mon[31360]: pgmap v280: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:15 smithi022 ceph-mon[40250]: pgmap v280: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:15 smithi122 ceph-mon[32255]: pgmap v280: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:27:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:27:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:17 smithi022 ceph-mon[31360]: pgmap v281: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:17 smithi022 ceph-mon[40250]: pgmap v281: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:17 smithi122 ceph-mon[32255]: pgmap v281: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:17.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:17] "GET /metrics HTTP/1.1" 200 201948 "" "Prometheus/2.18.1" 2021-12-15T17:27:17.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:17.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:17.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3834887151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3834887151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3834887151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:19 smithi022 ceph-mon[31360]: pgmap v282: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:19 smithi022 ceph-mon[40250]: pgmap v282: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:19 smithi122 ceph-mon[32255]: pgmap v282: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:27:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:21 smithi022 ceph-mon[31360]: pgmap v283: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:21 smithi022 ceph-mon[40250]: pgmap v283: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:21 smithi122 ceph-mon[32255]: pgmap v283: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:22.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:23.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:23.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:23.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:23.251 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:23 smithi022 ceph-mon[40250]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:23.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:23 smithi022 ceph-mon[31360]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:23 smithi122 ceph-mon[32255]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1056599470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1056599470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1056599470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:25 smithi022 ceph-mon[31360]: pgmap v285: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:25 smithi022 ceph-mon[40250]: pgmap v285: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:25 smithi122 ceph-mon[32255]: pgmap v285: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:25.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:27:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:27:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:27 smithi022 ceph-mon[31360]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:27 smithi022 ceph-mon[40250]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:27 smithi122 ceph-mon[32255]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:27] "GET /metrics HTTP/1.1" 200 201947 "" "Prometheus/2.18.1" 2021-12-15T17:27:28.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:28.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:28.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:28.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:29 smithi022 ceph-mon[31360]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2042701550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:29 smithi022 ceph-mon[40250]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2042701550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:29 smithi122 ceph-mon[32255]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2042701550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:27:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:31 smithi022 ceph-mon[31360]: pgmap v288: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:31 smithi022 ceph-mon[40250]: pgmap v288: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:31 smithi122 ceph-mon[32255]: pgmap v288: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:33 smithi122 ceph-mon[32255]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:33 smithi022 ceph-mon[31360]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:33 smithi022 ceph-mon[40250]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:33.619 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:33.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:33.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:33.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1274228024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1274228024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1274228024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:35 smithi122 ceph-mon[32255]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:35 smithi022 ceph-mon[31360]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:35 smithi022 ceph-mon[40250]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:35.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:27:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:27:37.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:37 smithi022 ceph-mon[40250]: pgmap v291: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:37.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:37 smithi022 ceph-mon[31360]: pgmap v291: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:37 smithi122 ceph-mon[32255]: pgmap v291: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:37] "GET /metrics HTTP/1.1" 200 201944 "" "Prometheus/2.18.1" 2021-12-15T17:27:38.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:39 smithi022 ceph-mon[31360]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:39 smithi022 ceph-mon[40250]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:39.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:39.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:39.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:39 smithi122 ceph-mon[32255]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:27:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4272337983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4272337983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4272337983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:41 smithi122 ceph-mon[32255]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:41 smithi022 ceph-mon[31360]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:41 smithi022 ceph-mon[40250]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:43 smithi122 ceph-mon[32255]: pgmap v294: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:43 smithi022 ceph-mon[31360]: pgmap v294: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:43 smithi022 ceph-mon[40250]: pgmap v294: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:44.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:44.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:44.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:45 smithi122 ceph-mon[32255]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1716237430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:45 smithi022 ceph-mon[31360]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1716237430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:45 smithi022 ceph-mon[40250]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1716237430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:45.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:27:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:27:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:47 smithi022 ceph-mon[31360]: pgmap v296: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:47 smithi022 ceph-mon[40250]: pgmap v296: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:47 smithi122 ceph-mon[32255]: pgmap v296: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:47] "GET /metrics HTTP/1.1" 200 201944 "" "Prometheus/2.18.1" 2021-12-15T17:27:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:49 smithi122 ceph-mon[32255]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:49 smithi022 ceph-mon[31360]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:49 smithi022 ceph-mon[40250]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:49.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:50.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:50.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:50.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:27:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:51 smithi122 ceph-mon[32255]: pgmap v298: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1095232559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:51 smithi022 ceph-mon[31360]: pgmap v298: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1095232559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:51 smithi022 ceph-mon[40250]: pgmap v298: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:27:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1095232559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:53 smithi122 ceph-mon[32255]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:53 smithi022 ceph-mon[31360]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:53 smithi022 ceph-mon[40250]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:55.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:27:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:55 smithi122 ceph-mon[32255]: pgmap v300: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:55.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:27:55.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:27:55.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:27:55.475 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:55 smithi022 ceph-mon[31360]: pgmap v300: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:55.475 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:55 smithi022 ceph-mon[40250]: pgmap v300: 132 pgs: 132 active+clean; 582 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:27:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:27:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:27:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/84852412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/84852412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/84852412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:27:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:57 smithi022 ceph-mon[31360]: pgmap v301: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:27:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:57 smithi022 ceph-mon[40250]: pgmap v301: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:27:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:57 smithi122 ceph-mon[32255]: pgmap v301: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:27:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:27:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:27:57] "GET /metrics HTTP/1.1" 200 201954 "" "Prometheus/2.18.1" 2021-12-15T17:27:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:27:59 smithi122 ceph-mon[32255]: pgmap v302: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:27:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:27:59 smithi022 ceph-mon[31360]: pgmap v302: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:27:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:27:59 smithi022 ceph-mon[40250]: pgmap v302: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:28:00.475 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:00.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:00.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:00.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:01 smithi122 ceph-mon[32255]: pgmap v303: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2131456408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:28:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:28:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:28:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[31360]: pgmap v303: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2131456408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:28:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:28:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:28:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[40250]: pgmap v303: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2131456408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:28:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:28:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:28:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:28:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:02 smithi122 ceph-mon[32255]: pgmap v304: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:28:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:02 smithi022 ceph-mon[31360]: pgmap v304: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:28:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:02 smithi022 ceph-mon[40250]: pgmap v304: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:28:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:28:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:28:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:28:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:28:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:28:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:04 smithi122 ceph-mon[32255]: pgmap v305: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:04 smithi022 ceph-mon[31360]: pgmap v305: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:04 smithi022 ceph-mon[40250]: pgmap v305: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:05.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:05.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:28:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:28:06.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:06.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:06.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:06 smithi122 ceph-mon[32255]: pgmap v306: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4279210236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:06 smithi022 ceph-mon[31360]: pgmap v306: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4279210236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:06 smithi022 ceph-mon[40250]: pgmap v306: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:28:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4279210236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:07] "GET /metrics HTTP/1.1" 200 201977 "" "Prometheus/2.18.1" 2021-12-15T17:28:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:08 smithi122 ceph-mon[32255]: pgmap v307: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:08 smithi022 ceph-mon[31360]: pgmap v307: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:08 smithi022 ceph-mon[40250]: pgmap v307: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:28:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:10 smithi122 ceph-mon[32255]: pgmap v308: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:11.214 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:11.215 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:10 smithi022 ceph-mon[31360]: pgmap v308: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:11.216 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:10 smithi022 ceph-mon[40250]: pgmap v308: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:11.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:11.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:11.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:11.915 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1808052621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1808052621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1808052621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:13.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:12 smithi122 ceph-mon[32255]: pgmap v309: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:12 smithi022 ceph-mon[31360]: pgmap v309: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:12 smithi022 ceph-mon[40250]: pgmap v309: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:14 smithi122 ceph-mon[32255]: pgmap v310: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:14 smithi022 ceph-mon[31360]: pgmap v310: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:14 smithi022 ceph-mon[40250]: pgmap v310: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:28:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:28:16.587 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:16.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:16.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:16.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:16 smithi122 ceph-mon[32255]: pgmap v311: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:16 smithi022 ceph-mon[31360]: pgmap v311: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:16 smithi022 ceph-mon[40250]: pgmap v311: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:17] "GET /metrics HTTP/1.1" 200 201977 "" "Prometheus/2.18.1" 2021-12-15T17:28:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3460132104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3460132104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3460132104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:18 smithi122 ceph-mon[32255]: pgmap v312: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:18 smithi022 ceph-mon[31360]: pgmap v312: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:18 smithi022 ceph-mon[40250]: pgmap v312: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:28:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:20 smithi122 ceph-mon[32255]: pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:20 smithi022 ceph-mon[31360]: pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:20 smithi022 ceph-mon[40250]: pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:21.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:22.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:22.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:22.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:22 smithi122 ceph-mon[32255]: pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1945999219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:22 smithi022 ceph-mon[31360]: pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1945999219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:22 smithi022 ceph-mon[40250]: pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1945999219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:24 smithi022 ceph-mon[31360]: pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:24 smithi022 ceph-mon[40250]: pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:24 smithi122 ceph-mon[32255]: pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:26.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:28:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:28:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:26 smithi022 ceph-mon[31360]: pgmap v316: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:26 smithi022 ceph-mon[40250]: pgmap v316: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:27.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:26 smithi122 ceph-mon[32255]: pgmap v316: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:27.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:27.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:27.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:27.704 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:27] "GET /metrics HTTP/1.1" 200 201974 "" "Prometheus/2.18.1" 2021-12-15T17:28:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3059433053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3059433053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3059433053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:28 smithi022 ceph-mon[31360]: pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:28 smithi022 ceph-mon[40250]: pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:28 smithi122 ceph-mon[32255]: pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:28:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:30 smithi022 ceph-mon[31360]: pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:30 smithi022 ceph-mon[40250]: pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:30 smithi122 ceph-mon[32255]: pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:32.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:32 smithi022 ceph-mon[31360]: pgmap v319: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:32 smithi022 ceph-mon[40250]: pgmap v319: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:33.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:33.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:33.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:32 smithi122 ceph-mon[32255]: pgmap v319: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1437345401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1437345401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1437345401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:34 smithi022 ceph-mon[31360]: pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:34 smithi022 ceph-mon[40250]: pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:34 smithi122 ceph-mon[32255]: pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:36.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:28:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:28:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:36 smithi022 ceph-mon[31360]: pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:36 smithi022 ceph-mon[40250]: pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:36 smithi122 ceph-mon[32255]: pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:37] "GET /metrics HTTP/1.1" 200 201972 "" "Prometheus/2.18.1" 2021-12-15T17:28:38.081 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:38.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:38.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:38.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:38 smithi022 ceph-mon[31360]: pgmap v322: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1119308948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:38 smithi022 ceph-mon[40250]: pgmap v322: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1119308948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:38 smithi122 ceph-mon[32255]: pgmap v322: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1119308948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:28:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:40 smithi022 ceph-mon[31360]: pgmap v323: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:40 smithi022 ceph-mon[40250]: pgmap v323: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:40 smithi122 ceph-mon[32255]: pgmap v323: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:42 smithi022 ceph-mon[31360]: pgmap v324: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:42 smithi022 ceph-mon[40250]: pgmap v324: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:42 smithi122 ceph-mon[32255]: pgmap v324: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:43.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:43.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:43.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:43.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3389228241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3389228241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3389228241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:44 smithi022 ceph-mon[31360]: pgmap v325: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:44 smithi022 ceph-mon[40250]: pgmap v325: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:44 smithi122 ceph-mon[32255]: pgmap v325: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:28:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:28:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:46 smithi022 ceph-mon[31360]: pgmap v326: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:46 smithi022 ceph-mon[40250]: pgmap v326: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:46 smithi122 ceph-mon[32255]: pgmap v326: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:47] "GET /metrics HTTP/1.1" 200 201972 "" "Prometheus/2.18.1" 2021-12-15T17:28:48.823 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:49.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:49.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:49.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:49.193 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:48 smithi022 ceph-mon[31360]: pgmap v327: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:49.193 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:48 smithi022 ceph-mon[40250]: pgmap v327: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:48 smithi122 ceph-mon[32255]: pgmap v327: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1928686618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1928686618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:28:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1928686618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:50 smithi022 ceph-mon[31360]: pgmap v328: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:50 smithi022 ceph-mon[40250]: pgmap v328: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:50 smithi122 ceph-mon[32255]: pgmap v328: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:52 smithi022 ceph-mon[31360]: pgmap v329: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:53.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:52 smithi022 ceph-mon[40250]: pgmap v329: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:52 smithi122 ceph-mon[32255]: pgmap v329: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:54.194 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:54.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:54.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:54.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:28:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:54 smithi022 ceph-mon[31360]: pgmap v330: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2375730493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:54 smithi022 ceph-mon[40250]: pgmap v330: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2375730493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:54 smithi122 ceph-mon[32255]: pgmap v330: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2375730493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:28:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:28:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:28:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:56 smithi022 ceph-mon[31360]: pgmap v331: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:56 smithi022 ceph-mon[40250]: pgmap v331: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:56 smithi122 ceph-mon[32255]: pgmap v331: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:28:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:28:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:28:57] "GET /metrics HTTP/1.1" 200 201981 "" "Prometheus/2.18.1" 2021-12-15T17:28:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:28:58 smithi022 ceph-mon[31360]: pgmap v332: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:28:58 smithi022 ceph-mon[40250]: pgmap v332: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:28:58 smithi122 ceph-mon[32255]: pgmap v332: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:28:59.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:28:59.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:28:59.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:28:59.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:29:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:00 smithi122 ceph-mon[32255]: pgmap v333: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/632791596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:00 smithi022 ceph-mon[31360]: pgmap v333: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/632791596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:00 smithi022 ceph-mon[40250]: pgmap v333: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/632791596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:01.922 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:29:01.922 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:29:01.922 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:29:01.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:01.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:01.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:29:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:29:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:29:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:02.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:02.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:02.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:29:02.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:29:02.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:29:02.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:02.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T17:29:02.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:02.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:29:03.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:02 smithi122 ceph-mon[32255]: pgmap v334: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:02 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3269M 2021-12-15T17:29:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:02 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3270M 2021-12-15T17:29:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:02 smithi022 ceph-mon[31360]: pgmap v334: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:02 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3269M 2021-12-15T17:29:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:02 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3270M 2021-12-15T17:29:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:02 smithi022 ceph-mon[40250]: pgmap v334: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:02 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3269M 2021-12-15T17:29:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:02 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3270M 2021-12-15T17:29:04.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:29:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:29:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:29:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:29:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:29:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:29:04.935 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:04 smithi122 ceph-mon[32255]: pgmap v335: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:04 smithi022 ceph-mon[31360]: pgmap v335: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:04 smithi022 ceph-mon[40250]: pgmap v335: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:05.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:05.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:05.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2198141653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:29:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:29:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2198141653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2198141653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:06 smithi122 ceph-mon[32255]: pgmap v336: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:06 smithi022 ceph-mon[31360]: pgmap v336: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:06 smithi022 ceph-mon[40250]: pgmap v336: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:07] "GET /metrics HTTP/1.1" 200 201974 "" "Prometheus/2.18.1" 2021-12-15T17:29:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:08 smithi022 ceph-mon[31360]: pgmap v337: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:08 smithi022 ceph-mon[40250]: pgmap v337: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:08 smithi122 ceph-mon[32255]: pgmap v337: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:10.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:29:10.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:10.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:10.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:10 smithi022 ceph-mon[31360]: pgmap v338: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1771562617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:10 smithi022 ceph-mon[40250]: pgmap v338: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1771562617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:10 smithi122 ceph-mon[32255]: pgmap v338: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1771562617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:12 smithi022 ceph-mon[31360]: pgmap v339: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:12 smithi022 ceph-mon[40250]: pgmap v339: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:12 smithi122 ceph-mon[32255]: pgmap v339: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:14 smithi022 ceph-mon[31360]: pgmap v340: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:14 smithi022 ceph-mon[40250]: pgmap v340: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:14 smithi122 ceph-mon[32255]: pgmap v340: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:16.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:16.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:16.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:29:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:29:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:16 smithi022 ceph-mon[31360]: pgmap v341: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/5818631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:16 smithi022 ceph-mon[40250]: pgmap v341: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/5818631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:16 smithi122 ceph-mon[32255]: pgmap v341: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/5818631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:17] "GET /metrics HTTP/1.1" 200 201974 "" "Prometheus/2.18.1" 2021-12-15T17:29:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:18 smithi022 ceph-mon[40250]: pgmap v342: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:18 smithi022 ceph-mon[31360]: pgmap v342: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:18 smithi122 ceph-mon[32255]: pgmap v342: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:29:21.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:20 smithi022 ceph-mon[31360]: pgmap v343: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:20 smithi022 ceph-mon[40250]: pgmap v343: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:21.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:21.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:21.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:20 smithi122 ceph-mon[32255]: pgmap v343: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1055856966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1055856966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1055856966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:22 smithi022 ceph-mon[31360]: pgmap v344: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:22 smithi022 ceph-mon[40250]: pgmap v344: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:22 smithi122 ceph-mon[32255]: pgmap v344: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:24 smithi022 ceph-mon[31360]: pgmap v345: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:24 smithi022 ceph-mon[40250]: pgmap v345: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:24 smithi122 ceph-mon[32255]: pgmap v345: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:29:26.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:29:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:29:26.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:26.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:26.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:26.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:26 smithi022 ceph-mon[40250]: pgmap v346: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3921432854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:26 smithi022 ceph-mon[31360]: pgmap v346: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3921432854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:26 smithi122 ceph-mon[32255]: pgmap v346: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3921432854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:27] "GET /metrics HTTP/1.1" 200 201972 "" "Prometheus/2.18.1" 2021-12-15T17:29:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:28 smithi022 ceph-mon[31360]: pgmap v347: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:28 smithi022 ceph-mon[40250]: pgmap v347: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:28 smithi122 ceph-mon[32255]: pgmap v347: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:29:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:30 smithi022 ceph-mon[31360]: pgmap v348: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:30 smithi022 ceph-mon[40250]: pgmap v348: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:30 smithi122 ceph-mon[32255]: pgmap v348: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:31.786 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:32.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:32.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:32.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:32 smithi022 ceph-mon[31360]: pgmap v349: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/778697830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:32 smithi022 ceph-mon[40250]: pgmap v349: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/778697830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:32 smithi122 ceph-mon[32255]: pgmap v349: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/778697830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:34 smithi022 ceph-mon[31360]: pgmap v350: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:34 smithi022 ceph-mon[40250]: pgmap v350: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:34 smithi122 ceph-mon[32255]: pgmap v350: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:36.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:29:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:29:37.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:36 smithi022 ceph-mon[31360]: pgmap v351: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:36 smithi022 ceph-mon[40250]: pgmap v351: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:36 smithi122 ceph-mon[32255]: pgmap v351: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:37] "GET /metrics HTTP/1.1" 200 201976 "" "Prometheus/2.18.1" 2021-12-15T17:29:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3717389037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3717389037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3717389037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:38 smithi022 ceph-mon[31360]: pgmap v352: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:38 smithi022 ceph-mon[40250]: pgmap v352: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:38 smithi122 ceph-mon[32255]: pgmap v352: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:29:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:40 smithi022 ceph-mon[31360]: pgmap v353: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:40 smithi022 ceph-mon[40250]: pgmap v353: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:40 smithi122 ceph-mon[32255]: pgmap v353: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:42.529 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:42.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:42.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:42.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:42 smithi022 ceph-mon[31360]: pgmap v354: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2208022962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:42 smithi022 ceph-mon[40250]: pgmap v354: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2208022962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:42 smithi122 ceph-mon[32255]: pgmap v354: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2208022962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:44 smithi022 ceph-mon[31360]: pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:44 smithi022 ceph-mon[40250]: pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:44 smithi122 ceph-mon[32255]: pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:46.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:29:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:29:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:47 smithi022 ceph-mon[31360]: pgmap v356: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:47 smithi022 ceph-mon[40250]: pgmap v356: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:47 smithi122 ceph-mon[32255]: pgmap v356: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:47] "GET /metrics HTTP/1.1" 200 201976 "" "Prometheus/2.18.1" 2021-12-15T17:29:47.901 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:48.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:49 smithi022 ceph-mon[31360]: pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/437256749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:49 smithi022 ceph-mon[40250]: pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/437256749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:49 smithi122 ceph-mon[32255]: pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/437256749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:29:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:51 smithi022 ceph-mon[31360]: pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:51 smithi022 ceph-mon[40250]: pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:51 smithi122 ceph-mon[32255]: pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:53.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:53 smithi022 ceph-mon[31360]: pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:53.274 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:53 smithi022 ceph-mon[40250]: pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:53 smithi122 ceph-mon[32255]: pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:53.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:53.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:53.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3594946138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3594946138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3594946138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:55 smithi022 ceph-mon[31360]: pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:55 smithi022 ceph-mon[40250]: pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:55 smithi122 ceph-mon[32255]: pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:56.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:29:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:29:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:57 smithi022 ceph-mon[31360]: pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:57 smithi022 ceph-mon[40250]: pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:57 smithi122 ceph-mon[32255]: pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:29:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:29:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:29:57] "GET /metrics HTTP/1.1" 200 201984 "" "Prometheus/2.18.1" 2021-12-15T17:29:58.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:29:59.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:29:59.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:29:59.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:29:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:59 smithi022 ceph-mon[31360]: pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:29:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4182340539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:59 smithi022 ceph-mon[40250]: pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:29:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4182340539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:29:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:59 smithi122 ceph-mon[32255]: pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:29:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:29:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4182340539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:30:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:00 smithi122 ceph-mon[32255]: pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T17:30:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:00 smithi022 ceph-mon[31360]: pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T17:30:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:00 smithi022 ceph-mon[40250]: pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T17:30:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:30:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:30:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:30:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:30:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:30:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:30:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:30:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:30:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:30:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:02 smithi022 ceph-mon[31360]: pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:02 smithi022 ceph-mon[40250]: pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:02 smithi122 ceph-mon[32255]: pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:04.014 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:30:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:30:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:30:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:30:04.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:04.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:04.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:30:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:30:05.230 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:04 smithi022 ceph-mon[31360]: pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:05.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1967549690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:05.231 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:04 smithi022 ceph-mon[40250]: pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:05.231 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1967549690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:04 smithi122 ceph-mon[32255]: pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1967549690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:06.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:30:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:30:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:06 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:06 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:06 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:07] "GET /metrics HTTP/1.1" 200 201983 "" "Prometheus/2.18.1" 2021-12-15T17:30:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[31360]: pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[40250]: pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:07 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:07 smithi122 ceph-mon[32255]: pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:07 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:07 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:30:09.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:09 smithi022 ceph-mon[31360]: pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:09 smithi022 ceph-mon[40250]: pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:09 smithi122 ceph-mon[32255]: pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:09.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:09.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:09.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:30:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4211420131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4211420131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4211420131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:11 smithi022 ceph-mon[31360]: pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:11 smithi022 ceph-mon[40250]: pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:11 smithi122 ceph-mon[32255]: pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:13 smithi022 ceph-mon[31360]: pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:13 smithi022 ceph-mon[40250]: pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:13 smithi122 ceph-mon[32255]: pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:14.765 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:15.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:15.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:15.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:15 smithi022 ceph-mon[31360]: pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3066416734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:15 smithi022 ceph-mon[40250]: pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3066416734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:15 smithi122 ceph-mon[32255]: pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3066416734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:16.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:30:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:30:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:17] "GET /metrics HTTP/1.1" 200 201983 "" "Prometheus/2.18.1" 2021-12-15T17:30:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:17 smithi022 ceph-mon[31360]: pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:17 smithi022 ceph-mon[40250]: pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:17 smithi122 ceph-mon[32255]: pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:17.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:19 smithi022 ceph-mon[31360]: pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:19 smithi022 ceph-mon[40250]: pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:19 smithi122 ceph-mon[32255]: pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:20.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:30:20.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:20.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:20.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:21 smithi022 ceph-mon[31360]: pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2523729182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:21 smithi022 ceph-mon[40250]: pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2523729182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:21 smithi122 ceph-mon[32255]: pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2523729182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:23 smithi022 ceph-mon[31360]: pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:23 smithi022 ceph-mon[40250]: pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:23 smithi122 ceph-mon[32255]: pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:25.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:25 smithi022 ceph-mon[31360]: pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:25 smithi022 ceph-mon[40250]: pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:25 smithi122 ceph-mon[32255]: pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:26.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:30:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:30:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/996262198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/996262198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/996262198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:27 smithi122 ceph-mon[32255]: pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:27 smithi022 ceph-mon[31360]: pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:27] "GET /metrics HTTP/1.1" 200 201973 "" "Prometheus/2.18.1" 2021-12-15T17:30:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:27 smithi022 ceph-mon[40250]: pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:29 smithi122 ceph-mon[32255]: pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:29 smithi022 ceph-mon[31360]: pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:29 smithi022 ceph-mon[40250]: pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:30:30.878 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:31.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:31.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:31.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:31 smithi022 ceph-mon[31360]: pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3887967645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:31 smithi022 ceph-mon[40250]: pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3887967645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:31 smithi122 ceph-mon[32255]: pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3887967645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:33 smithi122 ceph-mon[32255]: pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:33 smithi022 ceph-mon[31360]: pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:33 smithi022 ceph-mon[40250]: pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:35 smithi122 ceph-mon[32255]: pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:35 smithi022 ceph-mon[31360]: pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:35 smithi022 ceph-mon[40250]: pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:36.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:30:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:30:36.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:36 smithi022 ceph-mon[31360]: pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:36 smithi022 ceph-mon[40250]: pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:36.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:36.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:36.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:36 smithi122 ceph-mon[32255]: pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2233054490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:37] "GET /metrics HTTP/1.1" 200 201978 "" "Prometheus/2.18.1" 2021-12-15T17:30:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2233054490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2233054490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:38 smithi122 ceph-mon[32255]: pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:38 smithi022 ceph-mon[31360]: pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:38 smithi022 ceph-mon[40250]: pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:30:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:40 smithi022 ceph-mon[31360]: pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:40 smithi022 ceph-mon[40250]: pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:40 smithi122 ceph-mon[32255]: pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:41.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:41.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:41.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:41.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:42 smithi022 ceph-mon[31360]: pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3652489596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:42 smithi022 ceph-mon[40250]: pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3652489596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:42 smithi122 ceph-mon[32255]: pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:43.569 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3652489596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:44 smithi022 ceph-mon[31360]: pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:44 smithi022 ceph-mon[40250]: pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:44 smithi122 ceph-mon[32255]: pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:30:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:30:46.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:46 smithi022 ceph-mon[31360]: pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:46 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:46 smithi022 ceph-mon[40250]: pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:46 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:47.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:47.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:47.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:46 smithi122 ceph-mon[32255]: pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:46 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:47] "GET /metrics HTTP/1.1" 200 201978 "" "Prometheus/2.18.1" 2021-12-15T17:30:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/374166575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/374166575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/374166575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:48 smithi022 ceph-mon[31360]: pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:48 smithi022 ceph-mon[40250]: pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:48 smithi122 ceph-mon[32255]: pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:30:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:50 smithi022 ceph-mon[31360]: pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:50 smithi022 ceph-mon[40250]: pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:50 smithi122 ceph-mon[32255]: pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:52.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:52.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:52.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:52.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:52 smithi022 ceph-mon[31360]: pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3198440508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:52 smithi022 ceph-mon[40250]: pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3198440508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:52 smithi122 ceph-mon[32255]: pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3198440508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:54 smithi022 ceph-mon[31360]: pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:54 smithi022 ceph-mon[40250]: pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:54 smithi122 ceph-mon[32255]: pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:30:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:30:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:56 smithi022 ceph-mon[31360]: pgmap v391: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:56 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:56 smithi022 ceph-mon[40250]: pgmap v391: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:56 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:56 smithi122 ceph-mon[32255]: pgmap v391: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:30:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:56 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:30:57.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:30:57.737 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:30:57] "GET /metrics HTTP/1.1" 200 201982 "" "Prometheus/2.18.1" 2021-12-15T17:30:58.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:30:58.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:30:58.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:30:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:58 smithi022 ceph-mon[31360]: pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:30:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/279475779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:58 smithi022 ceph-mon[40250]: pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:30:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/279475779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:30:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:58 smithi122 ceph-mon[32255]: pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:30:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:30:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/279475779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:31:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:00 smithi022 ceph-mon[31360]: pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:00 smithi022 ceph-mon[40250]: pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:00 smithi122 ceph-mon[32255]: pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:03.107 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:02 smithi022 ceph-mon[31360]: pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:02 smithi022 ceph-mon[40250]: pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:02 smithi122 ceph-mon[32255]: pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:03.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:03.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:03.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:31:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:31:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/722727306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:31:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:31:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/722727306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:31:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:31:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/722727306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:04 smithi022 ceph-mon[31360]: pgmap v395: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:04 smithi022 ceph-mon[40250]: pgmap v395: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:04 smithi122 ceph-mon[32255]: pgmap v395: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:31:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:31:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[31360]: pgmap v396: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:31:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:31:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:31:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[40250]: pgmap v396: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:31:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:31:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:06 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:31:07.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:06 smithi122 ceph-mon[32255]: pgmap v396: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:07.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:07.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:06 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:31:07.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:06 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:31:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:06 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:31:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:07] "GET /metrics HTTP/1.1" 200 201985 "" "Prometheus/2.18.1" 2021-12-15T17:31:08.489 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:08.490 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:08 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:08.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:08 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:08 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:08.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:08.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:08.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:09 smithi022 ceph-mon[31360]: pgmap v397: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3213231760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:09 smithi022 ceph-mon[40250]: pgmap v397: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3213231760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:09 smithi122 ceph-mon[32255]: pgmap v397: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3213231760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:31:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:11 smithi022 ceph-mon[31360]: pgmap v398: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:11 smithi022 ceph-mon[40250]: pgmap v398: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:11 smithi122 ceph-mon[32255]: pgmap v398: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:13 smithi022 ceph-mon[31360]: pgmap v399: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:13 smithi022 ceph-mon[40250]: pgmap v399: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:13 smithi122 ceph-mon[32255]: pgmap v399: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:13.858 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:14.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:14.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:14.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:15 smithi022 ceph-mon[31360]: pgmap v400: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1590318078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:15 smithi022 ceph-mon[40250]: pgmap v400: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1590318078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:15 smithi122 ceph-mon[32255]: pgmap v400: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1590318078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:16.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:31:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:31:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:17 smithi022 ceph-mon[31360]: pgmap v401: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:17] "GET /metrics HTTP/1.1" 200 201985 "" "Prometheus/2.18.1" 2021-12-15T17:31:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:17 smithi022 ceph-mon[40250]: pgmap v401: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:17 smithi122 ceph-mon[32255]: pgmap v401: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:19.228 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:19 smithi022 ceph-mon[31360]: pgmap v402: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:19 smithi022 ceph-mon[40250]: pgmap v402: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:19.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:19.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:19.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:19 smithi122 ceph-mon[32255]: pgmap v402: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2415073482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:31:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2415073482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2415073482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:21 smithi022 ceph-mon[31360]: pgmap v403: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:21 smithi022 ceph-mon[40250]: pgmap v403: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:21 smithi122 ceph-mon[32255]: pgmap v403: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:23 smithi022 ceph-mon[31360]: pgmap v404: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:23 smithi022 ceph-mon[40250]: pgmap v404: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:23 smithi122 ceph-mon[32255]: pgmap v404: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:24.599 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:24.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:24.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:25 smithi022 ceph-mon[31360]: pgmap v405: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3323949136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:25 smithi022 ceph-mon[40250]: pgmap v405: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3323949136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:25 smithi122 ceph-mon[32255]: pgmap v405: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3323949136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:26.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:31:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:31:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:27 smithi022 ceph-mon[31360]: pgmap v406: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:27] "GET /metrics HTTP/1.1" 200 201987 "" "Prometheus/2.18.1" 2021-12-15T17:31:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:27 smithi022 ceph-mon[40250]: pgmap v406: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:27 smithi122 ceph-mon[32255]: pgmap v406: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:29 smithi022 ceph-mon[31360]: pgmap v407: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:29 smithi022 ceph-mon[40250]: pgmap v407: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:29 smithi122 ceph-mon[32255]: pgmap v407: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:29.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:30.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:30.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:30.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:31:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:31 smithi022 ceph-mon[31360]: pgmap v408: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3797170891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:31 smithi022 ceph-mon[40250]: pgmap v408: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3797170891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:31 smithi122 ceph-mon[32255]: pgmap v408: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3797170891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:33 smithi022 ceph-mon[31360]: pgmap v409: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:33 smithi022 ceph-mon[40250]: pgmap v409: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:33 smithi122 ceph-mon[32255]: pgmap v409: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:35.343 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:35 smithi022 ceph-mon[31360]: pgmap v410: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:35 smithi022 ceph-mon[40250]: pgmap v410: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:35 smithi122 ceph-mon[32255]: pgmap v410: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:35.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:35.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:35.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:36.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:31:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:31:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2305508821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2305508821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2305508821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:37] "GET /metrics HTTP/1.1" 200 201984 "" "Prometheus/2.18.1" 2021-12-15T17:31:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:37 smithi022 ceph-mon[31360]: pgmap v411: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:37 smithi022 ceph-mon[40250]: pgmap v411: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:37 smithi122 ceph-mon[32255]: pgmap v411: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:39 smithi022 ceph-mon[31360]: pgmap v412: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:39 smithi022 ceph-mon[40250]: pgmap v412: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:39 smithi122 ceph-mon[32255]: pgmap v412: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:31:40.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:41.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:41.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:41.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:41 smithi022 ceph-mon[31360]: pgmap v413: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3906975884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:41 smithi022 ceph-mon[40250]: pgmap v413: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3906975884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:41 smithi122 ceph-mon[32255]: pgmap v413: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3906975884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:43 smithi022 ceph-mon[31360]: pgmap v414: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:43 smithi022 ceph-mon[40250]: pgmap v414: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:43 smithi122 ceph-mon[32255]: pgmap v414: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:45 smithi022 ceph-mon[31360]: pgmap v415: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:45.658 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:45 smithi022 ceph-mon[40250]: pgmap v415: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:45 smithi122 ceph-mon[32255]: pgmap v415: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:46.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:46.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:31:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:31:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:47] "GET /metrics HTTP/1.1" 200 201984 "" "Prometheus/2.18.1" 2021-12-15T17:31:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:47 smithi022 ceph-mon[31360]: pgmap v416: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2869412638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:47 smithi022 ceph-mon[40250]: pgmap v416: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2869412638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:47 smithi122 ceph-mon[32255]: pgmap v416: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2869412638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:49 smithi022 ceph-mon[31360]: pgmap v417: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:49 smithi022 ceph-mon[40250]: pgmap v417: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:49 smithi122 ceph-mon[32255]: pgmap v417: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:31:51.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:51 smithi022 ceph-mon[31360]: pgmap v418: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:51 smithi022 ceph-mon[40250]: pgmap v418: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:51 smithi122 ceph-mon[32255]: pgmap v418: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:51.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== 2021-12-15T17:31:51.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== ']' 2021-12-15T17:31:51.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3991046021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3991046021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3991046021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:53 smithi022 ceph-mon[31360]: pgmap v419: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:53 smithi022 ceph-mon[40250]: pgmap v419: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:53 smithi122 ceph-mon[32255]: pgmap v419: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:55 smithi022 ceph-mon[31360]: pgmap v420: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:55 smithi022 ceph-mon[40250]: pgmap v420: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:55 smithi122 ceph-mon[32255]: pgmap v420: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:56.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:31:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:31:56.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.2 2021-12-15T17:31:57.197 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.3 2021-12-15T17:31:57.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCvJLphxJssCRAAXCEHJo0K+oEVG/bSRp0JcA== 2021-12-15T17:31:57.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQB4IrphLaNoKRAA5L5sUozWRFqKl8cBel1ynQ== == AQCvJLphxJssCRAAXCEHJo0K+oEVG/bSRp0JcA== ']' 2021-12-15T17:31:57.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:31:57.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.3' 2021-12-15T17:31:57.198 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:31:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:31:57] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:31:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:57 smithi022 ceph-mon[31360]: pgmap v421: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1282598405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:57 smithi022 ceph-mon[40250]: pgmap v421: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1282598405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:57.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:31:57.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:31:57.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.3 2021-12-15T17:31:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:57 smithi122 ceph-mon[32255]: pgmap v421: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:31:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:31:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1282598405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2021-12-15T17:31:57.842 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.3 on host 'smithi022' 2021-12-15T17:31:57.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:31:57.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:31:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2907710956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:31:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:31:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:31:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:31:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2021-12-15T17:31:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2021-12-15T17:31:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:58 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2907710956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:31:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:31:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:31:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:31:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2021-12-15T17:31:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2021-12-15T17:31:58.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2907710956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:31:58.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:31:58.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:31:58.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:31:58.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:58.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2021-12-15T17:31:58.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2021-12-15T17:31:58.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:58 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:31:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:59 smithi122 ceph-mon[32255]: pgmap v422: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:59 smithi122 ceph-mon[32255]: from='client.25714 v1:172.21.15.22:0/2997451547' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:31:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:59 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.3 2021-12-15T17:31:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:31:59 smithi122 ceph-mon[32255]: Rotating authentication key for osd.3 2021-12-15T17:31:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[31360]: pgmap v422: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[31360]: from='client.25714 v1:172.21.15.22:0/2997451547' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:31:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.3 2021-12-15T17:31:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[31360]: Rotating authentication key for osd.3 2021-12-15T17:31:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[40250]: pgmap v422: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:31:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[40250]: from='client.25714 v1:172.21.15.22:0/2997451547' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:31:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.3 2021-12-15T17:31:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:31:59 smithi022 ceph-mon[40250]: Rotating authentication key for osd.3 2021-12-15T17:32:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:32:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:01 smithi122 ceph-mon[32255]: pgmap v423: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:01 smithi022 ceph-mon[31360]: pgmap v423: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:01 smithi022 ceph-mon[40250]: pgmap v423: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:02.858 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:03.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:03.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:03.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[31360]: pgmap v424: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2552797328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:32:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:32:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[40250]: pgmap v424: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2552797328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:32:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:32:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:03 smithi122 ceph-mon[32255]: pgmap v424: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2552797328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:32:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:32:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:05 smithi122 ceph-mon[32255]: pgmap v425: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:05 smithi022 ceph-mon[31360]: pgmap v425: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:05 smithi022 ceph-mon[40250]: pgmap v425: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:32:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:32:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:07 smithi122 ceph-mon[32255]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:07 smithi022 ceph-mon[31360]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:07] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:32:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:07 smithi022 ceph-mon[40250]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:08.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:08.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:08.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:08.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:09 smithi122 ceph-mon[32255]: pgmap v427: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/543090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:09 smithi022 ceph-mon[31360]: pgmap v427: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/543090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:09 smithi022 ceph-mon[40250]: pgmap v427: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/543090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:32:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:11 smithi122 ceph-mon[32255]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:11 smithi022 ceph-mon[31360]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:11 smithi022 ceph-mon[40250]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:13.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:13.603 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:13 smithi022 ceph-mon[31360]: pgmap v429: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:13.603 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:13 smithi022 ceph-mon[40250]: pgmap v429: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:13 smithi122 ceph-mon[32255]: pgmap v429: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:13.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:13.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:13.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3403204612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3403204612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3403204612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:15 smithi122 ceph-mon[32255]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:15 smithi022 ceph-mon[31360]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:15 smithi022 ceph-mon[40250]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:16.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:32:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:32:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:16 smithi122 ceph-mon[32255]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:16 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:16 smithi022 ceph-mon[31360]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:16 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:16 smithi022 ceph-mon[40250]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:16 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:17] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:32:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:18 smithi122 ceph-mon[32255]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:18 smithi022 ceph-mon[31360]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:18 smithi022 ceph-mon[40250]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:18.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:19.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:19.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:19.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1687521106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1687521106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1687521106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:32:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:20 smithi022 ceph-mon[31360]: pgmap v433: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:20 smithi022 ceph-mon[40250]: pgmap v433: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:20 smithi122 ceph-mon[32255]: pgmap v433: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:22 smithi022 ceph-mon[31360]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:23.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:22 smithi022 ceph-mon[40250]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:22 smithi122 ceph-mon[32255]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:24.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:24.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:24.974 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:24 smithi022 ceph-mon[40250]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:24.974 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3904316769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:24 smithi022 ceph-mon[31360]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3904316769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:24 smithi122 ceph-mon[32255]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3904316769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:26.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:32:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:32:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:26 smithi022 ceph-mon[31360]: pgmap v436: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:26 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:26 smithi022 ceph-mon[40250]: pgmap v436: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:26 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:26 smithi122 ceph-mon[32255]: pgmap v436: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:26 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:27.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:27] "GET /metrics HTTP/1.1" 200 201984 "" "Prometheus/2.18.1" 2021-12-15T17:32:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:28 smithi022 ceph-mon[31360]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:28 smithi022 ceph-mon[40250]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:28 smithi122 ceph-mon[32255]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:29.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:30.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:30.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:30.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:32:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:30 smithi022 ceph-mon[31360]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/787798234' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:30 smithi022 ceph-mon[40250]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/787798234' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:30 smithi122 ceph-mon[32255]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/787798234' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:32 smithi022 ceph-mon[31360]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:32 smithi022 ceph-mon[40250]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:32 smithi122 ceph-mon[32255]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:35.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:35.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:34 smithi022 ceph-mon[31360]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:34 smithi022 ceph-mon[40250]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:34 smithi122 ceph-mon[32255]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:35.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:36.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:32:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:32:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1363493393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1363493393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1363493393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:36 smithi022 ceph-mon[31360]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:36 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:36 smithi022 ceph-mon[40250]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:36 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:36 smithi122 ceph-mon[32255]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:36 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:37] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:32:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:38 smithi022 ceph-mon[31360]: pgmap v442: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:38 smithi022 ceph-mon[40250]: pgmap v442: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:38 smithi122 ceph-mon[32255]: pgmap v442: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:32:40.471 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:40.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:40.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:40.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:40 smithi022 ceph-mon[31360]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/991188231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:40 smithi022 ceph-mon[40250]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/991188231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:40 smithi122 ceph-mon[32255]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/991188231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:43 smithi022 ceph-mon[31360]: pgmap v444: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:43 smithi022 ceph-mon[40250]: pgmap v444: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:43 smithi122 ceph-mon[32255]: pgmap v444: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:45 smithi022 ceph-mon[31360]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:45 smithi022 ceph-mon[40250]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:45 smithi122 ceph-mon[32255]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:45.843 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:46.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:32:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:32:46.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:46.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:46.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:47 smithi022 ceph-mon[31360]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:47.307 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:47.307 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4049547839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:47.308 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:47 smithi022 ceph-mon[40250]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:47.308 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:47.308 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4049547839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:47 smithi122 ceph-mon[32255]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4049547839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:47] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:32:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:49 smithi022 ceph-mon[31360]: pgmap v447: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:49 smithi022 ceph-mon[40250]: pgmap v447: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:49 smithi122 ceph-mon[32255]: pgmap v447: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:32:51.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:51 smithi022 ceph-mon[31360]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:51 smithi022 ceph-mon[40250]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:51 smithi122 ceph-mon[32255]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:51.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:51.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:51.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:52.017 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/204623716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/204623716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/204623716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:53 smithi022 ceph-mon[31360]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:53.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:53 smithi022 ceph-mon[40250]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:53 smithi122 ceph-mon[32255]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:55 smithi022 ceph-mon[31360]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:55 smithi022 ceph-mon[40250]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:55 smithi122 ceph-mon[32255]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:56.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:32:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:32:56.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:32:56.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:32:56.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:32:56.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:32:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:57 smithi022 ceph-mon[31360]: pgmap v451: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/421154410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:57 smithi022 ceph-mon[40250]: pgmap v451: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/421154410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:57 smithi122 ceph-mon[32255]: pgmap v451: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:32:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:32:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/421154410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:32:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:32:57] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:32:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[31360]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:32:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:32:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:32:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:32:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[40250]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:32:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:32:59.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:32:59.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:32:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:32:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:59 smithi122 ceph-mon[32255]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:32:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:32:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:32:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:32:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:32:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:33:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:33:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:01 smithi122 ceph-mon[32255]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:01 smithi022 ceph-mon[31360]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:01 smithi022 ceph-mon[40250]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:01.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:02.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:02.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:02.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:03 smithi122 ceph-mon[32255]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3063726074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:03 smithi022 ceph-mon[31360]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3063726074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:03 smithi022 ceph-mon[40250]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3063726074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:33:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:33:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:33:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:33:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:33:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:33:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:05 smithi122 ceph-mon[32255]: pgmap v455: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:05 smithi022 ceph-mon[31360]: pgmap v455: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:05 smithi022 ceph-mon[40250]: pgmap v455: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:06.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:33:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:33:07.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:07.332 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:07 smithi022 ceph-mon[31360]: pgmap v456: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:07.332 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:07.332 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:07 smithi022 ceph-mon[40250]: pgmap v456: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:07.333 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:07 smithi122 ceph-mon[32255]: pgmap v456: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:07.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:07.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:07.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:07.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:07] "GET /metrics HTTP/1.1" 200 201993 "" "Prometheus/2.18.1" 2021-12-15T17:33:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2052477800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2052477800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2052477800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:09 smithi122 ceph-mon[32255]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:09 smithi022 ceph-mon[31360]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:09 smithi022 ceph-mon[40250]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:33:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:11 smithi122 ceph-mon[32255]: pgmap v458: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:11 smithi022 ceph-mon[31360]: pgmap v458: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:11 smithi022 ceph-mon[40250]: pgmap v458: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:12.702 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:13.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:13.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:13.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:13 smithi122 ceph-mon[32255]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:13 smithi022 ceph-mon[31360]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:13 smithi022 ceph-mon[40250]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1106210372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1106210372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1106210372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:15 smithi122 ceph-mon[32255]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:15 smithi022 ceph-mon[31360]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:15 smithi022 ceph-mon[40250]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:33:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:33:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:17 smithi022 ceph-mon[31360]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:17 smithi022 ceph-mon[40250]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:17 smithi122 ceph-mon[32255]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:17] "GET /metrics HTTP/1.1" 200 201993 "" "Prometheus/2.18.1" 2021-12-15T17:33:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:18.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:18.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:18.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:19 smithi122 ceph-mon[32255]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1494719910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:19 smithi022 ceph-mon[31360]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1494719910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:19 smithi022 ceph-mon[40250]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1494719910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:33:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:21 smithi122 ceph-mon[32255]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:21 smithi022 ceph-mon[31360]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:21 smithi022 ceph-mon[40250]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:23 smithi122 ceph-mon[32255]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:23.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:23.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:23 smithi022 ceph-mon[31360]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:23.445 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:23 smithi022 ceph-mon[40250]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:23.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:23.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:23.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1314707842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1314707842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1314707842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:25 smithi122 ceph-mon[32255]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:25 smithi022 ceph-mon[31360]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:25 smithi022 ceph-mon[40250]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:26.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:33:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:33:27.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:27 smithi022 ceph-mon[40250]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:27.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:27 smithi022 ceph-mon[31360]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:27 smithi122 ceph-mon[32255]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:27] "GET /metrics HTTP/1.1" 200 201993 "" "Prometheus/2.18.1" 2021-12-15T17:33:28.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:29.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:29.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:29.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:29 smithi122 ceph-mon[32255]: pgmap v467: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:29 smithi022 ceph-mon[31360]: pgmap v467: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:29 smithi022 ceph-mon[40250]: pgmap v467: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:33:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1337674738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1337674738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1337674738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:31 smithi122 ceph-mon[32255]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:31 smithi022 ceph-mon[31360]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:31 smithi022 ceph-mon[40250]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:33 smithi122 ceph-mon[32255]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:33 smithi022 ceph-mon[31360]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:33 smithi022 ceph-mon[40250]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:34.188 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:34.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:34.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:34.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:35 smithi122 ceph-mon[32255]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1244919847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:35 smithi022 ceph-mon[31360]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1244919847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:35 smithi022 ceph-mon[40250]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1244919847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:36.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:33:35 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:33:37.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:37 smithi022 ceph-mon[40250]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:37 smithi022 ceph-mon[31360]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:37 smithi122 ceph-mon[32255]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:37] "GET /metrics HTTP/1.1" 200 201985 "" "Prometheus/2.18.1" 2021-12-15T17:33:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:39 smithi122 ceph-mon[32255]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:39 smithi022 ceph-mon[31360]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:39 smithi022 ceph-mon[40250]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:39.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:39.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:39.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:39.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2083111790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2083111790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:33:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2083111790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:41 smithi122 ceph-mon[32255]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:41 smithi022 ceph-mon[31360]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:41 smithi022 ceph-mon[40250]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:43 smithi122 ceph-mon[32255]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:43 smithi022 ceph-mon[31360]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:43 smithi022 ceph-mon[40250]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:44.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:45 smithi022 ceph-mon[31360]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:45 smithi022 ceph-mon[40250]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:45.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:45.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:45.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:45 smithi122 ceph-mon[32255]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:46.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:33:45 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:33:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1553724538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1553724538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1553724538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:47.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:47 smithi022 ceph-mon[40250]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:47 smithi022 ceph-mon[31360]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:47 smithi122 ceph-mon[32255]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:47] "GET /metrics HTTP/1.1" 200 201985 "" "Prometheus/2.18.1" 2021-12-15T17:33:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:49 smithi122 ceph-mon[32255]: pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:49 smithi022 ceph-mon[31360]: pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:49 smithi022 ceph-mon[40250]: pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:50.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:33:50.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:50.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:50.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:51 smithi122 ceph-mon[32255]: pgmap v478: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/570609406' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:51 smithi022 ceph-mon[31360]: pgmap v478: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/570609406' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:51 smithi022 ceph-mon[40250]: pgmap v478: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/570609406' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:53 smithi122 ceph-mon[32255]: pgmap v479: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:53 smithi022 ceph-mon[31360]: pgmap v479: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:53 smithi022 ceph-mon[40250]: pgmap v479: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:55 smithi122 ceph-mon[32255]: pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:55 smithi022 ceph-mon[31360]: pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:55 smithi022 ceph-mon[40250]: pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:55.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:33:56.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:33:56.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:33:56.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:33:56.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:33:55 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:33:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3018083730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3018083730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3018083730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:33:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:57 smithi022 ceph-mon[31360]: pgmap v481: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:57 smithi022 ceph-mon[40250]: pgmap v481: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:57 smithi122 ceph-mon[32255]: pgmap v481: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:33:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:33:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:33:57] "GET /metrics HTTP/1.1" 200 201992 "" "Prometheus/2.18.1" 2021-12-15T17:33:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[31360]: pgmap v482: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:33:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:33:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:33:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:33:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:33:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[40250]: pgmap v482: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:33:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:33:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:33:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:33:59.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:33:59 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:33:59.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:59 smithi122 ceph-mon[32255]: pgmap v482: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:33:59.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:33:59.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:33:59.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:33:59.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:33:59.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:33:59 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:34:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:34:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:00 smithi022 ceph-mon[40250]: pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:34:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:00 smithi022 ceph-mon[31360]: pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:34:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:00 smithi122 ceph-mon[32255]: pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:34:01.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3213172965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3213172965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:01.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3213172965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:02 smithi022 ceph-mon[31360]: pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:02 smithi022 ceph-mon[40250]: pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:02.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:02 smithi122 ceph-mon[32255]: pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:34:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:34:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:34:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:34:03.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:34:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:34:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:04 smithi022 ceph-mon[31360]: pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:04 smithi022 ceph-mon[40250]: pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:04.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:04 smithi122 ceph-mon[32255]: pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:06.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:06.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:34:05 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:34:06.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:06.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:06.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:06 smithi022 ceph-mon[31360]: pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:06 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2484203366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:06 smithi022 ceph-mon[40250]: pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:06 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2484203366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:06 smithi122 ceph-mon[32255]: pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:06 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2484203366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:07] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:34:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:08 smithi022 ceph-mon[31360]: pgmap v487: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:08 smithi022 ceph-mon[40250]: pgmap v487: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:08 smithi122 ceph-mon[32255]: pgmap v487: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:34:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:10 smithi022 ceph-mon[31360]: pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:10 smithi022 ceph-mon[40250]: pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:10 smithi122 ceph-mon[32255]: pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:12.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:12.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:12.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:12 smithi022 ceph-mon[31360]: pgmap v489: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/809349358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:12 smithi022 ceph-mon[40250]: pgmap v489: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/809349358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:13.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:12 smithi122 ceph-mon[32255]: pgmap v489: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/809349358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:14 smithi022 ceph-mon[31360]: pgmap v490: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:14 smithi022 ceph-mon[40250]: pgmap v490: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:14 smithi122 ceph-mon[32255]: pgmap v490: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:16.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:34:15 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:34:17.158 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:17 smithi022 ceph-mon[31360]: pgmap v491: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:17 smithi022 ceph-mon[40250]: pgmap v491: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:17 smithi122 ceph-mon[32255]: pgmap v491: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:17.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:17.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:17.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:17.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:17] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:34:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2312028320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2312028320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2312028320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:19 smithi022 ceph-mon[31360]: pgmap v492: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:19 smithi022 ceph-mon[40250]: pgmap v492: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:19 smithi122 ceph-mon[32255]: pgmap v492: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:34:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:21 smithi022 ceph-mon[31360]: pgmap v493: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:21 smithi022 ceph-mon[40250]: pgmap v493: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:21 smithi122 ceph-mon[32255]: pgmap v493: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:22.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:22.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:22.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:22.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:23 smithi022 ceph-mon[31360]: pgmap v494: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2428146648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:23.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:23 smithi022 ceph-mon[40250]: pgmap v494: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2428146648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:23 smithi122 ceph-mon[32255]: pgmap v494: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2428146648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:25 smithi022 ceph-mon[31360]: pgmap v495: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:25 smithi022 ceph-mon[40250]: pgmap v495: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:25 smithi122 ceph-mon[32255]: pgmap v495: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:26.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:34:25 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:34:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:27 smithi022 ceph-mon[31360]: pgmap v496: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:27 smithi022 ceph-mon[40250]: pgmap v496: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:27 smithi122 ceph-mon[32255]: pgmap v496: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:27] "GET /metrics HTTP/1.1" 200 201990 "" "Prometheus/2.18.1" 2021-12-15T17:34:27.894 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:29 smithi022 ceph-mon[31360]: pgmap v497: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3973146554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:29 smithi022 ceph-mon[40250]: pgmap v497: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3973146554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:29 smithi122 ceph-mon[32255]: pgmap v497: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3973146554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:34:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:31 smithi022 ceph-mon[31360]: pgmap v498: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:31 smithi022 ceph-mon[40250]: pgmap v498: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:31 smithi122 ceph-mon[32255]: pgmap v498: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:33.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:33.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:33 smithi022 ceph-mon[31360]: pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:33 smithi022 ceph-mon[40250]: pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:33 smithi122 ceph-mon[32255]: pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:33.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:33.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:33.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3092669315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3092669315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3092669315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:35 smithi122 ceph-mon[32255]: pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:35 smithi022 ceph-mon[31360]: pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:35 smithi022 ceph-mon[40250]: pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:36.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:34:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:34:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:37 smithi022 ceph-mon[31360]: pgmap v501: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:37 smithi022 ceph-mon[40250]: pgmap v501: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:37 smithi122 ceph-mon[32255]: pgmap v501: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:37] "GET /metrics HTTP/1.1" 200 201981 "" "Prometheus/2.18.1" 2021-12-15T17:34:38.641 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:39.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:39.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:39.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:39 smithi022 ceph-mon[31360]: pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1401237850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:39 smithi022 ceph-mon[40250]: pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1401237850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:39 smithi122 ceph-mon[32255]: pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1401237850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:34:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:41 smithi122 ceph-mon[32255]: pgmap v503: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:41 smithi022 ceph-mon[31360]: pgmap v503: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:41 smithi022 ceph-mon[40250]: pgmap v503: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:43 smithi122 ceph-mon[32255]: pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:43 smithi022 ceph-mon[31360]: pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:43 smithi022 ceph-mon[40250]: pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:44.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:44.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:44.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:44.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:45 smithi122 ceph-mon[32255]: pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1561414675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:45 smithi022 ceph-mon[31360]: pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1561414675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:45 smithi022 ceph-mon[40250]: pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1561414675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:34:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:34:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:47 smithi022 ceph-mon[31360]: pgmap v506: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:47 smithi022 ceph-mon[40250]: pgmap v506: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:47 smithi122 ceph-mon[32255]: pgmap v506: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:47] "GET /metrics HTTP/1.1" 200 201981 "" "Prometheus/2.18.1" 2021-12-15T17:34:49.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:49.382 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:49 smithi022 ceph-mon[40250]: pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:49.383 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:49 smithi022 ceph-mon[31360]: pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:49 smithi122 ceph-mon[32255]: pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:49.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:49.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:49.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:34:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/150917981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/150917981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/150917981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:51 smithi122 ceph-mon[32255]: pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:51 smithi022 ceph-mon[31360]: pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:51 smithi022 ceph-mon[40250]: pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:53 smithi122 ceph-mon[32255]: pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:53 smithi022 ceph-mon[31360]: pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:53 smithi022 ceph-mon[40250]: pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:54.753 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:34:55.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:34:55.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:34:55.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:34:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:55 smithi122 ceph-mon[32255]: pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:55 smithi022 ceph-mon[31360]: pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:55 smithi022 ceph-mon[40250]: pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/29667192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:34:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:34:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/29667192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/29667192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:34:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:57 smithi022 ceph-mon[31360]: pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:57 smithi022 ceph-mon[40250]: pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:57 smithi122 ceph-mon[32255]: pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:34:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:34:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:34:57] "GET /metrics HTTP/1.1" 200 201994 "" "Prometheus/2.18.1" 2021-12-15T17:34:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:34:59 smithi122 ceph-mon[32255]: pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:59.508 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:34:59 smithi022 ceph-mon[31360]: pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:34:59.508 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:34:59 smithi022 ceph-mon[40250]: pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:00.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:35:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:35:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:35:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:35:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:35:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:35:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:35:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:35:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:35:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:35:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:35:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:35:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:35:00.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:00.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:00.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:01 smithi122 ceph-mon[32255]: pgmap v513: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1071352285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:01 smithi022 ceph-mon[31360]: pgmap v513: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1071352285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:01 smithi022 ceph-mon[40250]: pgmap v513: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1071352285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:03 smithi122 ceph-mon[32255]: pgmap v514: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:03 smithi022 ceph-mon[31360]: pgmap v514: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:03 smithi022 ceph-mon[40250]: pgmap v514: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:35:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:35:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:35:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:35:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:35:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:35:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:05 smithi122 ceph-mon[32255]: pgmap v515: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:05.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:05.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:05 smithi022 ceph-mon[31360]: pgmap v515: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:05.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:05 smithi022 ceph-mon[40250]: pgmap v515: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:05.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:05.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:05.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2096849335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2096849335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2096849335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:35:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:35:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:07 smithi022 ceph-mon[31360]: pgmap v516: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:07 smithi022 ceph-mon[40250]: pgmap v516: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:07 smithi122 ceph-mon[32255]: pgmap v516: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:07] "GET /metrics HTTP/1.1" 200 201983 "" "Prometheus/2.18.1" 2021-12-15T17:35:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:09 smithi122 ceph-mon[32255]: pgmap v517: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:09 smithi022 ceph-mon[31360]: pgmap v517: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:09 smithi022 ceph-mon[40250]: pgmap v517: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:35:10.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:11.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:11.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:11.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:11.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:11 smithi022 ceph-mon[31360]: pgmap v518: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:11.231 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:11 smithi022 ceph-mon[40250]: pgmap v518: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:11 smithi122 ceph-mon[32255]: pgmap v518: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4000479563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4000479563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4000479563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:13.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:13 smithi122 ceph-mon[32255]: pgmap v519: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:13 smithi022 ceph-mon[31360]: pgmap v519: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:13 smithi022 ceph-mon[40250]: pgmap v519: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:15 smithi122 ceph-mon[32255]: pgmap v520: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:15 smithi022 ceph-mon[31360]: pgmap v520: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:15 smithi022 ceph-mon[40250]: pgmap v520: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:16.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:16.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:35:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:35:16.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:16.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:16.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:17 smithi022 ceph-mon[31360]: pgmap v521: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/323600959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:17 smithi022 ceph-mon[40250]: pgmap v521: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/323600959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:17 smithi122 ceph-mon[32255]: pgmap v521: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/323600959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:17] "GET /metrics HTTP/1.1" 200 201983 "" "Prometheus/2.18.1" 2021-12-15T17:35:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:19 smithi122 ceph-mon[32255]: pgmap v522: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:19 smithi022 ceph-mon[31360]: pgmap v522: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:19 smithi022 ceph-mon[40250]: pgmap v522: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:35:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:21 smithi122 ceph-mon[32255]: pgmap v523: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:21 smithi022 ceph-mon[31360]: pgmap v523: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:21 smithi022 ceph-mon[40250]: pgmap v523: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:21.601 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:21.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:21.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:21.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2028839569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2028839569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:22.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2028839569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:23 smithi122 ceph-mon[32255]: pgmap v524: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:23 smithi022 ceph-mon[31360]: pgmap v524: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:23 smithi022 ceph-mon[40250]: pgmap v524: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:25 smithi122 ceph-mon[32255]: pgmap v525: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:25 smithi022 ceph-mon[31360]: pgmap v525: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:25 smithi022 ceph-mon[40250]: pgmap v525: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:26.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:35:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:35:26.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:27.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:27 smithi022 ceph-mon[31360]: pgmap v526: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:27 smithi022 ceph-mon[40250]: pgmap v526: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:27.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:27.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:27.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:27 smithi122 ceph-mon[32255]: pgmap v526: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:27] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:35:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/826138125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/826138125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/826138125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:29 smithi122 ceph-mon[32255]: pgmap v527: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:29 smithi022 ceph-mon[31360]: pgmap v527: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:29 smithi022 ceph-mon[40250]: pgmap v527: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:35:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:31 smithi122 ceph-mon[32255]: pgmap v528: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:31 smithi022 ceph-mon[31360]: pgmap v528: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:31 smithi022 ceph-mon[40250]: pgmap v528: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:32.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:32.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:32.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:33 smithi122 ceph-mon[32255]: pgmap v529: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2127374439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:33 smithi022 ceph-mon[31360]: pgmap v529: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2127374439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:33 smithi022 ceph-mon[40250]: pgmap v529: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2127374439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:35 smithi122 ceph-mon[32255]: pgmap v530: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:35 smithi022 ceph-mon[31360]: pgmap v530: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:35 smithi022 ceph-mon[40250]: pgmap v530: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:36.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:35:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:35:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:37 smithi022 ceph-mon[31360]: pgmap v531: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:37 smithi022 ceph-mon[40250]: pgmap v531: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:37 smithi122 ceph-mon[32255]: pgmap v531: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:37.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:37.716 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:37] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:35:38.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:38.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:38.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/900680255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/900680255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/900680255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:39 smithi122 ceph-mon[32255]: pgmap v532: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:39 smithi022 ceph-mon[31360]: pgmap v532: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:39 smithi022 ceph-mon[40250]: pgmap v532: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:35:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:41 smithi122 ceph-mon[32255]: pgmap v533: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:41 smithi022 ceph-mon[31360]: pgmap v533: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:41 smithi022 ceph-mon[40250]: pgmap v533: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:43.082 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:43 smithi122 ceph-mon[32255]: pgmap v534: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:43.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:43.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:43.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:43.457 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:43 smithi022 ceph-mon[31360]: pgmap v534: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:43.457 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:43 smithi022 ceph-mon[40250]: pgmap v534: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1560968561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1560968561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1560968561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:45 smithi122 ceph-mon[32255]: pgmap v535: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:45 smithi022 ceph-mon[31360]: pgmap v535: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:45 smithi022 ceph-mon[40250]: pgmap v535: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:46.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:35:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:35:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:47 smithi122 ceph-mon[32255]: pgmap v536: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:47] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:35:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:47 smithi022 ceph-mon[31360]: pgmap v536: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:47 smithi022 ceph-mon[40250]: pgmap v536: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:48.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:48.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:48.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:48.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:49 smithi122 ceph-mon[32255]: pgmap v537: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2055262984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:49 smithi022 ceph-mon[31360]: pgmap v537: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2055262984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:49 smithi022 ceph-mon[40250]: pgmap v537: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2055262984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:35:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:51 smithi122 ceph-mon[32255]: pgmap v538: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:51 smithi022 ceph-mon[31360]: pgmap v538: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:51 smithi022 ceph-mon[40250]: pgmap v538: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:53 smithi122 ceph-mon[32255]: pgmap v539: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:53 smithi022 ceph-mon[31360]: pgmap v539: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:53 smithi022 ceph-mon[40250]: pgmap v539: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:53.825 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:54.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:54.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:54.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:35:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:55 smithi122 ceph-mon[32255]: pgmap v540: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/241524862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:55 smithi022 ceph-mon[31360]: pgmap v540: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/241524862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:55 smithi022 ceph-mon[40250]: pgmap v540: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/241524862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:35:56.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:35:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:35:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:57 smithi122 ceph-mon[32255]: pgmap v541: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:35:57] "GET /metrics HTTP/1.1" 200 201995 "" "Prometheus/2.18.1" 2021-12-15T17:35:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:57 smithi022 ceph-mon[31360]: pgmap v541: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:57 smithi022 ceph-mon[40250]: pgmap v541: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:35:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:35:59.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:35:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:35:59 smithi122 ceph-mon[32255]: pgmap v542: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:35:59 smithi022 ceph-mon[31360]: pgmap v542: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:35:59 smithi022 ceph-mon[40250]: pgmap v542: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:35:59.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:35:59.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:35:59.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/888616855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:36:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:36:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:36:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/888616855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:36:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:36:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:00 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:36:00.360 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:36:00.360 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/888616855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:00.360 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:36:00.361 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:36:00.361 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:00 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:36:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:01 smithi122 ceph-mon[32255]: pgmap v543: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:36:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:01 smithi022 ceph-mon[31360]: pgmap v543: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:36:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:01 smithi022 ceph-mon[40250]: pgmap v543: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:36:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:03 smithi122 ceph-mon[32255]: pgmap v544: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:03 smithi022 ceph-mon[31360]: pgmap v544: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:03 smithi022 ceph-mon[40250]: pgmap v544: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:36:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:36:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:36:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:36:04.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:36:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:36:04.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:04.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:04.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:05 smithi022 ceph-mon[31360]: pgmap v545: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1980028887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:05 smithi022 ceph-mon[40250]: pgmap v545: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1980028887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:05 smithi122 ceph-mon[32255]: pgmap v545: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1980028887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:06.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:36:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:36:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:07] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:36:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:07 smithi022 ceph-mon[31360]: pgmap v546: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:07 smithi022 ceph-mon[40250]: pgmap v546: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:07 smithi122 ceph-mon[32255]: pgmap v546: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:09 smithi022 ceph-mon[31360]: pgmap v547: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:09 smithi022 ceph-mon[40250]: pgmap v547: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:09 smithi122 ceph-mon[32255]: pgmap v547: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:09.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:10.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:10.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:10.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:36:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:11 smithi022 ceph-mon[31360]: pgmap v548: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2409360124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:11 smithi022 ceph-mon[40250]: pgmap v548: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2409360124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:11 smithi122 ceph-mon[32255]: pgmap v548: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2409360124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:13 smithi022 ceph-mon[31360]: pgmap v549: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:13 smithi022 ceph-mon[40250]: pgmap v549: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:13 smithi122 ceph-mon[32255]: pgmap v549: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:15.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:15.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:15 smithi022 ceph-mon[31360]: pgmap v550: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:15 smithi022 ceph-mon[40250]: pgmap v550: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:15 smithi122 ceph-mon[32255]: pgmap v550: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:16.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:36:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:36:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/381960969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/381960969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/381960969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:17] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:36:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:17 smithi022 ceph-mon[31360]: pgmap v551: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:17 smithi022 ceph-mon[40250]: pgmap v551: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:17 smithi122 ceph-mon[32255]: pgmap v551: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:19 smithi022 ceph-mon[31360]: pgmap v552: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:19 smithi022 ceph-mon[40250]: pgmap v552: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:19 smithi122 ceph-mon[32255]: pgmap v552: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:36:20.674 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:21.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:21.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:21.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:21 smithi022 ceph-mon[31360]: pgmap v553: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/166563720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:21 smithi022 ceph-mon[40250]: pgmap v553: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/166563720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:21 smithi122 ceph-mon[32255]: pgmap v553: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/166563720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:23 smithi022 ceph-mon[31360]: pgmap v554: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:23 smithi022 ceph-mon[40250]: pgmap v554: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:23 smithi122 ceph-mon[32255]: pgmap v554: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:25 smithi022 ceph-mon[31360]: pgmap v555: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:25 smithi022 ceph-mon[40250]: pgmap v555: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:25 smithi122 ceph-mon[32255]: pgmap v555: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:26.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:26.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:26.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:26.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:26.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:36:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:36:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:27] "GET /metrics HTTP/1.1" 200 201988 "" "Prometheus/2.18.1" 2021-12-15T17:36:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:27 smithi022 ceph-mon[31360]: pgmap v556: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2261397773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:27 smithi022 ceph-mon[40250]: pgmap v556: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2261397773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:27 smithi122 ceph-mon[32255]: pgmap v556: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2261397773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:29 smithi022 ceph-mon[31360]: pgmap v557: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:29 smithi022 ceph-mon[40250]: pgmap v557: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:29 smithi122 ceph-mon[32255]: pgmap v557: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:36:31.417 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:31 smithi022 ceph-mon[31360]: pgmap v558: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:31 smithi022 ceph-mon[40250]: pgmap v558: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:31 smithi122 ceph-mon[32255]: pgmap v558: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:31.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:31.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:31.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2445054835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2445054835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2445054835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:33 smithi022 ceph-mon[31360]: pgmap v559: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:33 smithi022 ceph-mon[40250]: pgmap v559: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:33 smithi122 ceph-mon[32255]: pgmap v559: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:35 smithi022 ceph-mon[31360]: pgmap v560: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:35 smithi022 ceph-mon[40250]: pgmap v560: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:35 smithi122 ceph-mon[32255]: pgmap v560: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:36:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:36:36.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:37.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:37.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:37.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:37] "GET /metrics HTTP/1.1" 200 201996 "" "Prometheus/2.18.1" 2021-12-15T17:36:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:37 smithi022 ceph-mon[31360]: pgmap v561: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3179139595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:37 smithi022 ceph-mon[40250]: pgmap v561: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3179139595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:37 smithi122 ceph-mon[32255]: pgmap v561: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3179139595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:39 smithi022 ceph-mon[31360]: pgmap v562: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:39 smithi022 ceph-mon[40250]: pgmap v562: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:39 smithi122 ceph-mon[32255]: pgmap v562: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:36:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:41 smithi022 ceph-mon[31360]: pgmap v563: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:41 smithi022 ceph-mon[40250]: pgmap v563: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:41 smithi122 ceph-mon[32255]: pgmap v563: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:42.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:42.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:42.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:42.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:43 smithi022 ceph-mon[31360]: pgmap v564: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2381721181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:43 smithi022 ceph-mon[40250]: pgmap v564: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2381721181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:43 smithi122 ceph-mon[32255]: pgmap v564: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2381721181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:45.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:45 smithi022 ceph-mon[31360]: pgmap v565: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:45 smithi022 ceph-mon[40250]: pgmap v565: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:45 smithi122 ceph-mon[32255]: pgmap v565: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:36:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:36:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:47] "GET /metrics HTTP/1.1" 200 201996 "" "Prometheus/2.18.1" 2021-12-15T17:36:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:47 smithi022 ceph-mon[31360]: pgmap v566: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:47 smithi022 ceph-mon[40250]: pgmap v566: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:47.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:47 smithi122 ceph-mon[32255]: pgmap v566: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:47.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:47.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:47.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1554694935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1554694935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1554694935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:49 smithi022 ceph-mon[31360]: pgmap v567: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:49 smithi022 ceph-mon[40250]: pgmap v567: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:49 smithi122 ceph-mon[32255]: pgmap v567: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:36:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:51 smithi022 ceph-mon[31360]: pgmap v568: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:51 smithi022 ceph-mon[40250]: pgmap v568: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:51 smithi122 ceph-mon[32255]: pgmap v568: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:52.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:53 smithi022 ceph-mon[31360]: pgmap v569: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:53 smithi022 ceph-mon[40250]: pgmap v569: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:53 smithi122 ceph-mon[32255]: pgmap v569: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2780342708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2780342708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2780342708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:55 smithi022 ceph-mon[31360]: pgmap v570: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:55 smithi022 ceph-mon[40250]: pgmap v570: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:55 smithi122 ceph-mon[32255]: pgmap v570: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:56.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:36:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:36:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:36:57] "GET /metrics HTTP/1.1" 200 201996 "" "Prometheus/2.18.1" 2021-12-15T17:36:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:57 smithi022 ceph-mon[31360]: pgmap v571: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:57 smithi022 ceph-mon[40250]: pgmap v571: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:57 smithi122 ceph-mon[32255]: pgmap v571: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:36:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:36:58.274 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:36:58.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:36:58.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:36:58.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:36:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:59 smithi022 ceph-mon[31360]: pgmap v572: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:36:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2949682303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:59 smithi022 ceph-mon[40250]: pgmap v572: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:36:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2949682303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:36:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:59 smithi122 ceph-mon[32255]: pgmap v572: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:36:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:36:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2949682303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:00.426 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:37:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[31360]: pgmap v573: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:37:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:37:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:37:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:37:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[40250]: pgmap v573: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:37:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:37:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:37:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:37:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:01 smithi122 ceph-mon[32255]: pgmap v573: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:37:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:37:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:37:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:37:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:03 smithi022 ceph-mon[31360]: pgmap v574: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:37:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:37:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:03 smithi022 ceph-mon[40250]: pgmap v574: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:37:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:37:03.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:03 smithi122 ceph-mon[32255]: pgmap v574: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:37:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:37:04.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:04.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:04.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4147016326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4147016326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4147016326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:05 smithi122 ceph-mon[32255]: pgmap v575: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:05 smithi022 ceph-mon[31360]: pgmap v575: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:05 smithi022 ceph-mon[40250]: pgmap v575: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:06.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:37:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:37:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:07 smithi122 ceph-mon[32255]: pgmap v576: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:07] "GET /metrics HTTP/1.1" 200 201996 "" "Prometheus/2.18.1" 2021-12-15T17:37:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:07 smithi022 ceph-mon[31360]: pgmap v576: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:07 smithi022 ceph-mon[40250]: pgmap v576: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:09.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:09.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:09.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:09.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:09 smithi122 ceph-mon[32255]: pgmap v577: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:09 smithi022 ceph-mon[31360]: pgmap v577: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:09 smithi022 ceph-mon[40250]: pgmap v577: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:37:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3392543528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3392543528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3392543528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:11 smithi122 ceph-mon[32255]: pgmap v578: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:11 smithi022 ceph-mon[31360]: pgmap v578: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:11 smithi022 ceph-mon[40250]: pgmap v578: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:13 smithi122 ceph-mon[32255]: pgmap v579: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:13 smithi022 ceph-mon[31360]: pgmap v579: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:13 smithi022 ceph-mon[40250]: pgmap v579: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:14.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:14.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:14.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:14.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:15 smithi122 ceph-mon[32255]: pgmap v580: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4045110596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:15 smithi022 ceph-mon[31360]: pgmap v580: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4045110596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:15 smithi022 ceph-mon[40250]: pgmap v580: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4045110596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:16.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:37:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:37:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:17 smithi122 ceph-mon[32255]: pgmap v581: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:17 smithi022 ceph-mon[31360]: pgmap v581: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:17.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:17 smithi022 ceph-mon[40250]: pgmap v581: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:17] "GET /metrics HTTP/1.1" 200 201996 "" "Prometheus/2.18.1" 2021-12-15T17:37:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:19 smithi122 ceph-mon[32255]: pgmap v582: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:19.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:19.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:19 smithi022 ceph-mon[31360]: pgmap v582: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:19.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:19 smithi022 ceph-mon[40250]: pgmap v582: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:20.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:20.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:20.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:37:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:21 smithi122 ceph-mon[32255]: pgmap v583: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3258556494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:21 smithi022 ceph-mon[31360]: pgmap v583: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3258556494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:21 smithi022 ceph-mon[40250]: pgmap v583: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3258556494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:23 smithi122 ceph-mon[32255]: pgmap v584: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:23 smithi022 ceph-mon[31360]: pgmap v584: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:23 smithi022 ceph-mon[40250]: pgmap v584: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:25.122 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:25 smithi122 ceph-mon[32255]: pgmap v585: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:25.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:25.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:25.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:25.490 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:25 smithi022 ceph-mon[40250]: pgmap v585: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:25.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:25 smithi022 ceph-mon[31360]: pgmap v585: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/168278460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:26.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:37:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:37:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/168278460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/168278460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:27 smithi022 ceph-mon[31360]: pgmap v586: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:27 smithi022 ceph-mon[40250]: pgmap v586: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:27 smithi122 ceph-mon[32255]: pgmap v586: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:27] "GET /metrics HTTP/1.1" 200 201991 "" "Prometheus/2.18.1" 2021-12-15T17:37:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:29 smithi122 ceph-mon[32255]: pgmap v587: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:29 smithi022 ceph-mon[31360]: pgmap v587: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:29 smithi022 ceph-mon[40250]: pgmap v587: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:37:30.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:30.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:30.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:30.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:31 smithi022 ceph-mon[31360]: pgmap v588: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1252132157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:31 smithi022 ceph-mon[40250]: pgmap v588: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1252132157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:31 smithi122 ceph-mon[32255]: pgmap v588: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1252132157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:33 smithi122 ceph-mon[32255]: pgmap v589: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:33 smithi022 ceph-mon[31360]: pgmap v589: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:33 smithi022 ceph-mon[40250]: pgmap v589: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:35 smithi122 ceph-mon[32255]: pgmap v590: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:35 smithi022 ceph-mon[31360]: pgmap v590: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:35 smithi022 ceph-mon[40250]: pgmap v590: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:35.860 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:36.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:36.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:36.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:36.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:37:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:37:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:37 smithi022 ceph-mon[31360]: pgmap v591: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2249314802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:37 smithi022 ceph-mon[40250]: pgmap v591: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2249314802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:37 smithi122 ceph-mon[32255]: pgmap v591: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2249314802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:37.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:37] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:37:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:39 smithi122 ceph-mon[32255]: pgmap v592: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:39 smithi022 ceph-mon[31360]: pgmap v592: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:39 smithi022 ceph-mon[40250]: pgmap v592: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:37:41.229 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:41 smithi122 ceph-mon[32255]: pgmap v593: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:41 smithi022 ceph-mon[31360]: pgmap v593: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:41 smithi022 ceph-mon[40250]: pgmap v593: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:41.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:41.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:41.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:42.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3552829599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:42.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3552829599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3552829599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:43 smithi122 ceph-mon[32255]: pgmap v594: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:43 smithi022 ceph-mon[31360]: pgmap v594: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:43 smithi022 ceph-mon[40250]: pgmap v594: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:45 smithi122 ceph-mon[32255]: pgmap v595: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:45 smithi022 ceph-mon[31360]: pgmap v595: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:45 smithi022 ceph-mon[40250]: pgmap v595: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:46.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:37:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:37:46.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:46.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:46.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:46.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:47 smithi022 ceph-mon[31360]: pgmap v596: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2210519072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:47 smithi022 ceph-mon[40250]: pgmap v596: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2210519072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:47 smithi122 ceph-mon[32255]: pgmap v596: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2210519072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:47] "GET /metrics HTTP/1.1" 200 201989 "" "Prometheus/2.18.1" 2021-12-15T17:37:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:49 smithi122 ceph-mon[32255]: pgmap v597: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:49 smithi022 ceph-mon[31360]: pgmap v597: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:49 smithi022 ceph-mon[40250]: pgmap v597: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:37:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:51 smithi122 ceph-mon[32255]: pgmap v598: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:51 smithi022 ceph-mon[31360]: pgmap v598: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:51 smithi022 ceph-mon[40250]: pgmap v598: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:37:51.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:52.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:52.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:52.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:53 smithi122 ceph-mon[32255]: pgmap v599: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3154380615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:53 smithi022 ceph-mon[31360]: pgmap v599: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3154380615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:53 smithi022 ceph-mon[40250]: pgmap v599: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3154380615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:55 smithi122 ceph-mon[32255]: pgmap v600: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:55 smithi022 ceph-mon[31360]: pgmap v600: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:55 smithi022 ceph-mon[40250]: pgmap v600: 132 pgs: 132 active+clean; 582 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:37:56.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:37:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:37:57.387 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:37:57.387 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:57 smithi022 ceph-mon[31360]: pgmap v601: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:37:57.388 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:57.388 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:57 smithi022 ceph-mon[40250]: pgmap v601: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:37:57.388 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:57 smithi122 ceph-mon[32255]: pgmap v601: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:37:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:37:57.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:37:57.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:37:57.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:37:57.758 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:37:57] "GET /metrics HTTP/1.1" 200 201987 "" "Prometheus/2.18.1" 2021-12-15T17:37:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/260478438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/260478438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/260478438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:37:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:37:59 smithi122 ceph-mon[32255]: pgmap v602: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:37:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:37:59 smithi022 ceph-mon[31360]: pgmap v602: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:37:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:37:59 smithi022 ceph-mon[40250]: pgmap v602: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:38:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:01 smithi122 ceph-mon[32255]: pgmap v603: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:38:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:38:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:01 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:38:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[31360]: pgmap v603: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:38:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:38:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:38:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[40250]: pgmap v603: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:38:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:38:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:01 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:38:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:38:02.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:02.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:38:02.761 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:38:03.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:03.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:03.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[31360]: pgmap v604: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3394311792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:38:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:38:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[40250]: pgmap v604: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3394311792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:38:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:38:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:03 smithi122 ceph-mon[32255]: pgmap v604: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3394311792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:38:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:38:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:05 smithi122 ceph-mon[32255]: pgmap v605: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:05 smithi022 ceph-mon[31360]: pgmap v605: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:05 smithi022 ceph-mon[40250]: pgmap v605: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:06.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:38:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:38:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:07 smithi122 ceph-mon[32255]: pgmap v606: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:07] "GET /metrics HTTP/1.1" 200 202012 "" "Prometheus/2.18.1" 2021-12-15T17:38:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:07 smithi022 ceph-mon[40250]: pgmap v606: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:07 smithi022 ceph-mon[31360]: pgmap v606: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:38:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:08.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:08.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:08.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:08.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:09 smithi122 ceph-mon[32255]: pgmap v607: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1187731785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:09 smithi022 ceph-mon[40250]: pgmap v607: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1187731785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:09 smithi022 ceph-mon[31360]: pgmap v607: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1187731785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:38:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:11 smithi122 ceph-mon[32255]: pgmap v608: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:11 smithi022 ceph-mon[31360]: pgmap v608: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:11 smithi022 ceph-mon[40250]: pgmap v608: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:13.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:13 smithi122 ceph-mon[32255]: pgmap v609: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:13 smithi022 ceph-mon[31360]: pgmap v609: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:13 smithi022 ceph-mon[40250]: pgmap v609: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:13.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:13.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:13.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/768157227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/768157227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/768157227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:15 smithi122 ceph-mon[32255]: pgmap v610: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:15 smithi022 ceph-mon[31360]: pgmap v610: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:15 smithi022 ceph-mon[40250]: pgmap v610: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:16.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:38:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:38:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:17 smithi122 ceph-mon[32255]: pgmap v611: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:17] "GET /metrics HTTP/1.1" 200 202012 "" "Prometheus/2.18.1" 2021-12-15T17:38:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:17 smithi022 ceph-mon[31360]: pgmap v611: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:17 smithi022 ceph-mon[40250]: pgmap v611: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:18.878 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:19.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:19.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:19.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:19 smithi022 ceph-mon[31360]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1907055858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:19 smithi022 ceph-mon[40250]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1907055858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:19 smithi122 ceph-mon[32255]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1907055858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:38:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:21 smithi122 ceph-mon[32255]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:21 smithi022 ceph-mon[31360]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:21 smithi022 ceph-mon[40250]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:23 smithi122 ceph-mon[32255]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:23 smithi022 ceph-mon[31360]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:23 smithi022 ceph-mon[40250]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:24.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:24.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:24.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:24.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:25 smithi122 ceph-mon[32255]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:25.762 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/768082990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:25 smithi022 ceph-mon[31360]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/768082990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:25 smithi022 ceph-mon[40250]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/768082990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:26.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:38:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:38:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:27 smithi122 ceph-mon[32255]: pgmap v616: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:27 smithi022 ceph-mon[31360]: pgmap v616: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:27] "GET /metrics HTTP/1.1" 200 202016 "" "Prometheus/2.18.1" 2021-12-15T17:38:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:27 smithi022 ceph-mon[40250]: pgmap v616: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:29.616 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:29 smithi022 ceph-mon[31360]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:29.617 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:29 smithi022 ceph-mon[40250]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:29 smithi122 ceph-mon[32255]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:29.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:29.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:29.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/741781645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:38:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/741781645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/741781645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:31 smithi122 ceph-mon[32255]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:31 smithi022 ceph-mon[31360]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:31 smithi022 ceph-mon[40250]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:33 smithi122 ceph-mon[32255]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:33 smithi022 ceph-mon[31360]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:33 smithi022 ceph-mon[40250]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:34.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:35.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:35.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:35.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:35 smithi122 ceph-mon[32255]: pgmap v620: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3857696897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:35 smithi022 ceph-mon[31360]: pgmap v620: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3857696897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:35 smithi022 ceph-mon[40250]: pgmap v620: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3857696897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:36.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:38:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:38:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:37 smithi122 ceph-mon[32255]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:37 smithi022 ceph-mon[31360]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:37] "GET /metrics HTTP/1.1" 200 202018 "" "Prometheus/2.18.1" 2021-12-15T17:38:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:37 smithi022 ceph-mon[40250]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:39 smithi122 ceph-mon[32255]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:39 smithi022 ceph-mon[31360]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:39 smithi022 ceph-mon[40250]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:40.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:38:40.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:40.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:40.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:41 smithi122 ceph-mon[32255]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1402880470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:41 smithi022 ceph-mon[31360]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1402880470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:41 smithi022 ceph-mon[40250]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1402880470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:43 smithi122 ceph-mon[32255]: pgmap v624: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:43 smithi022 ceph-mon[31360]: pgmap v624: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:43 smithi022 ceph-mon[40250]: pgmap v624: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:44 smithi122 ceph-mon[32255]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:44 smithi022 ceph-mon[31360]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:44.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:44 smithi022 ceph-mon[40250]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:45.727 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:46.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:46.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:46.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:46.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:38:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:38:47.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:47 smithi022 ceph-mon[40250]: pgmap v626: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:47.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3967246812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:47.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:47 smithi022 ceph-mon[31360]: pgmap v626: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3967246812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:47 smithi122 ceph-mon[32255]: pgmap v626: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3967246812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:47.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:47] "GET /metrics HTTP/1.1" 200 202018 "" "Prometheus/2.18.1" 2021-12-15T17:38:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:49 smithi122 ceph-mon[32255]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:49 smithi022 ceph-mon[31360]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:49 smithi022 ceph-mon[40250]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:38:51.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:51 smithi122 ceph-mon[32255]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:51.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:51.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:51.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:51.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:51 smithi022 ceph-mon[31360]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:51.467 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:51 smithi022 ceph-mon[40250]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:52.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2973907015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2973907015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2973907015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:53 smithi122 ceph-mon[32255]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:53 smithi022 ceph-mon[31360]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:53 smithi022 ceph-mon[40250]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:55 smithi122 ceph-mon[32255]: pgmap v630: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:55 smithi022 ceph-mon[31360]: pgmap v630: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:55 smithi022 ceph-mon[40250]: pgmap v630: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:56.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:38:56.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:38:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:38:56.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:38:56.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:38:56.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:38:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:57 smithi022 ceph-mon[31360]: pgmap v631: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2263285487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:57 smithi022 ceph-mon[40250]: pgmap v631: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2263285487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:57 smithi122 ceph-mon[32255]: pgmap v631: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:38:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:38:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2263285487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:38:57.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:38:57] "GET /metrics HTTP/1.1" 200 202017 "" "Prometheus/2.18.1" 2021-12-15T17:38:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:38:59 smithi122 ceph-mon[32255]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:38:59 smithi022 ceph-mon[31360]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:38:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:38:59 smithi022 ceph-mon[40250]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:39:01.337 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:01 smithi122 ceph-mon[32255]: pgmap v633: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:01.349 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:01 smithi022 ceph-mon[40250]: pgmap v633: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:01.350 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:01 smithi022 ceph-mon[31360]: pgmap v633: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:01.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:02.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:02.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:02.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:39:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:39:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:39:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:39:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:39:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:39:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:39:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:39:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:39:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:39:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:39:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:39:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:03 smithi122 ceph-mon[32255]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4173024705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:03 smithi022 ceph-mon[31360]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4173024705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:03 smithi022 ceph-mon[40250]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4173024705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:39:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:39:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:39:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:39:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:39:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:39:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:05 smithi122 ceph-mon[32255]: pgmap v635: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:05 smithi022 ceph-mon[31360]: pgmap v635: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:05 smithi022 ceph-mon[40250]: pgmap v635: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:06.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:39:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:39:07.194 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:07 smithi022 ceph-mon[31360]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:07 smithi022 ceph-mon[40250]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:07 smithi122 ceph-mon[32255]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:07.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:07.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:07.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:07] "GET /metrics HTTP/1.1" 200 202016 "" "Prometheus/2.18.1" 2021-12-15T17:39:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3136623447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3136623447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3136623447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:09 smithi122 ceph-mon[32255]: pgmap v637: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:09 smithi022 ceph-mon[31360]: pgmap v637: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:09 smithi022 ceph-mon[40250]: pgmap v637: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:39:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:11 smithi122 ceph-mon[32255]: pgmap v638: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:11.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:11 smithi022 ceph-mon[31360]: pgmap v638: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:11 smithi022 ceph-mon[40250]: pgmap v638: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:12.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:12.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:12.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:12.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:13 smithi022 ceph-mon[31360]: pgmap v639: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/460631443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:13 smithi022 ceph-mon[40250]: pgmap v639: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/460631443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:13.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:13 smithi122 ceph-mon[32255]: pgmap v639: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/460631443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:15 smithi122 ceph-mon[32255]: pgmap v640: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:15 smithi022 ceph-mon[31360]: pgmap v640: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:15 smithi022 ceph-mon[40250]: pgmap v640: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:16.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:39:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:39:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:17 smithi022 ceph-mon[31360]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:17 smithi022 ceph-mon[40250]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:17 smithi122 ceph-mon[32255]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:17.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:17] "GET /metrics HTTP/1.1" 200 202016 "" "Prometheus/2.18.1" 2021-12-15T17:39:17.936 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:18.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:19.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:19 smithi122 ceph-mon[32255]: pgmap v642: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:19.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3624498264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:19.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:19 smithi022 ceph-mon[31360]: pgmap v642: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3624498264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:19 smithi022 ceph-mon[40250]: pgmap v642: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3624498264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:39:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:21 smithi122 ceph-mon[32255]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:21 smithi022 ceph-mon[31360]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:21 smithi022 ceph-mon[40250]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:23.308 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:23 smithi122 ceph-mon[32255]: pgmap v644: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:23 smithi022 ceph-mon[31360]: pgmap v644: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:23 smithi022 ceph-mon[40250]: pgmap v644: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:23.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:23.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:23.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2948440669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2948440669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2948440669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:25 smithi122 ceph-mon[32255]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:25 smithi022 ceph-mon[31360]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:25 smithi022 ceph-mon[40250]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:26.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:39:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:39:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:27 smithi022 ceph-mon[31360]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:27.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:27 smithi022 ceph-mon[40250]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:27 smithi122 ceph-mon[32255]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:27.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:27.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:27] "GET /metrics HTTP/1.1" 200 202019 "" "Prometheus/2.18.1" 2021-12-15T17:39:28.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:29.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:29.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:29.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:29 smithi122 ceph-mon[32255]: pgmap v647: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4130289107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:29 smithi022 ceph-mon[31360]: pgmap v647: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4130289107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:29 smithi022 ceph-mon[40250]: pgmap v647: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4130289107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:39:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:31 smithi122 ceph-mon[32255]: pgmap v648: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:31 smithi022 ceph-mon[31360]: pgmap v648: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:31 smithi022 ceph-mon[40250]: pgmap v648: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:33 smithi122 ceph-mon[32255]: pgmap v649: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:33 smithi022 ceph-mon[31360]: pgmap v649: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:33 smithi022 ceph-mon[40250]: pgmap v649: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:34.050 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:34.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:35 smithi122 ceph-mon[32255]: pgmap v650: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3220548750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:35 smithi022 ceph-mon[31360]: pgmap v650: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3220548750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:35 smithi022 ceph-mon[40250]: pgmap v650: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3220548750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:36.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:39:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:39:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:37 smithi022 ceph-mon[31360]: pgmap v651: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:37 smithi022 ceph-mon[40250]: pgmap v651: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:37 smithi122 ceph-mon[32255]: pgmap v651: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:37] "GET /metrics HTTP/1.1" 200 202017 "" "Prometheus/2.18.1" 2021-12-15T17:39:39.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:39.423 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:39 smithi022 ceph-mon[31360]: pgmap v652: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:39.424 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:39 smithi022 ceph-mon[40250]: pgmap v652: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:39 smithi122 ceph-mon[32255]: pgmap v652: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:39.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:39.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:39.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:39:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3108828904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:40.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3108828904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3108828904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:41 smithi122 ceph-mon[32255]: pgmap v653: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:41 smithi022 ceph-mon[31360]: pgmap v653: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:41 smithi022 ceph-mon[40250]: pgmap v653: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:43 smithi122 ceph-mon[32255]: pgmap v654: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:43 smithi022 ceph-mon[31360]: pgmap v654: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:43 smithi022 ceph-mon[40250]: pgmap v654: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:44.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:45.147 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:45 smithi022 ceph-mon[40250]: pgmap v655: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:45 smithi122 ceph-mon[32255]: pgmap v655: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:45 smithi022 ceph-mon[31360]: pgmap v655: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/341093298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:39:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:39:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/341093298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/341093298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:47 smithi122 ceph-mon[32255]: pgmap v656: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:47] "GET /metrics HTTP/1.1" 200 202017 "" "Prometheus/2.18.1" 2021-12-15T17:39:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:47 smithi022 ceph-mon[31360]: pgmap v656: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:47 smithi022 ceph-mon[40250]: pgmap v656: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:49 smithi122 ceph-mon[32255]: pgmap v657: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:49 smithi022 ceph-mon[31360]: pgmap v657: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:49 smithi022 ceph-mon[40250]: pgmap v657: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:50.166 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:39:50.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:50.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:50.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:51 smithi122 ceph-mon[32255]: pgmap v658: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/338447573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:51 smithi022 ceph-mon[31360]: pgmap v658: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/338447573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:51 smithi022 ceph-mon[40250]: pgmap v658: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/338447573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:53 smithi122 ceph-mon[32255]: pgmap v659: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:53 smithi022 ceph-mon[31360]: pgmap v659: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:53 smithi022 ceph-mon[40250]: pgmap v659: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:55 smithi122 ceph-mon[32255]: pgmap v660: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:55 smithi022 ceph-mon[31360]: pgmap v660: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:55 smithi022 ceph-mon[40250]: pgmap v660: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:55.536 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:39:55.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:39:55.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:39:55.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:39:56.168 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/748184473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:56.169 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/748184473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/748184473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:39:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:39:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:39:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:57 smithi122 ceph-mon[32255]: pgmap v661: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:39:57] "GET /metrics HTTP/1.1" 200 202015 "" "Prometheus/2.18.1" 2021-12-15T17:39:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:57 smithi022 ceph-mon[31360]: pgmap v661: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:57 smithi022 ceph-mon[40250]: pgmap v661: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:39:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:39:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:39:59 smithi122 ceph-mon[32255]: pgmap v662: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:39:59 smithi022 ceph-mon[31360]: pgmap v662: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:39:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:39:59 smithi022 ceph-mon[40250]: pgmap v662: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:40:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T17:40:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T17:40:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T17:40:00.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:40:01.177 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:01 smithi022 ceph-mon[40250]: pgmap v663: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:01.178 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:01 smithi022 ceph-mon[31360]: pgmap v663: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:01.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== 2021-12-15T17:40:01.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== ']' 2021-12-15T17:40:01.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:01 smithi122 ceph-mon[32255]: pgmap v663: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:02.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3356695516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:40:02.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:40:02.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:40:02.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:40:02.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3356695516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:40:02.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:40:02.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:40:02.185 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:02 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:40:02.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3356695516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:40:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:40:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:40:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:02 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:40:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:03 smithi022 ceph-mon[31360]: pgmap v664: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:03 smithi022 ceph-mon[40250]: pgmap v664: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:03 smithi122 ceph-mon[32255]: pgmap v664: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:40:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:40:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:40:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:40:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:40:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:40:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:05 smithi022 ceph-mon[31360]: pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:05 smithi022 ceph-mon[40250]: pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:05 smithi122 ceph-mon[32255]: pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.3 2021-12-15T17:40:06.648 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.4 2021-12-15T17:40:06.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQCNJrphWn36MhAAiBQpUOEsLzsZBVDIHH6yRA== 2021-12-15T17:40:06.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQCWIrphj40QEhAAheOgMkj8GLHGCAEMPF93Tw== == AQCNJrphWn36MhAAiBQpUOEsLzsZBVDIHH6yRA== ']' 2021-12-15T17:40:06.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:40:06.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.4' 2021-12-15T17:40:06.650 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:06.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:40:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:40:07.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:07.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:07.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.4 2021-12-15T17:40:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[31360]: pgmap v666: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1152801830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:40:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3953147122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[40250]: pgmap v666: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1152801830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:40:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3953147122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:07.293 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.4 on host 'smithi122' 2021-12-15T17:40:07.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:07.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:07 smithi122 ceph-mon[32255]: pgmap v666: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1152801830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2021-12-15T17:40:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3953147122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:07] "GET /metrics HTTP/1.1" 200 202012 "" "Prometheus/2.18.1" 2021-12-15T17:40:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[31360]: from='client.26284 v1:172.21.15.22:0/2384497733' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:40:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.4 2021-12-15T17:40:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:40:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:40:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:40:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[40250]: from='client.26284 v1:172.21.15.22:0/2384497733' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:40:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.4 2021-12-15T17:40:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:40:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:40:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:08 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:40:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:08 smithi122 ceph-mon[32255]: from='client.26284 v1:172.21.15.22:0/2384497733' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:08 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.4 2021-12-15T17:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:08 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:08 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:08 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:40:09.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:09 smithi022 ceph-mon[31360]: pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:09 smithi022 ceph-mon[40250]: pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:09 smithi122 ceph-mon[32255]: pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:40:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:11 smithi022 ceph-mon[31360]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:11 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:11 smithi022 ceph-mon[40250]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:11 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:11.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:11 smithi122 ceph-mon[32255]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:11.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:11 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:12.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:12.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:12.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:12.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:12 smithi122 ceph-mon[32255]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:12 smithi122 ceph-mon[32255]: Rotating authentication key for osd.4 2021-12-15T17:40:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2021-12-15T17:40:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2021-12-15T17:40:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[31360]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[31360]: Rotating authentication key for osd.4 2021-12-15T17:40:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2021-12-15T17:40:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2021-12-15T17:40:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[40250]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[40250]: Rotating authentication key for osd.4 2021-12-15T17:40:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2021-12-15T17:40:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2021-12-15T17:40:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:40:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/516232525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/516232525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/516232525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:14 smithi122 ceph-mon[32255]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:14 smithi022 ceph-mon[40250]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:14 smithi022 ceph-mon[31360]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:16.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:40:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:40:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:17 smithi022 ceph-mon[31360]: pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:17 smithi022 ceph-mon[40250]: pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:17 smithi122 ceph-mon[32255]: pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:17.673 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:17.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:17] "GET /metrics HTTP/1.1" 200 202012 "" "Prometheus/2.18.1" 2021-12-15T17:40:18.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:18.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:18.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1096262273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1096262273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1096262273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:19 smithi122 ceph-mon[32255]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:19 smithi022 ceph-mon[31360]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:19 smithi022 ceph-mon[40250]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:40:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:21 smithi122 ceph-mon[32255]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:21 smithi022 ceph-mon[31360]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:21 smithi022 ceph-mon[40250]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:23.044 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:23.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:23.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:23.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:23.418 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:23 smithi022 ceph-mon[31360]: pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:23.418 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:23 smithi022 ceph-mon[40250]: pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:23 smithi122 ceph-mon[32255]: pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/447329647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:24.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/447329647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/447329647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:25 smithi122 ceph-mon[32255]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:25 smithi022 ceph-mon[31360]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:25 smithi022 ceph-mon[40250]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:26.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:40:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:40:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:27 smithi022 ceph-mon[31360]: pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:27 smithi022 ceph-mon[40250]: pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:27 smithi122 ceph-mon[32255]: pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:27.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:27] "GET /metrics HTTP/1.1" 200 202017 "" "Prometheus/2.18.1" 2021-12-15T17:40:28.419 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:28.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:29 smithi122 ceph-mon[32255]: pgmap v677: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/519248202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:29 smithi022 ceph-mon[31360]: pgmap v677: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/519248202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:29 smithi022 ceph-mon[40250]: pgmap v677: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:40:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/519248202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:40:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:31 smithi122 ceph-mon[32255]: pgmap v678: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:31 smithi022 ceph-mon[31360]: pgmap v678: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:31 smithi022 ceph-mon[40250]: pgmap v678: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:33 smithi122 ceph-mon[32255]: pgmap v679: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:33 smithi022 ceph-mon[31360]: pgmap v679: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:33 smithi022 ceph-mon[40250]: pgmap v679: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:33.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:34.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:34.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:34.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:35 smithi122 ceph-mon[32255]: pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1001131165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:35 smithi022 ceph-mon[31360]: pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1001131165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:35 smithi022 ceph-mon[40250]: pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1001131165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:36.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:40:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:40:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:37 smithi022 ceph-mon[31360]: pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:37 smithi022 ceph-mon[40250]: pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:37 smithi122 ceph-mon[32255]: pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:37] "GET /metrics HTTP/1.1" 200 202016 "" "Prometheus/2.18.1" 2021-12-15T17:40:39.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:39 smithi122 ceph-mon[32255]: pgmap v682: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:39.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:39 smithi022 ceph-mon[31360]: pgmap v682: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:39 smithi022 ceph-mon[40250]: pgmap v682: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:39.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:39.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/738799266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:40:40.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/738799266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/738799266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:41 smithi122 ceph-mon[32255]: pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:41 smithi022 ceph-mon[31360]: pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:41 smithi022 ceph-mon[40250]: pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:43 smithi122 ceph-mon[32255]: pgmap v684: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:43 smithi022 ceph-mon[31360]: pgmap v684: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:43 smithi022 ceph-mon[40250]: pgmap v684: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:44.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:44.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:44.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:44.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:45.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:45 smithi022 ceph-mon[31360]: pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1611482878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:45 smithi022 ceph-mon[40250]: pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1611482878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:45 smithi122 ceph-mon[32255]: pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1611482878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:46.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:40:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:40:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:47 smithi022 ceph-mon[31360]: pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:47 smithi022 ceph-mon[40250]: pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:47 smithi122 ceph-mon[32255]: pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:47] "GET /metrics HTTP/1.1" 200 202016 "" "Prometheus/2.18.1" 2021-12-15T17:40:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:49 smithi122 ceph-mon[32255]: pgmap v687: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:49 smithi022 ceph-mon[31360]: pgmap v687: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:49 smithi022 ceph-mon[40250]: pgmap v687: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:49.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:40:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:51 smithi122 ceph-mon[32255]: pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/698094414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:51 smithi022 ceph-mon[31360]: pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/698094414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:51 smithi022 ceph-mon[40250]: pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/698094414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:53 smithi122 ceph-mon[32255]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:53 smithi022 ceph-mon[31360]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:53 smithi022 ceph-mon[40250]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:55.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:40:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:55 smithi122 ceph-mon[32255]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:55.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:55 smithi022 ceph-mon[31360]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:55 smithi022 ceph-mon[40250]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:55.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:40:55.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:40:55.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:40:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1489030101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:40:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:40:56.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1489030101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1489030101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:40:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:57 smithi022 ceph-mon[31360]: pgmap v691: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:57 smithi022 ceph-mon[40250]: pgmap v691: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:57 smithi122 ceph-mon[32255]: pgmap v691: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:40:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:40:57.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:40:57] "GET /metrics HTTP/1.1" 200 202018 "" "Prometheus/2.18.1" 2021-12-15T17:40:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:40:59 smithi122 ceph-mon[32255]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:40:59 smithi022 ceph-mon[31360]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:40:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:40:59 smithi022 ceph-mon[40250]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:41:00.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:01.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:01 smithi022 ceph-mon[31360]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2442604803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:01 smithi022 ceph-mon[40250]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2442604803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:01 smithi122 ceph-mon[32255]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2442604803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:03 smithi122 ceph-mon[32255]: pgmap v694: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:03 smithi022 ceph-mon[31360]: pgmap v694: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:03 smithi022 ceph-mon[40250]: pgmap v694: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:41:04.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:41:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:41:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:41:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:41:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:41:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:05 smithi122 ceph-mon[32255]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:05 smithi022 ceph-mon[31360]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:05 smithi022 ceph-mon[40250]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:06.025 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:06.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:06.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:06.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:06.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:41:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:41:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:07 smithi122 ceph-mon[32255]: pgmap v696: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2913808567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:07.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:07] "GET /metrics HTTP/1.1" 200 202011 "" "Prometheus/2.18.1" 2021-12-15T17:41:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:07 smithi022 ceph-mon[31360]: pgmap v696: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2913808567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:07 smithi022 ceph-mon[40250]: pgmap v696: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2913808567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:09 smithi122 ceph-mon[32255]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:09 smithi022 ceph-mon[31360]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:09 smithi022 ceph-mon[40250]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:41:11.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:11 smithi122 ceph-mon[32255]: pgmap v698: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:11.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:11 smithi022 ceph-mon[31360]: pgmap v698: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:11 smithi022 ceph-mon[40250]: pgmap v698: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:11.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:11.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:11.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:12.375 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/533054904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:12.375 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:41:12.375 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:41:12.376 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:12 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:41:12.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/533054904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:41:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:41:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:41:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/533054904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:41:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:41:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:12 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:41:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:13 smithi122 ceph-mon[32255]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:13 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:41:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:13 smithi022 ceph-mon[31360]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:13 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:41:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:13 smithi022 ceph-mon[40250]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:13 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:41:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:15 smithi122 ceph-mon[32255]: pgmap v700: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:15 smithi022 ceph-mon[31360]: pgmap v700: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:15 smithi022 ceph-mon[40250]: pgmap v700: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:16.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:41:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:41:16.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:17.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:17.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:17.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:17 smithi122 ceph-mon[32255]: pgmap v701: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1751631520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:17.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:17] "GET /metrics HTTP/1.1" 200 202011 "" "Prometheus/2.18.1" 2021-12-15T17:41:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:17 smithi022 ceph-mon[31360]: pgmap v701: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1751631520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:17 smithi022 ceph-mon[40250]: pgmap v701: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1751631520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:19 smithi122 ceph-mon[32255]: pgmap v702: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:19 smithi022 ceph-mon[31360]: pgmap v702: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:19 smithi022 ceph-mon[40250]: pgmap v702: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:41:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:21 smithi122 ceph-mon[32255]: pgmap v703: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:21 smithi022 ceph-mon[31360]: pgmap v703: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:21 smithi022 ceph-mon[40250]: pgmap v703: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:22.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:22.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:22.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:22.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:23 smithi122 ceph-mon[32255]: pgmap v704: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1361554249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:23 smithi022 ceph-mon[31360]: pgmap v704: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1361554249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:23 smithi022 ceph-mon[40250]: pgmap v704: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1361554249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:25 smithi122 ceph-mon[32255]: pgmap v705: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:25 smithi022 ceph-mon[31360]: pgmap v705: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:25 smithi022 ceph-mon[40250]: pgmap v705: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:26.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:41:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:41:27.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:27 smithi122 ceph-mon[32255]: pgmap v706: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:27 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:27.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:27] "GET /metrics HTTP/1.1" 200 202016 "" "Prometheus/2.18.1" 2021-12-15T17:41:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:27 smithi022 ceph-mon[31360]: pgmap v706: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:27 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:27 smithi022 ceph-mon[40250]: pgmap v706: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:27 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:27.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:27.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:27.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:28 smithi122 ceph-mon[32255]: pgmap v707: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2634111919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:28.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:28 smithi022 ceph-mon[31360]: pgmap v707: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2634111919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:28 smithi022 ceph-mon[40250]: pgmap v707: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2634111919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:41:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:31 smithi122 ceph-mon[32255]: pgmap v708: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:31 smithi022 ceph-mon[31360]: pgmap v708: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:31 smithi022 ceph-mon[40250]: pgmap v708: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:32.878 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:33.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:33.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:33.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:33.249 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:33 smithi022 ceph-mon[31360]: pgmap v709: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:33.250 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:33 smithi022 ceph-mon[40250]: pgmap v709: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:33 smithi122 ceph-mon[32255]: pgmap v709: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/53313058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/53313058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/53313058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:35 smithi122 ceph-mon[32255]: pgmap v710: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:35.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:35 smithi022 ceph-mon[31360]: pgmap v710: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:35 smithi022 ceph-mon[40250]: pgmap v710: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:36.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:41:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:41:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:37 smithi022 ceph-mon[31360]: pgmap v711: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:37 smithi022 ceph-mon[40250]: pgmap v711: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:37 smithi122 ceph-mon[32255]: pgmap v711: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:37.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:37] "GET /metrics HTTP/1.1" 200 202018 "" "Prometheus/2.18.1" 2021-12-15T17:41:38.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:38.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:38.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:38.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:39 smithi122 ceph-mon[32255]: pgmap v712: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/452205951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:39 smithi022 ceph-mon[31360]: pgmap v712: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/452205951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:39 smithi022 ceph-mon[40250]: pgmap v712: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/452205951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:41:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:41 smithi122 ceph-mon[32255]: pgmap v713: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:41 smithi022 ceph-mon[31360]: pgmap v713: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:41 smithi022 ceph-mon[40250]: pgmap v713: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:43 smithi122 ceph-mon[32255]: pgmap v714: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:43 smithi022 ceph-mon[31360]: pgmap v714: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:43 smithi022 ceph-mon[40250]: pgmap v714: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:43.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:43.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:43.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:44.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2884655844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2884655844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2884655844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:45 smithi122 ceph-mon[32255]: pgmap v715: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:45.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:45 smithi022 ceph-mon[31360]: pgmap v715: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:45 smithi022 ceph-mon[40250]: pgmap v715: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:46.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:41:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:41:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:47 smithi022 ceph-mon[31360]: pgmap v716: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:47 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:47 smithi022 ceph-mon[40250]: pgmap v716: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:47 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:47 smithi122 ceph-mon[32255]: pgmap v716: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:47 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:47] "GET /metrics HTTP/1.1" 200 202018 "" "Prometheus/2.18.1" 2021-12-15T17:41:48.993 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:49.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:49 smithi022 ceph-mon[31360]: pgmap v717: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:49 smithi022 ceph-mon[40250]: pgmap v717: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:49.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:49.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:49.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:49 smithi122 ceph-mon[32255]: pgmap v717: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:41:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3359508349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3359508349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3359508349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:51 smithi122 ceph-mon[32255]: pgmap v718: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:51 smithi022 ceph-mon[31360]: pgmap v718: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:51 smithi022 ceph-mon[40250]: pgmap v718: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:53 smithi122 ceph-mon[32255]: pgmap v719: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:53 smithi022 ceph-mon[31360]: pgmap v719: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:53 smithi022 ceph-mon[40250]: pgmap v719: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:54.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:41:54.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:41:54.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:41:54.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:41:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:55 smithi122 ceph-mon[32255]: pgmap v720: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1934196162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:55 smithi022 ceph-mon[31360]: pgmap v720: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1934196162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:55 smithi022 ceph-mon[40250]: pgmap v720: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1934196162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:41:56.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:41:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:41:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:57 smithi022 ceph-mon[31360]: pgmap v721: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:57 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:57 smithi022 ceph-mon[40250]: pgmap v721: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:57 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:57 smithi122 ceph-mon[32255]: pgmap v721: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:41:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:57 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:41:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:41:57] "GET /metrics HTTP/1.1" 200 202019 "" "Prometheus/2.18.1" 2021-12-15T17:41:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:41:59 smithi122 ceph-mon[32255]: pgmap v722: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:41:59 smithi022 ceph-mon[31360]: pgmap v722: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:41:59 smithi022 ceph-mon[40250]: pgmap v722: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:41:59.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:42:00.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:42:00.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:42:00.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:42:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:01 smithi122 ceph-mon[32255]: pgmap v723: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/466775980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:01.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:01 smithi022 ceph-mon[31360]: pgmap v723: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/466775980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:01 smithi022 ceph-mon[40250]: pgmap v723: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/466775980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:03 smithi122 ceph-mon[32255]: pgmap v724: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:03 smithi022 ceph-mon[31360]: pgmap v724: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:03 smithi022 ceph-mon[40250]: pgmap v724: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:42:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:42:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:42:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:42:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:42:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:42:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:42:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:05 smithi122 ceph-mon[32255]: pgmap v725: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:05.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:42:05.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:42:05.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:05.482 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:05 smithi022 ceph-mon[40250]: pgmap v725: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:05.483 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:05 smithi022 ceph-mon[31360]: pgmap v725: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:06.395 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/451388661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:06.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/451388661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/451388661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:06.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:42:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:42:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:07 smithi022 ceph-mon[31360]: pgmap v726: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:07 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:07 smithi022 ceph-mon[40250]: pgmap v726: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:07 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:07 smithi122 ceph-mon[32255]: pgmap v726: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:07 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:07] "GET /metrics HTTP/1.1" 200 202008 "" "Prometheus/2.18.1" 2021-12-15T17:42:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:09 smithi122 ceph-mon[32255]: pgmap v727: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:09.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:09 smithi022 ceph-mon[31360]: pgmap v727: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:09 smithi022 ceph-mon[40250]: pgmap v727: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:42:10.483 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:42:10.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== 2021-12-15T17:42:10.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== ']' 2021-12-15T17:42:10.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:11 smithi122 ceph-mon[32255]: pgmap v728: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1348654095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:11 smithi022 ceph-mon[31360]: pgmap v728: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1348654095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:11 smithi022 ceph-mon[40250]: pgmap v728: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1348654095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:13.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:13 smithi122 ceph-mon[32255]: pgmap v729: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:13 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:13 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:13 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:13 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[40250]: pgmap v729: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[31360]: pgmap v729: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:13 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:15 smithi122 ceph-mon[32255]: pgmap v730: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:15 smithi022 ceph-mon[31360]: pgmap v730: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:15 smithi022 ceph-mon[40250]: pgmap v730: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:15.851 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.4 2021-12-15T17:42:16.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQB7KLphq/xlJRAA9Wy1RSm0OHLHaLVqUOcdbg== 2021-12-15T17:42:16.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQC6IrphhnRmBhAAd/lVkdX/rLLKVX7sllEYVw== == AQB7KLphq/xlJRAA9Wy1RSm0OHLHaLVqUOcdbg== ']' 2021-12-15T17:42:16.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:42:16.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.5' 2021-12-15T17:42:16.220 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.5 2021-12-15T17:42:16.220 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.5 2021-12-15T17:42:16.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQDOIrphC9Y6HRAAzzWtuvaTNhCaRo+GETHFwA== 2021-12-15T17:42:16.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDOIrphC9Y6HRAAzzWtuvaTNhCaRo+GETHFwA== 2021-12-15T17:42:16.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.5 2021-12-15T17:42:16.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:42:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:42:16.864 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.5 on host 'smithi122' 2021-12-15T17:42:16.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDOIrphC9Y6HRAAzzWtuvaTNhCaRo+GETHFwA== == AQDOIrphC9Y6HRAAzzWtuvaTNhCaRo+GETHFwA== ']' 2021-12-15T17:42:16.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: pgmap v731: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2596069546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3758207179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2021-12-15T17:42:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2021-12-15T17:42:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2021-12-15T17:42:17.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: pgmap v731: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:17.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2596069546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:17.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:17.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3758207179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2021-12-15T17:42:17.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2021-12-15T17:42:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:17 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2021-12-15T17:42:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: pgmap v731: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2596069546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2021-12-15T17:42:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3758207179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2021-12-15T17:42:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2021-12-15T17:42:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:17 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2021-12-15T17:42:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:17] "GET /metrics HTTP/1.1" 200 202008 "" "Prometheus/2.18.1" 2021-12-15T17:42:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:18 smithi122 ceph-mon[32255]: from='client.26452 v1:172.21.15.22:0/287250474' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:42:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:18 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.5 2021-12-15T17:42:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:18 smithi122 ceph-mon[32255]: Rotating authentication key for osd.5 2021-12-15T17:42:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:18 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[31360]: from='client.26452 v1:172.21.15.22:0/287250474' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:42:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.5 2021-12-15T17:42:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[31360]: Rotating authentication key for osd.5 2021-12-15T17:42:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[40250]: from='client.26452 v1:172.21.15.22:0/287250474' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:42:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.5 2021-12-15T17:42:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[40250]: Rotating authentication key for osd.5 2021-12-15T17:42:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:18 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:19 smithi122 ceph-mon[32255]: pgmap v732: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:19 smithi022 ceph-mon[31360]: pgmap v732: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:19 smithi022 ceph-mon[40250]: pgmap v732: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:42:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:21 smithi122 ceph-mon[32255]: pgmap v733: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:21 smithi022 ceph-mon[31360]: pgmap v733: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:21 smithi022 ceph-mon[40250]: pgmap v733: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:21.880 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.5 2021-12-15T17:42:22.252 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.6 2021-12-15T17:42:22.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQD4KLphqmA/NBAA4jiBmh1Nx9H/4mn8DVM7dg== 2021-12-15T17:42:22.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDOIrphC9Y6HRAAzzWtuvaTNhCaRo+GETHFwA== == AQD4KLphqmA/NBAA4jiBmh1Nx9H/4mn8DVM7dg== ']' 2021-12-15T17:42:22.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:42:22.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.6' 2021-12-15T17:42:22.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:42:22.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:22.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:22.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.6 2021-12-15T17:42:22.897 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.6 on host 'smithi122' 2021-12-15T17:42:22.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:42:22.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: pgmap v734: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3723333272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2021-12-15T17:42:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1566525388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2021-12-15T17:42:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2021-12-15T17:42:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: pgmap v734: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3723333272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2021-12-15T17:42:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1566525388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2021-12-15T17:42:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2021-12-15T17:42:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: pgmap v734: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3723333272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2021-12-15T17:42:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1566525388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:42:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:42:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:42:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2021-12-15T17:42:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2021-12-15T17:42:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[31360]: from='client.16575 v1:172.21.15.22:0/3251516325' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:42:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.6 2021-12-15T17:42:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[31360]: Rotating authentication key for osd.6 2021-12-15T17:42:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[40250]: from='client.16575 v1:172.21.15.22:0/3251516325' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:42:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.6 2021-12-15T17:42:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[40250]: Rotating authentication key for osd.6 2021-12-15T17:42:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:24 smithi122 ceph-mon[32255]: from='client.16575 v1:172.21.15.22:0/3251516325' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:42:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:24 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.6 2021-12-15T17:42:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:24 smithi122 ceph-mon[32255]: Rotating authentication key for osd.6 2021-12-15T17:42:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:42:25.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:25 smithi022 ceph-mon[31360]: pgmap v735: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:25 smithi022 ceph-mon[40250]: pgmap v735: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:25 smithi122 ceph-mon[32255]: pgmap v735: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:26.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:42:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:42:27.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:27 smithi022 ceph-mon[31360]: pgmap v736: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:27] "GET /metrics HTTP/1.1" 200 202013 "" "Prometheus/2.18.1" 2021-12-15T17:42:27.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:27 smithi022 ceph-mon[40250]: pgmap v736: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:27 smithi122 ceph-mon[32255]: pgmap v736: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:27.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:42:28.213 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:28.214 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:28.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:28.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:42:28.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:29 smithi022 ceph-mon[31360]: pgmap v737: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3270116046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:29 smithi022 ceph-mon[40250]: pgmap v737: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3270116046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:29 smithi122 ceph-mon[32255]: pgmap v737: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3270116046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:42:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:31 smithi022 ceph-mon[31360]: pgmap v738: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:31 smithi022 ceph-mon[40250]: pgmap v738: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:31 smithi122 ceph-mon[32255]: pgmap v738: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:33.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:42:33.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:33 smithi022 ceph-mon[31360]: pgmap v739: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:33 smithi022 ceph-mon[40250]: pgmap v739: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:33.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:33.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:42:33.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:33 smithi122 ceph-mon[32255]: pgmap v739: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:34.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1183252782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1183252782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1183252782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:35 smithi022 ceph-mon[31360]: pgmap v740: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:35 smithi022 ceph-mon[40250]: pgmap v740: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:35 smithi122 ceph-mon[32255]: pgmap v740: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:36.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:42:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:42:37.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:37 smithi022 ceph-mon[31360]: pgmap v741: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:37] "GET /metrics HTTP/1.1" 200 202014 "" "Prometheus/2.18.1" 2021-12-15T17:42:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:37 smithi022 ceph-mon[40250]: pgmap v741: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:37 smithi122 ceph-mon[32255]: pgmap v741: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:38.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:38.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:42:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:39.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:39.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:42:39.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:39 smithi022 ceph-mon[31360]: pgmap v742: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2922783813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:39 smithi022 ceph-mon[40250]: pgmap v742: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2922783813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:39 smithi122 ceph-mon[32255]: pgmap v742: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2922783813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:42:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:41 smithi022 ceph-mon[31360]: pgmap v743: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:41 smithi022 ceph-mon[40250]: pgmap v743: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:41 smithi122 ceph-mon[32255]: pgmap v743: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:43 smithi022 ceph-mon[31360]: pgmap v744: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:43 smithi022 ceph-mon[40250]: pgmap v744: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:43 smithi122 ceph-mon[32255]: pgmap v744: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:44.025 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:42:44.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:44.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:42:44.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:45 smithi022 ceph-mon[31360]: pgmap v745: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1555756894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:45 smithi022 ceph-mon[40250]: pgmap v745: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1555756894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:45 smithi122 ceph-mon[32255]: pgmap v745: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1555756894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:46.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:42:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:42:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:47 smithi022 ceph-mon[31360]: pgmap v746: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:47] "GET /metrics HTTP/1.1" 200 202014 "" "Prometheus/2.18.1" 2021-12-15T17:42:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:47 smithi022 ceph-mon[40250]: pgmap v746: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:47 smithi122 ceph-mon[32255]: pgmap v746: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:48.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:49.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:42:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:49 smithi022 ceph-mon[31360]: pgmap v747: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:49 smithi022 ceph-mon[40250]: pgmap v747: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:49 smithi122 ceph-mon[32255]: pgmap v747: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:49.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:49.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:42:49.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:42:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1624190390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1624190390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1624190390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:51 smithi022 ceph-mon[31360]: pgmap v748: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:51 smithi022 ceph-mon[40250]: pgmap v748: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:51 smithi122 ceph-mon[32255]: pgmap v748: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:53.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:53 smithi022 ceph-mon[31360]: pgmap v749: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:53 smithi022 ceph-mon[40250]: pgmap v749: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:53 smithi122 ceph-mon[32255]: pgmap v749: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:42:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:42:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:42:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:42:55.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:55 smithi022 ceph-mon[31360]: pgmap v750: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/967384561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:55 smithi022 ceph-mon[40250]: pgmap v750: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/967384561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:55 smithi122 ceph-mon[32255]: pgmap v750: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/967384561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:42:56.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:42:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:42:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:57 smithi022 ceph-mon[31360]: pgmap v751: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:42:57] "GET /metrics HTTP/1.1" 200 202024 "" "Prometheus/2.18.1" 2021-12-15T17:42:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:57 smithi022 ceph-mon[40250]: pgmap v751: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:57 smithi122 ceph-mon[32255]: pgmap v751: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:42:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:42:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:42:59 smithi022 ceph-mon[31360]: pgmap v752: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:42:59 smithi022 ceph-mon[40250]: pgmap v752: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:42:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:42:59 smithi122 ceph-mon[32255]: pgmap v752: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:00.132 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:43:00.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:00.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:00.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:01 smithi022 ceph-mon[31360]: pgmap v753: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3851507851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:01 smithi022 ceph-mon[40250]: pgmap v753: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3851507851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:01 smithi122 ceph-mon[32255]: pgmap v753: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3851507851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:03 smithi022 ceph-mon[31360]: pgmap v754: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:03 smithi022 ceph-mon[40250]: pgmap v754: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:03 smithi122 ceph-mon[32255]: pgmap v754: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:43:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:43:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:43:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:43:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:43:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:43:05.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:05 smithi122 ceph-mon[32255]: pgmap v755: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:05 smithi022 ceph-mon[31360]: pgmap v755: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:05 smithi022 ceph-mon[40250]: pgmap v755: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:05.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:05.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:05.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1012270544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:06.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:43:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:43:06.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1012270544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1012270544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:07 smithi122 ceph-mon[32255]: pgmap v756: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:07 smithi022 ceph-mon[31360]: pgmap v756: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:07.847 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:07] "GET /metrics HTTP/1.1" 200 202021 "" "Prometheus/2.18.1" 2021-12-15T17:43:07.847 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:07 smithi022 ceph-mon[40250]: pgmap v756: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:09 smithi122 ceph-mon[32255]: pgmap v757: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:09 smithi022 ceph-mon[31360]: pgmap v757: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:09 smithi022 ceph-mon[40250]: pgmap v757: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:43:10.873 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:11.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:11.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:11.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:11.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:11 smithi022 ceph-mon[31360]: pgmap v758: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2463150352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:11 smithi022 ceph-mon[40250]: pgmap v758: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2463150352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:11 smithi122 ceph-mon[32255]: pgmap v758: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2463150352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:13 smithi122 ceph-mon[32255]: pgmap v759: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:13 smithi022 ceph-mon[31360]: pgmap v759: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:13 smithi022 ceph-mon[40250]: pgmap v759: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:15 smithi122 ceph-mon[32255]: pgmap v760: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:15 smithi022 ceph-mon[31360]: pgmap v760: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:15 smithi022 ceph-mon[40250]: pgmap v760: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:16.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:16.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:16.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:16.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:16.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:43:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:43:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:17 smithi122 ceph-mon[32255]: pgmap v761: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4195763965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:17.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:17] "GET /metrics HTTP/1.1" 200 202021 "" "Prometheus/2.18.1" 2021-12-15T17:43:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:17 smithi022 ceph-mon[31360]: pgmap v761: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4195763965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:17 smithi022 ceph-mon[40250]: pgmap v761: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4195763965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:19 smithi122 ceph-mon[32255]: pgmap v762: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:19.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:19 smithi022 ceph-mon[31360]: pgmap v762: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:19 smithi022 ceph-mon[40250]: pgmap v762: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:43:21.613 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:21.614 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:21 smithi022 ceph-mon[31360]: pgmap v763: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:21.615 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:21 smithi022 ceph-mon[40250]: pgmap v763: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:21 smithi122 ceph-mon[32255]: pgmap v763: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:21.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:21.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:21.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/759010976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/759010976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/759010976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[31360]: pgmap v764: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:43:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:43:23.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:43:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[40250]: pgmap v764: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:43:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:43:23.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:23 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:43:23.660 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:23 smithi122 ceph-mon[32255]: pgmap v764: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:23.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:43:23.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:43:23.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:23 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:43:24.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:24 smithi122 ceph-mon[32255]: pgmap v765: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:43:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:24 smithi022 ceph-mon[31360]: pgmap v765: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:43:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:24 smithi022 ceph-mon[40250]: pgmap v765: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:43:26.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:43:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:43:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:27 smithi022 ceph-mon[31360]: pgmap v766: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:27 smithi022 ceph-mon[40250]: pgmap v766: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:27.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:27.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:27.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:27 smithi122 ceph-mon[32255]: pgmap v766: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:27] "GET /metrics HTTP/1.1" 200 202028 "" "Prometheus/2.18.1" 2021-12-15T17:43:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4056751243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4056751243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4056751243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:29 smithi122 ceph-mon[32255]: pgmap v767: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:29.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:29 smithi022 ceph-mon[31360]: pgmap v767: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:29 smithi022 ceph-mon[40250]: pgmap v767: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:43:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:31 smithi122 ceph-mon[32255]: pgmap v768: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:31 smithi022 ceph-mon[31360]: pgmap v768: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:31 smithi022 ceph-mon[40250]: pgmap v768: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:32.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:32.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:32.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:32.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:33 smithi122 ceph-mon[32255]: pgmap v769: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/237388286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:33 smithi022 ceph-mon[31360]: pgmap v769: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/237388286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:33 smithi022 ceph-mon[40250]: pgmap v769: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/237388286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:35 smithi122 ceph-mon[32255]: pgmap v770: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:35.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:35 smithi022 ceph-mon[31360]: pgmap v770: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:35 smithi022 ceph-mon[40250]: pgmap v770: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:36.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:43:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:43:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:37 smithi022 ceph-mon[31360]: pgmap v771: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:37.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:37 smithi022 ceph-mon[40250]: pgmap v771: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:37 smithi122 ceph-mon[32255]: pgmap v771: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:37.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:37.724 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:37] "GET /metrics HTTP/1.1" 200 202023 "" "Prometheus/2.18.1" 2021-12-15T17:43:38.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:38.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:38.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3098390754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:38.543 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3098390754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:38.545 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:38.545 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3098390754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:39 smithi122 ceph-mon[32255]: pgmap v772: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:39.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:39 smithi022 ceph-mon[31360]: pgmap v772: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:39 smithi022 ceph-mon[40250]: pgmap v772: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:43:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:41 smithi122 ceph-mon[32255]: pgmap v773: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:41 smithi022 ceph-mon[31360]: pgmap v773: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:41 smithi022 ceph-mon[40250]: pgmap v773: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:43.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:43 smithi122 ceph-mon[32255]: pgmap v774: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:43.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:43 smithi022 ceph-mon[31360]: pgmap v774: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:43.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:43 smithi022 ceph-mon[40250]: pgmap v774: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:43.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:43.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:43.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1127730419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1127730419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1127730419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:45 smithi122 ceph-mon[32255]: pgmap v775: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:45 smithi022 ceph-mon[31360]: pgmap v775: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:45 smithi022 ceph-mon[40250]: pgmap v775: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:46.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:43:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:43:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:47 smithi122 ceph-mon[32255]: pgmap v776: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:47 smithi022 ceph-mon[31360]: pgmap v776: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:47] "GET /metrics HTTP/1.1" 200 202023 "" "Prometheus/2.18.1" 2021-12-15T17:43:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:47 smithi022 ceph-mon[40250]: pgmap v776: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:48.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:48.475 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:48.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:48.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:48.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:48.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:49 smithi122 ceph-mon[32255]: pgmap v777: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3748872434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:49 smithi022 ceph-mon[31360]: pgmap v777: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3748872434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:49 smithi022 ceph-mon[40250]: pgmap v777: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3748872434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:43:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:51 smithi122 ceph-mon[32255]: pgmap v778: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:51.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:51 smithi022 ceph-mon[31360]: pgmap v778: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:51 smithi022 ceph-mon[40250]: pgmap v778: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:53 smithi122 ceph-mon[32255]: pgmap v779: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:53.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:53 smithi022 ceph-mon[31360]: pgmap v779: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:53 smithi022 ceph-mon[40250]: pgmap v779: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:53.843 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:54.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:54.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:54.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:43:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:55 smithi122 ceph-mon[32255]: pgmap v780: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2751157341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:55.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:55 smithi022 ceph-mon[31360]: pgmap v780: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2751157341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:55 smithi022 ceph-mon[40250]: pgmap v780: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2751157341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:43:56.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:43:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:43:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:57 smithi122 ceph-mon[32255]: pgmap v781: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:57 smithi022 ceph-mon[31360]: pgmap v781: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:43:57] "GET /metrics HTTP/1.1" 200 202029 "" "Prometheus/2.18.1" 2021-12-15T17:43:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:57 smithi022 ceph-mon[40250]: pgmap v781: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:43:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:43:59.213 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:43:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:43:59 smithi122 ceph-mon[32255]: pgmap v782: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:43:59 smithi022 ceph-mon[31360]: pgmap v782: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:43:59 smithi022 ceph-mon[40250]: pgmap v782: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:43:59.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:43:59.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:43:59.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3785315071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:44:00.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3785315071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3785315071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:01 smithi122 ceph-mon[32255]: pgmap v783: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:01 smithi022 ceph-mon[31360]: pgmap v783: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:01 smithi022 ceph-mon[40250]: pgmap v783: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:03 smithi122 ceph-mon[32255]: pgmap v784: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:03 smithi022 ceph-mon[31360]: pgmap v784: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:03 smithi022 ceph-mon[40250]: pgmap v784: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:44:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:44:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:44:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:44:04.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:44:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:44:04.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:04.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:04.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:05.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:05 smithi022 ceph-mon[31360]: pgmap v785: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1677647545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:05 smithi022 ceph-mon[40250]: pgmap v785: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1677647545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:05 smithi122 ceph-mon[32255]: pgmap v785: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1677647545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:06.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:44:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:44:07.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:07 smithi022 ceph-mon[31360]: pgmap v786: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:07] "GET /metrics HTTP/1.1" 200 202022 "" "Prometheus/2.18.1" 2021-12-15T17:44:07.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:07 smithi022 ceph-mon[40250]: pgmap v786: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:07 smithi122 ceph-mon[32255]: pgmap v786: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:08.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:09 smithi022 ceph-mon[31360]: pgmap v787: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:09 smithi022 ceph-mon[40250]: pgmap v787: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:09 smithi122 ceph-mon[32255]: pgmap v787: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:09.964 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:10.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:10.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:10.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:44:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:11 smithi022 ceph-mon[31360]: pgmap v788: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3483230648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:11 smithi022 ceph-mon[40250]: pgmap v788: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3483230648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:11 smithi122 ceph-mon[32255]: pgmap v788: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3483230648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:13.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:13 smithi022 ceph-mon[31360]: pgmap v789: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:13 smithi022 ceph-mon[40250]: pgmap v789: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:13 smithi122 ceph-mon[32255]: pgmap v789: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:15.335 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:15.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:15 smithi022 ceph-mon[31360]: pgmap v790: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:15 smithi022 ceph-mon[40250]: pgmap v790: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:15 smithi122 ceph-mon[32255]: pgmap v790: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:15.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:15.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:15.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:16.507 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3892788079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3892788079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3892788079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:16.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:44:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:44:17.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:17 smithi022 ceph-mon[31360]: pgmap v791: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:17] "GET /metrics HTTP/1.1" 200 202022 "" "Prometheus/2.18.1" 2021-12-15T17:44:17.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:17 smithi022 ceph-mon[40250]: pgmap v791: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:17 smithi122 ceph-mon[32255]: pgmap v791: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:18.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:19 smithi022 ceph-mon[31360]: pgmap v792: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:19 smithi022 ceph-mon[40250]: pgmap v792: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:19 smithi122 ceph-mon[32255]: pgmap v792: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:44:20.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:21.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:21.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:21.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:21.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:21 smithi022 ceph-mon[31360]: pgmap v793: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3636568329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:21 smithi022 ceph-mon[40250]: pgmap v793: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3636568329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:21 smithi122 ceph-mon[32255]: pgmap v793: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3636568329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:23 smithi022 ceph-mon[31360]: pgmap v794: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:23 smithi022 ceph-mon[40250]: pgmap v794: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:23 smithi122 ceph-mon[32255]: pgmap v794: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:44:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:44:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:44:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:24 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:24.515 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:44:24.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:44:24.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:44:24.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:24.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:24.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:44:24.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:44:24.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:44:24.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:24.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:24 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:25 smithi022 ceph-mon[31360]: pgmap v795: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:25 smithi022 ceph-mon[40250]: pgmap v795: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:25 smithi122 ceph-mon[32255]: pgmap v795: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:44:26.078 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:26.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:26.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:44:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:44:27.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:27] "GET /metrics HTTP/1.1" 200 202024 "" "Prometheus/2.18.1" 2021-12-15T17:44:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:27 smithi022 ceph-mon[31360]: pgmap v796: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1467522721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:27 smithi022 ceph-mon[40250]: pgmap v796: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1467522721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:27 smithi122 ceph-mon[32255]: pgmap v796: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1467522721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:29 smithi022 ceph-mon[31360]: pgmap v797: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:29 smithi022 ceph-mon[40250]: pgmap v797: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:29 smithi122 ceph-mon[32255]: pgmap v797: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:44:31.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:31 smithi022 ceph-mon[31360]: pgmap v798: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:31 smithi022 ceph-mon[40250]: pgmap v798: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:31 smithi122 ceph-mon[32255]: pgmap v798: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:31.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:31.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:31.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:32.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2674273027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2674273027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2674273027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:33 smithi022 ceph-mon[31360]: pgmap v799: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:33 smithi022 ceph-mon[40250]: pgmap v799: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:33 smithi122 ceph-mon[32255]: pgmap v799: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:35 smithi022 ceph-mon[31360]: pgmap v800: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:35 smithi022 ceph-mon[40250]: pgmap v800: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:35 smithi122 ceph-mon[32255]: pgmap v800: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:36.820 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:36.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:44:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:44:37.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:37.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:37.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:37] "GET /metrics HTTP/1.1" 200 202032 "" "Prometheus/2.18.1" 2021-12-15T17:44:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:37 smithi022 ceph-mon[31360]: pgmap v801: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/630618829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:37 smithi022 ceph-mon[40250]: pgmap v801: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/630618829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:37 smithi122 ceph-mon[32255]: pgmap v801: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/630618829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:39 smithi022 ceph-mon[31360]: pgmap v802: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:39 smithi022 ceph-mon[40250]: pgmap v802: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:39 smithi122 ceph-mon[32255]: pgmap v802: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:44:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:41 smithi022 ceph-mon[31360]: pgmap v803: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:41 smithi022 ceph-mon[40250]: pgmap v803: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:41 smithi122 ceph-mon[32255]: pgmap v803: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:42.188 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:42.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:42.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:42.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:43 smithi022 ceph-mon[31360]: pgmap v804: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2930832522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:43 smithi022 ceph-mon[40250]: pgmap v804: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2930832522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:43 smithi122 ceph-mon[32255]: pgmap v804: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2930832522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:45 smithi022 ceph-mon[31360]: pgmap v805: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:45 smithi022 ceph-mon[40250]: pgmap v805: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:45 smithi122 ceph-mon[32255]: pgmap v805: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:46.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:44:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:44:47.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:47 smithi022 ceph-mon[31360]: pgmap v806: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:47] "GET /metrics HTTP/1.1" 200 202032 "" "Prometheus/2.18.1" 2021-12-15T17:44:47.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:47 smithi022 ceph-mon[40250]: pgmap v806: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:47.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:47 smithi122 ceph-mon[32255]: pgmap v806: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:47.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:47.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:47.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2509490396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2509490396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2509490396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:49 smithi022 ceph-mon[31360]: pgmap v807: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:49 smithi022 ceph-mon[40250]: pgmap v807: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:49 smithi122 ceph-mon[32255]: pgmap v807: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:44:51.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:51 smithi022 ceph-mon[31360]: pgmap v808: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:51 smithi022 ceph-mon[40250]: pgmap v808: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:51 smithi122 ceph-mon[32255]: pgmap v808: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:52.928 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:53.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:53.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:53.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:53 smithi122 ceph-mon[32255]: pgmap v809: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:53 smithi022 ceph-mon[31360]: pgmap v809: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:53 smithi022 ceph-mon[40250]: pgmap v809: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/564248891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/564248891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/564248891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:55 smithi122 ceph-mon[32255]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:55 smithi022 ceph-mon[31360]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:55 smithi022 ceph-mon[40250]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:56.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:44:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:44:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:57 smithi122 ceph-mon[32255]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:57 smithi022 ceph-mon[31360]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:44:57] "GET /metrics HTTP/1.1" 200 202025 "" "Prometheus/2.18.1" 2021-12-15T17:44:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:57 smithi022 ceph-mon[40250]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:44:58.302 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:44:58.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:44:58.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:44:58.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:44:58.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:58.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:44:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:59 smithi122 ceph-mon[32255]: pgmap v812: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:44:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2560738224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:59 smithi022 ceph-mon[31360]: pgmap v812: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:44:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2560738224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:44:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:59 smithi022 ceph-mon[40250]: pgmap v812: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:44:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:44:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2560738224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:45:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:01 smithi122 ceph-mon[32255]: pgmap v813: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:01 smithi022 ceph-mon[31360]: pgmap v813: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:01 smithi022 ceph-mon[40250]: pgmap v813: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:03.676 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:03.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:03 smithi022 ceph-mon[31360]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:03.677 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:03 smithi022 ceph-mon[40250]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:03 smithi122 ceph-mon[32255]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:04.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:04.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:04.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:45:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:45:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4023762084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:45:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:45:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4023762084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:45:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:45:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4023762084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:05 smithi122 ceph-mon[32255]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:05 smithi022 ceph-mon[31360]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:05 smithi022 ceph-mon[40250]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:45:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:45:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:07 smithi122 ceph-mon[32255]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:07 smithi022 ceph-mon[31360]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:07] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:45:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:07 smithi022 ceph-mon[40250]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:09.044 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:09 smithi122 ceph-mon[32255]: pgmap v817: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:09 smithi022 ceph-mon[31360]: pgmap v817: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:09 smithi022 ceph-mon[40250]: pgmap v817: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:45:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4094981437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4094981437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4094981437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:11 smithi122 ceph-mon[32255]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:11 smithi022 ceph-mon[31360]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:11 smithi022 ceph-mon[40250]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:13 smithi122 ceph-mon[32255]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:13 smithi022 ceph-mon[31360]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:13 smithi022 ceph-mon[40250]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:14.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:14.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:14.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:14.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:15 smithi122 ceph-mon[32255]: pgmap v820: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1632037204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:15 smithi022 ceph-mon[31360]: pgmap v820: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1632037204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:15 smithi022 ceph-mon[40250]: pgmap v820: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1632037204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:16.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:45:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:45:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:17 smithi122 ceph-mon[32255]: pgmap v821: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:17 smithi022 ceph-mon[31360]: pgmap v821: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:17] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:45:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:17 smithi022 ceph-mon[40250]: pgmap v821: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:19 smithi122 ceph-mon[32255]: pgmap v822: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:19 smithi022 ceph-mon[31360]: pgmap v822: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:19 smithi022 ceph-mon[40250]: pgmap v822: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:19.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:20.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:20.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:20.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:45:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:21 smithi122 ceph-mon[32255]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/312456055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:21 smithi022 ceph-mon[31360]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/312456055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:21 smithi022 ceph-mon[40250]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/312456055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:23 smithi122 ceph-mon[32255]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:23 smithi022 ceph-mon[31360]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:23 smithi022 ceph-mon[40250]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:25.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[31360]: pgmap v825: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:45:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:45:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:45:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:45:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[40250]: pgmap v825: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:45:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:45:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:45:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:45:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:25 smithi122 ceph-mon[32255]: pgmap v825: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:45:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:45:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:45:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:45:25.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:25.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:25.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3195753973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:26.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3195753973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3195753973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:26.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:45:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:45:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:27 smithi122 ceph-mon[32255]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:27 smithi022 ceph-mon[31360]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:27] "GET /metrics HTTP/1.1" 200 202020 "" "Prometheus/2.18.1" 2021-12-15T17:45:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:27 smithi022 ceph-mon[40250]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:29 smithi122 ceph-mon[32255]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:29.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:29 smithi022 ceph-mon[31360]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:29 smithi022 ceph-mon[40250]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:45:30.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:30.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:30.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:30.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:31 smithi122 ceph-mon[32255]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2346909634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:31 smithi022 ceph-mon[31360]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2346909634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:31 smithi022 ceph-mon[40250]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2346909634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:33 smithi122 ceph-mon[32255]: pgmap v829: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:33.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:33 smithi022 ceph-mon[31360]: pgmap v829: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:33 smithi022 ceph-mon[40250]: pgmap v829: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:35 smithi122 ceph-mon[32255]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:35 smithi022 ceph-mon[31360]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:35 smithi022 ceph-mon[40250]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:35.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:36.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:36.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:36.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:36.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:45:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:45:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:37 smithi122 ceph-mon[32255]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3007737607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:37.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:37] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:45:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:37 smithi022 ceph-mon[31360]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3007737607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:37 smithi022 ceph-mon[40250]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3007737607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:38.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:39 smithi122 ceph-mon[32255]: pgmap v832: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:39.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:39 smithi022 ceph-mon[31360]: pgmap v832: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:39 smithi022 ceph-mon[40250]: pgmap v832: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:45:41.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:41 smithi122 ceph-mon[32255]: pgmap v833: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:41 smithi022 ceph-mon[31360]: pgmap v833: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:41 smithi022 ceph-mon[40250]: pgmap v833: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:41.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:41.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:41.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:42.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4093057698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4093057698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4093057698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:43 smithi122 ceph-mon[32255]: pgmap v834: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:43 smithi022 ceph-mon[31360]: pgmap v834: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:43 smithi022 ceph-mon[40250]: pgmap v834: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:45 smithi122 ceph-mon[32255]: pgmap v835: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:45.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:45 smithi022 ceph-mon[31360]: pgmap v835: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:45 smithi022 ceph-mon[40250]: pgmap v835: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:46.635 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:46.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:45:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:45:47.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:47.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:47.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:47 smithi022 ceph-mon[31360]: pgmap v836: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2029936117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:47 smithi022 ceph-mon[40250]: pgmap v836: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2029936117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:47 smithi122 ceph-mon[32255]: pgmap v836: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2029936117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:47] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:45:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:49 smithi022 ceph-mon[31360]: pgmap v837: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:49 smithi022 ceph-mon[40250]: pgmap v837: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:49 smithi122 ceph-mon[32255]: pgmap v837: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:45:51.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:51 smithi022 ceph-mon[31360]: pgmap v838: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:51 smithi022 ceph-mon[40250]: pgmap v838: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:51 smithi122 ceph-mon[32255]: pgmap v838: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:52.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:52.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:52.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:53 smithi022 ceph-mon[31360]: pgmap v839: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1346685484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:53 smithi022 ceph-mon[40250]: pgmap v839: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1346685484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:53 smithi122 ceph-mon[32255]: pgmap v839: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1346685484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:55 smithi022 ceph-mon[31360]: pgmap v840: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:55 smithi022 ceph-mon[40250]: pgmap v840: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:55 smithi122 ceph-mon[32255]: pgmap v840: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:56.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:45:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:45:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:45:57.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:57 smithi022 ceph-mon[31360]: pgmap v841: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:45:57] "GET /metrics HTTP/1.1" 200 202035 "" "Prometheus/2.18.1" 2021-12-15T17:45:57.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:57 smithi022 ceph-mon[40250]: pgmap v841: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:57 smithi122 ceph-mon[32255]: pgmap v841: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:45:57.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:45:57.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:45:57.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:45:58.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2821103773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2821103773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:45:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2821103773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:45:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:45:59 smithi022 ceph-mon[31360]: pgmap v842: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:45:59 smithi022 ceph-mon[40250]: pgmap v842: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:45:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:45:59 smithi122 ceph-mon[32255]: pgmap v842: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:46:01.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:01 smithi022 ceph-mon[31360]: pgmap v843: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:01 smithi022 ceph-mon[40250]: pgmap v843: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:01 smithi122 ceph-mon[32255]: pgmap v843: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:02.748 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:03.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:03.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:03.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:03 smithi022 ceph-mon[31360]: pgmap v844: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1722052887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:03 smithi022 ceph-mon[40250]: pgmap v844: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1722052887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:03 smithi122 ceph-mon[32255]: pgmap v844: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1722052887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:46:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:46:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:46:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:46:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:46:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:46:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:05 smithi022 ceph-mon[31360]: pgmap v845: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:05 smithi022 ceph-mon[40250]: pgmap v845: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:05 smithi122 ceph-mon[32255]: pgmap v845: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:06.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:46:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:46:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:07 smithi022 ceph-mon[31360]: pgmap v846: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:07] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:46:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:07 smithi022 ceph-mon[40250]: pgmap v846: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:07 smithi122 ceph-mon[32255]: pgmap v846: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:08.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:08.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:08.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:08.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:08.492 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:08.492 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:09 smithi022 ceph-mon[40250]: pgmap v847: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/463361600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:09 smithi022 ceph-mon[31360]: pgmap v847: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/463361600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:09 smithi122 ceph-mon[32255]: pgmap v847: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/463361600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:46:11.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:11 smithi022 ceph-mon[40250]: pgmap v848: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:11 smithi022 ceph-mon[31360]: pgmap v848: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:11 smithi122 ceph-mon[32255]: pgmap v848: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:13.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:13.493 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:13 smithi022 ceph-mon[31360]: pgmap v849: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:13.494 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:13 smithi022 ceph-mon[40250]: pgmap v849: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:13 smithi122 ceph-mon[32255]: pgmap v849: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:13.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2957151402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2957151402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2957151402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:15 smithi022 ceph-mon[31360]: pgmap v850: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:15 smithi022 ceph-mon[40250]: pgmap v850: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:15 smithi122 ceph-mon[32255]: pgmap v850: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:16.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:46:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:46:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:17 smithi022 ceph-mon[31360]: pgmap v851: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:17] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:46:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:17 smithi022 ceph-mon[40250]: pgmap v851: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:17 smithi122 ceph-mon[32255]: pgmap v851: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:18.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:18.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:19.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:19.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:19.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:19.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:19 smithi022 ceph-mon[31360]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/548547007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:19 smithi022 ceph-mon[40250]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/548547007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:19 smithi122 ceph-mon[32255]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/548547007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:46:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:21 smithi022 ceph-mon[31360]: pgmap v853: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:21 smithi022 ceph-mon[40250]: pgmap v853: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:21 smithi122 ceph-mon[32255]: pgmap v853: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:23 smithi022 ceph-mon[31360]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:23 smithi022 ceph-mon[40250]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:23 smithi122 ceph-mon[32255]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:24.229 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:24.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:24.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:24.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[31360]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2652623061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:46:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:46:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:46:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[40250]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2652623061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:46:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:46:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:25 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:46:25.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:25 smithi122 ceph-mon[32255]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:25.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2652623061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:25.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:46:25.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:46:25.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:25 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:46:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:46:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:46:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:46:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:46:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:46:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:27 smithi022 ceph-mon[31360]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:27] "GET /metrics HTTP/1.1" 200 202033 "" "Prometheus/2.18.1" 2021-12-15T17:46:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:27 smithi022 ceph-mon[40250]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:27 smithi122 ceph-mon[32255]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:28.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:29.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:29 smithi022 ceph-mon[31360]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:29 smithi022 ceph-mon[40250]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:29 smithi122 ceph-mon[32255]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:29.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:29.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:29.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:46:30.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:30 smithi022 ceph-mon[31360]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2971515375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:30 smithi022 ceph-mon[40250]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2971515375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:30.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:30 smithi122 ceph-mon[32255]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2971515375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:32.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:32 smithi122 ceph-mon[32255]: pgmap v859: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:33.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:32 smithi022 ceph-mon[31360]: pgmap v859: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:32 smithi022 ceph-mon[40250]: pgmap v859: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:34 smithi122 ceph-mon[32255]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:34.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:34.976 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:34 smithi022 ceph-mon[31360]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:34.976 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:34 smithi022 ceph-mon[40250]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:35.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:35.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:35.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2401970442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:36.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2401970442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2401970442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:36.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:46:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:46:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:37 smithi122 ceph-mon[32255]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:37.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:37 smithi022 ceph-mon[31360]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:37] "GET /metrics HTTP/1.1" 200 202030 "" "Prometheus/2.18.1" 2021-12-15T17:46:37.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:37 smithi022 ceph-mon[40250]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:38.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:39 smithi122 ceph-mon[32255]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:39 smithi022 ceph-mon[31360]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:39 smithi022 ceph-mon[40250]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:40.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:46:40.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:40.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:40.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:41 smithi122 ceph-mon[32255]: pgmap v863: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3078388490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:41 smithi022 ceph-mon[31360]: pgmap v863: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3078388490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:41 smithi022 ceph-mon[40250]: pgmap v863: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3078388490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:43 smithi122 ceph-mon[32255]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:43 smithi022 ceph-mon[31360]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:43 smithi022 ceph-mon[40250]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:45 smithi122 ceph-mon[32255]: pgmap v865: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:45 smithi022 ceph-mon[31360]: pgmap v865: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:45 smithi022 ceph-mon[40250]: pgmap v865: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:45.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:46.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:46.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:46.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2417898946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:46.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2417898946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2417898946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:46.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:46:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:46:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:47 smithi122 ceph-mon[32255]: pgmap v866: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:47.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:47 smithi022 ceph-mon[31360]: pgmap v866: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:47] "GET /metrics HTTP/1.1" 200 202030 "" "Prometheus/2.18.1" 2021-12-15T17:46:47.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:47 smithi022 ceph-mon[40250]: pgmap v866: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:49 smithi122 ceph-mon[32255]: pgmap v867: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:49 smithi022 ceph-mon[31360]: pgmap v867: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:49 smithi022 ceph-mon[40250]: pgmap v867: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:46:51.085 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:51 smithi122 ceph-mon[32255]: pgmap v868: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:51.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:51.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:51.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:51.453 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:51 smithi022 ceph-mon[31360]: pgmap v868: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:51.454 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:51 smithi022 ceph-mon[40250]: pgmap v868: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1217083413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1217083413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1217083413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:53 smithi022 ceph-mon[31360]: pgmap v869: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:53 smithi022 ceph-mon[40250]: pgmap v869: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:53 smithi122 ceph-mon[32255]: pgmap v869: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:55 smithi022 ceph-mon[31360]: pgmap v870: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:55 smithi022 ceph-mon[40250]: pgmap v870: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:55 smithi122 ceph-mon[32255]: pgmap v870: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:56.454 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:46:56.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:46:56.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:46:56.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:46:56.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:46:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:46:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:46:57] "GET /metrics HTTP/1.1" 200 202033 "" "Prometheus/2.18.1" 2021-12-15T17:46:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:57 smithi022 ceph-mon[31360]: pgmap v871: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2181862451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:57 smithi022 ceph-mon[40250]: pgmap v871: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2181862451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:57 smithi122 ceph-mon[32255]: pgmap v871: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:46:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2181862451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:46:58.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:46:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:46:59 smithi022 ceph-mon[31360]: pgmap v872: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:46:59 smithi022 ceph-mon[40250]: pgmap v872: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:46:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:46:59 smithi122 ceph-mon[32255]: pgmap v872: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:47:01.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:01 smithi022 ceph-mon[31360]: pgmap v873: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:01 smithi022 ceph-mon[40250]: pgmap v873: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:01 smithi122 ceph-mon[32255]: pgmap v873: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:01.825 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:02.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:02.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:02.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/771267851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/771267851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/771267851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:03.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:03 smithi022 ceph-mon[31360]: pgmap v874: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:03 smithi022 ceph-mon[40250]: pgmap v874: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:03 smithi122 ceph-mon[32255]: pgmap v874: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:04.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:47:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:47:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:47:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:47:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:47:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:47:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:05 smithi022 ceph-mon[31360]: pgmap v875: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:05 smithi022 ceph-mon[40250]: pgmap v875: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:05 smithi122 ceph-mon[32255]: pgmap v875: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:06.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:47:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:47:07.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:07.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:07 smithi022 ceph-mon[31360]: pgmap v876: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:07] "GET /metrics HTTP/1.1" 200 202028 "" "Prometheus/2.18.1" 2021-12-15T17:47:07.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:07 smithi022 ceph-mon[40250]: pgmap v876: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:07.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:07.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:07.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:07 smithi122 ceph-mon[32255]: pgmap v876: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:08.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3775660513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3775660513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3775660513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:09.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:09 smithi022 ceph-mon[31360]: pgmap v877: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:09 smithi022 ceph-mon[40250]: pgmap v877: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:09 smithi122 ceph-mon[32255]: pgmap v877: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:47:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:11 smithi022 ceph-mon[31360]: pgmap v878: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:11 smithi022 ceph-mon[40250]: pgmap v878: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:11 smithi122 ceph-mon[32255]: pgmap v878: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:12.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:12.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:12.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:12.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:13 smithi022 ceph-mon[31360]: pgmap v879: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2636677107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:13 smithi022 ceph-mon[40250]: pgmap v879: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2636677107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:13 smithi122 ceph-mon[32255]: pgmap v879: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2636677107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:15.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:15 smithi022 ceph-mon[31360]: pgmap v880: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:15 smithi022 ceph-mon[40250]: pgmap v880: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:15 smithi122 ceph-mon[32255]: pgmap v880: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:16.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:47:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:47:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:17 smithi022 ceph-mon[31360]: pgmap v881: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:17] "GET /metrics HTTP/1.1" 200 202028 "" "Prometheus/2.18.1" 2021-12-15T17:47:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:17 smithi022 ceph-mon[40250]: pgmap v881: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:17 smithi122 ceph-mon[32255]: pgmap v881: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:17.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:18.241 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:18.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:18.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:18.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:18.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:19 smithi022 ceph-mon[31360]: pgmap v882: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1182874893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:19 smithi022 ceph-mon[40250]: pgmap v882: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1182874893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:19 smithi122 ceph-mon[32255]: pgmap v882: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1182874893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:47:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:21 smithi022 ceph-mon[31360]: pgmap v883: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:21 smithi022 ceph-mon[40250]: pgmap v883: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:21 smithi122 ceph-mon[32255]: pgmap v883: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:23.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:23.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:23 smithi022 ceph-mon[40250]: pgmap v884: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:23 smithi022 ceph-mon[31360]: pgmap v884: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:23 smithi122 ceph-mon[32255]: pgmap v884: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:23.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/903641686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/903641686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/903641686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:25.509 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:25 smithi022 ceph-mon[31360]: pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:25.510 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:25 smithi022 ceph-mon[40250]: pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:25.519 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:25 smithi122 ceph-mon[32255]: pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:47:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:47:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:47:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:47:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:47:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:47:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:47:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:47:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:47:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:47:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:47:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:47:26.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:47:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:47:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:27 smithi022 ceph-mon[31360]: pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:27] "GET /metrics HTTP/1.1" 200 202034 "" "Prometheus/2.18.1" 2021-12-15T17:47:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:27 smithi022 ceph-mon[40250]: pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:27 smithi122 ceph-mon[32255]: pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:28.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:28.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:29.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:29.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:29.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:29 smithi022 ceph-mon[31360]: pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3842100962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:29 smithi022 ceph-mon[40250]: pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3842100962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:29 smithi122 ceph-mon[32255]: pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3842100962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:47:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:31 smithi022 ceph-mon[31360]: pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:31 smithi022 ceph-mon[40250]: pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:31 smithi122 ceph-mon[32255]: pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:33.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:33 smithi022 ceph-mon[31360]: pgmap v889: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:33 smithi022 ceph-mon[40250]: pgmap v889: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:33 smithi122 ceph-mon[32255]: pgmap v889: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:34.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:34.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:34.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:34.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:35 smithi122 ceph-mon[32255]: pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/93457474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:35 smithi022 ceph-mon[31360]: pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/93457474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:35 smithi022 ceph-mon[40250]: pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/93457474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:37.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:47:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:47:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:37 smithi122 ceph-mon[32255]: pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:37 smithi022 ceph-mon[31360]: pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:37] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:47:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:37 smithi022 ceph-mon[40250]: pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:38.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:39.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:39 smithi122 ceph-mon[32255]: pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:39 smithi022 ceph-mon[31360]: pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:39 smithi022 ceph-mon[40250]: pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:39.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:39.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:39.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:47:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2574050307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:40.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2574050307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2574050307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:41 smithi122 ceph-mon[32255]: pgmap v893: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:41.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:41 smithi022 ceph-mon[31360]: pgmap v893: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:41 smithi022 ceph-mon[40250]: pgmap v893: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:43 smithi122 ceph-mon[32255]: pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:43 smithi022 ceph-mon[31360]: pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:43 smithi022 ceph-mon[40250]: pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:44.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:45.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:45 smithi022 ceph-mon[31360]: pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/662480471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:45 smithi022 ceph-mon[40250]: pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/662480471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:45 smithi122 ceph-mon[32255]: pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/662480471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:47.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:47:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:47:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:47 smithi122 ceph-mon[32255]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:47 smithi022 ceph-mon[31360]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:47] "GET /metrics HTTP/1.1" 200 202031 "" "Prometheus/2.18.1" 2021-12-15T17:47:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:47 smithi022 ceph-mon[40250]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:49 smithi122 ceph-mon[32255]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:49 smithi022 ceph-mon[31360]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:49 smithi022 ceph-mon[40250]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:50.182 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:47:50.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:51 smithi122 ceph-mon[32255]: pgmap v898: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1165229426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:51 smithi022 ceph-mon[31360]: pgmap v898: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1165229426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:51 smithi022 ceph-mon[40250]: pgmap v898: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:47:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1165229426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:53 smithi122 ceph-mon[32255]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:53 smithi022 ceph-mon[31360]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:53 smithi022 ceph-mon[40250]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:55.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:47:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:55 smithi122 ceph-mon[32255]: pgmap v900: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:55 smithi022 ceph-mon[31360]: pgmap v900: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:55 smithi022 ceph-mon[40250]: pgmap v900: 132 pgs: 132 active+clean; 582 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:47:55.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:47:55.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:47:55.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:47:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/529851935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:56.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/529851935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/529851935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:47:57.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:47:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:47:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:57 smithi122 ceph-mon[32255]: pgmap v901: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:47:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:47:57] "GET /metrics HTTP/1.1" 200 202036 "" "Prometheus/2.18.1" 2021-12-15T17:47:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:57 smithi022 ceph-mon[31360]: pgmap v901: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:47:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:57 smithi022 ceph-mon[40250]: pgmap v901: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:47:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:47:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:47:59 smithi122 ceph-mon[32255]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:47:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:47:59 smithi022 ceph-mon[31360]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:47:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:47:59 smithi022 ceph-mon[40250]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:48:00.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:01.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:01.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:01.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:01 smithi122 ceph-mon[32255]: pgmap v903: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3680390257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:01 smithi022 ceph-mon[31360]: pgmap v903: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3680390257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:01 smithi022 ceph-mon[40250]: pgmap v903: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3680390257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:03 smithi122 ceph-mon[32255]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:48:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:48:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:03 smithi022 ceph-mon[31360]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:48:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:48:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:03 smithi022 ceph-mon[40250]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:48:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:48:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:05 smithi122 ceph-mon[32255]: pgmap v905: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:05 smithi022 ceph-mon[31360]: pgmap v905: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:05 smithi022 ceph-mon[40250]: pgmap v905: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:06.285 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:06.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:06.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:06.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:07.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:48:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:48:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:07 smithi122 ceph-mon[32255]: pgmap v906: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3656649040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:07] "GET /metrics HTTP/1.1" 200 202047 "" "Prometheus/2.18.1" 2021-12-15T17:48:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:07 smithi022 ceph-mon[31360]: pgmap v906: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3656649040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:07 smithi022 ceph-mon[40250]: pgmap v906: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:48:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3656649040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:09 smithi122 ceph-mon[32255]: pgmap v907: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:09 smithi022 ceph-mon[31360]: pgmap v907: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:09 smithi022 ceph-mon[40250]: pgmap v907: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:48:11.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:11.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:11 smithi022 ceph-mon[40250]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:11.655 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:11 smithi022 ceph-mon[31360]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:11 smithi122 ceph-mon[32255]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:12.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:12.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:12.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1220388716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:12.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1220388716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:12.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1220388716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:13 smithi122 ceph-mon[32255]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:13 smithi022 ceph-mon[31360]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:13 smithi022 ceph-mon[40250]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:15 smithi122 ceph-mon[32255]: pgmap v910: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:15 smithi022 ceph-mon[31360]: pgmap v910: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:15 smithi022 ceph-mon[40250]: pgmap v910: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:17.025 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:17.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:48:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:48:17.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:17.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:17.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:17 smithi122 ceph-mon[32255]: pgmap v911: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:17 smithi022 ceph-mon[31360]: pgmap v911: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:17] "GET /metrics HTTP/1.1" 200 202047 "" "Prometheus/2.18.1" 2021-12-15T17:48:17.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:17 smithi022 ceph-mon[40250]: pgmap v911: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3980254694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3980254694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3980254694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:19 smithi122 ceph-mon[32255]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:19 smithi022 ceph-mon[31360]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:19 smithi022 ceph-mon[40250]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:48:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:21 smithi122 ceph-mon[32255]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:21.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:21 smithi022 ceph-mon[31360]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:21 smithi022 ceph-mon[40250]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:22.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:22.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:22.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:22.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:23 smithi122 ceph-mon[32255]: pgmap v914: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1254853006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:23 smithi022 ceph-mon[31360]: pgmap v914: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1254853006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:23 smithi022 ceph-mon[40250]: pgmap v914: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1254853006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:25 smithi122 ceph-mon[32255]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:25 smithi022 ceph-mon[31360]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:25 smithi022 ceph-mon[40250]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:48:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:48:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:26 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:48:26.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:48:26.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:48:26.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:26 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:48:26.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:48:26.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:48:26.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:26 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:48:27.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:48:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:48:27.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:27 smithi122 ceph-mon[32255]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:27.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:27 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:27.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:27 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:27.557 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:27] "GET /metrics HTTP/1.1" 200 202053 "" "Prometheus/2.18.1" 2021-12-15T17:48:27.558 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:27 smithi022 ceph-mon[31360]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:27.558 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:27 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:27.558 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:27 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:27.558 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:27 smithi022 ceph-mon[40250]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:27.559 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:27 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:27.559 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:27 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:27.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:28.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:28.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:28.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2781873099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:28.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2781873099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2781873099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:29 smithi122 ceph-mon[32255]: pgmap v917: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:29 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:29 smithi022 ceph-mon[31360]: pgmap v917: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:29 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:29 smithi022 ceph-mon[40250]: pgmap v917: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:29 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:48:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:30 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:30 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:30.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:30 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:30 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:30 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:30 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:48:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:31 smithi122 ceph-mon[32255]: pgmap v918: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:31 smithi022 ceph-mon[31360]: pgmap v918: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:31 smithi022 ceph-mon[40250]: pgmap v918: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:33.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:33.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:33.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:33.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:33 smithi122 ceph-mon[32255]: pgmap v919: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:33 smithi022 ceph-mon[31360]: pgmap v919: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:33 smithi022 ceph-mon[40250]: pgmap v919: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1088634909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:34.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1088634909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1088634909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:35 smithi122 ceph-mon[32255]: pgmap v920: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:35 smithi022 ceph-mon[31360]: pgmap v920: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:35 smithi022 ceph-mon[40250]: pgmap v920: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:37.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:48:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:48:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:37 smithi122 ceph-mon[32255]: pgmap v921: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:37 smithi022 ceph-mon[31360]: pgmap v921: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:37] "GET /metrics HTTP/1.1" 200 202053 "" "Prometheus/2.18.1" 2021-12-15T17:48:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:37 smithi022 ceph-mon[40250]: pgmap v921: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:38.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:38.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:38.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:38.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:38.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:39 smithi122 ceph-mon[32255]: pgmap v922: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1587659767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:39 smithi022 ceph-mon[31360]: pgmap v922: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1587659767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:39 smithi022 ceph-mon[40250]: pgmap v922: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1587659767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:48:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:41 smithi122 ceph-mon[32255]: pgmap v923: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:41.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:41 smithi022 ceph-mon[31360]: pgmap v923: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:41 smithi022 ceph-mon[40250]: pgmap v923: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:43 smithi122 ceph-mon[32255]: pgmap v924: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:43 smithi022 ceph-mon[31360]: pgmap v924: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:43 smithi022 ceph-mon[40250]: pgmap v924: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:43.882 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:44.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:44.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:44.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:44.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1840364673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1840364673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1840364673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:45 smithi122 ceph-mon[32255]: pgmap v925: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:45.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:45 smithi022 ceph-mon[31360]: pgmap v925: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:45 smithi022 ceph-mon[40250]: pgmap v925: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:47.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:48:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:48:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:47 smithi122 ceph-mon[32255]: pgmap v926: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:47 smithi022 ceph-mon[31360]: pgmap v926: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:47] "GET /metrics HTTP/1.1" 200 202053 "" "Prometheus/2.18.1" 2021-12-15T17:48:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:47 smithi022 ceph-mon[40250]: pgmap v926: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:49.253 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:49 smithi022 ceph-mon[31360]: pgmap v927: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:49 smithi022 ceph-mon[40250]: pgmap v927: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:49.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:49.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:49.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:49 smithi122 ceph-mon[32255]: pgmap v927: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:48:50.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1877562250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1877562250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1877562250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:51 smithi022 ceph-mon[31360]: pgmap v928: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:51 smithi022 ceph-mon[40250]: pgmap v928: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:51 smithi122 ceph-mon[32255]: pgmap v928: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:53 smithi022 ceph-mon[31360]: pgmap v929: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:53 smithi022 ceph-mon[40250]: pgmap v929: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:53 smithi122 ceph-mon[32255]: pgmap v929: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:54.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:48:54.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:48:54.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:48:54.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:48:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:55 smithi022 ceph-mon[31360]: pgmap v930: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3530331472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:55 smithi022 ceph-mon[40250]: pgmap v930: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3530331472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:55.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:55 smithi122 ceph-mon[32255]: pgmap v930: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3530331472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:48:57.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:48:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:48:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:57 smithi022 ceph-mon[31360]: pgmap v931: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:48:57] "GET /metrics HTTP/1.1" 200 202048 "" "Prometheus/2.18.1" 2021-12-15T17:48:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:57 smithi022 ceph-mon[40250]: pgmap v931: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:57 smithi122 ceph-mon[32255]: pgmap v931: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:48:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:48:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:48:59 smithi022 ceph-mon[31360]: pgmap v932: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:48:59 smithi022 ceph-mon[40250]: pgmap v932: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:48:59 smithi122 ceph-mon[32255]: pgmap v932: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:48:59.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:00.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:00.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:00.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:49:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:01 smithi122 ceph-mon[32255]: pgmap v933: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1188460324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:01.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:01 smithi022 ceph-mon[31360]: pgmap v933: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1188460324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:01 smithi022 ceph-mon[40250]: pgmap v933: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1188460324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:03.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:03 smithi022 ceph-mon[40250]: pgmap v934: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:03 smithi022 ceph-mon[31360]: pgmap v934: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:03 smithi122 ceph-mon[32255]: pgmap v934: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:04.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:49:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:49:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:49:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:49:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:49:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:49:05.366 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:05 smithi022 ceph-mon[31360]: pgmap v935: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:05 smithi022 ceph-mon[40250]: pgmap v935: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:05 smithi122 ceph-mon[32255]: pgmap v935: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:05.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:05.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:05.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/53360109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/53360109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/53360109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:07.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:49:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:49:07.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:07 smithi022 ceph-mon[31360]: pgmap v936: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:07] "GET /metrics HTTP/1.1" 200 202046 "" "Prometheus/2.18.1" 2021-12-15T17:49:07.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:07 smithi022 ceph-mon[40250]: pgmap v936: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:07 smithi122 ceph-mon[32255]: pgmap v936: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:08.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:09.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:09 smithi022 ceph-mon[31360]: pgmap v937: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:09 smithi022 ceph-mon[40250]: pgmap v937: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:09 smithi122 ceph-mon[32255]: pgmap v937: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:49:10.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:11.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:11 smithi022 ceph-mon[31360]: pgmap v938: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2334860217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:11 smithi022 ceph-mon[40250]: pgmap v938: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2334860217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:11 smithi122 ceph-mon[32255]: pgmap v938: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2334860217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:13 smithi022 ceph-mon[31360]: pgmap v939: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:13 smithi022 ceph-mon[40250]: pgmap v939: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:13 smithi122 ceph-mon[32255]: pgmap v939: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:15.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:15 smithi022 ceph-mon[31360]: pgmap v940: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:15 smithi022 ceph-mon[40250]: pgmap v940: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:15 smithi122 ceph-mon[32255]: pgmap v940: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:16.106 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:16.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:16.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:16.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:17.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:49:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:49:17.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:17] "GET /metrics HTTP/1.1" 200 202046 "" "Prometheus/2.18.1" 2021-12-15T17:49:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:17 smithi022 ceph-mon[31360]: pgmap v941: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1076927117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:17 smithi022 ceph-mon[40250]: pgmap v941: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1076927117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:17 smithi122 ceph-mon[32255]: pgmap v941: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1076927117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:18.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:19.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:19 smithi022 ceph-mon[31360]: pgmap v942: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:19 smithi022 ceph-mon[40250]: pgmap v942: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:19 smithi122 ceph-mon[32255]: pgmap v942: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:49:21.478 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:21.479 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:21 smithi022 ceph-mon[31360]: pgmap v943: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:21.479 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:21 smithi022 ceph-mon[40250]: pgmap v943: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:21 smithi122 ceph-mon[32255]: pgmap v943: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:21.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:21.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:21.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3265320896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3265320896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3265320896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:23 smithi022 ceph-mon[31360]: pgmap v944: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:23 smithi022 ceph-mon[40250]: pgmap v944: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:23 smithi122 ceph-mon[32255]: pgmap v944: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:25 smithi022 ceph-mon[31360]: pgmap v945: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:25 smithi022 ceph-mon[40250]: pgmap v945: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:25 smithi122 ceph-mon[32255]: pgmap v945: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:26.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:49:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:49:27.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:27.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:27.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:27.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:27] "GET /metrics HTTP/1.1" 200 202051 "" "Prometheus/2.18.1" 2021-12-15T17:49:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:27 smithi022 ceph-mon[31360]: pgmap v946: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3289304408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:27 smithi022 ceph-mon[40250]: pgmap v946: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3289304408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:27 smithi122 ceph-mon[32255]: pgmap v946: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3289304408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:28.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:29.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:29 smithi022 ceph-mon[31360]: pgmap v947: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:29 smithi022 ceph-mon[40250]: pgmap v947: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:29 smithi122 ceph-mon[32255]: pgmap v947: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:30.247 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:49:30.247 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:49:30.248 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:49:30.248 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:49:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:49:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:30 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:49:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:30 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:49:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:30 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:49:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:30 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:49:30.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:49:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:49:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:49:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:30 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:49:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:31 smithi022 ceph-mon[31360]: pgmap v948: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:31 smithi022 ceph-mon[40250]: pgmap v948: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:31 smithi122 ceph-mon[32255]: pgmap v948: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:32.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:32.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:32.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:32.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:33.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:33 smithi022 ceph-mon[31360]: pgmap v949: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2785210660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:33 smithi022 ceph-mon[40250]: pgmap v949: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2785210660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:33 smithi122 ceph-mon[32255]: pgmap v949: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2785210660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:35.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:35 smithi022 ceph-mon[31360]: pgmap v950: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:35 smithi022 ceph-mon[40250]: pgmap v950: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:35 smithi122 ceph-mon[32255]: pgmap v950: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:37.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:49:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:49:37.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:37] "GET /metrics HTTP/1.1" 200 202055 "" "Prometheus/2.18.1" 2021-12-15T17:49:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:37 smithi022 ceph-mon[31360]: pgmap v951: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:37.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:37 smithi022 ceph-mon[40250]: pgmap v951: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:37.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:37 smithi122 ceph-mon[32255]: pgmap v951: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:37.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:37.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:37.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1484823878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1484823878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1484823878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:39.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:39 smithi022 ceph-mon[31360]: pgmap v952: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:39 smithi022 ceph-mon[40250]: pgmap v952: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:39 smithi122 ceph-mon[32255]: pgmap v952: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:49:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:49:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:41 smithi022 ceph-mon[31360]: pgmap v953: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:41 smithi022 ceph-mon[40250]: pgmap v953: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:41 smithi122 ceph-mon[32255]: pgmap v953: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:42.954 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:43.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:43.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:43.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:43 smithi022 ceph-mon[31360]: pgmap v954: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:43 smithi022 ceph-mon[40250]: pgmap v954: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:43 smithi122 ceph-mon[32255]: pgmap v954: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2250821226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2250821226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2250821226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:45 smithi022 ceph-mon[31360]: pgmap v955: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:45 smithi022 ceph-mon[40250]: pgmap v955: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:45 smithi122 ceph-mon[32255]: pgmap v955: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:47.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:49:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:49:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:47 smithi022 ceph-mon[31360]: pgmap v956: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:47] "GET /metrics HTTP/1.1" 200 202055 "" "Prometheus/2.18.1" 2021-12-15T17:49:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:47 smithi022 ceph-mon[40250]: pgmap v956: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:47 smithi122 ceph-mon[32255]: pgmap v956: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:48.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:48.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:48.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:48.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:48.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:48.690 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:49 smithi122 ceph-mon[32255]: pgmap v957: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3171539587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:49 smithi022 ceph-mon[31360]: pgmap v957: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3171539587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:49 smithi022 ceph-mon[40250]: pgmap v957: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3171539587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:49:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:51 smithi122 ceph-mon[32255]: pgmap v958: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:51 smithi022 ceph-mon[31360]: pgmap v958: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:51 smithi022 ceph-mon[40250]: pgmap v958: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:53 smithi122 ceph-mon[32255]: pgmap v959: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:53.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:53.691 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:53 smithi022 ceph-mon[31360]: pgmap v959: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:53.691 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:53 smithi022 ceph-mon[40250]: pgmap v959: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:54.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:54.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:54.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:54.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3770754119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3770754119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3770754119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:49:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:55 smithi122 ceph-mon[32255]: pgmap v960: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:55 smithi022 ceph-mon[31360]: pgmap v960: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:55 smithi022 ceph-mon[40250]: pgmap v960: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:57.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:49:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:49:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:57 smithi122 ceph-mon[32255]: pgmap v961: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:57 smithi022 ceph-mon[31360]: pgmap v961: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:49:57] "GET /metrics HTTP/1.1" 200 202050 "" "Prometheus/2.18.1" 2021-12-15T17:49:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:57 smithi022 ceph-mon[40250]: pgmap v961: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:49:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:49:59.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:49:59.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:49:59.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:49:59.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:49:59.426 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:49:59 smithi022 ceph-mon[31360]: pgmap v962: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:59.427 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:49:59 smithi022 ceph-mon[40250]: pgmap v962: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:49:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:49:59 smithi122 ceph-mon[32255]: pgmap v962: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:50:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1298866778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T17:50:00.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1298866778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T17:50:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1298866778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T17:50:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:01 smithi122 ceph-mon[32255]: pgmap v963: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:01 smithi022 ceph-mon[31360]: pgmap v963: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:01 smithi022 ceph-mon[40250]: pgmap v963: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:03 smithi122 ceph-mon[32255]: pgmap v964: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:03 smithi022 ceph-mon[31360]: pgmap v964: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:03 smithi022 ceph-mon[40250]: pgmap v964: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:04.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:50:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:50:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:50:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:50:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:50:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:50:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:50:04.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:50:04.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:50:04.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:05 smithi122 ceph-mon[32255]: pgmap v965: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1427296157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:05 smithi022 ceph-mon[31360]: pgmap v965: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1427296157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:05 smithi022 ceph-mon[40250]: pgmap v965: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1427296157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:07.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:50:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:50:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:07 smithi122 ceph-mon[32255]: pgmap v966: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:07 smithi022 ceph-mon[31360]: pgmap v966: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:07] "GET /metrics HTTP/1.1" 200 202049 "" "Prometheus/2.18.1" 2021-12-15T17:50:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:07 smithi022 ceph-mon[40250]: pgmap v966: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:09 smithi122 ceph-mon[32255]: pgmap v967: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:09 smithi022 ceph-mon[31360]: pgmap v967: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:09 smithi022 ceph-mon[40250]: pgmap v967: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:09.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:50:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:50:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:50:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:50:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2249592514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:10.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2249592514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2249592514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:11 smithi122 ceph-mon[32255]: pgmap v968: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:11 smithi022 ceph-mon[31360]: pgmap v968: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:11 smithi022 ceph-mon[40250]: pgmap v968: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:13 smithi122 ceph-mon[32255]: pgmap v969: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:13 smithi022 ceph-mon[31360]: pgmap v969: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:13 smithi022 ceph-mon[40250]: pgmap v969: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:15.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:50:15.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:15 smithi022 ceph-mon[31360]: pgmap v970: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:15 smithi022 ceph-mon[40250]: pgmap v970: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:15.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:50:15.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:50:15.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:15 smithi122 ceph-mon[32255]: pgmap v970: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3484335407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:16.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3484335407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3484335407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:17.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:50:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:50:17.594 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:17] "GET /metrics HTTP/1.1" 200 202049 "" "Prometheus/2.18.1" 2021-12-15T17:50:17.595 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:17 smithi022 ceph-mon[31360]: pgmap v971: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:17.595 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:17 smithi022 ceph-mon[40250]: pgmap v971: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:17 smithi122 ceph-mon[32255]: pgmap v971: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:19 smithi122 ceph-mon[32255]: pgmap v972: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:19.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:19 smithi022 ceph-mon[31360]: pgmap v972: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:19 smithi022 ceph-mon[40250]: pgmap v972: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:50:20.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:50:20.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== 2021-12-15T17:50:20.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== ']' 2021-12-15T17:50:20.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:21 smithi122 ceph-mon[32255]: pgmap v973: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3640834801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:21.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:21 smithi022 ceph-mon[31360]: pgmap v973: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3640834801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:21 smithi022 ceph-mon[40250]: pgmap v973: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3640834801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:23 smithi122 ceph-mon[32255]: pgmap v974: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:23 smithi022 ceph-mon[31360]: pgmap v974: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:23 smithi022 ceph-mon[40250]: pgmap v974: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:25.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:24 smithi022 ceph-mon[31360]: pgmap v975: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:24 smithi022 ceph-mon[40250]: pgmap v975: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:25.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:24 smithi122 ceph-mon[32255]: pgmap v975: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:25.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.6 2021-12-15T17:50:26.281 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for osd.7 2021-12-15T17:50:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQD+KLphfwlGNhAAAHjsltwFOciNf4bC+2/K2Q== 2021-12-15T17:50:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDoIrphobPeMRAA20ZYo9a+1wTqLqnkxBqMMw== == AQD+KLphfwlGNhAAAHjsltwFOciNf4bC+2/K2Q== ']' 2021-12-15T17:50:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:50:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for osd.7' 2021-12-15T17:50:26.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:50:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key osd.7 2021-12-15T17:50:26.925 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key osd.7 on host 'smithi122' 2021-12-15T17:50:26.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:50:26.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:27.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:50:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:50:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:27] "GET /metrics HTTP/1.1" 200 202047 "" "Prometheus/2.18.1" 2021-12-15T17:50:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[31360]: pgmap v976: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2056192294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1250830062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:50:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:50:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:50:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[40250]: pgmap v976: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2056192294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1250830062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:50:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:50:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:27 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:50:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:27 smithi122 ceph-mon[32255]: pgmap v976: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2056192294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2021-12-15T17:50:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1250830062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:27 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:50:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:27 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:50:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:27 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:50:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:28 smithi022 ceph-mon[31360]: from='client.27046 v1:172.21.15.22:0/1860471660' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:50:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:28 smithi022 ceph-mon[31360]: Schedule rotate-key daemon osd.7 2021-12-15T17:50:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:28 smithi022 ceph-mon[40250]: from='client.27046 v1:172.21.15.22:0/1860471660' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:50:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:28 smithi022 ceph-mon[40250]: Schedule rotate-key daemon osd.7 2021-12-15T17:50:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:28 smithi122 ceph-mon[32255]: from='client.27046 v1:172.21.15.22:0/1860471660' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:50:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:28 smithi122 ceph-mon[32255]: Schedule rotate-key daemon osd.7 2021-12-15T17:50:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:29 smithi022 ceph-mon[31360]: pgmap v977: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:29 smithi022 ceph-mon[40250]: pgmap v977: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:29 smithi122 ceph-mon[32255]: pgmap v977: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:50:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:31 smithi022 ceph-mon[31360]: pgmap v978: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:31 smithi022 ceph-mon[40250]: pgmap v978: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:31 smithi122 ceph-mon[32255]: pgmap v978: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:31.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:50:32.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:32.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:50:32.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2021-12-15T17:50:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2021-12-15T17:50:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1420317153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2021-12-15T17:50:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2021-12-15T17:50:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1420317153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2021-12-15T17:50:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2021-12-15T17:50:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:50:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1420317153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:33 smithi122 ceph-mon[32255]: pgmap v979: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:33 smithi122 ceph-mon[32255]: Rotating authentication key for osd.7 2021-12-15T17:50:33.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:33 smithi022 ceph-mon[31360]: pgmap v979: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:33 smithi022 ceph-mon[31360]: Rotating authentication key for osd.7 2021-12-15T17:50:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:33 smithi022 ceph-mon[40250]: pgmap v979: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:33 smithi022 ceph-mon[40250]: Rotating authentication key for osd.7 2021-12-15T17:50:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:35 smithi122 ceph-mon[32255]: pgmap v980: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:35 smithi022 ceph-mon[31360]: pgmap v980: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:35 smithi022 ceph-mon[40250]: pgmap v980: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:37.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:50:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:50:37.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:50:37.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:37.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:50:37.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:37.679 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:37] "GET /metrics HTTP/1.1" 200 202051 "" "Prometheus/2.18.1" 2021-12-15T17:50:37.679 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:37 smithi022 ceph-mon[31360]: pgmap v981: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:37.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:37 smithi022 ceph-mon[40250]: pgmap v981: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:37 smithi122 ceph-mon[32255]: pgmap v981: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3916389540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3916389540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3916389540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:39 smithi122 ceph-mon[32255]: pgmap v982: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:39 smithi022 ceph-mon[31360]: pgmap v982: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:39 smithi022 ceph-mon[40250]: pgmap v982: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:50:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:41 smithi122 ceph-mon[32255]: pgmap v983: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:41.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:41 smithi022 ceph-mon[31360]: pgmap v983: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:41 smithi022 ceph-mon[40250]: pgmap v983: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:42.680 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:50:43.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:43.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:50:43.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:43 smithi122 ceph-mon[32255]: pgmap v984: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:43.789 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2406636270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:43.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:43 smithi022 ceph-mon[31360]: pgmap v984: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:43.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2406636270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:43.791 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:43 smithi022 ceph-mon[40250]: pgmap v984: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:43.792 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2406636270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:45 smithi122 ceph-mon[32255]: pgmap v985: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:45.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:45 smithi022 ceph-mon[31360]: pgmap v985: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:45 smithi022 ceph-mon[40250]: pgmap v985: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:47.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:50:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:50:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:47 smithi122 ceph-mon[32255]: pgmap v986: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:47 smithi022 ceph-mon[31360]: pgmap v986: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:47] "GET /metrics HTTP/1.1" 200 202051 "" "Prometheus/2.18.1" 2021-12-15T17:50:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:47 smithi022 ceph-mon[40250]: pgmap v986: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:48.048 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:50:48.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:48.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:50:48.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:49 smithi122 ceph-mon[32255]: pgmap v987: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3664387558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:49.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:49 smithi022 ceph-mon[31360]: pgmap v987: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3664387558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:49 smithi022 ceph-mon[40250]: pgmap v987: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3664387558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:50:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:51 smithi122 ceph-mon[32255]: pgmap v988: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:51 smithi022 ceph-mon[31360]: pgmap v988: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:51 smithi022 ceph-mon[40250]: pgmap v988: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:53.434 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:50:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:53 smithi122 ceph-mon[32255]: pgmap v989: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:53 smithi022 ceph-mon[31360]: pgmap v989: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:53 smithi022 ceph-mon[40250]: pgmap v989: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:53.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:53.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:50:53.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1857368501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1857368501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1857368501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:55 smithi122 ceph-mon[32255]: pgmap v990: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:55 smithi022 ceph-mon[31360]: pgmap v990: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:55 smithi022 ceph-mon[40250]: pgmap v990: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:57.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:50:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:50:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:57 smithi122 ceph-mon[32255]: pgmap v991: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:57 smithi022 ceph-mon[31360]: pgmap v991: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:50:57] "GET /metrics HTTP/1.1" 200 202046 "" "Prometheus/2.18.1" 2021-12-15T17:50:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:57 smithi022 ceph-mon[40250]: pgmap v991: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:50:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:58.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:50:58.806 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:50:59.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:50:59.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:50:59.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:50:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:59 smithi022 ceph-mon[31360]: pgmap v992: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:50:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3550653807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:59 smithi022 ceph-mon[40250]: pgmap v992: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:50:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3550653807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:50:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:59 smithi122 ceph-mon[32255]: pgmap v992: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:50:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:50:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3550653807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:51:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:01 smithi122 ceph-mon[32255]: pgmap v993: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:01.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:01 smithi022 ceph-mon[40250]: pgmap v993: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:01 smithi022 ceph-mon[31360]: pgmap v993: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:03 smithi122 ceph-mon[32255]: pgmap v994: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:51:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:51:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:03 smithi022 ceph-mon[31360]: pgmap v994: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:51:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:51:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:03 smithi022 ceph-mon[40250]: pgmap v994: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:51:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:51:04.176 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:04.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:04.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:04.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:05 smithi122 ceph-mon[32255]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2990816283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:05 smithi022 ceph-mon[31360]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2990816283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:05 smithi022 ceph-mon[40250]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2990816283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:07.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:51:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:51:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:07 smithi122 ceph-mon[32255]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:07.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:07 smithi022 ceph-mon[31360]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:07] "GET /metrics HTTP/1.1" 200 202055 "" "Prometheus/2.18.1" 2021-12-15T17:51:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:07 smithi022 ceph-mon[40250]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:09.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:09 smithi122 ceph-mon[32255]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:09 smithi022 ceph-mon[31360]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:09 smithi022 ceph-mon[40250]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:10.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:51:10.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3646202914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:10.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3646202914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3646202914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:11 smithi122 ceph-mon[32255]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:11 smithi022 ceph-mon[31360]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:11 smithi022 ceph-mon[40250]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:13 smithi122 ceph-mon[32255]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:13 smithi022 ceph-mon[31360]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:13 smithi022 ceph-mon[40250]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:14.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:15.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:15.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:15.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:15 smithi122 ceph-mon[32255]: pgmap v1000: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/409461806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:15 smithi022 ceph-mon[31360]: pgmap v1000: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/409461806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:15 smithi022 ceph-mon[40250]: pgmap v1000: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/409461806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:17.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:51:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:51:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:17 smithi122 ceph-mon[32255]: pgmap v1001: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:17 smithi022 ceph-mon[31360]: pgmap v1001: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:17] "GET /metrics HTTP/1.1" 200 202055 "" "Prometheus/2.18.1" 2021-12-15T17:51:17.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:17 smithi022 ceph-mon[40250]: pgmap v1001: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:19.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:19 smithi022 ceph-mon[31360]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:19 smithi022 ceph-mon[40250]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:19 smithi122 ceph-mon[32255]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:20.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:51:20.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:20.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:20.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:21 smithi022 ceph-mon[31360]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/811272826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:21 smithi022 ceph-mon[40250]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/811272826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:21 smithi122 ceph-mon[32255]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/811272826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:23 smithi022 ceph-mon[31360]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:23 smithi022 ceph-mon[40250]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:23.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:23 smithi122 ceph-mon[32255]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:25.662 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:25 smithi022 ceph-mon[31360]: pgmap v1005: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:25 smithi022 ceph-mon[40250]: pgmap v1005: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:25 smithi122 ceph-mon[32255]: pgmap v1005: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:26.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:26.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:26.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:26.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2257963190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2257963190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:26.884 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2257963190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:27.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:51:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:51:27.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:27 smithi022 ceph-mon[31360]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:27] "GET /metrics HTTP/1.1" 200 202057 "" "Prometheus/2.18.1" 2021-12-15T17:51:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:27 smithi022 ceph-mon[40250]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:27 smithi122 ceph-mon[32255]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:28.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:28.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:29 smithi022 ceph-mon[31360]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:29 smithi022 ceph-mon[40250]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:29 smithi122 ceph-mon[32255]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:51:31.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:31.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:31.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:31.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:31 smithi022 ceph-mon[31360]: pgmap v1008: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3251741170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:31 smithi022 ceph-mon[40250]: pgmap v1008: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3251741170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:31 smithi122 ceph-mon[32255]: pgmap v1008: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3251741170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:32.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:51:32.465 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:51:32.465 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:51:32.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:51:32.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:51:32.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:51:32.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:51:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:51:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:51:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:33 smithi022 ceph-mon[31360]: pgmap v1009: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:51:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:33 smithi022 ceph-mon[40250]: pgmap v1009: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:51:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:33 smithi122 ceph-mon[32255]: pgmap v1009: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:51:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:35 smithi022 ceph-mon[31360]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:35 smithi022 ceph-mon[40250]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:35 smithi122 ceph-mon[32255]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:36.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:36.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:36.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:36.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:37.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:51:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:51:37.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:37 smithi022 ceph-mon[31360]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4263480345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:37] "GET /metrics HTTP/1.1" 200 202054 "" "Prometheus/2.18.1" 2021-12-15T17:51:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:37 smithi022 ceph-mon[40250]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4263480345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:37 smithi122 ceph-mon[32255]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4263480345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:38.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:38.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:39 smithi022 ceph-mon[31360]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:39 smithi022 ceph-mon[40250]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:39.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:39 smithi122 ceph-mon[32255]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:51:41.774 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:41.775 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:41 smithi022 ceph-mon[31360]: pgmap v1013: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:41.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:41 smithi022 ceph-mon[40250]: pgmap v1013: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:41 smithi122 ceph-mon[32255]: pgmap v1013: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:42.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:42.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:42.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:42.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2919962349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:42.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2919962349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:42.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2919962349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:43 smithi022 ceph-mon[31360]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:43 smithi022 ceph-mon[40250]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:43.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:43 smithi122 ceph-mon[32255]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:45 smithi022 ceph-mon[31360]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:45 smithi022 ceph-mon[40250]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:45 smithi122 ceph-mon[32255]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:47.145 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:47.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:51:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:51:47.489 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:47] "GET /metrics HTTP/1.1" 200 202054 "" "Prometheus/2.18.1" 2021-12-15T17:51:47.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:47.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:47.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:47 smithi022 ceph-mon[31360]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:47 smithi022 ceph-mon[40250]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:47 smithi122 ceph-mon[32255]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1422648904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1422648904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:48.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1422648904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:49 smithi022 ceph-mon[31360]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:49 smithi022 ceph-mon[40250]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:49.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:49 smithi122 ceph-mon[32255]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:51:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:51 smithi022 ceph-mon[31360]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:51 smithi022 ceph-mon[40250]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:51 smithi122 ceph-mon[32255]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:52.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:52.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:52.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:53 smithi022 ceph-mon[31360]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/580218754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:53 smithi022 ceph-mon[40250]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/580218754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:53 smithi122 ceph-mon[32255]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/580218754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:55 smithi022 ceph-mon[31360]: pgmap v1020: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:55 smithi022 ceph-mon[40250]: pgmap v1020: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:55.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:55 smithi122 ceph-mon[32255]: pgmap v1020: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:57.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:51:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:51:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:57 smithi022 ceph-mon[31360]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:51:57] "GET /metrics HTTP/1.1" 200 202053 "" "Prometheus/2.18.1" 2021-12-15T17:51:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:57 smithi022 ceph-mon[40250]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:57.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:51:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:57 smithi122 ceph-mon[32255]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:51:58.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:51:58.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:51:58.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:51:58.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2923632798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2923632798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:51:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2923632798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:51:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:51:59 smithi022 ceph-mon[40250]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:51:59 smithi022 ceph-mon[31360]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:51:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:51:59 smithi122 ceph-mon[32255]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:52:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:01 smithi022 ceph-mon[31360]: pgmap v1023: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:01 smithi022 ceph-mon[40250]: pgmap v1023: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:01 smithi122 ceph-mon[32255]: pgmap v1023: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:03.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:03.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:03 smithi022 ceph-mon[31360]: pgmap v1024: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:03 smithi022 ceph-mon[40250]: pgmap v1024: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:03.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:03.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:03.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:03 smithi122 ceph-mon[32255]: pgmap v1024: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:52:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:52:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/44477888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:52:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:52:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/44477888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:52:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:52:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/44477888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:05.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:05 smithi022 ceph-mon[31360]: pgmap v1025: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:05 smithi022 ceph-mon[40250]: pgmap v1025: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:05 smithi122 ceph-mon[32255]: pgmap v1025: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:07.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:52:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:52:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:07 smithi022 ceph-mon[31360]: pgmap v1026: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:07] "GET /metrics HTTP/1.1" 200 202046 "" "Prometheus/2.18.1" 2021-12-15T17:52:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:07 smithi022 ceph-mon[40250]: pgmap v1026: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:07 smithi122 ceph-mon[32255]: pgmap v1026: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:08.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:08.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:09.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:09.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:09.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:09.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:09 smithi022 ceph-mon[31360]: pgmap v1027: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3066815413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:09 smithi022 ceph-mon[40250]: pgmap v1027: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3066815413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:09 smithi122 ceph-mon[32255]: pgmap v1027: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3066815413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:52:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:11 smithi022 ceph-mon[31360]: pgmap v1028: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:11 smithi022 ceph-mon[40250]: pgmap v1028: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:11 smithi122 ceph-mon[32255]: pgmap v1028: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:13 smithi022 ceph-mon[31360]: pgmap v1029: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:13 smithi022 ceph-mon[40250]: pgmap v1029: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:13 smithi122 ceph-mon[32255]: pgmap v1029: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:15 smithi022 ceph-mon[31360]: pgmap v1030: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3917235731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:15 smithi022 ceph-mon[40250]: pgmap v1030: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3917235731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:15 smithi122 ceph-mon[32255]: pgmap v1030: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3917235731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:17.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:52:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:52:17.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:17 smithi022 ceph-mon[31360]: pgmap v1031: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:17] "GET /metrics HTTP/1.1" 200 202046 "" "Prometheus/2.18.1" 2021-12-15T17:52:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:17 smithi022 ceph-mon[40250]: pgmap v1031: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:17 smithi122 ceph-mon[32255]: pgmap v1031: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:18.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:19.378 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:19.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:19 smithi022 ceph-mon[31360]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:19 smithi022 ceph-mon[40250]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:19 smithi122 ceph-mon[32255]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:19.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:19.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:19.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:52:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2290888568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2290888568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2290888568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:21.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:21 smithi022 ceph-mon[31360]: pgmap v1033: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:21 smithi022 ceph-mon[40250]: pgmap v1033: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:21 smithi122 ceph-mon[32255]: pgmap v1033: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:23 smithi022 ceph-mon[31360]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:23 smithi022 ceph-mon[40250]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:23 smithi122 ceph-mon[32255]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:24.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:25.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:25.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:25.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:25 smithi022 ceph-mon[31360]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2290387769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:25 smithi022 ceph-mon[40250]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2290387769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:25 smithi122 ceph-mon[32255]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2290387769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:27.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:52:26 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:52:27.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:27 smithi022 ceph-mon[31360]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:27] "GET /metrics HTTP/1.1" 200 202054 "" "Prometheus/2.18.1" 2021-12-15T17:52:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:27 smithi022 ceph-mon[40250]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:27 smithi122 ceph-mon[32255]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:28.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:29.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:29 smithi022 ceph-mon[31360]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:29 smithi022 ceph-mon[40250]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:29 smithi122 ceph-mon[32255]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:30.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:52:30.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:30.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:30.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:31 smithi022 ceph-mon[31360]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3175924294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:31 smithi022 ceph-mon[40250]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3175924294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:31 smithi122 ceph-mon[32255]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3175924294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:33 smithi122 ceph-mon[32255]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:52:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:52:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:52:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:52:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[31360]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:52:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:52:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:52:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:52:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[40250]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:52:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:52:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:52:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:52:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:35 smithi122 ceph-mon[32255]: pgmap v1040: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:35 smithi022 ceph-mon[31360]: pgmap v1040: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:35 smithi022 ceph-mon[40250]: pgmap v1040: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:35.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:35.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:35.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3270920540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:36.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3270920540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3270920540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:37.288 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:52:36 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:52:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:37 smithi122 ceph-mon[32255]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:37.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:37 smithi022 ceph-mon[31360]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:37] "GET /metrics HTTP/1.1" 200 202052 "" "Prometheus/2.18.1" 2021-12-15T17:52:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:37 smithi022 ceph-mon[40250]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:39 smithi122 ceph-mon[32255]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:39 smithi022 ceph-mon[31360]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:39 smithi022 ceph-mon[40250]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:52:40.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:41.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:41.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:41.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:41 smithi022 ceph-mon[31360]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3190917288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:41 smithi022 ceph-mon[40250]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3190917288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:41 smithi122 ceph-mon[32255]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3190917288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:43 smithi122 ceph-mon[32255]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:43 smithi022 ceph-mon[31360]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:43 smithi022 ceph-mon[40250]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:45 smithi122 ceph-mon[32255]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:45 smithi022 ceph-mon[31360]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:45 smithi022 ceph-mon[40250]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:46.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:46.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:46.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:46.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:47.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:52:46 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:52:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:47 smithi122 ceph-mon[32255]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3963702266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:47.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:47] "GET /metrics HTTP/1.1" 200 202052 "" "Prometheus/2.18.1" 2021-12-15T17:52:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:47 smithi022 ceph-mon[31360]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3963702266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:47 smithi022 ceph-mon[40250]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3963702266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:49 smithi122 ceph-mon[32255]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:49 smithi022 ceph-mon[31360]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:49 smithi022 ceph-mon[40250]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:52:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:51.608 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:51 smithi022 ceph-mon[31360]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:51.608 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:51 smithi022 ceph-mon[40250]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:51 smithi122 ceph-mon[32255]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:51.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3721249891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:52.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3721249891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3721249891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:53 smithi122 ceph-mon[32255]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:53 smithi022 ceph-mon[31360]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:53 smithi022 ceph-mon[40250]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:55 smithi122 ceph-mon[32255]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:55 smithi022 ceph-mon[31360]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:55 smithi022 ceph-mon[40250]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:56.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:52:57.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:52:56 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:52:57.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:52:57.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:52:57.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:52:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:57 smithi122 ceph-mon[32255]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:57 smithi022 ceph-mon[31360]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:52:57] "GET /metrics HTTP/1.1" 200 202053 "" "Prometheus/2.18.1" 2021-12-15T17:52:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:57 smithi022 ceph-mon[40250]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:52:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4130336603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4130336603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:52:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4130336603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:52:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:52:59 smithi122 ceph-mon[32255]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:52:59 smithi022 ceph-mon[31360]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:52:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:52:59 smithi022 ceph-mon[40250]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:53:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:01 smithi122 ceph-mon[32255]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:01 smithi022 ceph-mon[31360]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:01 smithi022 ceph-mon[40250]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:02.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:02.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:02.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:02.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:03 smithi122 ceph-mon[32255]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2768786769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:53:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:53:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[31360]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2768786769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:53:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:53:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[40250]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2768786769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:53:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:53:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:05 smithi122 ceph-mon[32255]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:05 smithi022 ceph-mon[31360]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:05 smithi022 ceph-mon[40250]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:07.341 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:53:06 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:53:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:07 smithi122 ceph-mon[32255]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:07.718 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:07 smithi022 ceph-mon[31360]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:07.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:07] "GET /metrics HTTP/1.1" 200 202051 "" "Prometheus/2.18.1" 2021-12-15T17:53:07.720 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:07 smithi022 ceph-mon[40250]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:07.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:08.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:08.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:08.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2709844255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2709844255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2709844255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:09 smithi122 ceph-mon[32255]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:09 smithi022 ceph-mon[31360]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:09 smithi022 ceph-mon[40250]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:53:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:11 smithi122 ceph-mon[32255]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:11 smithi022 ceph-mon[31360]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:11 smithi022 ceph-mon[40250]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:13.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:13.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:13.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:13.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:13 smithi122 ceph-mon[32255]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:13 smithi022 ceph-mon[31360]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:13 smithi022 ceph-mon[40250]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2869099531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:14.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2869099531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2869099531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:15 smithi122 ceph-mon[32255]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:15 smithi022 ceph-mon[31360]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:15 smithi022 ceph-mon[40250]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:17.361 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:53:16 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:53:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:17 smithi122 ceph-mon[32255]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:17 smithi022 ceph-mon[31360]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:17] "GET /metrics HTTP/1.1" 200 202051 "" "Prometheus/2.18.1" 2021-12-15T17:53:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:17 smithi022 ceph-mon[40250]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:18.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:18.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:18.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:18.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:19 smithi122 ceph-mon[32255]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1362271350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:19.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:19 smithi022 ceph-mon[31360]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1362271350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:19 smithi022 ceph-mon[40250]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1362271350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:53:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:21 smithi122 ceph-mon[32255]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:21.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:21 smithi022 ceph-mon[31360]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:21 smithi022 ceph-mon[40250]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:23 smithi122 ceph-mon[32255]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:23 smithi022 ceph-mon[31360]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:23 smithi022 ceph-mon[40250]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:23.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:24.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:24.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:24.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2464637161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2464637161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2464637161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:25 smithi122 ceph-mon[32255]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:25 smithi022 ceph-mon[31360]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:25 smithi022 ceph-mon[40250]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:27.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:53:27 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:53:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:27 smithi122 ceph-mon[32255]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:27.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:27 smithi022 ceph-mon[31360]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:27] "GET /metrics HTTP/1.1" 200 202055 "" "Prometheus/2.18.1" 2021-12-15T17:53:27.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:27 smithi022 ceph-mon[40250]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:28.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:29.202 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:29.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:29 smithi022 ceph-mon[31360]: pgmap v1067: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:29 smithi022 ceph-mon[40250]: pgmap v1067: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:29.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:29.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:29.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:29 smithi122 ceph-mon[32255]: pgmap v1067: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4078698194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:53:30.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4078698194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4078698194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:31 smithi122 ceph-mon[32255]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:31 smithi022 ceph-mon[31360]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:31 smithi022 ceph-mon[40250]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:33 smithi122 ceph-mon[32255]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:53:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:53:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:33 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:53:33.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[31360]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:53:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:53:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:53:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[40250]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:53:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:53:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:33 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:53:34.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:34.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:53:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:53:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:53:34.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:34.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:34.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:35 smithi022 ceph-mon[31360]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1537848820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:35 smithi022 ceph-mon[40250]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1537848820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:35 smithi122 ceph-mon[32255]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1537848820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:37.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:53:37 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:53:37.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:37 smithi022 ceph-mon[31360]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:37] "GET /metrics HTTP/1.1" 200 202056 "" "Prometheus/2.18.1" 2021-12-15T17:53:37.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:37 smithi022 ceph-mon[40250]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:37 smithi122 ceph-mon[32255]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:38.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:39 smithi022 ceph-mon[31360]: pgmap v1072: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:39 smithi022 ceph-mon[40250]: pgmap v1072: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:39.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:39 smithi122 ceph-mon[32255]: pgmap v1072: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:39.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:40.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:40.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:40.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:53:41.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:41 smithi022 ceph-mon[40250]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/668798334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:41 smithi022 ceph-mon[31360]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/668798334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:41 smithi122 ceph-mon[32255]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/668798334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:43 smithi022 ceph-mon[31360]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:43 smithi022 ceph-mon[40250]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:43 smithi122 ceph-mon[32255]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:45.314 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:45.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:45 smithi022 ceph-mon[31360]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:45 smithi022 ceph-mon[40250]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:45.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:45.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:45.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:45 smithi122 ceph-mon[32255]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:46.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1614189360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1614189360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1614189360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:47 smithi122 ceph-mon[32255]: pgmap v1076: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:53:47 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:53:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:47 smithi022 ceph-mon[31360]: pgmap v1076: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:47] "GET /metrics HTTP/1.1" 200 202056 "" "Prometheus/2.18.1" 2021-12-15T17:53:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:47 smithi022 ceph-mon[40250]: pgmap v1076: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:48.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:49 smithi022 ceph-mon[31360]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:49 smithi022 ceph-mon[40250]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:49 smithi122 ceph-mon[32255]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:53:50.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:51.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:51.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:51.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:51 smithi022 ceph-mon[31360]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1076427541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:51 smithi022 ceph-mon[40250]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1076427541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:51 smithi122 ceph-mon[32255]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1076427541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:53.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:53 smithi022 ceph-mon[31360]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:53 smithi022 ceph-mon[40250]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:53 smithi122 ceph-mon[32255]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:55 smithi022 ceph-mon[31360]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:55 smithi022 ceph-mon[40250]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:55 smithi122 ceph-mon[32255]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:53:56.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:53:56.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:53:56.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:53:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:57 smithi122 ceph-mon[32255]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2983314540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:57.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:53:57 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:53:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:53:57] "GET /metrics HTTP/1.1" 200 202055 "" "Prometheus/2.18.1" 2021-12-15T17:53:57.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:57 smithi022 ceph-mon[40250]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2983314540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:57 smithi022 ceph-mon[31360]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:53:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2983314540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:53:58.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:53:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:53:59 smithi022 ceph-mon[31360]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:53:59 smithi022 ceph-mon[40250]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:53:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:53:59 smithi122 ceph-mon[32255]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:54:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:01 smithi122 ceph-mon[32255]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:01 smithi022 ceph-mon[31360]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:01 smithi022 ceph-mon[40250]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4047650670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:02.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4047650670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4047650670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:03 smithi122 ceph-mon[32255]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:03 smithi022 ceph-mon[31360]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:03 smithi022 ceph-mon[40250]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:54:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:54:04.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:54:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:54:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:54:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:54:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:05 smithi122 ceph-mon[32255]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:05 smithi022 ceph-mon[31360]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:05 smithi022 ceph-mon[40250]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:06.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:07.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:07.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:07.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:07.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:54:07 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:54:07.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:07 smithi022 ceph-mon[31360]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/851625793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:07] "GET /metrics HTTP/1.1" 200 202058 "" "Prometheus/2.18.1" 2021-12-15T17:54:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:07 smithi022 ceph-mon[40250]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/851625793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:07 smithi122 ceph-mon[32255]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/851625793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:08.849 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:09 smithi122 ceph-mon[32255]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:09 smithi022 ceph-mon[31360]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:09 smithi022 ceph-mon[40250]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:54:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:11 smithi122 ceph-mon[32255]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:11 smithi022 ceph-mon[31360]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:11 smithi022 ceph-mon[40250]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:12.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:12.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:12.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:12.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:13 smithi122 ceph-mon[32255]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1109832541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:13 smithi022 ceph-mon[31360]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1109832541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:13 smithi022 ceph-mon[40250]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1109832541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:15 smithi122 ceph-mon[32255]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:15 smithi022 ceph-mon[31360]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:15 smithi022 ceph-mon[40250]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:17.310 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:54:17 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:54:17.538 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:17 smithi122 ceph-mon[32255]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:17 smithi022 ceph-mon[31360]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:17] "GET /metrics HTTP/1.1" 200 202058 "" "Prometheus/2.18.1" 2021-12-15T17:54:17.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:17 smithi022 ceph-mon[40250]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:17.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:17.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:17.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2044755443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2044755443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2044755443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:19 smithi122 ceph-mon[32255]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:19.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:19 smithi022 ceph-mon[31360]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:19 smithi022 ceph-mon[40250]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:54:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:21 smithi122 ceph-mon[32255]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:21 smithi022 ceph-mon[31360]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:21 smithi022 ceph-mon[40250]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:22.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:23.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:23.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:23.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:23 smithi022 ceph-mon[31360]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/551451489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:23 smithi022 ceph-mon[40250]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/551451489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:23 smithi122 ceph-mon[32255]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/551451489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:25 smithi122 ceph-mon[32255]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:25 smithi022 ceph-mon[31360]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:25 smithi022 ceph-mon[40250]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:27.326 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:54:27 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:54:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:27 smithi122 ceph-mon[32255]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:27.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:27 smithi022 ceph-mon[31360]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:27] "GET /metrics HTTP/1.1" 200 202057 "" "Prometheus/2.18.1" 2021-12-15T17:54:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:27 smithi022 ceph-mon[40250]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:28.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:28.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:28.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:28.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:28.653 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:29 smithi122 ceph-mon[32255]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2667889773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:29 smithi022 ceph-mon[31360]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2667889773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:29 smithi022 ceph-mon[40250]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2667889773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:54:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:31 smithi122 ceph-mon[32255]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:31.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:31 smithi022 ceph-mon[31360]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:31 smithi022 ceph-mon[40250]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:33.653 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:33.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:33 smithi022 ceph-mon[31360]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:33.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:33 smithi022 ceph-mon[40250]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:33 smithi122 ceph-mon[32255]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:34.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:34.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:34.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:54:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:54:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:54:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1579224079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:54:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:54:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:54:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1579224079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:54:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:54:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:54:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1579224079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:54:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:35 smithi122 ceph-mon[32255]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:35 smithi022 ceph-mon[31360]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:35 smithi022 ceph-mon[40250]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:37.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:54:37 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:54:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:37 smithi122 ceph-mon[32255]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:37] "GET /metrics HTTP/1.1" 200 202060 "" "Prometheus/2.18.1" 2021-12-15T17:54:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:37 smithi022 ceph-mon[31360]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:37 smithi022 ceph-mon[40250]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:38.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:39.023 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:39.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:39.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:39.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:39 smithi122 ceph-mon[32255]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:39 smithi022 ceph-mon[31360]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:39 smithi022 ceph-mon[40250]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2446548228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:54:40.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2446548228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2446548228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:41 smithi122 ceph-mon[32255]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:41 smithi022 ceph-mon[40250]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:41 smithi022 ceph-mon[31360]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:43 smithi122 ceph-mon[32255]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:43 smithi022 ceph-mon[31360]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:43 smithi022 ceph-mon[40250]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:44.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:44.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:44.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:44.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:45 smithi122 ceph-mon[32255]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1890426487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:45.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:45 smithi022 ceph-mon[31360]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1890426487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:45 smithi022 ceph-mon[40250]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1890426487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:47.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:54:47 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:54:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:47 smithi122 ceph-mon[32255]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:47 smithi022 ceph-mon[31360]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:47] "GET /metrics HTTP/1.1" 200 202060 "" "Prometheus/2.18.1" 2021-12-15T17:54:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:47 smithi022 ceph-mon[40250]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:49 smithi122 ceph-mon[32255]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:49.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:49.766 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:49 smithi022 ceph-mon[31360]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:49.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:49 smithi022 ceph-mon[40250]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:50.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:50.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:50.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/245346978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:54:50.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/245346978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/245346978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:51 smithi122 ceph-mon[32255]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:51 smithi022 ceph-mon[31360]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:51 smithi022 ceph-mon[40250]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:53 smithi122 ceph-mon[32255]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:53 smithi022 ceph-mon[31360]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:53 smithi022 ceph-mon[40250]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:55.137 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:54:55.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:54:55.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:54:55.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:54:55.509 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:55 smithi022 ceph-mon[40250]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:55.509 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:55 smithi022 ceph-mon[31360]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:55 smithi122 ceph-mon[32255]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/959709934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:56.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/959709934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/959709934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:54:57.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:54:57 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:54:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:57 smithi122 ceph-mon[32255]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:57 smithi022 ceph-mon[31360]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:54:57] "GET /metrics HTTP/1.1" 200 202050 "" "Prometheus/2.18.1" 2021-12-15T17:54:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:57 smithi022 ceph-mon[40250]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:54:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:58.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:54:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:54:59 smithi122 ceph-mon[32255]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:54:59 smithi022 ceph-mon[31360]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:54:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:54:59 smithi022 ceph-mon[40250]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:55:00.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:00.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:00.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:00.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:01 smithi122 ceph-mon[32255]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/179202959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:01 smithi022 ceph-mon[31360]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/179202959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:01 smithi022 ceph-mon[40250]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/179202959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:03 smithi122 ceph-mon[32255]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:55:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:55:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:03 smithi022 ceph-mon[31360]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:55:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:55:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:03 smithi022 ceph-mon[40250]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:55:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:55:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:05 smithi122 ceph-mon[32255]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:05 smithi022 ceph-mon[31360]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:05 smithi022 ceph-mon[40250]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:05.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:06.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:06.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:06.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:06.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3606112174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3606112174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3606112174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:07.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:55:07 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:55:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:07 smithi122 ceph-mon[32255]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:07.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:07 smithi022 ceph-mon[31360]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:07] "GET /metrics HTTP/1.1" 200 202059 "" "Prometheus/2.18.1" 2021-12-15T17:55:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:07 smithi022 ceph-mon[40250]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:09 smithi122 ceph-mon[32255]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:09 smithi022 ceph-mon[31360]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:09 smithi022 ceph-mon[40250]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:55:11.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:11.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:11 smithi022 ceph-mon[31360]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:11 smithi022 ceph-mon[40250]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:11.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:11.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:11.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:11 smithi122 ceph-mon[32255]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/320638046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:12.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/320638046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:12.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/320638046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:13 smithi122 ceph-mon[32255]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:13 smithi022 ceph-mon[31360]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:13 smithi022 ceph-mon[40250]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:15 smithi122 ceph-mon[32255]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:15 smithi022 ceph-mon[31360]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:15 smithi022 ceph-mon[40250]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:16.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:16.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:16.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:16.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:17.429 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:55:17 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:55:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:17 smithi122 ceph-mon[32255]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/779213844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:17 smithi022 ceph-mon[31360]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/779213844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:17] "GET /metrics HTTP/1.1" 200 202059 "" "Prometheus/2.18.1" 2021-12-15T17:55:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:17 smithi022 ceph-mon[40250]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/779213844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:19.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:19 smithi022 ceph-mon[31360]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:19 smithi022 ceph-mon[40250]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:19 smithi122 ceph-mon[32255]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:55:21.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:21 smithi022 ceph-mon[31360]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:21 smithi022 ceph-mon[40250]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:21 smithi122 ceph-mon[32255]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:21.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:22.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:22.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:22.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:22.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2976874647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2976874647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2976874647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:23 smithi022 ceph-mon[31360]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:23 smithi022 ceph-mon[40250]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:23.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:23 smithi122 ceph-mon[32255]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:25 smithi022 ceph-mon[31360]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:25 smithi022 ceph-mon[40250]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:25 smithi122 ceph-mon[32255]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:27.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:27.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:55:27 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:55:27.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:27.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:27.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:27.729 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:27] "GET /metrics HTTP/1.1" 200 202055 "" "Prometheus/2.18.1" 2021-12-15T17:55:27.729 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:27 smithi022 ceph-mon[40250]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:27.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:27 smithi022 ceph-mon[31360]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:27 smithi122 ceph-mon[32255]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:28.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1315111328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1315111328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:28.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1315111328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:29 smithi022 ceph-mon[31360]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:29 smithi022 ceph-mon[40250]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:29 smithi122 ceph-mon[32255]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:55:31.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:31 smithi022 ceph-mon[31360]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:31 smithi022 ceph-mon[40250]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:31 smithi122 ceph-mon[32255]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:32.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:33.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:33.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:33 smithi022 ceph-mon[31360]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/394951545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:33 smithi022 ceph-mon[40250]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/394951545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:33 smithi122 ceph-mon[32255]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/394951545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:55:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:55:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:34 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:55:34.774 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:55:34.775 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:55:34.775 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:34 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:55:34.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:55:34.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:55:34.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:34 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:55:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:35 smithi022 ceph-mon[31360]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:55:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:35 smithi022 ceph-mon[40250]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:55:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:35 smithi122 ceph-mon[32255]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:55:37.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:55:37 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:55:37.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:37 smithi022 ceph-mon[31360]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:37] "GET /metrics HTTP/1.1" 200 202056 "" "Prometheus/2.18.1" 2021-12-15T17:55:37.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:37 smithi022 ceph-mon[40250]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:37 smithi122 ceph-mon[32255]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:38.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:38.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:38.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:38.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:38.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/366702256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/366702256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:38.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/366702256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:39.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:39 smithi022 ceph-mon[40250]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:39 smithi022 ceph-mon[31360]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:39 smithi122 ceph-mon[32255]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:55:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:41 smithi022 ceph-mon[31360]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:41 smithi022 ceph-mon[40250]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:41 smithi122 ceph-mon[32255]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:43.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:43 smithi022 ceph-mon[31360]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:43 smithi022 ceph-mon[40250]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:43.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:43.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:43.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:43.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:43 smithi122 ceph-mon[32255]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:44.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1395894848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:44.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1395894848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1395894848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:45 smithi022 ceph-mon[31360]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:45 smithi022 ceph-mon[40250]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:45 smithi122 ceph-mon[32255]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:47.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:55:47 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:55:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:47 smithi022 ceph-mon[31360]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:47] "GET /metrics HTTP/1.1" 200 202056 "" "Prometheus/2.18.1" 2021-12-15T17:55:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:47 smithi022 ceph-mon[40250]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:47 smithi122 ceph-mon[32255]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:48.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:49.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:49.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:49.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:49.493 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:49 smithi022 ceph-mon[40250]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:49.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:49 smithi022 ceph-mon[31360]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/11174414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/11174414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:49.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:49 smithi122 ceph-mon[32255]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/11174414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:55:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:51 smithi022 ceph-mon[31360]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:51 smithi022 ceph-mon[40250]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:51 smithi122 ceph-mon[32255]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:53 smithi022 ceph-mon[31360]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:53 smithi022 ceph-mon[40250]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:53 smithi122 ceph-mon[32255]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:54.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:54.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:54.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:54.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:55:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:55 smithi022 ceph-mon[31360]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/713215592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:55 smithi022 ceph-mon[40250]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/713215592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:55.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:55 smithi122 ceph-mon[32255]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/713215592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:55:57.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:55:57 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:55:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:57 smithi022 ceph-mon[31360]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:55:57] "GET /metrics HTTP/1.1" 200 202060 "" "Prometheus/2.18.1" 2021-12-15T17:55:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:57 smithi022 ceph-mon[40250]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:57 smithi122 ceph-mon[32255]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:55:58.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:55:59.585 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:55:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:55:59 smithi022 ceph-mon[31360]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:55:59 smithi022 ceph-mon[40250]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:55:59 smithi122 ceph-mon[32255]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:55:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:55:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:55:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:56:00.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2936856913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2936856913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:00.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2936856913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:01 smithi022 ceph-mon[31360]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:01 smithi022 ceph-mon[40250]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:01.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:01 smithi122 ceph-mon[32255]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:03 smithi022 ceph-mon[31360]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:56:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:56:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:03 smithi022 ceph-mon[40250]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:56:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:56:03.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:03 smithi122 ceph-mon[32255]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:56:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:56:04.957 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:05.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:05.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:05.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:05 smithi022 ceph-mon[31360]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1749616090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:05 smithi022 ceph-mon[40250]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1749616090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:05.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:05 smithi122 ceph-mon[32255]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:05.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1749616090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:07.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:56:07 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:56:07.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:07 smithi022 ceph-mon[31360]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:07] "GET /metrics HTTP/1.1" 200 202058 "" "Prometheus/2.18.1" 2021-12-15T17:56:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:07 smithi022 ceph-mon[40250]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:07.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:07 smithi122 ceph-mon[32255]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:09.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:09.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:09 smithi122 ceph-mon[32255]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:10.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:09 smithi022 ceph-mon[31360]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:09 smithi022 ceph-mon[40250]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:10.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:56:10.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:10.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:10.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:11 smithi122 ceph-mon[32255]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/589330033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:12.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:11 smithi022 ceph-mon[31360]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/589330033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:11 smithi022 ceph-mon[40250]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/589330033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:13 smithi122 ceph-mon[32255]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:14.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:13 smithi022 ceph-mon[31360]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:13 smithi022 ceph-mon[40250]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:15.706 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:15.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:15 smithi122 ceph-mon[32255]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:16.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:15 smithi022 ceph-mon[31360]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:15 smithi022 ceph-mon[40250]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:16.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:16.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:16.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3283153906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3283153906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3283153906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:17.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:56:17 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:56:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:17 smithi022 ceph-mon[31360]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:17] "GET /metrics HTTP/1.1" 200 202058 "" "Prometheus/2.18.1" 2021-12-15T17:56:17.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:17 smithi022 ceph-mon[40250]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:17 smithi122 ceph-mon[32255]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:19.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:19 smithi122 ceph-mon[32255]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:20.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:19 smithi022 ceph-mon[31360]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:19 smithi022 ceph-mon[40250]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:56:21.074 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:21.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:21.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:21.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:21.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:21 smithi022 ceph-mon[31360]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2393844390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:21 smithi022 ceph-mon[40250]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2393844390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:21 smithi122 ceph-mon[32255]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2393844390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:22.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:22 smithi122 ceph-mon[32255]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:23.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:22 smithi022 ceph-mon[31360]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:22 smithi022 ceph-mon[40250]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:25.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:24 smithi122 ceph-mon[32255]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:24 smithi022 ceph-mon[31360]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:24 smithi022 ceph-mon[40250]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:26.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:26.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:26.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:27 smithi122 ceph-mon[32255]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1337555821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:27.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:56:27 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:56:27.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:27] "GET /metrics HTTP/1.1" 200 202048 "" "Prometheus/2.18.1" 2021-12-15T17:56:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:27 smithi022 ceph-mon[31360]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1337555821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:27 smithi022 ceph-mon[40250]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1337555821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:28.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:29 smithi122 ceph-mon[32255]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:29 smithi022 ceph-mon[31360]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:29 smithi022 ceph-mon[40250]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:56:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:31 smithi122 ceph-mon[32255]: pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:31.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:31 smithi022 ceph-mon[31360]: pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:31 smithi022 ceph-mon[40250]: pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:31.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:32.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:32.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:32.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:32.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2466110475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2466110475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2466110475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:33 smithi122 ceph-mon[32255]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:33 smithi022 ceph-mon[31360]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:33 smithi022 ceph-mon[40250]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:35.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[40250]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:56:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:56:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:56:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:56:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[31360]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:56:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:56:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:56:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:56:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:35 smithi122 ceph-mon[32255]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:56:37.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:37 smithi122 ceph-mon[32255]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:37.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:56:37 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:56:37.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:37 smithi022 ceph-mon[31360]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:37] "GET /metrics HTTP/1.1" 200 202049 "" "Prometheus/2.18.1" 2021-12-15T17:56:37.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:37 smithi022 ceph-mon[40250]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:37.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:37.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:37.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/637264189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:38.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/637264189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/637264189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:39 smithi122 ceph-mon[32255]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:39 smithi022 ceph-mon[31360]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:39 smithi022 ceph-mon[40250]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:56:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:41 smithi122 ceph-mon[32255]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:41 smithi022 ceph-mon[31360]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:41 smithi022 ceph-mon[40250]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:42.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:42.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:42.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:42.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:43 smithi122 ceph-mon[32255]: pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1129772295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:43 smithi022 ceph-mon[31360]: pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1129772295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:43 smithi022 ceph-mon[40250]: pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1129772295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:45 smithi122 ceph-mon[32255]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:45.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:45 smithi022 ceph-mon[31360]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:45 smithi022 ceph-mon[40250]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:47 smithi122 ceph-mon[32255]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:56:47 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:56:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:47 smithi022 ceph-mon[31360]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:47] "GET /metrics HTTP/1.1" 200 202049 "" "Prometheus/2.18.1" 2021-12-15T17:56:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:47 smithi022 ceph-mon[40250]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:47.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:48.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:48.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:48.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2742332511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2742332511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2742332511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:49 smithi122 ceph-mon[32255]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:49.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:49 smithi022 ceph-mon[31360]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:49 smithi022 ceph-mon[40250]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:56:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:51 smithi122 ceph-mon[32255]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:51 smithi022 ceph-mon[31360]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:51 smithi022 ceph-mon[40250]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:53 smithi122 ceph-mon[32255]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:53.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:53.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:53.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:53.687 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:53 smithi022 ceph-mon[31360]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:53.687 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:53 smithi022 ceph-mon[40250]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3959869318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:54.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3959869318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3959869318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:55 smithi122 ceph-mon[32255]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:55 smithi022 ceph-mon[31360]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:55 smithi022 ceph-mon[40250]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:57 smithi122 ceph-mon[32255]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:56:57 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:56:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:57 smithi022 ceph-mon[31360]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:56:57] "GET /metrics HTTP/1.1" 200 202054 "" "Prometheus/2.18.1" 2021-12-15T17:56:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:57 smithi022 ceph-mon[40250]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:56:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:58.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:56:58.687 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:58.687 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:56:59.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:56:59.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:56:59.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:56:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:59 smithi122 ceph-mon[32255]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:56:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3942479624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:59 smithi022 ceph-mon[31360]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:56:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3942479624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:56:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:59 smithi022 ceph-mon[40250]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:56:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:56:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3942479624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:57:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:01 smithi122 ceph-mon[32255]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:01 smithi022 ceph-mon[31360]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:01 smithi022 ceph-mon[40250]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:03 smithi122 ceph-mon[32255]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:57:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:03 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:57:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:03 smithi022 ceph-mon[31360]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:57:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:03 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:57:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:03 smithi022 ceph-mon[40250]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:57:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:03 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:57:04.056 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:04.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:04.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:04.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:05 smithi122 ceph-mon[32255]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2404543476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:05 smithi022 ceph-mon[31360]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2404543476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:05 smithi022 ceph-mon[40250]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2404543476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:07 smithi122 ceph-mon[32255]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:57:07 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:57:07.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:07 smithi022 ceph-mon[31360]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:07] "GET /metrics HTTP/1.1" 200 202054 "" "Prometheus/2.18.1" 2021-12-15T17:57:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:07 smithi022 ceph-mon[40250]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:09 smithi122 ceph-mon[32255]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:09 smithi022 ceph-mon[31360]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:09 smithi022 ceph-mon[40250]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:09.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:09.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:09.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/710809077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:57:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/710809077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/710809077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:11 smithi122 ceph-mon[32255]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:11 smithi022 ceph-mon[31360]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:11 smithi022 ceph-mon[40250]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:13 smithi122 ceph-mon[32255]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:13 smithi022 ceph-mon[31360]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:13 smithi022 ceph-mon[40250]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:14.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:15.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:15.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:15.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:15.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:15 smithi022 ceph-mon[31360]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2808782334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:15 smithi022 ceph-mon[40250]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2808782334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:15 smithi122 ceph-mon[32255]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2808782334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:17 smithi122 ceph-mon[32255]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:57:17 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:57:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:17 smithi022 ceph-mon[31360]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:17] "GET /metrics HTTP/1.1" 200 202054 "" "Prometheus/2.18.1" 2021-12-15T17:57:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:17 smithi022 ceph-mon[40250]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:19 smithi122 ceph-mon[32255]: pgmap v1182: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:19.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:19 smithi022 ceph-mon[31360]: pgmap v1182: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:19 smithi022 ceph-mon[40250]: pgmap v1182: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:20.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:57:20.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:21 smithi122 ceph-mon[32255]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1687461787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:21 smithi022 ceph-mon[31360]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1687461787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:21 smithi022 ceph-mon[40250]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1687461787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:23 smithi122 ceph-mon[32255]: pgmap v1184: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:23 smithi022 ceph-mon[31360]: pgmap v1184: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:23 smithi022 ceph-mon[40250]: pgmap v1184: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:25.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:25 smithi122 ceph-mon[32255]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:25 smithi022 ceph-mon[31360]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:25 smithi022 ceph-mon[40250]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:25.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:25.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:25.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/20830599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:26.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/20830599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/20830599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:27 smithi122 ceph-mon[32255]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:57:27 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:57:27.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:27 smithi022 ceph-mon[31360]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:27] "GET /metrics HTTP/1.1" 200 202059 "" "Prometheus/2.18.1" 2021-12-15T17:57:27.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:27 smithi022 ceph-mon[40250]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:28.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:29 smithi122 ceph-mon[32255]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:29 smithi022 ceph-mon[31360]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:29 smithi022 ceph-mon[40250]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:57:30.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:31.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:31.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:31.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:31 smithi122 ceph-mon[32255]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2565579201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:31.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:31 smithi022 ceph-mon[31360]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2565579201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:31 smithi022 ceph-mon[40250]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2565579201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:33 smithi122 ceph-mon[32255]: pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:33.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:33 smithi022 ceph-mon[31360]: pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:33 smithi022 ceph-mon[40250]: pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:35 smithi122 ceph-mon[32255]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:57:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:57:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:35 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:57:35.697 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[31360]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:35.697 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:57:35.698 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:57:35.698 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:57:35.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[40250]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:35.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:57:35.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:57:35.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:35 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:57:36.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:36.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:36.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:36.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:37.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:36 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:57:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:36 smithi022 ceph-mon[31360]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2011537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:36 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:57:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:36 smithi022 ceph-mon[40250]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2011537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:36 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:57:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:36 smithi122 ceph-mon[32255]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2011537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:37.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:57:37 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:57:37.709 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:37] "GET /metrics HTTP/1.1" 200 202056 "" "Prometheus/2.18.1" 2021-12-15T17:57:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:37 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:37 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:37 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:39.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:38 smithi022 ceph-mon[31360]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:38 smithi022 ceph-mon[40250]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:38 smithi122 ceph-mon[32255]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:41.659 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:41 smithi022 ceph-mon[31360]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:41.659 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:41 smithi022 ceph-mon[40250]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:41 smithi122 ceph-mon[32255]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:42.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:42.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:42.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3865643611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:42.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3865643611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:42.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3865643611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:43 smithi122 ceph-mon[32255]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:43 smithi022 ceph-mon[31360]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:43 smithi022 ceph-mon[40250]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:45 smithi122 ceph-mon[32255]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:45.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:45 smithi022 ceph-mon[31360]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:45 smithi022 ceph-mon[40250]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:47.023 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:47.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:47.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:47.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:47.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:57:47 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:57:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:47 smithi122 ceph-mon[32255]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:47 smithi022 ceph-mon[31360]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:47] "GET /metrics HTTP/1.1" 200 202056 "" "Prometheus/2.18.1" 2021-12-15T17:57:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:47 smithi022 ceph-mon[40250]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:48 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4038515653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:48.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:48 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4038515653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:48 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4038515653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:49 smithi122 ceph-mon[32255]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:49.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:49 smithi022 ceph-mon[31360]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:49 smithi022 ceph-mon[40250]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:57:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:51 smithi122 ceph-mon[32255]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:51 smithi022 ceph-mon[31360]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:51 smithi022 ceph-mon[40250]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:57:52.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:52.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:53 smithi122 ceph-mon[32255]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2813966313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:53 smithi022 ceph-mon[31360]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2813966313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:53 smithi022 ceph-mon[40250]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2813966313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:55 smithi122 ceph-mon[32255]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:55 smithi022 ceph-mon[31360]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:55 smithi022 ceph-mon[40250]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 83 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:57:57.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:57:57 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:57:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:57 smithi122 ceph-mon[32255]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:57:57.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:57:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:57:57] "GET /metrics HTTP/1.1" 200 202061 "" "Prometheus/2.18.1" 2021-12-15T17:57:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:57 smithi022 ceph-mon[31360]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:57:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:57 smithi022 ceph-mon[40250]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:57:58.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:57:58.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:57:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:57:58.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:58 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/906988241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:58 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/906988241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:58 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:57:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/906988241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:57:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:57:59 smithi122 ceph-mon[32255]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:57:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:57:59 smithi022 ceph-mon[31360]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:57:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:57:59 smithi022 ceph-mon[40250]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:58:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:01 smithi122 ceph-mon[32255]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:01 smithi022 ceph-mon[31360]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:01 smithi022 ceph-mon[40250]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:03.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:58:03.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:58:03.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:58:03.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:03.523 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:03 smithi022 ceph-mon[31360]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:03.524 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:03 smithi022 ceph-mon[40250]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:03 smithi122 ceph-mon[32255]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:04 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/511911515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:04.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:04 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/511911515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:04 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/511911515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:05 smithi122 ceph-mon[32255]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:05 smithi022 ceph-mon[31360]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:05 smithi022 ceph-mon[40250]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:07 smithi122 ceph-mon[32255]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:07 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:58:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:07 smithi022 ceph-mon[31360]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:07] "GET /metrics HTTP/1.1" 200 202095 "" "Prometheus/2.18.1" 2021-12-15T17:58:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:07 smithi022 ceph-mon[40250]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T17:58:08.525 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:58:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:08 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:08 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:08 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:08.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:58:08.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:58:08.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:09 smithi122 ceph-mon[32255]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3190338634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:09 smithi022 ceph-mon[31360]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3190338634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:09 smithi022 ceph-mon[40250]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3190338634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:58:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:11 smithi122 ceph-mon[32255]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:11 smithi022 ceph-mon[31360]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:11 smithi022 ceph-mon[40250]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:13 smithi122 ceph-mon[32255]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:13 smithi022 ceph-mon[31360]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:13 smithi022 ceph-mon[40250]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:13.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:58:14.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:58:14.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:58:14.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:14.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3906494270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3906494270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3906494270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:15 smithi122 ceph-mon[32255]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:15 smithi022 ceph-mon[31360]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:15 smithi022 ceph-mon[40250]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:17.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:17 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:58:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:17 smithi122 ceph-mon[32255]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:17 smithi022 ceph-mon[31360]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:17] "GET /metrics HTTP/1.1" 200 202095 "" "Prometheus/2.18.1" 2021-12-15T17:58:17.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:17 smithi022 ceph-mon[40250]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:18 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:18 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:18 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:19.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:58:19.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:19 smithi022 ceph-mon[31360]: pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:19 smithi022 ceph-mon[40250]: pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:19.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:58:19.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:58:19.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:19 smithi122 ceph-mon[32255]: pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:58:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/487530842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/487530842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/487530842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:21 smithi122 ceph-mon[32255]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:21.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:21 smithi022 ceph-mon[31360]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:21 smithi022 ceph-mon[40250]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:23 smithi122 ceph-mon[32255]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:23 smithi022 ceph-mon[31360]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:23 smithi022 ceph-mon[40250]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:24.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:58:25.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== 2021-12-15T17:58:25.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== ']' 2021-12-15T17:58:25.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:25 smithi122 ceph-mon[32255]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2231272410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:25 smithi022 ceph-mon[31360]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2231272410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:25 smithi022 ceph-mon[40250]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2231272410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:27 smithi122 ceph-mon[32255]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:27 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:58:27.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:27 smithi022 ceph-mon[31360]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:27] "GET /metrics HTTP/1.1" 200 202098 "" "Prometheus/2.18.1" 2021-12-15T17:58:27.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:27 smithi022 ceph-mon[40250]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:28 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:28.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:28 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:28 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:29 smithi122 ceph-mon[32255]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:29 smithi022 ceph-mon[31360]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:29 smithi022 ceph-mon[40250]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key osd.7 2021-12-15T17:58:30.379 INFO:teuthology.orchestra.run.smithi022.stdout:rotating key for mgr.y 2021-12-15T17:58:30.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDnKrphe2ulLhAApxXILXmelEjElpxw7Xpo9Q== 2021-12-15T17:58:30.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQAJI7phiXKWHBAAYvsXbDtTTUVoqBQz9XY9tQ== == AQDnKrphe2ulLhAApxXILXmelEjElpxw7Xpo9Q== ']' 2021-12-15T17:58:30.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2021-12-15T17:58:30.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ echo 'rotating key for mgr.y' 2021-12-15T17:58:30.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:58:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:58:30.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ K=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:58:30.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:58:30.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ ceph orch daemon rotate-key mgr.y 2021-12-15T17:58:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:Scheduled to rotate-key mgr.y on host 'smithi022' 2021-12-15T17:58:31.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:58:31.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:31.281 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.152+0000 7fa134995700 -1 mgr.server reply reply (13) Permission denied access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication 2021-12-15T17:58:31.282 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.154+0000 7fa106e71700 -1 log_channel(cephadm) log [ERR] : Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-y/keyring', '-n', 'mgr.y', 'tell', 'mgr.y', 'rotate-key', '-i', '-']: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 legacy=0x7f61f810bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:31.283 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:31.284 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> 172.21.15.22:0/231715338 conn(0x7f61f8066e90 msgr2=0x7f61f8069300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-12-15T17:58:31.284 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:31.285 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 wait complete. 2021-12-15T17:58:31.285 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 Processor -- start 2021-12-15T17:58:31.288 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- start start 2021-12-15T17:58:31.289 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:31.289 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:31.290 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:31.290 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 0x7f61f8102730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53672/0 (socket says 172.21.15.22:53672) 2021-12-15T17:58:31.290 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 -- 172.21.15.22:0/1216147801 learned_addr learned my addr 172.21.15.22:0/1216147801 (peer_addr_for_me v1:172.21.15.22:0/0) 2021-12-15T17:58:31.291 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61f6d9d700 1 --1- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 0x7f61f811cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34478/0 (socket says 172.21.15.22:34478) 2021-12-15T17:58:31.291 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770178561 0 0) 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:31.292 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:31.292 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542879245 0 0) 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:31.292 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:31.292 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149600500 0 0) 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:31.293 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:31.293 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 3543687569 0 0) 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:31.293 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:31.294 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0003f30 con 0x7f61f81096c0 2021-12-15T17:58:31.294 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1177221001 0 0) 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:31.294 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f8060d30 2021-12-15T17:58:31.294 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61ec0018d0 con 0x7f61f8060d30 2021-12-15T17:58:31.295 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1791241748 0 0) 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:31.295 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8061850 2021-12-15T17:58:31.296 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61e80017b0 con 0x7f61f8061850 2021-12-15T17:58:31.296 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 564+0+0 (unknown 2240922346 0 0) 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:31.296 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 legacy=0x7f61f8102730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:31.297 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.122:6789/0 conn(0x7f61f8061850 legacy=0x7f61f81193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:31.298 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8121c70 con 0x7f61f81096c0 2021-12-15T17:58:31.298 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f81096c0 2021-12-15T17:58:31.299 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f811fa60 con 0x7f61f81096c0 2021-12-15T17:58:31.300 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:31.300 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(39 keys) v1 ==== 1740+0+0 (unknown 2695402636 0 0) 0x7f61f0001550 con 0x7f61f81096c0 2021-12-15T17:58:31.301 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0007010 con 0x7f61f81096c0 2021-12-15T17:58:31.301 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 876258047 0 0) 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:31.302 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 8 ==== mgrmap(e 21) v1 ==== 82394+0+0 (unknown 2299499152 0 0) 0x7f61f0002230 con 0x7f61f81096c0 2021-12-15T17:58:31.302 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6800/3632387897 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:31.302 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 9 ==== osd_map(104..104 src has 1..104) v4 ==== 6255+0+0 (unknown 3456861732 0 0) 0x7f61f007e120 con 0x7f61f81096c0 2021-12-15T17:58:31.303 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.152+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mgr.34562 v1:172.21.15.22:6800/3632387897 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (unknown 3575743073 0 0) 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:31.303 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: Er 2021-12-15T17:58:31.304 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: ror EACCES: problem getting command descriptions from mgr.y 2021-12-15T17:58:31.304 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:31 smithi022 conmon[31781]: 2021-12-15T17:58:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2097302622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3336630035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:58:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2021-12-15T17:58:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2021-12-15T17:58:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:58:31.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:58:31.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:31.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2021-12-15T17:58:31.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:31 smithi122 ceph-mon[32255]: osdmap e105: 8 total, 8 up, 8 in 2021-12-15T17:58:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2097302622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3336630035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:58:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2021-12-15T17:58:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2021-12-15T17:58:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:58:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:58:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2021-12-15T17:58:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[31360]: osdmap e105: 8 total, 8 up, 8 in 2021-12-15T17:58:31.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:31.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2097302622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2021-12-15T17:58:31.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3336630035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:31.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:31.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:31.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:31.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' 2021-12-15T17:58:31.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2021-12-15T17:58:31.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2021-12-15T17:58:31.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-12-15T17:58:31.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-12-15T17:58:31.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:31.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2021-12-15T17:58:31.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:31 smithi022 ceph-mon[40250]: osdmap e105: 8 total, 8 up, 8 in 2021-12-15T17:58:32.312 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:32 smithi022 conmon[31781]: 2021-12-15T17:58:32.043+0000 7fa17084f700 -1 mgr handle_mgr_map I was active but no longer am 2021-12-15T17:58:32.312 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:32 smithi022 conmon[31781]: ignoring --setuser ceph since I am not root 2021-12-15T17:58:32.313 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:32 smithi022 conmon[31781]: ignoring --setgroup ceph since I am not root 2021-12-15T17:58:32.313 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:32 smithi022 conmon[31781]: 2021-12-15T17:58:32.207+0000 7f774a293180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2021-12-15T17:58:32.315 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:32 smithi122 conmon[33927]: [15/Dec/2021:17:58:32] ENGINE Bus STOPPING 2021-12-15T17:58:32.315 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:32 smithi122 conmon[33927]: [15/Dec/2021:17:58:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2021-12-15T17:58:32.315 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:32 smithi122 conmon[33927]: [15/Dec/2021:17:58:32] ENGINE Bus STOPPED 2021-12-15T17:58:32.315 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:32 smithi122 conmon[33927]: [15/Dec/2021:17:58:32] ENGINE Bus STARTING 2021-12-15T17:58:32.316 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:32 smithi122 conmon[33927]: [15/Dec/2021:17:58:32] ENGINE Serving on http://:::9283 2021-12-15T17:58:32.316 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:32 smithi122 conmon[33927]: [15/Dec/2021:17:58:32] ENGINE Bus STARTED 2021-12-15T17:58:32.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='client.17700 v1:172.21.15.22:0/736116666' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:58:32.318 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='client.17700 v1:172.21.15.22:0/736116666' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:58:32.319 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: Schedule rotate-key daemon mgr.y 2021-12-15T17:58:32.319 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: Rotating authentication key for mgr.y 2021-12-15T17:58:32.319 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='client.17706 v1:172.21.15.22:0/1216147801' entity='mgr.y' cmd=[{"prefix": "get_command_descriptions"}]: access denied 2021-12-15T17:58:32.319 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-y/keyring', '-n', 'mgr.y', 'tell', 'mgr.y', 'rotate-key', '-i', '-']: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 legacy=0x7f61f810bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.320 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:32.320 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> 172.21.15.22:0/231715338 conn(0x7f61f8066e90 msgr2=0x7f61f8069300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-12-15T17:58:32.320 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:32.321 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 wait complete. 2021-12-15T17:58:32.321 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 Processor -- start 2021-12-15T17:58:32.322 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- start start 2021-12-15T17:58:32.322 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:32.322 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:32.323 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:32.323 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 0x7f61f8102730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53672/0 (socket says 172.21.15.22:53672) 2021-12-15T17:58:32.323 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 -- 172.21.15.22:0/1216147801 learned_addr learned my addr 172.21.15.22:0/1216147801 (peer_addr_for_me v1:172.21.15.22:0/0) 2021-12-15T17:58:32.323 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61f6d9d700 1 --1- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 0x7f61f811cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34478/0 (socket says 172.21.15.22:34478) 2021-12-15T17:58:32.323 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770178561 0 0) 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:32.324 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:32.324 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542879245 0 0) 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:32.324 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:32.324 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149600500 0 0) 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:32.325 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:32.325 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 3543687569 0 0) 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:32.325 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:32.326 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0003f30 con 0x7f61f81096c0 2021-12-15T17:58:32.326 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1177221001 0 0) 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:32.326 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f8060d30 2021-12-15T17:58:32.326 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61ec0018d0 con 0x7f61f8060d30 2021-12-15T17:58:32.327 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1791241748 0 0) 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:32.327 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8061850 2021-12-15T17:58:32.327 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61e80017b0 con 0x7f61f8061850 2021-12-15T17:58:32.327 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 564+0+0 (unknown 2240922346 0 0) 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:32.327 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 legacy=0x7f61f8102730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.328 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.122:6789/0 conn(0x7f61f8061850 legacy=0x7f61f81193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.328 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8121c70 con 0x7f61f81096c0 2021-12-15T17:58:32.328 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f81096c0 2021-12-15T17:58:32.328 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f811fa60 con 0x7f61f81096c0 2021-12-15T17:58:32.329 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:32.329 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(39 keys) v1 ==== 1740+0+0 (unknown 2695402636 0 0) 0x7f61f0001550 con 0x7f61f81096c0 2021-12-15T17:58:32.329 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0007010 con 0x7f61f81096c0 2021-12-15T17:58:32.329 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 876258047 0 0) 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:32.329 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 8 ==== mgrmap(e 21) v1 ==== 82394+0+0 (unknown 2299499152 0 0) 0x7f61f0002230 con 0x7f61f81096c0 2021-12-15T17:58:32.330 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6800/3632387897 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:32.330 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 9 ==== osd_map(104..104 src has 1..104) v4 ==== 6255+0+0 (unknown 3456861732 0 0) 0x7f61f007e120 con 0x7f61f81096c0 2021-12-15T17:58:32.330 INFO:journalctl@ceph.mon.c.smithi022.stdout: 2021-12-15T17:58:31.152+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mgr.34562 v1:172.21.15.22:6800/3632387897 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (unknown 3575743073 0 0) 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:32.330 INFO:journalctl@ceph.mon.c.smithi022.stdout: Error EACCES: problem getting command descriptions from mgr.y 2021-12-15T17:58:32.331 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: Failing over to other MGR 2021-12-15T17:58:32.331 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: Standby manager daemon y started 2021-12-15T17:58:32.331 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2021-12-15T17:58:32.331 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: mgrmap e22: x(active, starting, since 0.887532s), standbys: y 2021-12-15T17:58:32.332 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:58:32.332 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:58:32.332 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:58:32.332 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:58:32.333 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:58:32.333 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:58:32.333 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:58:32.333 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:58:32.333 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:58:32.334 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:58:32.334 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:58:32.334 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:58:32.334 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:58:32.334 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:58:32.335 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:58:32.335 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:58:32.335 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: Manager daemon x is now available 2021-12-15T17:58:32.335 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:32.335 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:32.336 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:32.336 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:32.336 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:32.336 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:32.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:32.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: Schedule rotate-key daemon mgr.y 2021-12-15T17:58:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: Rotating authentication key for mgr.y 2021-12-15T17:58:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='client.17706 v1:172.21.15.22:0/1216147801' entity='mgr.y' cmd=[{"prefix": "get_command_descriptions"}]: access denied 2021-12-15T17:58:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-y/keyring', '-n', 'mgr.y', 'tell', 'mgr.y', 'rotate-key', '-i', '-']: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 legacy=0x7f61f810bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> 172.21.15.22:0/231715338 conn(0x7f61f8066e90 msgr2=0x7f61f8069300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-12-15T17:58:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 wait complete. 2021-12-15T17:58:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 Processor -- start 2021-12-15T17:58:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- start start 2021-12-15T17:58:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 0x7f61f8102730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53672/0 (socket says 172.21.15.22:53672) 2021-12-15T17:58:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 -- 172.21.15.22:0/1216147801 learned_addr learned my addr 172.21.15.22:0/1216147801 (peer_addr_for_me v1:172.21.15.22:0/0) 2021-12-15T17:58:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61f6d9d700 1 --1- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 0x7f61f811cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34478/0 (socket says 172.21.15.22:34478) 2021-12-15T17:58:32.689 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770178561 0 0) 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:32.689 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:32.689 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542879245 0 0) 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:32.689 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:32.689 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149600500 0 0) 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:32.690 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:32.690 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 3543687569 0 0) 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:32.690 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:32.690 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0003f30 con 0x7f61f81096c0 2021-12-15T17:58:32.691 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1177221001 0 0) 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:32.691 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f8060d30 2021-12-15T17:58:32.691 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61ec0018d0 con 0x7f61f8060d30 2021-12-15T17:58:32.691 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1791241748 0 0) 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:32.691 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8061850 2021-12-15T17:58:32.692 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61e80017b0 con 0x7f61f8061850 2021-12-15T17:58:32.692 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 564+0+0 (unknown 2240922346 0 0) 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:32.692 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 legacy=0x7f61f8102730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.692 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.122:6789/0 conn(0x7f61f8061850 legacy=0x7f61f81193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.693 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8121c70 con 0x7f61f81096c0 2021-12-15T17:58:32.693 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f81096c0 2021-12-15T17:58:32.693 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f811fa60 con 0x7f61f81096c0 2021-12-15T17:58:32.693 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:32.693 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(39 keys) v1 ==== 1740+0+0 (unknown 2695402636 0 0) 0x7f61f0001550 con 0x7f61f81096c0 2021-12-15T17:58:32.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0007010 con 0x7f61f81096c0 2021-12-15T17:58:32.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 876258047 0 0) 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:32.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 8 ==== mgrmap(e 21) v1 ==== 82394+0+0 (unknown 2299499152 0 0) 0x7f61f0002230 con 0x7f61f81096c0 2021-12-15T17:58:32.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6800/3632387897 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:32.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 9 ==== osd_map(104..104 src has 1..104) v4 ==== 6255+0+0 (unknown 3456861732 0 0) 0x7f61f007e120 con 0x7f61f81096c0 2021-12-15T17:58:32.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2021-12-15T17:58:31.152+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mgr.34562 v1:172.21.15.22:6800/3632387897 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (unknown 3575743073 0 0) 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:32.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: Error EACCES: problem getting command descriptions from mgr.y 2021-12-15T17:58:32.695 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: Failing over to other MGR 2021-12-15T17:58:32.695 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: Standby manager daemon y started 2021-12-15T17:58:32.696 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2021-12-15T17:58:32.696 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: mgrmap e22: x(active, starting, since 0.887532s), standbys: y 2021-12-15T17:58:32.696 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:58:32.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:58:32.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:58:32.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:58:32.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:58:32.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:58:32.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:58:32.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:58:32.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:58:32.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:58:32.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:58:32.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:58:32.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:58:32.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:58:32.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:58:32.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:58:32.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: Manager daemon x is now available 2021-12-15T17:58:32.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:32.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:32.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:32.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:32.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:32.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:32.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:32.704 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='client.17700 v1:172.21.15.22:0/736116666' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2021-12-15T17:58:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: Schedule rotate-key daemon mgr.y 2021-12-15T17:58:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: Rotating authentication key for mgr.y 2021-12-15T17:58:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='client.17706 v1:172.21.15.22:0/1216147801' entity='mgr.y' cmd=[{"prefix": "get_command_descriptions"}]: access denied 2021-12-15T17:58:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-y/keyring', '-n', 'mgr.y', 'tell', 'mgr.y', 'rotate-key', '-i', '-']: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 legacy=0x7f61f810bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 >> 172.21.15.22:0/231715338 conn(0x7f61f8066e90 msgr2=0x7f61f8069300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-12-15T17:58:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 shutdown_connections 2021-12-15T17:58:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- 172.21.15.22:0/231715338 wait complete. 2021-12-15T17:58:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 Processor -- start 2021-12-15T17:58:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.148+0000 7f61fcf8a700 1 -- start start 2021-12-15T17:58:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61fcf8a700 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 0x7f61f8102730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53672/0 (socket says 172.21.15.22:53672) 2021-12-15T17:58:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61f659c700 1 -- 172.21.15.22:0/1216147801 learned_addr learned my addr 172.21.15.22:0/1216147801 (peer_addr_for_me v1:172.21.15.22:0/0) 2021-12-15T17:58:32.782 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61f6d9d700 1 --1- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6789/0 conn(0x7f61f81096c0 0x7f61f811cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34478/0 (socket says 172.21.15.22:34478) 2021-12-15T17:58:32.782 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770178561 0 0) 0x7f61f811e690 con 0x7f61f81096c0 2021-12-15T17:58:32.782 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:32.782 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542879245 0 0) 0x7f61f8120a70 con 0x7f61f8060d30 2021-12-15T17:58:32.783 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:32.783 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149600500 0 0) 0x7f61f811f870 con 0x7f61f8061850 2021-12-15T17:58:32.783 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:32.783 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 3543687569 0 0) 0x7f61e0003570 con 0x7f61f81096c0 2021-12-15T17:58:32.784 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:32.784 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0003f30 con 0x7f61f81096c0 2021-12-15T17:58:32.784 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1177221001 0 0) 0x7f61f811e690 con 0x7f61f8060d30 2021-12-15T17:58:32.784 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61e0003570 con 0x7f61f8060d30 2021-12-15T17:58:32.784 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.1 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61ec0018d0 con 0x7f61f8060d30 2021-12-15T17:58:32.785 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 748+0+0 (unknown 1791241748 0 0) 0x7f61f8120a70 con 0x7f61f8061850 2021-12-15T17:58:32.785 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f811e690 con 0x7f61f8061850 2021-12-15T17:58:32.785 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.2 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61e80017b0 con 0x7f61f8061850 2021-12-15T17:58:32.785 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 564+0+0 (unknown 2240922346 0 0) 0x7f61f811f870 con 0x7f61f81096c0 2021-12-15T17:58:32.785 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.22:6790/0 conn(0x7f61f8060d30 legacy=0x7f61f8102730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.786 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.149+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 >> v1:172.21.15.122:6789/0 conn(0x7f61f8061850 legacy=0x7f61f81193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-12-15T17:58:32.786 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8121c70 con 0x7f61f81096c0 2021-12-15T17:58:32.786 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7f61f8120a70 con 0x7f61f81096c0 2021-12-15T17:58:32.786 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f811fa60 con 0x7f61f81096c0 2021-12-15T17:58:32.787 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61fcf8a700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:32.787 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(39 keys) v1 ==== 1740+0+0 (unknown 2695402636 0 0) 0x7f61f0001550 con 0x7f61f81096c0 2021-12-15T17:58:32.787 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3566035639 0 0) 0x7f61f0007010 con 0x7f61f81096c0 2021-12-15T17:58:32.787 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.150+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 876258047 0 0) 0x7f61f811fed0 con 0x7f61f81096c0 2021-12-15T17:58:32.788 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 8 ==== mgrmap(e 21) v1 ==== 82394+0+0 (unknown 2299499152 0 0) 0x7f61f0002230 con 0x7f61f81096c0 2021-12-15T17:58:32.788 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 --> v1:172.21.15.22:6800/3632387897 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:32.788 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.151+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mon.0 v1:172.21.15.22:6789/0 9 ==== osd_map(104..104 src has 1..104) v4 ==== 6255+0+0 (unknown 3456861732 0 0) 0x7f61f007e120 con 0x7f61f81096c0 2021-12-15T17:58:32.788 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2021-12-15T17:58:31.152+0000 7f61e6ffd700 1 -- 172.21.15.22:0/1216147801 <== mgr.34562 v1:172.21.15.22:6800/3632387897 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (unknown 3575743073 0 0) 0x7f61e006a450 con 0x7f61e0066790 2021-12-15T17:58:32.788 INFO:journalctl@ceph.mon.a.smithi022.stdout: Error EACCES: problem getting command descriptions from mgr.y 2021-12-15T17:58:32.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: Failing over to other MGR 2021-12-15T17:58:32.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: Standby manager daemon y started 2021-12-15T17:58:32.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34562 v1:172.21.15.22:0/1254464903' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2021-12-15T17:58:32.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: mgrmap e22: x(active, starting, since 0.887532s), standbys: y 2021-12-15T17:58:32.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-15T17:58:32.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-15T17:58:32.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-15T17:58:32.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-12-15T17:58:32.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-12-15T17:58:32.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-12-15T17:58:32.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-12-15T17:58:32.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-12-15T17:58:32.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-12-15T17:58:32.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-12-15T17:58:32.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-12-15T17:58:32.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-12-15T17:58:32.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-12-15T17:58:32.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-12-15T17:58:32.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-12-15T17:58:32.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-12-15T17:58:32.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: Manager daemon x is now available 2021-12-15T17:58:32.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:32.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:32.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:32.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:32.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:32.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:58:32.794 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:32.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:58:33.093 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:32 smithi022 conmon[31781]: 2021-12-15T17:58:32.787+0000 7f774a293180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2021-12-15T17:58:33.094 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:32 smithi022 conmon[31781]: 2021-12-15T17:58:32.916+0000 7f774a293180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2021-12-15T17:58:33.094 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:32 smithi022 conmon[31781]: 2021-12-15T17:58:32.998+0000 7f774a293180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2021-12-15T17:58:33.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:33 smithi022 conmon[31781]: 2021-12-15T17:58:33.094+0000 7f774a293180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2021-12-15T17:58:34.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:34 smithi122 ceph-mon[32255]: mgrmap e23: x(active, since 1.88762s), standbys: y 2021-12-15T17:58:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:34 smithi122 ceph-mon[32255]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:34.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:33 smithi122 conmon[33927]: [15/Dec/2021:17:58:33] ENGINE Bus STARTING 2021-12-15T17:58:34.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:33 smithi122 conmon[33927]: [15/Dec/2021:17:58:33] ENGINE Serving on https://172.21.15.122:7150 2021-12-15T17:58:34.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:33 smithi122 conmon[33927]: [15/Dec/2021:17:58:33] ENGINE Bus STARTED 2021-12-15T17:58:34.277 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:33 smithi022 conmon[31781]: 2021-12-15T17:58:33.857+0000 7f774a293180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:58:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:34 smithi022 ceph-mon[40250]: mgrmap e23: x(active, since 1.88762s), standbys: y 2021-12-15T17:58:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:34 smithi022 ceph-mon[40250]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:34 smithi022 ceph-mon[31360]: mgrmap e23: x(active, since 1.88762s), standbys: y 2021-12-15T17:58:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:34 smithi022 ceph-mon[31360]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:34.733 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:34 smithi022 conmon[31781]: 2021-12-15T17:58:34.452+0000 7f774a293180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2021-12-15T17:58:34.733 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:34 smithi022 conmon[31781]: 2021-12-15T17:58:34.545+0000 7f774a293180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2021-12-15T17:58:35.027 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:34 smithi022 conmon[31781]: 2021-12-15T17:58:34.734+0000 7f774a293180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2021-12-15T17:58:35.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:34 smithi022 conmon[31781]: 2021-12-15T17:58:34.826+0000 7f774a293180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2021-12-15T17:58:35.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:34 smithi022 conmon[31781]: 2021-12-15T17:58:34.924+0000 7f774a293180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2021-12-15T17:58:35.403 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:35 smithi022 conmon[31781]: 2021-12-15T17:58:35.112+0000 7f774a293180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2021-12-15T17:58:35.404 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[31360]: [15/Dec/2021:17:58:33] ENGINE Bus STARTING 2021-12-15T17:58:35.404 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[31360]: [15/Dec/2021:17:58:33] ENGINE Serving on https://172.21.15.122:7150 2021-12-15T17:58:35.404 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[31360]: [15/Dec/2021:17:58:33] ENGINE Bus STARTED 2021-12-15T17:58:35.405 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[31360]: mgrmap e24: x(active, since 2s), standbys: y 2021-12-15T17:58:35.405 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[31360]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:35.405 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[40250]: [15/Dec/2021:17:58:33] ENGINE Bus STARTING 2021-12-15T17:58:35.405 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[40250]: [15/Dec/2021:17:58:33] ENGINE Serving on https://172.21.15.122:7150 2021-12-15T17:58:35.406 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[40250]: [15/Dec/2021:17:58:33] ENGINE Bus STARTED 2021-12-15T17:58:35.406 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[40250]: mgrmap e24: x(active, since 2s), standbys: y 2021-12-15T17:58:35.406 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:35 smithi022 ceph-mon[40250]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:35 smithi122 ceph-mon[32255]: [15/Dec/2021:17:58:33] ENGINE Bus STARTING 2021-12-15T17:58:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:35 smithi122 ceph-mon[32255]: [15/Dec/2021:17:58:33] ENGINE Serving on https://172.21.15.122:7150 2021-12-15T17:58:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:35 smithi122 ceph-mon[32255]: [15/Dec/2021:17:58:33] ENGINE Bus STARTED 2021-12-15T17:58:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:35 smithi122 ceph-mon[32255]: mgrmap e24: x(active, since 2s), standbys: y 2021-12-15T17:58:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:35 smithi122 ceph-mon[32255]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:35.735 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:35 smithi022 conmon[31781]: 2021-12-15T17:58:35.404+0000 7f774a293180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2021-12-15T17:58:35.735 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:35 smithi022 conmon[31781]: 2021-12-15T17:58:35.545+0000 7f774a293180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2021-12-15T17:58:35.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:35 smithi022 conmon[31781]: 2021-12-15T17:58:35.637+0000 7f774a293180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2021-12-15T17:58:36.027 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:35 smithi022 conmon[31781]: 2021-12-15T17:58:35.736+0000 7f774a293180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2021-12-15T17:58:36.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:58:36.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:58:36.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:58:36.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:36.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:36 smithi022 conmon[31781]: 2021-12-15T17:58:36.598+0000 7f774a293180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2021-12-15T17:58:37.278 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:36 smithi022 conmon[31781]: 2021-12-15T17:58:36.976+0000 7f774a293180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2021-12-15T17:58:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[31360]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3112906874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[40250]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3112906874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:37 smithi122 conmon[70494]: debug there is no tcmu-runner data available 2021-12-15T17:58:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:37 smithi122 ceph-mon[32255]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3112906874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:37.580 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:37 smithi022 conmon[31781]: 2021-12-15T17:58:37.279+0000 7f774a293180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2021-12-15T17:58:37.581 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:37 smithi022 conmon[31781]: 2021-12-15T17:58:37.436+0000 7f774a293180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2021-12-15T17:58:37.844 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:37 smithi022 conmon[31781]: 2021-12-15T17:58:37.748+0000 7f774a293180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2021-12-15T17:58:38.142 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:37 smithi022 conmon[31781]: 2021-12-15T17:58:37.844+0000 7f774a293180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2021-12-15T17:58:38.143 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:38 smithi022 ceph-mon[31360]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:38.143 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:38.144 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:38 smithi022 ceph-mon[40250]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:38.144 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:38 smithi122 ceph-mon[32255]: from='client.34535 v1:172.21.15.122:0/719969562' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:38.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: 2021-12-15T17:58:38.220+0000 7f774a293180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2021-12-15T17:58:39.027 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: 2021-12-15T17:58:38.541+0000 7f774a293180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2021-12-15T17:58:39.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: [15/Dec/2021:17:58:38] ENGINE Bus STARTING 2021-12-15T17:58:39.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: CherryPy Checker: 2021-12-15T17:58:39.028 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: The Application mounted at '' has an empty config. 2021-12-15T17:58:39.029 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: 2021-12-15T17:58:39.029 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: [15/Dec/2021:17:58:38] ENGINE Serving on http://:::9283 2021-12-15T17:58:39.029 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:38 smithi022 conmon[31781]: [15/Dec/2021:17:58:38] ENGINE Bus STARTED 2021-12-15T17:58:39.381 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[31360]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:39.381 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[31360]: Standby manager daemon y restarted 2021-12-15T17:58:39.382 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[31360]: Standby manager daemon y started 2021-12-15T17:58:39.382 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2021-12-15T17:58:39.382 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:58:39.382 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2021-12-15T17:58:39.383 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[31360]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:58:39.383 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[40250]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:39.383 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[40250]: Standby manager daemon y restarted 2021-12-15T17:58:39.384 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[40250]: Standby manager daemon y started 2021-12-15T17:58:39.384 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2021-12-15T17:58:39.384 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:58:39.384 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2021-12-15T17:58:39.384 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:39 smithi022 ceph-mon[40250]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:58:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:39 smithi122 ceph-mon[32255]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2021-12-15T17:58:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:39 smithi122 ceph-mon[32255]: Standby manager daemon y restarted 2021-12-15T17:58:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:39 smithi122 ceph-mon[32255]: Standby manager daemon y started 2021-12-15T17:58:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:39 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2021-12-15T17:58:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:39 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-12-15T17:58:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:39 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2021-12-15T17:58:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:39 smithi122 ceph-mon[32255]: from='mgr.? v1:172.21.15.22:0/3192261549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-12-15T17:58:40.384 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:40] "GET /metrics HTTP/1.1" 200 25676 "" "Prometheus/2.18.1" 2021-12-15T17:58:40.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:40 smithi122 ceph-mon[32255]: mgrmap e25: x(active, since 7s), standbys: y 2021-12-15T17:58:40.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:40.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:40 smithi122 ceph-mon[32255]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:58:40.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:40.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[31360]: mgrmap e25: x(active, since 7s), standbys: y 2021-12-15T17:58:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[31360]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:58:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[40250]: mgrmap e25: x(active, since 7s), standbys: y 2021-12-15T17:58:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[40250]: Updating smithi022:/etc/ceph/ceph.conf 2021-12-15T17:58:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:58:41.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2021-12-15T17:58:41.383 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.383 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:58:41.383 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:58:41.383 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:58:41.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:58:41.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:41.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:58:41.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:58:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2021-12-15T17:58:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:58:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:58:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:58:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:58:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:58:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2021-12-15T17:58:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: Updating smithi122:/etc/ceph/ceph.conf 2021-12-15T17:58:41.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2021-12-15T17:58:41.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:41.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:58:41.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-12-15T17:58:41.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:41.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:58:41.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:58:41.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:41.916 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:41 smithi122 systemd[1]: Stopping Ceph iscsi.iscsi.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:58:41.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:41 smithi122 bash[88702]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-iscsi.iscsi.a" found: no such container 2021-12-15T17:58:41.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:41 smithi122 conmon[70494]: debug Shutdown received 2021-12-15T17:58:41.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:41 smithi122 conmon[70494]: debug No gateway configuration to remove on this host (smithi122) 2021-12-15T17:58:42.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 bash[88702]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-iscsi-iscsi-a 2021-12-15T17:58:42.176 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 bash[88702]: Error: no container with name or ID "ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-iscsi.iscsi.a" found: no such container 2021-12-15T17:58:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:42 smithi122 ceph-mon[32255]: Reconfiguring iscsi.iscsi.a (dependencies changed)... 2021-12-15T17:58:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:42 smithi122 ceph-mon[32255]: Reconfiguring daemon iscsi.iscsi.a on smithi122 2021-12-15T17:58:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/519263602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:42 smithi022 ceph-mon[31360]: Reconfiguring iscsi.iscsi.a (dependencies changed)... 2021-12-15T17:58:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:42 smithi022 ceph-mon[31360]: Reconfiguring daemon iscsi.iscsi.a on smithi122 2021-12-15T17:58:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/519263602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:42 smithi022 ceph-mon[40250]: Reconfiguring iscsi.iscsi.a (dependencies changed)... 2021-12-15T17:58:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:42 smithi022 ceph-mon[40250]: Reconfiguring daemon iscsi.iscsi.a on smithi122 2021-12-15T17:58:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/519263602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 bash[88864]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c-iscsi-iscsi-a-tcmu 2021-12-15T17:58:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@iscsi.iscsi.a.service: Succeeded. 2021-12-15T17:58:42.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 systemd[1]: Stopped Ceph iscsi.iscsi.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:58:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@iscsi.iscsi.a.service: Found left-over process 70123 (podman) in control group while starting unit. Ignoring. 2021-12-15T17:58:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2021-12-15T17:58:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 systemd[1]: Starting Ceph iscsi.iscsi.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c... 2021-12-15T17:58:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 systemd[1]: ceph-0ea1c4a2-5dca-11ec-8c31-001a4aab830c@iscsi.iscsi.a.service: Found left-over process 70123 (podman) in control group while starting unit. Ignoring. 2021-12-15T17:58:42.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. 2021-12-15T17:58:43.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 podman[89109]: 2021-12-15T17:58:43.176 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:42 smithi122 conmon[89198]: log file path now is '/var/log/tcmu-runner.log' 2021-12-15T17:58:43.176 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 bash[88955]: log file path now is '/var/log/tcmu-runner.log' 2021-12-15T17:58:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:43 smithi122 ceph-mon[32255]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-12-15T17:58:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 podman[89327]: 2021-12-15T17:58:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 bash[88955]: 4544b726e3861527541fadb3eb4ef4a39851a22079fc5212f68e994e1a83e813 2021-12-15T17:58:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 systemd[1]: Started Ceph iscsi.iscsi.a for 0ea1c4a2-5dca-11ec-8c31-001a4aab830c. 2021-12-15T17:58:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:43 smithi022 ceph-mon[31360]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-12-15T17:58:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:43 smithi022 ceph-mon[40250]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-12-15T17:58:43.908 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 bash[69936]: time="2021-12-15T17:58:43Z" level=error msg="Could not retrieve exit code from event: died not found: unable to find event" 2021-12-15T17:58:43.909 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 conmon[89367]: debug Started the configuration object watcher 2021-12-15T17:58:43.909 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 conmon[89367]: debug Checking for config object changes every 1s 2021-12-15T17:58:43.909 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 conmon[89367]: debug Processing osd blocklist entries for this node 2021-12-15T17:58:43.909 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 conmon[89367]: debug Reading the configuration object to update local LIO configuration 2021-12-15T17:58:43.910 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 conmon[89367]: debug Configuration does not have an entry for this host(smithi122) - nothing to define to LIO 2021-12-15T17:58:44.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:43 smithi122 conmon[89367]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2021-12-15T17:58:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-12-15T17:58:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-12-15T17:58:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: Adding iSCSI gateway http://:@172.21.15.122:5000 to Dashboard 2021-12-15T17:58:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-12-15T17:58:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-12-15T17:58:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi122"}]: dispatch 2021-12-15T17:58:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi122"}]: dispatch 2021-12-15T17:58:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.122:0/3525112698' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-12-15T17:58:44.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-12-15T17:58:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-12-15T17:58:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: Adding iSCSI gateway http://:@172.21.15.122:5000 to Dashboard 2021-12-15T17:58:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-12-15T17:58:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-12-15T17:58:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi122"}]: dispatch 2021-12-15T17:58:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi122"}]: dispatch 2021-12-15T17:58:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.122:0/3525112698' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-12-15T17:58:44.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-12-15T17:58:44.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-12-15T17:58:44.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: Adding iSCSI gateway http://:@172.21.15.122:5000 to Dashboard 2021-12-15T17:58:44.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-12-15T17:58:44.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-12-15T17:58:44.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi122"}]: dispatch 2021-12-15T17:58:44.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mon.? v1:172.21.15.122:0/409581631' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi122"}]: dispatch 2021-12-15T17:58:44.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:44.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:58:44.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:58:44.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:58:44.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.122:0/3525112698' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-12-15T17:58:45.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:45 smithi022 ceph-mon[31360]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2021-12-15T17:58:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:45 smithi022 ceph-mon[40250]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2021-12-15T17:58:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:45 smithi122 ceph-mon[32255]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2021-12-15T17:58:46.808 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:58:47.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:58:47.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:58:47.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:47.434 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:58:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:47 smithi022 ceph-mon[31360]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1398826302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:47 smithi022 ceph-mon[40250]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1398826302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:47 smithi122 ceph-mon[32255]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1398826302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:49.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:48 smithi022 ceph-mon[31360]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:48 smithi022 ceph-mon[40250]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:58:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:48 smithi122 ceph-mon[32255]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:50] "GET /metrics HTTP/1.1" 200 202091 "" "Prometheus/2.18.1" 2021-12-15T17:58:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:51 smithi122 ceph-mon[32255]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:51 smithi022 ceph-mon[31360]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:51 smithi022 ceph-mon[40250]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:58:52.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:58:52.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:58:52.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:52.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T17:58:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T17:58:53.605 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:53 smithi122 ceph-mon[32255]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:53.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/100026504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:53 smithi022 ceph-mon[31360]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/100026504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:53 smithi022 ceph-mon[40250]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T17:58:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/100026504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:53.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:58:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T17:58:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:55 smithi122 ceph-mon[32255]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:55 smithi022 ceph-mon[31360]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:58:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:55 smithi022 ceph-mon[40250]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:57.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:58:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:57 smithi122 ceph-mon[32255]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:57 smithi022 ceph-mon[31360]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:58:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:57 smithi022 ceph-mon[40250]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:58:57.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:58:57.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:58:57.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:58:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3303012799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:58.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3303012799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3303012799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:58:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:58:59 smithi122 ceph-mon[32255]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:58:59 smithi022 ceph-mon[31360]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:58:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:58:59 smithi022 ceph-mon[40250]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:00] "GET /metrics HTTP/1.1" 200 202091 "" "Prometheus/2.18.1" 2021-12-15T17:59:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:01 smithi122 ceph-mon[32255]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:01 smithi022 ceph-mon[31360]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:01 smithi022 ceph-mon[40250]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T17:59:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T17:59:02.921 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:03.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:03.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:03.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:03.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:03 smithi122 ceph-mon[32255]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:03.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3670652513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:03 smithi022 ceph-mon[31360]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3670652513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:03 smithi022 ceph-mon[40250]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3670652513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:03.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:59:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T17:59:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:05 smithi122 ceph-mon[32255]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:05 smithi022 ceph-mon[31360]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:05 smithi022 ceph-mon[40250]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:07 smithi122 ceph-mon[32255]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:07 smithi022 ceph-mon[31360]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:59:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:07 smithi022 ceph-mon[40250]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:08.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:08.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:08.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:08.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:09 smithi122 ceph-mon[32255]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2331812998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:09 smithi022 ceph-mon[31360]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2331812998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:09 smithi022 ceph-mon[40250]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2331812998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:10] "GET /metrics HTTP/1.1" 200 202090 "" "Prometheus/2.18.1" 2021-12-15T17:59:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:11 smithi122 ceph-mon[32255]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:11 smithi022 ceph-mon[31360]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:11 smithi022 ceph-mon[40250]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T17:59:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T17:59:13.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:13 smithi122 ceph-mon[32255]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:13.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:13.668 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:13 smithi022 ceph-mon[31360]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:13.669 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:13 smithi022 ceph-mon[40250]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:13.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:59:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T17:59:14.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:14.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:14.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3744288963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:14.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3744288963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3744288963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:15 smithi122 ceph-mon[32255]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:15 smithi022 ceph-mon[31360]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:15 smithi022 ceph-mon[40250]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:17 smithi122 ceph-mon[32255]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:17 smithi022 ceph-mon[31360]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:59:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:17 smithi022 ceph-mon[40250]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:19.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:19.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:19.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:19.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:19 smithi122 ceph-mon[32255]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:19 smithi022 ceph-mon[31360]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:19 smithi022 ceph-mon[40250]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1213100433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:20] "GET /metrics HTTP/1.1" 200 202090 "" "Prometheus/2.18.1" 2021-12-15T17:59:20.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1213100433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:20.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1213100433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:21 smithi122 ceph-mon[32255]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:21.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:21 smithi022 ceph-mon[31360]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:21 smithi022 ceph-mon[40250]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T17:59:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T17:59:23.626 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:23 smithi122 ceph-mon[32255]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:23 smithi022 ceph-mon[31360]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:23 smithi022 ceph-mon[40250]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:23.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:59:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T17:59:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:25 smithi122 ceph-mon[32255]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2468471992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:25 smithi022 ceph-mon[31360]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2468471992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:25 smithi022 ceph-mon[40250]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2468471992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:27 smithi122 ceph-mon[32255]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:27 smithi022 ceph-mon[31360]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:59:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:27 smithi022 ceph-mon[40250]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:29 smithi122 ceph-mon[32255]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:29.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:29 smithi022 ceph-mon[31360]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:29 smithi022 ceph-mon[40250]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:29.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:30.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:30.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:30.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3637110824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:30] "GET /metrics HTTP/1.1" 200 202090 "" "Prometheus/2.18.1" 2021-12-15T17:59:30.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3637110824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3637110824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:31 smithi122 ceph-mon[32255]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:31.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:31 smithi022 ceph-mon[31360]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:31 smithi022 ceph-mon[40250]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T17:59:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T17:59:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:59:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:59:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:59:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:59:32.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:59:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:59:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:59:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:59:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:59:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T17:59:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:59:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T17:59:33.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:59:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T17:59:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:33 smithi122 ceph-mon[32255]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:33.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:33 smithi022 ceph-mon[31360]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:33 smithi022 ceph-mon[40250]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:35.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:35.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:35.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:35.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:35.520 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:35.521 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:35 smithi022 ceph-mon[31360]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:35.521 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:35.522 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:35 smithi022 ceph-mon[40250]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:35 smithi122 ceph-mon[32255]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3654903812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:36.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3654903812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3654903812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:37 smithi122 ceph-mon[32255]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:37 smithi022 ceph-mon[31360]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:59:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:37 smithi022 ceph-mon[40250]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:39 smithi122 ceph-mon[32255]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:39 smithi022 ceph-mon[31360]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:39 smithi022 ceph-mon[40250]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:40] "GET /metrics HTTP/1.1" 200 202086 "" "Prometheus/2.18.1" 2021-12-15T17:59:40.520 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:40.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:40.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:40.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:41 smithi122 ceph-mon[32255]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2333694997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:41.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:41 smithi022 ceph-mon[31360]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2333694997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:41 smithi022 ceph-mon[40250]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2333694997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T17:59:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T17:59:43.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:59:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T17:59:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:43 smithi122 ceph-mon[32255]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:43 smithi022 ceph-mon[31360]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:43 smithi022 ceph-mon[40250]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:45 smithi122 ceph-mon[32255]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:45 smithi022 ceph-mon[31360]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:45 smithi022 ceph-mon[40250]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:45.888 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:46.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:46.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:46.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:46.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1572398114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1572398114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1572398114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:47 smithi122 ceph-mon[32255]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:47 smithi022 ceph-mon[31360]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:59:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:47 smithi022 ceph-mon[40250]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:59:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:59:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:59:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:59:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:59:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:59:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T17:59:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T17:59:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T17:59:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:49 smithi022 ceph-mon[31360]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:59:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:49 smithi022 ceph-mon[40250]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:59:49.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:49 smithi122 ceph-mon[32255]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T17:59:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 17:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:50] "GET /metrics HTTP/1.1" 200 202095 "" "Prometheus/2.18.1" 2021-12-15T17:59:51.261 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:51.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:51 smithi022 ceph-mon[31360]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:51 smithi022 ceph-mon[40250]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:51.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:51 smithi122 ceph-mon[32255]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2945608180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 17:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T17:59:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T17:59:52.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2945608180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2945608180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:53 smithi022 ceph-mon[31360]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:53 smithi022 ceph-mon[40250]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:53 smithi122 ceph-mon[32255]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 17:59:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T17:59:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:55 smithi022 ceph-mon[31360]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:55 smithi022 ceph-mon[40250]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:55.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T17:59:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:55 smithi122 ceph-mon[32255]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:56.636 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T17:59:57.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T17:59:57.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T17:59:57.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T17:59:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 17:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:17:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T17:59:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:57 smithi022 ceph-mon[31360]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2146499245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:57 smithi022 ceph-mon[40250]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2146499245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:57 smithi122 ceph-mon[32255]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T17:59:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2146499245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T17:59:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 17:59:59 smithi022 ceph-mon[31360]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 17:59:59 smithi022 ceph-mon[40250]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T17:59:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 17:59:59 smithi122 ceph-mon[32255]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:00] "GET /metrics HTTP/1.1" 200 202095 "" "Prometheus/2.18.1" 2021-12-15T18:00:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T18:00:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T18:00:00.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T18:00:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:01 smithi022 ceph-mon[31360]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:01 smithi022 ceph-mon[40250]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:01.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:01 smithi122 ceph-mon[32255]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:02.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:02.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:02.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:02.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/196064625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:00:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:00:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/196064625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/196064625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:03 smithi022 ceph-mon[31360]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:03 smithi022 ceph-mon[40250]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:03.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:00:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:00:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:03 smithi122 ceph-mon[32255]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:05 smithi022 ceph-mon[31360]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:05 smithi022 ceph-mon[40250]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:05.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:05 smithi122 ceph-mon[32255]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:07.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:07.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:07.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:07.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:07.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:07 smithi022 ceph-mon[31360]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:07.760 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:00:07.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:07 smithi022 ceph-mon[40250]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:07 smithi122 ceph-mon[32255]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1869047123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1869047123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1869047123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:09 smithi022 ceph-mon[31360]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:09 smithi022 ceph-mon[40250]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:09 smithi122 ceph-mon[32255]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:10] "GET /metrics HTTP/1.1" 200 202092 "" "Prometheus/2.18.1" 2021-12-15T18:00:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:11 smithi022 ceph-mon[31360]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:11 smithi022 ceph-mon[40250]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:11 smithi122 ceph-mon[32255]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:00:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:00:12.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:13.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:13.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:13.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:13 smithi022 ceph-mon[31360]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2389362997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:13 smithi022 ceph-mon[40250]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2389362997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:13.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:00:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:00:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:13 smithi122 ceph-mon[32255]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2389362997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:15 smithi022 ceph-mon[31360]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:15 smithi022 ceph-mon[40250]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:15 smithi122 ceph-mon[32255]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:17 smithi022 ceph-mon[31360]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:00:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:17 smithi022 ceph-mon[40250]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:17 smithi122 ceph-mon[32255]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:18.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1116130310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1116130310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1116130310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:19 smithi022 ceph-mon[31360]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:19 smithi022 ceph-mon[40250]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:19 smithi122 ceph-mon[32255]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:20] "GET /metrics HTTP/1.1" 200 202091 "" "Prometheus/2.18.1" 2021-12-15T18:00:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:21 smithi022 ceph-mon[31360]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:21 smithi022 ceph-mon[40250]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:21 smithi122 ceph-mon[32255]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:00:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:00:23.493 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:23 smithi022 ceph-mon[31360]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:23 smithi022 ceph-mon[40250]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:23.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:23.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:23.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:23.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:00:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:00:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:23 smithi122 ceph-mon[32255]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2239862430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2239862430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:24.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2239862430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:25 smithi022 ceph-mon[31360]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:25 smithi022 ceph-mon[40250]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:25 smithi122 ceph-mon[32255]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:27 smithi022 ceph-mon[31360]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:00:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:27 smithi022 ceph-mon[40250]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:27 smithi122 ceph-mon[32255]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:28.865 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:29.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:29.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:29.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:29 smithi022 ceph-mon[40250]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2512009862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:29 smithi022 ceph-mon[31360]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2512009862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:29 smithi122 ceph-mon[32255]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2512009862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:30] "GET /metrics HTTP/1.1" 200 202091 "" "Prometheus/2.18.1" 2021-12-15T18:00:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:31 smithi122 ceph-mon[32255]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:31 smithi022 ceph-mon[31360]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:31 smithi022 ceph-mon[40250]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:00:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:00:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:00:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:00:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:00:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:00:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:00:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:00:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:00:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:00:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:00:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:00:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:00:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:00:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:33 smithi122 ceph-mon[32255]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:00:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:00:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:33 smithi022 ceph-mon[31360]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:33 smithi022 ceph-mon[40250]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:34.234 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:34.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:34.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:34.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:35 smithi122 ceph-mon[32255]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/803928482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:35 smithi022 ceph-mon[31360]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/803928482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:35 smithi022 ceph-mon[40250]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/803928482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:37.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:37 smithi022 ceph-mon[31360]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:00:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:37 smithi022 ceph-mon[40250]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:37 smithi122 ceph-mon[32255]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:39.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:39.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:39 smithi122 ceph-mon[32255]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:39.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:39.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:39.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:39.975 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:39 smithi022 ceph-mon[31360]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:39.976 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:39 smithi022 ceph-mon[40250]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:40] "GET /metrics HTTP/1.1" 200 202090 "" "Prometheus/2.18.1" 2021-12-15T18:00:40.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/666310160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/666310160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/666310160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:41 smithi122 ceph-mon[32255]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:41 smithi022 ceph-mon[31360]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:41 smithi022 ceph-mon[40250]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:00:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:00:43.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:43 smithi122 ceph-mon[32255]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:00:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:00:44.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:43 smithi022 ceph-mon[31360]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:43 smithi022 ceph-mon[40250]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:44 smithi122 ceph-mon[32255]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:44.976 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:44.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:44.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:44 smithi022 ceph-mon[40250]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:44.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:44.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:44 smithi022 ceph-mon[31360]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:45.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:45.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:45.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3204769649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3204769649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3204769649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:46.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:46 smithi122 ceph-mon[32255]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:46 smithi022 ceph-mon[31360]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:46 smithi022 ceph-mon[40250]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:00:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:49 smithi122 ceph-mon[32255]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:00:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:00:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:00:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:00:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[31360]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:00:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:00:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:00:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:00:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[40250]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:00:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:00:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:00:49.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:00:50.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:50] "GET /metrics HTTP/1.1" 200 202105 "" "Prometheus/2.18.1" 2021-12-15T18:00:50.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:50.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:50.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:51 smithi122 ceph-mon[32255]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4205111109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:51 smithi022 ceph-mon[31360]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4205111109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:51 smithi022 ceph-mon[40250]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4205111109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:00:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:00:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:53 smithi122 ceph-mon[32255]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:53 smithi022 ceph-mon[31360]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:53 smithi022 ceph-mon[40250]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:54.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:00:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:00:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:55 smithi122 ceph-mon[32255]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:00:55.717 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:55.718 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:55 smithi022 ceph-mon[31360]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:55.718 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:00:55.718 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:55 smithi022 ceph-mon[40250]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:56.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:00:56.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:00:56.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:00:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1419463094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:56.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1419463094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1419463094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:00:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:57 smithi122 ceph-mon[32255]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:57 smithi022 ceph-mon[31360]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:00:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:57 smithi022 ceph-mon[40250]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:00:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:00:59 smithi122 ceph-mon[32255]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:00:59 smithi022 ceph-mon[31360]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:00:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:00:59 smithi022 ceph-mon[40250]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:00] "GET /metrics HTTP/1.1" 200 202105 "" "Prometheus/2.18.1" 2021-12-15T18:01:01.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:01.356 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:01 smithi022 ceph-mon[31360]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:01.356 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:01 smithi022 ceph-mon[40250]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:01.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:01.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:01.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:01 smithi122 ceph-mon[32255]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:01:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:01:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2777282901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2777282901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2777282901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:03 smithi122 ceph-mon[32255]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:03 smithi022 ceph-mon[31360]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:03 smithi022 ceph-mon[40250]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:04.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:01:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:01:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:05 smithi122 ceph-mon[32255]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:05 smithi022 ceph-mon[31360]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:05 smithi022 ceph-mon[40250]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:06.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:06.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:06.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:06.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:07 smithi122 ceph-mon[32255]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/982501208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:07 smithi022 ceph-mon[31360]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/982501208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:01:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:07 smithi022 ceph-mon[40250]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/982501208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:09 smithi122 ceph-mon[32255]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:09 smithi022 ceph-mon[31360]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:09 smithi022 ceph-mon[40250]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:10] "GET /metrics HTTP/1.1" 200 202089 "" "Prometheus/2.18.1" 2021-12-15T18:01:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:11 smithi122 ceph-mon[32255]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:11 smithi022 ceph-mon[31360]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:11 smithi022 ceph-mon[40250]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:11.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:12.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:12.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:12.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/65349424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/65349424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/65349424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:01:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:01:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:13 smithi122 ceph-mon[32255]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:13 smithi022 ceph-mon[31360]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:13 smithi022 ceph-mon[40250]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:14.189 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:01:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:01:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:15 smithi122 ceph-mon[32255]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:15 smithi022 ceph-mon[31360]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:15 smithi022 ceph-mon[40250]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:17.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:17.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:17 smithi022 ceph-mon[31360]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:01:17.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:17 smithi022 ceph-mon[40250]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:17.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:17.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:17.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:17 smithi122 ceph-mon[32255]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/663911005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/663911005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/663911005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:19 smithi122 ceph-mon[32255]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:19 smithi022 ceph-mon[31360]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:19 smithi022 ceph-mon[40250]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:20] "GET /metrics HTTP/1.1" 200 202101 "" "Prometheus/2.18.1" 2021-12-15T18:01:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:21 smithi122 ceph-mon[32255]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:21 smithi022 ceph-mon[31360]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:21 smithi022 ceph-mon[40250]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:22.579 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:01:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:01:22.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:22.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:22.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:23 smithi122 ceph-mon[32255]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4032822014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:23 smithi022 ceph-mon[31360]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4032822014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:23 smithi022 ceph-mon[40250]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4032822014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:24.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:01:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:01:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:25 smithi122 ceph-mon[32255]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:25 smithi022 ceph-mon[31360]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:25 smithi022 ceph-mon[40250]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:27 smithi122 ceph-mon[32255]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:27 smithi022 ceph-mon[31360]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:01:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:27 smithi022 ceph-mon[40250]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:27.954 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:28.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:29.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:29.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:29.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3465665214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:29.130 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3465665214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:29.131 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3465665214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:29 smithi122 ceph-mon[32255]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:29 smithi022 ceph-mon[31360]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:29 smithi022 ceph-mon[40250]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:30] "GET /metrics HTTP/1.1" 200 202101 "" "Prometheus/2.18.1" 2021-12-15T18:01:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:31 smithi122 ceph-mon[32255]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:31 smithi022 ceph-mon[31360]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:31 smithi022 ceph-mon[40250]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:01:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:01:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:01:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:01:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:01:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:01:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:01:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:01:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:01:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:01:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:01:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:01:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:01:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:01:33.326 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:33 smithi122 ceph-mon[32255]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:33.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:33.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:33.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:33.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:33 smithi022 ceph-mon[40250]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:33.697 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:33 smithi022 ceph-mon[31360]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:34.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:01:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:01:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/173654830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/173654830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/173654830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:35 smithi122 ceph-mon[32255]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:35 smithi022 ceph-mon[31360]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:35 smithi022 ceph-mon[40250]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:37 smithi122 ceph-mon[32255]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:37 smithi022 ceph-mon[31360]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:01:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:37 smithi022 ceph-mon[40250]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:38.698 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:39.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:39.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:39.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:39 smithi122 ceph-mon[32255]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/94383041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:39 smithi022 ceph-mon[31360]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/94383041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:39 smithi022 ceph-mon[40250]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/94383041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:40] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:01:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:41 smithi122 ceph-mon[32255]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:41 smithi022 ceph-mon[31360]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:41 smithi022 ceph-mon[40250]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:01:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:01:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:43 smithi122 ceph-mon[32255]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:43 smithi022 ceph-mon[31360]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:43 smithi022 ceph-mon[40250]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:44.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:44.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:01:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:01:44.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:44.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:44.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1854669087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1854669087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1854669087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:45 smithi022 ceph-mon[31360]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:45 smithi022 ceph-mon[40250]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:45 smithi122 ceph-mon[32255]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:47 smithi022 ceph-mon[31360]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:01:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:47 smithi022 ceph-mon[40250]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:47 smithi122 ceph-mon[32255]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:49.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:49 smithi122 ceph-mon[32255]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:01:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:01:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:01:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:01:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[31360]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:01:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:01:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:01:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:01:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[40250]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:01:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:01:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:01:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:01:49.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:49.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:49.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:50] "GET /metrics HTTP/1.1" 200 202102 "" "Prometheus/2.18.1" 2021-12-15T18:01:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1404725643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1404725643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1404725643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:51.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:51 smithi022 ceph-mon[31360]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:51 smithi022 ceph-mon[40250]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:51 smithi122 ceph-mon[32255]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:01:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:01:53.756 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:53 smithi122 ceph-mon[32255]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:53 smithi022 ceph-mon[31360]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:53 smithi022 ceph-mon[40250]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:54.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:01:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:01:54.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:01:55.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:01:55.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:01:55.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:01:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:55 smithi022 ceph-mon[31360]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3511769243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:55 smithi022 ceph-mon[40250]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3511769243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:55.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:01:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:55 smithi122 ceph-mon[32255]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3511769243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:01:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:57 smithi022 ceph-mon[31360]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:01:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:57 smithi022 ceph-mon[40250]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:57 smithi122 ceph-mon[32255]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:01:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:01:59 smithi022 ceph-mon[31360]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:01:59 smithi022 ceph-mon[40250]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:01:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:01:59 smithi122 ceph-mon[32255]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:00] "GET /metrics HTTP/1.1" 200 202102 "" "Prometheus/2.18.1" 2021-12-15T18:02:00.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:00.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:00.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:01 smithi022 ceph-mon[31360]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/372839135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:01 smithi022 ceph-mon[40250]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/372839135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:01 smithi122 ceph-mon[32255]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/372839135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:02:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:02:03.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:03 smithi122 ceph-mon[32255]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:03 smithi022 ceph-mon[31360]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:03 smithi022 ceph-mon[40250]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:04.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:02:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:02:05.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:05 smithi022 ceph-mon[31360]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:05 smithi022 ceph-mon[40250]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:05.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:05.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:05.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:05 smithi122 ceph-mon[32255]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3794806241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3794806241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:06.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3794806241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:07 smithi022 ceph-mon[31360]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:02:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:07 smithi022 ceph-mon[40250]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:07 smithi122 ceph-mon[32255]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:09 smithi022 ceph-mon[31360]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:09 smithi022 ceph-mon[40250]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:09 smithi122 ceph-mon[32255]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:10] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:02:10.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:11.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:11.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:11.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:11 smithi022 ceph-mon[31360]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4039899669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:11 smithi022 ceph-mon[40250]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4039899669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:11 smithi122 ceph-mon[32255]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4039899669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:02:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:02:13.773 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:13 smithi122 ceph-mon[32255]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:13 smithi022 ceph-mon[31360]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:13 smithi022 ceph-mon[40250]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:14.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:02:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:02:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:15 smithi022 ceph-mon[31360]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:15 smithi022 ceph-mon[40250]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:15 smithi122 ceph-mon[32255]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:16.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:16.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:16.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:16.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:02:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:17 smithi022 ceph-mon[31360]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1514112046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:17 smithi022 ceph-mon[40250]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1514112046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:17 smithi122 ceph-mon[32255]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1514112046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:19 smithi022 ceph-mon[31360]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:19 smithi022 ceph-mon[40250]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:19 smithi122 ceph-mon[32255]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:20] "GET /metrics HTTP/1.1" 200 202100 "" "Prometheus/2.18.1" 2021-12-15T18:02:21.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:21 smithi022 ceph-mon[31360]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:21 smithi022 ceph-mon[40250]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:21 smithi122 ceph-mon[32255]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:22.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:22.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:22.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2946227771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:02:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:02:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2946227771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2946227771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:23 smithi022 ceph-mon[31360]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:23 smithi022 ceph-mon[40250]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:23.784 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:23 smithi122 ceph-mon[32255]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:24.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:02:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:02:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:25 smithi022 ceph-mon[31360]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:25 smithi022 ceph-mon[40250]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:25 smithi122 ceph-mon[32255]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:27.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:27.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:27.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:02:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:27 smithi022 ceph-mon[31360]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2368527761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:27 smithi022 ceph-mon[40250]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2368527761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:27 smithi122 ceph-mon[32255]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2368527761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:29 smithi022 ceph-mon[31360]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:29 smithi022 ceph-mon[40250]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:29 smithi122 ceph-mon[32255]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:30] "GET /metrics HTTP/1.1" 200 202100 "" "Prometheus/2.18.1" 2021-12-15T18:02:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:31 smithi022 ceph-mon[31360]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:31 smithi022 ceph-mon[40250]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:31 smithi122 ceph-mon[32255]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:32.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:02:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:02:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:02:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:02:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:02:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:02:32.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:32.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:32.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:32.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:02:32.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:02:32.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:02:32.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:02:32.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:02:32.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:02:32.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:02:32.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:02:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:33 smithi022 ceph-mon[31360]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1128142407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:33 smithi022 ceph-mon[40250]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1128142407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:33.795 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:33 smithi122 ceph-mon[32255]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:33.795 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1128142407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:34.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:02:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:02:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:35 smithi022 ceph-mon[31360]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:35 smithi022 ceph-mon[40250]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:35 smithi122 ceph-mon[32255]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:37.756 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:37.756 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:02:37.757 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:37 smithi022 ceph-mon[31360]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:37.757 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:37 smithi022 ceph-mon[40250]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:37 smithi122 ceph-mon[32255]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:38.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:38.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:38.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1514893414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1514893414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1514893414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:39.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:39 smithi122 ceph-mon[32255]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:39 smithi022 ceph-mon[31360]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:39 smithi022 ceph-mon[40250]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:40] "GET /metrics HTTP/1.1" 200 202095 "" "Prometheus/2.18.1" 2021-12-15T18:02:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:41 smithi122 ceph-mon[32255]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:41 smithi022 ceph-mon[31360]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:41 smithi022 ceph-mon[40250]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:02:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:02:43.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:43.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:43.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:43.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:43 smithi022 ceph-mon[31360]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3134924846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:43 smithi022 ceph-mon[40250]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3134924846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:43.805 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:43 smithi122 ceph-mon[32255]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:43.806 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3134924846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:44.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:02:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:02:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:45 smithi122 ceph-mon[32255]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:45 smithi022 ceph-mon[31360]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:45 smithi022 ceph-mon[40250]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:47 smithi022 ceph-mon[31360]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:02:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:47 smithi022 ceph-mon[40250]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:47 smithi122 ceph-mon[32255]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:48.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:48.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:48.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[31360]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1206696885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:02:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:02:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:02:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[40250]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1206696885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:02:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:02:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:02:49.840 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:49 smithi122 ceph-mon[32255]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:49.842 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1206696885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:49.843 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:02:49.843 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:02:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:02:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:50] "GET /metrics HTTP/1.1" 200 202101 "" "Prometheus/2.18.1" 2021-12-15T18:02:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:02:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:50 smithi122 ceph-mon[32255]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:02:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:50 smithi022 ceph-mon[31360]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:02:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:50 smithi022 ceph-mon[40250]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:02:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:02:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:53 smithi122 ceph-mon[32255]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:53 smithi022 ceph-mon[31360]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:53 smithi022 ceph-mon[40250]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:53.866 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:54.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:02:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:02:54.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:54.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:54.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:55 smithi122 ceph-mon[32255]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3919236116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:55 smithi022 ceph-mon[31360]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3919236116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:02:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:55 smithi022 ceph-mon[40250]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3919236116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:02:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:57 smithi122 ceph-mon[32255]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:57 smithi022 ceph-mon[31360]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:02:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:57 smithi022 ceph-mon[40250]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:02:59.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:02:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:02:59 smithi022 ceph-mon[31360]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:02:59 smithi022 ceph-mon[40250]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:02:59.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:02:59.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:02:59.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:02:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:02:59 smithi122 ceph-mon[32255]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/832693251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:00] "GET /metrics HTTP/1.1" 200 202101 "" "Prometheus/2.18.1" 2021-12-15T18:03:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/832693251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/832693251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:01 smithi122 ceph-mon[32255]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:01 smithi022 ceph-mon[31360]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:01 smithi022 ceph-mon[40250]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:03:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:03:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:03 smithi122 ceph-mon[32255]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:03 smithi022 ceph-mon[31360]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:03 smithi022 ceph-mon[40250]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:04.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:03:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:03:04.609 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:04.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:04.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:04.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:05 smithi122 ceph-mon[32255]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1161080903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:05 smithi022 ceph-mon[31360]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1161080903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:05 smithi022 ceph-mon[40250]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1161080903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:07 smithi122 ceph-mon[32255]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:07 smithi022 ceph-mon[31360]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:03:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:07 smithi022 ceph-mon[40250]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:09 smithi122 ceph-mon[32255]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:09 smithi022 ceph-mon[31360]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:09 smithi022 ceph-mon[40250]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:09.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:10.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:10.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:10.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:10] "GET /metrics HTTP/1.1" 200 202097 "" "Prometheus/2.18.1" 2021-12-15T18:03:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3698381052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3698381052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3698381052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:11 smithi122 ceph-mon[32255]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:11 smithi022 ceph-mon[31360]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:11 smithi022 ceph-mon[40250]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:03:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:03:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:13 smithi122 ceph-mon[32255]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:13 smithi022 ceph-mon[31360]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:13 smithi022 ceph-mon[40250]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:14.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:03:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:03:15.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:15 smithi122 ceph-mon[32255]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:15.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:15.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:15.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:15.719 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:15.720 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:15 smithi022 ceph-mon[40250]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:15.720 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:15.720 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:15 smithi022 ceph-mon[31360]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3785473229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:16.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3785473229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:16.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3785473229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:17 smithi122 ceph-mon[32255]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:17 smithi022 ceph-mon[31360]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:03:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:17 smithi022 ceph-mon[40250]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:19 smithi122 ceph-mon[32255]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:19 smithi022 ceph-mon[31360]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:19 smithi022 ceph-mon[40250]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:20] "GET /metrics HTTP/1.1" 200 202094 "" "Prometheus/2.18.1" 2021-12-15T18:03:20.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:21.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:21.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:21.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:21 smithi122 ceph-mon[32255]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/35896161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:21 smithi022 ceph-mon[31360]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/35896161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:21 smithi022 ceph-mon[40250]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/35896161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:03:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:03:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:23 smithi122 ceph-mon[32255]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:23 smithi022 ceph-mon[31360]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:23 smithi022 ceph-mon[40250]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:24.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:03:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:03:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:25 smithi122 ceph-mon[32255]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:25 smithi022 ceph-mon[31360]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:25 smithi022 ceph-mon[40250]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:26.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:27 smithi122 ceph-mon[32255]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/108312830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:03:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:27 smithi022 ceph-mon[31360]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/108312830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:27 smithi022 ceph-mon[40250]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/108312830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:29 smithi122 ceph-mon[32255]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:29 smithi022 ceph-mon[31360]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:29 smithi022 ceph-mon[40250]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:30] "GET /metrics HTTP/1.1" 200 202094 "" "Prometheus/2.18.1" 2021-12-15T18:03:31.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:31 smithi122 ceph-mon[32255]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:31 smithi022 ceph-mon[31360]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:31 smithi022 ceph-mon[40250]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:31.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:31.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:31.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:03:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:03:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2118630509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:03:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:03:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:03:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:03:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2118630509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:03:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:03:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:03:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:03:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2118630509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:03:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:03:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:03:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:03:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:33 smithi122 ceph-mon[32255]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:33 smithi022 ceph-mon[31360]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:33 smithi022 ceph-mon[40250]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:34.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:03:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:03:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:35 smithi122 ceph-mon[32255]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:35 smithi022 ceph-mon[31360]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:35 smithi022 ceph-mon[40250]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:36.825 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:37.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:37.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:37.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:03:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:37 smithi022 ceph-mon[31360]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2188737855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:37 smithi022 ceph-mon[40250]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2188737855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:37 smithi122 ceph-mon[32255]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2188737855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:39 smithi022 ceph-mon[31360]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:39 smithi022 ceph-mon[40250]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:39 smithi122 ceph-mon[32255]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:40] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:03:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:41 smithi022 ceph-mon[31360]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:41 smithi022 ceph-mon[40250]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:41 smithi122 ceph-mon[32255]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:42.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:42.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:42.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:42.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:03:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:03:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:43 smithi022 ceph-mon[31360]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2447280154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:43 smithi022 ceph-mon[40250]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2447280154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:43.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:43 smithi122 ceph-mon[32255]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:43.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2447280154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:44.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:03:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:03:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:45 smithi022 ceph-mon[31360]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:45 smithi022 ceph-mon[40250]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:45 smithi122 ceph-mon[32255]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:47.562 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:47 smithi022 ceph-mon[31360]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:03:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:47 smithi022 ceph-mon[40250]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:47.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:47.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:47.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:47 smithi122 ceph-mon[32255]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2442339573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2442339573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2442339573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:49 smithi022 ceph-mon[31360]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:49 smithi022 ceph-mon[40250]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:49.892 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:49 smithi122 ceph-mon[32255]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:50] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:03:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:03:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:03:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:03:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:03:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:03:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:03:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:03:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:03:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:03:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:03:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:03:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:03:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:51 smithi022 ceph-mon[31360]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:51 smithi022 ceph-mon[40250]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:51 smithi122 ceph-mon[32255]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:03:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:03:52.932 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:53.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:53.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:53.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:53 smithi022 ceph-mon[31360]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2277222022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:53 smithi022 ceph-mon[40250]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2277222022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:53.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:53 smithi122 ceph-mon[32255]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:03:53.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2277222022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:54.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:03:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:03:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:55 smithi022 ceph-mon[31360]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:55 smithi022 ceph-mon[40250]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:03:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:55 smithi122 ceph-mon[32255]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:57 smithi022 ceph-mon[31360]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:03:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:57 smithi022 ceph-mon[40250]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:57 smithi122 ceph-mon[32255]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:03:58.305 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:03:58.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:03:58.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:03:58.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:03:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:59 smithi022 ceph-mon[31360]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:03:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1818668182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:59 smithi022 ceph-mon[40250]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:03:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1818668182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:03:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:59 smithi122 ceph-mon[32255]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:03:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:03:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1818668182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:00] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:04:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:01 smithi022 ceph-mon[31360]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:01 smithi022 ceph-mon[40250]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:01 smithi122 ceph-mon[32255]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:04:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:04:03.678 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:03 smithi022 ceph-mon[31360]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:03 smithi022 ceph-mon[40250]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:03.878 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:03 smithi122 ceph-mon[32255]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:04.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:04.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:04:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:04:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/153930381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/153930381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/153930381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:05 smithi022 ceph-mon[31360]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:05 smithi022 ceph-mon[40250]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:05 smithi122 ceph-mon[32255]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:07 smithi022 ceph-mon[31360]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:04:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:07 smithi022 ceph-mon[40250]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:07 smithi122 ceph-mon[32255]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:09.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:09.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:09.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:09 smithi022 ceph-mon[31360]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3991408166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:09 smithi022 ceph-mon[40250]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3991408166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:09 smithi122 ceph-mon[32255]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3991408166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:10] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:04:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:11 smithi022 ceph-mon[31360]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:11 smithi022 ceph-mon[40250]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:11 smithi122 ceph-mon[32255]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:04:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:04:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:13 smithi022 ceph-mon[31360]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:13 smithi022 ceph-mon[40250]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:13.884 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:13 smithi122 ceph-mon[32255]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:14.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:04:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:04:14.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:14.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:14.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:14.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:15 smithi022 ceph-mon[31360]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3429179883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:15 smithi022 ceph-mon[40250]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3429179883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:15 smithi122 ceph-mon[32255]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3429179883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:17 smithi022 ceph-mon[31360]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:04:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:17 smithi022 ceph-mon[40250]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:17 smithi122 ceph-mon[32255]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:19 smithi022 ceph-mon[31360]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:19 smithi022 ceph-mon[40250]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:19.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:19 smithi122 ceph-mon[32255]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:20.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:20.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:20.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:20] "GET /metrics HTTP/1.1" 200 202096 "" "Prometheus/2.18.1" 2021-12-15T18:04:20.504 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3620030429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3620030429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3620030429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:21 smithi022 ceph-mon[31360]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:21 smithi022 ceph-mon[40250]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:21 smithi122 ceph-mon[32255]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:04:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:04:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:23 smithi022 ceph-mon[31360]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:23 smithi022 ceph-mon[40250]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:23.890 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:23 smithi122 ceph-mon[32255]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:24.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:04:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:04:25.150 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:25.511 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:25.512 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:25 smithi022 ceph-mon[31360]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:25.512 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/382014134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:25.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:25.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:25.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:25 smithi022 ceph-mon[40250]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/382014134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:25 smithi122 ceph-mon[32255]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/382014134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:27 smithi022 ceph-mon[31360]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:04:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:27 smithi022 ceph-mon[40250]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:27 smithi122 ceph-mon[32255]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:29 smithi022 ceph-mon[31360]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:29 smithi022 ceph-mon[40250]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:29 smithi122 ceph-mon[32255]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:30] "GET /metrics HTTP/1.1" 200 202096 "" "Prometheus/2.18.1" 2021-12-15T18:04:30.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:30.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:30.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:30.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:31 smithi022 ceph-mon[31360]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/281559987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:31 smithi022 ceph-mon[40250]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/281559987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:31 smithi122 ceph-mon[32255]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/281559987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:04:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:04:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:04:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:04:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:04:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:04:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:04:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:04:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:04:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:04:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:04:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:04:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:04:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:04:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:33 smithi022 ceph-mon[31360]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:33 smithi022 ceph-mon[40250]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:33.895 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:33 smithi122 ceph-mon[32255]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:34.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:04:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:04:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:35 smithi022 ceph-mon[31360]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:35 smithi022 ceph-mon[40250]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:35.889 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:35 smithi122 ceph-mon[32255]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:36.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:36.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:36.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3413149514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3413149514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3413149514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:37 smithi022 ceph-mon[31360]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:04:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:37 smithi022 ceph-mon[40250]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:37 smithi122 ceph-mon[32255]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:39 smithi122 ceph-mon[32255]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:39 smithi022 ceph-mon[31360]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:39 smithi022 ceph-mon[40250]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:40] "GET /metrics HTTP/1.1" 200 202094 "" "Prometheus/2.18.1" 2021-12-15T18:04:41.258 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:41.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:41.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:41.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:41 smithi122 ceph-mon[32255]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:41 smithi022 ceph-mon[31360]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:41 smithi022 ceph-mon[40250]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2100144395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:04:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:04:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2100144395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2100144395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:43.905 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:43 smithi122 ceph-mon[32255]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:43 smithi022 ceph-mon[31360]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:43 smithi022 ceph-mon[40250]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:44.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:04:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:04:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:45 smithi122 ceph-mon[32255]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:45 smithi022 ceph-mon[31360]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:45 smithi022 ceph-mon[40250]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:46.631 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:47.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:47.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:47.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:04:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:47 smithi022 ceph-mon[31360]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3677498641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:47 smithi022 ceph-mon[40250]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3677498641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:47 smithi122 ceph-mon[32255]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3677498641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:49 smithi122 ceph-mon[32255]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:49 smithi022 ceph-mon[31360]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:49 smithi022 ceph-mon[40250]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:50] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:04:50.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:04:50.730 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:04:50.730 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:04:50.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:04:50.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:04:50.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:04:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:04:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:04:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:04:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:51 smithi122 ceph-mon[32255]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:52.004 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[31360]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:52.004 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:52.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:52.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:52.005 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[40250]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:52.006 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:52.006 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:52.006 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:04:52.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:52.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:52.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:52.562 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:04:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:04:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1791843063' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1791843063' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1791843063' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:53.916 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:53 smithi122 ceph-mon[32255]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:53 smithi022 ceph-mon[31360]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:53 smithi022 ceph-mon[40250]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:54.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:04:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:04:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:55 smithi122 ceph-mon[32255]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:55 smithi022 ceph-mon[31360]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:04:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:55 smithi022 ceph-mon[40250]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:04:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:04:57.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:04:57.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:04:57.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:04:57.748 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:04:57.749 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:57 smithi022 ceph-mon[31360]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:57.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:57 smithi022 ceph-mon[40250]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:57.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:57 smithi122 ceph-mon[32255]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:04:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2149160139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2149160139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2149160139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:04:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:04:59 smithi122 ceph-mon[32255]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:04:59 smithi022 ceph-mon[31360]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:04:59 smithi022 ceph-mon[40250]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:00] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:05:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:01 smithi122 ceph-mon[32255]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:01 smithi022 ceph-mon[31360]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:01 smithi022 ceph-mon[40250]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:05:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:05:02.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:03.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:03.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:03.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:03.927 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:03 smithi122 ceph-mon[32255]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:03.927 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3189838049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:03 smithi022 ceph-mon[40250]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3189838049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:03 smithi022 ceph-mon[31360]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3189838049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:04.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:05:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:05:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:05 smithi122 ceph-mon[32255]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:05 smithi022 ceph-mon[31360]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:05 smithi022 ceph-mon[40250]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:07 smithi022 ceph-mon[31360]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:05:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:07 smithi022 ceph-mon[40250]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:07 smithi122 ceph-mon[32255]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:08.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:08.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:08.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:08.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/453342526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/453342526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/453342526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:09 smithi122 ceph-mon[32255]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:09 smithi022 ceph-mon[31360]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:09 smithi022 ceph-mon[40250]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:10] "GET /metrics HTTP/1.1" 200 202097 "" "Prometheus/2.18.1" 2021-12-15T18:05:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:11 smithi122 ceph-mon[32255]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:11 smithi022 ceph-mon[31360]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:11 smithi022 ceph-mon[40250]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:05:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:05:13.486 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:13 smithi022 ceph-mon[31360]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:13 smithi022 ceph-mon[40250]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:13.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:13.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:13.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:13.933 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:13 smithi122 ceph-mon[32255]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:14.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:05:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:05:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/641142681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/641142681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/641142681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:15 smithi122 ceph-mon[32255]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:15 smithi022 ceph-mon[31360]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:15 smithi022 ceph-mon[40250]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:16 smithi122 ceph-mon[32255]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:16 smithi022 ceph-mon[31360]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:16 smithi022 ceph-mon[40250]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:05:18.850 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:19.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:19.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:19.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:19 smithi022 ceph-mon[31360]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3017265081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:19 smithi022 ceph-mon[40250]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3017265081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:19 smithi122 ceph-mon[32255]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3017265081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:20] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:05:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:21 smithi122 ceph-mon[32255]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:21 smithi022 ceph-mon[31360]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:21 smithi022 ceph-mon[40250]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:05:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:05:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:23 smithi122 ceph-mon[32255]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:23 smithi022 ceph-mon[31360]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:23 smithi022 ceph-mon[40250]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:05:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:05:24.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:24.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:24.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:25 smithi122 ceph-mon[32255]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2536119674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:25 smithi022 ceph-mon[40250]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2536119674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:25 smithi022 ceph-mon[31360]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2536119674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:27 smithi122 ceph-mon[32255]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:05:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:27 smithi022 ceph-mon[40250]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:27 smithi022 ceph-mon[31360]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:29.591 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:29 smithi122 ceph-mon[32255]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:29 smithi022 ceph-mon[31360]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:29 smithi022 ceph-mon[40250]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:29.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:29.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:29.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1458846642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:30] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:05:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1458846642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1458846642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:31 smithi122 ceph-mon[32255]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:31 smithi022 ceph-mon[31360]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:31 smithi022 ceph-mon[40250]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:05:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:05:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:05:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:05:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:05:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:05:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:05:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:05:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:05:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:05:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:05:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:05:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:05:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:05:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:33 smithi122 ceph-mon[32255]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:33 smithi022 ceph-mon[31360]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:33 smithi022 ceph-mon[40250]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:05:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:05:34.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:35.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:35.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:35.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:35 smithi122 ceph-mon[32255]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1387460321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:35 smithi022 ceph-mon[31360]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1387460321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:35 smithi022 ceph-mon[40250]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1387460321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:37 smithi122 ceph-mon[32255]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:37 smithi022 ceph-mon[31360]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:05:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:37 smithi022 ceph-mon[40250]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:39 smithi122 ceph-mon[32255]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:39 smithi022 ceph-mon[31360]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:39 smithi022 ceph-mon[40250]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:40.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:40] "GET /metrics HTTP/1.1" 200 202098 "" "Prometheus/2.18.1" 2021-12-15T18:05:40.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:40.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:40.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:41 smithi122 ceph-mon[32255]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1642891109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:41 smithi022 ceph-mon[31360]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1642891109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:41 smithi022 ceph-mon[40250]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1642891109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:05:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:05:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:43 smithi122 ceph-mon[32255]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:43 smithi022 ceph-mon[31360]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:43 smithi022 ceph-mon[40250]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:05:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:05:45.698 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:45.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:45.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:45 smithi022 ceph-mon[31360]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:45.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:45.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:45 smithi022 ceph-mon[40250]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:45 smithi122 ceph-mon[32255]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:46.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:46.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:46.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3235882800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3235882800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3235882800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:47 smithi022 ceph-mon[31360]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:05:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:47 smithi022 ceph-mon[40250]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:47 smithi122 ceph-mon[32255]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:49 smithi022 ceph-mon[31360]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:49 smithi022 ceph-mon[40250]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:49 smithi122 ceph-mon[32255]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:50] "GET /metrics HTTP/1.1" 200 202093 "" "Prometheus/2.18.1" 2021-12-15T18:05:51.067 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:51.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:51.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:51.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:51 smithi122 ceph-mon[32255]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:05:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:05:51.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:05:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[31360]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:05:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:05:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:05:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/714449060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[40250]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:05:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:05:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:05:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/714449060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/714449060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:52.617 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:05:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:05:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:05:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:52 smithi122 ceph-mon[32255]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:05:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:52 smithi022 ceph-mon[31360]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:05:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:52 smithi022 ceph-mon[40250]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:05:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:05:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:55 smithi022 ceph-mon[31360]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:55 smithi022 ceph-mon[40250]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:05:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:55 smithi122 ceph-mon[32255]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:56.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:05:56.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:05:56.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:05:56.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:05:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:57 smithi122 ceph-mon[32255]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2189142841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:05:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:57 smithi022 ceph-mon[31360]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2189142841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:57 smithi022 ceph-mon[40250]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:05:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2189142841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:05:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:05:59 smithi122 ceph-mon[32255]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:05:59 smithi022 ceph-mon[31360]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:05:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:05:59 smithi022 ceph-mon[40250]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:00] "GET /metrics HTTP/1.1" 200 202093 "" "Prometheus/2.18.1" 2021-12-15T18:06:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:01 smithi122 ceph-mon[32255]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:01 smithi022 ceph-mon[31360]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:01 smithi022 ceph-mon[40250]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:01.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:02.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:02.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:02.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1877102790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1877102790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1877102790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:06:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:06:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:03 smithi122 ceph-mon[32255]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:03 smithi022 ceph-mon[31360]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:03 smithi022 ceph-mon[40250]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:04.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:06:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:06:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:05 smithi122 ceph-mon[32255]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:05 smithi022 ceph-mon[31360]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:05 smithi022 ceph-mon[40250]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:07.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:07 smithi022 ceph-mon[31360]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:06:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:07 smithi022 ceph-mon[40250]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:07.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:07.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:07.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:07 smithi122 ceph-mon[32255]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/155268887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/155268887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/155268887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:09 smithi122 ceph-mon[32255]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:09 smithi022 ceph-mon[31360]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:09 smithi022 ceph-mon[40250]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:10] "GET /metrics HTTP/1.1" 200 202097 "" "Prometheus/2.18.1" 2021-12-15T18:06:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:11 smithi022 ceph-mon[31360]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:11 smithi022 ceph-mon[40250]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:11 smithi122 ceph-mon[32255]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:12.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:06:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:06:12.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:12.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:12.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:13 smithi022 ceph-mon[31360]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1644778759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:13 smithi022 ceph-mon[40250]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1644778759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:13 smithi122 ceph-mon[32255]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1644778759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:06:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:06:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:15 smithi022 ceph-mon[31360]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:15 smithi022 ceph-mon[40250]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:15 smithi122 ceph-mon[32255]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:17 smithi022 ceph-mon[31360]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:06:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:17 smithi022 ceph-mon[40250]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:17 smithi122 ceph-mon[32255]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:18.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:18.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:18.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3067615253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3067615253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3067615253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:19 smithi022 ceph-mon[31360]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:19 smithi022 ceph-mon[40250]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:19 smithi122 ceph-mon[32255]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:20] "GET /metrics HTTP/1.1" 200 202096 "" "Prometheus/2.18.1" 2021-12-15T18:06:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:21 smithi022 ceph-mon[31360]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:21 smithi022 ceph-mon[40250]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:21 smithi122 ceph-mon[32255]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:06:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:06:23.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:23.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:23.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:23.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:23.670 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:23 smithi022 ceph-mon[31360]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:23.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:23 smithi022 ceph-mon[40250]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:23 smithi122 ceph-mon[32255]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:06:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:06:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187329112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:24.884 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187329112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187329112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:25 smithi022 ceph-mon[31360]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:25 smithi022 ceph-mon[40250]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:25 smithi122 ceph-mon[32255]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:27 smithi022 ceph-mon[31360]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:06:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:27 smithi022 ceph-mon[40250]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:27 smithi122 ceph-mon[32255]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:28.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:29.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:29.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:29.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:29 smithi022 ceph-mon[31360]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/513592620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:29 smithi022 ceph-mon[40250]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/513592620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:29 smithi122 ceph-mon[32255]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/513592620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:30] "GET /metrics HTTP/1.1" 200 202096 "" "Prometheus/2.18.1" 2021-12-15T18:06:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:31 smithi022 ceph-mon[31360]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:31 smithi022 ceph-mon[40250]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:31 smithi122 ceph-mon[32255]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:06:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:06:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:06:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:06:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:06:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:06:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:06:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:06:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:06:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:06:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:06:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:33 smithi022 ceph-mon[31360]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:33 smithi022 ceph-mon[40250]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:33 smithi122 ceph-mon[32255]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:34.042 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:34.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:34.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:34.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:06:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:06:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4209569673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4209569673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4209569673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:35 smithi022 ceph-mon[31360]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:35 smithi022 ceph-mon[40250]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:35 smithi122 ceph-mon[32255]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:37 smithi022 ceph-mon[31360]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:06:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:37 smithi022 ceph-mon[40250]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:37 smithi122 ceph-mon[32255]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:39.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:39 smithi022 ceph-mon[31360]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:39 smithi022 ceph-mon[40250]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:39.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:39.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:39.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:39 smithi122 ceph-mon[32255]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:40] "GET /metrics HTTP/1.1" 200 202097 "" "Prometheus/2.18.1" 2021-12-15T18:06:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1886110275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1886110275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1886110275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:41 smithi022 ceph-mon[31360]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:41 smithi022 ceph-mon[40250]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:41 smithi122 ceph-mon[32255]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:06:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:06:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:43 smithi022 ceph-mon[31360]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:43 smithi022 ceph-mon[40250]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:43 smithi122 ceph-mon[32255]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:06:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:06:44.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:45 smithi022 ceph-mon[31360]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1052605046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:45 smithi022 ceph-mon[40250]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1052605046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:45 smithi122 ceph-mon[32255]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1052605046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:47 smithi022 ceph-mon[31360]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:06:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:47 smithi022 ceph-mon[40250]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:47 smithi122 ceph-mon[32255]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:49 smithi022 ceph-mon[31360]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:49 smithi022 ceph-mon[40250]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:49 smithi122 ceph-mon[32255]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:50.160 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:50] "GET /metrics HTTP/1.1" 200 202096 "" "Prometheus/2.18.1" 2021-12-15T18:06:50.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:50.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:50.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:51 smithi022 ceph-mon[31360]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1348872422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:51 smithi022 ceph-mon[40250]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1348872422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:51 smithi122 ceph-mon[32255]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1348872422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:06:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:06:52.508 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:06:52.509 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:06:52.509 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:06:52.510 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:06:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:06:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:06:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:06:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:06:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:06:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:06:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:06:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:06:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:53 smithi022 ceph-mon[31360]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:53 smithi022 ceph-mon[40250]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:53 smithi122 ceph-mon[32255]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:06:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:06:55.536 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:06:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:55 smithi022 ceph-mon[31360]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:55 smithi022 ceph-mon[40250]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:06:55.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:06:55.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:06:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:06:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:55 smithi122 ceph-mon[32255]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/428431712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/428431712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/428431712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:06:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:57 smithi022 ceph-mon[31360]: pgmap v255: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:06:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:57 smithi022 ceph-mon[40250]: pgmap v255: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:57 smithi122 ceph-mon[32255]: pgmap v255: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:06:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:06:59 smithi022 ceph-mon[31360]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:06:59 smithi022 ceph-mon[40250]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:06:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:06:59 smithi122 ceph-mon[32255]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:00] "GET /metrics HTTP/1.1" 200 202096 "" "Prometheus/2.18.1" 2021-12-15T18:07:00.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:01.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:01.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:01.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:01 smithi022 ceph-mon[31360]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2571664797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:01 smithi022 ceph-mon[40250]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2571664797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:01 smithi122 ceph-mon[32255]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2571664797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:07:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:07:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:03 smithi022 ceph-mon[31360]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:03 smithi022 ceph-mon[40250]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:03 smithi122 ceph-mon[32255]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:04.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:07:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:07:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:05 smithi022 ceph-mon[31360]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:05 smithi022 ceph-mon[40250]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:05 smithi122 ceph-mon[32255]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:06.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:06.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:06.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:06.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:07:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:07 smithi022 ceph-mon[31360]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/520581989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:07 smithi022 ceph-mon[40250]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/520581989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:07 smithi122 ceph-mon[32255]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/520581989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:09 smithi122 ceph-mon[32255]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:09 smithi022 ceph-mon[31360]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:09 smithi022 ceph-mon[40250]: pgmap v261: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:10] "GET /metrics HTTP/1.1" 200 202099 "" "Prometheus/2.18.1" 2021-12-15T18:07:11.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:11 smithi122 ceph-mon[32255]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:12.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:12.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:12.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:12.016 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:11 smithi022 ceph-mon[31360]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:12.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:11 smithi022 ceph-mon[40250]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1831077844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:07:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:07:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1831077844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1831077844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:13 smithi122 ceph-mon[32255]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:13 smithi022 ceph-mon[31360]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:13 smithi022 ceph-mon[40250]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:07:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:07:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:15 smithi122 ceph-mon[32255]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:15 smithi022 ceph-mon[31360]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:15 smithi022 ceph-mon[40250]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:17.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:17.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:17.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:17.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:07:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:17 smithi022 ceph-mon[31360]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2127021981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:17 smithi022 ceph-mon[40250]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2127021981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:17 smithi122 ceph-mon[32255]: pgmap v265: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2127021981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:19 smithi122 ceph-mon[32255]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:19 smithi022 ceph-mon[31360]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:19 smithi022 ceph-mon[40250]: pgmap v266: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:20] "GET /metrics HTTP/1.1" 200 202100 "" "Prometheus/2.18.1" 2021-12-15T18:07:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:21 smithi122 ceph-mon[32255]: pgmap v267: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:21 smithi022 ceph-mon[31360]: pgmap v267: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:21 smithi022 ceph-mon[40250]: pgmap v267: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:22.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:07:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:07:22.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:22.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:22.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:23 smithi122 ceph-mon[32255]: pgmap v268: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2623065331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:23 smithi022 ceph-mon[31360]: pgmap v268: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2623065331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:23 smithi022 ceph-mon[40250]: pgmap v268: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2623065331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:07:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:07:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:25 smithi122 ceph-mon[32255]: pgmap v269: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:25 smithi022 ceph-mon[31360]: pgmap v269: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:25 smithi022 ceph-mon[40250]: pgmap v269: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:27.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:27.761 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:07:27.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:27 smithi022 ceph-mon[31360]: pgmap v270: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:27.762 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:27 smithi022 ceph-mon[40250]: pgmap v270: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:27 smithi122 ceph-mon[32255]: pgmap v270: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:28.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:28.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:28.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1123170663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1123170663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1123170663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:29 smithi122 ceph-mon[32255]: pgmap v271: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:29 smithi022 ceph-mon[40250]: pgmap v271: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:29 smithi022 ceph-mon[31360]: pgmap v271: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:30] "GET /metrics HTTP/1.1" 200 202100 "" "Prometheus/2.18.1" 2021-12-15T18:07:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:31 smithi122 ceph-mon[32255]: pgmap v272: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:31 smithi022 ceph-mon[31360]: pgmap v272: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:31 smithi022 ceph-mon[40250]: pgmap v272: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:32.580 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:07:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:07:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:07:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:07:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:07:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:07:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:07:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:07:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:07:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:07:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:07:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:07:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:07:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:07:33.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:33.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:33.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:33.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:33 smithi022 ceph-mon[31360]: pgmap v273: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2071231979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:33 smithi022 ceph-mon[40250]: pgmap v273: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2071231979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:33 smithi122 ceph-mon[32255]: pgmap v273: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2071231979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:07:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:07:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:35 smithi122 ceph-mon[32255]: pgmap v274: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:35 smithi022 ceph-mon[31360]: pgmap v274: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:35 smithi022 ceph-mon[40250]: pgmap v274: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:37 smithi022 ceph-mon[31360]: pgmap v275: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:07:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:37 smithi022 ceph-mon[40250]: pgmap v275: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:37 smithi122 ceph-mon[32255]: pgmap v275: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:38.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:38.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:38.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:38.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:39 smithi122 ceph-mon[32255]: pgmap v276: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2993973195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:39 smithi022 ceph-mon[40250]: pgmap v276: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2993973195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:39 smithi022 ceph-mon[31360]: pgmap v276: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2993973195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:40] "GET /metrics HTTP/1.1" 200 202091 "" "Prometheus/2.18.1" 2021-12-15T18:07:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:41 smithi122 ceph-mon[32255]: pgmap v277: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:41 smithi022 ceph-mon[31360]: pgmap v277: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:41 smithi022 ceph-mon[40250]: pgmap v277: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:07:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:07:43.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:43.870 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:43 smithi022 ceph-mon[40250]: pgmap v278: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:43.871 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:43 smithi022 ceph-mon[31360]: pgmap v278: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:43 smithi122 ceph-mon[32255]: pgmap v278: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:44.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:44.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:44.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:07:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:07:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3290090686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3290090686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3290090686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:45 smithi122 ceph-mon[32255]: pgmap v279: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:45 smithi022 ceph-mon[31360]: pgmap v279: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:45 smithi022 ceph-mon[40250]: pgmap v279: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:47 smithi022 ceph-mon[31360]: pgmap v280: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:07:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:47 smithi022 ceph-mon[40250]: pgmap v280: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:47 smithi122 ceph-mon[32255]: pgmap v280: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:49.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:49.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:49.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:49.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:49 smithi122 ceph-mon[32255]: pgmap v281: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3261126185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:49 smithi022 ceph-mon[31360]: pgmap v281: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3261126185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:49 smithi022 ceph-mon[40250]: pgmap v281: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:07:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3261126185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:50] "GET /metrics HTTP/1.1" 200 202100 "" "Prometheus/2.18.1" 2021-12-15T18:07:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:51 smithi122 ceph-mon[32255]: pgmap v282: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:51 smithi022 ceph-mon[31360]: pgmap v282: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:51 smithi022 ceph-mon[40250]: pgmap v282: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:07:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:07:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:07:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:07:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:07:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:07:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:07:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:07:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:07:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:07:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:07:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:07:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:07:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:07:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:53 smithi122 ceph-mon[32255]: pgmap v283: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:53 smithi022 ceph-mon[31360]: pgmap v283: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:53 smithi022 ceph-mon[40250]: pgmap v283: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:07:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:07:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:07:54.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:54.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:07:54.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:07:54.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:07:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:55 smithi122 ceph-mon[32255]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T18:07:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2327963086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:55 smithi022 ceph-mon[31360]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T18:07:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2327963086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:07:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:55 smithi022 ceph-mon[40250]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T18:07:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2327963086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:07:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:57 smithi022 ceph-mon[31360]: pgmap v285: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T18:07:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:07:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:57 smithi022 ceph-mon[40250]: pgmap v285: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T18:07:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:57 smithi122 ceph-mon[32255]: pgmap v285: 132 pgs: 132 active+clean; 582 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T18:07:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:07:59 smithi122 ceph-mon[32255]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:07:59.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:07:59.979 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:07:59 smithi022 ceph-mon[31360]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:07:59.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:07:59 smithi022 ceph-mon[40250]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:00.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:00.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:00.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:00] "GET /metrics HTTP/1.1" 200 202100 "" "Prometheus/2.18.1" 2021-12-15T18:08:00.623 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4054639767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4054639767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4054639767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:01 smithi122 ceph-mon[32255]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:01 smithi022 ceph-mon[31360]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:01 smithi022 ceph-mon[40250]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:08:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:08:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:03 smithi122 ceph-mon[32255]: pgmap v288: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:03 smithi022 ceph-mon[31360]: pgmap v288: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:03 smithi022 ceph-mon[40250]: pgmap v288: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:04.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:08:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:08:05.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:05.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:05.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:05.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:05 smithi122 ceph-mon[32255]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:05 smithi022 ceph-mon[31360]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:05 smithi022 ceph-mon[40250]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:08:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/121780597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/121780597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/121780597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:07 smithi022 ceph-mon[31360]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2021-12-15T18:08:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:08:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:07 smithi022 ceph-mon[40250]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2021-12-15T18:08:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:07 smithi122 ceph-mon[32255]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2021-12-15T18:08:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:08 smithi122 ceph-mon[32255]: pgmap v291: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T18:08:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:08 smithi022 ceph-mon[31360]: pgmap v291: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T18:08:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:08 smithi022 ceph-mon[40250]: pgmap v291: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T18:08:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:10] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:08:10.725 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:11.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:11.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:11.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:11 smithi022 ceph-mon[31360]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2811194899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:11 smithi022 ceph-mon[40250]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2811194899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:11 smithi122 ceph-mon[32255]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2811194899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:08:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:08:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:13 smithi022 ceph-mon[31360]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:13 smithi022 ceph-mon[40250]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:13 smithi122 ceph-mon[32255]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:08:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:08:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:15 smithi022 ceph-mon[31360]: pgmap v294: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:15 smithi022 ceph-mon[40250]: pgmap v294: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:15 smithi122 ceph-mon[32255]: pgmap v294: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:16.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:16.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:16.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:08:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:17 smithi022 ceph-mon[31360]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4266028900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:17 smithi022 ceph-mon[40250]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4266028900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:17 smithi122 ceph-mon[32255]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4266028900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:19 smithi022 ceph-mon[31360]: pgmap v296: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:19 smithi022 ceph-mon[40250]: pgmap v296: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:19 smithi122 ceph-mon[32255]: pgmap v296: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:20] "GET /metrics HTTP/1.1" 200 202145 "" "Prometheus/2.18.1" 2021-12-15T18:08:21.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:21 smithi022 ceph-mon[31360]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:21 smithi022 ceph-mon[40250]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:21.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:21.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:21.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:21 smithi122 ceph-mon[32255]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1048815699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:08:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:08:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1048815699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1048815699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:23 smithi022 ceph-mon[31360]: pgmap v298: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:23 smithi022 ceph-mon[40250]: pgmap v298: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:23 smithi122 ceph-mon[32255]: pgmap v298: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:08:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:08:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:25 smithi022 ceph-mon[31360]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:25 smithi022 ceph-mon[40250]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:25 smithi122 ceph-mon[32255]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:26.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:27.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:27.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:27.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:27.465 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:08:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:27 smithi022 ceph-mon[31360]: pgmap v300: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2834079329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:27 smithi022 ceph-mon[40250]: pgmap v300: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2834079329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:27 smithi122 ceph-mon[32255]: pgmap v300: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2834079329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:29 smithi022 ceph-mon[31360]: pgmap v301: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:29 smithi022 ceph-mon[40250]: pgmap v301: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:29 smithi122 ceph-mon[32255]: pgmap v301: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:30] "GET /metrics HTTP/1.1" 200 202145 "" "Prometheus/2.18.1" 2021-12-15T18:08:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:31 smithi022 ceph-mon[31360]: pgmap v302: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:31 smithi022 ceph-mon[40250]: pgmap v302: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:31 smithi122 ceph-mon[32255]: pgmap v302: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:32.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:32.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:08:32.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:08:32.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:08:32.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:08:32.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:32.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:32.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:08:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:08:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:08:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:08:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:08:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:08:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:08:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:08:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:08:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:08:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:33 smithi022 ceph-mon[31360]: pgmap v303: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/512896405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:33 smithi022 ceph-mon[40250]: pgmap v303: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/512896405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:33 smithi122 ceph-mon[32255]: pgmap v303: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/512896405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:08:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:08:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:35 smithi022 ceph-mon[31360]: pgmap v304: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:35 smithi022 ceph-mon[40250]: pgmap v304: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:35 smithi122 ceph-mon[32255]: pgmap v304: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:37.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:37 smithi022 ceph-mon[31360]: pgmap v305: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:08:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:37 smithi022 ceph-mon[40250]: pgmap v305: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:37.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:37 smithi122 ceph-mon[32255]: pgmap v305: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:38.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:38.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:38.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2502024827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2502024827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2502024827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:39 smithi022 ceph-mon[31360]: pgmap v306: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:39 smithi022 ceph-mon[40250]: pgmap v306: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:39 smithi122 ceph-mon[32255]: pgmap v306: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:40] "GET /metrics HTTP/1.1" 200 202141 "" "Prometheus/2.18.1" 2021-12-15T18:08:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:41 smithi022 ceph-mon[31360]: pgmap v307: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:41 smithi022 ceph-mon[40250]: pgmap v307: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:41 smithi122 ceph-mon[32255]: pgmap v307: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:08:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:08:42.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:43.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:43.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:43.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:43 smithi022 ceph-mon[40250]: pgmap v308: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/443342305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:43 smithi022 ceph-mon[31360]: pgmap v308: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/443342305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:43 smithi122 ceph-mon[32255]: pgmap v308: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/443342305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:08:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:08:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:45 smithi022 ceph-mon[31360]: pgmap v309: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:45 smithi022 ceph-mon[40250]: pgmap v309: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:45 smithi122 ceph-mon[32255]: pgmap v309: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:47 smithi022 ceph-mon[31360]: pgmap v310: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:08:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:47 smithi022 ceph-mon[40250]: pgmap v310: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:47 smithi122 ceph-mon[32255]: pgmap v310: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:48.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:48.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:48.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:48.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:49 smithi022 ceph-mon[31360]: pgmap v311: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1101766529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:49 smithi022 ceph-mon[40250]: pgmap v311: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1101766529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:49 smithi122 ceph-mon[32255]: pgmap v311: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1101766529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:50] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:08:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:51 smithi022 ceph-mon[31360]: pgmap v312: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:51 smithi022 ceph-mon[40250]: pgmap v312: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:51 smithi122 ceph-mon[32255]: pgmap v312: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:52.584 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:08:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:08:53.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[31360]: pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:08:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:08:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:08:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[40250]: pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:08:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:08:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:08:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:53 smithi122 ceph-mon[32255]: pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:08:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:08:53.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:08:54.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:54.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:54.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:08:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:08:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2491273783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2491273783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2491273783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:08:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:55 smithi022 ceph-mon[31360]: pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:55 smithi022 ceph-mon[40250]: pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:08:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:55 smithi122 ceph-mon[32255]: pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:57.381 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:57.381 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:57 smithi122 ceph-mon[32255]: pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:57.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:57 smithi022 ceph-mon[31360]: pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:57.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:57 smithi022 ceph-mon[40250]: pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:08:57.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:08:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:08:59.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:08:59.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:08:59 smithi022 ceph-mon[31360]: pgmap v316: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:59.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:08:59.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:08:59.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:08:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:08:59 smithi122 ceph-mon[32255]: pgmap v316: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:08:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:08:59 smithi022 ceph-mon[40250]: pgmap v316: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:00] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:09:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/992747412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/992747412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/992747412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:01 smithi022 ceph-mon[31360]: pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:01 smithi022 ceph-mon[40250]: pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:01 smithi122 ceph-mon[32255]: pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:09:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:09:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:03 smithi022 ceph-mon[31360]: pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:03 smithi022 ceph-mon[40250]: pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:03 smithi122 ceph-mon[32255]: pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:04.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:04.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:09:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:09:04.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:04.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:04.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:05 smithi022 ceph-mon[31360]: pgmap v319: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2291363432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:05 smithi022 ceph-mon[40250]: pgmap v319: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2291363432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:05 smithi122 ceph-mon[32255]: pgmap v319: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2291363432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:07 smithi022 ceph-mon[31360]: pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:09:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:07 smithi022 ceph-mon[40250]: pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:07 smithi122 ceph-mon[32255]: pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:09 smithi022 ceph-mon[31360]: pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:09 smithi022 ceph-mon[40250]: pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:09.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:09 smithi122 ceph-mon[32255]: pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:10.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:10.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:10.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:10] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:09:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1913577392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1913577392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1913577392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:11 smithi022 ceph-mon[31360]: pgmap v322: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:11 smithi022 ceph-mon[40250]: pgmap v322: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:11 smithi122 ceph-mon[32255]: pgmap v322: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:09:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:09:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:13 smithi022 ceph-mon[31360]: pgmap v323: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:13 smithi022 ceph-mon[40250]: pgmap v323: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:13 smithi122 ceph-mon[32255]: pgmap v323: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:09:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:09:15.176 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:15.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:15.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:15.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:15 smithi022 ceph-mon[31360]: pgmap v324: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:15 smithi022 ceph-mon[40250]: pgmap v324: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:15 smithi122 ceph-mon[32255]: pgmap v324: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3012620452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3012620452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3012620452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:17 smithi022 ceph-mon[31360]: pgmap v325: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:09:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:17 smithi022 ceph-mon[40250]: pgmap v325: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:17 smithi122 ceph-mon[32255]: pgmap v325: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:19 smithi022 ceph-mon[31360]: pgmap v326: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:19 smithi022 ceph-mon[40250]: pgmap v326: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:19 smithi122 ceph-mon[32255]: pgmap v326: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:20] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:09:20.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:20.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:20.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:20.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:21 smithi022 ceph-mon[31360]: pgmap v327: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744995512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:21 smithi022 ceph-mon[40250]: pgmap v327: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744995512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:21 smithi122 ceph-mon[32255]: pgmap v327: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744995512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:09:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:09:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:23 smithi022 ceph-mon[31360]: pgmap v328: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:23 smithi022 ceph-mon[40250]: pgmap v328: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:23 smithi122 ceph-mon[32255]: pgmap v328: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:09:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:09:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:25 smithi022 ceph-mon[31360]: pgmap v329: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:25 smithi022 ceph-mon[40250]: pgmap v329: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:25.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:25 smithi122 ceph-mon[32255]: pgmap v329: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2568889157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2568889157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2568889157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:27 smithi022 ceph-mon[31360]: pgmap v330: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:09:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:27 smithi022 ceph-mon[40250]: pgmap v330: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:27 smithi122 ceph-mon[32255]: pgmap v330: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:29 smithi022 ceph-mon[31360]: pgmap v331: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:29 smithi022 ceph-mon[40250]: pgmap v331: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:29 smithi122 ceph-mon[32255]: pgmap v331: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:30] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:09:31.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:31.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:31.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:31.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:31.659 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:31 smithi022 ceph-mon[31360]: pgmap v332: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:31.659 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:31 smithi022 ceph-mon[40250]: pgmap v332: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:31 smithi122 ceph-mon[32255]: pgmap v332: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:09:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:09:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3395971802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:09:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:09:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:09:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:09:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3395971802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:09:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:09:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:09:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:09:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3395971802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:09:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:09:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:09:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:09:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:33 smithi022 ceph-mon[31360]: pgmap v333: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:33 smithi022 ceph-mon[40250]: pgmap v333: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:33 smithi122 ceph-mon[32255]: pgmap v333: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:09:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:09:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:35 smithi022 ceph-mon[31360]: pgmap v334: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:35 smithi022 ceph-mon[40250]: pgmap v334: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:35 smithi122 ceph-mon[32255]: pgmap v334: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:36.659 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:37.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:37.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:37.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:09:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:37 smithi022 ceph-mon[40250]: pgmap v335: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1462804004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:37 smithi022 ceph-mon[31360]: pgmap v335: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1462804004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:37 smithi122 ceph-mon[32255]: pgmap v335: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1462804004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:39 smithi022 ceph-mon[31360]: pgmap v336: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:39 smithi022 ceph-mon[40250]: pgmap v336: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:39 smithi122 ceph-mon[32255]: pgmap v336: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:40] "GET /metrics HTTP/1.1" 200 202150 "" "Prometheus/2.18.1" 2021-12-15T18:09:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:41 smithi022 ceph-mon[31360]: pgmap v337: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:41 smithi022 ceph-mon[40250]: pgmap v337: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:41 smithi122 ceph-mon[32255]: pgmap v337: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:42.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:42.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:42.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:42.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2942334573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:09:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:09:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2942334573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2942334573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:43 smithi022 ceph-mon[31360]: pgmap v338: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:43 smithi022 ceph-mon[40250]: pgmap v338: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:43 smithi122 ceph-mon[32255]: pgmap v338: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:09:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:09:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:45 smithi022 ceph-mon[31360]: pgmap v339: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:45 smithi022 ceph-mon[40250]: pgmap v339: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:45 smithi122 ceph-mon[32255]: pgmap v339: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:47.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:47.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:47.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:47.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:09:47.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:47 smithi022 ceph-mon[31360]: pgmap v340: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:47 smithi022 ceph-mon[40250]: pgmap v340: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:47 smithi122 ceph-mon[32255]: pgmap v340: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3091260421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3091260421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3091260421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:49 smithi122 ceph-mon[32255]: pgmap v341: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:49 smithi022 ceph-mon[31360]: pgmap v341: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:49 smithi022 ceph-mon[40250]: pgmap v341: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:50] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:09:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:51 smithi122 ceph-mon[32255]: pgmap v342: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:51 smithi022 ceph-mon[31360]: pgmap v342: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:51 smithi022 ceph-mon[40250]: pgmap v342: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:09:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:09:52.778 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:53.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:53.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:53.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:53 smithi122 ceph-mon[32255]: pgmap v343: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/255446040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:53 smithi022 ceph-mon[31360]: pgmap v343: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/255446040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:53 smithi022 ceph-mon[40250]: pgmap v343: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/255446040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:09:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:09:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:55 smithi122 ceph-mon[32255]: pgmap v344: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:55 smithi022 ceph-mon[31360]: pgmap v344: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:09:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:55 smithi022 ceph-mon[40250]: pgmap v344: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:57 smithi122 ceph-mon[32255]: pgmap v345: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:09:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:09:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:09:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:09:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[31360]: pgmap v345: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:09:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:09:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:09:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[40250]: pgmap v345: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:09:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:09:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:09:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:09:58.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:09:58.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:09:58.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:09:58.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:09:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:09:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:58 smithi122 ceph-mon[32255]: pgmap v346: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:09:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2821123850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:09:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:58 smithi022 ceph-mon[40250]: pgmap v346: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:09:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2821123850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:09:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:09:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:58 smithi022 ceph-mon[31360]: pgmap v346: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:09:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:09:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2821123850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:00] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:10:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T18:10:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T18:10:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T18:10:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:01 smithi022 ceph-mon[40250]: pgmap v347: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:01 smithi022 ceph-mon[31360]: pgmap v347: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:01.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:01 smithi122 ceph-mon[32255]: pgmap v347: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:10:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:10:03.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:03 smithi022 ceph-mon[31360]: pgmap v348: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:03 smithi022 ceph-mon[40250]: pgmap v348: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:03.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:03.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:03.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:03 smithi122 ceph-mon[32255]: pgmap v348: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:04.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:10:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:10:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3313314961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3313314961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3313314961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:05.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:05 smithi022 ceph-mon[31360]: pgmap v349: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:05.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:05.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:05 smithi022 ceph-mon[40250]: pgmap v349: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:05 smithi122 ceph-mon[32255]: pgmap v349: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:07 smithi022 ceph-mon[31360]: pgmap v350: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:10:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:07 smithi022 ceph-mon[40250]: pgmap v350: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:07 smithi122 ceph-mon[32255]: pgmap v350: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:08.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:09.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:09.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:09.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:09 smithi022 ceph-mon[31360]: pgmap v351: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1897166133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:09 smithi022 ceph-mon[40250]: pgmap v351: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1897166133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:09 smithi122 ceph-mon[32255]: pgmap v351: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1897166133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:10] "GET /metrics HTTP/1.1" 200 202152 "" "Prometheus/2.18.1" 2021-12-15T18:10:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:11 smithi022 ceph-mon[31360]: pgmap v352: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:11 smithi022 ceph-mon[40250]: pgmap v352: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:11 smithi122 ceph-mon[32255]: pgmap v352: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:10:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:10:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:13 smithi022 ceph-mon[31360]: pgmap v353: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:13 smithi022 ceph-mon[40250]: pgmap v353: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:13 smithi122 ceph-mon[32255]: pgmap v353: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:14.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:10:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:10:14.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:14.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:14.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:15 smithi022 ceph-mon[31360]: pgmap v354: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/337113765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:15 smithi022 ceph-mon[40250]: pgmap v354: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/337113765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:15 smithi122 ceph-mon[32255]: pgmap v354: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/337113765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:17 smithi022 ceph-mon[31360]: pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:10:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:17 smithi022 ceph-mon[40250]: pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:17 smithi122 ceph-mon[32255]: pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:19.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:19 smithi022 ceph-mon[31360]: pgmap v356: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:19 smithi022 ceph-mon[40250]: pgmap v356: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:19 smithi122 ceph-mon[32255]: pgmap v356: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:19.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:19.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:19.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:20] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:10:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/940653200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/940653200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/940653200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:21 smithi022 ceph-mon[31360]: pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:21 smithi022 ceph-mon[40250]: pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:21 smithi122 ceph-mon[32255]: pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:10:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:10:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:23 smithi022 ceph-mon[31360]: pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:23 smithi022 ceph-mon[40250]: pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:23 smithi122 ceph-mon[32255]: pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:10:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:10:24.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:25.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:25.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:25.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:25 smithi022 ceph-mon[31360]: pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1414754911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:25 smithi022 ceph-mon[40250]: pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1414754911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:25 smithi122 ceph-mon[32255]: pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1414754911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:27 smithi022 ceph-mon[31360]: pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:10:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:27 smithi022 ceph-mon[40250]: pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:27 smithi122 ceph-mon[32255]: pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:29 smithi022 ceph-mon[31360]: pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:29 smithi022 ceph-mon[40250]: pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:29 smithi122 ceph-mon[32255]: pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:30.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:30] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:10:30.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:30.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:30.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:31 smithi022 ceph-mon[31360]: pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1884067940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:31 smithi022 ceph-mon[40250]: pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1884067940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:31 smithi122 ceph-mon[32255]: pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1884067940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:10:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:10:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:10:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:10:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:10:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:10:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:10:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:10:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:10:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:10:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:10:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:10:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:10:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:10:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:33 smithi022 ceph-mon[31360]: pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:33 smithi022 ceph-mon[40250]: pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:33 smithi122 ceph-mon[32255]: pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:34.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:10:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:10:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:35 smithi022 ceph-mon[31360]: pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:35 smithi022 ceph-mon[40250]: pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:35 smithi122 ceph-mon[32255]: pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:36.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:36.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:36.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:36.521 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3852160020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3852160020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3852160020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:10:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:37 smithi022 ceph-mon[31360]: pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:37 smithi022 ceph-mon[40250]: pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:37 smithi122 ceph-mon[32255]: pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:39 smithi022 ceph-mon[31360]: pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:39 smithi022 ceph-mon[40250]: pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:39 smithi122 ceph-mon[32255]: pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:40] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:10:41.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:41.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:41.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:41.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:41 smithi022 ceph-mon[31360]: pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/805141829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:41 smithi022 ceph-mon[40250]: pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/805141829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:41 smithi122 ceph-mon[32255]: pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/805141829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:10:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:10:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:43 smithi122 ceph-mon[32255]: pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:43 smithi022 ceph-mon[31360]: pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:43 smithi022 ceph-mon[40250]: pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:44.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:10:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:10:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:45 smithi122 ceph-mon[32255]: pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:45 smithi022 ceph-mon[31360]: pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:45 smithi022 ceph-mon[40250]: pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:46.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:46.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:46.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:10:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:47 smithi022 ceph-mon[31360]: pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1772376652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:47 smithi022 ceph-mon[40250]: pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1772376652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:47 smithi122 ceph-mon[32255]: pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1772376652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:49 smithi122 ceph-mon[32255]: pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:49 smithi022 ceph-mon[31360]: pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:49 smithi022 ceph-mon[40250]: pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:50] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:10:51.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:51.833 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:51 smithi022 ceph-mon[40250]: pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:51.834 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:51 smithi022 ceph-mon[31360]: pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:51 smithi122 ceph-mon[32255]: pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:52.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:52.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:52.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3842147989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:10:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:10:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3842147989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3842147989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:53 smithi122 ceph-mon[32255]: pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:53 smithi022 ceph-mon[31360]: pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:53 smithi022 ceph-mon[40250]: pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:54.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:10:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:10:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:55 smithi122 ceph-mon[32255]: pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:55 smithi022 ceph-mon[31360]: pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:10:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:55 smithi022 ceph-mon[40250]: pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:10:57.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:10:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:10:57.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:10:57.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:10:57.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:10:57.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:57 smithi022 ceph-mon[31360]: pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:57.906 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:57 smithi022 ceph-mon[40250]: pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:57.908 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:57 smithi122 ceph-mon[32255]: pgmap v375: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:10:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2488302641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:10:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:10:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:10:58.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:10:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2488302641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:10:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:10:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:10:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:10:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2488302641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:10:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:10:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:10:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:10:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:10:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:10:59 smithi122 ceph-mon[32255]: pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:10:59 smithi022 ceph-mon[31360]: pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:10:59 smithi022 ceph-mon[40250]: pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:00] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:11:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:01 smithi122 ceph-mon[32255]: pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:01 smithi022 ceph-mon[31360]: pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:01 smithi022 ceph-mon[40250]: pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:02.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:11:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:11:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:02.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:02.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:03 smithi122 ceph-mon[32255]: pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1779728360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:03 smithi022 ceph-mon[31360]: pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1779728360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:03 smithi022 ceph-mon[40250]: pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1779728360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:04.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:11:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:11:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:05 smithi122 ceph-mon[32255]: pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:05 smithi022 ceph-mon[31360]: pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:05 smithi022 ceph-mon[40250]: pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:07 smithi022 ceph-mon[31360]: pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:11:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:07 smithi022 ceph-mon[40250]: pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:07 smithi122 ceph-mon[32255]: pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:07.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:08.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:08.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:08.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:08.572 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3574908814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3574908814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3574908814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:09 smithi122 ceph-mon[32255]: pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:09 smithi022 ceph-mon[31360]: pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:09 smithi022 ceph-mon[40250]: pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:10] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:11:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:11 smithi122 ceph-mon[32255]: pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:11 smithi022 ceph-mon[31360]: pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:11 smithi022 ceph-mon[40250]: pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:11:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:11:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:13.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:13.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:13.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:13 smithi122 ceph-mon[32255]: pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:13 smithi022 ceph-mon[31360]: pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:13 smithi022 ceph-mon[40250]: pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:14.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:11:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:11:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2439526231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2439526231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2439526231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:15 smithi122 ceph-mon[32255]: pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:15 smithi022 ceph-mon[31360]: pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:15 smithi022 ceph-mon[40250]: pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:17 smithi022 ceph-mon[31360]: pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:11:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:17 smithi022 ceph-mon[40250]: pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:17 smithi122 ceph-mon[32255]: pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:18.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:19.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:19.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:19.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:19 smithi122 ceph-mon[32255]: pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2556714115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:19 smithi022 ceph-mon[31360]: pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2556714115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:19 smithi022 ceph-mon[40250]: pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2556714115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:20] "GET /metrics HTTP/1.1" 200 202149 "" "Prometheus/2.18.1" 2021-12-15T18:11:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:21 smithi122 ceph-mon[32255]: pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:21 smithi022 ceph-mon[31360]: pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:21 smithi022 ceph-mon[40250]: pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:11:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:11:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:23 smithi122 ceph-mon[32255]: pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:23 smithi022 ceph-mon[31360]: pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:23 smithi022 ceph-mon[40250]: pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:24.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:24.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:24.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:24.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:11:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:11:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2642222825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2642222825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2642222825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:25 smithi122 ceph-mon[32255]: pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:25 smithi022 ceph-mon[31360]: pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:25 smithi022 ceph-mon[40250]: pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:27 smithi022 ceph-mon[31360]: pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:11:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:27 smithi022 ceph-mon[40250]: pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:27 smithi122 ceph-mon[32255]: pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:29.426 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:29 smithi022 ceph-mon[31360]: pgmap v391: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:29 smithi022 ceph-mon[40250]: pgmap v391: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:29.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:29.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:29.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:29 smithi122 ceph-mon[32255]: pgmap v391: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:30] "GET /metrics HTTP/1.1" 200 202149 "" "Prometheus/2.18.1" 2021-12-15T18:11:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1866819860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1866819860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1866819860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:31 smithi122 ceph-mon[32255]: pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:32.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:31 smithi022 ceph-mon[31360]: pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:32.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:31 smithi022 ceph-mon[40250]: pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:32.622 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:11:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:11:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:11:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:11:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:11:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:11:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:11:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:11:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:11:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:11:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:11:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:11:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:11:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:11:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:33 smithi122 ceph-mon[32255]: pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:33 smithi022 ceph-mon[31360]: pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:33 smithi022 ceph-mon[40250]: pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:34.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:11:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:11:34.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:35.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:35.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:35.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:35 smithi122 ceph-mon[32255]: pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/665969426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:35 smithi022 ceph-mon[31360]: pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/665969426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:35 smithi022 ceph-mon[40250]: pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/665969426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:11:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:37 smithi022 ceph-mon[31360]: pgmap v395: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:37 smithi022 ceph-mon[40250]: pgmap v395: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:37 smithi122 ceph-mon[32255]: pgmap v395: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:39 smithi122 ceph-mon[32255]: pgmap v396: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:39 smithi022 ceph-mon[31360]: pgmap v396: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:39 smithi022 ceph-mon[40250]: pgmap v396: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:40.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:40] "GET /metrics HTTP/1.1" 200 202150 "" "Prometheus/2.18.1" 2021-12-15T18:11:40.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:40.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:40.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:41 smithi022 ceph-mon[31360]: pgmap v397: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/852461151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:41 smithi022 ceph-mon[40250]: pgmap v397: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/852461151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:41 smithi122 ceph-mon[32255]: pgmap v397: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/852461151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:11:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:11:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:43 smithi022 ceph-mon[31360]: pgmap v398: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:43 smithi022 ceph-mon[40250]: pgmap v398: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:43 smithi122 ceph-mon[32255]: pgmap v398: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:44.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:11:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:11:45.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:45 smithi022 ceph-mon[31360]: pgmap v399: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:45 smithi022 ceph-mon[40250]: pgmap v399: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:45.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:45.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:45.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:45 smithi122 ceph-mon[32255]: pgmap v399: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/578843414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/578843414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/578843414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:47 smithi022 ceph-mon[31360]: pgmap v400: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:11:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:47 smithi022 ceph-mon[40250]: pgmap v400: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:47 smithi122 ceph-mon[32255]: pgmap v400: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:49 smithi022 ceph-mon[31360]: pgmap v401: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:49 smithi022 ceph-mon[40250]: pgmap v401: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:49 smithi122 ceph-mon[32255]: pgmap v401: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:50] "GET /metrics HTTP/1.1" 200 202151 "" "Prometheus/2.18.1" 2021-12-15T18:11:50.905 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:51.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:51.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:51.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:51 smithi022 ceph-mon[31360]: pgmap v402: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3744255435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:51 smithi022 ceph-mon[40250]: pgmap v402: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3744255435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:51 smithi122 ceph-mon[32255]: pgmap v402: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3744255435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:11:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:11:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:53 smithi022 ceph-mon[31360]: pgmap v403: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:53 smithi022 ceph-mon[40250]: pgmap v403: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:53 smithi122 ceph-mon[32255]: pgmap v403: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:54.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:11:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:11:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:55 smithi022 ceph-mon[40250]: pgmap v404: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:55 smithi022 ceph-mon[31360]: pgmap v404: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:11:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:55 smithi122 ceph-mon[32255]: pgmap v404: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:56.274 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:11:56.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:11:56.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:11:56.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:11:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:11:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:57 smithi022 ceph-mon[31360]: pgmap v405: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1755858558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:57 smithi022 ceph-mon[40250]: pgmap v405: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1755858558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:57 smithi122 ceph-mon[32255]: pgmap v405: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:11:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1755858558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:11:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:11:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:11:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:11:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:11:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:11:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:11:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:11:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:11:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:11:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:59 smithi022 ceph-mon[31360]: pgmap v406: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:11:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:11:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:59 smithi022 ceph-mon[40250]: pgmap v406: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:11:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:11:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:59 smithi122 ceph-mon[32255]: pgmap v406: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:11:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:11:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:12:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:00] "GET /metrics HTTP/1.1" 200 202151 "" "Prometheus/2.18.1" 2021-12-15T18:12:01.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:01 smithi022 ceph-mon[31360]: pgmap v407: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:01 smithi022 ceph-mon[40250]: pgmap v407: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:01 smithi122 ceph-mon[32255]: pgmap v407: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:02.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:02.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:02.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2610562800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:12:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:12:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2610562800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2610562800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:03 smithi022 ceph-mon[31360]: pgmap v408: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:03 smithi022 ceph-mon[40250]: pgmap v408: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:03 smithi122 ceph-mon[32255]: pgmap v408: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:04.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:12:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:12:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:05 smithi022 ceph-mon[31360]: pgmap v409: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:05 smithi022 ceph-mon[40250]: pgmap v409: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:05 smithi122 ceph-mon[32255]: pgmap v409: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:07.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:07.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:07.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:07.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:12:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:07 smithi022 ceph-mon[31360]: pgmap v410: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3756058663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:07 smithi022 ceph-mon[40250]: pgmap v410: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3756058663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:07 smithi122 ceph-mon[32255]: pgmap v410: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3756058663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:09 smithi022 ceph-mon[31360]: pgmap v411: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:09 smithi022 ceph-mon[40250]: pgmap v411: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:09 smithi122 ceph-mon[32255]: pgmap v411: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:10] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:12:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:11 smithi122 ceph-mon[32255]: pgmap v412: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:11 smithi022 ceph-mon[31360]: pgmap v412: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:11 smithi022 ceph-mon[40250]: pgmap v412: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:12.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:12:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:12:12.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:12.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:13 smithi122 ceph-mon[32255]: pgmap v413: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1157977216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:13 smithi022 ceph-mon[31360]: pgmap v413: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1157977216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:13 smithi022 ceph-mon[40250]: pgmap v413: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1157977216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:14.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:12:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:12:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:15 smithi122 ceph-mon[32255]: pgmap v414: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:15 smithi022 ceph-mon[31360]: pgmap v414: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:15 smithi022 ceph-mon[40250]: pgmap v414: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:17.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:17.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:17 smithi022 ceph-mon[31360]: pgmap v415: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:17.753 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:12:17.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:17 smithi022 ceph-mon[40250]: pgmap v415: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:17 smithi122 ceph-mon[32255]: pgmap v415: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:18.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:18.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:18.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3035248261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3035248261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3035248261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:19 smithi122 ceph-mon[32255]: pgmap v416: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:19 smithi022 ceph-mon[31360]: pgmap v416: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:19 smithi022 ceph-mon[40250]: pgmap v416: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:20] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:12:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:21 smithi122 ceph-mon[32255]: pgmap v417: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:21 smithi022 ceph-mon[31360]: pgmap v417: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:21 smithi022 ceph-mon[40250]: pgmap v417: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:12:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:12:23.123 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:23.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:23.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:23.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:23 smithi022 ceph-mon[31360]: pgmap v418: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3730268103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:23 smithi022 ceph-mon[40250]: pgmap v418: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3730268103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:23 smithi122 ceph-mon[32255]: pgmap v418: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3730268103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:24.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:12:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:12:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:25 smithi122 ceph-mon[32255]: pgmap v419: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:25 smithi022 ceph-mon[31360]: pgmap v419: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:25 smithi022 ceph-mon[40250]: pgmap v419: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:27 smithi022 ceph-mon[31360]: pgmap v420: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:12:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:27 smithi022 ceph-mon[40250]: pgmap v420: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:27 smithi122 ceph-mon[32255]: pgmap v420: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:28.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:28.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:28.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:28.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:29 smithi122 ceph-mon[32255]: pgmap v421: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1540963968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:29 smithi022 ceph-mon[31360]: pgmap v421: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1540963968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:29 smithi022 ceph-mon[40250]: pgmap v421: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1540963968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:30] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:12:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:31 smithi122 ceph-mon[32255]: pgmap v422: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:31 smithi022 ceph-mon[31360]: pgmap v422: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:31 smithi022 ceph-mon[40250]: pgmap v422: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:32.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:12:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:12:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:12:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:12:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:12:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:12:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:12:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:12:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:12:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:12:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:12:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:12:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:12:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:12:33.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:33.865 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:33 smithi022 ceph-mon[31360]: pgmap v423: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:33.865 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:33 smithi022 ceph-mon[40250]: pgmap v423: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:33 smithi122 ceph-mon[32255]: pgmap v423: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:34.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:34.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:34.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:12:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:12:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3537853588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3537853588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3537853588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:35 smithi122 ceph-mon[32255]: pgmap v424: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:35 smithi022 ceph-mon[31360]: pgmap v424: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:35 smithi022 ceph-mon[40250]: pgmap v424: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:37 smithi022 ceph-mon[31360]: pgmap v425: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:12:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:37 smithi022 ceph-mon[40250]: pgmap v425: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:37 smithi122 ceph-mon[32255]: pgmap v425: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:39.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:39.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:39.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:39.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:39 smithi122 ceph-mon[32255]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:39 smithi022 ceph-mon[31360]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:39 smithi022 ceph-mon[40250]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:40] "GET /metrics HTTP/1.1" 200 202149 "" "Prometheus/2.18.1" 2021-12-15T18:12:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/840706118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/840706118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/840706118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:41 smithi122 ceph-mon[32255]: pgmap v427: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:41 smithi022 ceph-mon[31360]: pgmap v427: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:41 smithi022 ceph-mon[40250]: pgmap v427: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:12:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:12:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:43 smithi122 ceph-mon[32255]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:43 smithi022 ceph-mon[31360]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:43 smithi022 ceph-mon[40250]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:44.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:44.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:12:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:12:44.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:44.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:44.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:45 smithi122 ceph-mon[32255]: pgmap v429: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3133966043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:45 smithi022 ceph-mon[31360]: pgmap v429: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3133966043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:45 smithi022 ceph-mon[40250]: pgmap v429: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:12:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3133966043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:47 smithi022 ceph-mon[31360]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:12:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:47 smithi022 ceph-mon[40250]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:47 smithi122 ceph-mon[32255]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:12:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:49 smithi122 ceph-mon[32255]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:12:49.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:49.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:49 smithi022 ceph-mon[31360]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:12:49.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:49 smithi022 ceph-mon[40250]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:12:50.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:50.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:50.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:50] "GET /metrics HTTP/1.1" 200 202154 "" "Prometheus/2.18.1" 2021-12-15T18:12:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2456905097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2456905097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2456905097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:51 smithi122 ceph-mon[32255]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:51 smithi022 ceph-mon[31360]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:51 smithi022 ceph-mon[40250]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:12:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:12:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:53 smithi122 ceph-mon[32255]: pgmap v433: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:53 smithi022 ceph-mon[31360]: pgmap v433: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:53 smithi022 ceph-mon[40250]: pgmap v433: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:54.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:12:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:12:55.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:12:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:12:55.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:12:55.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:12:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:55 smithi122 ceph-mon[32255]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:12:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:55 smithi022 ceph-mon[31360]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:12:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:55 smithi022 ceph-mon[40250]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:12:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:12:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/763265666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/763265666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/763265666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:12:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:57 smithi022 ceph-mon[31360]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:12:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:57 smithi022 ceph-mon[40250]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:57 smithi122 ceph-mon[32255]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T18:12:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:59 smithi122 ceph-mon[32255]: pgmap v436: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:12:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:12:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:12:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:12:59.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:12:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:13:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[31360]: pgmap v436: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:13:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:13:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:13:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:13:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:13:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[40250]: pgmap v436: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T18:13:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:13:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:13:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:13:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:12:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:13:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:00] "GET /metrics HTTP/1.1" 200 202154 "" "Prometheus/2.18.1" 2021-12-15T18:13:00.725 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:01.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:01.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:01.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:01 smithi122 ceph-mon[32255]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/186349627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:01 smithi022 ceph-mon[31360]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/186349627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:01 smithi022 ceph-mon[40250]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/186349627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:13:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:13:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:03 smithi122 ceph-mon[32255]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:03 smithi022 ceph-mon[31360]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:03 smithi022 ceph-mon[40250]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:04.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:13:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:13:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:05 smithi122 ceph-mon[32255]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:05 smithi022 ceph-mon[31360]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:05 smithi022 ceph-mon[40250]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:06.097 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:06.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:06.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:06.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/942209088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/942209088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/942209088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:07 smithi022 ceph-mon[31360]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:13:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:07 smithi022 ceph-mon[40250]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:07 smithi122 ceph-mon[32255]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:09 smithi122 ceph-mon[32255]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:09 smithi022 ceph-mon[31360]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:09 smithi022 ceph-mon[40250]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:10] "GET /metrics HTTP/1.1" 200 202152 "" "Prometheus/2.18.1" 2021-12-15T18:13:11.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:11 smithi022 ceph-mon[31360]: pgmap v442: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:11 smithi022 ceph-mon[40250]: pgmap v442: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:11.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:11.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:11.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:11 smithi122 ceph-mon[32255]: pgmap v442: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:12.640 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:13:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:13:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2834486198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2834486198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2834486198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:13 smithi122 ceph-mon[32255]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:13 smithi022 ceph-mon[31360]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:13 smithi022 ceph-mon[40250]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:14.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:13:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:13:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:15 smithi122 ceph-mon[32255]: pgmap v444: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:15 smithi022 ceph-mon[31360]: pgmap v444: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:15 smithi022 ceph-mon[40250]: pgmap v444: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:16.844 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:17.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:17.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:17.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:13:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:17 smithi122 ceph-mon[32255]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/128633007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:17 smithi022 ceph-mon[31360]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/128633007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:17 smithi022 ceph-mon[40250]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/128633007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:19 smithi122 ceph-mon[32255]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:19 smithi022 ceph-mon[31360]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:19 smithi022 ceph-mon[40250]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:20] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:13:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:21 smithi122 ceph-mon[32255]: pgmap v447: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:21 smithi022 ceph-mon[31360]: pgmap v447: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:21 smithi022 ceph-mon[40250]: pgmap v447: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:22.213 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:22.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:22.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:22.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:22.661 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:13:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:13:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/487080859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/487080859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/487080859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:23 smithi122 ceph-mon[32255]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:23 smithi022 ceph-mon[31360]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:23 smithi022 ceph-mon[40250]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:24.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:13:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:13:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:25 smithi122 ceph-mon[32255]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:25 smithi022 ceph-mon[31360]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:26.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:25 smithi022 ceph-mon[40250]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:26.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:27.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:27.663 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:13:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:27 smithi122 ceph-mon[32255]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:27.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:27.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:27.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:27.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:27 smithi022 ceph-mon[31360]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:27.951 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:27 smithi022 ceph-mon[40250]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3210192582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3210192582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3210192582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:29 smithi122 ceph-mon[32255]: pgmap v451: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:29 smithi022 ceph-mon[40250]: pgmap v451: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:29 smithi022 ceph-mon[31360]: pgmap v451: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:30] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:13:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:31 smithi122 ceph-mon[32255]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:31 smithi022 ceph-mon[31360]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:31 smithi022 ceph-mon[40250]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:32.678 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:13:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:13:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:13:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:13:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:13:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:13:32.951 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:32.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:13:32.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:13:32.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:13:32.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:13:32.953 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:13:32.953 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:13:32.953 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:13:32.953 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:13:33.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:33.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:33.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:33 smithi122 ceph-mon[32255]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1565830466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:33 smithi022 ceph-mon[31360]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1565830466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:33 smithi022 ceph-mon[40250]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1565830466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:34.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:13:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:13:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:35 smithi122 ceph-mon[32255]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:35 smithi022 ceph-mon[31360]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:35 smithi022 ceph-mon[40250]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:37.684 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:13:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:37 smithi122 ceph-mon[32255]: pgmap v455: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:37 smithi022 ceph-mon[31360]: pgmap v455: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:37 smithi022 ceph-mon[40250]: pgmap v455: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:38.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:38.690 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3439462304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:38.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:38.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:38.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3439462304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3439462304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:39 smithi022 ceph-mon[31360]: pgmap v456: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:39 smithi022 ceph-mon[40250]: pgmap v456: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:39 smithi122 ceph-mon[32255]: pgmap v456: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:40] "GET /metrics HTTP/1.1" 200 202149 "" "Prometheus/2.18.1" 2021-12-15T18:13:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:41 smithi022 ceph-mon[31360]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:41 smithi022 ceph-mon[40250]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:41 smithi122 ceph-mon[32255]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:13:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:13:43.696 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:43 smithi022 ceph-mon[31360]: pgmap v458: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:43 smithi022 ceph-mon[40250]: pgmap v458: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:44.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:44.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:43 smithi122 ceph-mon[32255]: pgmap v458: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:44.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:13:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:13:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2831836369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:44 smithi022 ceph-mon[31360]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2831836369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:44 smithi022 ceph-mon[40250]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2831836369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:44 smithi122 ceph-mon[32255]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:47 smithi022 ceph-mon[31360]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:13:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:47 smithi022 ceph-mon[40250]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:47 smithi122 ceph-mon[32255]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:49.067 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:49.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:49.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:49.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:49 smithi022 ceph-mon[31360]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/913139123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:49 smithi022 ceph-mon[40250]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/913139123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:49 smithi122 ceph-mon[32255]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/913139123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:50] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:13:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:51 smithi022 ceph-mon[31360]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:51 smithi022 ceph-mon[40250]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:51 smithi122 ceph-mon[32255]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:13:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:13:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:53 smithi022 ceph-mon[31360]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:53 smithi022 ceph-mon[40250]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:53 smithi122 ceph-mon[32255]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:54.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:13:54.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:13:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:13:54.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:13:54.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:13:54.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:13:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:55 smithi022 ceph-mon[31360]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1116828705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:55 smithi022 ceph-mon[40250]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1116828705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:55 smithi122 ceph-mon[32255]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:13:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1116828705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:13:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:57 smithi022 ceph-mon[31360]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:13:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:57 smithi022 ceph-mon[40250]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:57 smithi122 ceph-mon[32255]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:13:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:59 smithi122 ceph-mon[32255]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:13:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:13:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:13:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:13:59.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[31360]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:59.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:13:59.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:13:59.707 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:13:59.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[40250]: pgmap v466: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:13:59.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:13:59.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:13:59.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:13:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:13:59.806 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:00.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:00.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:00.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:00] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:14:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:14:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3671294889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:00 smithi022 ceph-mon[31360]: pgmap v467: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:14:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3671294889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:00 smithi022 ceph-mon[40250]: pgmap v467: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:14:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3671294889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:00 smithi122 ceph-mon[32255]: pgmap v467: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:14:02.319Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:14:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:03 smithi122 ceph-mon[32255]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:03 smithi022 ceph-mon[31360]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:03 smithi022 ceph-mon[40250]: pgmap v468: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:04.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:14:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:14:05.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:05 smithi022 ceph-mon[31360]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:05 smithi022 ceph-mon[40250]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:05.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:05.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:05.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:05 smithi122 ceph-mon[32255]: pgmap v469: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2906999507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2906999507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2906999507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:07 smithi022 ceph-mon[31360]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:14:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:07 smithi022 ceph-mon[40250]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:07 smithi122 ceph-mon[32255]: pgmap v470: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:09 smithi022 ceph-mon[31360]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:09 smithi022 ceph-mon[40250]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:09 smithi122 ceph-mon[32255]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:10] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:14:10.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:10.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:10.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:10.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:11 smithi022 ceph-mon[31360]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4164739945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:11 smithi022 ceph-mon[40250]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4164739945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:11 smithi122 ceph-mon[32255]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4164739945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:14:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:14:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:13 smithi022 ceph-mon[31360]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:13 smithi022 ceph-mon[40250]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:13 smithi122 ceph-mon[32255]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:14.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:14:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:14:15.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:15.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:15 smithi022 ceph-mon[31360]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:15.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:15.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:15 smithi022 ceph-mon[40250]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:15.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:15 smithi122 ceph-mon[32255]: pgmap v474: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:16.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:16.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:16.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1374841875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1374841875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1374841875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:17 smithi022 ceph-mon[31360]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:14:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:17 smithi022 ceph-mon[40250]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:17 smithi122 ceph-mon[32255]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:19 smithi122 ceph-mon[32255]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:19 smithi022 ceph-mon[31360]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:19 smithi022 ceph-mon[40250]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:20] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:14:21.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:21.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:21.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:21.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:21 smithi122 ceph-mon[32255]: pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:21 smithi022 ceph-mon[31360]: pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:21 smithi022 ceph-mon[40250]: pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1046647991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:14:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:14:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1046647991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1046647991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:23 smithi122 ceph-mon[32255]: pgmap v478: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:23 smithi022 ceph-mon[31360]: pgmap v478: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:23 smithi022 ceph-mon[40250]: pgmap v478: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:24.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:14:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:14:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:25 smithi122 ceph-mon[32255]: pgmap v479: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:25 smithi022 ceph-mon[31360]: pgmap v479: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:25 smithi022 ceph-mon[40250]: pgmap v479: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:26.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:27.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:14:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:27 smithi022 ceph-mon[31360]: pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4156331462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:27 smithi022 ceph-mon[40250]: pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4156331462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:27 smithi122 ceph-mon[32255]: pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4156331462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:29 smithi122 ceph-mon[32255]: pgmap v481: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:29 smithi022 ceph-mon[31360]: pgmap v481: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:29 smithi022 ceph-mon[40250]: pgmap v481: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:30] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:14:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:31 smithi122 ceph-mon[32255]: pgmap v482: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:32.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:32.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:31 smithi022 ceph-mon[31360]: pgmap v482: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:32.019 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:31 smithi022 ceph-mon[40250]: pgmap v482: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:32.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:32.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:32.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:32.568 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:14:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:14:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:14:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:14:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:14:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:14:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3880854639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:14:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:14:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3880854639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:14:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:14:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:14:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:14:32.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3880854639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:33 smithi122 ceph-mon[32255]: pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:33 smithi022 ceph-mon[31360]: pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:33 smithi022 ceph-mon[40250]: pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:34.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:14:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:14:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:35 smithi122 ceph-mon[32255]: pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:35 smithi022 ceph-mon[31360]: pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:35 smithi022 ceph-mon[40250]: pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:37.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:37.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:37.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:37.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:37.762 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:14:37.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:37 smithi022 ceph-mon[31360]: pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:37.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:37 smithi022 ceph-mon[40250]: pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:37 smithi122 ceph-mon[32255]: pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4081139614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4081139614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4081139614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:39 smithi122 ceph-mon[32255]: pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:39 smithi022 ceph-mon[31360]: pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:39 smithi022 ceph-mon[40250]: pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:40] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:14:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:41 smithi122 ceph-mon[32255]: pgmap v487: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:41 smithi022 ceph-mon[31360]: pgmap v487: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:41 smithi022 ceph-mon[40250]: pgmap v487: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:14:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:14:42.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:43.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:43.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:43.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:43 smithi122 ceph-mon[32255]: pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1120140436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:43 smithi022 ceph-mon[40250]: pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1120140436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:43 smithi022 ceph-mon[31360]: pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1120140436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:44.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:14:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:14:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:45 smithi122 ceph-mon[32255]: pgmap v489: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:45 smithi022 ceph-mon[31360]: pgmap v489: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:45 smithi022 ceph-mon[40250]: pgmap v489: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:47 smithi022 ceph-mon[31360]: pgmap v490: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:14:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:47 smithi022 ceph-mon[40250]: pgmap v490: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:47 smithi122 ceph-mon[32255]: pgmap v490: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:48.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1475277108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1475277108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1475277108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:49 smithi122 ceph-mon[32255]: pgmap v491: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:49 smithi022 ceph-mon[31360]: pgmap v491: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:49 smithi022 ceph-mon[40250]: pgmap v491: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:50] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:14:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:51 smithi122 ceph-mon[32255]: pgmap v492: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:51 smithi022 ceph-mon[31360]: pgmap v492: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:51 smithi022 ceph-mon[40250]: pgmap v492: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:14:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:14:53.502 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:53 smithi022 ceph-mon[31360]: pgmap v493: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:53 smithi022 ceph-mon[40250]: pgmap v493: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:53.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:53.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:53.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:53 smithi122 ceph-mon[32255]: pgmap v493: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:54.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:14:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:14:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1624701938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1624701938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1624701938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:14:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:55 smithi122 ceph-mon[32255]: pgmap v494: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:55 smithi022 ceph-mon[31360]: pgmap v494: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:55 smithi022 ceph-mon[40250]: pgmap v494: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:14:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:57 smithi022 ceph-mon[31360]: pgmap v495: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:14:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:57 smithi022 ceph-mon[40250]: pgmap v495: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:57 smithi122 ceph-mon[32255]: pgmap v495: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:14:58.874 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:14:59.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:14:59.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:14:59.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:14:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:59 smithi122 ceph-mon[32255]: pgmap v496: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:14:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:14:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3096239169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:59 smithi022 ceph-mon[31360]: pgmap v496: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:14:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3096239169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:59 smithi022 ceph-mon[40250]: pgmap v496: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:14:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3096239169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:00] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:15:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:15:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:15:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:15:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:15:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:15:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:15:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:01 smithi122 ceph-mon[32255]: pgmap v497: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:01 smithi022 ceph-mon[31360]: pgmap v497: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:01 smithi022 ceph-mon[40250]: pgmap v497: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:15:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:15:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:03 smithi122 ceph-mon[32255]: pgmap v498: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:03 smithi022 ceph-mon[31360]: pgmap v498: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:03 smithi022 ceph-mon[40250]: pgmap v498: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:04.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:04.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:04.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:04.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:04.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:15:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:15:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2240143040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2240143040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2240143040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:05 smithi122 ceph-mon[32255]: pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:05 smithi022 ceph-mon[31360]: pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:05 smithi022 ceph-mon[40250]: pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:07 smithi022 ceph-mon[31360]: pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:15:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:07 smithi022 ceph-mon[40250]: pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:07 smithi122 ceph-mon[32255]: pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:15:09.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:09 smithi122 ceph-mon[32255]: pgmap v501: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:09.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:09.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:09.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:09.980 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:09 smithi022 ceph-mon[31360]: pgmap v501: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:09.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:09 smithi022 ceph-mon[40250]: pgmap v501: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:10] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:15:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/210661815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/210661815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/210661815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:11 smithi122 ceph-mon[32255]: pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:11 smithi022 ceph-mon[31360]: pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:11 smithi022 ceph-mon[40250]: pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:15:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:15:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:13 smithi122 ceph-mon[32255]: pgmap v503: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:13 smithi022 ceph-mon[31360]: pgmap v503: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:13 smithi022 ceph-mon[40250]: pgmap v503: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:14.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:15:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:15:14.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:15.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:15.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:15.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:15 smithi122 ceph-mon[32255]: pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2138884742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:15 smithi022 ceph-mon[31360]: pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2138884742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:15 smithi022 ceph-mon[40250]: pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2138884742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:17.643 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:15:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:17 smithi122 ceph-mon[32255]: pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:17 smithi022 ceph-mon[31360]: pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:17 smithi022 ceph-mon[40250]: pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:19 smithi122 ceph-mon[32255]: pgmap v506: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:19 smithi022 ceph-mon[31360]: pgmap v506: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:19 smithi022 ceph-mon[40250]: pgmap v506: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:20.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:20] "GET /metrics HTTP/1.1" 200 202150 "" "Prometheus/2.18.1" 2021-12-15T18:15:20.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:20.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:20.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:21 smithi122 ceph-mon[32255]: pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1363135587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:21 smithi022 ceph-mon[31360]: pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1363135587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:21 smithi022 ceph-mon[40250]: pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1363135587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:15:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:15:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:23 smithi122 ceph-mon[32255]: pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:23 smithi022 ceph-mon[31360]: pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:23 smithi022 ceph-mon[40250]: pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:24.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:15:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:15:25.722 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:25 smithi122 ceph-mon[32255]: pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:25 smithi022 ceph-mon[31360]: pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:25 smithi022 ceph-mon[40250]: pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:26.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:26.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:26.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2325285819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2325285819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2325285819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:27.661 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:15:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:27 smithi122 ceph-mon[32255]: pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:27 smithi022 ceph-mon[31360]: pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:27 smithi022 ceph-mon[40250]: pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:29 smithi122 ceph-mon[32255]: pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:29 smithi022 ceph-mon[31360]: pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:29 smithi022 ceph-mon[40250]: pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:30] "GET /metrics HTTP/1.1" 200 202150 "" "Prometheus/2.18.1" 2021-12-15T18:15:31.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:31 smithi022 ceph-mon[31360]: pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1601826734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:31 smithi022 ceph-mon[40250]: pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1601826734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:31 smithi122 ceph-mon[32255]: pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1601826734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:32.674 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:15:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:15:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:15:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:15:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:15:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:15:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:15:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:15:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:15:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:15:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:15:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:15:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:15:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:15:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:33 smithi122 ceph-mon[32255]: pgmap v513: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:33 smithi022 ceph-mon[31360]: pgmap v513: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:33 smithi022 ceph-mon[40250]: pgmap v513: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:15:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:15:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:35 smithi122 ceph-mon[32255]: pgmap v514: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:35 smithi022 ceph-mon[31360]: pgmap v514: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:35 smithi022 ceph-mon[40250]: pgmap v514: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:36.462 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:37.680 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:15:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:37 smithi122 ceph-mon[32255]: pgmap v515: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/409333409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:37 smithi022 ceph-mon[31360]: pgmap v515: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/409333409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:37 smithi022 ceph-mon[40250]: pgmap v515: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/409333409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:39 smithi122 ceph-mon[32255]: pgmap v516: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:39 smithi022 ceph-mon[31360]: pgmap v516: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:39 smithi022 ceph-mon[40250]: pgmap v516: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:40] "GET /metrics HTTP/1.1" 200 202150 "" "Prometheus/2.18.1" 2021-12-15T18:15:41.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:41 smithi022 ceph-mon[31360]: pgmap v517: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:41 smithi022 ceph-mon[40250]: pgmap v517: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:41 smithi122 ceph-mon[32255]: pgmap v517: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:42.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:42.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:42.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:15:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:15:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1534686725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1534686725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1534686725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:43 smithi022 ceph-mon[31360]: pgmap v518: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:43 smithi022 ceph-mon[40250]: pgmap v518: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:43 smithi122 ceph-mon[32255]: pgmap v518: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:44.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:15:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:15:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:45 smithi022 ceph-mon[31360]: pgmap v519: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:45 smithi022 ceph-mon[40250]: pgmap v519: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:45 smithi122 ceph-mon[32255]: pgmap v519: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:47.205 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:15:47.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:47.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:47.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:47 smithi022 ceph-mon[31360]: pgmap v520: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2620164914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:47 smithi022 ceph-mon[40250]: pgmap v520: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2620164914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:47 smithi122 ceph-mon[32255]: pgmap v520: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2620164914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:49 smithi022 ceph-mon[31360]: pgmap v521: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:49 smithi022 ceph-mon[40250]: pgmap v521: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:49 smithi122 ceph-mon[32255]: pgmap v521: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:50] "GET /metrics HTTP/1.1" 200 202149 "" "Prometheus/2.18.1" 2021-12-15T18:15:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:51 smithi022 ceph-mon[31360]: pgmap v522: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:51 smithi022 ceph-mon[40250]: pgmap v522: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:51 smithi122 ceph-mon[32255]: pgmap v522: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:15:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:15:52.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:52.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:52.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:53 smithi022 ceph-mon[31360]: pgmap v523: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3579911192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:53 smithi022 ceph-mon[40250]: pgmap v523: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3579911192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:53 smithi122 ceph-mon[32255]: pgmap v523: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3579911192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:15:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:15:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:55 smithi022 ceph-mon[31360]: pgmap v524: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:55 smithi022 ceph-mon[40250]: pgmap v524: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:55 smithi122 ceph-mon[32255]: pgmap v524: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:15:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:15:57.709 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:15:57.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:15:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:57 smithi022 ceph-mon[31360]: pgmap v525: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:57 smithi022 ceph-mon[40250]: pgmap v525: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:57 smithi122 ceph-mon[32255]: pgmap v525: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:15:58.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:15:58.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:15:58.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:15:58.714 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2444069421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2444069421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:15:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2444069421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:15:59 smithi022 ceph-mon[31360]: pgmap v526: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:15:59 smithi022 ceph-mon[40250]: pgmap v526: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:15:59 smithi122 ceph-mon[32255]: pgmap v526: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:00] "GET /metrics HTTP/1.1" 200 202149 "" "Prometheus/2.18.1" 2021-12-15T18:16:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:01 smithi122 ceph-mon[32255]: pgmap v527: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:16:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:16:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:16:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:16:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[40250]: pgmap v527: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:16:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:16:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:16:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:16:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[31360]: pgmap v527: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:16:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:16:02.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:16:02.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:16:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:16:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:16:03.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:03.685 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:03 smithi022 ceph-mon[31360]: pgmap v528: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:03.686 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:03 smithi022 ceph-mon[40250]: pgmap v528: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:03.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:03.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:03.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:03 smithi122 ceph-mon[32255]: pgmap v528: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:04.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:16:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:16:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3512957880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:05.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3512957880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:05.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3512957880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:05 smithi122 ceph-mon[32255]: pgmap v529: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:05 smithi022 ceph-mon[31360]: pgmap v529: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:05 smithi022 ceph-mon[40250]: pgmap v529: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:07 smithi022 ceph-mon[31360]: pgmap v530: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:16:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:07 smithi022 ceph-mon[40250]: pgmap v530: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:07 smithi122 ceph-mon[32255]: pgmap v530: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:08.688 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:09.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:09.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:09.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:09 smithi122 ceph-mon[32255]: pgmap v531: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/932298649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:09 smithi022 ceph-mon[31360]: pgmap v531: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/932298649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:09 smithi022 ceph-mon[40250]: pgmap v531: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/932298649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:10] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:16:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:11 smithi122 ceph-mon[32255]: pgmap v532: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:11 smithi022 ceph-mon[31360]: pgmap v532: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:11 smithi022 ceph-mon[40250]: pgmap v532: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:16:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:16:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:13 smithi122 ceph-mon[32255]: pgmap v533: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:13 smithi022 ceph-mon[31360]: pgmap v533: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:13 smithi022 ceph-mon[40250]: pgmap v533: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:14.064 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:14.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2846637523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2846637523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:14.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:16:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:16:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2846637523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:15 smithi122 ceph-mon[32255]: pgmap v534: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:15 smithi022 ceph-mon[31360]: pgmap v534: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:15 smithi022 ceph-mon[40250]: pgmap v534: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:17 smithi022 ceph-mon[31360]: pgmap v535: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:16:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:17 smithi022 ceph-mon[40250]: pgmap v535: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:17 smithi122 ceph-mon[32255]: pgmap v535: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:19.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:19 smithi022 ceph-mon[31360]: pgmap v536: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:19 smithi022 ceph-mon[40250]: pgmap v536: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:19.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:19.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:19.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:19 smithi122 ceph-mon[32255]: pgmap v536: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:20] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:16:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1856304009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1856304009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1856304009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:21 smithi122 ceph-mon[32255]: pgmap v537: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:21 smithi022 ceph-mon[31360]: pgmap v537: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:21 smithi022 ceph-mon[40250]: pgmap v537: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:16:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:16:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:23 smithi122 ceph-mon[32255]: pgmap v538: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:23 smithi022 ceph-mon[31360]: pgmap v538: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:23 smithi022 ceph-mon[40250]: pgmap v538: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:24.802 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:24.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:16:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:16:25.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:25.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:25.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:25 smithi122 ceph-mon[32255]: pgmap v539: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4691260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:25 smithi022 ceph-mon[31360]: pgmap v539: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4691260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:25 smithi022 ceph-mon[40250]: pgmap v539: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4691260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:27 smithi022 ceph-mon[31360]: pgmap v540: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:16:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:27 smithi022 ceph-mon[40250]: pgmap v540: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:27 smithi122 ceph-mon[32255]: pgmap v540: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:29 smithi122 ceph-mon[32255]: pgmap v541: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:29 smithi022 ceph-mon[31360]: pgmap v541: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:29 smithi022 ceph-mon[40250]: pgmap v541: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:30.171 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:30] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:16:30.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:30.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:30.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/891137272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/891137272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/891137272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:31 smithi122 ceph-mon[32255]: pgmap v542: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:31 smithi022 ceph-mon[31360]: pgmap v542: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:31 smithi022 ceph-mon[40250]: pgmap v542: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:32.594 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:16:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:16:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:16:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:16:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:16:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:16:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:16:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:16:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:16:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:16:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:16:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:16:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:16:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:16:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:33 smithi122 ceph-mon[32255]: pgmap v543: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:33 smithi022 ceph-mon[31360]: pgmap v543: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:33 smithi022 ceph-mon[40250]: pgmap v543: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:16:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:16:35.541 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:35.913 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:35 smithi022 ceph-mon[31360]: pgmap v544: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:35.913 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:35.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:35 smithi022 ceph-mon[40250]: pgmap v544: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:35.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:35 smithi122 ceph-mon[32255]: pgmap v544: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4205666703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4205666703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4205666703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:37 smithi022 ceph-mon[31360]: pgmap v545: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:16:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:37 smithi022 ceph-mon[40250]: pgmap v545: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:37.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:37 smithi122 ceph-mon[32255]: pgmap v545: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:39 smithi122 ceph-mon[32255]: pgmap v546: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:39 smithi022 ceph-mon[31360]: pgmap v546: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:39 smithi022 ceph-mon[40250]: pgmap v546: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:40] "GET /metrics HTTP/1.1" 200 202149 "" "Prometheus/2.18.1" 2021-12-15T18:16:40.914 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:41.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:41.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:41.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:41 smithi122 ceph-mon[32255]: pgmap v547: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3234587745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:41 smithi022 ceph-mon[31360]: pgmap v547: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3234587745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:41 smithi022 ceph-mon[40250]: pgmap v547: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3234587745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:16:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:16:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:43 smithi122 ceph-mon[32255]: pgmap v548: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:43 smithi022 ceph-mon[31360]: pgmap v548: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:43 smithi022 ceph-mon[40250]: pgmap v548: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:44.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:16:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:16:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:45 smithi122 ceph-mon[32255]: pgmap v549: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:45 smithi022 ceph-mon[31360]: pgmap v549: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:45 smithi022 ceph-mon[40250]: pgmap v549: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:46.285 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:46.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:46.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:46.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:16:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:47 smithi022 ceph-mon[31360]: pgmap v550: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2539114639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:47 smithi022 ceph-mon[40250]: pgmap v550: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2539114639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:47 smithi122 ceph-mon[32255]: pgmap v550: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2539114639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:49 smithi122 ceph-mon[32255]: pgmap v551: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:49 smithi022 ceph-mon[31360]: pgmap v551: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:49 smithi022 ceph-mon[40250]: pgmap v551: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:50] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:16:51.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:51 smithi122 ceph-mon[32255]: pgmap v552: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:52.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:52.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:52.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:52.023 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:51 smithi022 ceph-mon[31360]: pgmap v552: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:52.023 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:51 smithi022 ceph-mon[40250]: pgmap v552: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:52.632 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:16:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:16:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1848472826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1848472826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1848472826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:53 smithi122 ceph-mon[32255]: pgmap v553: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:53 smithi022 ceph-mon[31360]: pgmap v553: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:53 smithi022 ceph-mon[40250]: pgmap v553: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:16:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:16:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:55 smithi122 ceph-mon[32255]: pgmap v554: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:55 smithi022 ceph-mon[31360]: pgmap v554: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:55 smithi022 ceph-mon[40250]: pgmap v554: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:16:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:16:57.023 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:16:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:16:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:16:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:16:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:57 smithi022 ceph-mon[31360]: pgmap v555: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/276264075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:16:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:57 smithi022 ceph-mon[40250]: pgmap v555: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/276264075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:57 smithi122 ceph-mon[32255]: pgmap v555: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:16:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/276264075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:16:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:16:59 smithi122 ceph-mon[32255]: pgmap v556: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:16:59 smithi022 ceph-mon[31360]: pgmap v556: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:16:59 smithi022 ceph-mon[40250]: pgmap v556: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:00] "GET /metrics HTTP/1.1" 200 202144 "" "Prometheus/2.18.1" 2021-12-15T18:17:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[31360]: pgmap v557: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:17:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:17:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:17:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:17:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[40250]: pgmap v557: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:17:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:17:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:17:01.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:17:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:01 smithi122 ceph-mon[32255]: pgmap v557: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:17:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:17:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:17:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:17:02.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:17:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:17:02.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:02.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:02.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:03 smithi122 ceph-mon[32255]: pgmap v558: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/899368489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:03 smithi022 ceph-mon[31360]: pgmap v558: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/899368489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:03 smithi022 ceph-mon[40250]: pgmap v558: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:04.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/899368489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:04.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:17:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:17:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:05 smithi122 ceph-mon[32255]: pgmap v559: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:05 smithi022 ceph-mon[31360]: pgmap v559: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:05 smithi022 ceph-mon[40250]: pgmap v559: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:07.650 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:17:07.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:07.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:07 smithi122 ceph-mon[32255]: pgmap v560: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:07 smithi022 ceph-mon[31360]: pgmap v560: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:07 smithi022 ceph-mon[40250]: pgmap v560: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:08.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:08.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:08.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1635274088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1635274088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1635274088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:09 smithi122 ceph-mon[32255]: pgmap v561: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:09 smithi022 ceph-mon[31360]: pgmap v561: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:09 smithi022 ceph-mon[40250]: pgmap v561: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:10] "GET /metrics HTTP/1.1" 200 202147 "" "Prometheus/2.18.1" 2021-12-15T18:17:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:11 smithi122 ceph-mon[32255]: pgmap v562: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:11 smithi022 ceph-mon[31360]: pgmap v562: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:11 smithi022 ceph-mon[40250]: pgmap v562: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:17:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:17:13.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:13.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:13.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:13.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:13 smithi022 ceph-mon[31360]: pgmap v563: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3514893575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:13 smithi022 ceph-mon[40250]: pgmap v563: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3514893575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:13 smithi122 ceph-mon[32255]: pgmap v563: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3514893575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:14.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:17:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:17:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:15 smithi122 ceph-mon[32255]: pgmap v564: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:15 smithi022 ceph-mon[31360]: pgmap v564: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:15 smithi022 ceph-mon[40250]: pgmap v564: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:17.667 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:17:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:17 smithi122 ceph-mon[32255]: pgmap v565: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:17 smithi022 ceph-mon[31360]: pgmap v565: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:17 smithi022 ceph-mon[40250]: pgmap v565: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:18.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:18.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:18.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:18.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:19 smithi122 ceph-mon[32255]: pgmap v566: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1542343793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:19 smithi022 ceph-mon[31360]: pgmap v566: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1542343793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:19 smithi022 ceph-mon[40250]: pgmap v566: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1542343793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:20] "GET /metrics HTTP/1.1" 200 202150 "" "Prometheus/2.18.1" 2021-12-15T18:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:21 smithi122 ceph-mon[32255]: pgmap v567: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:21 smithi022 ceph-mon[31360]: pgmap v567: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:21 smithi022 ceph-mon[40250]: pgmap v567: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:17:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:17:23.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:23 smithi122 ceph-mon[32255]: pgmap v568: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:23 smithi022 ceph-mon[31360]: pgmap v568: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:23 smithi022 ceph-mon[40250]: pgmap v568: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:24.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:24.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:24.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4075357946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:24.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:17:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:17:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4075357946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4075357946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:25 smithi122 ceph-mon[32255]: pgmap v569: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:25 smithi022 ceph-mon[31360]: pgmap v569: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:25 smithi022 ceph-mon[40250]: pgmap v569: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:27.684 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:17:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:27 smithi122 ceph-mon[32255]: pgmap v570: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:27 smithi022 ceph-mon[31360]: pgmap v570: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:27 smithi022 ceph-mon[40250]: pgmap v570: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:29.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:29 smithi022 ceph-mon[31360]: pgmap v571: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:30.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3590654594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:30.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:29 smithi022 ceph-mon[40250]: pgmap v571: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:30.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3590654594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:29 smithi122 ceph-mon[32255]: pgmap v571: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3590654594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:30] "GET /metrics HTTP/1.1" 200 202150 "" "Prometheus/2.18.1" 2021-12-15T18:17:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:31 smithi022 ceph-mon[31360]: pgmap v572: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:31 smithi022 ceph-mon[40250]: pgmap v572: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:31 smithi122 ceph-mon[32255]: pgmap v572: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:17:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:17:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:17:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:17:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:17:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:17:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:17:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:17:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:17:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:17:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:17:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:17:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:17:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:17:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:33 smithi022 ceph-mon[31360]: pgmap v573: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:33 smithi022 ceph-mon[40250]: pgmap v573: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:33 smithi122 ceph-mon[32255]: pgmap v573: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:34.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:17:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:17:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:34.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:35 smithi022 ceph-mon[31360]: pgmap v574: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2560171417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:35 smithi022 ceph-mon[40250]: pgmap v574: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2560171417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:35 smithi122 ceph-mon[32255]: pgmap v574: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2560171417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:37.707 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:17:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:37 smithi022 ceph-mon[40250]: pgmap v575: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:38.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:37 smithi022 ceph-mon[31360]: pgmap v575: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:37 smithi122 ceph-mon[32255]: pgmap v575: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:39.989 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:39.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:39 smithi022 ceph-mon[31360]: pgmap v576: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:39.989 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:39 smithi022 ceph-mon[40250]: pgmap v576: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:39 smithi122 ceph-mon[32255]: pgmap v576: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:40.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:40.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:40.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:40] "GET /metrics HTTP/1.1" 200 202146 "" "Prometheus/2.18.1" 2021-12-15T18:17:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2789250360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2789250360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2789250360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:41 smithi022 ceph-mon[31360]: pgmap v577: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:41 smithi022 ceph-mon[40250]: pgmap v577: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:41 smithi122 ceph-mon[32255]: pgmap v577: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:17:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:17:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:43 smithi022 ceph-mon[31360]: pgmap v578: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:43 smithi022 ceph-mon[40250]: pgmap v578: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:43 smithi122 ceph-mon[32255]: pgmap v578: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:44.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:17:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:17:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:45.724 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:45 smithi022 ceph-mon[40250]: pgmap v579: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:45.724 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:45.724 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2195956302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:45.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:45.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:45.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:45 smithi022 ceph-mon[31360]: pgmap v579: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2195956302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:45 smithi122 ceph-mon[32255]: pgmap v579: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2195956302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:47.726 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:17:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:47 smithi022 ceph-mon[31360]: pgmap v580: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:47 smithi022 ceph-mon[40250]: pgmap v580: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:47 smithi122 ceph-mon[32255]: pgmap v580: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:49 smithi022 ceph-mon[40250]: pgmap v581: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:49 smithi022 ceph-mon[31360]: pgmap v581: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:49 smithi122 ceph-mon[32255]: pgmap v581: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:17:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:50] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:17:50.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:51.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:51.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:51.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:51 smithi022 ceph-mon[31360]: pgmap v582: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/516446582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:51 smithi022 ceph-mon[40250]: pgmap v582: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/516446582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:51 smithi122 ceph-mon[32255]: pgmap v582: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/516446582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:17:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:17:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:53 smithi022 ceph-mon[31360]: pgmap v583: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:53 smithi022 ceph-mon[40250]: pgmap v583: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:53 smithi122 ceph-mon[32255]: pgmap v583: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:17:54.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:17:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:17:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:55 smithi022 ceph-mon[31360]: pgmap v584: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T18:17:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:55 smithi022 ceph-mon[40250]: pgmap v584: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T18:17:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:56.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:17:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:55 smithi122 ceph-mon[32255]: pgmap v584: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T18:17:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:17:56.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:17:56.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:17:56.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:17:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/687500747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/687500747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/687500747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:17:57.749 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:17:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:57 smithi022 ceph-mon[31360]: pgmap v585: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T18:17:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:57 smithi022 ceph-mon[40250]: pgmap v585: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T18:17:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:57 smithi122 ceph-mon[32255]: pgmap v585: 132 pgs: 132 active+clean; 582 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T18:17:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:17:58 smithi022 ceph-mon[31360]: pgmap v586: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T18:17:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:17:58 smithi022 ceph-mon[40250]: pgmap v586: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T18:17:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:17:58 smithi122 ceph-mon[32255]: pgmap v586: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T18:18:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:00] "GET /metrics HTTP/1.1" 200 202148 "" "Prometheus/2.18.1" 2021-12-15T18:18:01.478 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[31360]: pgmap v587: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T18:18:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:18:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:18:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:18:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[40250]: pgmap v587: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T18:18:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:18:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:18:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:18:01.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:01.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:01.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:01 smithi122 ceph-mon[32255]: pgmap v587: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T18:18:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:18:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:18:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:18:02.563 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:18:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:18:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3003932016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:18:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3003932016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:18:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3003932016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:18:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:03 smithi122 ceph-mon[32255]: pgmap v588: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T18:18:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:03 smithi022 ceph-mon[31360]: pgmap v588: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T18:18:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:03 smithi022 ceph-mon[40250]: pgmap v588: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T18:18:04.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:18:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:18:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:05 smithi122 ceph-mon[32255]: pgmap v589: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T18:18:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:05 smithi022 ceph-mon[31360]: pgmap v589: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T18:18:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:05 smithi022 ceph-mon[40250]: pgmap v589: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T18:18:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:06.873 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:07.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:07.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:07.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:18:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:07 smithi122 ceph-mon[32255]: pgmap v590: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:18:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2596419822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:07 smithi022 ceph-mon[31360]: pgmap v590: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:18:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2596419822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:07 smithi022 ceph-mon[40250]: pgmap v590: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:18:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2596419822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:09 smithi122 ceph-mon[32255]: pgmap v591: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:18:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:09 smithi022 ceph-mon[31360]: pgmap v591: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:18:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:09 smithi022 ceph-mon[40250]: pgmap v591: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:18:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:10] "GET /metrics HTTP/1.1" 200 202186 "" "Prometheus/2.18.1" 2021-12-15T18:18:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:11 smithi122 ceph-mon[32255]: pgmap v592: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:11 smithi022 ceph-mon[31360]: pgmap v592: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:11 smithi022 ceph-mon[40250]: pgmap v592: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:12.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:12.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:12.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:12.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:18:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:18:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:13 smithi122 ceph-mon[32255]: pgmap v593: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/714472502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:13 smithi022 ceph-mon[31360]: pgmap v593: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/714472502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:13 smithi022 ceph-mon[40250]: pgmap v593: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/714472502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:14.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:18:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:18:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:15 smithi122 ceph-mon[32255]: pgmap v594: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:15 smithi022 ceph-mon[31360]: pgmap v594: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:15 smithi022 ceph-mon[40250]: pgmap v594: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:17.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:17 smithi022 ceph-mon[31360]: pgmap v595: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:18:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:17 smithi022 ceph-mon[40250]: pgmap v595: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:17 smithi122 ceph-mon[32255]: pgmap v595: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:17.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:17.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:17.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4241114064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4241114064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4241114064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:19 smithi122 ceph-mon[32255]: pgmap v596: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:19 smithi022 ceph-mon[31360]: pgmap v596: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:19 smithi022 ceph-mon[40250]: pgmap v596: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:20] "GET /metrics HTTP/1.1" 200 202188 "" "Prometheus/2.18.1" 2021-12-15T18:18:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:21 smithi122 ceph-mon[32255]: pgmap v597: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:21 smithi022 ceph-mon[31360]: pgmap v597: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:21 smithi022 ceph-mon[40250]: pgmap v597: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:18:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:18:22.989 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:23.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:23.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:23.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:23 smithi022 ceph-mon[31360]: pgmap v598: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1544430684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:23 smithi022 ceph-mon[40250]: pgmap v598: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1544430684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:23 smithi122 ceph-mon[32255]: pgmap v598: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1544430684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:24.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:18:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:18:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:25 smithi122 ceph-mon[32255]: pgmap v599: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:25 smithi022 ceph-mon[31360]: pgmap v599: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:25 smithi022 ceph-mon[40250]: pgmap v599: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:27 smithi022 ceph-mon[31360]: pgmap v600: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:18:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:27 smithi022 ceph-mon[40250]: pgmap v600: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:27 smithi122 ceph-mon[32255]: pgmap v600: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:28.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:28.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:28.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:28.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:29 smithi122 ceph-mon[32255]: pgmap v601: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2138948147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:29 smithi022 ceph-mon[31360]: pgmap v601: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2138948147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:29 smithi022 ceph-mon[40250]: pgmap v601: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2138948147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:30] "GET /metrics HTTP/1.1" 200 202188 "" "Prometheus/2.18.1" 2021-12-15T18:18:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:31 smithi122 ceph-mon[32255]: pgmap v602: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:31 smithi022 ceph-mon[31360]: pgmap v602: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:31 smithi022 ceph-mon[40250]: pgmap v602: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:32.613 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:18:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:18:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:18:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:18:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:18:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:18:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:18:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:18:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:18:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:18:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:18:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:18:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:18:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:18:33.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:33 smithi122 ceph-mon[32255]: pgmap v603: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:33 smithi022 ceph-mon[31360]: pgmap v603: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:33 smithi022 ceph-mon[40250]: pgmap v603: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:34.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:34.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:34.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/449837560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:34.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:18:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:18:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/449837560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/449837560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:35 smithi122 ceph-mon[32255]: pgmap v604: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:35 smithi022 ceph-mon[31360]: pgmap v604: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:35 smithi022 ceph-mon[40250]: pgmap v604: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:37 smithi022 ceph-mon[31360]: pgmap v605: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:18:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:37 smithi022 ceph-mon[40250]: pgmap v605: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:37 smithi122 ceph-mon[32255]: pgmap v605: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:39.105 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:39.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:39.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:39.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:39 smithi022 ceph-mon[31360]: pgmap v606: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2097025268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:39 smithi022 ceph-mon[40250]: pgmap v606: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2097025268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:39 smithi122 ceph-mon[32255]: pgmap v606: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2097025268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:40] "GET /metrics HTTP/1.1" 200 202185 "" "Prometheus/2.18.1" 2021-12-15T18:18:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:41 smithi122 ceph-mon[32255]: pgmap v607: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:41 smithi022 ceph-mon[31360]: pgmap v607: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:41 smithi022 ceph-mon[40250]: pgmap v607: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:18:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:18:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:43 smithi122 ceph-mon[32255]: pgmap v608: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:43 smithi022 ceph-mon[31360]: pgmap v608: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:43 smithi022 ceph-mon[40250]: pgmap v608: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:44.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:44.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:44.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:44.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:44.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:18:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:18:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:45 smithi122 ceph-mon[32255]: pgmap v609: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2911351087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:45 smithi022 ceph-mon[31360]: pgmap v609: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2911351087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:45 smithi022 ceph-mon[40250]: pgmap v609: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2911351087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:47 smithi022 ceph-mon[31360]: pgmap v610: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:18:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:47 smithi022 ceph-mon[40250]: pgmap v610: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:47 smithi122 ceph-mon[32255]: pgmap v610: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:49 smithi122 ceph-mon[32255]: pgmap v611: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:49 smithi022 ceph-mon[31360]: pgmap v611: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:49 smithi022 ceph-mon[40250]: pgmap v611: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:50.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:50.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:50.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:50] "GET /metrics HTTP/1.1" 200 202187 "" "Prometheus/2.18.1" 2021-12-15T18:18:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3624528936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3624528936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3624528936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:51 smithi122 ceph-mon[32255]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:51 smithi022 ceph-mon[31360]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:51 smithi022 ceph-mon[40250]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:18:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:18:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:53 smithi122 ceph-mon[32255]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:53 smithi022 ceph-mon[31360]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:53 smithi022 ceph-mon[40250]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:18:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:18:55.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:18:55.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:18:55.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:18:55.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:18:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:55 smithi122 ceph-mon[32255]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3866763389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:55 smithi022 ceph-mon[31360]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3866763389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:55 smithi022 ceph-mon[40250]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:18:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:18:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3866763389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:18:57.650 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:18:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:57 smithi122 ceph-mon[32255]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:57 smithi022 ceph-mon[31360]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:57 smithi022 ceph-mon[40250]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:18:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:18:59 smithi122 ceph-mon[32255]: pgmap v616: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:18:59 smithi022 ceph-mon[31360]: pgmap v616: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:18:59 smithi022 ceph-mon[40250]: pgmap v616: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:00] "GET /metrics HTTP/1.1" 200 202187 "" "Prometheus/2.18.1" 2021-12-15T18:19:00.587 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:00.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:00.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:00.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:01 smithi122 ceph-mon[32255]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1295752094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:01.995 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:01 smithi022 ceph-mon[31360]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:01.996 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1295752094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:01.996 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:01 smithi022 ceph-mon[40250]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:01.996 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1295752094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:02.663 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:19:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:19:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:19:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:19:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:19:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:19:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:19:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:19:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:19:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:19:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:19:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:03 smithi122 ceph-mon[32255]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:03 smithi022 ceph-mon[31360]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:03 smithi022 ceph-mon[40250]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:04.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:19:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:19:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:05 smithi122 ceph-mon[32255]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:05.950 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:05 smithi022 ceph-mon[40250]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:05.950 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:05.951 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:05.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:05 smithi022 ceph-mon[31360]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:05.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:06.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:06.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:06.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:06.914 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:06.914 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2466763002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:06.914 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2466763002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2466763002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:07.727 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:19:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:07 smithi022 ceph-mon[31360]: pgmap v620: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:07 smithi022 ceph-mon[40250]: pgmap v620: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:07 smithi122 ceph-mon[32255]: pgmap v620: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:19:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:09 smithi022 ceph-mon[31360]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:09 smithi022 ceph-mon[40250]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:09 smithi122 ceph-mon[32255]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:10] "GET /metrics HTTP/1.1" 200 202185 "" "Prometheus/2.18.1" 2021-12-15T18:19:11.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:11.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:11.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:11.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:11 smithi022 ceph-mon[31360]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/586298384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:11 smithi022 ceph-mon[40250]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/586298384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:11 smithi122 ceph-mon[32255]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/586298384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:19:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:19:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:13 smithi022 ceph-mon[31360]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:13 smithi022 ceph-mon[40250]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:13 smithi122 ceph-mon[32255]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:14.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:19:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:19:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:15 smithi022 ceph-mon[31360]: pgmap v624: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:15 smithi022 ceph-mon[40250]: pgmap v624: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:15 smithi122 ceph-mon[32255]: pgmap v624: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:16.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:17.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:17.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:17.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:17.739 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:19:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:17 smithi022 ceph-mon[31360]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4176805820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:17 smithi022 ceph-mon[40250]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4176805820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:17 smithi122 ceph-mon[32255]: pgmap v625: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4176805820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:19 smithi022 ceph-mon[31360]: pgmap v626: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:19 smithi022 ceph-mon[40250]: pgmap v626: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:19 smithi122 ceph-mon[32255]: pgmap v626: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:20] "GET /metrics HTTP/1.1" 200 202182 "" "Prometheus/2.18.1" 2021-12-15T18:19:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:21 smithi022 ceph-mon[31360]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:21 smithi022 ceph-mon[40250]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:22.063 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:21 smithi122 ceph-mon[32255]: pgmap v627: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:22.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:22.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:22.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:19:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:19:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3855950742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3855950742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3855950742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:23 smithi022 ceph-mon[31360]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:23 smithi022 ceph-mon[40250]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:23 smithi122 ceph-mon[32255]: pgmap v628: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:24.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:19:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:19:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:25 smithi022 ceph-mon[31360]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:25 smithi022 ceph-mon[40250]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:25 smithi122 ceph-mon[32255]: pgmap v629: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:26 smithi122 ceph-mon[32255]: pgmap v630: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:26 smithi022 ceph-mon[31360]: pgmap v630: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:26 smithi022 ceph-mon[40250]: pgmap v630: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:27.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:19:27.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:27.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:27.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/848722711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/848722711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/848722711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:28 smithi122 ceph-mon[32255]: pgmap v631: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:28 smithi022 ceph-mon[31360]: pgmap v631: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:28 smithi022 ceph-mon[40250]: pgmap v631: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:30] "GET /metrics HTTP/1.1" 200 202182 "" "Prometheus/2.18.1" 2021-12-15T18:19:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:31 smithi122 ceph-mon[32255]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:31 smithi022 ceph-mon[31360]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:31 smithi022 ceph-mon[40250]: pgmap v632: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:32.587 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:19:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:19:32.808 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:19:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:19:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:19:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:19:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:19:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:19:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:19:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:19:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:19:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:19:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:19:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:19:33.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:33.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:33.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:33 smithi122 ceph-mon[32255]: pgmap v633: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3951227002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:33 smithi022 ceph-mon[31360]: pgmap v633: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3951227002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:33 smithi022 ceph-mon[40250]: pgmap v633: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3951227002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:19:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:19:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:35 smithi122 ceph-mon[32255]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:35 smithi022 ceph-mon[31360]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:35 smithi022 ceph-mon[40250]: pgmap v634: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:37 smithi022 ceph-mon[31360]: pgmap v635: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:19:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:37 smithi022 ceph-mon[40250]: pgmap v635: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:37 smithi122 ceph-mon[32255]: pgmap v635: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:38.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:38.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:38.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:38.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2065677945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2065677945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2065677945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:39 smithi122 ceph-mon[32255]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:39 smithi022 ceph-mon[31360]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:39 smithi022 ceph-mon[40250]: pgmap v636: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:40] "GET /metrics HTTP/1.1" 200 202185 "" "Prometheus/2.18.1" 2021-12-15T18:19:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:41 smithi122 ceph-mon[32255]: pgmap v637: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:41 smithi022 ceph-mon[31360]: pgmap v637: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:41 smithi022 ceph-mon[40250]: pgmap v637: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:19:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:19:43.549 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:43.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:43.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:43.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:43.922 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:43 smithi022 ceph-mon[31360]: pgmap v638: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:43.923 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:43 smithi022 ceph-mon[40250]: pgmap v638: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:43 smithi122 ceph-mon[32255]: pgmap v638: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4218293862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:44.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:19:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:19:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4218293862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4218293862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:45 smithi122 ceph-mon[32255]: pgmap v639: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:45 smithi022 ceph-mon[31360]: pgmap v639: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:45 smithi022 ceph-mon[40250]: pgmap v639: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:47 smithi022 ceph-mon[31360]: pgmap v640: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:19:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:47 smithi022 ceph-mon[40250]: pgmap v640: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:47 smithi122 ceph-mon[32255]: pgmap v640: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:48.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:49.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:49.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:49.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:49 smithi122 ceph-mon[32255]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/467248031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:49 smithi022 ceph-mon[31360]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/467248031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:49 smithi022 ceph-mon[40250]: pgmap v641: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/467248031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:50] "GET /metrics HTTP/1.1" 200 202180 "" "Prometheus/2.18.1" 2021-12-15T18:19:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:51 smithi122 ceph-mon[32255]: pgmap v642: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:51 smithi022 ceph-mon[31360]: pgmap v642: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:51 smithi022 ceph-mon[40250]: pgmap v642: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:19:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:19:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:53 smithi122 ceph-mon[32255]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:53 smithi022 ceph-mon[31360]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:53 smithi022 ceph-mon[40250]: pgmap v643: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:54.294 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:54.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:19:54.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:19:54.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:19:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:19:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:19:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:55 smithi122 ceph-mon[32255]: pgmap v644: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/855605396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:55 smithi022 ceph-mon[31360]: pgmap v644: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/855605396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:55 smithi022 ceph-mon[40250]: pgmap v644: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:19:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:19:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/855605396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:19:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:57 smithi022 ceph-mon[31360]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:19:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:57 smithi022 ceph-mon[40250]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:57 smithi122 ceph-mon[32255]: pgmap v645: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:19:59.663 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:19:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:19:59 smithi122 ceph-mon[32255]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:19:59 smithi022 ceph-mon[31360]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:19:59 smithi022 ceph-mon[40250]: pgmap v646: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:00.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:00.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:00.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:00] "GET /metrics HTTP/1.1" 200 202180 "" "Prometheus/2.18.1" 2021-12-15T18:20:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T18:20:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3479310119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T18:20:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3479310119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T18:20:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3479310119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:01 smithi122 ceph-mon[32255]: pgmap v647: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:01 smithi022 ceph-mon[31360]: pgmap v647: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:01 smithi022 ceph-mon[40250]: pgmap v647: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:20:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:20:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:03 smithi122 ceph-mon[32255]: pgmap v648: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:03 smithi022 ceph-mon[31360]: pgmap v648: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:04.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:03 smithi022 ceph-mon[40250]: pgmap v648: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:04.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:20:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:20:05.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:05.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:05.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:05.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:05 smithi022 ceph-mon[31360]: pgmap v649: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2835632332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:05 smithi022 ceph-mon[40250]: pgmap v649: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2835632332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:05 smithi122 ceph-mon[32255]: pgmap v649: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2835632332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:20:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[31360]: pgmap v650: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:20:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:20:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:20:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:20:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[40250]: pgmap v650: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:20:07.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:20:07.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:20:07.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:20:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:07 smithi122 ceph-mon[32255]: pgmap v650: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:20:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:09 smithi122 ceph-mon[32255]: pgmap v651: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:09 smithi022 ceph-mon[31360]: pgmap v651: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:09 smithi022 ceph-mon[40250]: pgmap v651: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:10.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:10] "GET /metrics HTTP/1.1" 200 202186 "" "Prometheus/2.18.1" 2021-12-15T18:20:10.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:10.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:10.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:11 smithi122 ceph-mon[32255]: pgmap v652: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3123934361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:11 smithi022 ceph-mon[31360]: pgmap v652: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3123934361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:11 smithi022 ceph-mon[40250]: pgmap v652: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3123934361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:20:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:20:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:13 smithi122 ceph-mon[32255]: pgmap v653: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:13 smithi022 ceph-mon[31360]: pgmap v653: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:13 smithi022 ceph-mon[40250]: pgmap v653: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:14.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:20:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:20:15.785 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:15 smithi122 ceph-mon[32255]: pgmap v654: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:15 smithi022 ceph-mon[31360]: pgmap v654: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:15 smithi022 ceph-mon[40250]: pgmap v654: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:16.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:16.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:16.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/393113655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/393113655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/393113655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:17.654 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:20:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:17 smithi122 ceph-mon[32255]: pgmap v655: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:17 smithi022 ceph-mon[40250]: pgmap v655: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:17 smithi022 ceph-mon[31360]: pgmap v655: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:19 smithi122 ceph-mon[32255]: pgmap v656: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:19 smithi022 ceph-mon[40250]: pgmap v656: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:19 smithi022 ceph-mon[31360]: pgmap v656: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:20] "GET /metrics HTTP/1.1" 200 202183 "" "Prometheus/2.18.1" 2021-12-15T18:20:21.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:21.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:21.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:21.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:21 smithi022 ceph-mon[31360]: pgmap v657: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1690157924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:21 smithi022 ceph-mon[40250]: pgmap v657: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1690157924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:21 smithi122 ceph-mon[32255]: pgmap v657: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1690157924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:20:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:20:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:23 smithi122 ceph-mon[32255]: pgmap v658: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:23 smithi022 ceph-mon[31360]: pgmap v658: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:23 smithi022 ceph-mon[40250]: pgmap v658: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:24.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:20:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:20:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:25 smithi122 ceph-mon[32255]: pgmap v659: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:25 smithi022 ceph-mon[31360]: pgmap v659: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:25 smithi022 ceph-mon[40250]: pgmap v659: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:26.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:26.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:26.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:27.667 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:20:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:27 smithi122 ceph-mon[32255]: pgmap v660: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3929918786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:27 smithi022 ceph-mon[31360]: pgmap v660: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3929918786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:27 smithi022 ceph-mon[40250]: pgmap v660: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3929918786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:29 smithi122 ceph-mon[32255]: pgmap v661: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:29 smithi022 ceph-mon[31360]: pgmap v661: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:29 smithi022 ceph-mon[40250]: pgmap v661: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:30] "GET /metrics HTTP/1.1" 200 202183 "" "Prometheus/2.18.1" 2021-12-15T18:20:31.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:31 smithi122 ceph-mon[32255]: pgmap v662: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:31 smithi022 ceph-mon[31360]: pgmap v662: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:31 smithi022 ceph-mon[40250]: pgmap v662: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:32.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:32.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:32.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:32.678 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:20:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:20:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3500077398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:20:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:20:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:20:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:20:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3500077398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:20:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:20:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:20:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:20:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3500077398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:33 smithi122 ceph-mon[32255]: pgmap v663: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:33 smithi022 ceph-mon[40250]: pgmap v663: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:33 smithi022 ceph-mon[31360]: pgmap v663: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:20:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:20:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:35 smithi122 ceph-mon[32255]: pgmap v664: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:35 smithi022 ceph-mon[31360]: pgmap v664: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:35 smithi022 ceph-mon[40250]: pgmap v664: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:20:37.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:37.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:37.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:37 smithi122 ceph-mon[32255]: pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/347212974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:37 smithi022 ceph-mon[31360]: pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/347212974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:37 smithi022 ceph-mon[40250]: pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/347212974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:39 smithi022 ceph-mon[31360]: pgmap v666: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:39 smithi022 ceph-mon[40250]: pgmap v666: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:39 smithi122 ceph-mon[32255]: pgmap v666: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:40] "GET /metrics HTTP/1.1" 200 202175 "" "Prometheus/2.18.1" 2021-12-15T18:20:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:41 smithi022 ceph-mon[31360]: pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:41 smithi022 ceph-mon[40250]: pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:41 smithi122 ceph-mon[32255]: pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:42.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:20:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:20:43.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:43.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:43.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:43 smithi022 ceph-mon[31360]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/43916519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:43 smithi022 ceph-mon[40250]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/43916519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:43 smithi122 ceph-mon[32255]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/43916519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:20:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:20:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:45 smithi022 ceph-mon[31360]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:45 smithi022 ceph-mon[40250]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:45 smithi122 ceph-mon[32255]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:47.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:20:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:48.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:47 smithi022 ceph-mon[31360]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:48.012 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:47 smithi022 ceph-mon[40250]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:47 smithi122 ceph-mon[32255]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:48.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:48.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:48.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:48.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4203991309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:48.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4203991309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4203991309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:49 smithi022 ceph-mon[31360]: pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:49 smithi022 ceph-mon[40250]: pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:49 smithi122 ceph-mon[32255]: pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:50] "GET /metrics HTTP/1.1" 200 202183 "" "Prometheus/2.18.1" 2021-12-15T18:20:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:51 smithi022 ceph-mon[31360]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:51 smithi022 ceph-mon[40250]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:51 smithi122 ceph-mon[32255]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:20:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:20:53.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:53.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:53.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:53.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:20:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:53 smithi022 ceph-mon[31360]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:53 smithi022 ceph-mon[40250]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:53 smithi122 ceph-mon[32255]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1731244746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1731244746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1731244746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:20:55.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:20:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:20:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:55 smithi022 ceph-mon[31360]: pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:55 smithi022 ceph-mon[40250]: pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:55 smithi122 ceph-mon[32255]: pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:20:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:20:57.721 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:20:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:57 smithi022 ceph-mon[31360]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:57 smithi022 ceph-mon[40250]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:57 smithi122 ceph-mon[32255]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:20:58.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:20:59.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:20:59.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:20:59.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:59 smithi022 ceph-mon[31360]: pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:20:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2762829170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:59 smithi022 ceph-mon[40250]: pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:20:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2762829170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:59 smithi122 ceph-mon[32255]: pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:20:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2762829170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:00] "GET /metrics HTTP/1.1" 200 202183 "" "Prometheus/2.18.1" 2021-12-15T18:21:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:01 smithi022 ceph-mon[31360]: pgmap v677: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:01 smithi022 ceph-mon[40250]: pgmap v677: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:01 smithi122 ceph-mon[32255]: pgmap v677: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:21:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:21:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:03 smithi022 ceph-mon[31360]: pgmap v678: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:03 smithi022 ceph-mon[40250]: pgmap v678: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:04.130 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:03 smithi122 ceph-mon[32255]: pgmap v678: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:04.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:04.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:04.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/494837637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/494837637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:21:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:21:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/494837637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:05 smithi022 ceph-mon[31360]: pgmap v679: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:05 smithi022 ceph-mon[40250]: pgmap v679: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:05 smithi122 ceph-mon[32255]: pgmap v679: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:07.737 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:21:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[31360]: pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:21:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:21:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:21:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[40250]: pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:21:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:21:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:21:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:07 smithi122 ceph-mon[32255]: pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:21:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:21:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:21:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:21:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:08 smithi122 ceph-mon[32255]: pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:21:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:08 smithi022 ceph-mon[31360]: pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:21:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:08 smithi022 ceph-mon[40250]: pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:09.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:09.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2499890986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2499890986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2499890986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:10] "GET /metrics HTTP/1.1" 200 202184 "" "Prometheus/2.18.1" 2021-12-15T18:21:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:11 smithi122 ceph-mon[32255]: pgmap v682: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:11 smithi022 ceph-mon[31360]: pgmap v682: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:11 smithi022 ceph-mon[40250]: pgmap v682: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:21:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:21:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:13 smithi122 ceph-mon[32255]: pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:13 smithi022 ceph-mon[31360]: pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:13 smithi022 ceph-mon[40250]: pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:14.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:15.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:21:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:21:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:15.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:15.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:15 smithi122 ceph-mon[32255]: pgmap v684: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/601821501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:15 smithi022 ceph-mon[31360]: pgmap v684: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/601821501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:15 smithi022 ceph-mon[40250]: pgmap v684: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/601821501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:21:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:17 smithi022 ceph-mon[40250]: pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:17 smithi022 ceph-mon[31360]: pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:17 smithi122 ceph-mon[32255]: pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:19 smithi122 ceph-mon[32255]: pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:19 smithi022 ceph-mon[31360]: pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:19 smithi022 ceph-mon[40250]: pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:20.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:20] "GET /metrics HTTP/1.1" 200 202189 "" "Prometheus/2.18.1" 2021-12-15T18:21:20.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:20.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:20.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1821963157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1821963157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1821963157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:21 smithi122 ceph-mon[32255]: pgmap v687: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:21 smithi022 ceph-mon[31360]: pgmap v687: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:21 smithi022 ceph-mon[40250]: pgmap v687: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:21:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:21:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:23 smithi122 ceph-mon[32255]: pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:23 smithi022 ceph-mon[31360]: pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:23 smithi022 ceph-mon[40250]: pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:21:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:21:25.609 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:25 smithi122 ceph-mon[32255]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:25.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:25.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:25.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:25.978 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:25 smithi022 ceph-mon[40250]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:25.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:25.979 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:25 smithi022 ceph-mon[31360]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:25.979 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/93965723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/93965723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/93965723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:27 smithi022 ceph-mon[31360]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:21:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:27 smithi022 ceph-mon[40250]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:27 smithi122 ceph-mon[32255]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:29 smithi122 ceph-mon[32255]: pgmap v691: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:29 smithi022 ceph-mon[31360]: pgmap v691: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:29 smithi022 ceph-mon[40250]: pgmap v691: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:30] "GET /metrics HTTP/1.1" 200 202189 "" "Prometheus/2.18.1" 2021-12-15T18:21:30.979 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:31.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:31.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:31.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:31 smithi122 ceph-mon[32255]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/635572230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:31 smithi022 ceph-mon[31360]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/635572230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:31 smithi022 ceph-mon[40250]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/635572230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:32.644 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:21:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:21:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:21:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:21:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:21:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:21:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:21:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:21:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:21:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:21:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:21:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:21:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:21:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:21:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:33 smithi122 ceph-mon[32255]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:33 smithi022 ceph-mon[31360]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:33 smithi022 ceph-mon[40250]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:35.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:21:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:21:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:35 smithi122 ceph-mon[32255]: pgmap v694: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:35 smithi022 ceph-mon[31360]: pgmap v694: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:35 smithi022 ceph-mon[40250]: pgmap v694: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:36.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:37.650 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:21:37.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:37 smithi122 ceph-mon[32255]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:37.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2343990539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:37 smithi022 ceph-mon[31360]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2343990539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:37 smithi022 ceph-mon[40250]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2343990539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:39 smithi122 ceph-mon[32255]: pgmap v696: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:39 smithi022 ceph-mon[31360]: pgmap v696: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:39 smithi022 ceph-mon[40250]: pgmap v696: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:40] "GET /metrics HTTP/1.1" 200 202185 "" "Prometheus/2.18.1" 2021-12-15T18:21:41.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:41 smithi122 ceph-mon[32255]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:41 smithi022 ceph-mon[31360]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:41 smithi022 ceph-mon[40250]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:42.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:42.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:42.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:42.661 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:21:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:21:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2205881547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2205881547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2205881547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:43 smithi122 ceph-mon[32255]: pgmap v698: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:43 smithi022 ceph-mon[31360]: pgmap v698: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:43 smithi022 ceph-mon[40250]: pgmap v698: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:21:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:21:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:45 smithi122 ceph-mon[32255]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:45 smithi022 ceph-mon[31360]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:45 smithi022 ceph-mon[40250]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:47.089 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:47.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:47.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:47.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:47.667 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:21:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:47 smithi122 ceph-mon[32255]: pgmap v700: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3502322113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:47 smithi022 ceph-mon[31360]: pgmap v700: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3502322113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:47 smithi022 ceph-mon[40250]: pgmap v700: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3502322113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:49 smithi122 ceph-mon[32255]: pgmap v701: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:50.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:49 smithi022 ceph-mon[31360]: pgmap v701: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:50.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:49 smithi022 ceph-mon[40250]: pgmap v701: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:50] "GET /metrics HTTP/1.1" 200 202188 "" "Prometheus/2.18.1" 2021-12-15T18:21:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:51 smithi122 ceph-mon[32255]: pgmap v702: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:51 smithi022 ceph-mon[31360]: pgmap v702: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:51 smithi022 ceph-mon[40250]: pgmap v702: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:52.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:21:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:21:52.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:52.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:52.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:53 smithi122 ceph-mon[32255]: pgmap v703: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3870527895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:53 smithi022 ceph-mon[31360]: pgmap v703: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3870527895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:53 smithi022 ceph-mon[40250]: pgmap v703: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3870527895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:21:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:21:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:55 smithi122 ceph-mon[32255]: pgmap v704: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:55 smithi022 ceph-mon[31360]: pgmap v704: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:55 smithi022 ceph-mon[40250]: pgmap v704: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:21:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:21:57.680 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:21:57.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:21:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:57 smithi122 ceph-mon[32255]: pgmap v705: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:57 smithi022 ceph-mon[31360]: pgmap v705: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:57 smithi022 ceph-mon[40250]: pgmap v705: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:21:58.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:21:58.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:21:58.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:21:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221012869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221012869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:21:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221012869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:21:59 smithi022 ceph-mon[31360]: pgmap v706: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:21:59 smithi022 ceph-mon[40250]: pgmap v706: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:21:59 smithi122 ceph-mon[32255]: pgmap v706: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:00] "GET /metrics HTTP/1.1" 200 202188 "" "Prometheus/2.18.1" 2021-12-15T18:22:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:01 smithi022 ceph-mon[31360]: pgmap v707: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:01 smithi022 ceph-mon[40250]: pgmap v707: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:01 smithi122 ceph-mon[32255]: pgmap v707: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:22:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:22:03.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:03.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:03.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:03.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:03 smithi022 ceph-mon[31360]: pgmap v708: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/805868681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:03 smithi022 ceph-mon[40250]: pgmap v708: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/805868681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:03 smithi122 ceph-mon[32255]: pgmap v708: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/805868681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:22:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:22:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:05 smithi022 ceph-mon[31360]: pgmap v709: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:05 smithi022 ceph-mon[40250]: pgmap v709: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:05 smithi122 ceph-mon[32255]: pgmap v709: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:07.696 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:22:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:07 smithi022 ceph-mon[31360]: pgmap v710: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:07 smithi022 ceph-mon[40250]: pgmap v710: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:07 smithi122 ceph-mon[32255]: pgmap v710: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:08.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:08.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:08.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:08.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:08.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:22:08.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:22:08.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:22:08.931 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:22:08.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:22:08.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:22:08.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:22:08.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:22:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:22:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:22:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:22:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:22:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:09 smithi022 ceph-mon[31360]: pgmap v711: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1627174140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:09 smithi022 ceph-mon[40250]: pgmap v711: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1627174140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:09 smithi122 ceph-mon[32255]: pgmap v711: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1627174140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:10] "GET /metrics HTTP/1.1" 200 202187 "" "Prometheus/2.18.1" 2021-12-15T18:22:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:11 smithi022 ceph-mon[31360]: pgmap v712: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:11 smithi022 ceph-mon[40250]: pgmap v712: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:11 smithi122 ceph-mon[32255]: pgmap v712: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:22:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:22:13.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:13 smithi022 ceph-mon[31360]: pgmap v713: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:13 smithi022 ceph-mon[40250]: pgmap v713: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:13 smithi122 ceph-mon[32255]: pgmap v713: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:14.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:14.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:14.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/768736027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/768736027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/768736027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:15.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:22:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:22:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:15 smithi022 ceph-mon[31360]: pgmap v714: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:15 smithi022 ceph-mon[40250]: pgmap v714: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:15 smithi122 ceph-mon[32255]: pgmap v714: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:17.713 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:22:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:17 smithi022 ceph-mon[31360]: pgmap v715: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:17 smithi022 ceph-mon[40250]: pgmap v715: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:17 smithi122 ceph-mon[32255]: pgmap v715: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:19.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:19.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:19.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:19.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:19 smithi022 ceph-mon[31360]: pgmap v716: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2417817473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:19 smithi022 ceph-mon[40250]: pgmap v716: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2417817473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:19 smithi122 ceph-mon[32255]: pgmap v716: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2417817473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:20] "GET /metrics HTTP/1.1" 200 202181 "" "Prometheus/2.18.1" 2021-12-15T18:22:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:21 smithi022 ceph-mon[31360]: pgmap v717: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:21 smithi022 ceph-mon[40250]: pgmap v717: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:21 smithi122 ceph-mon[32255]: pgmap v717: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:22:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:22:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:23 smithi022 ceph-mon[31360]: pgmap v718: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:23 smithi022 ceph-mon[40250]: pgmap v718: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:23 smithi122 ceph-mon[32255]: pgmap v718: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:24.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:25.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:25.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:25.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:22:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:22:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:25 smithi022 ceph-mon[31360]: pgmap v719: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/470526228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:25 smithi022 ceph-mon[40250]: pgmap v719: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/470526228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:25 smithi122 ceph-mon[32255]: pgmap v719: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/470526228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:27.729 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:22:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:27 smithi022 ceph-mon[31360]: pgmap v720: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:27 smithi022 ceph-mon[40250]: pgmap v720: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:27 smithi122 ceph-mon[32255]: pgmap v720: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:29 smithi022 ceph-mon[31360]: pgmap v721: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:29 smithi022 ceph-mon[40250]: pgmap v721: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:30.043 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:29 smithi122 ceph-mon[32255]: pgmap v721: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:30.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:30.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:30.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:30] "GET /metrics HTTP/1.1" 200 202181 "" "Prometheus/2.18.1" 2021-12-15T18:22:30.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/475918947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/475918947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/475918947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:31 smithi022 ceph-mon[31360]: pgmap v722: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:31 smithi022 ceph-mon[40250]: pgmap v722: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:31 smithi122 ceph-mon[32255]: pgmap v722: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:22:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:22:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:22:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:22:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:22:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:22:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:22:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:22:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:22:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:22:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:22:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:22:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:22:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:22:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:33 smithi022 ceph-mon[31360]: pgmap v723: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:33 smithi022 ceph-mon[40250]: pgmap v723: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:33 smithi122 ceph-mon[32255]: pgmap v723: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:35.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:22:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:22:35.412 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:35.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:35.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:35.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:35 smithi022 ceph-mon[31360]: pgmap v724: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:35 smithi022 ceph-mon[40250]: pgmap v724: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:35 smithi122 ceph-mon[32255]: pgmap v724: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2070496838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2070496838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2070496838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:37.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:22:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:37 smithi022 ceph-mon[31360]: pgmap v725: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:37 smithi022 ceph-mon[40250]: pgmap v725: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:37 smithi122 ceph-mon[32255]: pgmap v725: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:39 smithi022 ceph-mon[31360]: pgmap v726: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:39 smithi022 ceph-mon[40250]: pgmap v726: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:39 smithi122 ceph-mon[32255]: pgmap v726: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:40] "GET /metrics HTTP/1.1" 200 202185 "" "Prometheus/2.18.1" 2021-12-15T18:22:40.786 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:41 smithi022 ceph-mon[31360]: pgmap v727: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4024914120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:41 smithi022 ceph-mon[40250]: pgmap v727: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4024914120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:41 smithi122 ceph-mon[32255]: pgmap v727: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4024914120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:22:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:22:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:43 smithi022 ceph-mon[31360]: pgmap v728: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:43 smithi022 ceph-mon[40250]: pgmap v728: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:43 smithi122 ceph-mon[32255]: pgmap v728: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:45.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:22:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:22:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:45 smithi022 ceph-mon[31360]: pgmap v729: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:45 smithi022 ceph-mon[40250]: pgmap v729: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:46.154 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:45 smithi122 ceph-mon[32255]: pgmap v729: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:46.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:46.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:46.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3830250961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:46.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3830250961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3830250961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:47.769 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:22:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:47 smithi022 ceph-mon[31360]: pgmap v730: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:47 smithi022 ceph-mon[40250]: pgmap v730: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:47 smithi122 ceph-mon[32255]: pgmap v730: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:49 smithi022 ceph-mon[31360]: pgmap v731: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:49 smithi022 ceph-mon[40250]: pgmap v731: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:49 smithi122 ceph-mon[32255]: pgmap v731: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:50] "GET /metrics HTTP/1.1" 200 202188 "" "Prometheus/2.18.1" 2021-12-15T18:22:51.522 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:51.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:51.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:51.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:51 smithi022 ceph-mon[31360]: pgmap v732: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:51 smithi022 ceph-mon[40250]: pgmap v732: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:51 smithi122 ceph-mon[32255]: pgmap v732: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:22:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:22:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1878458560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1878458560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1878458560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:53 smithi122 ceph-mon[32255]: pgmap v733: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:53 smithi022 ceph-mon[31360]: pgmap v733: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:53 smithi022 ceph-mon[40250]: pgmap v733: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:22:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:22:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:55 smithi122 ceph-mon[32255]: pgmap v734: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:55 smithi022 ceph-mon[31360]: pgmap v734: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:55 smithi022 ceph-mon[40250]: pgmap v734: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:56.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:22:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:57.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:22:57.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:22:57.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:22:57.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:57.262 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:22:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:22:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:57 smithi122 ceph-mon[32255]: pgmap v735: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/313277812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:57 smithi022 ceph-mon[31360]: pgmap v735: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/313277812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:57 smithi022 ceph-mon[40250]: pgmap v735: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:22:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/313277812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:22:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:22:58 smithi122 ceph-mon[32255]: pgmap v736: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:22:58 smithi022 ceph-mon[31360]: pgmap v736: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:22:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:22:58 smithi022 ceph-mon[40250]: pgmap v736: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:00] "GET /metrics HTTP/1.1" 200 202188 "" "Prometheus/2.18.1" 2021-12-15T18:23:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:01 smithi122 ceph-mon[32255]: pgmap v737: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:01 smithi022 ceph-mon[31360]: pgmap v737: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:01 smithi022 ceph-mon[40250]: pgmap v737: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:02.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:02.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:02.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:02.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:02.639 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:23:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:23:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1896221231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1896221231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1896221231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:03 smithi122 ceph-mon[32255]: pgmap v738: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:03 smithi022 ceph-mon[31360]: pgmap v738: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:03 smithi022 ceph-mon[40250]: pgmap v738: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:23:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:23:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:05 smithi122 ceph-mon[32255]: pgmap v739: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:05 smithi022 ceph-mon[31360]: pgmap v739: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:05 smithi022 ceph-mon[40250]: pgmap v739: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:07.636 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:07.646 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:23:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:07 smithi122 ceph-mon[32255]: pgmap v740: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:08.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:08.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:08.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:08.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:07 smithi022 ceph-mon[31360]: pgmap v740: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:08.006 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:07 smithi022 ceph-mon[40250]: pgmap v740: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:08.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/362036071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:08.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:23:08.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:23:08.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:23:08.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/362036071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:08.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:23:08.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:23:08.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:23:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/362036071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:23:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:23:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:23:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:09 smithi122 ceph-mon[32255]: pgmap v741: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:10.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:23:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:09 smithi022 ceph-mon[31360]: pgmap v741: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:23:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:09 smithi022 ceph-mon[40250]: pgmap v741: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:23:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:10] "GET /metrics HTTP/1.1" 200 202190 "" "Prometheus/2.18.1" 2021-12-15T18:23:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:11 smithi122 ceph-mon[32255]: pgmap v742: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:11 smithi022 ceph-mon[31360]: pgmap v742: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:11 smithi022 ceph-mon[40250]: pgmap v742: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:23:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:23:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:13.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:13.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:13.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:13 smithi122 ceph-mon[32255]: pgmap v743: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/330684875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:13 smithi022 ceph-mon[31360]: pgmap v743: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/330684875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:13 smithi022 ceph-mon[40250]: pgmap v743: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/330684875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:15.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:23:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:23:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:15 smithi122 ceph-mon[32255]: pgmap v744: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:15 smithi022 ceph-mon[31360]: pgmap v744: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:15 smithi022 ceph-mon[40250]: pgmap v744: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:23:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:17 smithi122 ceph-mon[32255]: pgmap v745: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:17 smithi022 ceph-mon[31360]: pgmap v745: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:17 smithi022 ceph-mon[40250]: pgmap v745: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:18.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:18.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:18 smithi022 ceph-mon[31360]: pgmap v746: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1644826740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:18 smithi022 ceph-mon[40250]: pgmap v746: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1644826740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:18 smithi122 ceph-mon[32255]: pgmap v746: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1644826740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:20] "GET /metrics HTTP/1.1" 200 202204 "" "Prometheus/2.18.1" 2021-12-15T18:23:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:21 smithi122 ceph-mon[32255]: pgmap v747: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:21 smithi022 ceph-mon[31360]: pgmap v747: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:21 smithi022 ceph-mon[40250]: pgmap v747: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:23:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:23:23.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:23 smithi122 ceph-mon[32255]: pgmap v748: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:23 smithi022 ceph-mon[31360]: pgmap v748: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:23 smithi022 ceph-mon[40250]: pgmap v748: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:24.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:24.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:24.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4147898886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:24.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:23:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:23:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4147898886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4147898886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:25 smithi122 ceph-mon[32255]: pgmap v749: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:25 smithi022 ceph-mon[31360]: pgmap v749: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:25 smithi022 ceph-mon[40250]: pgmap v749: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:27.653 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:23:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:27 smithi122 ceph-mon[32255]: pgmap v750: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:27 smithi022 ceph-mon[31360]: pgmap v750: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:27 smithi022 ceph-mon[40250]: pgmap v750: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:29.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:29.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:29.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:29 smithi022 ceph-mon[31360]: pgmap v751: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3568161233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:29 smithi022 ceph-mon[40250]: pgmap v751: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3568161233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:29 smithi122 ceph-mon[32255]: pgmap v751: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3568161233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:30] "GET /metrics HTTP/1.1" 200 202204 "" "Prometheus/2.18.1" 2021-12-15T18:23:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:31 smithi122 ceph-mon[32255]: pgmap v752: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:31 smithi022 ceph-mon[31360]: pgmap v752: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:31 smithi022 ceph-mon[40250]: pgmap v752: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:32.664 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:23:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:23:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:23:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:23:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:23:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:23:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:23:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:23:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:23:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:23:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:23:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:23:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:23:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:23:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:33 smithi122 ceph-mon[32255]: pgmap v753: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:33 smithi022 ceph-mon[31360]: pgmap v753: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:33 smithi022 ceph-mon[40250]: pgmap v753: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:34.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:34.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:34.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:34.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:35.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:23:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:23:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:35 smithi122 ceph-mon[32255]: pgmap v754: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1388287600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:35 smithi022 ceph-mon[31360]: pgmap v754: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1388287600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:35 smithi022 ceph-mon[40250]: pgmap v754: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1388287600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:37.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:23:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:37 smithi122 ceph-mon[32255]: pgmap v755: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:37 smithi022 ceph-mon[31360]: pgmap v755: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:37 smithi022 ceph-mon[40250]: pgmap v755: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:39.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:39 smithi122 ceph-mon[32255]: pgmap v756: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:39 smithi022 ceph-mon[31360]: pgmap v756: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:39 smithi022 ceph-mon[40250]: pgmap v756: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:40.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:40.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:40.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:40] "GET /metrics HTTP/1.1" 200 202201 "" "Prometheus/2.18.1" 2021-12-15T18:23:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2793209755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2793209755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2793209755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:41 smithi122 ceph-mon[32255]: pgmap v757: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:41 smithi022 ceph-mon[31360]: pgmap v757: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:41 smithi022 ceph-mon[40250]: pgmap v757: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:23:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:23:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:43 smithi022 ceph-mon[31360]: pgmap v758: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:43 smithi022 ceph-mon[40250]: pgmap v758: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:43 smithi122 ceph-mon[32255]: pgmap v758: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:23:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:23:45.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:45.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:45.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:45.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:45 smithi022 ceph-mon[31360]: pgmap v759: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4101822527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:45 smithi022 ceph-mon[40250]: pgmap v759: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4101822527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:45 smithi122 ceph-mon[32255]: pgmap v759: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4101822527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:47.699 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:23:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:47 smithi022 ceph-mon[31360]: pgmap v760: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:47 smithi022 ceph-mon[40250]: pgmap v760: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:47 smithi122 ceph-mon[32255]: pgmap v760: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:49 smithi022 ceph-mon[31360]: pgmap v761: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:49 smithi022 ceph-mon[40250]: pgmap v761: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:49 smithi122 ceph-mon[32255]: pgmap v761: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:50] "GET /metrics HTTP/1.1" 200 202209 "" "Prometheus/2.18.1" 2021-12-15T18:23:50.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:50.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:50.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:50.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:51 smithi022 ceph-mon[31360]: pgmap v762: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/672840405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:51 smithi022 ceph-mon[40250]: pgmap v762: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/672840405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:51 smithi122 ceph-mon[32255]: pgmap v762: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/672840405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:23:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:23:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:53 smithi022 ceph-mon[31360]: pgmap v763: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:53 smithi022 ceph-mon[40250]: pgmap v763: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:53 smithi122 ceph-mon[32255]: pgmap v763: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:23:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:23:55.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:23:55.984 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:55 smithi022 ceph-mon[31360]: pgmap v764: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:55.984 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:55 smithi022 ceph-mon[40250]: pgmap v764: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:55 smithi122 ceph-mon[32255]: pgmap v764: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:23:56.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:23:56.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:23:56.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:23:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1197475174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1197475174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:23:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1197475174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:23:57.714 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:23:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:57 smithi022 ceph-mon[31360]: pgmap v765: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:57 smithi022 ceph-mon[40250]: pgmap v765: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:23:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:57 smithi122 ceph-mon[32255]: pgmap v765: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:23:59 smithi022 ceph-mon[31360]: pgmap v766: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:23:59 smithi022 ceph-mon[40250]: pgmap v766: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:23:59 smithi122 ceph-mon[32255]: pgmap v766: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:00] "GET /metrics HTTP/1.1" 200 202209 "" "Prometheus/2.18.1" 2021-12-15T18:24:01.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:01.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:01.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:01.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:01 smithi022 ceph-mon[31360]: pgmap v767: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/334554123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:01 smithi022 ceph-mon[40250]: pgmap v767: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/334554123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:01 smithi122 ceph-mon[32255]: pgmap v767: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/334554123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:24:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:24:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:03 smithi022 ceph-mon[31360]: pgmap v768: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:03 smithi022 ceph-mon[40250]: pgmap v768: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:03 smithi122 ceph-mon[32255]: pgmap v768: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:24:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:24:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:05 smithi022 ceph-mon[31360]: pgmap v769: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:05 smithi022 ceph-mon[40250]: pgmap v769: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:05 smithi122 ceph-mon[32255]: pgmap v769: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:06.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:07.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:07.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:07.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:07.732 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:24:08.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:07 smithi022 ceph-mon[40250]: pgmap v770: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1684704805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:07 smithi022 ceph-mon[31360]: pgmap v770: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1684704805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:07 smithi122 ceph-mon[32255]: pgmap v770: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1684704805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[31360]: pgmap v771: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:24:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:24:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:24:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:24:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[40250]: pgmap v771: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:24:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:24:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:24:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:24:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:09 smithi122 ceph-mon[32255]: pgmap v771: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:24:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:24:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:24:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:24:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:10] "GET /metrics HTTP/1.1" 200 202201 "" "Prometheus/2.18.1" 2021-12-15T18:24:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:11 smithi022 ceph-mon[31360]: pgmap v772: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:11 smithi022 ceph-mon[40250]: pgmap v772: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:12.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:11 smithi122 ceph-mon[32255]: pgmap v772: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:12.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:12.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:12.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:24:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:24:12.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3914988327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3914988327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3914988327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:13 smithi022 ceph-mon[31360]: pgmap v773: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:13 smithi022 ceph-mon[40250]: pgmap v773: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:13 smithi122 ceph-mon[32255]: pgmap v773: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:15.189 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:24:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:24:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:15 smithi022 ceph-mon[31360]: pgmap v774: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:15 smithi022 ceph-mon[40250]: pgmap v774: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:15 smithi122 ceph-mon[32255]: pgmap v774: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:17.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:17.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:24:17.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:17.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:17.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:17 smithi022 ceph-mon[31360]: pgmap v775: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:17 smithi022 ceph-mon[40250]: pgmap v775: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:17 smithi122 ceph-mon[32255]: pgmap v775: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3677476048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3677476048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3677476048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:19 smithi022 ceph-mon[31360]: pgmap v776: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:19 smithi022 ceph-mon[40250]: pgmap v776: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:19 smithi122 ceph-mon[32255]: pgmap v776: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:20] "GET /metrics HTTP/1.1" 200 202191 "" "Prometheus/2.18.1" 2021-12-15T18:24:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:21 smithi022 ceph-mon[31360]: pgmap v777: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:21 smithi022 ceph-mon[40250]: pgmap v777: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:21 smithi122 ceph-mon[32255]: pgmap v777: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:24:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:24:22.846 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:23.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:23.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:23.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:23 smithi022 ceph-mon[31360]: pgmap v778: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3729524348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:23 smithi022 ceph-mon[40250]: pgmap v778: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3729524348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:23 smithi122 ceph-mon[32255]: pgmap v778: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3729524348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:24:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:24:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:25 smithi022 ceph-mon[31360]: pgmap v779: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:25 smithi022 ceph-mon[40250]: pgmap v779: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:25 smithi122 ceph-mon[32255]: pgmap v779: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:27.772 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:24:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:27 smithi022 ceph-mon[31360]: pgmap v780: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:27 smithi022 ceph-mon[40250]: pgmap v780: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:27 smithi122 ceph-mon[32255]: pgmap v780: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:28.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:28.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2991079502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2991079502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2991079502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:29 smithi022 ceph-mon[31360]: pgmap v781: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:29 smithi022 ceph-mon[40250]: pgmap v781: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:29 smithi122 ceph-mon[32255]: pgmap v781: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:30] "GET /metrics HTTP/1.1" 200 202191 "" "Prometheus/2.18.1" 2021-12-15T18:24:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:31 smithi122 ceph-mon[32255]: pgmap v782: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:32.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:31 smithi022 ceph-mon[31360]: pgmap v782: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:31 smithi022 ceph-mon[40250]: pgmap v782: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:24:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:24:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:24:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:24:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:24:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:24:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:24:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:24:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:24:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:24:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:24:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:24:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:24:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:24:33.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:33.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:33.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:33.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:33.960 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:33 smithi022 ceph-mon[31360]: pgmap v783: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:33.960 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:33 smithi022 ceph-mon[40250]: pgmap v783: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:33 smithi122 ceph-mon[32255]: pgmap v783: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1103034968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:35.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:24:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:24:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1103034968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1103034968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:35 smithi122 ceph-mon[32255]: pgmap v784: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:35 smithi022 ceph-mon[31360]: pgmap v784: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:35 smithi022 ceph-mon[40250]: pgmap v784: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:24:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:37 smithi122 ceph-mon[32255]: pgmap v785: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:37 smithi022 ceph-mon[31360]: pgmap v785: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:37 smithi022 ceph-mon[40250]: pgmap v785: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:38.960 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:39.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:39.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:39.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:39 smithi122 ceph-mon[32255]: pgmap v786: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2700326005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:39 smithi022 ceph-mon[31360]: pgmap v786: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2700326005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:39 smithi022 ceph-mon[40250]: pgmap v786: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2700326005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:40] "GET /metrics HTTP/1.1" 200 202203 "" "Prometheus/2.18.1" 2021-12-15T18:24:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:41 smithi122 ceph-mon[32255]: pgmap v787: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:41 smithi022 ceph-mon[31360]: pgmap v787: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:41 smithi022 ceph-mon[40250]: pgmap v787: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:24:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:24:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:42 smithi122 ceph-mon[32255]: pgmap v788: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:42 smithi022 ceph-mon[31360]: pgmap v788: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:42 smithi022 ceph-mon[40250]: pgmap v788: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:44.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:44.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:44.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:24:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:24:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:45 smithi122 ceph-mon[32255]: pgmap v789: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1698601820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:45 smithi022 ceph-mon[31360]: pgmap v789: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1698601820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:45 smithi022 ceph-mon[40250]: pgmap v789: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1698601820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:47.661 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:24:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:47 smithi122 ceph-mon[32255]: pgmap v790: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:47 smithi022 ceph-mon[31360]: pgmap v790: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:47 smithi022 ceph-mon[40250]: pgmap v790: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:49.702 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:49 smithi122 ceph-mon[32255]: pgmap v791: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:49 smithi022 ceph-mon[31360]: pgmap v791: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:49 smithi022 ceph-mon[40250]: pgmap v791: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:50.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:50.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:50.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:50] "GET /metrics HTTP/1.1" 200 202203 "" "Prometheus/2.18.1" 2021-12-15T18:24:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3552253292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3552253292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3552253292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:51 smithi122 ceph-mon[32255]: pgmap v792: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:51 smithi022 ceph-mon[31360]: pgmap v792: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:51 smithi022 ceph-mon[40250]: pgmap v792: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:24:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:24:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:53 smithi122 ceph-mon[32255]: pgmap v793: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:53 smithi022 ceph-mon[31360]: pgmap v793: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:53 smithi022 ceph-mon[40250]: pgmap v793: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:55.073 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:24:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:24:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:24:55.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:24:55.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:24:55.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:24:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:55 smithi022 ceph-mon[31360]: pgmap v794: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1907414957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:55 smithi022 ceph-mon[40250]: pgmap v794: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1907414957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:55 smithi122 ceph-mon[32255]: pgmap v794: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:24:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1907414957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:24:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:24:57.681 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:24:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:57 smithi122 ceph-mon[32255]: pgmap v795: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:57 smithi022 ceph-mon[31360]: pgmap v795: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:57 smithi022 ceph-mon[40250]: pgmap v795: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:24:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:24:59 smithi122 ceph-mon[32255]: pgmap v796: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:24:59 smithi022 ceph-mon[31360]: pgmap v796: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:24:59 smithi022 ceph-mon[40250]: pgmap v796: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:00] "GET /metrics HTTP/1.1" 200 202203 "" "Prometheus/2.18.1" 2021-12-15T18:25:00.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:00.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:00.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:00.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:01 smithi022 ceph-mon[31360]: pgmap v797: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2459755650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:01 smithi022 ceph-mon[40250]: pgmap v797: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2459755650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:01 smithi122 ceph-mon[32255]: pgmap v797: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2459755650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:25:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:25:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:03 smithi022 ceph-mon[31360]: pgmap v798: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:03 smithi022 ceph-mon[40250]: pgmap v798: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:03 smithi122 ceph-mon[32255]: pgmap v798: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:25:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:25:05.810 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:05 smithi022 ceph-mon[31360]: pgmap v799: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:05 smithi022 ceph-mon[40250]: pgmap v799: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:06.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:06.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:06.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:05 smithi122 ceph-mon[32255]: pgmap v799: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4267470880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4267470880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4267470880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:07.697 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:25:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:07 smithi022 ceph-mon[31360]: pgmap v800: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:07 smithi022 ceph-mon[40250]: pgmap v800: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:07 smithi122 ceph-mon[32255]: pgmap v800: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:09 smithi122 ceph-mon[32255]: pgmap v801: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:25:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:25:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:25:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[31360]: pgmap v801: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:25:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:25:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:25:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[40250]: pgmap v801: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:25:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:25:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:25:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:10] "GET /metrics HTTP/1.1" 200 202206 "" "Prometheus/2.18.1" 2021-12-15T18:25:11.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:11 smithi122 ceph-mon[32255]: pgmap v802: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:11 smithi022 ceph-mon[31360]: pgmap v802: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:25:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:11 smithi022 ceph-mon[40250]: pgmap v802: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:11.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:11.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:11.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1558890513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:25:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:25:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1558890513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1558890513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:13 smithi122 ceph-mon[32255]: pgmap v803: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:13 smithi022 ceph-mon[31360]: pgmap v803: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:13 smithi022 ceph-mon[40250]: pgmap v803: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:15.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:25:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:25:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:15 smithi122 ceph-mon[32255]: pgmap v804: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:15 smithi022 ceph-mon[31360]: pgmap v804: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:15 smithi022 ceph-mon[40250]: pgmap v804: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:16.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:16.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:16.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:16.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:16.917 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:16.917 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:17.666 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:25:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:17 smithi122 ceph-mon[32255]: pgmap v805: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/985727661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:17 smithi022 ceph-mon[31360]: pgmap v805: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/985727661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:17 smithi022 ceph-mon[40250]: pgmap v805: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/985727661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:19 smithi122 ceph-mon[32255]: pgmap v806: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:19 smithi022 ceph-mon[31360]: pgmap v806: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:19 smithi022 ceph-mon[40250]: pgmap v806: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:20] "GET /metrics HTTP/1.1" 200 202201 "" "Prometheus/2.18.1" 2021-12-15T18:25:21.918 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:21 smithi122 ceph-mon[32255]: pgmap v807: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:21 smithi022 ceph-mon[31360]: pgmap v807: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:21 smithi022 ceph-mon[40250]: pgmap v807: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:22.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:22.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:22.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2245334212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:22.677 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2245334212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:22.678 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:25:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:25:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2245334212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:23 smithi122 ceph-mon[32255]: pgmap v808: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:23 smithi022 ceph-mon[31360]: pgmap v808: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:23 smithi022 ceph-mon[40250]: pgmap v808: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:25:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:25:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:25 smithi122 ceph-mon[32255]: pgmap v809: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:25 smithi022 ceph-mon[40250]: pgmap v809: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:25 smithi022 ceph-mon[31360]: pgmap v809: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:27.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:27.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:27.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:27.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:27.655 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:25:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:27 smithi022 ceph-mon[31360]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3489734450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:27 smithi022 ceph-mon[40250]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3489734450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:27 smithi122 ceph-mon[32255]: pgmap v810: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3489734450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:29 smithi022 ceph-mon[31360]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:29 smithi022 ceph-mon[40250]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:29 smithi122 ceph-mon[32255]: pgmap v811: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:30] "GET /metrics HTTP/1.1" 200 202201 "" "Prometheus/2.18.1" 2021-12-15T18:25:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:31 smithi022 ceph-mon[31360]: pgmap v812: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:31 smithi022 ceph-mon[40250]: pgmap v812: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:31 smithi122 ceph-mon[32255]: pgmap v812: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:32.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:25:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:25:33.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:33.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:33.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:33.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:25:33.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:25:33.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:25:33.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:25:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:25:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:25:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:25:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:25:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:25:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:25:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:25:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:25:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:33 smithi022 ceph-mon[31360]: pgmap v813: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3735827086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:33 smithi022 ceph-mon[40250]: pgmap v813: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3735827086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:33 smithi122 ceph-mon[32255]: pgmap v813: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3735827086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:25:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:25:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:35 smithi022 ceph-mon[31360]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:35 smithi022 ceph-mon[40250]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:35 smithi122 ceph-mon[32255]: pgmap v814: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:37.707 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:25:38.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:37 smithi022 ceph-mon[31360]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:37 smithi022 ceph-mon[40250]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:37 smithi122 ceph-mon[32255]: pgmap v815: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:38.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:38.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:38.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:38.711 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1751928590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1751928590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1751928590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:39 smithi022 ceph-mon[31360]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:39 smithi022 ceph-mon[40250]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:39 smithi122 ceph-mon[32255]: pgmap v816: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:40] "GET /metrics HTTP/1.1" 200 202202 "" "Prometheus/2.18.1" 2021-12-15T18:25:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:41 smithi022 ceph-mon[31360]: pgmap v817: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:41 smithi022 ceph-mon[40250]: pgmap v817: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:41 smithi122 ceph-mon[32255]: pgmap v817: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:25:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:25:43.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:43.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:43.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:43.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:43 smithi022 ceph-mon[31360]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:43 smithi022 ceph-mon[40250]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:43 smithi122 ceph-mon[32255]: pgmap v818: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2861799404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2861799404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:25:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:25:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2861799404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:45 smithi022 ceph-mon[31360]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:45 smithi022 ceph-mon[40250]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:45 smithi122 ceph-mon[32255]: pgmap v819: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:47.730 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:25:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:47 smithi022 ceph-mon[31360]: pgmap v820: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:47 smithi022 ceph-mon[40250]: pgmap v820: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:47 smithi122 ceph-mon[32255]: pgmap v820: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:48.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:49.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:49.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:49.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:49 smithi022 ceph-mon[31360]: pgmap v821: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187379336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:49 smithi022 ceph-mon[40250]: pgmap v821: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187379336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:49 smithi122 ceph-mon[32255]: pgmap v821: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187379336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:50] "GET /metrics HTTP/1.1" 200 202202 "" "Prometheus/2.18.1" 2021-12-15T18:25:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:51 smithi022 ceph-mon[31360]: pgmap v822: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:51 smithi022 ceph-mon[40250]: pgmap v822: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:51 smithi122 ceph-mon[32255]: pgmap v822: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:25:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:25:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:53 smithi022 ceph-mon[31360]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:53 smithi022 ceph-mon[40250]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:54.140 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:53 smithi122 ceph-mon[32255]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:54.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:54.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:54.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:25:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2753753665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2753753665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:25:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:25:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2753753665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:25:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:55 smithi022 ceph-mon[31360]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:55 smithi022 ceph-mon[40250]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:55 smithi122 ceph-mon[32255]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:25:57.751 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:25:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:57 smithi022 ceph-mon[31360]: pgmap v825: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:57 smithi022 ceph-mon[40250]: pgmap v825: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:57 smithi122 ceph-mon[32255]: pgmap v825: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:25:59.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:25:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:25:59 smithi022 ceph-mon[31360]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:25:59 smithi022 ceph-mon[40250]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:25:59.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:25:59.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:25:59.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:25:59 smithi122 ceph-mon[32255]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:00] "GET /metrics HTTP/1.1" 200 202202 "" "Prometheus/2.18.1" 2021-12-15T18:26:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/7281533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/7281533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/7281533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:01 smithi022 ceph-mon[31360]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:01 smithi022 ceph-mon[40250]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:01 smithi122 ceph-mon[32255]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:26:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:26:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:03 smithi022 ceph-mon[31360]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:03 smithi022 ceph-mon[40250]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:03 smithi122 ceph-mon[32255]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:04.886 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:05.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:05.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:05.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:05.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:26:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:26:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:05 smithi022 ceph-mon[31360]: pgmap v829: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2990849689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:05 smithi022 ceph-mon[40250]: pgmap v829: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2990849689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:05 smithi122 ceph-mon[32255]: pgmap v829: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2990849689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:07.772 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:26:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:07 smithi022 ceph-mon[31360]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:07 smithi022 ceph-mon[40250]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:07 smithi122 ceph-mon[32255]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:09 smithi022 ceph-mon[31360]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:09 smithi022 ceph-mon[40250]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:10.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:09 smithi122 ceph-mon[32255]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:10.258 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:10] "GET /metrics HTTP/1.1" 200 202204 "" "Prometheus/2.18.1" 2021-12-15T18:26:10.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:10.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:10.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:26:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:26:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:26:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:26:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3611853605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:26:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:26:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:26:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:26:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3611853605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:26:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:26:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:26:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:26:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3611853605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:11 smithi122 ceph-mon[32255]: pgmap v832: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:11 smithi022 ceph-mon[31360]: pgmap v832: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:11 smithi022 ceph-mon[40250]: pgmap v832: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:26:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:26:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:13 smithi122 ceph-mon[32255]: pgmap v833: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:13 smithi022 ceph-mon[31360]: pgmap v833: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:13 smithi022 ceph-mon[40250]: pgmap v833: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:26:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:26:15.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:16.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:16.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:16.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:16.007 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:15 smithi022 ceph-mon[31360]: pgmap v834: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:16.007 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:15 smithi022 ceph-mon[40250]: pgmap v834: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:15 smithi122 ceph-mon[32255]: pgmap v834: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1273216078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1273216078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1273216078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:26:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:17 smithi122 ceph-mon[32255]: pgmap v835: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:17 smithi022 ceph-mon[31360]: pgmap v835: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:17 smithi022 ceph-mon[40250]: pgmap v835: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:20.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:19 smithi122 ceph-mon[32255]: pgmap v836: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:19 smithi022 ceph-mon[31360]: pgmap v836: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:19 smithi022 ceph-mon[40250]: pgmap v836: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:20] "GET /metrics HTTP/1.1" 200 202206 "" "Prometheus/2.18.1" 2021-12-15T18:26:21.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:21.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:21.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:21.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:21 smithi122 ceph-mon[32255]: pgmap v837: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3672066803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:21 smithi022 ceph-mon[31360]: pgmap v837: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3672066803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:21 smithi022 ceph-mon[40250]: pgmap v837: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3672066803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:26:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:26:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:23 smithi122 ceph-mon[32255]: pgmap v838: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:23 smithi022 ceph-mon[31360]: pgmap v838: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:23 smithi022 ceph-mon[40250]: pgmap v838: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:26:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:26:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:26:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:25 smithi122 ceph-mon[32255]: pgmap v839: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:25 smithi022 ceph-mon[31360]: pgmap v839: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:25 smithi022 ceph-mon[40250]: pgmap v839: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:26.379 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:26.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:26.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:26.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3446199777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3446199777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3446199777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:26:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:27 smithi122 ceph-mon[32255]: pgmap v840: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:27 smithi022 ceph-mon[31360]: pgmap v840: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:27 smithi022 ceph-mon[40250]: pgmap v840: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:29 smithi122 ceph-mon[32255]: pgmap v841: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:29 smithi022 ceph-mon[31360]: pgmap v841: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:29 smithi022 ceph-mon[40250]: pgmap v841: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:30] "GET /metrics HTTP/1.1" 200 202206 "" "Prometheus/2.18.1" 2021-12-15T18:26:31.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:31 smithi022 ceph-mon[31360]: pgmap v842: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:31 smithi022 ceph-mon[40250]: pgmap v842: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:32.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:32.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:32.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:31 smithi122 ceph-mon[32255]: pgmap v842: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:26:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:26:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2315806961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:26:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:26:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:26:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:26:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2315806961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:26:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:26:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:26:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:26:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2315806961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:26:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:26:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:26:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:26:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:33 smithi122 ceph-mon[32255]: pgmap v843: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:33 smithi022 ceph-mon[31360]: pgmap v843: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:33 smithi022 ceph-mon[40250]: pgmap v843: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:26:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:26:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:35 smithi122 ceph-mon[32255]: pgmap v844: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:35 smithi022 ceph-mon[31360]: pgmap v844: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:35 smithi022 ceph-mon[40250]: pgmap v844: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:37.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:37.118 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:37.118 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:36 smithi022 ceph-mon[40250]: pgmap v845: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:37.119 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:37.119 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:36 smithi022 ceph-mon[31360]: pgmap v845: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:36 smithi122 ceph-mon[32255]: pgmap v845: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:37.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:37.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:37.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:26:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/570485812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/570485812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/570485812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:38 smithi122 ceph-mon[32255]: pgmap v846: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:38 smithi022 ceph-mon[31360]: pgmap v846: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:38 smithi022 ceph-mon[40250]: pgmap v846: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:40] "GET /metrics HTTP/1.1" 200 202205 "" "Prometheus/2.18.1" 2021-12-15T18:26:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:41 smithi122 ceph-mon[32255]: pgmap v847: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:41 smithi022 ceph-mon[31360]: pgmap v847: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:41 smithi022 ceph-mon[40250]: pgmap v847: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:42.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:26:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:26:42.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:42.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:42.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:43 smithi122 ceph-mon[32255]: pgmap v848: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/801290691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:43 smithi022 ceph-mon[31360]: pgmap v848: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/801290691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:43 smithi022 ceph-mon[40250]: pgmap v848: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/801290691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:26:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:26:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:45 smithi122 ceph-mon[32255]: pgmap v849: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:45 smithi022 ceph-mon[31360]: pgmap v849: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:45 smithi022 ceph-mon[40250]: pgmap v849: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:47.690 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:26:47.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:47 smithi022 ceph-mon[31360]: pgmap v850: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:47 smithi022 ceph-mon[40250]: pgmap v850: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:47 smithi122 ceph-mon[32255]: pgmap v850: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:48.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:48.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:48.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/389387145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/389387145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/389387145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:49 smithi022 ceph-mon[31360]: pgmap v851: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:49 smithi022 ceph-mon[40250]: pgmap v851: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:49 smithi122 ceph-mon[32255]: pgmap v851: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:50] "GET /metrics HTTP/1.1" 200 202210 "" "Prometheus/2.18.1" 2021-12-15T18:26:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:51 smithi022 ceph-mon[31360]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:51 smithi022 ceph-mon[40250]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:51 smithi122 ceph-mon[32255]: pgmap v852: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:26:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:26:53.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:53.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:53.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:53.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:26:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:53 smithi022 ceph-mon[31360]: pgmap v853: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3977906550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:53 smithi022 ceph-mon[40250]: pgmap v853: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3977906550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:53 smithi122 ceph-mon[32255]: pgmap v853: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3977906550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:26:55.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:26:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:26:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:55 smithi022 ceph-mon[31360]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:55 smithi022 ceph-mon[40250]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:55 smithi122 ceph-mon[32255]: pgmap v854: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:26:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:26:57.711 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:26:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:57 smithi022 ceph-mon[31360]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:57 smithi022 ceph-mon[40250]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:57 smithi122 ceph-mon[32255]: pgmap v855: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:26:58.601 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:26:58.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:26:58.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:26:58.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:59 smithi022 ceph-mon[31360]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:26:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1642931856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:59 smithi022 ceph-mon[40250]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:26:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1642931856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:59 smithi122 ceph-mon[32255]: pgmap v856: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:26:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1642931856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:00] "GET /metrics HTTP/1.1" 200 202210 "" "Prometheus/2.18.1" 2021-12-15T18:27:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:01 smithi022 ceph-mon[31360]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:01 smithi022 ceph-mon[40250]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:01 smithi122 ceph-mon[32255]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:27:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:27:03.970 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:03 smithi022 ceph-mon[31360]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:03 smithi022 ceph-mon[40250]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:03 smithi122 ceph-mon[32255]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:04.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:04.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:04.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/312136962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/312136962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:05.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/312136962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:05.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:27:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:27:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:05 smithi022 ceph-mon[31360]: pgmap v859: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:05 smithi022 ceph-mon[40250]: pgmap v859: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:05 smithi122 ceph-mon[32255]: pgmap v859: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:07.733 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:27:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:07 smithi022 ceph-mon[31360]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:07 smithi022 ceph-mon[40250]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:07 smithi122 ceph-mon[32255]: pgmap v860: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:09.338 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:09.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:09.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:09.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:09 smithi022 ceph-mon[31360]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2860075376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:09 smithi022 ceph-mon[40250]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2860075376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:09 smithi122 ceph-mon[32255]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2860075376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:10] "GET /metrics HTTP/1.1" 200 202211 "" "Prometheus/2.18.1" 2021-12-15T18:27:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:27:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:27:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:27:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:27:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:27:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:27:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:27:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:27:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:27:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:11 smithi022 ceph-mon[31360]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:27:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:11 smithi022 ceph-mon[40250]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:27:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:11 smithi122 ceph-mon[32255]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:27:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:27:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:27:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:13 smithi022 ceph-mon[31360]: pgmap v863: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:13 smithi022 ceph-mon[40250]: pgmap v863: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:13 smithi122 ceph-mon[32255]: pgmap v863: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:14.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:15.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:15.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:15.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:27:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:27:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:15 smithi022 ceph-mon[31360]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4142942777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:15 smithi022 ceph-mon[40250]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4142942777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:15 smithi122 ceph-mon[32255]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4142942777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:17.754 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:27:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:17 smithi022 ceph-mon[31360]: pgmap v865: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:17 smithi022 ceph-mon[40250]: pgmap v865: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:17 smithi122 ceph-mon[32255]: pgmap v865: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:19 smithi022 ceph-mon[31360]: pgmap v866: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:19 smithi022 ceph-mon[40250]: pgmap v866: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:20.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:19 smithi122 ceph-mon[32255]: pgmap v866: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:20] "GET /metrics HTTP/1.1" 200 202212 "" "Prometheus/2.18.1" 2021-12-15T18:27:20.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:20.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:20.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3219479282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3219479282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3219479282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:21 smithi022 ceph-mon[31360]: pgmap v867: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:21 smithi022 ceph-mon[40250]: pgmap v867: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:21 smithi122 ceph-mon[32255]: pgmap v867: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:27:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:27:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:23 smithi022 ceph-mon[31360]: pgmap v868: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:23 smithi022 ceph-mon[40250]: pgmap v868: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:23 smithi122 ceph-mon[32255]: pgmap v868: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:27:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:27:25.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:25.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:25.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:25 smithi022 ceph-mon[31360]: pgmap v869: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:25 smithi022 ceph-mon[40250]: pgmap v869: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:25 smithi122 ceph-mon[32255]: pgmap v869: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2566296594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2566296594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2566296594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:27.774 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:27:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:27 smithi022 ceph-mon[31360]: pgmap v870: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:27 smithi022 ceph-mon[40250]: pgmap v870: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:27 smithi122 ceph-mon[32255]: pgmap v870: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:29 smithi022 ceph-mon[31360]: pgmap v871: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:29 smithi022 ceph-mon[40250]: pgmap v871: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:29 smithi122 ceph-mon[32255]: pgmap v871: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:30] "GET /metrics HTTP/1.1" 200 202212 "" "Prometheus/2.18.1" 2021-12-15T18:27:30.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:31 smithi122 ceph-mon[32255]: pgmap v872: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3052918989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:31 smithi022 ceph-mon[31360]: pgmap v872: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3052918989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:31 smithi022 ceph-mon[40250]: pgmap v872: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3052918989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:27:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:27:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:27:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:27:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:27:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:27:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:27:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:27:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:27:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:27:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:27:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:27:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:27:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:27:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:33 smithi122 ceph-mon[32255]: pgmap v873: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:33 smithi022 ceph-mon[31360]: pgmap v873: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:33 smithi022 ceph-mon[40250]: pgmap v873: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:27:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:27:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:35 smithi122 ceph-mon[32255]: pgmap v874: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:36.191 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:36.191 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:35 smithi022 ceph-mon[40250]: pgmap v874: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:36.192 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:35 smithi022 ceph-mon[31360]: pgmap v874: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:36.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:36.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:36.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3862018501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3862018501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3862018501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:27:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:37 smithi122 ceph-mon[32255]: pgmap v875: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:37 smithi022 ceph-mon[31360]: pgmap v875: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:37 smithi022 ceph-mon[40250]: pgmap v875: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:39 smithi122 ceph-mon[32255]: pgmap v876: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:39 smithi022 ceph-mon[31360]: pgmap v876: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:39 smithi022 ceph-mon[40250]: pgmap v876: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:40] "GET /metrics HTTP/1.1" 200 202210 "" "Prometheus/2.18.1" 2021-12-15T18:27:41.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:41.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:41.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:41.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:41.936 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:41 smithi022 ceph-mon[31360]: pgmap v877: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:41.936 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:41 smithi022 ceph-mon[40250]: pgmap v877: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:41 smithi122 ceph-mon[32255]: pgmap v877: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:27:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:27:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1132343657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1132343657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1132343657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:43 smithi122 ceph-mon[32255]: pgmap v878: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:43 smithi022 ceph-mon[31360]: pgmap v878: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:43 smithi022 ceph-mon[40250]: pgmap v878: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:27:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:27:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:45 smithi122 ceph-mon[32255]: pgmap v879: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:45 smithi022 ceph-mon[31360]: pgmap v879: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:45 smithi022 ceph-mon[40250]: pgmap v879: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:46.936 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:47.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:47.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:47.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:27:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:47 smithi122 ceph-mon[32255]: pgmap v880: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4046697709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:47 smithi022 ceph-mon[31360]: pgmap v880: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4046697709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:47 smithi022 ceph-mon[40250]: pgmap v880: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4046697709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:49 smithi122 ceph-mon[32255]: pgmap v881: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:49 smithi022 ceph-mon[31360]: pgmap v881: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:49 smithi022 ceph-mon[40250]: pgmap v881: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:27:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:50] "GET /metrics HTTP/1.1" 200 202211 "" "Prometheus/2.18.1" 2021-12-15T18:27:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:51 smithi122 ceph-mon[32255]: pgmap v882: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:51 smithi022 ceph-mon[31360]: pgmap v882: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:51 smithi022 ceph-mon[40250]: pgmap v882: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:52.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:52.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:27:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:27:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3621042587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3621042587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3621042587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:53 smithi122 ceph-mon[32255]: pgmap v883: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:53 smithi022 ceph-mon[31360]: pgmap v883: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:53 smithi022 ceph-mon[40250]: pgmap v883: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:27:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:27:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:27:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:55 smithi122 ceph-mon[32255]: pgmap v884: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:27:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:55 smithi022 ceph-mon[31360]: pgmap v884: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:27:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:55 smithi022 ceph-mon[40250]: pgmap v884: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:27:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:27:57.678 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:27:57.679 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:27:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:57 smithi022 ceph-mon[31360]: pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:27:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:57 smithi022 ceph-mon[40250]: pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:27:58.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:27:58.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:27:58.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:27:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:57 smithi122 ceph-mon[32255]: pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:27:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1697393910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1697393910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:27:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1697393910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:27:59 smithi122 ceph-mon[32255]: pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:27:59 smithi022 ceph-mon[31360]: pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:27:59 smithi022 ceph-mon[40250]: pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:00] "GET /metrics HTTP/1.1" 200 202211 "" "Prometheus/2.18.1" 2021-12-15T18:28:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:01 smithi022 ceph-mon[31360]: pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:02.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:01 smithi022 ceph-mon[40250]: pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:01 smithi122 ceph-mon[32255]: pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:28:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:03.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:03.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:03.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:03 smithi022 ceph-mon[31360]: pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2394453360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:03 smithi022 ceph-mon[40250]: pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2394453360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:03 smithi122 ceph-mon[32255]: pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2394453360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:05.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:28:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:28:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:05 smithi022 ceph-mon[31360]: pgmap v889: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:05 smithi022 ceph-mon[40250]: pgmap v889: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:05 smithi122 ceph-mon[32255]: pgmap v889: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:07.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:07.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:28:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:07 smithi022 ceph-mon[31360]: pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:28:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:07 smithi022 ceph-mon[40250]: pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:28:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:07 smithi122 ceph-mon[32255]: pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:28:08.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:08.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:08.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:08.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:09 smithi022 ceph-mon[31360]: pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:28:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3846351335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:09 smithi022 ceph-mon[40250]: pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:28:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3846351335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:09 smithi122 ceph-mon[32255]: pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:28:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3846351335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:10] "GET /metrics HTTP/1.1" 200 202231 "" "Prometheus/2.18.1" 2021-12-15T18:28:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[31360]: pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:28:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:28:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:28:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:28:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[40250]: pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:28:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:28:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:28:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:28:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:11 smithi122 ceph-mon[32255]: pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:28:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:28:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:28:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:28:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:28:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:28:13.783 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:13 smithi022 ceph-mon[31360]: pgmap v893: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:13 smithi022 ceph-mon[40250]: pgmap v893: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:14.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:14.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:14.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:13 smithi122 ceph-mon[32255]: pgmap v893: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1146627804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1146627804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:15.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1146627804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:28:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:28:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:15 smithi022 ceph-mon[31360]: pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:15 smithi022 ceph-mon[40250]: pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:15 smithi122 ceph-mon[32255]: pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:17.727 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:28:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:17 smithi022 ceph-mon[31360]: pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:17 smithi022 ceph-mon[40250]: pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:17 smithi122 ceph-mon[32255]: pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:19.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:19.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:19.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:19.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:19 smithi022 ceph-mon[31360]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2137835013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:19 smithi022 ceph-mon[40250]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2137835013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:19 smithi122 ceph-mon[32255]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2137835013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:20] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:28:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:21 smithi022 ceph-mon[31360]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:21 smithi022 ceph-mon[40250]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:21 smithi122 ceph-mon[32255]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:28:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:28:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:23 smithi022 ceph-mon[31360]: pgmap v898: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:23 smithi022 ceph-mon[40250]: pgmap v898: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:23 smithi122 ceph-mon[32255]: pgmap v898: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:24.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:24.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:24.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:28:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:28:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:25 smithi022 ceph-mon[31360]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3451885843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:25 smithi022 ceph-mon[40250]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3451885843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:25 smithi122 ceph-mon[32255]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3451885843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:27.744 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:28:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:27 smithi022 ceph-mon[31360]: pgmap v900: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:27 smithi022 ceph-mon[40250]: pgmap v900: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:27 smithi122 ceph-mon[32255]: pgmap v900: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:29.894 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:29 smithi022 ceph-mon[31360]: pgmap v901: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:29 smithi022 ceph-mon[40250]: pgmap v901: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:29 smithi122 ceph-mon[32255]: pgmap v901: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:30.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:30.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:30.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:30] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:28:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3625589036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3625589036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3625589036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:31 smithi022 ceph-mon[31360]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:31 smithi022 ceph-mon[40250]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:31 smithi122 ceph-mon[32255]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:28:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:28:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:28:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:28:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:28:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:28:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:28:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:28:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:28:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:28:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:28:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:28:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:28:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:28:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:33 smithi022 ceph-mon[31360]: pgmap v903: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:33 smithi022 ceph-mon[40250]: pgmap v903: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:33 smithi122 ceph-mon[32255]: pgmap v903: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:28:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:28:35.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:35.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:35.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:35 smithi022 ceph-mon[31360]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2311394896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:35 smithi022 ceph-mon[40250]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2311394896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:35 smithi122 ceph-mon[32255]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2311394896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:37.768 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:28:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:37 smithi022 ceph-mon[31360]: pgmap v905: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:37 smithi022 ceph-mon[40250]: pgmap v905: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:37 smithi122 ceph-mon[32255]: pgmap v905: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:39 smithi022 ceph-mon[31360]: pgmap v906: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:39 smithi022 ceph-mon[40250]: pgmap v906: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:39 smithi122 ceph-mon[32255]: pgmap v906: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:40] "GET /metrics HTTP/1.1" 200 202234 "" "Prometheus/2.18.1" 2021-12-15T18:28:40.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:41.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:41.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:41.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:41 smithi022 ceph-mon[31360]: pgmap v907: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/298802795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:41 smithi022 ceph-mon[40250]: pgmap v907: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/298802795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:41 smithi122 ceph-mon[32255]: pgmap v907: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/298802795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:28:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:28:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:43 smithi022 ceph-mon[31360]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:43 smithi022 ceph-mon[40250]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:43 smithi122 ceph-mon[32255]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:28:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:28:46.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:45 smithi022 ceph-mon[31360]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:45 smithi022 ceph-mon[40250]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:45 smithi122 ceph-mon[32255]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:46.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:46.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:46.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1684016369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1684016369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1684016369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:28:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:47 smithi122 ceph-mon[32255]: pgmap v910: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:47 smithi022 ceph-mon[31360]: pgmap v910: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:47 smithi022 ceph-mon[40250]: pgmap v910: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:49 smithi122 ceph-mon[32255]: pgmap v911: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:49 smithi022 ceph-mon[31360]: pgmap v911: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:49 smithi022 ceph-mon[40250]: pgmap v911: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:50] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:28:51.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:51.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:51.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:51.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:51 smithi022 ceph-mon[31360]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/666386130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:51 smithi022 ceph-mon[40250]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/666386130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:51 smithi122 ceph-mon[32255]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/666386130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:28:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:28:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:53 smithi122 ceph-mon[32255]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:53 smithi022 ceph-mon[31360]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:53 smithi022 ceph-mon[40250]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:28:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:28:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:55 smithi122 ceph-mon[32255]: pgmap v914: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:55 smithi022 ceph-mon[31360]: pgmap v914: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:55 smithi022 ceph-mon[40250]: pgmap v914: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:28:56.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:28:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:57.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:28:57.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:28:57.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:28:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:28:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:28:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:57 smithi122 ceph-mon[32255]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/635471166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:57 smithi022 ceph-mon[31360]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/635471166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:28:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:57 smithi022 ceph-mon[40250]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:28:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/635471166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:28:59 smithi122 ceph-mon[32255]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:28:59 smithi022 ceph-mon[31360]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:28:59 smithi022 ceph-mon[40250]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:00] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:29:02.121 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:02.122 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:01 smithi022 ceph-mon[31360]: pgmap v917: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:02.122 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:01 smithi022 ceph-mon[40250]: pgmap v917: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:01 smithi122 ceph-mon[32255]: pgmap v917: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:02.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:02.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:02.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:29:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:29:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1089906966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1089906966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1089906966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:03 smithi122 ceph-mon[32255]: pgmap v918: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:03 smithi022 ceph-mon[31360]: pgmap v918: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:03 smithi022 ceph-mon[40250]: pgmap v918: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:05.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:29:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:29:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:05 smithi122 ceph-mon[32255]: pgmap v919: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:05 smithi022 ceph-mon[31360]: pgmap v919: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:05 smithi022 ceph-mon[40250]: pgmap v919: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:07.490 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:29:07.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:07.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:07.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:07 smithi122 ceph-mon[32255]: pgmap v920: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:07 smithi022 ceph-mon[31360]: pgmap v920: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:07 smithi022 ceph-mon[40250]: pgmap v920: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3144387460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3144387460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3144387460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:09 smithi122 ceph-mon[32255]: pgmap v921: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:09 smithi022 ceph-mon[31360]: pgmap v921: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:09 smithi022 ceph-mon[40250]: pgmap v921: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:10] "GET /metrics HTTP/1.1" 200 202235 "" "Prometheus/2.18.1" 2021-12-15T18:29:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:11 smithi122 ceph-mon[32255]: pgmap v922: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:29:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:29:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:29:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[31360]: pgmap v922: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:29:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:29:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:29:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[40250]: pgmap v922: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:29:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:29:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:29:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:29:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:29:12.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:13.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:13.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:13 smithi122 ceph-mon[32255]: pgmap v923: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3465974066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:13 smithi022 ceph-mon[31360]: pgmap v923: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3465974066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:13 smithi022 ceph-mon[40250]: pgmap v923: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3465974066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:29:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:29:15.962 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:15 smithi022 ceph-mon[31360]: pgmap v924: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:15.962 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:15.963 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:15 smithi022 ceph-mon[40250]: pgmap v924: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:15.963 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:15 smithi122 ceph-mon[32255]: pgmap v924: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:16.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:16.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:16.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:16.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:16.955 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:16.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:16.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:16.957 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:17.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:29:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:17 smithi122 ceph-mon[32255]: pgmap v925: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:18.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:17 smithi022 ceph-mon[31360]: pgmap v925: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:18.232 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:17 smithi022 ceph-mon[40250]: pgmap v925: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:18.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:18.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/523955664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/523955664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:19.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/523955664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:19.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:19 smithi122 ceph-mon[32255]: pgmap v926: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:19 smithi022 ceph-mon[31360]: pgmap v926: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:19 smithi022 ceph-mon[40250]: pgmap v926: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:29:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:20] "GET /metrics HTTP/1.1" 200 202233 "" "Prometheus/2.18.1" 2021-12-15T18:29:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:21 smithi122 ceph-mon[32255]: pgmap v927: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:21 smithi022 ceph-mon[31360]: pgmap v927: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:21 smithi022 ceph-mon[40250]: pgmap v927: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:29:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:29:23.599 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:23.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:23.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:23.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:23.970 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:23 smithi022 ceph-mon[31360]: pgmap v928: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:23.970 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:23 smithi022 ceph-mon[40250]: pgmap v928: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:23 smithi122 ceph-mon[32255]: pgmap v928: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:25.144 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/278139706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:25.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:24 smithi122 ceph-mon[32255]: pgmap v929: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/278139706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:24 smithi022 ceph-mon[40250]: pgmap v929: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/278139706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:24 smithi022 ceph-mon[31360]: pgmap v929: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:29:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:29:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:27.723 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:29:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:27 smithi022 ceph-mon[31360]: pgmap v930: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:27 smithi022 ceph-mon[40250]: pgmap v930: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:27 smithi122 ceph-mon[32255]: pgmap v930: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:28.970 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:29.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:29.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:29.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:29 smithi022 ceph-mon[31360]: pgmap v931: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/542509246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:29 smithi022 ceph-mon[40250]: pgmap v931: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/542509246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:29 smithi122 ceph-mon[32255]: pgmap v931: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/542509246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:30] "GET /metrics HTTP/1.1" 200 202233 "" "Prometheus/2.18.1" 2021-12-15T18:29:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:31 smithi022 ceph-mon[31360]: pgmap v932: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:31 smithi022 ceph-mon[40250]: pgmap v932: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:31 smithi122 ceph-mon[32255]: pgmap v932: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:29:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:29:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:29:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:29:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:29:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:29:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:29:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:29:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:29:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:29:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:29:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:29:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:29:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:29:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:33 smithi022 ceph-mon[31360]: pgmap v933: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:33 smithi022 ceph-mon[40250]: pgmap v933: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:33 smithi122 ceph-mon[32255]: pgmap v933: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:34.338 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:34.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:34.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:34.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1261952410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1261952410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1261952410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:29:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:29:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:35 smithi022 ceph-mon[31360]: pgmap v934: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:35 smithi022 ceph-mon[40250]: pgmap v934: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:35 smithi122 ceph-mon[32255]: pgmap v934: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:37.754 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:29:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:37 smithi022 ceph-mon[31360]: pgmap v935: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:37 smithi022 ceph-mon[40250]: pgmap v935: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:37 smithi122 ceph-mon[32255]: pgmap v935: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:39.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:39 smithi022 ceph-mon[31360]: pgmap v936: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:39 smithi022 ceph-mon[40250]: pgmap v936: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:40.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:40.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:40.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:39 smithi122 ceph-mon[32255]: pgmap v936: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:40] "GET /metrics HTTP/1.1" 200 202238 "" "Prometheus/2.18.1" 2021-12-15T18:29:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3042330992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3042330992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3042330992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:41 smithi022 ceph-mon[31360]: pgmap v937: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:41 smithi022 ceph-mon[40250]: pgmap v937: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:41 smithi122 ceph-mon[32255]: pgmap v937: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:29:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:29:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:43 smithi022 ceph-mon[31360]: pgmap v938: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:43 smithi022 ceph-mon[40250]: pgmap v938: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:43 smithi122 ceph-mon[32255]: pgmap v938: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:45.081 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:29:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:29:45.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:45.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:45.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:45 smithi022 ceph-mon[31360]: pgmap v939: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3334709998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:45 smithi022 ceph-mon[40250]: pgmap v939: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3334709998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:45 smithi122 ceph-mon[32255]: pgmap v939: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3334709998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:47.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:29:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:47 smithi022 ceph-mon[31360]: pgmap v940: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:47 smithi022 ceph-mon[40250]: pgmap v940: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:47 smithi122 ceph-mon[32255]: pgmap v940: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:49 smithi022 ceph-mon[31360]: pgmap v941: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:49 smithi022 ceph-mon[40250]: pgmap v941: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:49 smithi122 ceph-mon[32255]: pgmap v941: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:50] "GET /metrics HTTP/1.1" 200 202237 "" "Prometheus/2.18.1" 2021-12-15T18:29:50.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:50.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:50.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:50.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:51 smithi022 ceph-mon[31360]: pgmap v942: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2006908947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:51 smithi022 ceph-mon[40250]: pgmap v942: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2006908947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:51 smithi122 ceph-mon[32255]: pgmap v942: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2006908947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:29:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:29:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:53 smithi122 ceph-mon[32255]: pgmap v943: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:53 smithi022 ceph-mon[31360]: pgmap v943: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:53 smithi022 ceph-mon[40250]: pgmap v943: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:29:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:29:55.823 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:29:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:55 smithi122 ceph-mon[32255]: pgmap v944: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:56.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:29:56.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:29:56.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:29:56.192 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:55 smithi022 ceph-mon[40250]: pgmap v944: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:56.192 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:55 smithi022 ceph-mon[31360]: pgmap v944: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:29:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1880593353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1880593353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:29:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1880593353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:29:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:29:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:57 smithi122 ceph-mon[32255]: pgmap v945: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:57 smithi022 ceph-mon[31360]: pgmap v945: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:29:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:57 smithi022 ceph-mon[40250]: pgmap v945: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:29:59 smithi122 ceph-mon[32255]: pgmap v946: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:29:59 smithi022 ceph-mon[31360]: pgmap v946: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:29:59 smithi022 ceph-mon[40250]: pgmap v946: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:00] "GET /metrics HTTP/1.1" 200 202237 "" "Prometheus/2.18.1" 2021-12-15T18:30:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T18:30:01.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:01.194 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T18:30:01.194 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T18:30:01.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:01.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:01.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:01 smithi022 ceph-mon[31360]: pgmap v947: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3481534394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:01 smithi022 ceph-mon[40250]: pgmap v947: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3481534394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:01 smithi122 ceph-mon[32255]: pgmap v947: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3481534394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:30:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:30:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:03 smithi122 ceph-mon[32255]: pgmap v948: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:03 smithi022 ceph-mon[31360]: pgmap v948: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:03 smithi022 ceph-mon[40250]: pgmap v948: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:05.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:30:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:30:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:05 smithi122 ceph-mon[32255]: pgmap v949: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:05 smithi022 ceph-mon[31360]: pgmap v949: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:05 smithi022 ceph-mon[40250]: pgmap v949: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:06.563 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:06.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:06.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:06.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:06.936 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:06.936 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:30:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:07 smithi122 ceph-mon[32255]: pgmap v950: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1925599308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:07 smithi022 ceph-mon[31360]: pgmap v950: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1925599308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:07 smithi022 ceph-mon[40250]: pgmap v950: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1925599308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:09 smithi122 ceph-mon[32255]: pgmap v951: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:09 smithi022 ceph-mon[31360]: pgmap v951: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:09 smithi022 ceph-mon[40250]: pgmap v951: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:10] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:30:11.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:11 smithi122 ceph-mon[32255]: pgmap v952: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:11 smithi022 ceph-mon[31360]: pgmap v952: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:11 smithi022 ceph-mon[40250]: pgmap v952: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:12.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:12.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:12.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:30:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:30:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1927788340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1927788340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1927788340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:13 smithi122 ceph-mon[32255]: pgmap v953: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:13 smithi022 ceph-mon[31360]: pgmap v953: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:13 smithi022 ceph-mon[40250]: pgmap v953: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:15.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:30:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:30:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:15 smithi122 ceph-mon[32255]: pgmap v954: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:15 smithi022 ceph-mon[31360]: pgmap v954: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:15 smithi022 ceph-mon[40250]: pgmap v954: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:17.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:17.588 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:30:17.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:17.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:17.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:17 smithi022 ceph-mon[31360]: pgmap v955: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4148645747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:17 smithi022 ceph-mon[40250]: pgmap v955: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4148645747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:17 smithi122 ceph-mon[32255]: pgmap v955: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4148645747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:19 smithi122 ceph-mon[32255]: pgmap v956: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:30:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:30:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:30:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[31360]: pgmap v956: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:30:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:30:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:30:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[40250]: pgmap v956: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:30:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:30:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:30:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:20] "GET /metrics HTTP/1.1" 200 202234 "" "Prometheus/2.18.1" 2021-12-15T18:30:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:30:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:30:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:30:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:21 smithi122 ceph-mon[32255]: pgmap v957: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:21 smithi022 ceph-mon[31360]: pgmap v957: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:21 smithi022 ceph-mon[40250]: pgmap v957: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:22.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:30:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:30:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:22 smithi022 ceph-mon[31360]: pgmap v958: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:22 smithi022 ceph-mon[40250]: pgmap v958: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:23.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:23.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:23.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:22 smithi122 ceph-mon[32255]: pgmap v958: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/389712585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/389712585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/389712585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:25.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:24 smithi122 ceph-mon[32255]: pgmap v959: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:24 smithi022 ceph-mon[31360]: pgmap v959: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:24 smithi022 ceph-mon[40250]: pgmap v959: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:30:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:30:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:27.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:30:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:27 smithi022 ceph-mon[31360]: pgmap v960: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:27 smithi022 ceph-mon[40250]: pgmap v960: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:28.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:27 smithi122 ceph-mon[32255]: pgmap v960: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:28.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:28.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:28.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/400210582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/400210582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/400210582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:29 smithi022 ceph-mon[31360]: pgmap v961: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:29 smithi022 ceph-mon[40250]: pgmap v961: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:29 smithi122 ceph-mon[32255]: pgmap v961: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:30] "GET /metrics HTTP/1.1" 200 202234 "" "Prometheus/2.18.1" 2021-12-15T18:30:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:31 smithi022 ceph-mon[31360]: pgmap v962: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:31 smithi022 ceph-mon[40250]: pgmap v962: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:31 smithi122 ceph-mon[32255]: pgmap v962: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:30:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:30:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:30:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:30:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:30:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:30:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:30:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:30:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:30:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:30:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:30:33.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:33.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:33.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:33.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:33 smithi022 ceph-mon[31360]: pgmap v963: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:33 smithi022 ceph-mon[40250]: pgmap v963: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:33 smithi122 ceph-mon[32255]: pgmap v963: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1485464731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1485464731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1485464731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:30:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:30:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:35 smithi022 ceph-mon[31360]: pgmap v964: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:35 smithi022 ceph-mon[40250]: pgmap v964: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:35 smithi122 ceph-mon[32255]: pgmap v964: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:37.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:30:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:37 smithi022 ceph-mon[31360]: pgmap v965: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:37 smithi022 ceph-mon[40250]: pgmap v965: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:37 smithi122 ceph-mon[32255]: pgmap v965: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:38.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:39.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:39.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:39.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:39 smithi022 ceph-mon[31360]: pgmap v966: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1064272390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:39 smithi022 ceph-mon[40250]: pgmap v966: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1064272390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:39 smithi122 ceph-mon[32255]: pgmap v966: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1064272390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:40] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:30:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:41 smithi022 ceph-mon[40250]: pgmap v967: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:41 smithi022 ceph-mon[31360]: pgmap v967: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:41 smithi122 ceph-mon[32255]: pgmap v967: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:30:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:30:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:43 smithi022 ceph-mon[31360]: pgmap v968: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:43 smithi022 ceph-mon[40250]: pgmap v968: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:44.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:43 smithi122 ceph-mon[32255]: pgmap v968: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:44.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:44.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:44.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3323236503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3323236503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3323236503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:30:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:30:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:45 smithi022 ceph-mon[31360]: pgmap v969: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:45 smithi022 ceph-mon[40250]: pgmap v969: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:45 smithi122 ceph-mon[32255]: pgmap v969: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:30:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:47 smithi122 ceph-mon[32255]: pgmap v970: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:47 smithi022 ceph-mon[31360]: pgmap v970: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:47 smithi022 ceph-mon[40250]: pgmap v970: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:49.549 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:49.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:49.919 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:49 smithi022 ceph-mon[40250]: pgmap v971: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:49.920 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:49 smithi022 ceph-mon[31360]: pgmap v971: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:49 smithi122 ceph-mon[32255]: pgmap v971: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:50] "GET /metrics HTTP/1.1" 200 202226 "" "Prometheus/2.18.1" 2021-12-15T18:30:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/807354685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/807354685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/807354685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:51 smithi122 ceph-mon[32255]: pgmap v972: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:51 smithi022 ceph-mon[31360]: pgmap v972: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:51 smithi022 ceph-mon[40250]: pgmap v972: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:30:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:30:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:53 smithi122 ceph-mon[32255]: pgmap v973: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:53 smithi022 ceph-mon[31360]: pgmap v973: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:53 smithi022 ceph-mon[40250]: pgmap v973: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:54.920 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:30:55.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:30:55.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:30:55.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:30:55.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:30:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:30:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:55 smithi122 ceph-mon[32255]: pgmap v974: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/663862581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:55 smithi022 ceph-mon[40250]: pgmap v974: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/663862581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:55 smithi022 ceph-mon[31360]: pgmap v974: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:30:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/663862581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:30:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:30:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:30:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:57 smithi122 ceph-mon[32255]: pgmap v975: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:57 smithi022 ceph-mon[31360]: pgmap v975: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:30:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:57 smithi022 ceph-mon[40250]: pgmap v975: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:30:59 smithi122 ceph-mon[32255]: pgmap v976: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:30:59 smithi022 ceph-mon[31360]: pgmap v976: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:30:59 smithi022 ceph-mon[40250]: pgmap v976: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:00.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:00] "GET /metrics HTTP/1.1" 200 202226 "" "Prometheus/2.18.1" 2021-12-15T18:31:00.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:00.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:00.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3752179784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3752179784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3752179784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:01 smithi122 ceph-mon[32255]: pgmap v977: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:01 smithi022 ceph-mon[31360]: pgmap v977: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:01 smithi022 ceph-mon[40250]: pgmap v977: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:31:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:31:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:03 smithi122 ceph-mon[32255]: pgmap v978: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:03 smithi022 ceph-mon[31360]: pgmap v978: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:03 smithi022 ceph-mon[40250]: pgmap v978: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:05.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:05.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:31:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:31:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:05 smithi022 ceph-mon[31360]: pgmap v979: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:05 smithi022 ceph-mon[40250]: pgmap v979: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:06.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:05 smithi122 ceph-mon[32255]: pgmap v979: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4255180180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4255180180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4255180180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:31:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:07 smithi122 ceph-mon[32255]: pgmap v980: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:07 smithi022 ceph-mon[31360]: pgmap v980: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:07 smithi022 ceph-mon[40250]: pgmap v980: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:09 smithi122 ceph-mon[32255]: pgmap v981: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:09 smithi022 ceph-mon[31360]: pgmap v981: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:09 smithi022 ceph-mon[40250]: pgmap v981: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:10] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:31:11.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:11.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:11.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:11.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:11 smithi122 ceph-mon[32255]: pgmap v982: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3851006499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:11 smithi022 ceph-mon[31360]: pgmap v982: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3851006499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:11 smithi022 ceph-mon[40250]: pgmap v982: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3851006499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:31:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:31:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:13 smithi122 ceph-mon[32255]: pgmap v983: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:13 smithi022 ceph-mon[31360]: pgmap v983: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:13 smithi022 ceph-mon[40250]: pgmap v983: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:15.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:31:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:31:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:15 smithi122 ceph-mon[32255]: pgmap v984: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:15 smithi022 ceph-mon[31360]: pgmap v984: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:15 smithi022 ceph-mon[40250]: pgmap v984: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:16.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:16.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:16.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:16.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3286944138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3286944138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3286944138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:31:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:17 smithi122 ceph-mon[32255]: pgmap v985: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:17 smithi022 ceph-mon[31360]: pgmap v985: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:17 smithi022 ceph-mon[40250]: pgmap v985: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:19 smithi122 ceph-mon[32255]: pgmap v986: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:19 smithi022 ceph-mon[31360]: pgmap v986: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:19 smithi022 ceph-mon[40250]: pgmap v986: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:20] "GET /metrics HTTP/1.1" 200 202239 "" "Prometheus/2.18.1" 2021-12-15T18:31:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:31:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:31:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:31:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:31:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:31:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:31:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:31:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:31:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:31:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:31:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:31:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:31:21.769 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:21 smithi022 ceph-mon[31360]: pgmap v987: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:21 smithi022 ceph-mon[40250]: pgmap v987: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:22.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:22.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:22.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:21 smithi122 ceph-mon[32255]: pgmap v987: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:31:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:31:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1389615635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1389615635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1389615635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:23 smithi122 ceph-mon[32255]: pgmap v988: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:23 smithi022 ceph-mon[31360]: pgmap v988: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:23 smithi022 ceph-mon[40250]: pgmap v988: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:25.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:31:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:31:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:25 smithi122 ceph-mon[32255]: pgmap v989: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:25 smithi022 ceph-mon[31360]: pgmap v989: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:25 smithi022 ceph-mon[40250]: pgmap v989: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:27.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:27.141 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:27.142 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:27.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:27.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:27.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:31:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:27 smithi122 ceph-mon[32255]: pgmap v990: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3053285154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:27 smithi022 ceph-mon[31360]: pgmap v990: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3053285154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:27 smithi022 ceph-mon[40250]: pgmap v990: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3053285154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:29 smithi122 ceph-mon[32255]: pgmap v991: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:29 smithi022 ceph-mon[31360]: pgmap v991: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:29 smithi022 ceph-mon[40250]: pgmap v991: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:30] "GET /metrics HTTP/1.1" 200 202239 "" "Prometheus/2.18.1" 2021-12-15T18:31:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:31 smithi122 ceph-mon[32255]: pgmap v992: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:31 smithi022 ceph-mon[31360]: pgmap v992: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:31 smithi022 ceph-mon[40250]: pgmap v992: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:32.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:31:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:31:32.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:32.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:32.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:31:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:31:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:31:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:31:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:31:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:31:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:31:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:31:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:31:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:31:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:31:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:31:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:33 smithi122 ceph-mon[32255]: pgmap v993: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2059428752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:33 smithi022 ceph-mon[31360]: pgmap v993: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2059428752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:33 smithi022 ceph-mon[40250]: pgmap v993: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2059428752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:34 smithi122 ceph-mon[32255]: pgmap v994: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:34 smithi022 ceph-mon[31360]: pgmap v994: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:34 smithi022 ceph-mon[40250]: pgmap v994: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:31:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:31:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:37.750 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:31:37.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:37 smithi022 ceph-mon[31360]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:37 smithi022 ceph-mon[40250]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:37 smithi122 ceph-mon[32255]: pgmap v995: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:38.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:38.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:38.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2567125393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2567125393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2567125393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:39 smithi022 ceph-mon[31360]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:39 smithi022 ceph-mon[40250]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:39 smithi122 ceph-mon[32255]: pgmap v996: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:40] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:31:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:41 smithi022 ceph-mon[31360]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:41 smithi022 ceph-mon[40250]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:41 smithi122 ceph-mon[32255]: pgmap v997: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:31:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:31:43.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:43.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:43.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:43.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:43 smithi022 ceph-mon[31360]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2389234286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:43 smithi022 ceph-mon[40250]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2389234286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:43 smithi122 ceph-mon[32255]: pgmap v998: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2389234286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:31:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:31:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:45 smithi022 ceph-mon[31360]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:45 smithi022 ceph-mon[40250]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:45 smithi122 ceph-mon[32255]: pgmap v999: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:47.770 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:31:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:47 smithi022 ceph-mon[31360]: pgmap v1000: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:47 smithi022 ceph-mon[40250]: pgmap v1000: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:47 smithi122 ceph-mon[32255]: pgmap v1000: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:48.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:48.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:48.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:48.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:49 smithi022 ceph-mon[31360]: pgmap v1001: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/835597392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:49 smithi022 ceph-mon[40250]: pgmap v1001: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/835597392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:49 smithi122 ceph-mon[32255]: pgmap v1001: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/835597392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:50] "GET /metrics HTTP/1.1" 200 202235 "" "Prometheus/2.18.1" 2021-12-15T18:31:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:51 smithi022 ceph-mon[31360]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:51 smithi022 ceph-mon[40250]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:51 smithi122 ceph-mon[32255]: pgmap v1002: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:31:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:31:53.993 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:53 smithi022 ceph-mon[31360]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:53 smithi022 ceph-mon[40250]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:53 smithi122 ceph-mon[32255]: pgmap v1003: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:54.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:54.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:54.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:31:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1120745387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1120745387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1120745387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:31:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:31:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:31:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:55 smithi122 ceph-mon[32255]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:55 smithi022 ceph-mon[31360]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:55 smithi022 ceph-mon[40250]: pgmap v1004: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:31:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:31:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:31:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:57 smithi122 ceph-mon[32255]: pgmap v1005: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:57 smithi022 ceph-mon[31360]: pgmap v1005: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:57 smithi022 ceph-mon[40250]: pgmap v1005: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:31:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:31:59.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:31:59.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:31:59.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:59 smithi022 ceph-mon[31360]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:31:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3695373247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:59 smithi022 ceph-mon[40250]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:31:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3695373247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:59 smithi122 ceph-mon[32255]: pgmap v1006: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:31:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3695373247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:00] "GET /metrics HTTP/1.1" 200 202235 "" "Prometheus/2.18.1" 2021-12-15T18:32:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:01 smithi122 ceph-mon[32255]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:01 smithi022 ceph-mon[31360]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:01 smithi022 ceph-mon[40250]: pgmap v1007: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:32:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:32:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:03 smithi122 ceph-mon[32255]: pgmap v1008: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:03 smithi022 ceph-mon[31360]: pgmap v1008: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:03 smithi022 ceph-mon[40250]: pgmap v1008: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:04.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:05.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:05.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:05.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:05.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:32:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:32:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:05 smithi122 ceph-mon[32255]: pgmap v1009: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1769585374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:05 smithi022 ceph-mon[31360]: pgmap v1009: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1769585374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:05 smithi022 ceph-mon[40250]: pgmap v1009: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1769585374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:32:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:07 smithi122 ceph-mon[32255]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:07 smithi022 ceph-mon[31360]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:07 smithi022 ceph-mon[40250]: pgmap v1010: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:10.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:10.104 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:09 smithi022 ceph-mon[31360]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:10.105 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:09 smithi022 ceph-mon[40250]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:09 smithi122 ceph-mon[32255]: pgmap v1011: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:10] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:32:10.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:10.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:10.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3710887918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3710887918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3710887918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:11 smithi122 ceph-mon[32255]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:11 smithi022 ceph-mon[31360]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:11 smithi022 ceph-mon[40250]: pgmap v1012: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:32:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:32:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:13 smithi122 ceph-mon[32255]: pgmap v1013: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:13 smithi022 ceph-mon[31360]: pgmap v1013: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:13 smithi022 ceph-mon[40250]: pgmap v1013: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:15.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:15.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:32:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:32:15.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:15.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:15.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:15 smithi122 ceph-mon[32255]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:15 smithi022 ceph-mon[31360]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:15 smithi022 ceph-mon[40250]: pgmap v1014: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/422891342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/422891342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/422891342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:32:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:17 smithi122 ceph-mon[32255]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:17 smithi022 ceph-mon[31360]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:17 smithi022 ceph-mon[40250]: pgmap v1015: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:19 smithi122 ceph-mon[32255]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:19 smithi022 ceph-mon[31360]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:19 smithi022 ceph-mon[40250]: pgmap v1016: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:20] "GET /metrics HTTP/1.1" 200 202232 "" "Prometheus/2.18.1" 2021-12-15T18:32:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:32:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:32:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:32:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:32:21.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:21.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:21.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:21.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:32:21.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:32:21.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:32:21.205 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:32:21.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:32:21.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:32:21.207 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:32:21.207 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:32:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:21 smithi122 ceph-mon[32255]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3352452381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:21 smithi022 ceph-mon[40250]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3352452381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:21 smithi022 ceph-mon[31360]: pgmap v1017: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3352452381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:32:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:32:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:23 smithi122 ceph-mon[32255]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:23 smithi022 ceph-mon[40250]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:23 smithi022 ceph-mon[31360]: pgmap v1018: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:32:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:32:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:25 smithi122 ceph-mon[32255]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:26.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:25 smithi022 ceph-mon[31360]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:26.204 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:25 smithi022 ceph-mon[40250]: pgmap v1019: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:26.856 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:26.857 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1101463896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1101463896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1101463896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:32:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:27 smithi122 ceph-mon[32255]: pgmap v1020: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:27 smithi022 ceph-mon[31360]: pgmap v1020: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:27 smithi022 ceph-mon[40250]: pgmap v1020: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:29 smithi122 ceph-mon[32255]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:29 smithi022 ceph-mon[31360]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:29 smithi022 ceph-mon[40250]: pgmap v1021: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:30] "GET /metrics HTTP/1.1" 200 202232 "" "Prometheus/2.18.1" 2021-12-15T18:32:31.577 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:31.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:31.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:31.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:31 smithi122 ceph-mon[32255]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:31 smithi022 ceph-mon[31360]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:31 smithi022 ceph-mon[40250]: pgmap v1022: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:32:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:32:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1908660760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:32:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:32:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:32:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:32:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1908660760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:33.368 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:32:33.368 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:32:33.368 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:32:33.369 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:32:33.369 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1908660760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:33.369 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:32:33.370 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:32:33.370 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:32:33.370 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:32:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:33 smithi122 ceph-mon[32255]: pgmap v1023: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:33 smithi022 ceph-mon[31360]: pgmap v1023: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:33 smithi022 ceph-mon[40250]: pgmap v1023: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:35.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:32:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:32:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:35 smithi122 ceph-mon[32255]: pgmap v1024: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:35 smithi022 ceph-mon[31360]: pgmap v1024: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:35 smithi022 ceph-mon[40250]: pgmap v1024: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:36.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:37.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:37.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:37.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:32:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:37 smithi122 ceph-mon[32255]: pgmap v1025: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4186547330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:37 smithi022 ceph-mon[31360]: pgmap v1025: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4186547330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:37 smithi022 ceph-mon[40250]: pgmap v1025: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4186547330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:39 smithi122 ceph-mon[32255]: pgmap v1026: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:39 smithi022 ceph-mon[31360]: pgmap v1026: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:39 smithi022 ceph-mon[40250]: pgmap v1026: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:40] "GET /metrics HTTP/1.1" 200 202235 "" "Prometheus/2.18.1" 2021-12-15T18:32:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:41 smithi022 ceph-mon[31360]: pgmap v1027: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:41 smithi022 ceph-mon[40250]: pgmap v1027: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:41 smithi122 ceph-mon[32255]: pgmap v1027: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:42.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:32:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:32:42.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:42.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:42.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/98953690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/98953690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/98953690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:43 smithi022 ceph-mon[31360]: pgmap v1028: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:43 smithi022 ceph-mon[40250]: pgmap v1028: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:43 smithi122 ceph-mon[32255]: pgmap v1028: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:45.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:32:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:32:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:45 smithi022 ceph-mon[31360]: pgmap v1029: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:45 smithi022 ceph-mon[40250]: pgmap v1029: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:45 smithi122 ceph-mon[32255]: pgmap v1029: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:47.698 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:47.698 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:32:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:47 smithi022 ceph-mon[31360]: pgmap v1030: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:47 smithi022 ceph-mon[40250]: pgmap v1030: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:48.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:48.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:48.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:47 smithi122 ceph-mon[32255]: pgmap v1030: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4044287045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4044287045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4044287045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:49 smithi122 ceph-mon[32255]: pgmap v1031: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:49 smithi022 ceph-mon[31360]: pgmap v1031: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:49 smithi022 ceph-mon[40250]: pgmap v1031: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:50] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:32:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:51 smithi122 ceph-mon[32255]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:51 smithi022 ceph-mon[31360]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:51 smithi022 ceph-mon[40250]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:32:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:32:53.069 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:53.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:53.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:53.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:32:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:53 smithi122 ceph-mon[32255]: pgmap v1033: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3397287379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:53 smithi022 ceph-mon[31360]: pgmap v1033: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3397287379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:53 smithi022 ceph-mon[40250]: pgmap v1033: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3397287379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:32:55.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:32:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:32:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:55 smithi122 ceph-mon[32255]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:55 smithi022 ceph-mon[31360]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:55 smithi022 ceph-mon[40250]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:32:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:32:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:32:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:57 smithi122 ceph-mon[32255]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:57 smithi022 ceph-mon[31360]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:57 smithi022 ceph-mon[40250]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:32:58.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:32:58.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:32:58.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:32:58.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:59 smithi122 ceph-mon[32255]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:32:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3148169693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:59 smithi022 ceph-mon[31360]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:32:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3148169693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:59 smithi022 ceph-mon[40250]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:32:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3148169693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:00] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:33:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:01 smithi122 ceph-mon[32255]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:01 smithi022 ceph-mon[31360]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:01 smithi022 ceph-mon[40250]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:33:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:33:03.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:03 smithi122 ceph-mon[32255]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:04.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:04.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:04.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:04.190 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:03 smithi022 ceph-mon[40250]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:04.191 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:03 smithi022 ceph-mon[31360]: pgmap v1038: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3786734923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3786734923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3786734923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:05.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:33:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:33:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:05 smithi122 ceph-mon[32255]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:05 smithi022 ceph-mon[31360]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:05 smithi022 ceph-mon[40250]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:33:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:07 smithi122 ceph-mon[32255]: pgmap v1040: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:07 smithi022 ceph-mon[31360]: pgmap v1040: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:07 smithi022 ceph-mon[40250]: pgmap v1040: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:09.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:09.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:09.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:09.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:09 smithi122 ceph-mon[32255]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3228651637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:09 smithi022 ceph-mon[31360]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3228651637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:09 smithi022 ceph-mon[40250]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3228651637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:10] "GET /metrics HTTP/1.1" 200 202235 "" "Prometheus/2.18.1" 2021-12-15T18:33:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:11 smithi122 ceph-mon[32255]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:11 smithi022 ceph-mon[31360]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:11 smithi022 ceph-mon[40250]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:33:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:33:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:13 smithi122 ceph-mon[32255]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:13 smithi022 ceph-mon[40250]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:13 smithi022 ceph-mon[31360]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:14.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:14.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:14.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:15.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:33:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:33:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:15 smithi122 ceph-mon[32255]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4146027991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:15 smithi022 ceph-mon[31360]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4146027991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:15 smithi022 ceph-mon[40250]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4146027991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:33:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:17 smithi122 ceph-mon[32255]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:17 smithi022 ceph-mon[31360]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:17 smithi022 ceph-mon[40250]: pgmap v1045: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:19.935 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:19 smithi122 ceph-mon[32255]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:19 smithi022 ceph-mon[31360]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:19 smithi022 ceph-mon[40250]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:20.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:20.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:20.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:20] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:33:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3693509950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:33:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:33:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:33:21.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3693509950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:21.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:33:21.239 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:33:21.239 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:33:21.239 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3693509950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:21.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:33:21.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:33:21.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:33:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:21 smithi122 ceph-mon[32255]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:33:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:21 smithi022 ceph-mon[31360]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:33:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:21 smithi022 ceph-mon[40250]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:33:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:33:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:33:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:23 smithi122 ceph-mon[32255]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:23 smithi022 ceph-mon[31360]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:23 smithi022 ceph-mon[40250]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:25.307 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:25.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:25.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:25.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:33:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:33:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:25 smithi022 ceph-mon[40250]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/355485341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:25 smithi022 ceph-mon[31360]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/355485341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:25 smithi122 ceph-mon[32255]: pgmap v1049: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/355485341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:33:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:27 smithi122 ceph-mon[32255]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:27 smithi022 ceph-mon[31360]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:27 smithi022 ceph-mon[40250]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:29 smithi122 ceph-mon[32255]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:29 smithi022 ceph-mon[31360]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:29 smithi022 ceph-mon[40250]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:30] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:33:30.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:31.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:31.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:31.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:31 smithi122 ceph-mon[32255]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3891009997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:31 smithi022 ceph-mon[31360]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3891009997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:31 smithi022 ceph-mon[40250]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3891009997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:33:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:33:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:33:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:33:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:33:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:33:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:33:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:33:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:33:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:33:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:33:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:33:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:33:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:33:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:33 smithi122 ceph-mon[32255]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:33 smithi022 ceph-mon[31360]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:33 smithi022 ceph-mon[40250]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:33:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:33:36.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:35 smithi122 ceph-mon[32255]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:35 smithi022 ceph-mon[31360]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:35 smithi022 ceph-mon[40250]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/759596654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/759596654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/759596654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:33:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:37 smithi122 ceph-mon[32255]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:37 smithi022 ceph-mon[31360]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:37 smithi022 ceph-mon[40250]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:39 smithi122 ceph-mon[32255]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:39 smithi022 ceph-mon[31360]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:39 smithi022 ceph-mon[40250]: pgmap v1056: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:40] "GET /metrics HTTP/1.1" 200 202237 "" "Prometheus/2.18.1" 2021-12-15T18:33:41.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:41.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:41.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:41.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:41 smithi122 ceph-mon[32255]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/707782102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:41 smithi022 ceph-mon[31360]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/707782102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:41 smithi022 ceph-mon[40250]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/707782102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:33:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:33:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:43 smithi122 ceph-mon[32255]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:43 smithi022 ceph-mon[31360]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:43 smithi022 ceph-mon[40250]: pgmap v1058: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:33:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:33:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:45 smithi122 ceph-mon[32255]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:45 smithi022 ceph-mon[31360]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:45 smithi022 ceph-mon[40250]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:46.793 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:47.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:47.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:47.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:47.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:47.164 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:33:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:47 smithi122 ceph-mon[32255]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2653911264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:47 smithi022 ceph-mon[31360]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2653911264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:47 smithi022 ceph-mon[40250]: pgmap v1060: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2653911264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:48 smithi122 ceph-mon[32255]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:48 smithi022 ceph-mon[31360]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:48 smithi022 ceph-mon[40250]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:50] "GET /metrics HTTP/1.1" 200 202230 "" "Prometheus/2.18.1" 2021-12-15T18:33:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:51 smithi022 ceph-mon[31360]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:51 smithi022 ceph-mon[40250]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:52.163 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:51 smithi122 ceph-mon[32255]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:52.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:52.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:52.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:33:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:33:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3117567900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3117567900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3117567900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:53 smithi122 ceph-mon[32255]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:53 smithi022 ceph-mon[31360]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:53 smithi022 ceph-mon[40250]: pgmap v1063: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:33:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:33:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:55 smithi122 ceph-mon[32255]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:55 smithi022 ceph-mon[31360]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:55 smithi022 ceph-mon[40250]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:33:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:33:57.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:33:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:33:57.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:33:57.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:33:57.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:33:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:57 smithi122 ceph-mon[32255]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:57 smithi022 ceph-mon[31360]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:57 smithi022 ceph-mon[40250]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:33:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2402550681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2402550681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:33:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2402550681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:33:59 smithi122 ceph-mon[32255]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:33:59 smithi022 ceph-mon[31360]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:33:59 smithi022 ceph-mon[40250]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:00] "GET /metrics HTTP/1.1" 200 202230 "" "Prometheus/2.18.1" 2021-12-15T18:34:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:01 smithi122 ceph-mon[32255]: pgmap v1067: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:01 smithi022 ceph-mon[31360]: pgmap v1067: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:01 smithi022 ceph-mon[40250]: pgmap v1067: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:34:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:34:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:03.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:03.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:03.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:03 smithi122 ceph-mon[32255]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3492665653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:03 smithi022 ceph-mon[31360]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3492665653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:03 smithi022 ceph-mon[40250]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3492665653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:05.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:34:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:34:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:05 smithi122 ceph-mon[32255]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:05 smithi022 ceph-mon[31360]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:05 smithi022 ceph-mon[40250]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:34:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:07 smithi122 ceph-mon[32255]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:07 smithi022 ceph-mon[31360]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:07 smithi022 ceph-mon[40250]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:08.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:08.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:08.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:08.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2254326253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2254326253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2254326253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:09 smithi122 ceph-mon[32255]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:09 smithi022 ceph-mon[31360]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:09 smithi022 ceph-mon[40250]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:10] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:34:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:11 smithi122 ceph-mon[32255]: pgmap v1072: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:11 smithi022 ceph-mon[31360]: pgmap v1072: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:11 smithi022 ceph-mon[40250]: pgmap v1072: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:34:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:34:13.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:14.016 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:13 smithi022 ceph-mon[31360]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:14.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:13 smithi022 ceph-mon[40250]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:13 smithi122 ceph-mon[32255]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/213278029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/213278029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/213278029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:15.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:34:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:34:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:15 smithi122 ceph-mon[32255]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:15 smithi022 ceph-mon[31360]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:15 smithi022 ceph-mon[40250]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:34:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:17 smithi122 ceph-mon[32255]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:17 smithi022 ceph-mon[31360]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:17 smithi022 ceph-mon[40250]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:19.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:19.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:19.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:19 smithi122 ceph-mon[32255]: pgmap v1076: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/221502582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:19 smithi022 ceph-mon[31360]: pgmap v1076: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/221502582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:19 smithi022 ceph-mon[40250]: pgmap v1076: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/221502582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:20] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:34:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[31360]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:34:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:34:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:34:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:34:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[40250]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:34:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:34:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:34:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:34:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:21 smithi122 ceph-mon[32255]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:34:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:34:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:34:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:34:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:34:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:34:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:23 smithi122 ceph-mon[32255]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:23 smithi022 ceph-mon[31360]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:23 smithi022 ceph-mon[40250]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:24.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:24.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:24.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:24.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3159194325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3159194325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3159194325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:34:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:34:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:25 smithi122 ceph-mon[32255]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:25 smithi022 ceph-mon[31360]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:25 smithi022 ceph-mon[40250]: pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:34:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:27 smithi122 ceph-mon[32255]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:27 smithi022 ceph-mon[31360]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:27 smithi022 ceph-mon[40250]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:29.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:29 smithi022 ceph-mon[31360]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:29 smithi022 ceph-mon[40250]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:30.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:30.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:30.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:29 smithi122 ceph-mon[32255]: pgmap v1081: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:30] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:34:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/530746173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/530746173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/530746173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:31 smithi122 ceph-mon[32255]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:31 smithi022 ceph-mon[31360]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:31 smithi022 ceph-mon[40250]: pgmap v1082: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:34:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:34:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:34:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:34:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:34:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:34:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:34:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:34:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:34:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:34:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:34:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:34:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:34:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:34:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:33 smithi122 ceph-mon[32255]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:33 smithi022 ceph-mon[31360]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:33 smithi022 ceph-mon[40250]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:35.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:35.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:35.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:35.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:34:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:34:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:35 smithi122 ceph-mon[32255]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3487380252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:35 smithi022 ceph-mon[40250]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3487380252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:35 smithi022 ceph-mon[31360]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3487380252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:34:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:37 smithi122 ceph-mon[32255]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:37 smithi022 ceph-mon[31360]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:37 smithi022 ceph-mon[40250]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:39 smithi122 ceph-mon[32255]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:39 smithi022 ceph-mon[31360]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:39 smithi022 ceph-mon[40250]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:40] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:34:40.502 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:40.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:40.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:40.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:41 smithi122 ceph-mon[32255]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3680462315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:41 smithi022 ceph-mon[31360]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3680462315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:41 smithi022 ceph-mon[40250]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3680462315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:34:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:34:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:43 smithi122 ceph-mon[32255]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:43 smithi022 ceph-mon[31360]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:43 smithi022 ceph-mon[40250]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:34:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:34:45.874 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:45 smithi122 ceph-mon[32255]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:46.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:46.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:46.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:46.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:45 smithi022 ceph-mon[31360]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:46.245 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:45 smithi022 ceph-mon[40250]: pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1869607012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1869607012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1869607012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:34:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:47 smithi122 ceph-mon[32255]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:47 smithi022 ceph-mon[31360]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:47 smithi022 ceph-mon[40250]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:49 smithi122 ceph-mon[32255]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:49 smithi022 ceph-mon[31360]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:49 smithi022 ceph-mon[40250]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:50] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:34:51.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:51.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:51.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:51.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:51 smithi022 ceph-mon[31360]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1312178798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:51 smithi022 ceph-mon[40250]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1312178798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:51 smithi122 ceph-mon[32255]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1312178798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:34:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:34:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:53 smithi122 ceph-mon[32255]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:53 smithi022 ceph-mon[31360]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:53 smithi022 ceph-mon[40250]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:55.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:34:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:34:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:55 smithi122 ceph-mon[32255]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:55 smithi022 ceph-mon[31360]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:55 smithi022 ceph-mon[40250]: pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:34:56.613 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:34:56.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:34:56.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:34:56.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:34:56.984 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:56.984 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:34:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:34:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:57 smithi122 ceph-mon[32255]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1629253185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:57 smithi022 ceph-mon[31360]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1629253185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:34:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:57 smithi022 ceph-mon[40250]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:34:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1629253185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:34:59 smithi122 ceph-mon[32255]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:34:59 smithi022 ceph-mon[31360]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:34:59 smithi022 ceph-mon[40250]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:00] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:35:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:02.091 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:01 smithi122 ceph-mon[32255]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:01 smithi022 ceph-mon[31360]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:01 smithi022 ceph-mon[40250]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:02.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:02.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:02.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:35:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:35:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1914545205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1914545205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1914545205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:03 smithi122 ceph-mon[32255]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:03 smithi022 ceph-mon[31360]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:03 smithi022 ceph-mon[40250]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:05.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:35:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:35:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:05 smithi122 ceph-mon[32255]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:05 smithi022 ceph-mon[31360]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:05 smithi022 ceph-mon[40250]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:07.351 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:07.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:07.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:07.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:07.723 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:35:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:07 smithi022 ceph-mon[31360]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4265411488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:07 smithi022 ceph-mon[40250]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4265411488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:07 smithi122 ceph-mon[32255]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4265411488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:09 smithi122 ceph-mon[32255]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:09 smithi022 ceph-mon[31360]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:09 smithi022 ceph-mon[40250]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:10] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:35:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:11 smithi122 ceph-mon[32255]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:11 smithi022 ceph-mon[31360]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:11 smithi022 ceph-mon[40250]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:35:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:35:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:13.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:13.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:13.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:13 smithi122 ceph-mon[32255]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1226998020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:13 smithi022 ceph-mon[31360]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1226998020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:13 smithi022 ceph-mon[40250]: pgmap v1103: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1226998020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:15.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:35:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:35:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:15 smithi122 ceph-mon[32255]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:15 smithi022 ceph-mon[31360]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:15 smithi022 ceph-mon[40250]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:35:18.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:17 smithi122 ceph-mon[32255]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:17 smithi022 ceph-mon[31360]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:17 smithi022 ceph-mon[40250]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:18.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1940784821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1940784821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1940784821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:19 smithi122 ceph-mon[32255]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:19 smithi022 ceph-mon[31360]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:19 smithi022 ceph-mon[40250]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:20] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:35:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[31360]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:35:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:35:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:35:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[40250]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:35:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:35:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:35:22.126 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:21 smithi122 ceph-mon[32255]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:22.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:35:22.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:35:22.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:35:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:35:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:35:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:23 smithi122 ceph-mon[32255]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:23.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[31360]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:23.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:35:23.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:23 smithi022 ceph-mon[40250]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:23.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:23.838 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:23.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:23.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/655631978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/655631978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/655631978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:25 smithi122 ceph-mon[32255]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:25.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:35:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:35:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:25 smithi022 ceph-mon[31360]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:25 smithi022 ceph-mon[40250]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:35:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:27 smithi122 ceph-mon[32255]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:27 smithi022 ceph-mon[31360]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:27 smithi022 ceph-mon[40250]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:28.840 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:29.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:29.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:29.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:29 smithi122 ceph-mon[32255]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/734052095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:29 smithi022 ceph-mon[31360]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/734052095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:29 smithi022 ceph-mon[40250]: pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/734052095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:30] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:35:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:31 smithi122 ceph-mon[32255]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:31 smithi022 ceph-mon[31360]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:31 smithi022 ceph-mon[40250]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:35:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:35:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:35:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:35:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:35:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:35:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:35:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:35:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:35:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:35:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:35:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:35:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:35:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:35:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:33 smithi122 ceph-mon[32255]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:34.209 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:34.210 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:33 smithi022 ceph-mon[31360]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:34.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:33 smithi022 ceph-mon[40250]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:34.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:34.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:34.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3490643560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3490643560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3490643560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:35:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:35:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:35 smithi122 ceph-mon[32255]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:35 smithi022 ceph-mon[31360]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:35 smithi022 ceph-mon[40250]: pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:35:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:37 smithi122 ceph-mon[32255]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:37 smithi022 ceph-mon[31360]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:37 smithi022 ceph-mon[40250]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:39.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:39.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:39.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:39.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:39.955 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:39 smithi022 ceph-mon[31360]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:39.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:39 smithi022 ceph-mon[40250]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:39 smithi122 ceph-mon[32255]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:40] "GET /metrics HTTP/1.1" 200 202246 "" "Prometheus/2.18.1" 2021-12-15T18:35:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/704892198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/704892198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/704892198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:41 smithi122 ceph-mon[32255]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:41 smithi022 ceph-mon[31360]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:41 smithi022 ceph-mon[40250]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:35:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:35:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:43 smithi122 ceph-mon[32255]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:43 smithi022 ceph-mon[31360]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:43 smithi022 ceph-mon[40250]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:35:44.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:45.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:45.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:45.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:35:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:35:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:45 smithi122 ceph-mon[32255]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1554562160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:45 smithi022 ceph-mon[31360]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1554562160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:45 smithi022 ceph-mon[40250]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1554562160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:35:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:47 smithi122 ceph-mon[32255]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:47 smithi022 ceph-mon[40250]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:47 smithi022 ceph-mon[31360]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:49 smithi122 ceph-mon[32255]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:49 smithi022 ceph-mon[31360]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:49 smithi022 ceph-mon[40250]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:50.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:50] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:35:50.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:50.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:50.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2080523039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2080523039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2080523039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:51 smithi122 ceph-mon[32255]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:51 smithi022 ceph-mon[31360]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:51 smithi022 ceph-mon[40250]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:35:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:35:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:53 smithi122 ceph-mon[32255]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:53 smithi022 ceph-mon[31360]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:53 smithi022 ceph-mon[40250]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:55.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:35:55.857 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:35:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:35:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:55 smithi022 ceph-mon[31360]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:55 smithi022 ceph-mon[40250]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:56.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:35:56.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:35:56.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:35:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:55 smithi122 ceph-mon[32255]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:35:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2547117122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2547117122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:35:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2547117122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:35:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:35:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:57 smithi122 ceph-mon[32255]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:57 smithi022 ceph-mon[31360]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:35:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:57 smithi022 ceph-mon[40250]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:35:59 smithi122 ceph-mon[32255]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:35:59 smithi022 ceph-mon[31360]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:35:59 smithi022 ceph-mon[40250]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:00] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:36:01.065 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:01.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:01.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:01.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:01 smithi122 ceph-mon[32255]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/923634053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:01 smithi022 ceph-mon[40250]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/923634053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:01 smithi022 ceph-mon[31360]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/923634053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:36:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:36:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:03 smithi122 ceph-mon[32255]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:03 smithi022 ceph-mon[31360]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:03 smithi022 ceph-mon[40250]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:05.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:36:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:36:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:05 smithi122 ceph-mon[32255]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:05 smithi022 ceph-mon[31360]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:05 smithi022 ceph-mon[40250]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:06.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:06.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:06.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:06.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1105223454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1105223454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1105223454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:36:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:07 smithi122 ceph-mon[32255]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:07 smithi022 ceph-mon[31360]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:07 smithi022 ceph-mon[40250]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:09 smithi122 ceph-mon[32255]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:09 smithi022 ceph-mon[31360]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:09 smithi022 ceph-mon[40250]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:10] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:36:11.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:12.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:12.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:12.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:12.174 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:11 smithi022 ceph-mon[31360]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:12.174 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:11 smithi022 ceph-mon[40250]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:11 smithi122 ceph-mon[32255]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:36:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:36:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/487933502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/487933502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/487933502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:13 smithi122 ceph-mon[32255]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:13 smithi022 ceph-mon[31360]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:13 smithi022 ceph-mon[40250]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:15.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:36:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:36:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:15 smithi122 ceph-mon[32255]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:15 smithi022 ceph-mon[31360]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:15 smithi022 ceph-mon[40250]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:17.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:17.175 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:17.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:36:17.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:17.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:17.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:17 smithi122 ceph-mon[32255]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2529491306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:17 smithi022 ceph-mon[31360]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2529491306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:17 smithi022 ceph-mon[40250]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2529491306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:19 smithi122 ceph-mon[32255]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:19 smithi022 ceph-mon[31360]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:19 smithi022 ceph-mon[40250]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:20] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:36:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:21 smithi122 ceph-mon[32255]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:21 smithi022 ceph-mon[31360]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:21 smithi022 ceph-mon[40250]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:22.612 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:36:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:36:22.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:22.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:22.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:36:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:36:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:36:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:36:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:36:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:36:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:36:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:36:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:36:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:23 smithi122 ceph-mon[32255]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/64394346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:36:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:23 smithi022 ceph-mon[31360]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/64394346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:36:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:23 smithi022 ceph-mon[40250]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/64394346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:36:25.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:36:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:36:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:25 smithi122 ceph-mon[32255]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:25 smithi022 ceph-mon[31360]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:25 smithi022 ceph-mon[40250]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:36:27.933 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:27 smithi122 ceph-mon[32255]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:27 smithi022 ceph-mon[31360]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:27 smithi022 ceph-mon[40250]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3113912706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3113912706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3113912706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:30.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:30] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:36:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:29 smithi122 ceph-mon[32255]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:29 smithi022 ceph-mon[40250]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:29 smithi022 ceph-mon[31360]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:31 smithi122 ceph-mon[32255]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:31 smithi022 ceph-mon[31360]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:31 smithi022 ceph-mon[40250]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:36:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:36:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:36:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:36:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:36:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:36:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:36:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:36:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:36:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:36:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:36:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:36:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:36:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:36:33.304 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:33.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:33 smithi022 ceph-mon[31360]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1205890192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:33 smithi022 ceph-mon[40250]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1205890192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:33 smithi122 ceph-mon[32255]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1205890192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:35.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:36:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:36:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:35 smithi122 ceph-mon[32255]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:35 smithi022 ceph-mon[31360]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:35 smithi022 ceph-mon[40250]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:36:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:37 smithi122 ceph-mon[32255]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:37 smithi022 ceph-mon[31360]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:37 smithi022 ceph-mon[40250]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:38.673 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:39.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:39.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:39.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:39 smithi122 ceph-mon[32255]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3217580970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:39 smithi022 ceph-mon[31360]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3217580970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:39 smithi022 ceph-mon[40250]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3217580970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:40] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:36:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:41 smithi122 ceph-mon[32255]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:41 smithi022 ceph-mon[31360]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:41 smithi022 ceph-mon[40250]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:36:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:36:44.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:43 smithi122 ceph-mon[32255]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:43 smithi022 ceph-mon[31360]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:43 smithi022 ceph-mon[40250]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:44.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:44.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:44.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2390923069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2390923069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2390923069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:45.846 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:36:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:36:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:45 smithi122 ceph-mon[32255]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:45 smithi022 ceph-mon[31360]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:45 smithi022 ceph-mon[40250]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:36:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:47 smithi122 ceph-mon[32255]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:47 smithi022 ceph-mon[31360]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:47 smithi022 ceph-mon[40250]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:49.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:49.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:49.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:49.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:49 smithi122 ceph-mon[32255]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1421056951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:49 smithi022 ceph-mon[31360]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1421056951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:49 smithi022 ceph-mon[40250]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1421056951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:50] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:36:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:51 smithi122 ceph-mon[32255]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:51 smithi022 ceph-mon[31360]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:51 smithi022 ceph-mon[40250]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:36:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:36:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:53 smithi122 ceph-mon[32255]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:53 smithi022 ceph-mon[31360]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:53 smithi022 ceph-mon[40250]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:36:55.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:36:55.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:36:55.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:36:55.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:36:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:36:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:55 smithi122 ceph-mon[32255]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/370044306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:55 smithi022 ceph-mon[31360]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/370044306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:55 smithi022 ceph-mon[40250]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:36:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/370044306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:36:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:36:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:36:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:57 smithi122 ceph-mon[32255]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:57 smithi022 ceph-mon[31360]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:36:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:57 smithi022 ceph-mon[40250]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:00.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:00.165 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:36:59 smithi022 ceph-mon[40250]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:00.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:36:59 smithi022 ceph-mon[31360]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:36:59 smithi122 ceph-mon[32255]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:00] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:37:00.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:00.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:00.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3089529980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3089529980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3089529980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:01 smithi122 ceph-mon[32255]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:01 smithi022 ceph-mon[31360]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:01 smithi022 ceph-mon[40250]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:37:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:37:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:03 smithi122 ceph-mon[32255]: pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:03 smithi022 ceph-mon[31360]: pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:03 smithi022 ceph-mon[40250]: pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:05.536 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:05.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:37:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:37:05.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:05.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:05.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:05 smithi122 ceph-mon[32255]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:05 smithi022 ceph-mon[31360]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:05 smithi022 ceph-mon[40250]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/993134543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/993134543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/993134543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:37:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:07 smithi122 ceph-mon[32255]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:07 smithi022 ceph-mon[31360]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:07 smithi022 ceph-mon[40250]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:09 smithi122 ceph-mon[32255]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:09 smithi022 ceph-mon[31360]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:09 smithi022 ceph-mon[40250]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:10] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:37:10.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:11.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:11.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:11.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:11 smithi122 ceph-mon[32255]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2256403881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:11 smithi022 ceph-mon[31360]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2256403881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:11 smithi022 ceph-mon[40250]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2256403881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:37:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:37:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:13 smithi122 ceph-mon[32255]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:13 smithi022 ceph-mon[31360]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:13 smithi022 ceph-mon[40250]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:15.899 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:37:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:37:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:15 smithi122 ceph-mon[32255]: pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:15 smithi022 ceph-mon[31360]: pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:15 smithi022 ceph-mon[40250]: pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:16.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:16.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:16.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:16.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:16.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:16.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/503491166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/503491166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/503491166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:37:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:17 smithi122 ceph-mon[32255]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:17 smithi022 ceph-mon[31360]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:17 smithi022 ceph-mon[40250]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:19 smithi122 ceph-mon[32255]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:19 smithi022 ceph-mon[31360]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:19 smithi022 ceph-mon[40250]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:20] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:37:21.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:22.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:22.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:22.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:21 smithi122 ceph-mon[32255]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:21 smithi022 ceph-mon[31360]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:21 smithi022 ceph-mon[40250]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:37:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:37:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3577291445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3577291445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3577291445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:23 smithi122 ceph-mon[32255]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:37:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:37:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:37:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:37:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[31360]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:37:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:37:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:37:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:37:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[40250]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:37:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:37:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:37:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:37:25.919 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:37:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:37:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:25 smithi122 ceph-mon[32255]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:25 smithi022 ceph-mon[31360]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:25 smithi022 ceph-mon[40250]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:27.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:27.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:27.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:27.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:37:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:27 smithi122 ceph-mon[32255]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2426137525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:27 smithi022 ceph-mon[31360]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2426137525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:27 smithi022 ceph-mon[40250]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2426137525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:29 smithi122 ceph-mon[32255]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:30.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:30] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:37:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:29 smithi022 ceph-mon[31360]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:29 smithi022 ceph-mon[40250]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:31 smithi122 ceph-mon[32255]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:31 smithi022 ceph-mon[31360]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:31 smithi022 ceph-mon[40250]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:32.393 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:37:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:37:32.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:32.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:32.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:37:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:37:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:37:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:37:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2317783378' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:37:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:37:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:37:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:37:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2317783378' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:37:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:37:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:37:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:37:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2317783378' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:33 smithi122 ceph-mon[32255]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:33 smithi022 ceph-mon[31360]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:33 smithi022 ceph-mon[40250]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:34 smithi022 ceph-mon[31360]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:34 smithi022 ceph-mon[40250]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:34 smithi122 ceph-mon[32255]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:35.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:37:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:37:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:37.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:37.765 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:37:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:37 smithi022 ceph-mon[31360]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:37 smithi022 ceph-mon[40250]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:38.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:38.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:38.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:37 smithi122 ceph-mon[32255]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/394717032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/394717032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/394717032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:39 smithi122 ceph-mon[32255]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:39 smithi022 ceph-mon[31360]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:39 smithi022 ceph-mon[40250]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:40] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:37:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:41 smithi122 ceph-mon[32255]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:41 smithi022 ceph-mon[31360]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:41 smithi022 ceph-mon[40250]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:37:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:37:43.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:43.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:43.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:43.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:43 smithi122 ceph-mon[32255]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/350968958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:43 smithi022 ceph-mon[31360]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/350968958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:43 smithi022 ceph-mon[40250]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/350968958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:45.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:37:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:37:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:45 smithi122 ceph-mon[32255]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:45 smithi022 ceph-mon[31360]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:45 smithi022 ceph-mon[40250]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:37:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:47 smithi122 ceph-mon[32255]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:47 smithi022 ceph-mon[31360]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:47 smithi022 ceph-mon[40250]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:48.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:48.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:48.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:48.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:49 smithi122 ceph-mon[32255]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1553392776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:49 smithi022 ceph-mon[31360]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1553392776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:49 smithi022 ceph-mon[40250]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:37:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1553392776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:50] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:37:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:51 smithi122 ceph-mon[32255]: pgmap v1182: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:51 smithi022 ceph-mon[31360]: pgmap v1182: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:51 smithi022 ceph-mon[40250]: pgmap v1182: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:37:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:37:53.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:53 smithi122 ceph-mon[32255]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:53 smithi022 ceph-mon[31360]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:53 smithi022 ceph-mon[40250]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:37:54.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:54.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:54.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:37:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3066091313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3066091313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3066091313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:37:55.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:37:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:37:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:55 smithi122 ceph-mon[32255]: pgmap v1184: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:37:56.521 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:55 smithi022 ceph-mon[31360]: pgmap v1184: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:37:56.522 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:55 smithi022 ceph-mon[40250]: pgmap v1184: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:37:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:37:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:37:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:57 smithi122 ceph-mon[32255]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:37:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:57 smithi022 ceph-mon[31360]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:37:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:57 smithi022 ceph-mon[40250]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:37:59.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:37:59.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:37:59.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:37:59.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:59 smithi022 ceph-mon[31360]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:37:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4127658635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:59 smithi022 ceph-mon[40250]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:37:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4127658635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:59 smithi122 ceph-mon[32255]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:37:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4127658635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:00] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:38:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:01 smithi122 ceph-mon[32255]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:01 smithi022 ceph-mon[31360]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:01 smithi022 ceph-mon[40250]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:38:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:38:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:03 smithi122 ceph-mon[32255]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:03 smithi022 ceph-mon[31360]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:03 smithi022 ceph-mon[40250]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:04.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:05.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:05.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:05.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:05.885 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:38:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:38:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:05 smithi122 ceph-mon[32255]: pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1050124039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:05 smithi022 ceph-mon[31360]: pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1050124039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:05 smithi022 ceph-mon[40250]: pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:38:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1050124039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:38:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:07 smithi122 ceph-mon[32255]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:38:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:07 smithi022 ceph-mon[31360]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:38:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:07 smithi022 ceph-mon[40250]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:38:10.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:09 smithi122 ceph-mon[32255]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:38:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:09 smithi022 ceph-mon[31360]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:38:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:09 smithi022 ceph-mon[40250]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:38:10.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:10.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:10.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:10] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:38:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/780670461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/780670461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/780670461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:11 smithi122 ceph-mon[32255]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:11 smithi022 ceph-mon[31360]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:11 smithi022 ceph-mon[40250]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:38:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:38:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:13 smithi122 ceph-mon[32255]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:13 smithi022 ceph-mon[31360]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:13 smithi022 ceph-mon[40250]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:15.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:15.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:15.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:15.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:15.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:38:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:38:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:15 smithi122 ceph-mon[32255]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3307096822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:15 smithi022 ceph-mon[31360]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3307096822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:15 smithi022 ceph-mon[40250]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3307096822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:38:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:17 smithi122 ceph-mon[32255]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:17 smithi022 ceph-mon[31360]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:17 smithi022 ceph-mon[40250]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:19 smithi122 ceph-mon[32255]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:19 smithi022 ceph-mon[31360]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:19 smithi022 ceph-mon[40250]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:20] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:38:20.782 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:21.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:21.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:21.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:21 smithi122 ceph-mon[32255]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2201073913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:21 smithi022 ceph-mon[31360]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2201073913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:21 smithi022 ceph-mon[40250]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2201073913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:38:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:38:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:23 smithi122 ceph-mon[32255]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:38:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:38:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:38:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[31360]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:38:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:38:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:38:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[40250]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:38:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:38:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:38:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:38:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:24 smithi022 ceph-mon[31360]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:38:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:24 smithi022 ceph-mon[40250]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:38:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:24 smithi122 ceph-mon[32255]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:25.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:38:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:38:26.151 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:26.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:26.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:26.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1756933397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1756933397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1756933397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:38:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:27 smithi122 ceph-mon[32255]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:27 smithi022 ceph-mon[31360]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:27 smithi022 ceph-mon[40250]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:29 smithi122 ceph-mon[32255]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:29 smithi022 ceph-mon[31360]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:29 smithi022 ceph-mon[40250]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:30] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:38:31.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:31.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:31 smithi122 ceph-mon[32255]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:31 smithi022 ceph-mon[31360]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:31 smithi022 ceph-mon[40250]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:38:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:38:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2386806125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:38:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:38:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:38:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:38:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2386806125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:38:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:38:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:38:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:38:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2386806125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:38:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:38:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:38:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:38:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:33 smithi122 ceph-mon[32255]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:33 smithi022 ceph-mon[31360]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:33 smithi022 ceph-mon[40250]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:35.860 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:38:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:38:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:35 smithi122 ceph-mon[32255]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:35 smithi022 ceph-mon[31360]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:35 smithi022 ceph-mon[40250]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:36.893 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:37.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:37.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:37.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:37.261 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:38:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:37 smithi122 ceph-mon[32255]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3945435133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:37 smithi022 ceph-mon[31360]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3945435133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:37 smithi022 ceph-mon[40250]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3945435133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:39 smithi122 ceph-mon[32255]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:39 smithi022 ceph-mon[31360]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:39 smithi022 ceph-mon[40250]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:40] "GET /metrics HTTP/1.1" 200 202246 "" "Prometheus/2.18.1" 2021-12-15T18:38:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:41 smithi122 ceph-mon[32255]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:42.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:41 smithi022 ceph-mon[31360]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:42.262 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:41 smithi022 ceph-mon[40250]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:42.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:42.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:42.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:42.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:38:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:38:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1808258997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1808258997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1808258997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:43 smithi122 ceph-mon[32255]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:43 smithi022 ceph-mon[31360]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:43 smithi022 ceph-mon[40250]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:45.880 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:38:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:38:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:45 smithi122 ceph-mon[32255]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:45 smithi022 ceph-mon[31360]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:45 smithi022 ceph-mon[40250]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:47.635 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:38:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:47 smithi122 ceph-mon[32255]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:47 smithi022 ceph-mon[31360]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:47 smithi022 ceph-mon[40250]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1139962731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1139962731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1139962731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:49 smithi122 ceph-mon[32255]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:49 smithi022 ceph-mon[31360]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:49 smithi022 ceph-mon[40250]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:50] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:38:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:51 smithi122 ceph-mon[32255]: pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:51 smithi022 ceph-mon[31360]: pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:51 smithi022 ceph-mon[40250]: pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:38:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:38:53.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:53.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:53.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:53.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:53 smithi122 ceph-mon[32255]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2485756414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:53 smithi022 ceph-mon[31360]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2485756414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:53 smithi022 ceph-mon[40250]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2485756414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:55.900 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:38:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:38:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:55 smithi122 ceph-mon[32255]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:55 smithi022 ceph-mon[31360]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:55 smithi022 ceph-mon[40250]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:38:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:38:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:38:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:57 smithi122 ceph-mon[32255]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:57 smithi022 ceph-mon[31360]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:57 smithi022 ceph-mon[40250]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:38:58.382 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:38:58.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:38:58.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:38:58.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:38:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1741160812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1741160812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:38:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1741160812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:38:59 smithi122 ceph-mon[32255]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:38:59 smithi022 ceph-mon[31360]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:38:59 smithi022 ceph-mon[40250]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:00] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:39:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:01 smithi122 ceph-mon[32255]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:01 smithi022 ceph-mon[31360]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:01 smithi022 ceph-mon[40250]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:39:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:39:03.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:03 smithi022 ceph-mon[40250]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:03 smithi022 ceph-mon[31360]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:04.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:04.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:04.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:03 smithi122 ceph-mon[32255]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1545062673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1545062673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1545062673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:05.923 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:39:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:39:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:05 smithi122 ceph-mon[32255]: pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:05 smithi022 ceph-mon[31360]: pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:05 smithi022 ceph-mon[40250]: pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:39:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:07 smithi122 ceph-mon[32255]: pgmap v1220: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:07 smithi022 ceph-mon[31360]: pgmap v1220: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:07 smithi022 ceph-mon[40250]: pgmap v1220: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:09.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:09.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:09.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:09.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:10.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:10] "GET /metrics HTTP/1.1" 200 202246 "" "Prometheus/2.18.1" 2021-12-15T18:39:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:09 smithi122 ceph-mon[32255]: pgmap v1221: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1436811684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:09 smithi022 ceph-mon[31360]: pgmap v1221: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1436811684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:09 smithi022 ceph-mon[40250]: pgmap v1221: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1436811684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:11 smithi122 ceph-mon[32255]: pgmap v1222: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:11 smithi022 ceph-mon[31360]: pgmap v1222: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:11 smithi022 ceph-mon[40250]: pgmap v1222: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:39:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:39:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:13 smithi022 ceph-mon[31360]: pgmap v1223: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:13 smithi022 ceph-mon[40250]: pgmap v1223: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:13 smithi122 ceph-mon[32255]: pgmap v1223: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:14.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:14.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:14.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:14.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1809003387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1809003387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1809003387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:39:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:39:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:15 smithi022 ceph-mon[31360]: pgmap v1224: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:15 smithi022 ceph-mon[40250]: pgmap v1224: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:15 smithi122 ceph-mon[32255]: pgmap v1224: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:39:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:17 smithi022 ceph-mon[31360]: pgmap v1225: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:17 smithi022 ceph-mon[40250]: pgmap v1225: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:17 smithi122 ceph-mon[32255]: pgmap v1225: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:19 smithi022 ceph-mon[31360]: pgmap v1226: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:19 smithi022 ceph-mon[40250]: pgmap v1226: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:19 smithi122 ceph-mon[32255]: pgmap v1226: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:19.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:20.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:20.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:20.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2067374156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:20] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:39:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2067374156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2067374156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:21 smithi122 ceph-mon[32255]: pgmap v1227: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:21 smithi022 ceph-mon[31360]: pgmap v1227: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:21 smithi022 ceph-mon[40250]: pgmap v1227: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:39:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:39:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:23 smithi122 ceph-mon[32255]: pgmap v1228: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:23 smithi022 ceph-mon[31360]: pgmap v1228: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:23 smithi022 ceph-mon[40250]: pgmap v1228: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:39:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:39:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:39:25.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:39:25.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:39:25.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:39:25.242 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:39:25.242 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:39:25.242 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:39:25.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:25 smithi122 ceph-mon[32255]: pgmap v1229: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4237637235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:25.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:39:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:39:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:25 smithi022 ceph-mon[31360]: pgmap v1229: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4237637235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:25 smithi022 ceph-mon[40250]: pgmap v1229: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4237637235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:39:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:27 smithi122 ceph-mon[32255]: pgmap v1230: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:27 smithi022 ceph-mon[31360]: pgmap v1230: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:27 smithi022 ceph-mon[40250]: pgmap v1230: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:29 smithi122 ceph-mon[32255]: pgmap v1231: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:29 smithi022 ceph-mon[31360]: pgmap v1231: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:29 smithi022 ceph-mon[40250]: pgmap v1231: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:39:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:30] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:39:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:31 smithi122 ceph-mon[32255]: pgmap v1232: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1700049130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:31 smithi022 ceph-mon[31360]: pgmap v1232: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1700049130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:31 smithi022 ceph-mon[40250]: pgmap v1232: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1700049130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:39:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:39:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:39:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:39:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:39:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:39:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:39:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:39:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:39:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:39:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:39:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:39:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:39:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:39:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:33 smithi122 ceph-mon[32255]: pgmap v1233: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:33 smithi022 ceph-mon[31360]: pgmap v1233: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:33 smithi022 ceph-mon[40250]: pgmap v1233: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:35 smithi122 ceph-mon[32255]: pgmap v1234: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:35.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:39:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:39:35.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:35 smithi022 ceph-mon[31360]: pgmap v1234: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:35 smithi022 ceph-mon[40250]: pgmap v1234: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:36.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:36.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:36.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1426620420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1426620420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1426620420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:39:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:37 smithi122 ceph-mon[32255]: pgmap v1235: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:37 smithi022 ceph-mon[31360]: pgmap v1235: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:37 smithi022 ceph-mon[40250]: pgmap v1235: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:39 smithi122 ceph-mon[32255]: pgmap v1236: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:39 smithi022 ceph-mon[31360]: pgmap v1236: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:39 smithi022 ceph-mon[40250]: pgmap v1236: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:40] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:39:41.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:41.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:41 smithi022 ceph-mon[31360]: pgmap v1237: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2165738657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:41 smithi022 ceph-mon[40250]: pgmap v1237: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2165738657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:41 smithi122 ceph-mon[32255]: pgmap v1237: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2165738657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:39:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:39:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:43 smithi122 ceph-mon[32255]: pgmap v1238: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:43 smithi022 ceph-mon[31360]: pgmap v1238: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:43 smithi022 ceph-mon[40250]: pgmap v1238: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:45.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:39:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:39:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:45 smithi122 ceph-mon[32255]: pgmap v1239: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:45 smithi022 ceph-mon[31360]: pgmap v1239: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:45 smithi022 ceph-mon[40250]: pgmap v1239: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:47.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:47.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:47.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:39:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:47 smithi122 ceph-mon[32255]: pgmap v1240: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2450200100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:47 smithi022 ceph-mon[31360]: pgmap v1240: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2450200100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:47 smithi022 ceph-mon[40250]: pgmap v1240: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2450200100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:49 smithi122 ceph-mon[32255]: pgmap v1241: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:49 smithi022 ceph-mon[31360]: pgmap v1241: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:49 smithi022 ceph-mon[40250]: pgmap v1241: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:50] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:39:52.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:51 smithi122 ceph-mon[32255]: pgmap v1242: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:51 smithi022 ceph-mon[31360]: pgmap v1242: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:51 smithi022 ceph-mon[40250]: pgmap v1242: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:52.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:52.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:52.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:39:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:39:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3714212070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3714212070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3714212070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:53 smithi122 ceph-mon[32255]: pgmap v1243: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:53 smithi022 ceph-mon[31360]: pgmap v1243: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:53 smithi022 ceph-mon[40250]: pgmap v1243: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:55.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:39:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:39:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:55 smithi122 ceph-mon[32255]: pgmap v1244: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:55 smithi022 ceph-mon[31360]: pgmap v1244: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:55 smithi022 ceph-mon[40250]: pgmap v1244: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:39:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:39:57.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:39:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:39:57.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:39:57.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:39:57.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:39:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:57 smithi122 ceph-mon[32255]: pgmap v1245: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3376993564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:57 smithi022 ceph-mon[31360]: pgmap v1245: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3376993564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:39:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:57 smithi022 ceph-mon[40250]: pgmap v1245: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:39:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3376993564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:39:59 smithi122 ceph-mon[32255]: pgmap v1246: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:39:59 smithi022 ceph-mon[31360]: pgmap v1246: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:39:59 smithi022 ceph-mon[40250]: pgmap v1246: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:00] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:40:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T18:40:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T18:40:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T18:40:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:01 smithi122 ceph-mon[32255]: pgmap v1247: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:02.227 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:01 smithi022 ceph-mon[31360]: pgmap v1247: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:02.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:01 smithi022 ceph-mon[40250]: pgmap v1247: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:40:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:40:02.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:03.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:03.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:03.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:03 smithi122 ceph-mon[32255]: pgmap v1248: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1849039315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:03 smithi022 ceph-mon[31360]: pgmap v1248: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1849039315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:03 smithi022 ceph-mon[40250]: pgmap v1248: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1849039315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:05.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:40:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:40:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:05 smithi122 ceph-mon[32255]: pgmap v1249: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:05 smithi022 ceph-mon[31360]: pgmap v1249: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:05 smithi022 ceph-mon[40250]: pgmap v1249: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:40:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:07 smithi122 ceph-mon[32255]: pgmap v1250: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:08.198 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:08.199 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:07 smithi022 ceph-mon[31360]: pgmap v1250: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:08.199 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:07 smithi022 ceph-mon[40250]: pgmap v1250: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:08.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:08.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:08.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:08.933 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2203972006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2203972006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2203972006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:10.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:10] "GET /metrics HTTP/1.1" 200 202252 "" "Prometheus/2.18.1" 2021-12-15T18:40:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:09 smithi122 ceph-mon[32255]: pgmap v1251: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:09 smithi022 ceph-mon[31360]: pgmap v1251: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:09 smithi022 ceph-mon[40250]: pgmap v1251: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:11 smithi022 ceph-mon[31360]: pgmap v1252: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:11 smithi022 ceph-mon[40250]: pgmap v1252: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:11 smithi122 ceph-mon[32255]: pgmap v1252: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:40:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:40:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:13.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:13.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:13.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:13 smithi022 ceph-mon[40250]: pgmap v1253: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3871086515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:13 smithi022 ceph-mon[31360]: pgmap v1253: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3871086515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:13 smithi122 ceph-mon[32255]: pgmap v1253: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3871086515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:40:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:40:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:15 smithi022 ceph-mon[31360]: pgmap v1254: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:15 smithi022 ceph-mon[40250]: pgmap v1254: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:15 smithi122 ceph-mon[32255]: pgmap v1254: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:40:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:17 smithi022 ceph-mon[31360]: pgmap v1255: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:17 smithi022 ceph-mon[40250]: pgmap v1255: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:17 smithi122 ceph-mon[32255]: pgmap v1255: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:18.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:19.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:19.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:19.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:19 smithi022 ceph-mon[31360]: pgmap v1256: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/295970818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:19 smithi022 ceph-mon[40250]: pgmap v1256: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/295970818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:20] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:40:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:19 smithi122 ceph-mon[32255]: pgmap v1256: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/295970818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:21 smithi022 ceph-mon[31360]: pgmap v1257: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:21 smithi022 ceph-mon[40250]: pgmap v1257: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:22.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:21 smithi122 ceph-mon[32255]: pgmap v1257: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:40:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:40:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:23 smithi022 ceph-mon[31360]: pgmap v1258: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:23 smithi022 ceph-mon[40250]: pgmap v1258: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:24.310 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:23 smithi122 ceph-mon[32255]: pgmap v1258: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:24.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:24.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:24.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/933625857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/933625857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/933625857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:25.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:40:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:40:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:25 smithi022 ceph-mon[31360]: pgmap v1259: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:25 smithi022 ceph-mon[40250]: pgmap v1259: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:25 smithi122 ceph-mon[32255]: pgmap v1259: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:26 smithi022 ceph-mon[31360]: pgmap v1260: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:26 smithi022 ceph-mon[40250]: pgmap v1260: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:26 smithi122 ceph-mon[32255]: pgmap v1260: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:40:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:40:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:40:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:40:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:40:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:40:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:40:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:40:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:40:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:40:29.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:29 smithi022 ceph-mon[31360]: pgmap v1261: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:40:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:29 smithi022 ceph-mon[40250]: pgmap v1261: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:40:30.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:30.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:30.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:29 smithi122 ceph-mon[32255]: pgmap v1261: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:40:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:30] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:40:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3074233401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3074233401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3074233401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:31 smithi122 ceph-mon[32255]: pgmap v1262: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:31 smithi022 ceph-mon[31360]: pgmap v1262: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:31 smithi022 ceph-mon[40250]: pgmap v1262: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:40:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:40:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:40:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:40:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:40:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:40:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:40:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:40:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:40:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:40:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:40:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:40:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:40:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:40:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:33 smithi122 ceph-mon[32255]: pgmap v1263: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:33 smithi022 ceph-mon[31360]: pgmap v1263: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:33 smithi022 ceph-mon[40250]: pgmap v1263: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:35.044 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:35.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:35.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:35.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:35 smithi122 ceph-mon[32255]: pgmap v1264: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3686252386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:35.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:40:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:40:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:35 smithi022 ceph-mon[31360]: pgmap v1264: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3686252386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:35 smithi022 ceph-mon[40250]: pgmap v1264: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3686252386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:40:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:37 smithi122 ceph-mon[32255]: pgmap v1265: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:37 smithi022 ceph-mon[31360]: pgmap v1265: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:37 smithi022 ceph-mon[40250]: pgmap v1265: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:39 smithi122 ceph-mon[32255]: pgmap v1266: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:39 smithi022 ceph-mon[31360]: pgmap v1266: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:39 smithi022 ceph-mon[40250]: pgmap v1266: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:40.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:40] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:40:40.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:40.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:40.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3988335074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3988335074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3988335074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:41 smithi122 ceph-mon[32255]: pgmap v1267: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:41 smithi022 ceph-mon[31360]: pgmap v1267: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:41 smithi022 ceph-mon[40250]: pgmap v1267: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:40:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:40:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:43 smithi122 ceph-mon[32255]: pgmap v1268: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:43 smithi022 ceph-mon[31360]: pgmap v1268: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:43 smithi022 ceph-mon[40250]: pgmap v1268: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:45.783 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:46.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:46.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:46.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:46.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:45 smithi022 ceph-mon[40250]: pgmap v1269: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:46.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:45 smithi022 ceph-mon[31360]: pgmap v1269: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:45 smithi122 ceph-mon[32255]: pgmap v1269: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:46.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:40:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:40:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2523731510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2523731510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2523731510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:40:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:47 smithi122 ceph-mon[32255]: pgmap v1270: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:47 smithi022 ceph-mon[31360]: pgmap v1270: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:47 smithi022 ceph-mon[40250]: pgmap v1270: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:49 smithi122 ceph-mon[32255]: pgmap v1271: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:49 smithi022 ceph-mon[31360]: pgmap v1271: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:49 smithi022 ceph-mon[40250]: pgmap v1271: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:50] "GET /metrics HTTP/1.1" 200 202239 "" "Prometheus/2.18.1" 2021-12-15T18:40:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:51.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:51.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:51.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:51 smithi122 ceph-mon[32255]: pgmap v1272: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1697101136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:51 smithi022 ceph-mon[31360]: pgmap v1272: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1697101136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:51 smithi022 ceph-mon[40250]: pgmap v1272: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1697101136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:40:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:40:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:53 smithi122 ceph-mon[32255]: pgmap v1273: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:53 smithi022 ceph-mon[31360]: pgmap v1273: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:53 smithi022 ceph-mon[40250]: pgmap v1273: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:40:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:40:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:55 smithi122 ceph-mon[32255]: pgmap v1274: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:55 smithi022 ceph-mon[31360]: pgmap v1274: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:55 smithi022 ceph-mon[40250]: pgmap v1274: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:40:56.527 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:40:56.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:40:56.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:40:56.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:40:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3943809057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3943809057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:40:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3943809057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:40:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:40:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:57 smithi122 ceph-mon[32255]: pgmap v1275: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:57 smithi022 ceph-mon[40250]: pgmap v1275: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:40:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:57 smithi022 ceph-mon[31360]: pgmap v1275: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:00.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:00] "GET /metrics HTTP/1.1" 200 202239 "" "Prometheus/2.18.1" 2021-12-15T18:41:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:40:59 smithi122 ceph-mon[32255]: pgmap v1276: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:40:59 smithi022 ceph-mon[31360]: pgmap v1276: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:40:59 smithi022 ceph-mon[40250]: pgmap v1276: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:01.897 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:01 smithi122 ceph-mon[32255]: pgmap v1277: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:02.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:02.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:02.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:02.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:01 smithi022 ceph-mon[40250]: pgmap v1277: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:02.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:01 smithi022 ceph-mon[31360]: pgmap v1277: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:41:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:41:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1078393065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1078393065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1078393065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:03 smithi022 ceph-mon[31360]: pgmap v1278: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:03 smithi022 ceph-mon[40250]: pgmap v1278: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:03 smithi122 ceph-mon[32255]: pgmap v1278: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:05 smithi122 ceph-mon[32255]: pgmap v1279: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:06.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:41:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:41:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:05 smithi022 ceph-mon[31360]: pgmap v1279: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:05 smithi022 ceph-mon[40250]: pgmap v1279: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:07.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:07.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:07.268 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:41:07.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:07.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:07.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:07 smithi022 ceph-mon[31360]: pgmap v1280: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/171607962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:07 smithi022 ceph-mon[40250]: pgmap v1280: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/171607962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:07 smithi122 ceph-mon[32255]: pgmap v1280: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/171607962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:09 smithi022 ceph-mon[31360]: pgmap v1281: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:09 smithi022 ceph-mon[40250]: pgmap v1281: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:09 smithi122 ceph-mon[32255]: pgmap v1281: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:10] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:41:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:11 smithi022 ceph-mon[31360]: pgmap v1282: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:11 smithi022 ceph-mon[40250]: pgmap v1282: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:11 smithi122 ceph-mon[32255]: pgmap v1282: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:12.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:41:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:41:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:13 smithi022 ceph-mon[31360]: pgmap v1283: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3640339943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:13 smithi022 ceph-mon[40250]: pgmap v1283: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3640339943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:13 smithi122 ceph-mon[32255]: pgmap v1283: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3640339943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:15.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:41:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:41:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:15 smithi022 ceph-mon[31360]: pgmap v1284: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:15 smithi022 ceph-mon[40250]: pgmap v1284: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:15 smithi122 ceph-mon[32255]: pgmap v1284: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:41:18.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:17 smithi022 ceph-mon[31360]: pgmap v1285: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:17 smithi022 ceph-mon[40250]: pgmap v1285: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:18.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:18.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:17 smithi122 ceph-mon[32255]: pgmap v1285: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/27761582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/27761582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/27761582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:19 smithi022 ceph-mon[31360]: pgmap v1286: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:19 smithi022 ceph-mon[40250]: pgmap v1286: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:19 smithi122 ceph-mon[32255]: pgmap v1286: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:20] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:41:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:21 smithi122 ceph-mon[32255]: pgmap v1287: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:21 smithi022 ceph-mon[31360]: pgmap v1287: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:21 smithi022 ceph-mon[40250]: pgmap v1287: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:41:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:41:23.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:23.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:23.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:23 smithi022 ceph-mon[31360]: pgmap v1288: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1103067721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:23 smithi022 ceph-mon[40250]: pgmap v1288: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1103067721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:23 smithi122 ceph-mon[32255]: pgmap v1288: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1103067721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:25 smithi122 ceph-mon[32255]: pgmap v1289: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:26.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:41:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:41:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:25 smithi022 ceph-mon[31360]: pgmap v1289: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:25 smithi022 ceph-mon[40250]: pgmap v1289: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:41:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:27 smithi122 ceph-mon[32255]: pgmap v1290: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:27 smithi022 ceph-mon[31360]: pgmap v1290: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:27 smithi022 ceph-mon[40250]: pgmap v1290: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:28.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:29.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:29.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:29.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[31360]: pgmap v1291: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2106242133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:41:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:41:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:41:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:41:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[40250]: pgmap v1291: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2106242133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:41:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:41:30.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:41:30.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:41:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:29 smithi122 ceph-mon[32255]: pgmap v1291: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2106242133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:41:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:41:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:41:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:41:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:30] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:41:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:31 smithi122 ceph-mon[32255]: pgmap v1292: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:31 smithi022 ceph-mon[31360]: pgmap v1292: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:31 smithi022 ceph-mon[40250]: pgmap v1292: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:41:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:41:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:41:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:41:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:41:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:41:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:41:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:41:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:41:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:41:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:41:34.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:33 smithi122 ceph-mon[32255]: pgmap v1293: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:33 smithi022 ceph-mon[31360]: pgmap v1293: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:33 smithi022 ceph-mon[40250]: pgmap v1293: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:34.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:34.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:34.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3767585934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3767585934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3767585934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:35 smithi122 ceph-mon[32255]: pgmap v1294: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:36.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:41:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:41:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:35 smithi022 ceph-mon[31360]: pgmap v1294: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:36.475 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:35 smithi022 ceph-mon[40250]: pgmap v1294: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:41:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:37 smithi122 ceph-mon[32255]: pgmap v1295: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:37 smithi022 ceph-mon[31360]: pgmap v1295: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:37 smithi022 ceph-mon[40250]: pgmap v1295: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:39.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:39.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:39.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:39 smithi122 ceph-mon[32255]: pgmap v1296: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187152292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:39 smithi022 ceph-mon[31360]: pgmap v1296: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187152292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:39 smithi022 ceph-mon[40250]: pgmap v1296: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187152292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:40] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:41:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:41 smithi122 ceph-mon[32255]: pgmap v1297: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:41 smithi022 ceph-mon[31360]: pgmap v1297: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:41 smithi022 ceph-mon[40250]: pgmap v1297: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:41:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:41:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:43 smithi122 ceph-mon[32255]: pgmap v1298: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:43 smithi022 ceph-mon[31360]: pgmap v1298: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:43 smithi022 ceph-mon[40250]: pgmap v1298: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:44.873 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:45.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:45.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:45.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:41:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:41:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:45 smithi122 ceph-mon[32255]: pgmap v1299: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/200849331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:45 smithi022 ceph-mon[31360]: pgmap v1299: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/200849331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:45 smithi022 ceph-mon[40250]: pgmap v1299: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/200849331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:41:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:47 smithi122 ceph-mon[32255]: pgmap v1300: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:47 smithi022 ceph-mon[31360]: pgmap v1300: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:47 smithi022 ceph-mon[40250]: pgmap v1300: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:50.245 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:49 smithi022 ceph-mon[31360]: pgmap v1301: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:50.246 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:49 smithi022 ceph-mon[40250]: pgmap v1301: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:49 smithi122 ceph-mon[32255]: pgmap v1301: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:50] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:41:50.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:50.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:50.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:50.939 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2180031057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2180031057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2180031057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:51 smithi022 ceph-mon[31360]: pgmap v1302: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:51 smithi022 ceph-mon[40250]: pgmap v1302: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:51 smithi122 ceph-mon[32255]: pgmap v1302: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:41:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:41:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:53 smithi022 ceph-mon[31360]: pgmap v1303: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:53 smithi022 ceph-mon[40250]: pgmap v1303: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:53 smithi122 ceph-mon[32255]: pgmap v1303: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:55.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:41:55.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:41:55.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:41:55.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:41:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:41:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:41:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:55 smithi122 ceph-mon[32255]: pgmap v1304: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:55 smithi022 ceph-mon[31360]: pgmap v1304: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:55 smithi022 ceph-mon[40250]: pgmap v1304: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:41:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2195602343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2195602343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:41:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2195602343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:41:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:41:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:57 smithi022 ceph-mon[31360]: pgmap v1305: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:57 smithi022 ceph-mon[40250]: pgmap v1305: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:41:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:57 smithi122 ceph-mon[32255]: pgmap v1305: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:41:59 smithi022 ceph-mon[31360]: pgmap v1306: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:41:59 smithi022 ceph-mon[40250]: pgmap v1306: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:41:59 smithi122 ceph-mon[32255]: pgmap v1306: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:00] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:42:00.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:01.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:01.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:01.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:01 smithi022 ceph-mon[31360]: pgmap v1307: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2868478946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:01 smithi022 ceph-mon[40250]: pgmap v1307: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2868478946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:01 smithi122 ceph-mon[32255]: pgmap v1307: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2868478946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:42:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:42:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:03 smithi022 ceph-mon[31360]: pgmap v1308: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:03 smithi022 ceph-mon[40250]: pgmap v1308: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:03 smithi122 ceph-mon[32255]: pgmap v1308: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:05 smithi122 ceph-mon[32255]: pgmap v1309: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:06.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:42:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:42:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:05 smithi022 ceph-mon[31360]: pgmap v1309: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:05 smithi022 ceph-mon[40250]: pgmap v1309: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:06.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:06.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:06.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:06.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:07.093 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3450900089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:07.094 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:07.095 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3450900089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3450900089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:42:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:07 smithi022 ceph-mon[31360]: pgmap v1310: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:07 smithi022 ceph-mon[40250]: pgmap v1310: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:07 smithi122 ceph-mon[32255]: pgmap v1310: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:09 smithi022 ceph-mon[31360]: pgmap v1311: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:09 smithi022 ceph-mon[40250]: pgmap v1311: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:09 smithi122 ceph-mon[32255]: pgmap v1311: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:10] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:42:11.722 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:12.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:12.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:12.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:11 smithi022 ceph-mon[31360]: pgmap v1312: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:11 smithi022 ceph-mon[40250]: pgmap v1312: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:11 smithi122 ceph-mon[32255]: pgmap v1312: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:42:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:42:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3454390248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3454390248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3454390248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:13 smithi022 ceph-mon[31360]: pgmap v1313: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:13 smithi022 ceph-mon[40250]: pgmap v1313: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:13 smithi122 ceph-mon[32255]: pgmap v1313: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:15 smithi122 ceph-mon[32255]: pgmap v1314: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:15 smithi022 ceph-mon[31360]: pgmap v1314: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:15 smithi022 ceph-mon[40250]: pgmap v1314: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:42:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:42:17.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:17.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:17.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:17.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:42:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:17 smithi122 ceph-mon[32255]: pgmap v1315: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3551263646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:17 smithi022 ceph-mon[31360]: pgmap v1315: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3551263646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:17 smithi022 ceph-mon[40250]: pgmap v1315: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3551263646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:19 smithi122 ceph-mon[32255]: pgmap v1316: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:19 smithi022 ceph-mon[31360]: pgmap v1316: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:19 smithi022 ceph-mon[40250]: pgmap v1316: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:20] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:42:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:21 smithi122 ceph-mon[32255]: pgmap v1317: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:21 smithi022 ceph-mon[31360]: pgmap v1317: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:21 smithi022 ceph-mon[40250]: pgmap v1317: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:22.459 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:42:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:42:22.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:22.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:22.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2566639090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2566639090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2566639090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:23 smithi122 ceph-mon[32255]: pgmap v1318: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:23 smithi022 ceph-mon[31360]: pgmap v1318: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:23 smithi022 ceph-mon[40250]: pgmap v1318: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:26.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:42:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:42:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:25 smithi122 ceph-mon[32255]: pgmap v1319: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:25 smithi022 ceph-mon[31360]: pgmap v1319: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:25 smithi022 ceph-mon[40250]: pgmap v1319: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:42:27.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:27 smithi122 ceph-mon[32255]: pgmap v1320: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:28.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:28.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:28.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:28.198 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:27 smithi022 ceph-mon[31360]: pgmap v1320: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:28.198 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:27 smithi022 ceph-mon[40250]: pgmap v1320: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3261296065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3261296065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3261296065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:29 smithi122 ceph-mon[32255]: pgmap v1321: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:42:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:42:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:42:30.210 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[31360]: pgmap v1321: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:30.210 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:42:30.210 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:42:30.211 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:42:30.211 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[40250]: pgmap v1321: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:30.212 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:42:30.212 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:42:30.212 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:42:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:30] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:42:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:42:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:31 smithi022 ceph-mon[31360]: pgmap v1322: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:42:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:31 smithi022 ceph-mon[40250]: pgmap v1322: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:42:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:31 smithi122 ceph-mon[32255]: pgmap v1322: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:42:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:42:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:42:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:42:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:42:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:42:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:42:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:42:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:42:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:42:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:42:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:42:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:42:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:42:33.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:33 smithi022 ceph-mon[31360]: pgmap v1323: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:33 smithi022 ceph-mon[40250]: pgmap v1323: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:33.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:33.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:33.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:33 smithi122 ceph-mon[32255]: pgmap v1323: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3482788089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3482788089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3482788089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:35 smithi122 ceph-mon[32255]: pgmap v1324: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:35 smithi022 ceph-mon[31360]: pgmap v1324: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:35 smithi022 ceph-mon[40250]: pgmap v1324: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:36.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:42:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:42:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:42:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:37 smithi122 ceph-mon[32255]: pgmap v1325: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:37 smithi022 ceph-mon[31360]: pgmap v1325: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:37 smithi022 ceph-mon[40250]: pgmap v1325: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:38.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:38.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:38.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:38.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:39 smithi122 ceph-mon[32255]: pgmap v1326: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3777512681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:39 smithi022 ceph-mon[31360]: pgmap v1326: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3777512681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:39 smithi022 ceph-mon[40250]: pgmap v1326: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3777512681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:40] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:42:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:41 smithi122 ceph-mon[32255]: pgmap v1327: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:41 smithi022 ceph-mon[40250]: pgmap v1327: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:41 smithi022 ceph-mon[31360]: pgmap v1327: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:42:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:42:43.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:43 smithi122 ceph-mon[32255]: pgmap v1328: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:43 smithi022 ceph-mon[31360]: pgmap v1328: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:43 smithi022 ceph-mon[40250]: pgmap v1328: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:44.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:44.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:44.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3870353784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3870353784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3870353784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:42:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:42:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:45 smithi122 ceph-mon[32255]: pgmap v1329: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:45 smithi022 ceph-mon[31360]: pgmap v1329: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:45 smithi022 ceph-mon[40250]: pgmap v1329: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:42:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:47 smithi122 ceph-mon[32255]: pgmap v1330: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:47 smithi022 ceph-mon[31360]: pgmap v1330: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:47 smithi022 ceph-mon[40250]: pgmap v1330: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:49.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:49.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:49.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:49.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:49.926 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:49 smithi022 ceph-mon[31360]: pgmap v1331: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:49.927 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3971021941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:50.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:50] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:42:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:49 smithi122 ceph-mon[32255]: pgmap v1331: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:50.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3971021941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:49 smithi022 ceph-mon[40250]: pgmap v1331: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3971021941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:51 smithi122 ceph-mon[32255]: pgmap v1332: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:51 smithi022 ceph-mon[31360]: pgmap v1332: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:51 smithi022 ceph-mon[40250]: pgmap v1332: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:42:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:42:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:53 smithi122 ceph-mon[32255]: pgmap v1333: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:53 smithi022 ceph-mon[31360]: pgmap v1333: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:53 smithi022 ceph-mon[40250]: pgmap v1333: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:54.678 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:42:55.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:42:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:42:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:42:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:42:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:42:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:55 smithi122 ceph-mon[32255]: pgmap v1334: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1184041364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:55 smithi022 ceph-mon[31360]: pgmap v1334: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1184041364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:55 smithi022 ceph-mon[40250]: pgmap v1334: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:42:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1184041364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:42:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:42:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:42:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:57 smithi022 ceph-mon[31360]: pgmap v1335: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:57 smithi022 ceph-mon[40250]: pgmap v1335: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:42:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:57 smithi122 ceph-mon[32255]: pgmap v1335: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:00.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:42:59 smithi022 ceph-mon[31360]: pgmap v1336: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:42:59 smithi022 ceph-mon[40250]: pgmap v1336: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:00.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:00.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:00.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:00] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:43:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:42:59 smithi122 ceph-mon[32255]: pgmap v1336: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3120209118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3120209118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3120209118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:01 smithi022 ceph-mon[31360]: pgmap v1337: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:01 smithi022 ceph-mon[40250]: pgmap v1337: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:01 smithi122 ceph-mon[32255]: pgmap v1337: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:43:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:43:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:03 smithi022 ceph-mon[31360]: pgmap v1338: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:03 smithi022 ceph-mon[40250]: pgmap v1338: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:03 smithi122 ceph-mon[32255]: pgmap v1338: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:05.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:05.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:05.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:05.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:05 smithi122 ceph-mon[32255]: pgmap v1339: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/154186661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:06.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:43:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:43:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:05 smithi022 ceph-mon[31360]: pgmap v1339: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/154186661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:05 smithi022 ceph-mon[40250]: pgmap v1339: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/154186661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:43:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:07 smithi022 ceph-mon[31360]: pgmap v1340: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:07 smithi022 ceph-mon[40250]: pgmap v1340: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:07 smithi122 ceph-mon[32255]: pgmap v1340: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:09 smithi022 ceph-mon[31360]: pgmap v1341: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:09 smithi022 ceph-mon[40250]: pgmap v1341: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:10] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:43:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:09 smithi122 ceph-mon[32255]: pgmap v1341: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:10.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:11.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:11.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:11.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:11 smithi022 ceph-mon[40250]: pgmap v1342: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1954093622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:11 smithi022 ceph-mon[31360]: pgmap v1342: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1954093622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:11 smithi122 ceph-mon[32255]: pgmap v1342: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1954093622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:43:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:43:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:13 smithi022 ceph-mon[31360]: pgmap v1343: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:13 smithi022 ceph-mon[40250]: pgmap v1343: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:13 smithi122 ceph-mon[32255]: pgmap v1343: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:16.161 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:43:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:43:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:15 smithi122 ceph-mon[32255]: pgmap v1344: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:15 smithi022 ceph-mon[31360]: pgmap v1344: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:15 smithi022 ceph-mon[40250]: pgmap v1344: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1508035399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1508035399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1508035399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:43:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:17 smithi022 ceph-mon[31360]: pgmap v1345: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:17 smithi022 ceph-mon[40250]: pgmap v1345: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:17 smithi122 ceph-mon[32255]: pgmap v1345: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:19 smithi022 ceph-mon[31360]: pgmap v1346: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:19 smithi022 ceph-mon[40250]: pgmap v1346: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:19 smithi122 ceph-mon[32255]: pgmap v1346: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:20] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:43:21.529 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:21.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:21.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:21.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:21 smithi022 ceph-mon[31360]: pgmap v1347: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/406289049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:21 smithi022 ceph-mon[40250]: pgmap v1347: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/406289049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:21 smithi122 ceph-mon[32255]: pgmap v1347: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/406289049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:43:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:43:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:23 smithi022 ceph-mon[31360]: pgmap v1348: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:23 smithi022 ceph-mon[40250]: pgmap v1348: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:23 smithi122 ceph-mon[32255]: pgmap v1348: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:26.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:43:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:43:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:25 smithi122 ceph-mon[32255]: pgmap v1349: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:25 smithi022 ceph-mon[31360]: pgmap v1349: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:25 smithi022 ceph-mon[40250]: pgmap v1349: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:26.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:27.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:27.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:27.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:27.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:27.268 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:43:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:27 smithi022 ceph-mon[31360]: pgmap v1350: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2530938111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:27 smithi022 ceph-mon[40250]: pgmap v1350: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2530938111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:27 smithi122 ceph-mon[32255]: pgmap v1350: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2530938111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:29 smithi122 ceph-mon[32255]: pgmap v1351: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:29 smithi022 ceph-mon[31360]: pgmap v1351: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:29 smithi022 ceph-mon[40250]: pgmap v1351: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:30] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:43:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:43:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:43:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:43:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:43:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:43:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:43:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:43:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:43:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:43:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:43:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:31 smithi122 ceph-mon[32255]: pgmap v1352: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:43:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:31 smithi022 ceph-mon[31360]: pgmap v1352: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:43:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:31 smithi022 ceph-mon[40250]: pgmap v1352: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:32.269 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:32.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:32.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:32.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:43:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:43:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:43:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:43:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:43:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:43:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2092780271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:43:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:43:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:43:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:43:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2092780271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:43:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:43:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:43:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:43:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2092780271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:33 smithi022 ceph-mon[31360]: pgmap v1353: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:33 smithi022 ceph-mon[40250]: pgmap v1353: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:33 smithi122 ceph-mon[32255]: pgmap v1353: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:35 smithi122 ceph-mon[32255]: pgmap v1354: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:36.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:43:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:43:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:35 smithi022 ceph-mon[31360]: pgmap v1354: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:35 smithi022 ceph-mon[40250]: pgmap v1354: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:37.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:43:38.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:38.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:38.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:37 smithi122 ceph-mon[32255]: pgmap v1355: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:37 smithi022 ceph-mon[31360]: pgmap v1355: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:37 smithi022 ceph-mon[40250]: pgmap v1355: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1395022358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1395022358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1395022358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:40.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:39 smithi122 ceph-mon[32255]: pgmap v1356: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:40.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:40] "GET /metrics HTTP/1.1" 200 202236 "" "Prometheus/2.18.1" 2021-12-15T18:43:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:39 smithi022 ceph-mon[31360]: pgmap v1356: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:39 smithi022 ceph-mon[40250]: pgmap v1356: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:41 smithi122 ceph-mon[32255]: pgmap v1357: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:41 smithi022 ceph-mon[31360]: pgmap v1357: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:41 smithi022 ceph-mon[40250]: pgmap v1357: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:43:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:43:43.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:43.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:43.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:43.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:43 smithi122 ceph-mon[32255]: pgmap v1358: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/885990971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:43 smithi022 ceph-mon[40250]: pgmap v1358: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/885990971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:43 smithi022 ceph-mon[31360]: pgmap v1358: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/885990971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:45 smithi122 ceph-mon[32255]: pgmap v1359: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:46.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:43:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:43:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:45 smithi022 ceph-mon[31360]: pgmap v1359: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:45 smithi022 ceph-mon[40250]: pgmap v1359: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:43:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:47 smithi022 ceph-mon[31360]: pgmap v1360: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:47 smithi022 ceph-mon[40250]: pgmap v1360: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:48.387 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:47 smithi122 ceph-mon[32255]: pgmap v1360: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:48.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:48.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:48.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3159637412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3159637412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3159637412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:49 smithi022 ceph-mon[31360]: pgmap v1361: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:49 smithi022 ceph-mon[40250]: pgmap v1361: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:49 smithi122 ceph-mon[32255]: pgmap v1361: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:50] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:43:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:51 smithi022 ceph-mon[31360]: pgmap v1362: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:51 smithi022 ceph-mon[40250]: pgmap v1362: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:51 smithi122 ceph-mon[32255]: pgmap v1362: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:43:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:43:53.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:53 smithi022 ceph-mon[31360]: pgmap v1363: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:53 smithi022 ceph-mon[40250]: pgmap v1363: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:54.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:54.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:54.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:43:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:53 smithi122 ceph-mon[32255]: pgmap v1363: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2240184294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2240184294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2240184294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:43:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:55 smithi122 ceph-mon[32255]: pgmap v1364: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:56.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:43:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:43:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:55 smithi022 ceph-mon[31360]: pgmap v1364: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:55 smithi022 ceph-mon[40250]: pgmap v1364: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:43:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:43:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:43:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:57 smithi022 ceph-mon[31360]: pgmap v1365: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:57 smithi022 ceph-mon[40250]: pgmap v1365: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:57 smithi122 ceph-mon[32255]: pgmap v1365: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:43:59.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:43:59.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:43:59.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:43:59.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:59 smithi022 ceph-mon[31360]: pgmap v1366: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:43:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3873552328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:59 smithi022 ceph-mon[40250]: pgmap v1366: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:43:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3873552328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:00] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:44:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:59 smithi122 ceph-mon[32255]: pgmap v1366: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:43:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3873552328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:01 smithi022 ceph-mon[31360]: pgmap v1367: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:01 smithi022 ceph-mon[40250]: pgmap v1367: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:01 smithi122 ceph-mon[32255]: pgmap v1367: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:44:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:44:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:03 smithi022 ceph-mon[31360]: pgmap v1368: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:03 smithi022 ceph-mon[40250]: pgmap v1368: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:03 smithi122 ceph-mon[32255]: pgmap v1368: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:04.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:04.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:04.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3415799597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3415799597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3415799597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:44:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:44:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:05 smithi122 ceph-mon[32255]: pgmap v1369: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:05 smithi022 ceph-mon[31360]: pgmap v1369: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:05 smithi022 ceph-mon[40250]: pgmap v1369: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:44:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:07 smithi022 ceph-mon[31360]: pgmap v1370: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:07 smithi022 ceph-mon[40250]: pgmap v1370: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:07 smithi122 ceph-mon[32255]: pgmap v1370: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:10.238 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:09 smithi022 ceph-mon[31360]: pgmap v1371: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:10.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:09 smithi022 ceph-mon[40250]: pgmap v1371: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:10] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:44:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:09 smithi122 ceph-mon[32255]: pgmap v1371: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2102244851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2102244851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2102244851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:11 smithi022 ceph-mon[31360]: pgmap v1372: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:11 smithi022 ceph-mon[40250]: pgmap v1372: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:11 smithi122 ceph-mon[32255]: pgmap v1372: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:44:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:44:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:13 smithi022 ceph-mon[31360]: pgmap v1373: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:13 smithi022 ceph-mon[40250]: pgmap v1373: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:13 smithi122 ceph-mon[32255]: pgmap v1373: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:15.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:15.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:15.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:44:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:44:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:16 smithi122 ceph-mon[32255]: pgmap v1374: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3500552474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:16 smithi022 ceph-mon[31360]: pgmap v1374: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3500552474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:16 smithi022 ceph-mon[40250]: pgmap v1374: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3500552474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:44:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:18 smithi022 ceph-mon[31360]: pgmap v1375: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:18 smithi022 ceph-mon[40250]: pgmap v1375: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:18 smithi122 ceph-mon[32255]: pgmap v1375: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:19 smithi022 ceph-mon[31360]: pgmap v1376: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:19 smithi022 ceph-mon[40250]: pgmap v1376: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:19 smithi122 ceph-mon[32255]: pgmap v1376: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:20] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:44:20.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:20.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:20.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:20.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:21 smithi122 ceph-mon[32255]: pgmap v1377: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3695598469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:21 smithi022 ceph-mon[40250]: pgmap v1377: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3695598469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:21 smithi022 ceph-mon[31360]: pgmap v1377: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3695598469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:44:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:44:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:23 smithi122 ceph-mon[32255]: pgmap v1378: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:23 smithi022 ceph-mon[31360]: pgmap v1378: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:23 smithi022 ceph-mon[40250]: pgmap v1378: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:25.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:26.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:44:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:44:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:25 smithi122 ceph-mon[32255]: pgmap v1379: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:25 smithi022 ceph-mon[31360]: pgmap v1379: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:25 smithi022 ceph-mon[40250]: pgmap v1379: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:26.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:26.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:26.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/256282414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/256282414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/256282414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:44:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:27 smithi022 ceph-mon[31360]: pgmap v1380: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:27 smithi022 ceph-mon[40250]: pgmap v1380: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:27 smithi122 ceph-mon[32255]: pgmap v1380: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:29 smithi022 ceph-mon[31360]: pgmap v1381: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:29 smithi022 ceph-mon[40250]: pgmap v1381: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:30] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:44:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:29 smithi122 ceph-mon[32255]: pgmap v1381: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:44:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:44:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:44:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:44:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:44:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:44:31.117 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:44:31.118 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:44:31.118 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:44:31.351 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:31.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:31.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:31.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:31 smithi022 ceph-mon[31360]: pgmap v1382: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:44:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/893039602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:31 smithi022 ceph-mon[40250]: pgmap v1382: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:44:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/893039602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:31 smithi122 ceph-mon[32255]: pgmap v1382: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:44:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/893039602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:44:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:44:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:44:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:44:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:44:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:44:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:44:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:44:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:44:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:44:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:44:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:44:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:44:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:44:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:33 smithi022 ceph-mon[31360]: pgmap v1383: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:33 smithi022 ceph-mon[40250]: pgmap v1383: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:33 smithi122 ceph-mon[32255]: pgmap v1383: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:35 smithi122 ceph-mon[32255]: pgmap v1384: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:36.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:44:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:44:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:35 smithi022 ceph-mon[31360]: pgmap v1384: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:35 smithi022 ceph-mon[40250]: pgmap v1384: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:37.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:37.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:37.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:44:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:37 smithi022 ceph-mon[31360]: pgmap v1385: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3771868611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:37 smithi022 ceph-mon[40250]: pgmap v1385: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3771868611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:37 smithi122 ceph-mon[32255]: pgmap v1385: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3771868611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:39 smithi022 ceph-mon[31360]: pgmap v1386: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:39 smithi022 ceph-mon[40250]: pgmap v1386: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:39 smithi122 ceph-mon[32255]: pgmap v1386: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:40] "GET /metrics HTTP/1.1" 200 202246 "" "Prometheus/2.18.1" 2021-12-15T18:44:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:41 smithi022 ceph-mon[31360]: pgmap v1387: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:41 smithi022 ceph-mon[40250]: pgmap v1387: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:42.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:41 smithi122 ceph-mon[32255]: pgmap v1387: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:42.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:42.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:42.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:44:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:44:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3443306187' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3443306187' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3443306187' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:43 smithi022 ceph-mon[31360]: pgmap v1388: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:43 smithi022 ceph-mon[40250]: pgmap v1388: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:43 smithi122 ceph-mon[32255]: pgmap v1388: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:44:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:44:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:45 smithi122 ceph-mon[32255]: pgmap v1389: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:45 smithi022 ceph-mon[31360]: pgmap v1389: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:45 smithi022 ceph-mon[40250]: pgmap v1389: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:47.459 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:44:47.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:47.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:47.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:47 smithi022 ceph-mon[31360]: pgmap v1390: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3925898592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:47 smithi022 ceph-mon[40250]: pgmap v1390: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3925898592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:47 smithi122 ceph-mon[32255]: pgmap v1390: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3925898592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:49 smithi022 ceph-mon[31360]: pgmap v1391: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:49 smithi022 ceph-mon[40250]: pgmap v1391: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:50] "GET /metrics HTTP/1.1" 200 202238 "" "Prometheus/2.18.1" 2021-12-15T18:44:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:49 smithi122 ceph-mon[32255]: pgmap v1391: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:51 smithi022 ceph-mon[31360]: pgmap v1392: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:51 smithi022 ceph-mon[40250]: pgmap v1392: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:51 smithi122 ceph-mon[32255]: pgmap v1392: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:44:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:44:52.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:53.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:53.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:53.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:53 smithi022 ceph-mon[40250]: pgmap v1393: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1385205304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:53 smithi022 ceph-mon[31360]: pgmap v1393: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1385205304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:53 smithi122 ceph-mon[32255]: pgmap v1393: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1385205304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:44:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:44:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:55 smithi122 ceph-mon[32255]: pgmap v1394: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:55 smithi022 ceph-mon[31360]: pgmap v1394: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:55 smithi022 ceph-mon[40250]: pgmap v1394: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:44:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:44:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:44:58.202 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:44:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:57 smithi022 ceph-mon[31360]: pgmap v1395: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:57 smithi022 ceph-mon[40250]: pgmap v1395: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:57 smithi122 ceph-mon[32255]: pgmap v1395: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:44:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:44:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:44:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:44:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:44:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2380076337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:44:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2380076337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:44:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:44:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2380076337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:00 smithi022 ceph-mon[31360]: pgmap v1396: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:00 smithi022 ceph-mon[40250]: pgmap v1396: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:00] "GET /metrics HTTP/1.1" 200 202238 "" "Prometheus/2.18.1" 2021-12-15T18:45:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:00 smithi122 ceph-mon[32255]: pgmap v1396: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:02 smithi022 ceph-mon[31360]: pgmap v1397: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:02 smithi022 ceph-mon[40250]: pgmap v1397: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:02 smithi122 ceph-mon[32255]: pgmap v1397: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:45:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:45:03.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:03.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:03.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:04 smithi022 ceph-mon[31360]: pgmap v1398: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1846722371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:04 smithi022 ceph-mon[40250]: pgmap v1398: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1846722371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:04 smithi122 ceph-mon[32255]: pgmap v1398: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1846722371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:45:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:45:06.257 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:06 smithi122 ceph-mon[32255]: pgmap v1399: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:06 smithi022 ceph-mon[31360]: pgmap v1399: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:06 smithi022 ceph-mon[40250]: pgmap v1399: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:45:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:08 smithi022 ceph-mon[31360]: pgmap v1400: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:08 smithi022 ceph-mon[40250]: pgmap v1400: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:08 smithi122 ceph-mon[32255]: pgmap v1400: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:08.943 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:09 smithi022 ceph-mon[31360]: pgmap v1401: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:09 smithi022 ceph-mon[40250]: pgmap v1401: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:09.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:09.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:09.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:09 smithi122 ceph-mon[32255]: pgmap v1401: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1909629191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1909629191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:10] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:45:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1909629191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:11 smithi122 ceph-mon[32255]: pgmap v1402: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:11 smithi022 ceph-mon[31360]: pgmap v1402: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:11 smithi022 ceph-mon[40250]: pgmap v1402: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:45:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:45:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:13 smithi122 ceph-mon[32255]: pgmap v1403: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:13 smithi022 ceph-mon[31360]: pgmap v1403: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:13 smithi022 ceph-mon[40250]: pgmap v1403: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:14.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:14.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:14.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:14.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/884696618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/884696618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/884696618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:15 smithi122 ceph-mon[32255]: pgmap v1404: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:16.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:45:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:45:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:15 smithi022 ceph-mon[31360]: pgmap v1404: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:15 smithi022 ceph-mon[40250]: pgmap v1404: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:45:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:17 smithi022 ceph-mon[31360]: pgmap v1405: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:17 smithi022 ceph-mon[40250]: pgmap v1405: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:17 smithi122 ceph-mon[32255]: pgmap v1405: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:19.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:20.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:20.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:20.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:19 smithi022 ceph-mon[31360]: pgmap v1406: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:19 smithi022 ceph-mon[40250]: pgmap v1406: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:20] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:45:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:19 smithi122 ceph-mon[32255]: pgmap v1406: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/412236138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/412236138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/412236138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:21 smithi022 ceph-mon[31360]: pgmap v1407: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:21 smithi022 ceph-mon[40250]: pgmap v1407: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:21 smithi122 ceph-mon[32255]: pgmap v1407: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:45:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:45:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:23 smithi022 ceph-mon[31360]: pgmap v1408: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:23 smithi022 ceph-mon[40250]: pgmap v1408: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:23 smithi122 ceph-mon[32255]: pgmap v1408: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:25.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:25.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:25.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:25.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:25 smithi122 ceph-mon[32255]: pgmap v1409: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/823759533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:26.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:45:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:45:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:25 smithi022 ceph-mon[31360]: pgmap v1409: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/823759533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:25 smithi022 ceph-mon[40250]: pgmap v1409: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/823759533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:45:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:27 smithi022 ceph-mon[31360]: pgmap v1410: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:27 smithi022 ceph-mon[40250]: pgmap v1410: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:27 smithi122 ceph-mon[32255]: pgmap v1410: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:29 smithi022 ceph-mon[31360]: pgmap v1411: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:29 smithi022 ceph-mon[40250]: pgmap v1411: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:30.424 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:30] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:45:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:29 smithi122 ceph-mon[32255]: pgmap v1411: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:30.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:30.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:30.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:31.190 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3713245060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:31.190 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3713245060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3713245060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[31360]: pgmap v1412: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:45:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:45:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:45:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[40250]: pgmap v1412: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:45:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:45:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:45:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:32.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:31 smithi122 ceph-mon[32255]: pgmap v1412: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:45:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:45:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:45:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:45:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:45:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:45:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:45:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:45:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:45:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:45:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:45:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:45:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:45:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:45:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:45:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:45:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:45:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:45:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:33 smithi022 ceph-mon[31360]: pgmap v1413: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:33 smithi022 ceph-mon[40250]: pgmap v1413: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:33 smithi122 ceph-mon[32255]: pgmap v1413: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:35.793 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:36.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:36.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:36.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:36.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:35 smithi022 ceph-mon[31360]: pgmap v1414: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:36.166 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:35 smithi022 ceph-mon[40250]: pgmap v1414: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:35 smithi122 ceph-mon[32255]: pgmap v1414: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:36.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:45:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:45:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2173502860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2173502860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2173502860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:45:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:38 smithi022 ceph-mon[31360]: pgmap v1415: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:38 smithi022 ceph-mon[40250]: pgmap v1415: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:38 smithi122 ceph-mon[32255]: pgmap v1415: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:40 smithi022 ceph-mon[31360]: pgmap v1416: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:40 smithi022 ceph-mon[40250]: pgmap v1416: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:40 smithi122 ceph-mon[32255]: pgmap v1416: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:40] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:45:41.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:41.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:41.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:41.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:42 smithi022 ceph-mon[31360]: pgmap v1417: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4057188066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:42 smithi022 ceph-mon[40250]: pgmap v1417: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4057188066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:42 smithi122 ceph-mon[32255]: pgmap v1417: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4057188066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:45:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:45:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:44 smithi022 ceph-mon[31360]: pgmap v1418: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:44 smithi022 ceph-mon[40250]: pgmap v1418: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:44 smithi122 ceph-mon[32255]: pgmap v1418: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:46 smithi122 ceph-mon[32255]: pgmap v1419: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:46.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:45:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:45:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:46 smithi022 ceph-mon[31360]: pgmap v1419: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:46 smithi022 ceph-mon[40250]: pgmap v1419: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:46.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:46.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:46.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:46.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3082088130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:47 smithi022 ceph-mon[31360]: pgmap v1420: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3082088130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:47 smithi022 ceph-mon[40250]: pgmap v1420: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3082088130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:47 smithi122 ceph-mon[32255]: pgmap v1420: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:45:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:49 smithi022 ceph-mon[31360]: pgmap v1421: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:49 smithi022 ceph-mon[40250]: pgmap v1421: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:49 smithi122 ceph-mon[32255]: pgmap v1421: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:50] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:45:51.907 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:52.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:52.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:52.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:51 smithi022 ceph-mon[31360]: pgmap v1422: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:51 smithi022 ceph-mon[40250]: pgmap v1422: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:51 smithi122 ceph-mon[32255]: pgmap v1422: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:45:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:45:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1279269475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1279269475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1279269475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:53 smithi022 ceph-mon[31360]: pgmap v1423: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:53 smithi022 ceph-mon[40250]: pgmap v1423: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:53 smithi122 ceph-mon[32255]: pgmap v1423: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:55 smithi022 ceph-mon[31360]: pgmap v1424: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:55 smithi022 ceph-mon[40250]: pgmap v1424: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:56.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:55 smithi122 ceph-mon[32255]: pgmap v1424: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:45:56.440 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:45:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:45:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:57.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:45:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:45:57.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:45:57.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:45:57.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:45:57.651 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:45:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:57 smithi022 ceph-mon[31360]: pgmap v1425: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2766750145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:57 smithi022 ceph-mon[40250]: pgmap v1425: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2766750145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:45:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:57 smithi122 ceph-mon[32255]: pgmap v1425: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:45:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2766750145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:45:59 smithi022 ceph-mon[31360]: pgmap v1426: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:45:59 smithi022 ceph-mon[40250]: pgmap v1426: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:45:59 smithi122 ceph-mon[32255]: pgmap v1426: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:00] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:46:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:01 smithi022 ceph-mon[31360]: pgmap v1427: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:01 smithi022 ceph-mon[40250]: pgmap v1427: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:01 smithi122 ceph-mon[32255]: pgmap v1427: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:02.652 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:46:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:46:03.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:03.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:03.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:03 smithi022 ceph-mon[31360]: pgmap v1428: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1529533442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:03 smithi022 ceph-mon[40250]: pgmap v1428: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1529533442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:03 smithi122 ceph-mon[32255]: pgmap v1428: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1529533442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:05 smithi022 ceph-mon[31360]: pgmap v1429: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:05 smithi022 ceph-mon[40250]: pgmap v1429: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:46:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:46:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:05 smithi122 ceph-mon[32255]: pgmap v1429: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:46:08.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:07 smithi022 ceph-mon[31360]: pgmap v1430: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:07 smithi022 ceph-mon[40250]: pgmap v1430: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:08.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:08.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:08.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:07 smithi122 ceph-mon[32255]: pgmap v1430: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3060279707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3060279707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3060279707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:09 smithi022 ceph-mon[31360]: pgmap v1431: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:09 smithi022 ceph-mon[40250]: pgmap v1431: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:09 smithi122 ceph-mon[32255]: pgmap v1431: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:10] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:46:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:11 smithi022 ceph-mon[31360]: pgmap v1432: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:11 smithi022 ceph-mon[40250]: pgmap v1432: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:11 smithi122 ceph-mon[32255]: pgmap v1432: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:46:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:46:13.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:13.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:13.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:13.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:14 smithi022 ceph-mon[31360]: pgmap v1433: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3053648696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:14 smithi022 ceph-mon[40250]: pgmap v1433: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3053648696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:14 smithi122 ceph-mon[32255]: pgmap v1433: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3053648696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:16 smithi022 ceph-mon[31360]: pgmap v1434: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:16 smithi022 ceph-mon[40250]: pgmap v1434: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:46:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:46:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:16 smithi122 ceph-mon[32255]: pgmap v1434: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:46:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:18 smithi022 ceph-mon[31360]: pgmap v1435: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:18 smithi022 ceph-mon[40250]: pgmap v1435: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:18 smithi122 ceph-mon[32255]: pgmap v1435: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:18.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:19.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:19.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:19.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:20 smithi022 ceph-mon[31360]: pgmap v1436: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1381750072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:20 smithi022 ceph-mon[40250]: pgmap v1436: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1381750072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:20] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:46:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:20 smithi122 ceph-mon[32255]: pgmap v1436: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1381750072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:22 smithi022 ceph-mon[31360]: pgmap v1437: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:22 smithi022 ceph-mon[40250]: pgmap v1437: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:22.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:22 smithi122 ceph-mon[32255]: pgmap v1437: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:46:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:46:24.130 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:24 smithi022 ceph-mon[31360]: pgmap v1438: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:24 smithi022 ceph-mon[40250]: pgmap v1438: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:24 smithi122 ceph-mon[32255]: pgmap v1438: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2382126947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2382126947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2382126947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:26 smithi022 ceph-mon[31360]: pgmap v1439: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:26 smithi022 ceph-mon[40250]: pgmap v1439: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:26 smithi122 ceph-mon[32255]: pgmap v1439: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:46:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:46:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:46:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:28 smithi022 ceph-mon[31360]: pgmap v1440: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:28 smithi022 ceph-mon[40250]: pgmap v1440: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:28 smithi122 ceph-mon[32255]: pgmap v1440: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:29 smithi122 ceph-mon[32255]: pgmap v1441: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:29.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:29.501 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:29 smithi022 ceph-mon[31360]: pgmap v1441: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:29.501 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:29 smithi022 ceph-mon[40250]: pgmap v1441: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/227878710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/227878710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/227878710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:30] "GET /metrics HTTP/1.1" 200 202241 "" "Prometheus/2.18.1" 2021-12-15T18:46:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:31 smithi022 ceph-mon[31360]: pgmap v1442: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:31 smithi022 ceph-mon[40250]: pgmap v1442: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:32.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:31 smithi122 ceph-mon[32255]: pgmap v1442: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:46:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:46:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:46:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:46:33.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:46:33.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:46:33.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:46:33.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:46:33.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:46:33.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:46:33.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:46:33.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:46:33.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:46:33.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:46:33.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:46:33.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:46:33.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:46:33.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:46:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:46:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:46:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:46:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:46:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:46:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:46:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:46:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:46:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:33 smithi022 ceph-mon[31360]: pgmap v1443: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:33 smithi022 ceph-mon[40250]: pgmap v1443: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:33 smithi122 ceph-mon[32255]: pgmap v1443: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:34.872 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:35.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:35.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:35.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:35 smithi022 ceph-mon[31360]: pgmap v1444: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3577956584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:35 smithi022 ceph-mon[40250]: pgmap v1444: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3577956584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:46:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:46:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:35 smithi122 ceph-mon[32255]: pgmap v1444: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3577956584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:46:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:37 smithi022 ceph-mon[31360]: pgmap v1445: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:37 smithi022 ceph-mon[40250]: pgmap v1445: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:38.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:37 smithi122 ceph-mon[32255]: pgmap v1445: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:40.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:40.246 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:39 smithi022 ceph-mon[40250]: pgmap v1446: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:40.247 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:39 smithi022 ceph-mon[31360]: pgmap v1446: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:40] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:46:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:39 smithi122 ceph-mon[32255]: pgmap v1446: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:40.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:40.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:40.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/394748152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/394748152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/394748152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:41 smithi022 ceph-mon[31360]: pgmap v1447: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:41 smithi022 ceph-mon[40250]: pgmap v1447: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:42.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:41 smithi122 ceph-mon[32255]: pgmap v1447: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:46:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:46:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:43 smithi022 ceph-mon[31360]: pgmap v1448: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:43 smithi022 ceph-mon[40250]: pgmap v1448: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:43 smithi122 ceph-mon[32255]: pgmap v1448: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:45.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:45.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:45.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:45.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:45 smithi022 ceph-mon[31360]: pgmap v1449: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:45 smithi022 ceph-mon[40250]: pgmap v1449: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:46:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:46:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:45 smithi122 ceph-mon[32255]: pgmap v1449: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3306027652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3306027652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3306027652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:46:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:47 smithi022 ceph-mon[31360]: pgmap v1450: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:47 smithi022 ceph-mon[40250]: pgmap v1450: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:47 smithi122 ceph-mon[32255]: pgmap v1450: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:46:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:49 smithi022 ceph-mon[31360]: pgmap v1451: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:49 smithi022 ceph-mon[40250]: pgmap v1451: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:50] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:46:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:49 smithi122 ceph-mon[32255]: pgmap v1451: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:50.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:51 smithi022 ceph-mon[31360]: pgmap v1452: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1998456147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:51 smithi022 ceph-mon[40250]: pgmap v1452: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1998456147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:51 smithi122 ceph-mon[32255]: pgmap v1452: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1998456147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:46:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:46:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:53 smithi022 ceph-mon[31360]: pgmap v1453: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:53 smithi022 ceph-mon[40250]: pgmap v1453: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:53 smithi122 ceph-mon[32255]: pgmap v1453: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:55 smithi022 ceph-mon[31360]: pgmap v1454: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:55 smithi022 ceph-mon[40250]: pgmap v1454: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:56.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:46:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:46:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:46:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:55 smithi122 ceph-mon[32255]: pgmap v1454: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:46:56.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:46:56.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:46:56.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:46:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2833331471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2833331471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:46:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2833331471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:46:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:46:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:57 smithi022 ceph-mon[31360]: pgmap v1455: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:57 smithi022 ceph-mon[40250]: pgmap v1455: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:46:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:57 smithi122 ceph-mon[32255]: pgmap v1455: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:47:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:46:59 smithi022 ceph-mon[31360]: pgmap v1456: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:46:59 smithi022 ceph-mon[40250]: pgmap v1456: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:00] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:47:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:46:59 smithi122 ceph-mon[32255]: pgmap v1456: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:01.727 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:02.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:02.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:02.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:02 smithi022 ceph-mon[31360]: pgmap v1457: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:02 smithi022 ceph-mon[40250]: pgmap v1457: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:02.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:02 smithi122 ceph-mon[32255]: pgmap v1457: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:47:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:47:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/369613170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/369613170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/369613170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:04 smithi022 ceph-mon[31360]: pgmap v1458: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:04 smithi022 ceph-mon[40250]: pgmap v1458: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:04 smithi122 ceph-mon[32255]: pgmap v1458: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:06 smithi022 ceph-mon[31360]: pgmap v1459: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:06 smithi022 ceph-mon[40250]: pgmap v1459: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:06 smithi122 ceph-mon[32255]: pgmap v1459: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:06.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:47:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:47:07.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:07.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:07.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:07.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:47:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:08 smithi022 ceph-mon[31360]: pgmap v1460: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2581924267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:08 smithi022 ceph-mon[40250]: pgmap v1460: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2581924267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:08 smithi122 ceph-mon[32255]: pgmap v1460: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2581924267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:10 smithi022 ceph-mon[31360]: pgmap v1461: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:10 smithi022 ceph-mon[40250]: pgmap v1461: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:10 smithi122 ceph-mon[32255]: pgmap v1461: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:10] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:47:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:12 smithi022 ceph-mon[31360]: pgmap v1462: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:12 smithi022 ceph-mon[40250]: pgmap v1462: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:12 smithi122 ceph-mon[32255]: pgmap v1462: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:12.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:47:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:47:12.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:12.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:12.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/673909539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/673909539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/673909539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:14 smithi122 ceph-mon[32255]: pgmap v1463: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:14 smithi022 ceph-mon[31360]: pgmap v1463: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:14 smithi022 ceph-mon[40250]: pgmap v1463: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:15 smithi122 ceph-mon[32255]: pgmap v1464: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:15 smithi022 ceph-mon[31360]: pgmap v1464: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:15 smithi022 ceph-mon[40250]: pgmap v1464: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:47:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:47:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:47:17.847 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:18.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:18.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:18.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:18.216 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:17 smithi022 ceph-mon[31360]: pgmap v1465: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:18.217 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:17 smithi022 ceph-mon[40250]: pgmap v1465: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:17 smithi122 ceph-mon[32255]: pgmap v1465: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3304614420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3304614420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3304614420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:19 smithi022 ceph-mon[31360]: pgmap v1466: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:19 smithi022 ceph-mon[40250]: pgmap v1466: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:20] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:47:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:19 smithi122 ceph-mon[32255]: pgmap v1466: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:21 smithi022 ceph-mon[31360]: pgmap v1467: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:21 smithi022 ceph-mon[40250]: pgmap v1467: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:22.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:21 smithi122 ceph-mon[32255]: pgmap v1467: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:47:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:47:23.217 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:23.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:23.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:23.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:23 smithi022 ceph-mon[31360]: pgmap v1468: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3919659047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:23 smithi022 ceph-mon[40250]: pgmap v1468: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3919659047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:23 smithi122 ceph-mon[32255]: pgmap v1468: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3919659047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:25 smithi022 ceph-mon[31360]: pgmap v1469: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:25 smithi022 ceph-mon[40250]: pgmap v1469: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:25 smithi122 ceph-mon[32255]: pgmap v1469: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:47:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:47:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:47:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:27 smithi022 ceph-mon[31360]: pgmap v1470: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:27 smithi022 ceph-mon[40250]: pgmap v1470: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:27 smithi122 ceph-mon[32255]: pgmap v1470: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:28.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:28.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:28.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1534058014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1534058014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1534058014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:29 smithi022 ceph-mon[31360]: pgmap v1471: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:29 smithi022 ceph-mon[40250]: pgmap v1471: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:29 smithi122 ceph-mon[32255]: pgmap v1471: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:30] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:47:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:31 smithi022 ceph-mon[40250]: pgmap v1472: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:31 smithi022 ceph-mon[31360]: pgmap v1472: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:31 smithi122 ceph-mon[32255]: pgmap v1472: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:47:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:47:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:47:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:47:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:47:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:47:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:47:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:47:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:47:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:47:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:47:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:47:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:47:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:47:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:47:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:47:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:47:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:47:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:47:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:47:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:47:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:47:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:47:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:47:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:47:33.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:47:33.959 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:34 smithi022 ceph-mon[31360]: pgmap v1473: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:34 smithi022 ceph-mon[40250]: pgmap v1473: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:34.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:34 smithi122 ceph-mon[32255]: pgmap v1473: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1887253835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1887253835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1887253835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:36 smithi022 ceph-mon[31360]: pgmap v1474: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:36 smithi022 ceph-mon[40250]: pgmap v1474: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:36 smithi122 ceph-mon[32255]: pgmap v1474: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:36.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:47:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:47:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:47:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:38 smithi022 ceph-mon[31360]: pgmap v1475: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:38 smithi022 ceph-mon[40250]: pgmap v1475: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:38 smithi122 ceph-mon[32255]: pgmap v1475: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:39.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:39.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:39.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:39.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:40 smithi022 ceph-mon[31360]: pgmap v1476: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2955698677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:40 smithi022 ceph-mon[40250]: pgmap v1476: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2955698677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:40] "GET /metrics HTTP/1.1" 200 202240 "" "Prometheus/2.18.1" 2021-12-15T18:47:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:40 smithi122 ceph-mon[32255]: pgmap v1476: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2955698677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:42 smithi022 ceph-mon[31360]: pgmap v1477: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:42 smithi022 ceph-mon[40250]: pgmap v1477: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:42.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:42 smithi122 ceph-mon[32255]: pgmap v1477: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:47:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:47:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:44 smithi022 ceph-mon[31360]: pgmap v1478: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:44 smithi022 ceph-mon[40250]: pgmap v1478: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:44 smithi122 ceph-mon[32255]: pgmap v1478: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:44.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:45.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:45.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:45.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:46 smithi022 ceph-mon[31360]: pgmap v1479: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1062119617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:46 smithi022 ceph-mon[40250]: pgmap v1479: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1062119617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:47:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:47:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:46 smithi122 ceph-mon[32255]: pgmap v1479: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1062119617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:47:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:48 smithi022 ceph-mon[31360]: pgmap v1480: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:48 smithi022 ceph-mon[40250]: pgmap v1480: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:48 smithi122 ceph-mon[32255]: pgmap v1480: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:50.064 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:50] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:47:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:50 smithi122 ceph-mon[32255]: pgmap v1481: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:50.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:50.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:50.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:50.437 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:50 smithi022 ceph-mon[40250]: pgmap v1481: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:50.438 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:50 smithi022 ceph-mon[31360]: pgmap v1481: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:47:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3022856988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3022856988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3022856988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:52 smithi122 ceph-mon[32255]: pgmap v1482: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:52 smithi022 ceph-mon[31360]: pgmap v1482: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:52 smithi022 ceph-mon[40250]: pgmap v1482: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:47:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:47:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:53 smithi122 ceph-mon[32255]: pgmap v1483: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:53 smithi022 ceph-mon[31360]: pgmap v1483: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:53 smithi022 ceph-mon[40250]: pgmap v1483: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:47:55.437 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:47:55.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:47:55.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:47:55.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:47:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:55 smithi022 ceph-mon[31360]: pgmap v1484: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:47:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3781757848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:55 smithi022 ceph-mon[40250]: pgmap v1484: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:47:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3781757848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:55 smithi122 ceph-mon[32255]: pgmap v1484: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:47:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3781757848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:47:56.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:47:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:47:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:47:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:57 smithi022 ceph-mon[31360]: pgmap v1485: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:47:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:57 smithi022 ceph-mon[40250]: pgmap v1485: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:47:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:47:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:57 smithi122 ceph-mon[32255]: pgmap v1485: 132 pgs: 132 active+clean; 582 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:48:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:47:59 smithi022 ceph-mon[31360]: pgmap v1486: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:47:59 smithi022 ceph-mon[40250]: pgmap v1486: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:00] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:48:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:47:59 smithi122 ceph-mon[32255]: pgmap v1486: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:00.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:01.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:01.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:01.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:02 smithi022 ceph-mon[31360]: pgmap v1487: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4280683525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:02 smithi022 ceph-mon[40250]: pgmap v1487: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4280683525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:02.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:02 smithi122 ceph-mon[32255]: pgmap v1487: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4280683525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:48:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:48:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:04 smithi022 ceph-mon[31360]: pgmap v1488: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:04 smithi022 ceph-mon[40250]: pgmap v1488: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:04 smithi122 ceph-mon[32255]: pgmap v1488: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:06.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:06 smithi022 ceph-mon[31360]: pgmap v1489: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:06 smithi022 ceph-mon[40250]: pgmap v1489: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:48:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:48:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:06 smithi122 ceph-mon[32255]: pgmap v1489: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:48:06.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:06.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:06.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3639027332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3639027332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3639027332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:48:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:08 smithi022 ceph-mon[31360]: pgmap v1490: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:48:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:08 smithi022 ceph-mon[40250]: pgmap v1490: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:48:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:08 smithi122 ceph-mon[32255]: pgmap v1490: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:48:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:10 smithi022 ceph-mon[31360]: pgmap v1491: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:48:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:10 smithi022 ceph-mon[40250]: pgmap v1491: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:48:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:10 smithi122 ceph-mon[32255]: pgmap v1491: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:48:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:10] "GET /metrics HTTP/1.1" 200 202248 "" "Prometheus/2.18.1" 2021-12-15T18:48:11.555 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:11.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:11.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:11.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:12 smithi022 ceph-mon[31360]: pgmap v1492: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2970594376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:12 smithi022 ceph-mon[40250]: pgmap v1492: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2970594376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:12 smithi122 ceph-mon[32255]: pgmap v1492: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2970594376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:48:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:48:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:14 smithi022 ceph-mon[31360]: pgmap v1493: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:14 smithi022 ceph-mon[40250]: pgmap v1493: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:14 smithi122 ceph-mon[32255]: pgmap v1493: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:16 smithi022 ceph-mon[31360]: pgmap v1494: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:16 smithi022 ceph-mon[40250]: pgmap v1494: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:48:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:48:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:16 smithi122 ceph-mon[32255]: pgmap v1494: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:16.926 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:17.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:17.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:48:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:18 smithi022 ceph-mon[31360]: pgmap v1495: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3021946926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:18 smithi022 ceph-mon[40250]: pgmap v1495: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3021946926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:18 smithi122 ceph-mon[32255]: pgmap v1495: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3021946926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:20 smithi122 ceph-mon[32255]: pgmap v1496: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:20] "GET /metrics HTTP/1.1" 200 202246 "" "Prometheus/2.18.1" 2021-12-15T18:48:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:20 smithi022 ceph-mon[31360]: pgmap v1496: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:20 smithi022 ceph-mon[40250]: pgmap v1496: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:22.296 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:22.297 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:22 smithi022 ceph-mon[31360]: pgmap v1497: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:22.298 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:22 smithi022 ceph-mon[40250]: pgmap v1497: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:22 smithi122 ceph-mon[32255]: pgmap v1497: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:22.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:22.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:22.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:48:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:48:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3426740514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:23.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3426740514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:23.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3426740514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:24 smithi122 ceph-mon[32255]: pgmap v1498: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:24 smithi022 ceph-mon[31360]: pgmap v1498: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:24 smithi022 ceph-mon[40250]: pgmap v1498: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:26 smithi122 ceph-mon[32255]: pgmap v1499: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:48:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:48:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:26 smithi022 ceph-mon[31360]: pgmap v1499: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:26 smithi022 ceph-mon[40250]: pgmap v1499: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:27 smithi022 ceph-mon[31360]: pgmap v1500: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:27 smithi022 ceph-mon[40250]: pgmap v1500: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:27 smithi122 ceph-mon[32255]: pgmap v1500: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:27.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:27.668 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:48:28.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:28.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:28.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3992074742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3992074742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3992074742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:29 smithi122 ceph-mon[32255]: pgmap v1501: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:29 smithi022 ceph-mon[31360]: pgmap v1501: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:29 smithi022 ceph-mon[40250]: pgmap v1501: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:30] "GET /metrics HTTP/1.1" 200 202246 "" "Prometheus/2.18.1" 2021-12-15T18:48:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:32 smithi022 ceph-mon[31360]: pgmap v1502: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:32 smithi022 ceph-mon[40250]: pgmap v1502: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:32 smithi122 ceph-mon[32255]: pgmap v1502: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:48:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:48:33.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:48:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:48:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:48:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:48:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:48:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:48:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:48:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:48:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:48:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:48:33.378 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:48:33.378 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:48:33.379 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:48:33.379 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:48:33.380 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:48:33.380 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:48:33.381 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:48:33.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:34 smithi022 ceph-mon[31360]: pgmap v1503: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:48:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3676392064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:34 smithi022 ceph-mon[40250]: pgmap v1503: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:48:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3676392064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:34 smithi122 ceph-mon[32255]: pgmap v1503: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:48:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3676392064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:36 smithi022 ceph-mon[31360]: pgmap v1504: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:36 smithi022 ceph-mon[40250]: pgmap v1504: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:36 smithi122 ceph-mon[32255]: pgmap v1504: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:36.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:48:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:48:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:48:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:38 smithi022 ceph-mon[31360]: pgmap v1505: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:38 smithi022 ceph-mon[40250]: pgmap v1505: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:38 smithi122 ceph-mon[32255]: pgmap v1505: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:38.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:38.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:38.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:38.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4013494659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4013494659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4013494659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:40 smithi022 ceph-mon[31360]: pgmap v1506: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:40 smithi022 ceph-mon[40250]: pgmap v1506: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:40] "GET /metrics HTTP/1.1" 200 202252 "" "Prometheus/2.18.1" 2021-12-15T18:48:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:40 smithi122 ceph-mon[32255]: pgmap v1506: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:42 smithi022 ceph-mon[31360]: pgmap v1507: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:42 smithi022 ceph-mon[40250]: pgmap v1507: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:42 smithi122 ceph-mon[32255]: pgmap v1507: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:48:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:48:43.810 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:44.181 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:44 smithi022 ceph-mon[31360]: pgmap v1508: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:44.182 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:44 smithi022 ceph-mon[40250]: pgmap v1508: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:44 smithi122 ceph-mon[32255]: pgmap v1508: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4157353966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4157353966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4157353966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:48:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:48:46.459 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:46 smithi122 ceph-mon[32255]: pgmap v1509: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:46 smithi022 ceph-mon[31360]: pgmap v1509: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:46 smithi022 ceph-mon[40250]: pgmap v1509: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:48:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:48 smithi122 ceph-mon[32255]: pgmap v1510: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:48 smithi022 ceph-mon[31360]: pgmap v1510: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:48 smithi022 ceph-mon[40250]: pgmap v1510: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:49.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:49.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:49.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:49.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:50] "GET /metrics HTTP/1.1" 200 202248 "" "Prometheus/2.18.1" 2021-12-15T18:48:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:50 smithi122 ceph-mon[32255]: pgmap v1511: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1431027007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:50 smithi022 ceph-mon[31360]: pgmap v1511: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1431027007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:50 smithi022 ceph-mon[40250]: pgmap v1511: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1431027007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:52 smithi122 ceph-mon[32255]: pgmap v1512: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:52 smithi022 ceph-mon[31360]: pgmap v1512: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:52 smithi022 ceph-mon[40250]: pgmap v1512: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:48:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:48:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:54 smithi122 ceph-mon[32255]: pgmap v1513: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:54 smithi022 ceph-mon[31360]: pgmap v1513: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:54 smithi022 ceph-mon[40250]: pgmap v1513: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:54.551 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:48:54.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:48:54.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:48:54.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:48:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3454345480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3454345480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3454345480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:48:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:56 smithi122 ceph-mon[32255]: pgmap v1514: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:56.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:48:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:48:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:56 smithi022 ceph-mon[31360]: pgmap v1514: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:56 smithi022 ceph-mon[40250]: pgmap v1514: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:48:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:48:57 smithi022 ceph-mon[31360]: pgmap v1515: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:48:57 smithi022 ceph-mon[40250]: pgmap v1515: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:48:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:48:57 smithi122 ceph-mon[32255]: pgmap v1515: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:48:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:48:59.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:00 smithi022 ceph-mon[31360]: pgmap v1516: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:00 smithi022 ceph-mon[40250]: pgmap v1516: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:00.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:00.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:00.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:00] "GET /metrics HTTP/1.1" 200 202248 "" "Prometheus/2.18.1" 2021-12-15T18:49:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:00 smithi122 ceph-mon[32255]: pgmap v1516: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2752733618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2752733618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2752733618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:02 smithi022 ceph-mon[31360]: pgmap v1517: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:02 smithi022 ceph-mon[40250]: pgmap v1517: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:02.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:02 smithi122 ceph-mon[32255]: pgmap v1517: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:49:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:49:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:04 smithi022 ceph-mon[31360]: pgmap v1518: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:04 smithi022 ceph-mon[40250]: pgmap v1518: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:04 smithi122 ceph-mon[32255]: pgmap v1518: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:05.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:05.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:05.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:05.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:06 smithi022 ceph-mon[31360]: pgmap v1519: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301316950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:06 smithi022 ceph-mon[40250]: pgmap v1519: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301316950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:49:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:49:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:06 smithi122 ceph-mon[32255]: pgmap v1519: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301316950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:49:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:08 smithi022 ceph-mon[31360]: pgmap v1520: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:08 smithi022 ceph-mon[40250]: pgmap v1520: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:08 smithi122 ceph-mon[32255]: pgmap v1520: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:10] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:49:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:10 smithi122 ceph-mon[32255]: pgmap v1521: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:10 smithi022 ceph-mon[31360]: pgmap v1521: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:10 smithi022 ceph-mon[40250]: pgmap v1521: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:10.661 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:11.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:11.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:11.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:11 smithi022 ceph-mon[31360]: pgmap v1522: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/426601859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:11 smithi022 ceph-mon[40250]: pgmap v1522: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/426601859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:11 smithi122 ceph-mon[32255]: pgmap v1522: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/426601859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:49:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:49:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:13 smithi022 ceph-mon[31360]: pgmap v1523: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:13 smithi022 ceph-mon[40250]: pgmap v1523: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:14.369 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:13 smithi122 ceph-mon[32255]: pgmap v1523: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:16.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:16.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:49:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:49:16.219 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:15 smithi122 ceph-mon[32255]: pgmap v1524: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:15 smithi022 ceph-mon[31360]: pgmap v1524: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:15 smithi022 ceph-mon[40250]: pgmap v1524: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:16.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:16.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:16.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3384486886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3384486886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3384486886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:49:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:17 smithi022 ceph-mon[31360]: pgmap v1525: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:17 smithi022 ceph-mon[40250]: pgmap v1525: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:17 smithi122 ceph-mon[32255]: pgmap v1525: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:20 smithi022 ceph-mon[31360]: pgmap v1526: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:20 smithi022 ceph-mon[40250]: pgmap v1526: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:20] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:49:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:20 smithi122 ceph-mon[32255]: pgmap v1526: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:21.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:21.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:21.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:21.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:22 smithi022 ceph-mon[31360]: pgmap v1527: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1665013295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:22 smithi022 ceph-mon[40250]: pgmap v1527: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1665013295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:22.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:22 smithi122 ceph-mon[32255]: pgmap v1527: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1665013295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:49:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:49:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:24 smithi022 ceph-mon[31360]: pgmap v1528: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:24 smithi022 ceph-mon[40250]: pgmap v1528: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:24 smithi122 ceph-mon[32255]: pgmap v1528: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:26 smithi022 ceph-mon[31360]: pgmap v1529: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:26 smithi022 ceph-mon[40250]: pgmap v1529: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:26 smithi122 ceph-mon[32255]: pgmap v1529: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:49:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:49:26.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:27.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:27.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:27.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:49:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:28 smithi022 ceph-mon[31360]: pgmap v1530: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1704798062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:28 smithi022 ceph-mon[40250]: pgmap v1530: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:28.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1704798062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:28 smithi122 ceph-mon[32255]: pgmap v1530: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1704798062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:30 smithi022 ceph-mon[31360]: pgmap v1531: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:30 smithi022 ceph-mon[40250]: pgmap v1531: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:30] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:49:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:30 smithi122 ceph-mon[32255]: pgmap v1531: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:32.139 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:32 smithi022 ceph-mon[31360]: pgmap v1532: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:32 smithi022 ceph-mon[40250]: pgmap v1532: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:32 smithi122 ceph-mon[32255]: pgmap v1532: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:32.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:32.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:32.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:49:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:49:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:49:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:49:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:49:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:49:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/920294587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:49:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:49:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:49:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:49:33.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/920294587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:33.417 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:49:33.417 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:49:33.418 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:49:33.418 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:49:33.418 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/920294587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[31360]: pgmap v1533: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:49:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:49:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:49:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[40250]: pgmap v1533: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:49:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:49:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:49:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:34 smithi122 ceph-mon[32255]: pgmap v1533: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:49:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:49:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:49:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:36 smithi022 ceph-mon[31360]: pgmap v1534: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:36 smithi022 ceph-mon[40250]: pgmap v1534: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:36 smithi122 ceph-mon[32255]: pgmap v1534: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:36.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:49:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:49:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:49:37.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:37.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:37.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:37 smithi122 ceph-mon[32255]: pgmap v1535: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1138887146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[31360]: pgmap v1535: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1138887146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[40250]: pgmap v1535: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1138887146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:49:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:39 smithi022 ceph-mon[31360]: pgmap v1536: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:39 smithi022 ceph-mon[40250]: pgmap v1536: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:39 smithi122 ceph-mon[32255]: pgmap v1536: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:40] "GET /metrics HTTP/1.1" 200 202246 "" "Prometheus/2.18.1" 2021-12-15T18:49:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:42 smithi022 ceph-mon[31360]: pgmap v1537: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:42 smithi022 ceph-mon[40250]: pgmap v1537: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:42 smithi122 ceph-mon[32255]: pgmap v1537: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:49:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:49:42.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:43.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:43.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:43.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:44 smithi022 ceph-mon[31360]: pgmap v1538: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/614401952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:44 smithi022 ceph-mon[40250]: pgmap v1538: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/614401952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:44 smithi122 ceph-mon[32255]: pgmap v1538: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/614401952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:46 smithi122 ceph-mon[32255]: pgmap v1539: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:46.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:49:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:49:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:46 smithi022 ceph-mon[31360]: pgmap v1539: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:46 smithi022 ceph-mon[40250]: pgmap v1539: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:49:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:48 smithi122 ceph-mon[32255]: pgmap v1540: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:48 smithi022 ceph-mon[31360]: pgmap v1540: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:48 smithi022 ceph-mon[40250]: pgmap v1540: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:48.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:48.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:48.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3627300119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3627300119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3627300119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:50] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:49:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:50 smithi122 ceph-mon[32255]: pgmap v1541: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:50 smithi022 ceph-mon[31360]: pgmap v1541: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:50 smithi022 ceph-mon[40250]: pgmap v1541: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:52 smithi122 ceph-mon[32255]: pgmap v1542: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:52 smithi022 ceph-mon[31360]: pgmap v1542: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:52 smithi022 ceph-mon[40250]: pgmap v1542: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:49:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:49:53.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:53.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:53.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:53.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:49:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:54 smithi022 ceph-mon[31360]: pgmap v1543: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/435913760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:54 smithi022 ceph-mon[40250]: pgmap v1543: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/435913760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:54 smithi122 ceph-mon[32255]: pgmap v1543: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/435913760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:49:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:56 smithi122 ceph-mon[32255]: pgmap v1544: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:56.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:49:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:49:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:56 smithi022 ceph-mon[31360]: pgmap v1544: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:56 smithi022 ceph-mon[40250]: pgmap v1544: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:49:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:49:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:49:58 smithi122 ceph-mon[32255]: pgmap v1545: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:49:58 smithi022 ceph-mon[31360]: pgmap v1545: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:49:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:49:58 smithi022 ceph-mon[40250]: pgmap v1545: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:49:58.982 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:00] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:00 smithi122 ceph-mon[32255]: pgmap v1546: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2344933473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T18:50:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:00 smithi022 ceph-mon[31360]: pgmap v1546: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2344933473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T18:50:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:00 smithi022 ceph-mon[40250]: pgmap v1546: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2344933473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T18:50:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:02 smithi122 ceph-mon[32255]: pgmap v1547: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:02 smithi022 ceph-mon[31360]: pgmap v1547: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:02 smithi022 ceph-mon[40250]: pgmap v1547: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:50:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:50:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:04.354 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:04 smithi022 ceph-mon[31360]: pgmap v1548: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:04.355 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:04 smithi022 ceph-mon[40250]: pgmap v1548: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:04 smithi122 ceph-mon[32255]: pgmap v1548: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:04.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:04.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:04.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1158852688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:05 smithi122 ceph-mon[32255]: pgmap v1549: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1158852688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:05 smithi022 ceph-mon[31360]: pgmap v1549: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1158852688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:05 smithi022 ceph-mon[40250]: pgmap v1549: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:50:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:50:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:50:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:08 smithi022 ceph-mon[31360]: pgmap v1550: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:08 smithi022 ceph-mon[40250]: pgmap v1550: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:08 smithi122 ceph-mon[32255]: pgmap v1550: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:09.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:10.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:10.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:10.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:10 smithi022 ceph-mon[31360]: pgmap v1551: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:10 smithi022 ceph-mon[40250]: pgmap v1551: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:10] "GET /metrics HTTP/1.1" 200 202244 "" "Prometheus/2.18.1" 2021-12-15T18:50:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:10 smithi122 ceph-mon[32255]: pgmap v1551: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1856076899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1856076899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1856076899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:12 smithi022 ceph-mon[31360]: pgmap v1552: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:12 smithi022 ceph-mon[40250]: pgmap v1552: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:12 smithi122 ceph-mon[32255]: pgmap v1552: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:50:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:50:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:14 smithi022 ceph-mon[31360]: pgmap v1553: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:14 smithi022 ceph-mon[40250]: pgmap v1553: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:14 smithi122 ceph-mon[32255]: pgmap v1553: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:15.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:15.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:15.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:15.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:16 smithi022 ceph-mon[31360]: pgmap v1554: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3299076424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:16 smithi022 ceph-mon[40250]: pgmap v1554: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3299076424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:50:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:50:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:16 smithi122 ceph-mon[32255]: pgmap v1554: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3299076424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:50:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:18 smithi022 ceph-mon[31360]: pgmap v1555: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:18 smithi022 ceph-mon[40250]: pgmap v1555: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:18 smithi122 ceph-mon[32255]: pgmap v1555: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:20 smithi022 ceph-mon[31360]: pgmap v1556: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:20 smithi022 ceph-mon[40250]: pgmap v1556: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:20] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:50:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:20 smithi122 ceph-mon[32255]: pgmap v1556: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:20.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:20.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:20.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2496034688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:21.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2496034688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2496034688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:22.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:22 smithi122 ceph-mon[32255]: pgmap v1557: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:22 smithi022 ceph-mon[31360]: pgmap v1557: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:22 smithi022 ceph-mon[40250]: pgmap v1557: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:50:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:50:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:24 smithi122 ceph-mon[32255]: pgmap v1558: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:24 smithi022 ceph-mon[31360]: pgmap v1558: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:24 smithi022 ceph-mon[40250]: pgmap v1558: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:25.825 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:26.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:26.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:26.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:26.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:26 smithi022 ceph-mon[31360]: pgmap v1559: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:26.196 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:26 smithi022 ceph-mon[40250]: pgmap v1559: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:26 smithi122 ceph-mon[32255]: pgmap v1559: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:50:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:50:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3568045807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3568045807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3568045807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:50:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:28 smithi122 ceph-mon[32255]: pgmap v1560: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:28 smithi022 ceph-mon[31360]: pgmap v1560: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:28 smithi022 ceph-mon[40250]: pgmap v1560: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:30 smithi122 ceph-mon[32255]: pgmap v1561: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:30] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:50:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:30 smithi022 ceph-mon[31360]: pgmap v1561: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:30 smithi022 ceph-mon[40250]: pgmap v1561: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:31.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:31.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:31.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:31.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:32 smithi122 ceph-mon[32255]: pgmap v1562: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3462594404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:32 smithi022 ceph-mon[31360]: pgmap v1562: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3462594404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:32 smithi022 ceph-mon[40250]: pgmap v1562: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3462594404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:50:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:50:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:50:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:50:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:50:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:50:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:50:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:50:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:50:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:50:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:50:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:50:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:50:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:50:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:34 smithi122 ceph-mon[32255]: pgmap v1563: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:34 smithi022 ceph-mon[31360]: pgmap v1563: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:34 smithi022 ceph-mon[40250]: pgmap v1563: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:50:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:50:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:36 smithi122 ceph-mon[32255]: pgmap v1564: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:36 smithi022 ceph-mon[31360]: pgmap v1564: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:36 smithi022 ceph-mon[40250]: pgmap v1564: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:36.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:36.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:36.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:36.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1967819572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1967819572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1967819572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:50:38.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:38.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:38 smithi122 ceph-mon[32255]: pgmap v1565: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:38.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:38.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:38 smithi022 ceph-mon[31360]: pgmap v1565: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:38.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:38.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:38 smithi022 ceph-mon[40250]: pgmap v1565: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:50:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:50:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:50:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:50:39.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:50:39.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:50:39.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:50:39.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:50:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:50:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:50:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:50:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:50:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:40 smithi122 ceph-mon[32255]: pgmap v1566: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:40] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:50:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:40 smithi022 ceph-mon[31360]: pgmap v1566: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:40 smithi022 ceph-mon[40250]: pgmap v1566: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:41.935 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:42 smithi022 ceph-mon[31360]: pgmap v1567: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:42 smithi022 ceph-mon[40250]: pgmap v1567: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:42.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:42.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:42.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:42.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:42 smithi122 ceph-mon[32255]: pgmap v1567: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:50:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:50:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/391769987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/391769987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/391769987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:44 smithi022 ceph-mon[31360]: pgmap v1568: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:44 smithi022 ceph-mon[40250]: pgmap v1568: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:44 smithi122 ceph-mon[32255]: pgmap v1568: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:46 smithi022 ceph-mon[31360]: pgmap v1569: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:46 smithi022 ceph-mon[40250]: pgmap v1569: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:46 smithi122 ceph-mon[32255]: pgmap v1569: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:46.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:50:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:50:47.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:47.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:47.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:47.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:47.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:50:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:48 smithi022 ceph-mon[31360]: pgmap v1570: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1448042537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:48 smithi022 ceph-mon[40250]: pgmap v1570: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1448042537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:48 smithi122 ceph-mon[32255]: pgmap v1570: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1448042537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:50 smithi022 ceph-mon[31360]: pgmap v1571: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:50 smithi022 ceph-mon[40250]: pgmap v1571: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:50] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:50:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:50 smithi122 ceph-mon[32255]: pgmap v1571: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:52 smithi022 ceph-mon[31360]: pgmap v1572: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:52 smithi022 ceph-mon[40250]: pgmap v1572: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:52 smithi122 ceph-mon[32255]: pgmap v1572: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:52.676 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:50:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:50:53.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:53.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:53.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:54 smithi022 ceph-mon[31360]: pgmap v1573: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1065878580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:54 smithi022 ceph-mon[40250]: pgmap v1573: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1065878580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:54 smithi122 ceph-mon[32255]: pgmap v1573: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1065878580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:56 smithi122 ceph-mon[32255]: pgmap v1574: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:56.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:50:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:50:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:56 smithi022 ceph-mon[31360]: pgmap v1574: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:56 smithi022 ceph-mon[40250]: pgmap v1574: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:50:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:50:58.048 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:50:58.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:50:58.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:50:58.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:50:58.416 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:58.416 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:58 smithi022 ceph-mon[40250]: pgmap v1575: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:58.417 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:58.417 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:58 smithi022 ceph-mon[31360]: pgmap v1575: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:50:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:58 smithi122 ceph-mon[32255]: pgmap v1575: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:50:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:50:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/169749742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:59.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:50:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/169749742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:50:59.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:50:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/169749742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:00] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:51:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:00 smithi122 ceph-mon[32255]: pgmap v1576: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:00 smithi022 ceph-mon[31360]: pgmap v1576: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:00 smithi022 ceph-mon[40250]: pgmap v1576: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:02 smithi122 ceph-mon[32255]: pgmap v1577: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:02 smithi022 ceph-mon[31360]: pgmap v1577: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:02 smithi022 ceph-mon[40250]: pgmap v1577: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:51:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:51:03.417 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:03.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:03.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:03.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:04 smithi122 ceph-mon[32255]: pgmap v1578: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/555275463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:04 smithi022 ceph-mon[31360]: pgmap v1578: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/555275463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:04 smithi022 ceph-mon[40250]: pgmap v1578: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/555275463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:51:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:51:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:06 smithi122 ceph-mon[32255]: pgmap v1579: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:06 smithi022 ceph-mon[31360]: pgmap v1579: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:06 smithi022 ceph-mon[40250]: pgmap v1579: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:51:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:08 smithi122 ceph-mon[32255]: pgmap v1580: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:08 smithi022 ceph-mon[31360]: pgmap v1580: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:08 smithi022 ceph-mon[40250]: pgmap v1580: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:08.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:09.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:09.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:09.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:10] "GET /metrics HTTP/1.1" 200 202249 "" "Prometheus/2.18.1" 2021-12-15T18:51:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:10 smithi122 ceph-mon[32255]: pgmap v1581: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1782506646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:10 smithi022 ceph-mon[31360]: pgmap v1581: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1782506646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:10 smithi022 ceph-mon[40250]: pgmap v1581: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1782506646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:12 smithi122 ceph-mon[32255]: pgmap v1582: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:12 smithi022 ceph-mon[31360]: pgmap v1582: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:12 smithi022 ceph-mon[40250]: pgmap v1582: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:51:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:51:14.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:14 smithi122 ceph-mon[32255]: pgmap v1583: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:14 smithi022 ceph-mon[31360]: pgmap v1583: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:14 smithi022 ceph-mon[40250]: pgmap v1583: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:14.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:14.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:14.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2206055320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2206055320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2206055320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:16 smithi122 ceph-mon[32255]: pgmap v1584: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:16.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:51:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:51:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:16 smithi022 ceph-mon[31360]: pgmap v1584: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:16 smithi022 ceph-mon[40250]: pgmap v1584: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:51:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:18 smithi122 ceph-mon[32255]: pgmap v1585: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:18 smithi022 ceph-mon[31360]: pgmap v1585: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:18 smithi022 ceph-mon[40250]: pgmap v1585: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:19.538 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:19.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:19.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:19.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:20 smithi022 ceph-mon[31360]: pgmap v1586: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1750576542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:20 smithi022 ceph-mon[40250]: pgmap v1586: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1750576542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:20 smithi122 ceph-mon[32255]: pgmap v1586: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1750576542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:20] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:51:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:51:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:51:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:22 smithi122 ceph-mon[32255]: pgmap v1587: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:22 smithi022 ceph-mon[31360]: pgmap v1587: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:22 smithi022 ceph-mon[40250]: pgmap v1587: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:23 smithi122 ceph-mon[32255]: pgmap v1588: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:23 smithi022 ceph-mon[31360]: pgmap v1588: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:23 smithi022 ceph-mon[40250]: pgmap v1588: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:24.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:25.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:25.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:25.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:26 smithi122 ceph-mon[32255]: pgmap v1589: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3372760833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:51:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:51:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:26 smithi022 ceph-mon[31360]: pgmap v1589: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3372760833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:26 smithi022 ceph-mon[40250]: pgmap v1589: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3372760833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:51:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:28 smithi122 ceph-mon[32255]: pgmap v1590: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:28 smithi022 ceph-mon[31360]: pgmap v1590: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:28 smithi022 ceph-mon[40250]: pgmap v1590: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:30.278 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:30 smithi122 ceph-mon[32255]: pgmap v1591: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:30.483 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:30] "GET /metrics HTTP/1.1" 200 202243 "" "Prometheus/2.18.1" 2021-12-15T18:51:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:30 smithi022 ceph-mon[31360]: pgmap v1591: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:30 smithi022 ceph-mon[40250]: pgmap v1591: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:30.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:30.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:30.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1387076934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1387076934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1387076934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:32 smithi122 ceph-mon[32255]: pgmap v1592: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:51:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:51:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:32 smithi022 ceph-mon[31360]: pgmap v1592: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:32 smithi022 ceph-mon[40250]: pgmap v1592: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:51:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:51:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:51:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:51:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:51:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:51:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:51:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:51:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:51:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:51:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:51:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:51:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:34 smithi122 ceph-mon[32255]: pgmap v1593: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:34 smithi022 ceph-mon[31360]: pgmap v1593: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:34 smithi022 ceph-mon[40250]: pgmap v1593: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:35 smithi122 ceph-mon[32255]: pgmap v1594: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:35 smithi022 ceph-mon[31360]: pgmap v1594: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:35 smithi022 ceph-mon[40250]: pgmap v1594: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:36.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:36.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:36.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1348184985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1348184985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:51:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:51:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1348184985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:37 smithi022 ceph-mon[31360]: pgmap v1595: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:37 smithi022 ceph-mon[40250]: pgmap v1595: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:37 smithi122 ceph-mon[32255]: pgmap v1595: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:51:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:51:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:51:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:51:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:51:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:51:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:51:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:51:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:51:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:51:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:39 smithi122 ceph-mon[32255]: pgmap v1596: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:51:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:39 smithi022 ceph-mon[31360]: pgmap v1596: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:51:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:39 smithi022 ceph-mon[40250]: pgmap v1596: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:51:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:40] "GET /metrics HTTP/1.1" 200 202251 "" "Prometheus/2.18.1" 2021-12-15T18:51:41.014 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:41.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:41.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:41.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:42 smithi122 ceph-mon[32255]: pgmap v1597: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2185597254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:42 smithi022 ceph-mon[31360]: pgmap v1597: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2185597254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:42 smithi022 ceph-mon[40250]: pgmap v1597: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2185597254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:51:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:51:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:44 smithi122 ceph-mon[32255]: pgmap v1598: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:44 smithi022 ceph-mon[31360]: pgmap v1598: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:44 smithi022 ceph-mon[40250]: pgmap v1598: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:46.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:46.390 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:46 smithi022 ceph-mon[31360]: pgmap v1599: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:46.390 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:46 smithi022 ceph-mon[40250]: pgmap v1599: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:46 smithi122 ceph-mon[32255]: pgmap v1599: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:46.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:51:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:51:46.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:46.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:46.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/223258110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/223258110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/223258110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:51:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:48 smithi122 ceph-mon[32255]: pgmap v1600: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:48 smithi022 ceph-mon[31360]: pgmap v1600: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:48 smithi022 ceph-mon[40250]: pgmap v1600: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:49 smithi122 ceph-mon[32255]: pgmap v1601: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:49 smithi022 ceph-mon[31360]: pgmap v1601: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:49 smithi022 ceph-mon[40250]: pgmap v1601: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:50] "GET /metrics HTTP/1.1" 200 202256 "" "Prometheus/2.18.1" 2021-12-15T18:51:51.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:52.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:52.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:52.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:52 smithi122 ceph-mon[32255]: pgmap v1602: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:51:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:51:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:52 smithi022 ceph-mon[31360]: pgmap v1602: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:52 smithi022 ceph-mon[40250]: pgmap v1602: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301984732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301984732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301984732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:54 smithi122 ceph-mon[32255]: pgmap v1603: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:54 smithi022 ceph-mon[31360]: pgmap v1603: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:54 smithi022 ceph-mon[40250]: pgmap v1603: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:56 smithi122 ceph-mon[32255]: pgmap v1604: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:56.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:51:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:51:56.634 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:56 smithi022 ceph-mon[31360]: pgmap v1604: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:56.635 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:56 smithi022 ceph-mon[40250]: pgmap v1604: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:57.123 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:51:57.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:51:57.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:51:57.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:51:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:51:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:58 smithi122 ceph-mon[32255]: pgmap v1605: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4229627392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:58 smithi022 ceph-mon[31360]: pgmap v1605: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4229627392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:51:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:58 smithi022 ceph-mon[40250]: pgmap v1605: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:51:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4229627392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:51:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:51:59 smithi122 ceph-mon[32255]: pgmap v1606: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:51:59 smithi022 ceph-mon[31360]: pgmap v1606: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:51:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:51:59 smithi022 ceph-mon[40250]: pgmap v1606: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:00] "GET /metrics HTTP/1.1" 200 202256 "" "Prometheus/2.18.1" 2021-12-15T18:52:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:02 smithi122 ceph-mon[32255]: pgmap v1607: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:52:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:52:02.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:02.494 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:02 smithi022 ceph-mon[31360]: pgmap v1607: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:02.495 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:02 smithi022 ceph-mon[40250]: pgmap v1607: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:02.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:02.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:02.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3194199655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3194199655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3194199655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:04 smithi122 ceph-mon[32255]: pgmap v1608: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:04 smithi022 ceph-mon[31360]: pgmap v1608: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:04 smithi022 ceph-mon[40250]: pgmap v1608: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:06 smithi122 ceph-mon[32255]: pgmap v1609: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:06.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:52:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:52:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:06 smithi022 ceph-mon[31360]: pgmap v1609: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:06 smithi022 ceph-mon[40250]: pgmap v1609: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:52:07.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:08.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:08.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:08.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:08.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:08.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:08 smithi022 ceph-mon[31360]: pgmap v1610: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:08.232 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:08.232 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:08 smithi022 ceph-mon[40250]: pgmap v1610: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:08 smithi122 ceph-mon[32255]: pgmap v1610: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1564177622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:09 smithi122 ceph-mon[32255]: pgmap v1611: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1564177622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:09 smithi022 ceph-mon[31360]: pgmap v1611: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1564177622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:09 smithi022 ceph-mon[40250]: pgmap v1611: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:10] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:52:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:12 smithi122 ceph-mon[32255]: pgmap v1612: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:52:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:52:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:12 smithi022 ceph-mon[31360]: pgmap v1612: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:12 smithi022 ceph-mon[40250]: pgmap v1612: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:13.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:13.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:13.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:14 smithi122 ceph-mon[32255]: pgmap v1613: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/547306031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:14 smithi022 ceph-mon[31360]: pgmap v1613: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/547306031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:14 smithi022 ceph-mon[40250]: pgmap v1613: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/547306031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:52:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:52:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:16 smithi122 ceph-mon[32255]: pgmap v1614: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:16 smithi022 ceph-mon[31360]: pgmap v1614: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:16 smithi022 ceph-mon[40250]: pgmap v1614: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:52:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:18 smithi122 ceph-mon[32255]: pgmap v1615: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:18 smithi022 ceph-mon[31360]: pgmap v1615: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:18 smithi022 ceph-mon[40250]: pgmap v1615: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:18.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:18.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:18.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:18.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/907563763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:19 smithi022 ceph-mon[31360]: pgmap v1616: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/907563763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:19 smithi022 ceph-mon[40250]: pgmap v1616: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/907563763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:19 smithi122 ceph-mon[32255]: pgmap v1616: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:20] "GET /metrics HTTP/1.1" 200 202254 "" "Prometheus/2.18.1" 2021-12-15T18:52:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:22 smithi122 ceph-mon[32255]: pgmap v1617: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:52:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:52:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:22 smithi022 ceph-mon[31360]: pgmap v1617: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:22 smithi022 ceph-mon[40250]: pgmap v1617: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:23.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:24 smithi022 ceph-mon[31360]: pgmap v1618: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:24 smithi022 ceph-mon[40250]: pgmap v1618: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:24.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:24.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:24.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:24 smithi122 ceph-mon[32255]: pgmap v1618: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4207344401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4207344401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4207344401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:26 smithi122 ceph-mon[32255]: pgmap v1619: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:26.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:52:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:52:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:26 smithi022 ceph-mon[31360]: pgmap v1619: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:26 smithi022 ceph-mon[40250]: pgmap v1619: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:27 smithi022 ceph-mon[31360]: pgmap v1620: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:27 smithi022 ceph-mon[40250]: pgmap v1620: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:27 smithi122 ceph-mon[32255]: pgmap v1620: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:52:29.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:29.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:29.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:29.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:30] "GET /metrics HTTP/1.1" 200 202254 "" "Prometheus/2.18.1" 2021-12-15T18:52:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:30 smithi122 ceph-mon[32255]: pgmap v1621: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/203409924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:30 smithi022 ceph-mon[31360]: pgmap v1621: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/203409924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:30 smithi022 ceph-mon[40250]: pgmap v1621: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/203409924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:32.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:32 smithi122 ceph-mon[32255]: pgmap v1622: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:32.439 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:52:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:52:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:32 smithi022 ceph-mon[31360]: pgmap v1622: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:32 smithi022 ceph-mon[40250]: pgmap v1622: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:52:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:52:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:52:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:52:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:33 smithi122 ceph-mon[32255]: pgmap v1623: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:52:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:52:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:52:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:52:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[31360]: pgmap v1623: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:52:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:52:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:52:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:52:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:33 smithi022 ceph-mon[40250]: pgmap v1623: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:34.713 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:35.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:35.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:35.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:35 smithi122 ceph-mon[32255]: pgmap v1624: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1604840464' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:35 smithi022 ceph-mon[31360]: pgmap v1624: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1604840464' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:35 smithi022 ceph-mon[40250]: pgmap v1624: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1604840464' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:36.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:52:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:52:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:52:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:38 smithi122 ceph-mon[32255]: pgmap v1625: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:38 smithi022 ceph-mon[31360]: pgmap v1625: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:38 smithi022 ceph-mon[40250]: pgmap v1625: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:40] "GET /metrics HTTP/1.1" 200 202252 "" "Prometheus/2.18.1" 2021-12-15T18:52:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:40 smithi122 ceph-mon[32255]: pgmap v1626: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:52:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:52:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:52:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:52:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:40.444 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[31360]: pgmap v1626: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:40.444 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:52:40.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:52:40.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:52:40.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:52:40.445 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[40250]: pgmap v1626: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:40.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:52:40.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:52:40.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:52:40.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:52:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1726801731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1726801731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1726801731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:42.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:42 smithi122 ceph-mon[32255]: pgmap v1627: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:42 smithi022 ceph-mon[31360]: pgmap v1627: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:42 smithi022 ceph-mon[40250]: pgmap v1627: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:52:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:52:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:44 smithi122 ceph-mon[32255]: pgmap v1628: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:44 smithi022 ceph-mon[31360]: pgmap v1628: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:44 smithi022 ceph-mon[40250]: pgmap v1628: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:45.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:45.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:45.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:45.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:46 smithi022 ceph-mon[31360]: pgmap v1629: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/586220468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:46 smithi022 ceph-mon[40250]: pgmap v1629: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/586220468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:52:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:52:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:46 smithi122 ceph-mon[32255]: pgmap v1629: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/586220468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:52:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:48 smithi122 ceph-mon[32255]: pgmap v1630: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:48 smithi022 ceph-mon[31360]: pgmap v1630: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:48 smithi022 ceph-mon[40250]: pgmap v1630: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:50 smithi122 ceph-mon[32255]: pgmap v1631: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:50] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:52:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:50 smithi022 ceph-mon[31360]: pgmap v1631: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:50 smithi022 ceph-mon[40250]: pgmap v1631: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:50.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:51.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:51.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:51.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:52 smithi122 ceph-mon[32255]: pgmap v1632: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1443222780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:52 smithi022 ceph-mon[31360]: pgmap v1632: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1443222780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:52 smithi022 ceph-mon[40250]: pgmap v1632: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1443222780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:52:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:52:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:54 smithi122 ceph-mon[32255]: pgmap v1633: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:54 smithi022 ceph-mon[31360]: pgmap v1633: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:54 smithi022 ceph-mon[40250]: pgmap v1633: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:56.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:52:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:56 smithi122 ceph-mon[32255]: pgmap v1634: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:56.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:52:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:52:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:56 smithi022 ceph-mon[31360]: pgmap v1634: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:56 smithi022 ceph-mon[40250]: pgmap v1634: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:52:56.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:52:56.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:52:56.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:52:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3276003900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3276003900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3276003900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:52:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:52:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:52:58 smithi122 ceph-mon[32255]: pgmap v1635: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:52:58 smithi022 ceph-mon[31360]: pgmap v1635: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:52:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:52:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:52:58 smithi022 ceph-mon[40250]: pgmap v1635: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:00 smithi122 ceph-mon[32255]: pgmap v1636: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:00] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:53:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:00 smithi022 ceph-mon[31360]: pgmap v1636: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:00 smithi022 ceph-mon[40250]: pgmap v1636: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:01.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:01.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:01.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:01.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:02 smithi022 ceph-mon[31360]: pgmap v1637: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/132033080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:02 smithi022 ceph-mon[40250]: pgmap v1637: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/132033080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:02 smithi122 ceph-mon[32255]: pgmap v1637: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/132033080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:53:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:53:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:04 smithi122 ceph-mon[32255]: pgmap v1638: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:04 smithi022 ceph-mon[31360]: pgmap v1638: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:04 smithi022 ceph-mon[40250]: pgmap v1638: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:53:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:53:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:06 smithi122 ceph-mon[32255]: pgmap v1639: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:06 smithi022 ceph-mon[31360]: pgmap v1639: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:06 smithi022 ceph-mon[40250]: pgmap v1639: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:06.930 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:07.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:07.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:07.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:53:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:08 smithi122 ceph-mon[32255]: pgmap v1640: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3179176070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:08 smithi022 ceph-mon[31360]: pgmap v1640: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3179176070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:08 smithi022 ceph-mon[40250]: pgmap v1640: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3179176070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:10 smithi122 ceph-mon[32255]: pgmap v1641: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:10] "GET /metrics HTTP/1.1" 200 202254 "" "Prometheus/2.18.1" 2021-12-15T18:53:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:10 smithi022 ceph-mon[31360]: pgmap v1641: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:10 smithi022 ceph-mon[40250]: pgmap v1641: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:12.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:12 smithi122 ceph-mon[32255]: pgmap v1642: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:53:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:53:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:12 smithi022 ceph-mon[31360]: pgmap v1642: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:12 smithi022 ceph-mon[40250]: pgmap v1642: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:12.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:12.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:12.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3481129370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:13.623 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3481129370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:13.624 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3481129370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:14 smithi122 ceph-mon[32255]: pgmap v1643: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:14 smithi022 ceph-mon[31360]: pgmap v1643: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:14 smithi022 ceph-mon[40250]: pgmap v1643: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:15 smithi122 ceph-mon[32255]: pgmap v1644: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:15 smithi022 ceph-mon[31360]: pgmap v1644: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:15 smithi022 ceph-mon[40250]: pgmap v1644: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:16.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:53:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:53:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:17.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:53:18.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:18.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:18.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:18 smithi122 ceph-mon[32255]: pgmap v1645: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/196287270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:18 smithi022 ceph-mon[31360]: pgmap v1645: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/196287270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:18 smithi022 ceph-mon[40250]: pgmap v1645: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/196287270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:20] "GET /metrics HTTP/1.1" 200 202252 "" "Prometheus/2.18.1" 2021-12-15T18:53:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:20 smithi122 ceph-mon[32255]: pgmap v1646: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:20 smithi022 ceph-mon[31360]: pgmap v1646: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:20 smithi022 ceph-mon[40250]: pgmap v1646: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:22 smithi122 ceph-mon[32255]: pgmap v1647: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:22 smithi022 ceph-mon[31360]: pgmap v1647: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:22 smithi022 ceph-mon[40250]: pgmap v1647: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:53:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:53:23.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:23.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:23.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:23.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:24 smithi122 ceph-mon[32255]: pgmap v1648: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/395768902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:24 smithi022 ceph-mon[31360]: pgmap v1648: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/395768902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:24 smithi022 ceph-mon[40250]: pgmap v1648: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/395768902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:26.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:26 smithi122 ceph-mon[32255]: pgmap v1649: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:26 smithi022 ceph-mon[31360]: pgmap v1649: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:26 smithi022 ceph-mon[40250]: pgmap v1649: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:26.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:53:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:53:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:53:28.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:28.420 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:28.421 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:28 smithi022 ceph-mon[31360]: pgmap v1650: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:28.421 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:28.421 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:28 smithi022 ceph-mon[40250]: pgmap v1650: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:28 smithi122 ceph-mon[32255]: pgmap v1650: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:28.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:28.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:28.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3363930443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3363930443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3363930443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:30] "GET /metrics HTTP/1.1" 200 202252 "" "Prometheus/2.18.1" 2021-12-15T18:53:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:30 smithi122 ceph-mon[32255]: pgmap v1651: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:30 smithi022 ceph-mon[31360]: pgmap v1651: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:30 smithi022 ceph-mon[40250]: pgmap v1651: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:32 smithi122 ceph-mon[32255]: pgmap v1652: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:53:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:53:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:32 smithi022 ceph-mon[31360]: pgmap v1652: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:32 smithi022 ceph-mon[40250]: pgmap v1652: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:53:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:53:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:53:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:53:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:53:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:53:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:53:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:53:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:53:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:53:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:53:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:53:33.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:34.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:34.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:34.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:34 smithi122 ceph-mon[32255]: pgmap v1653: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:34 smithi022 ceph-mon[31360]: pgmap v1653: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:34 smithi022 ceph-mon[40250]: pgmap v1653: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/498775494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/498775494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/498775494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:53:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:53:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:36 smithi122 ceph-mon[32255]: pgmap v1654: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:36 smithi022 ceph-mon[31360]: pgmap v1654: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:36 smithi022 ceph-mon[40250]: pgmap v1654: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:53:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:38 smithi122 ceph-mon[32255]: pgmap v1655: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:38 smithi022 ceph-mon[31360]: pgmap v1655: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:38 smithi022 ceph-mon[40250]: pgmap v1655: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:39.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:40] "GET /metrics HTTP/1.1" 200 202242 "" "Prometheus/2.18.1" 2021-12-15T18:53:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:40 smithi122 ceph-mon[32255]: pgmap v1656: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1633349926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:53:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:53:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:53:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[40250]: pgmap v1656: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1633349926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:53:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:53:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:53:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[31360]: pgmap v1656: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1633349926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:53:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:53:40.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:53:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:53:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:41 smithi022 ceph-mon[31360]: pgmap v1657: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:53:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:41 smithi022 ceph-mon[40250]: pgmap v1657: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:53:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:41 smithi122 ceph-mon[32255]: pgmap v1657: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:53:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:53:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:43 smithi122 ceph-mon[32255]: pgmap v1658: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:43 smithi022 ceph-mon[31360]: pgmap v1658: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:43 smithi022 ceph-mon[40250]: pgmap v1658: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:44.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:44.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:44.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:44.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2138463507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2138463507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2138463507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:45 smithi022 ceph-mon[31360]: pgmap v1659: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:45 smithi022 ceph-mon[40250]: pgmap v1659: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:46.331 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:45 smithi122 ceph-mon[32255]: pgmap v1659: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:46.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:53:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:53:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:53:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:48 smithi122 ceph-mon[32255]: pgmap v1660: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:48 smithi022 ceph-mon[31360]: pgmap v1660: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:48 smithi022 ceph-mon[40250]: pgmap v1660: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:49.905 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:50.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:50.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:50.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:50.275 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:50 smithi022 ceph-mon[31360]: pgmap v1661: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:50.275 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:50 smithi022 ceph-mon[40250]: pgmap v1661: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:50 smithi122 ceph-mon[32255]: pgmap v1661: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:50] "GET /metrics HTTP/1.1" 200 202254 "" "Prometheus/2.18.1" 2021-12-15T18:53:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2221698435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2221698435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2221698435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:52 smithi122 ceph-mon[32255]: pgmap v1662: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:52 smithi022 ceph-mon[31360]: pgmap v1662: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:52 smithi022 ceph-mon[40250]: pgmap v1662: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:53:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:53:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:54 smithi122 ceph-mon[32255]: pgmap v1663: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:54 smithi022 ceph-mon[31360]: pgmap v1663: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:54 smithi022 ceph-mon[40250]: pgmap v1663: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:55.274 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:53:55.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:53:55.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:53:55.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:53:56.342 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:56 smithi122 ceph-mon[32255]: pgmap v1664: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:56.342 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/847793910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:56 smithi022 ceph-mon[31360]: pgmap v1664: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/847793910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:56 smithi022 ceph-mon[40250]: pgmap v1664: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:53:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/847793910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:53:56.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:53:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:53:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:53:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:53:58 smithi122 ceph-mon[32255]: pgmap v1665: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:53:58 smithi022 ceph-mon[31360]: pgmap v1665: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:53:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:53:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:53:58 smithi022 ceph-mon[40250]: pgmap v1665: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:00 smithi122 ceph-mon[32255]: pgmap v1666: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:00] "GET /metrics HTTP/1.1" 200 202254 "" "Prometheus/2.18.1" 2021-12-15T18:54:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:00 smithi022 ceph-mon[31360]: pgmap v1666: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:00 smithi022 ceph-mon[40250]: pgmap v1666: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:00.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:01.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:01.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:01.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/208880414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/208880414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/208880414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:02 smithi122 ceph-mon[32255]: pgmap v1667: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:54:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:54:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:02 smithi022 ceph-mon[31360]: pgmap v1667: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:02 smithi022 ceph-mon[40250]: pgmap v1667: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:04 smithi122 ceph-mon[32255]: pgmap v1668: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:04 smithi022 ceph-mon[31360]: pgmap v1668: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:04 smithi022 ceph-mon[40250]: pgmap v1668: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:06.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:06 smithi022 ceph-mon[31360]: pgmap v1669: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:06 smithi022 ceph-mon[40250]: pgmap v1669: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:06.343 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:06 smithi122 ceph-mon[32255]: pgmap v1669: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:06.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:06.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:06.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:06.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:54:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:54:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1209598618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1209598618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1209598618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:54:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:08 smithi122 ceph-mon[32255]: pgmap v1670: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:08 smithi022 ceph-mon[31360]: pgmap v1670: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:08 smithi022 ceph-mon[40250]: pgmap v1670: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:10] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:54:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:10 smithi122 ceph-mon[32255]: pgmap v1671: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:10 smithi022 ceph-mon[31360]: pgmap v1671: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:10 smithi022 ceph-mon[40250]: pgmap v1671: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:11.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:11.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:11.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:12 smithi122 ceph-mon[32255]: pgmap v1672: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/330006048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:54:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:54:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:12 smithi022 ceph-mon[31360]: pgmap v1672: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/330006048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:12 smithi022 ceph-mon[40250]: pgmap v1672: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/330006048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:14 smithi122 ceph-mon[32255]: pgmap v1673: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:14 smithi022 ceph-mon[31360]: pgmap v1673: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:14 smithi022 ceph-mon[40250]: pgmap v1673: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:15 smithi122 ceph-mon[32255]: pgmap v1674: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:15 smithi022 ceph-mon[31360]: pgmap v1674: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:15 smithi022 ceph-mon[40250]: pgmap v1674: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:16.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:54:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:54:16.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:17.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:17.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:17.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:54:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:18 smithi122 ceph-mon[32255]: pgmap v1675: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1748534502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:18 smithi022 ceph-mon[31360]: pgmap v1675: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1748534502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:18 smithi022 ceph-mon[40250]: pgmap v1675: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1748534502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:20] "GET /metrics HTTP/1.1" 200 202249 "" "Prometheus/2.18.1" 2021-12-15T18:54:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:20 smithi122 ceph-mon[32255]: pgmap v1676: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:20 smithi022 ceph-mon[31360]: pgmap v1676: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:20 smithi022 ceph-mon[40250]: pgmap v1676: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:22.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:22 smithi122 ceph-mon[32255]: pgmap v1677: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:54:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:54:22.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:22.496 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:22 smithi022 ceph-mon[31360]: pgmap v1677: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:22.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:22 smithi022 ceph-mon[40250]: pgmap v1677: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3401267461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3401267461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3401267461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:24 smithi122 ceph-mon[32255]: pgmap v1678: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:24 smithi022 ceph-mon[31360]: pgmap v1678: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:24 smithi022 ceph-mon[40250]: pgmap v1678: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:26.365 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:26 smithi122 ceph-mon[32255]: pgmap v1679: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:26 smithi022 ceph-mon[31360]: pgmap v1679: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:26 smithi022 ceph-mon[40250]: pgmap v1679: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:26.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:54:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:54:27.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:54:27.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:27.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:27.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:28 smithi022 ceph-mon[31360]: pgmap v1680: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2184853353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:28 smithi022 ceph-mon[40250]: pgmap v1680: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2184853353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:28 smithi122 ceph-mon[32255]: pgmap v1680: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2184853353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:30] "GET /metrics HTTP/1.1" 200 202249 "" "Prometheus/2.18.1" 2021-12-15T18:54:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:30 smithi122 ceph-mon[32255]: pgmap v1681: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:30 smithi022 ceph-mon[31360]: pgmap v1681: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:30 smithi022 ceph-mon[40250]: pgmap v1681: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:32 smithi122 ceph-mon[32255]: pgmap v1682: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:54:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:54:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:32 smithi022 ceph-mon[31360]: pgmap v1682: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:32 smithi022 ceph-mon[40250]: pgmap v1682: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:32.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:33.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:33.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:33.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:33.242 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:54:33.243 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:54:33.243 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:54:33.243 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:54:33.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:54:33.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:54:33.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:54:33.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:54:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:54:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:54:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:54:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:54:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:34 smithi122 ceph-mon[32255]: pgmap v1683: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/303271940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:34 smithi022 ceph-mon[31360]: pgmap v1683: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/303271940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:34 smithi022 ceph-mon[40250]: pgmap v1683: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/303271940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:36.375 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:36 smithi122 ceph-mon[32255]: pgmap v1684: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:36 smithi022 ceph-mon[31360]: pgmap v1684: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:36 smithi022 ceph-mon[40250]: pgmap v1684: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:36.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:54:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:54:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:54:38.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:38 smithi122 ceph-mon[32255]: pgmap v1685: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:38 smithi022 ceph-mon[31360]: pgmap v1685: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:38 smithi022 ceph-mon[40250]: pgmap v1685: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:38.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:38.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:38.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3222857844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3222857844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3222857844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:40 smithi122 ceph-mon[32255]: pgmap v1686: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:40] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:54:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:40 smithi022 ceph-mon[31360]: pgmap v1686: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:40 smithi022 ceph-mon[40250]: pgmap v1686: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:54:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:54:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:54:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:54:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:54:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:54:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:54:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:54:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:54:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:54:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:54:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:54:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:42 smithi122 ceph-mon[32255]: pgmap v1687: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:54:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:54:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:42 smithi022 ceph-mon[31360]: pgmap v1687: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:42 smithi022 ceph-mon[40250]: pgmap v1687: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:43.613 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:43.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:43.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:43.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:44 smithi022 ceph-mon[31360]: pgmap v1688: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2438375355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:44 smithi022 ceph-mon[40250]: pgmap v1688: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2438375355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:44 smithi122 ceph-mon[32255]: pgmap v1688: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2438375355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:45 smithi122 ceph-mon[32255]: pgmap v1689: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:45 smithi022 ceph-mon[31360]: pgmap v1689: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:45 smithi022 ceph-mon[40250]: pgmap v1689: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:46.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:54:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:54:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:47 smithi122 ceph-mon[32255]: pgmap v1690: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:54:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:47 smithi022 ceph-mon[31360]: pgmap v1690: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:47 smithi022 ceph-mon[40250]: pgmap v1690: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:48.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:49 smithi022 ceph-mon[31360]: pgmap v1691: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:49 smithi022 ceph-mon[40250]: pgmap v1691: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:49.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:49.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:49.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:49 smithi122 ceph-mon[32255]: pgmap v1691: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/695125810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:50] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:54:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/695125810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/695125810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:52 smithi122 ceph-mon[32255]: pgmap v1692: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:54:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:54:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:52 smithi022 ceph-mon[31360]: pgmap v1692: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:52 smithi022 ceph-mon[40250]: pgmap v1692: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:54.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:54:54.356 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:54 smithi022 ceph-mon[31360]: pgmap v1693: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:54.356 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:54 smithi022 ceph-mon[40250]: pgmap v1693: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:54 smithi122 ceph-mon[32255]: pgmap v1693: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:54.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:54:54.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:54:54.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:54:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4168072833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4168072833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4168072833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:54:56.397 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:56 smithi122 ceph-mon[32255]: pgmap v1694: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:56 smithi022 ceph-mon[31360]: pgmap v1694: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:56 smithi022 ceph-mon[40250]: pgmap v1694: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:54:56.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:54:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:54:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:54:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:54:58 smithi122 ceph-mon[32255]: pgmap v1695: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:54:58 smithi022 ceph-mon[31360]: pgmap v1695: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:54:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:54:58 smithi022 ceph-mon[40250]: pgmap v1695: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:54:59.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:00.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:00.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:00.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:00] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:55:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:00 smithi122 ceph-mon[32255]: pgmap v1696: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:00 smithi022 ceph-mon[31360]: pgmap v1696: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:00 smithi022 ceph-mon[40250]: pgmap v1696: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/681520085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/681520085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/681520085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:02 smithi122 ceph-mon[32255]: pgmap v1697: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:02 smithi022 ceph-mon[31360]: pgmap v1697: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:02 smithi022 ceph-mon[40250]: pgmap v1697: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:55:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:55:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:04 smithi122 ceph-mon[32255]: pgmap v1698: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:04 smithi022 ceph-mon[31360]: pgmap v1698: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:04 smithi022 ceph-mon[40250]: pgmap v1698: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:05.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:05.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:05.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:05.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:06.402 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:06 smithi122 ceph-mon[32255]: pgmap v1699: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:06.402 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/579562184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:06 smithi022 ceph-mon[31360]: pgmap v1699: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/579562184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:06 smithi022 ceph-mon[40250]: pgmap v1699: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/579562184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:06.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:55:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:55:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:55:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:08 smithi122 ceph-mon[32255]: pgmap v1700: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:08 smithi022 ceph-mon[31360]: pgmap v1700: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:08 smithi022 ceph-mon[40250]: pgmap v1700: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:10] "GET /metrics HTTP/1.1" 200 202256 "" "Prometheus/2.18.1" 2021-12-15T18:55:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:10 smithi122 ceph-mon[32255]: pgmap v1701: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:10.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:10 smithi022 ceph-mon[40250]: pgmap v1701: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:10.473 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:10 smithi022 ceph-mon[31360]: pgmap v1701: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:10.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3821433936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3821433936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3821433936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:55:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:55:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:12 smithi122 ceph-mon[32255]: pgmap v1702: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:12 smithi022 ceph-mon[31360]: pgmap v1702: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:12 smithi022 ceph-mon[40250]: pgmap v1702: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:14 smithi122 ceph-mon[32255]: pgmap v1703: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:14 smithi022 ceph-mon[31360]: pgmap v1703: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:14 smithi022 ceph-mon[40250]: pgmap v1703: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:15.843 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:16.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:16.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:16.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:16.403 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:16 smithi122 ceph-mon[32255]: pgmap v1704: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:16 smithi022 ceph-mon[31360]: pgmap v1704: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:16 smithi022 ceph-mon[40250]: pgmap v1704: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:16.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:55:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:55:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4241248002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4241248002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4241248002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:55:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:18 smithi122 ceph-mon[32255]: pgmap v1705: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:18 smithi022 ceph-mon[31360]: pgmap v1705: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:18 smithi022 ceph-mon[40250]: pgmap v1705: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:19 smithi122 ceph-mon[32255]: pgmap v1706: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:19 smithi022 ceph-mon[31360]: pgmap v1706: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:19 smithi022 ceph-mon[40250]: pgmap v1706: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:20] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:55:21.214 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:21.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:21.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:21.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:22 smithi122 ceph-mon[32255]: pgmap v1707: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/6198145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:55:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:55:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:22 smithi022 ceph-mon[31360]: pgmap v1707: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/6198145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:22 smithi022 ceph-mon[40250]: pgmap v1707: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/6198145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:23 smithi122 ceph-mon[32255]: pgmap v1708: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:23 smithi022 ceph-mon[31360]: pgmap v1708: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:23 smithi022 ceph-mon[40250]: pgmap v1708: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:25 smithi122 ceph-mon[32255]: pgmap v1709: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:25 smithi022 ceph-mon[31360]: pgmap v1709: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:25 smithi022 ceph-mon[40250]: pgmap v1709: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:26.582 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:26.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:55:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:55:26.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:26.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:26.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3995797952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3995797952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3995797952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:55:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:28 smithi122 ceph-mon[32255]: pgmap v1710: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:28 smithi022 ceph-mon[31360]: pgmap v1710: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:28 smithi022 ceph-mon[40250]: pgmap v1710: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:30 smithi122 ceph-mon[32255]: pgmap v1711: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:30] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:55:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:30 smithi022 ceph-mon[31360]: pgmap v1711: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:30 smithi022 ceph-mon[40250]: pgmap v1711: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:31.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:32 smithi022 ceph-mon[31360]: pgmap v1712: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:32 smithi022 ceph-mon[40250]: pgmap v1712: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:32.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:32.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:32.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:32 smithi122 ceph-mon[32255]: pgmap v1712: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:55:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:55:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/275977415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:55:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:55:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:55:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:55:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/275977415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:55:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:55:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:55:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:55:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/275977415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:55:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:55:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:55:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:55:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:34 smithi122 ceph-mon[32255]: pgmap v1713: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:34 smithi022 ceph-mon[31360]: pgmap v1713: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:34 smithi022 ceph-mon[40250]: pgmap v1713: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:36.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:36 smithi122 ceph-mon[32255]: pgmap v1714: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:36 smithi022 ceph-mon[31360]: pgmap v1714: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:36 smithi022 ceph-mon[40250]: pgmap v1714: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:36.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:55:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:55:37.327 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:37.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:37.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:37.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:37.696 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:55:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:38 smithi122 ceph-mon[32255]: pgmap v1715: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2390707173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:38 smithi022 ceph-mon[31360]: pgmap v1715: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2390707173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:38 smithi022 ceph-mon[40250]: pgmap v1715: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2390707173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:40] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:55:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:40 smithi122 ceph-mon[32255]: pgmap v1716: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:40 smithi022 ceph-mon[31360]: pgmap v1716: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:40 smithi022 ceph-mon[40250]: pgmap v1716: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:55:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:55:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:55:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:41.522 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:55:41.523 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:55:41.523 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:55:41.523 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:41.524 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:55:41.524 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:55:41.524 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:55:41.524 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:42 smithi122 ceph-mon[32255]: pgmap v1717: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:55:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:55:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:42 smithi022 ceph-mon[31360]: pgmap v1717: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:42 smithi022 ceph-mon[40250]: pgmap v1717: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:55:42.698 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:43.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:43.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:43.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/531926834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/531926834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/531926834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:44 smithi122 ceph-mon[32255]: pgmap v1718: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:44 smithi022 ceph-mon[31360]: pgmap v1718: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:44 smithi022 ceph-mon[40250]: pgmap v1718: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:46.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:46 smithi122 ceph-mon[32255]: pgmap v1719: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:46 smithi022 ceph-mon[31360]: pgmap v1719: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:46 smithi022 ceph-mon[40250]: pgmap v1719: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:46.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:55:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:55:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:55:48.067 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:48 smithi122 ceph-mon[32255]: pgmap v1720: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:48.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:48.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:48.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:48.442 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:48.442 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:48 smithi022 ceph-mon[31360]: pgmap v1720: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:48.442 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:48.443 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:48 smithi022 ceph-mon[40250]: pgmap v1720: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4213900690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4213900690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4213900690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:50 smithi122 ceph-mon[32255]: pgmap v1721: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:50] "GET /metrics HTTP/1.1" 200 202255 "" "Prometheus/2.18.1" 2021-12-15T18:55:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:50 smithi022 ceph-mon[31360]: pgmap v1721: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:50 smithi022 ceph-mon[40250]: pgmap v1721: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:52 smithi122 ceph-mon[32255]: pgmap v1722: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:55:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:55:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:52 smithi022 ceph-mon[31360]: pgmap v1722: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:52 smithi022 ceph-mon[40250]: pgmap v1722: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:53 smithi122 ceph-mon[32255]: pgmap v1723: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:53.443 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:53.443 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:53 smithi022 ceph-mon[31360]: pgmap v1723: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:53.444 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:53 smithi022 ceph-mon[40250]: pgmap v1723: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:53.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:53.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:53.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3188284638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3188284638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3188284638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:55:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:55 smithi122 ceph-mon[32255]: pgmap v1724: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:55 smithi022 ceph-mon[31360]: pgmap v1724: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:55 smithi022 ceph-mon[40250]: pgmap v1724: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:55:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:55:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:57 smithi122 ceph-mon[32255]: pgmap v1725: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:57 smithi022 ceph-mon[31360]: pgmap v1725: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:55:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:55:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:57 smithi022 ceph-mon[40250]: pgmap v1725: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:55:58.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:55:59.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:55:59.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:55:59.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:55:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:55:59 smithi122 ceph-mon[32255]: pgmap v1726: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:55:59 smithi022 ceph-mon[31360]: pgmap v1726: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:55:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:55:59 smithi022 ceph-mon[40250]: pgmap v1726: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:00] "GET /metrics HTTP/1.1" 200 202255 "" "Prometheus/2.18.1" 2021-12-15T18:56:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806775926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806775926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806775926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:02 smithi122 ceph-mon[32255]: pgmap v1727: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:56:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:56:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:02 smithi022 ceph-mon[31360]: pgmap v1727: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:02 smithi022 ceph-mon[40250]: pgmap v1727: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:03 smithi122 ceph-mon[32255]: pgmap v1728: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:03 smithi022 ceph-mon[31360]: pgmap v1728: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:03 smithi022 ceph-mon[40250]: pgmap v1728: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:04.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:04.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:04.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:04.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1001770408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1001770408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1001770408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:05 smithi122 ceph-mon[32255]: pgmap v1729: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:05 smithi022 ceph-mon[31360]: pgmap v1729: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:05 smithi022 ceph-mon[40250]: pgmap v1729: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:56:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:56:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:07 smithi122 ceph-mon[32255]: pgmap v1730: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:56:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:07 smithi022 ceph-mon[31360]: pgmap v1730: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:07 smithi022 ceph-mon[40250]: pgmap v1730: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:09 smithi122 ceph-mon[32255]: pgmap v1731: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:09 smithi022 ceph-mon[31360]: pgmap v1731: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:09 smithi022 ceph-mon[40250]: pgmap v1731: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:09.558 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:09.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:09.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:09.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2378321801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2378321801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:10] "GET /metrics HTTP/1.1" 200 202255 "" "Prometheus/2.18.1" 2021-12-15T18:56:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2378321801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:12 smithi122 ceph-mon[32255]: pgmap v1732: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:56:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:56:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:12 smithi022 ceph-mon[31360]: pgmap v1732: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:12 smithi022 ceph-mon[40250]: pgmap v1732: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:13 smithi122 ceph-mon[32255]: pgmap v1733: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:13 smithi022 ceph-mon[31360]: pgmap v1733: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:13 smithi022 ceph-mon[40250]: pgmap v1733: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:14.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:15 smithi022 ceph-mon[31360]: pgmap v1734: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:15 smithi022 ceph-mon[40250]: pgmap v1734: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:15.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:15.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:15.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:15 smithi122 ceph-mon[32255]: pgmap v1734: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1195100882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1195100882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1195100882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:56:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:56:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:17 smithi122 ceph-mon[32255]: pgmap v1735: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:17 smithi022 ceph-mon[31360]: pgmap v1735: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:56:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:17 smithi022 ceph-mon[40250]: pgmap v1735: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:19 smithi122 ceph-mon[32255]: pgmap v1736: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:19 smithi022 ceph-mon[31360]: pgmap v1736: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:19 smithi022 ceph-mon[40250]: pgmap v1736: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:20.301 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:20] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:56:20.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:20.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:20.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1253483655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1253483655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1253483655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:22 smithi122 ceph-mon[32255]: pgmap v1737: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:56:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:56:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:22 smithi022 ceph-mon[31360]: pgmap v1737: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:22 smithi022 ceph-mon[40250]: pgmap v1737: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:23 smithi122 ceph-mon[32255]: pgmap v1738: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:23 smithi022 ceph-mon[31360]: pgmap v1738: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:23 smithi022 ceph-mon[40250]: pgmap v1738: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:25 smithi122 ceph-mon[32255]: pgmap v1739: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:25 smithi022 ceph-mon[31360]: pgmap v1739: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:25 smithi022 ceph-mon[40250]: pgmap v1739: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:25.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:26.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:26.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:26.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1834075767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1834075767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1834075767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:56:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:56:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:27 smithi122 ceph-mon[32255]: pgmap v1740: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:56:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:27 smithi022 ceph-mon[31360]: pgmap v1740: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:27 smithi022 ceph-mon[40250]: pgmap v1740: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:29 smithi122 ceph-mon[32255]: pgmap v1741: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:29 smithi022 ceph-mon[31360]: pgmap v1741: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:29 smithi022 ceph-mon[40250]: pgmap v1741: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:30] "GET /metrics HTTP/1.1" 200 202247 "" "Prometheus/2.18.1" 2021-12-15T18:56:31.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:31.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:31.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:31.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:32 smithi122 ceph-mon[32255]: pgmap v1742: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/243255051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:56:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:56:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:32 smithi022 ceph-mon[31360]: pgmap v1742: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/243255051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:32 smithi022 ceph-mon[40250]: pgmap v1742: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/243255051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:56:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:56:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:56:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:56:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:56:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:56:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:56:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:56:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:56:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:56:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:56:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:56:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:34 smithi122 ceph-mon[32255]: pgmap v1743: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:34 smithi022 ceph-mon[31360]: pgmap v1743: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:34 smithi022 ceph-mon[40250]: pgmap v1743: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:35 smithi122 ceph-mon[32255]: pgmap v1744: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:35 smithi022 ceph-mon[31360]: pgmap v1744: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:35 smithi022 ceph-mon[40250]: pgmap v1744: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:36.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:36.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:36.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:36.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:36.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:56:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:56:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2541854444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2541854444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2541854444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:56:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:38 smithi122 ceph-mon[32255]: pgmap v1745: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:38 smithi022 ceph-mon[31360]: pgmap v1745: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:38 smithi022 ceph-mon[40250]: pgmap v1745: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:39 smithi122 ceph-mon[32255]: pgmap v1746: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:39 smithi022 ceph-mon[31360]: pgmap v1746: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:39 smithi022 ceph-mon[40250]: pgmap v1746: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:40] "GET /metrics HTTP/1.1" 200 202249 "" "Prometheus/2.18.1" 2021-12-15T18:56:41.780 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:42.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:42.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:42.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[31360]: pgmap v1747: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:56:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:56:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:56:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:56:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[40250]: pgmap v1747: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:56:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:56:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:56:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:56:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:42 smithi122 ceph-mon[32255]: pgmap v1747: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:56:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:56:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:56:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:56:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:56:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:56:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2985315898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2985315898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2985315898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:44 smithi122 ceph-mon[32255]: pgmap v1748: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:44 smithi022 ceph-mon[31360]: pgmap v1748: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:44 smithi022 ceph-mon[40250]: pgmap v1748: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:45 smithi122 ceph-mon[32255]: pgmap v1749: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:45 smithi022 ceph-mon[31360]: pgmap v1749: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:45 smithi022 ceph-mon[40250]: pgmap v1749: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:56:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:56:47.144 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:47 smithi122 ceph-mon[32255]: pgmap v1750: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:47.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:47.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:47.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:47.516 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:56:47.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:47.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:47 smithi022 ceph-mon[40250]: pgmap v1750: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:47.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:47.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:47 smithi022 ceph-mon[31360]: pgmap v1750: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4236615324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4236615324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4236615324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:49 smithi122 ceph-mon[32255]: pgmap v1751: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:49 smithi022 ceph-mon[31360]: pgmap v1751: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:49 smithi022 ceph-mon[40250]: pgmap v1751: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:50] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:56:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:52 smithi122 ceph-mon[32255]: pgmap v1752: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:56:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:56:52.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:52.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:52 smithi022 ceph-mon[31360]: pgmap v1752: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:52.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:52 smithi022 ceph-mon[40250]: pgmap v1752: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3590394613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3590394613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3590394613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:54 smithi122 ceph-mon[32255]: pgmap v1753: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:54 smithi022 ceph-mon[31360]: pgmap v1753: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:54 smithi022 ceph-mon[40250]: pgmap v1753: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:55 smithi122 ceph-mon[32255]: pgmap v1754: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:55 smithi022 ceph-mon[31360]: pgmap v1754: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:55 smithi022 ceph-mon[40250]: pgmap v1754: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:56:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:56:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:57 smithi122 ceph-mon[32255]: pgmap v1755: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:57 smithi022 ceph-mon[31360]: pgmap v1755: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:56:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:56:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:57 smithi022 ceph-mon[40250]: pgmap v1755: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:56:57.889 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:56:58.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:56:58.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:56:58.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:56:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4170048189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4170048189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4170048189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:56:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:56:59 smithi122 ceph-mon[32255]: pgmap v1756: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:56:59 smithi022 ceph-mon[31360]: pgmap v1756: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:56:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:56:59 smithi022 ceph-mon[40250]: pgmap v1756: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:00] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:57:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:02 smithi122 ceph-mon[32255]: pgmap v1757: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:57:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:57:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:02 smithi022 ceph-mon[31360]: pgmap v1757: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:02 smithi022 ceph-mon[40250]: pgmap v1757: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:03.257 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:03 smithi122 ceph-mon[32255]: pgmap v1758: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:03 smithi022 ceph-mon[31360]: pgmap v1758: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:03 smithi022 ceph-mon[40250]: pgmap v1758: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:03.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:03.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:03.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/365748617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/365748617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/365748617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:05 smithi122 ceph-mon[32255]: pgmap v1759: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:05 smithi022 ceph-mon[31360]: pgmap v1759: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:05 smithi022 ceph-mon[40250]: pgmap v1759: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:57:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:57:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:07 smithi122 ceph-mon[32255]: pgmap v1760: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:57:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:07 smithi022 ceph-mon[31360]: pgmap v1760: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:07 smithi022 ceph-mon[40250]: pgmap v1760: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:08.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:08.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:08.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:08.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/418450571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/418450571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/418450571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:10] "GET /metrics HTTP/1.1" 200 202250 "" "Prometheus/2.18.1" 2021-12-15T18:57:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:10 smithi122 ceph-mon[32255]: pgmap v1761: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:10 smithi022 ceph-mon[31360]: pgmap v1761: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:10 smithi022 ceph-mon[40250]: pgmap v1761: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:12 smithi122 ceph-mon[32255]: pgmap v1762: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:57:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:57:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:12 smithi022 ceph-mon[31360]: pgmap v1762: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:12 smithi022 ceph-mon[40250]: pgmap v1762: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:13 smithi122 ceph-mon[32255]: pgmap v1763: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:13 smithi022 ceph-mon[31360]: pgmap v1763: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:13 smithi022 ceph-mon[40250]: pgmap v1763: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:13.998 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:14.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:14.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:14.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/732760955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/732760955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/732760955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:15 smithi122 ceph-mon[32255]: pgmap v1764: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:15 smithi022 ceph-mon[31360]: pgmap v1764: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:15 smithi022 ceph-mon[40250]: pgmap v1764: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:57:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:57:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:17 smithi122 ceph-mon[32255]: pgmap v1765: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:57:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:17 smithi022 ceph-mon[31360]: pgmap v1765: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:17 smithi022 ceph-mon[40250]: pgmap v1765: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:19 smithi122 ceph-mon[32255]: pgmap v1766: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:19 smithi022 ceph-mon[31360]: pgmap v1766: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:19 smithi022 ceph-mon[40250]: pgmap v1766: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:19.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:19.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:19.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3893322498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:20] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:57:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3893322498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3893322498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:22 smithi122 ceph-mon[32255]: pgmap v1767: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:57:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:57:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:22 smithi022 ceph-mon[31360]: pgmap v1767: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:22 smithi022 ceph-mon[40250]: pgmap v1767: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:23 smithi122 ceph-mon[32255]: pgmap v1768: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:23 smithi022 ceph-mon[31360]: pgmap v1768: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:23 smithi022 ceph-mon[40250]: pgmap v1768: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:25.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:25.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2211348126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:25 smithi122 ceph-mon[32255]: pgmap v1769: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2211348126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:25 smithi022 ceph-mon[31360]: pgmap v1769: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2211348126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:25 smithi022 ceph-mon[40250]: pgmap v1769: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:57:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:57:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:27.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:27 smithi122 ceph-mon[32255]: pgmap v1770: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:57:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:27 smithi022 ceph-mon[31360]: pgmap v1770: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:27 smithi022 ceph-mon[40250]: pgmap v1770: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:29 smithi122 ceph-mon[32255]: pgmap v1771: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:29 smithi022 ceph-mon[31360]: pgmap v1771: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:29 smithi022 ceph-mon[40250]: pgmap v1771: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:30.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:30] "GET /metrics HTTP/1.1" 200 202253 "" "Prometheus/2.18.1" 2021-12-15T18:57:30.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:30.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:30.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1821127887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1821127887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1821127887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:32 smithi122 ceph-mon[32255]: pgmap v1772: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:57:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:57:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:32 smithi022 ceph-mon[31360]: pgmap v1772: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:32 smithi022 ceph-mon[40250]: pgmap v1772: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:57:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:57:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:57:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:57:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:57:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:57:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:57:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:57:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:57:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:57:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:57:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:57:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:34 smithi122 ceph-mon[32255]: pgmap v1773: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:34 smithi022 ceph-mon[31360]: pgmap v1773: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:34 smithi022 ceph-mon[40250]: pgmap v1773: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:35 smithi122 ceph-mon[32255]: pgmap v1774: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:35.478 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:35.478 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:35 smithi022 ceph-mon[31360]: pgmap v1774: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:35.479 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:35 smithi022 ceph-mon[40250]: pgmap v1774: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:35.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:35.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:35.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2705302381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2705302381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2705302381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:36.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:57:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:57:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:57:37.559 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:37.559 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:37 smithi022 ceph-mon[31360]: pgmap v1775: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:37.560 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:37.560 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:37 smithi022 ceph-mon[40250]: pgmap v1775: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:37 smithi122 ceph-mon[32255]: pgmap v1775: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:39 smithi122 ceph-mon[32255]: pgmap v1776: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:39 smithi022 ceph-mon[31360]: pgmap v1776: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:39 smithi022 ceph-mon[40250]: pgmap v1776: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:40] "GET /metrics HTTP/1.1" 200 202245 "" "Prometheus/2.18.1" 2021-12-15T18:57:40.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:42 smithi122 ceph-mon[32255]: pgmap v1777: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1396517864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:57:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:57:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:57:42.458 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[31360]: pgmap v1777: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:42.458 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1396517864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:42.458 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:57:42.458 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:57:42.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:57:42.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[40250]: pgmap v1777: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:42.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1396517864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:42.460 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:57:42.460 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:57:42.460 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:57:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:57:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:57:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:57:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:43 smithi022 ceph-mon[40250]: pgmap v1778: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:57:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:43 smithi022 ceph-mon[31360]: pgmap v1778: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:57:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:43 smithi122 ceph-mon[32255]: pgmap v1778: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:45 smithi122 ceph-mon[32255]: pgmap v1779: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:45 smithi022 ceph-mon[31360]: pgmap v1779: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:45 smithi022 ceph-mon[40250]: pgmap v1779: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:46.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:46.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:46.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:46.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:57:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:57:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2620876702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2620876702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2620876702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:57:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:48 smithi122 ceph-mon[32255]: pgmap v1780: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:48 smithi022 ceph-mon[31360]: pgmap v1780: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:48 smithi022 ceph-mon[40250]: pgmap v1780: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:49 smithi122 ceph-mon[32255]: pgmap v1781: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:49 smithi022 ceph-mon[31360]: pgmap v1781: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:49 smithi022 ceph-mon[40250]: pgmap v1781: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:57:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:50] "GET /metrics HTTP/1.1" 200 202251 "" "Prometheus/2.18.1" 2021-12-15T18:57:51.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:52 smithi022 ceph-mon[31360]: pgmap v1782: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/389884011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:52 smithi022 ceph-mon[40250]: pgmap v1782: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/389884011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:52 smithi122 ceph-mon[32255]: pgmap v1782: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/389884011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:57:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:57:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:53 smithi122 ceph-mon[32255]: pgmap v1783: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:53 smithi022 ceph-mon[31360]: pgmap v1783: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:53 smithi022 ceph-mon[40250]: pgmap v1783: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:57:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:55 smithi022 ceph-mon[31360]: pgmap v1784: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:57:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:55 smithi022 ceph-mon[40250]: pgmap v1784: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:57:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:55 smithi122 ceph-mon[32255]: pgmap v1784: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:57:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:57:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:57:56.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:57:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:57 smithi022 ceph-mon[31360]: pgmap v1785: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:57:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:57 smithi022 ceph-mon[40250]: pgmap v1785: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:57:57.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:57:57.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:57:57.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:57:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:57:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:57 smithi122 ceph-mon[32255]: pgmap v1785: 132 pgs: 132 active+clean; 582 KiB data, 105 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T18:57:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:57:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1965850981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1965850981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1965850981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:57:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:57:59 smithi022 ceph-mon[31360]: pgmap v1786: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:57:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:57:59 smithi022 ceph-mon[40250]: pgmap v1786: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:57:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:57:59 smithi122 ceph-mon[32255]: pgmap v1786: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:00] "GET /metrics HTTP/1.1" 200 202251 "" "Prometheus/2.18.1" 2021-12-15T18:58:02.332 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:02 smithi122 ceph-mon[32255]: pgmap v1787: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:58:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:58:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:02 smithi022 ceph-mon[31360]: pgmap v1787: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:02 smithi022 ceph-mon[40250]: pgmap v1787: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:02.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:02.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:02.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3421901230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3421901230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3421901230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:04 smithi122 ceph-mon[32255]: pgmap v1788: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:04 smithi022 ceph-mon[31360]: pgmap v1788: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:04 smithi022 ceph-mon[40250]: pgmap v1788: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:05 smithi022 ceph-mon[31360]: pgmap v1789: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:05 smithi022 ceph-mon[40250]: pgmap v1789: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:05 smithi122 ceph-mon[32255]: pgmap v1789: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T18:58:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:58:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:58:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:58:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:07 smithi022 ceph-mon[31360]: pgmap v1790: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:58:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:07 smithi022 ceph-mon[40250]: pgmap v1790: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:58:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:07 smithi122 ceph-mon[32255]: pgmap v1790: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T18:58:07.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:08.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:08.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:08.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3810717718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3810717718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3810717718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:09 smithi022 ceph-mon[31360]: pgmap v1791: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:58:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:09 smithi022 ceph-mon[40250]: pgmap v1791: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:58:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:09 smithi122 ceph-mon[32255]: pgmap v1791: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T18:58:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:10] "GET /metrics HTTP/1.1" 200 202258 "" "Prometheus/2.18.1" 2021-12-15T18:58:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:12 smithi122 ceph-mon[32255]: pgmap v1792: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:58:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:58:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:12 smithi022 ceph-mon[31360]: pgmap v1792: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:12 smithi022 ceph-mon[40250]: pgmap v1792: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:13.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:13.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:13.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:13.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:13.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:13 smithi022 ceph-mon[31360]: pgmap v1793: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:13.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:13 smithi022 ceph-mon[40250]: pgmap v1793: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:13 smithi122 ceph-mon[32255]: pgmap v1793: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/829546856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/829546856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/829546856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:15 smithi022 ceph-mon[31360]: pgmap v1794: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:15 smithi022 ceph-mon[40250]: pgmap v1794: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:15 smithi122 ceph-mon[32255]: pgmap v1794: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:58:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:58:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:17 smithi022 ceph-mon[31360]: pgmap v1795: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:58:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:17 smithi022 ceph-mon[40250]: pgmap v1795: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:17 smithi122 ceph-mon[32255]: pgmap v1795: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T18:58:18.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:18.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:18.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:18.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3489685538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3489685538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3489685538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:19 smithi122 ceph-mon[32255]: pgmap v1796: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:19 smithi022 ceph-mon[31360]: pgmap v1796: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:19 smithi022 ceph-mon[40250]: pgmap v1796: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:20] "GET /metrics HTTP/1.1" 200 202259 "" "Prometheus/2.18.1" 2021-12-15T18:58:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:22 smithi122 ceph-mon[32255]: pgmap v1797: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:58:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:58:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:22 smithi022 ceph-mon[31360]: pgmap v1797: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:22 smithi022 ceph-mon[40250]: pgmap v1797: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:23 smithi022 ceph-mon[31360]: pgmap v1798: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:23 smithi022 ceph-mon[40250]: pgmap v1798: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:23 smithi122 ceph-mon[32255]: pgmap v1798: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:23.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:24.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:24.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:24.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/343949184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/343949184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/343949184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:25 smithi022 ceph-mon[31360]: pgmap v1799: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:25 smithi022 ceph-mon[40250]: pgmap v1799: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:25 smithi122 ceph-mon[32255]: pgmap v1799: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:58:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:58:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:58:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:27 smithi022 ceph-mon[31360]: pgmap v1800: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:27 smithi022 ceph-mon[40250]: pgmap v1800: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:27 smithi122 ceph-mon[32255]: pgmap v1800: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:29.188 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:29 smithi022 ceph-mon[31360]: pgmap v1801: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:29 smithi022 ceph-mon[40250]: pgmap v1801: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:29.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:29.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:29.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:29 smithi122 ceph-mon[32255]: pgmap v1801: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3155777321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:30] "GET /metrics HTTP/1.1" 200 202259 "" "Prometheus/2.18.1" 2021-12-15T18:58:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3155777321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3155777321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:32 smithi122 ceph-mon[32255]: pgmap v1802: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:58:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:58:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:32 smithi022 ceph-mon[31360]: pgmap v1802: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:32 smithi022 ceph-mon[40250]: pgmap v1802: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:58:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:58:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:58:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:58:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:58:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:58:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:58:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:58:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:58:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:58:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:58:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:58:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:34 smithi122 ceph-mon[32255]: pgmap v1803: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:34 smithi022 ceph-mon[31360]: pgmap v1803: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:34 smithi022 ceph-mon[40250]: pgmap v1803: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:34.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:34.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:34.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:34.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3192565219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3192565219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3192565219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:36 smithi122 ceph-mon[32255]: pgmap v1804: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:36 smithi022 ceph-mon[31360]: pgmap v1804: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:36 smithi022 ceph-mon[40250]: pgmap v1804: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:36.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:58:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:58:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:58:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:37 smithi022 ceph-mon[31360]: pgmap v1805: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:37 smithi022 ceph-mon[40250]: pgmap v1805: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:37 smithi122 ceph-mon[32255]: pgmap v1805: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:39 smithi022 ceph-mon[31360]: pgmap v1806: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:39 smithi022 ceph-mon[40250]: pgmap v1806: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:39 smithi122 ceph-mon[32255]: pgmap v1806: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:39.927 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:40.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:40.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:40.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:40] "GET /metrics HTTP/1.1" 200 202263 "" "Prometheus/2.18.1" 2021-12-15T18:58:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/336469253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/336469253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/336469253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:42 smithi122 ceph-mon[32255]: pgmap v1807: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:58:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:58:42.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:42 smithi022 ceph-mon[31360]: pgmap v1807: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:42.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:42 smithi022 ceph-mon[40250]: pgmap v1807: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:58:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:58:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:58:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:58:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:58:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:58:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:58:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:58:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:58:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:58:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:58:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:58:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:44 smithi122 ceph-mon[32255]: pgmap v1808: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:44 smithi022 ceph-mon[31360]: pgmap v1808: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:44 smithi022 ceph-mon[40250]: pgmap v1808: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:45.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:45 smithi022 ceph-mon[31360]: pgmap v1809: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:45 smithi022 ceph-mon[40250]: pgmap v1809: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:45.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:45.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:45.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:45 smithi122 ceph-mon[32255]: pgmap v1809: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2322392486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2322392486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:46.586 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2322392486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:58:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:58:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:47 smithi022 ceph-mon[31360]: pgmap v1810: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:58:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:47 smithi022 ceph-mon[40250]: pgmap v1810: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:47 smithi122 ceph-mon[32255]: pgmap v1810: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:49 smithi022 ceph-mon[31360]: pgmap v1811: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:49 smithi022 ceph-mon[40250]: pgmap v1811: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:49 smithi122 ceph-mon[32255]: pgmap v1811: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:50] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T18:58:50.667 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:51.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:51.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:51.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2448322436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2448322436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2448322436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:52 smithi122 ceph-mon[32255]: pgmap v1812: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:58:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:58:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:52 smithi022 ceph-mon[31360]: pgmap v1812: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:52 smithi022 ceph-mon[40250]: pgmap v1812: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:53 smithi022 ceph-mon[31360]: pgmap v1813: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:53 smithi022 ceph-mon[40250]: pgmap v1813: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:53 smithi122 ceph-mon[32255]: pgmap v1813: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:55 smithi022 ceph-mon[31360]: pgmap v1814: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:55 smithi022 ceph-mon[40250]: pgmap v1814: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:55 smithi122 ceph-mon[32255]: pgmap v1814: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:56.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:58:56.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:58:56.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:58:56.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:58:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:58:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:58:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1705536887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1705536887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1705536887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:58:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:58:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:58 smithi122 ceph-mon[32255]: pgmap v1815: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:58 smithi022 ceph-mon[31360]: pgmap v1815: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:58:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:58 smithi022 ceph-mon[40250]: pgmap v1815: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:58:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:58:59 smithi022 ceph-mon[31360]: pgmap v1816: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:58:59 smithi022 ceph-mon[40250]: pgmap v1816: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:58:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:58:59 smithi122 ceph-mon[32255]: pgmap v1816: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:00] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T18:59:01.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:59:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:59:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:02 smithi122 ceph-mon[32255]: pgmap v1817: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/28132401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:02 smithi022 ceph-mon[31360]: pgmap v1817: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/28132401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:02 smithi022 ceph-mon[40250]: pgmap v1817: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/28132401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:03 smithi022 ceph-mon[31360]: pgmap v1818: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:03 smithi022 ceph-mon[40250]: pgmap v1818: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:03 smithi122 ceph-mon[32255]: pgmap v1818: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:05 smithi022 ceph-mon[31360]: pgmap v1819: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:05 smithi022 ceph-mon[40250]: pgmap v1819: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:05 smithi122 ceph-mon[32255]: pgmap v1819: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:06.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:59:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:59:07.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:07.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:07.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1191723735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:07 smithi022 ceph-mon[31360]: pgmap v1820: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1191723735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:07 smithi022 ceph-mon[40250]: pgmap v1820: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1191723735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:07 smithi122 ceph-mon[32255]: pgmap v1820: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:59:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:09 smithi022 ceph-mon[31360]: pgmap v1821: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:09 smithi022 ceph-mon[40250]: pgmap v1821: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:09 smithi122 ceph-mon[32255]: pgmap v1821: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:10] "GET /metrics HTTP/1.1" 200 202261 "" "Prometheus/2.18.1" 2021-12-15T18:59:12.142 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:12 smithi122 ceph-mon[32255]: pgmap v1822: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:59:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:59:12.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:12.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:12.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:12.509 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:12 smithi022 ceph-mon[31360]: pgmap v1822: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:12.510 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:12 smithi022 ceph-mon[40250]: pgmap v1822: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/524526936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/524526936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/524526936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:14 smithi122 ceph-mon[32255]: pgmap v1823: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:14 smithi022 ceph-mon[31360]: pgmap v1823: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:14 smithi022 ceph-mon[40250]: pgmap v1823: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:15 smithi022 ceph-mon[31360]: pgmap v1824: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:15 smithi022 ceph-mon[40250]: pgmap v1824: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:15 smithi122 ceph-mon[32255]: pgmap v1824: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:59:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:59:17.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:17.511 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:59:17.511 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:17.512 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:17 smithi022 ceph-mon[40250]: pgmap v1825: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:17.512 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:17.512 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:17 smithi022 ceph-mon[31360]: pgmap v1825: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:17 smithi122 ceph-mon[32255]: pgmap v1825: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:17.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:17.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:17.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/731296912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/731296912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/731296912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:19 smithi022 ceph-mon[31360]: pgmap v1826: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:19 smithi022 ceph-mon[40250]: pgmap v1826: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:19 smithi122 ceph-mon[32255]: pgmap v1826: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:20] "GET /metrics HTTP/1.1" 200 202265 "" "Prometheus/2.18.1" 2021-12-15T18:59:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:22 smithi122 ceph-mon[32255]: pgmap v1827: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:59:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:59:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:22 smithi022 ceph-mon[31360]: pgmap v1827: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:22 smithi022 ceph-mon[40250]: pgmap v1827: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:22.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:23.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:23.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:23.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:23 smithi022 ceph-mon[31360]: pgmap v1828: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:23 smithi022 ceph-mon[40250]: pgmap v1828: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:23 smithi122 ceph-mon[32255]: pgmap v1828: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2956442814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2956442814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2956442814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:25 smithi022 ceph-mon[31360]: pgmap v1829: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:25 smithi022 ceph-mon[40250]: pgmap v1829: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:25 smithi122 ceph-mon[32255]: pgmap v1829: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:59:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:59:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:59:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:27 smithi022 ceph-mon[31360]: pgmap v1830: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:27 smithi022 ceph-mon[40250]: pgmap v1830: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:27 smithi122 ceph-mon[32255]: pgmap v1830: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:28.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:28.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:28.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2316989358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2316989358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2316989358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:30] "GET /metrics HTTP/1.1" 200 202265 "" "Prometheus/2.18.1" 2021-12-15T18:59:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:30 smithi122 ceph-mon[32255]: pgmap v1831: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:30 smithi022 ceph-mon[31360]: pgmap v1831: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:30 smithi022 ceph-mon[40250]: pgmap v1831: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:59:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:59:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:33 smithi122 ceph-mon[32255]: pgmap v1832: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:59:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:59:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:59:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:59:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[31360]: pgmap v1832: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:59:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:59:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:59:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:59:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[40250]: pgmap v1832: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:59:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:59:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T18:59:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T18:59:33.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:33.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:33.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:33.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3559325660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3559325660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3559325660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:35 smithi122 ceph-mon[32255]: pgmap v1833: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:35 smithi022 ceph-mon[31360]: pgmap v1833: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:35 smithi022 ceph-mon[40250]: pgmap v1833: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:36.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:59:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:59:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:37 smithi022 ceph-mon[31360]: pgmap v1834: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:37 smithi022 ceph-mon[40250]: pgmap v1834: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:37 smithi122 ceph-mon[32255]: pgmap v1834: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:59:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:38.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:39 smithi022 ceph-mon[31360]: pgmap v1835: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:39 smithi022 ceph-mon[40250]: pgmap v1835: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:39.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:39.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:39.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:39 smithi122 ceph-mon[32255]: pgmap v1835: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/24273073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:40.887 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:40] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T18:59:40.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/24273073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:40.888 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/24273073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:41 smithi122 ceph-mon[32255]: pgmap v1836: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:41 smithi022 ceph-mon[31360]: pgmap v1836: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:41 smithi022 ceph-mon[40250]: pgmap v1836: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:59:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:59:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[31360]: pgmap v1837: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:59:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:59:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:59:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[40250]: pgmap v1837: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:59:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:59:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:59:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:43 smithi122 ceph-mon[32255]: pgmap v1837: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T18:59:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T18:59:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T18:59:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:44.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:44.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:44.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:45 smithi122 ceph-mon[32255]: pgmap v1838: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3789416672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:45 smithi022 ceph-mon[31360]: pgmap v1838: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3789416672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:45 smithi022 ceph-mon[40250]: pgmap v1838: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3789416672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:59:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:59:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:47 smithi022 ceph-mon[31360]: pgmap v1839: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:47 smithi022 ceph-mon[40250]: pgmap v1839: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:47 smithi122 ceph-mon[32255]: pgmap v1839: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:47.771 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:59:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:49 smithi122 ceph-mon[32255]: pgmap v1840: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:49.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:49 smithi022 ceph-mon[31360]: pgmap v1840: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:49.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:49 smithi022 ceph-mon[40250]: pgmap v1840: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:49.743 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:50.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:50.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:50.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 18:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:50] "GET /metrics HTTP/1.1" 200 202265 "" "Prometheus/2.18.1" 2021-12-15T18:59:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:50 smithi122 ceph-mon[32255]: pgmap v1841: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/965110120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[31360]: pgmap v1841: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/965110120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[40250]: pgmap v1841: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/965110120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:51.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T18:59:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 18:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T18:59:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T18:59:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:53 smithi122 ceph-mon[32255]: pgmap v1842: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:53 smithi022 ceph-mon[31360]: pgmap v1842: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:53 smithi022 ceph-mon[40250]: pgmap v1842: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:55.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T18:59:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:55 smithi122 ceph-mon[32255]: pgmap v1843: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:55.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T18:59:55.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T18:59:55.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T18:59:55.474 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:55 smithi022 ceph-mon[31360]: pgmap v1843: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:55.475 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:55 smithi022 ceph-mon[40250]: pgmap v1843: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T18:59:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1048588232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1048588232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1048588232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T18:59:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 18:59:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T18:59:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:57 smithi022 ceph-mon[31360]: pgmap v1844: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:57 smithi022 ceph-mon[40250]: pgmap v1844: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:57 smithi122 ceph-mon[32255]: pgmap v1844: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 18:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:18:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T18:59:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T18:59:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 18:59:59 smithi122 ceph-mon[32255]: pgmap v1845: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 18:59:59 smithi022 ceph-mon[31360]: pgmap v1845: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T18:59:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 18:59:59 smithi022 ceph-mon[40250]: pgmap v1845: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:00] "GET /metrics HTTP/1.1" 200 202265 "" "Prometheus/2.18.1" 2021-12-15T19:00:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T19:00:00.475 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:00.476 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T19:00:00.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T19:00:00.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:00.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:00.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:01.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:01 smithi022 ceph-mon[31360]: pgmap v1846: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:01.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2885272704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:01 smithi122 ceph-mon[32255]: pgmap v1846: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2885272704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:01 smithi022 ceph-mon[40250]: pgmap v1846: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2885272704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:02 smithi022 ceph-mon[31360]: pgmap v1847: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:02 smithi022 ceph-mon[40250]: pgmap v1847: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:02 smithi122 ceph-mon[32255]: pgmap v1847: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:00:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:00:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:04 smithi022 ceph-mon[31360]: pgmap v1848: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:04 smithi022 ceph-mon[40250]: pgmap v1848: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:04 smithi122 ceph-mon[32255]: pgmap v1848: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:05.846 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:06.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:06.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:06.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:00:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:00:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:07 smithi022 ceph-mon[31360]: pgmap v1849: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3472007215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:07 smithi022 ceph-mon[40250]: pgmap v1849: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3472007215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:07 smithi122 ceph-mon[32255]: pgmap v1849: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3472007215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:00:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:09 smithi122 ceph-mon[32255]: pgmap v1850: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:09 smithi022 ceph-mon[31360]: pgmap v1850: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:09 smithi022 ceph-mon[40250]: pgmap v1850: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:10] "GET /metrics HTTP/1.1" 200 202268 "" "Prometheus/2.18.1" 2021-12-15T19:00:11.219 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:11 smithi122 ceph-mon[32255]: pgmap v1851: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:11 smithi022 ceph-mon[31360]: pgmap v1851: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:11.861 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:11 smithi022 ceph-mon[40250]: pgmap v1851: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:11.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:11.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:11.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2806626452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:00:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:00:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2806626452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2806626452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:13 smithi122 ceph-mon[32255]: pgmap v1852: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:13 smithi022 ceph-mon[31360]: pgmap v1852: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:13 smithi022 ceph-mon[40250]: pgmap v1852: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:15.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:15 smithi122 ceph-mon[32255]: pgmap v1853: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:15 smithi022 ceph-mon[31360]: pgmap v1853: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:15 smithi022 ceph-mon[40250]: pgmap v1853: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:16.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:00:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:00:16.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:16.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:16.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:17 smithi022 ceph-mon[31360]: pgmap v1854: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/398565104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:17 smithi022 ceph-mon[40250]: pgmap v1854: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/398565104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:17 smithi122 ceph-mon[32255]: pgmap v1854: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/398565104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:00:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:19 smithi122 ceph-mon[32255]: pgmap v1855: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:19 smithi022 ceph-mon[31360]: pgmap v1855: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:19 smithi022 ceph-mon[40250]: pgmap v1855: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:20 smithi122 ceph-mon[32255]: pgmap v1856: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:20] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T19:00:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:20 smithi022 ceph-mon[31360]: pgmap v1856: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:20 smithi022 ceph-mon[40250]: pgmap v1856: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:21.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:22.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:22.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:22.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:00:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:00:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:23 smithi122 ceph-mon[32255]: pgmap v1857: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/322797036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:23 smithi022 ceph-mon[31360]: pgmap v1857: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/322797036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:23 smithi022 ceph-mon[40250]: pgmap v1857: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/322797036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:25 smithi122 ceph-mon[32255]: pgmap v1858: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:25 smithi022 ceph-mon[31360]: pgmap v1858: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:25 smithi022 ceph-mon[40250]: pgmap v1858: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:00:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:00:27.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:27 smithi122 ceph-mon[32255]: pgmap v1859: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:27 smithi022 ceph-mon[31360]: pgmap v1859: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:00:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:27 smithi022 ceph-mon[40250]: pgmap v1859: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:27.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:27.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:27.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2279434518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2279434518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2279434518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:29 smithi122 ceph-mon[32255]: pgmap v1860: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:29 smithi022 ceph-mon[31360]: pgmap v1860: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:29 smithi022 ceph-mon[40250]: pgmap v1860: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:30] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T19:00:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:31 smithi122 ceph-mon[32255]: pgmap v1861: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:31 smithi022 ceph-mon[31360]: pgmap v1861: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:31 smithi022 ceph-mon[40250]: pgmap v1861: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:00:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:00:32.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:33.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:33.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:33.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:33 smithi122 ceph-mon[32255]: pgmap v1862: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:00:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:00:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:00:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:00:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1493402286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[31360]: pgmap v1862: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:00:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:00:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:00:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:00:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1493402286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[40250]: pgmap v1862: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:00:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:00:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:00:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:00:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1493402286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:35 smithi122 ceph-mon[32255]: pgmap v1863: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:35 smithi022 ceph-mon[31360]: pgmap v1863: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:35 smithi022 ceph-mon[40250]: pgmap v1863: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:37.140 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:00:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:00:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:37 smithi022 ceph-mon[31360]: pgmap v1864: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:37 smithi022 ceph-mon[40250]: pgmap v1864: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:37 smithi122 ceph-mon[32255]: pgmap v1864: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:00:38.081 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:38.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:38.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:38.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:38.449 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:38.450 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:39 smithi122 ceph-mon[32255]: pgmap v1865: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1019146307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:39 smithi022 ceph-mon[31360]: pgmap v1865: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1019146307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:39 smithi022 ceph-mon[40250]: pgmap v1865: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1019146307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:40] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:00:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:41 smithi122 ceph-mon[32255]: pgmap v1866: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:41 smithi022 ceph-mon[31360]: pgmap v1866: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:41 smithi022 ceph-mon[40250]: pgmap v1866: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:00:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:00:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:43 smithi122 ceph-mon[32255]: pgmap v1867: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:43.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:43.451 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:43 smithi022 ceph-mon[31360]: pgmap v1867: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:43.452 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:43 smithi022 ceph-mon[40250]: pgmap v1867: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:43.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:44.156 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3302938690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3302938690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3302938690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:45 smithi122 ceph-mon[32255]: pgmap v1868: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:45 smithi022 ceph-mon[31360]: pgmap v1868: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:45 smithi022 ceph-mon[40250]: pgmap v1868: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:47.160 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:00:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:00:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:47 smithi122 ceph-mon[32255]: pgmap v1869: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:47 smithi022 ceph-mon[31360]: pgmap v1869: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:00:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:47 smithi022 ceph-mon[40250]: pgmap v1869: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:48.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:49.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:49.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:49.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:49 smithi122 ceph-mon[32255]: pgmap v1870: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:49 smithi022 ceph-mon[31360]: pgmap v1870: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:49 smithi022 ceph-mon[40250]: pgmap v1870: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:50] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:00:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:50 smithi122 ceph-mon[32255]: pgmap v1871: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1841458340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:50 smithi022 ceph-mon[31360]: pgmap v1871: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1841458340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:50 smithi022 ceph-mon[40250]: pgmap v1871: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1841458340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:00:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:00:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:00:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:00:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:00:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:00:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:00:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:00:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:00:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:00:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:00:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:00:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:52 smithi122 ceph-mon[32255]: pgmap v1872: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:00:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:00:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:52 smithi022 ceph-mon[31360]: pgmap v1872: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:52 smithi022 ceph-mon[40250]: pgmap v1872: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:54.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:54 smithi122 ceph-mon[32255]: pgmap v1873: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:54 smithi022 ceph-mon[31360]: pgmap v1873: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:54 smithi022 ceph-mon[40250]: pgmap v1873: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:00:54.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:54.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:54.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:00:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2553985269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2553985269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2553985269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:00:57.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:00:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:00:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:57 smithi122 ceph-mon[32255]: pgmap v1874: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:57 smithi022 ceph-mon[31360]: pgmap v1874: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:00:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:57 smithi022 ceph-mon[40250]: pgmap v1874: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:00:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:00:59 smithi122 ceph-mon[32255]: pgmap v1875: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:00:59 smithi022 ceph-mon[31360]: pgmap v1875: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:00:59 smithi022 ceph-mon[40250]: pgmap v1875: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:00:59.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:00:59.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:00:59.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:00:59.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3701172541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3701172541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:00] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:01:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3701172541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:01 smithi122 ceph-mon[32255]: pgmap v1876: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:01 smithi022 ceph-mon[31360]: pgmap v1876: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:01 smithi022 ceph-mon[40250]: pgmap v1876: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:01:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:01:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:03 smithi122 ceph-mon[32255]: pgmap v1877: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:03 smithi022 ceph-mon[31360]: pgmap v1877: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:03 smithi022 ceph-mon[40250]: pgmap v1877: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:04.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:05 smithi022 ceph-mon[31360]: pgmap v1878: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:05 smithi022 ceph-mon[40250]: pgmap v1878: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:05 smithi122 ceph-mon[32255]: pgmap v1878: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3682723682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3682723682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3682723682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:07.169 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:01:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:01:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:07 smithi122 ceph-mon[32255]: pgmap v1879: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:01:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:07 smithi022 ceph-mon[31360]: pgmap v1879: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:07 smithi022 ceph-mon[40250]: pgmap v1879: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:08 smithi122 ceph-mon[32255]: pgmap v1880: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:08 smithi022 ceph-mon[31360]: pgmap v1880: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:08 smithi022 ceph-mon[40250]: pgmap v1880: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:10.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:10] "GET /metrics HTTP/1.1" 200 202263 "" "Prometheus/2.18.1" 2021-12-15T19:01:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:10 smithi122 ceph-mon[32255]: pgmap v1881: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:10 smithi022 ceph-mon[31360]: pgmap v1881: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:10 smithi022 ceph-mon[40250]: pgmap v1881: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:10.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:10.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:10.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4282604581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4282604581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4282604581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:01:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:01:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:13 smithi122 ceph-mon[32255]: pgmap v1882: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:13 smithi022 ceph-mon[31360]: pgmap v1882: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:13 smithi022 ceph-mon[40250]: pgmap v1882: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:15 smithi122 ceph-mon[32255]: pgmap v1883: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:15 smithi022 ceph-mon[31360]: pgmap v1883: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:15 smithi022 ceph-mon[40250]: pgmap v1883: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:15.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:16.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:16.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:16.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3700629513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3700629513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3700629513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:17.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:01:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:01:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:17 smithi122 ceph-mon[32255]: pgmap v1884: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:17 smithi022 ceph-mon[31360]: pgmap v1884: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:01:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:17 smithi022 ceph-mon[40250]: pgmap v1884: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:19 smithi122 ceph-mon[32255]: pgmap v1885: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:19 smithi022 ceph-mon[31360]: pgmap v1885: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:19 smithi022 ceph-mon[40250]: pgmap v1885: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:20] "GET /metrics HTTP/1.1" 200 202265 "" "Prometheus/2.18.1" 2021-12-15T19:01:21.050 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:21.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:21.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:21.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:21.426 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:21 smithi022 ceph-mon[31360]: pgmap v1886: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:21.426 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:21 smithi022 ceph-mon[40250]: pgmap v1886: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:21 smithi122 ceph-mon[32255]: pgmap v1886: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3167531659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:01:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:01:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3167531659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3167531659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:23 smithi122 ceph-mon[32255]: pgmap v1887: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:23 smithi022 ceph-mon[31360]: pgmap v1887: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:23 smithi022 ceph-mon[40250]: pgmap v1887: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:24 smithi122 ceph-mon[32255]: pgmap v1888: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:24 smithi022 ceph-mon[31360]: pgmap v1888: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:24 smithi022 ceph-mon[40250]: pgmap v1888: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:26.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:26.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:26.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:27.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:01:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:01:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:27 smithi122 ceph-mon[32255]: pgmap v1889: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3695703974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:01:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:27 smithi022 ceph-mon[31360]: pgmap v1889: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3695703974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:27 smithi022 ceph-mon[40250]: pgmap v1889: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3695703974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:29 smithi122 ceph-mon[32255]: pgmap v1890: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:29 smithi022 ceph-mon[31360]: pgmap v1890: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:29 smithi022 ceph-mon[40250]: pgmap v1890: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:30] "GET /metrics HTTP/1.1" 200 202265 "" "Prometheus/2.18.1" 2021-12-15T19:01:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:31 smithi122 ceph-mon[32255]: pgmap v1891: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:31 smithi022 ceph-mon[31360]: pgmap v1891: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:31 smithi022 ceph-mon[40250]: pgmap v1891: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:31.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:32.164 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1270156968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:32.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:32.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:32.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1270156968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:01:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:01:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1270156968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:33 smithi122 ceph-mon[32255]: pgmap v1892: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:01:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:01:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:01:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:01:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[31360]: pgmap v1892: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:01:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:01:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:01:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:01:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[40250]: pgmap v1892: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:01:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:01:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:01:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:01:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:35 smithi122 ceph-mon[32255]: pgmap v1893: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:35 smithi022 ceph-mon[31360]: pgmap v1893: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:35 smithi022 ceph-mon[40250]: pgmap v1893: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:37.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:37.170 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:01:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:01:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:37 smithi122 ceph-mon[32255]: pgmap v1894: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:01:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:37 smithi022 ceph-mon[40250]: pgmap v1894: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:37 smithi022 ceph-mon[31360]: pgmap v1894: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:37.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:37.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:38 smithi122 ceph-mon[32255]: pgmap v1895: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/81523122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:38 smithi022 ceph-mon[31360]: pgmap v1895: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/81523122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:38 smithi022 ceph-mon[40250]: pgmap v1895: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/81523122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:40 smithi122 ceph-mon[32255]: pgmap v1896: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:40] "GET /metrics HTTP/1.1" 200 202257 "" "Prometheus/2.18.1" 2021-12-15T19:01:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:40 smithi022 ceph-mon[31360]: pgmap v1896: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:40 smithi022 ceph-mon[40250]: pgmap v1896: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:42.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:01:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:01:42.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:42.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:42.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:43 smithi122 ceph-mon[32255]: pgmap v1897: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3966678825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:43 smithi022 ceph-mon[31360]: pgmap v1897: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3966678825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:43 smithi022 ceph-mon[40250]: pgmap v1897: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3966678825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:45 smithi122 ceph-mon[32255]: pgmap v1898: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:45 smithi022 ceph-mon[40250]: pgmap v1898: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:45 smithi022 ceph-mon[31360]: pgmap v1898: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:47.160 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:01:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:01:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:47 smithi122 ceph-mon[32255]: pgmap v1899: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:47 smithi022 ceph-mon[31360]: pgmap v1899: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:01:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:47 smithi022 ceph-mon[40250]: pgmap v1899: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:47.904 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:49 smithi122 ceph-mon[32255]: pgmap v1900: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2119054578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:49 smithi022 ceph-mon[31360]: pgmap v1900: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2119054578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:49 smithi022 ceph-mon[40250]: pgmap v1900: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2119054578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:50] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:01:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:51 smithi122 ceph-mon[32255]: pgmap v1901: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:01:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:01:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:01:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[31360]: pgmap v1901: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:01:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:01:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:01:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[40250]: pgmap v1901: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:01:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:01:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:01:52.576 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:01:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:01:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:52 smithi122 ceph-mon[32255]: pgmap v1902: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:01:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:52 smithi022 ceph-mon[31360]: pgmap v1902: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:01:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:52 smithi022 ceph-mon[40250]: pgmap v1902: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:01:53.278 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:53.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:53.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:53.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:54 smithi122 ceph-mon[32255]: pgmap v1903: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4201681160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:54 smithi022 ceph-mon[31360]: pgmap v1903: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4201681160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:54 smithi022 ceph-mon[40250]: pgmap v1903: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:01:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4201681160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:57.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:01:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:01:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:57 smithi122 ceph-mon[32255]: pgmap v1904: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:57 smithi022 ceph-mon[31360]: pgmap v1904: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:01:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:57 smithi022 ceph-mon[40250]: pgmap v1904: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:01:58.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:01:59.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:01:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:01:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:01:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:59 smithi022 ceph-mon[31360]: pgmap v1905: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:01:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2569683938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:59 smithi022 ceph-mon[40250]: pgmap v1905: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:01:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2569683938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:01:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:59 smithi122 ceph-mon[32255]: pgmap v1905: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:01:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:01:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2569683938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:00] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:02:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:01 smithi122 ceph-mon[32255]: pgmap v1906: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:01 smithi022 ceph-mon[31360]: pgmap v1906: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:01 smithi022 ceph-mon[40250]: pgmap v1906: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:02:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:02:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:03 smithi122 ceph-mon[32255]: pgmap v1907: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:03 smithi022 ceph-mon[31360]: pgmap v1907: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:03 smithi022 ceph-mon[40250]: pgmap v1907: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:04.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:04.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:04.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:04.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:05 smithi122 ceph-mon[32255]: pgmap v1908: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3903096291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:05 smithi022 ceph-mon[31360]: pgmap v1908: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3903096291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:05 smithi022 ceph-mon[40250]: pgmap v1908: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3903096291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:07.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:02:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:02:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:07 smithi122 ceph-mon[32255]: pgmap v1909: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:07 smithi022 ceph-mon[31360]: pgmap v1909: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:02:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:07 smithi022 ceph-mon[40250]: pgmap v1909: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:09.387 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:09 smithi122 ceph-mon[32255]: pgmap v1910: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:09 smithi022 ceph-mon[31360]: pgmap v1910: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:09 smithi022 ceph-mon[40250]: pgmap v1910: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:09.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:09.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:09.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:10 smithi122 ceph-mon[32255]: pgmap v1911: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/627156490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:10] "GET /metrics HTTP/1.1" 200 202264 "" "Prometheus/2.18.1" 2021-12-15T19:02:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:10 smithi022 ceph-mon[31360]: pgmap v1911: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/627156490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:10 smithi022 ceph-mon[40250]: pgmap v1911: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/627156490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:02:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:02:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:13 smithi122 ceph-mon[32255]: pgmap v1912: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:13 smithi022 ceph-mon[31360]: pgmap v1912: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:13 smithi022 ceph-mon[40250]: pgmap v1912: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:14.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:15.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:15.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:15.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:15 smithi122 ceph-mon[32255]: pgmap v1913: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1681014001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:15 smithi022 ceph-mon[31360]: pgmap v1913: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1681014001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:15 smithi022 ceph-mon[40250]: pgmap v1913: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1681014001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:17.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:02:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:02:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:17 smithi122 ceph-mon[32255]: pgmap v1914: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:17 smithi022 ceph-mon[31360]: pgmap v1914: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:02:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:17 smithi022 ceph-mon[40250]: pgmap v1914: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:19 smithi122 ceph-mon[32255]: pgmap v1915: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:19 smithi022 ceph-mon[31360]: pgmap v1915: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:19 smithi022 ceph-mon[40250]: pgmap v1915: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:20.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:20] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T19:02:20.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:20.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:20.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:21 smithi122 ceph-mon[32255]: pgmap v1916: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2644437797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:21 smithi022 ceph-mon[31360]: pgmap v1916: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2644437797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:21 smithi022 ceph-mon[40250]: pgmap v1916: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2644437797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:22 smithi122 ceph-mon[32255]: pgmap v1917: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:02:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:02:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:22 smithi022 ceph-mon[31360]: pgmap v1917: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:22 smithi022 ceph-mon[40250]: pgmap v1917: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:24 smithi022 ceph-mon[31360]: pgmap v1918: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:24 smithi022 ceph-mon[40250]: pgmap v1918: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:24 smithi122 ceph-mon[32255]: pgmap v1918: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:25.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:25.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:26 smithi022 ceph-mon[31360]: pgmap v1919: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2267401856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:26 smithi022 ceph-mon[40250]: pgmap v1919: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2267401856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:26 smithi122 ceph-mon[32255]: pgmap v1919: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2267401856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:02:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:02:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:02:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:28 smithi022 ceph-mon[31360]: pgmap v1920: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:28 smithi022 ceph-mon[40250]: pgmap v1920: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:28 smithi122 ceph-mon[32255]: pgmap v1920: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:30 smithi122 ceph-mon[32255]: pgmap v1921: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:30] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T19:02:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:30 smithi022 ceph-mon[31360]: pgmap v1921: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:30 smithi022 ceph-mon[40250]: pgmap v1921: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:30.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:31.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:31.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:31.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3092179789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:02:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:02:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3092179789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3092179789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:33 smithi122 ceph-mon[32255]: pgmap v1922: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:02:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:02:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:02:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:02:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[31360]: pgmap v1922: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:02:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:02:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:02:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:02:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[40250]: pgmap v1922: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:02:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:02:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:02:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:02:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:35 smithi122 ceph-mon[32255]: pgmap v1923: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:35 smithi022 ceph-mon[31360]: pgmap v1923: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:35 smithi022 ceph-mon[40250]: pgmap v1923: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:36.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:36.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:36.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:36.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:37.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:02:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:02:37.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:37 smithi122 ceph-mon[32255]: pgmap v1924: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:37.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1541402061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:37 smithi022 ceph-mon[31360]: pgmap v1924: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1541402061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:02:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:37 smithi022 ceph-mon[40250]: pgmap v1924: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1541402061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:38 smithi122 ceph-mon[32255]: pgmap v1925: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:38 smithi022 ceph-mon[31360]: pgmap v1925: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:38 smithi022 ceph-mon[40250]: pgmap v1925: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:40 smithi122 ceph-mon[32255]: pgmap v1926: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:40.449 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:40] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:02:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:40 smithi022 ceph-mon[31360]: pgmap v1926: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:40 smithi022 ceph-mon[40250]: pgmap v1926: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:41.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:41.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:41.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:41.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/845521921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/845521921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/845521921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:02:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:02:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:43 smithi122 ceph-mon[32255]: pgmap v1927: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:43 smithi022 ceph-mon[31360]: pgmap v1927: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:43 smithi022 ceph-mon[40250]: pgmap v1927: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:45 smithi122 ceph-mon[32255]: pgmap v1928: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:45 smithi022 ceph-mon[31360]: pgmap v1928: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:45 smithi022 ceph-mon[40250]: pgmap v1928: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:46.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:47.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:02:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:02:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:47 smithi022 ceph-mon[31360]: pgmap v1929: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:47 smithi022 ceph-mon[40250]: pgmap v1929: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:47.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:47.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:47.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:47 smithi122 ceph-mon[32255]: pgmap v1929: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:02:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:48 smithi022 ceph-mon[31360]: pgmap v1930: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3337658798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:48 smithi022 ceph-mon[40250]: pgmap v1930: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3337658798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:48 smithi122 ceph-mon[32255]: pgmap v1930: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3337658798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:50 smithi122 ceph-mon[32255]: pgmap v1931: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:50] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:02:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:50 smithi022 ceph-mon[31360]: pgmap v1931: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:50 smithi022 ceph-mon[40250]: pgmap v1931: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:02:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:02:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:02:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:02:52.354 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:02:52.355 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:02:52.355 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:02:52.356 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:02:52.356 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:02:52.357 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:02:52.357 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:02:52.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:02:52.358 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:02:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:02:52.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:52.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:52.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:53 smithi122 ceph-mon[32255]: pgmap v1932: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/435747792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:53 smithi022 ceph-mon[31360]: pgmap v1932: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/435747792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:53 smithi022 ceph-mon[40250]: pgmap v1932: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/435747792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:54 smithi022 ceph-mon[31360]: pgmap v1933: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:54 smithi022 ceph-mon[40250]: pgmap v1933: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:54 smithi122 ceph-mon[32255]: pgmap v1933: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:02:57.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:02:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:02:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:57 smithi122 ceph-mon[32255]: pgmap v1934: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:57 smithi022 ceph-mon[31360]: pgmap v1934: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:02:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:57 smithi022 ceph-mon[40250]: pgmap v1934: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:57.718 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:02:58.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:02:58.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:02:58.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:02:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/7860568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/7860568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:02:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/7860568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:02:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:02:59 smithi122 ceph-mon[32255]: pgmap v1935: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:02:59 smithi022 ceph-mon[31360]: pgmap v1935: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:02:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:02:59 smithi022 ceph-mon[40250]: pgmap v1935: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:00] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:03:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:01 smithi122 ceph-mon[32255]: pgmap v1936: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:01 smithi022 ceph-mon[31360]: pgmap v1936: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:01 smithi022 ceph-mon[40250]: pgmap v1936: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:02 smithi022 ceph-mon[31360]: pgmap v1937: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:02 smithi022 ceph-mon[40250]: pgmap v1937: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:02 smithi122 ceph-mon[32255]: pgmap v1937: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:03:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:03:03.089 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:03.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:04 smithi022 ceph-mon[31360]: pgmap v1938: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1668380281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:04 smithi022 ceph-mon[40250]: pgmap v1938: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1668380281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:04 smithi122 ceph-mon[32255]: pgmap v1938: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1668380281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:07.177 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:03:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:03:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:07 smithi122 ceph-mon[32255]: pgmap v1939: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:03:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:07 smithi022 ceph-mon[31360]: pgmap v1939: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:07 smithi022 ceph-mon[40250]: pgmap v1939: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:08.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:08.462 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:08.462 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:08.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:08.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:08.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:09 smithi122 ceph-mon[32255]: pgmap v1940: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1531041196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:09 smithi022 ceph-mon[31360]: pgmap v1940: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1531041196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:09 smithi022 ceph-mon[40250]: pgmap v1940: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1531041196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:10] "GET /metrics HTTP/1.1" 200 202263 "" "Prometheus/2.18.1" 2021-12-15T19:03:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:11 smithi022 ceph-mon[31360]: pgmap v1941: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:11 smithi022 ceph-mon[40250]: pgmap v1941: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:11 smithi122 ceph-mon[32255]: pgmap v1941: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:12 smithi022 ceph-mon[31360]: pgmap v1942: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:12 smithi022 ceph-mon[40250]: pgmap v1942: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:12.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:12 smithi122 ceph-mon[32255]: pgmap v1942: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:03:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:03:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:14.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:14.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:14.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:14 smithi022 ceph-mon[31360]: pgmap v1943: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3485594237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:14 smithi022 ceph-mon[40250]: pgmap v1943: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3485594237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:14 smithi122 ceph-mon[32255]: pgmap v1943: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3485594237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:17.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:03:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:03:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:17 smithi122 ceph-mon[32255]: pgmap v1944: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:17 smithi022 ceph-mon[31360]: pgmap v1944: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:03:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:17 smithi022 ceph-mon[40250]: pgmap v1944: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:19.208 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:19 smithi022 ceph-mon[31360]: pgmap v1945: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:19 smithi022 ceph-mon[40250]: pgmap v1945: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:19.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:19.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:19.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:19 smithi122 ceph-mon[32255]: pgmap v1945: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:20] "GET /metrics HTTP/1.1" 200 202266 "" "Prometheus/2.18.1" 2021-12-15T19:03:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:20 smithi122 ceph-mon[32255]: pgmap v1946: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3681346521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:20 smithi022 ceph-mon[31360]: pgmap v1946: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3681346521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:20 smithi022 ceph-mon[40250]: pgmap v1946: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3681346521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:03:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:03:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:23 smithi122 ceph-mon[32255]: pgmap v1947: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:23 smithi022 ceph-mon[31360]: pgmap v1947: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:23 smithi022 ceph-mon[40250]: pgmap v1947: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:24.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:24.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:24.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:24.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:25 smithi022 ceph-mon[31360]: pgmap v1948: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/512765595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:25 smithi022 ceph-mon[40250]: pgmap v1948: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/512765595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:25 smithi122 ceph-mon[32255]: pgmap v1948: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/512765595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:03:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:03:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:27 smithi022 ceph-mon[31360]: pgmap v1949: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:03:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:27 smithi022 ceph-mon[40250]: pgmap v1949: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:27 smithi122 ceph-mon[32255]: pgmap v1949: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:29 smithi022 ceph-mon[31360]: pgmap v1950: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:29 smithi022 ceph-mon[40250]: pgmap v1950: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:29 smithi122 ceph-mon[32255]: pgmap v1950: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:29.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:30.197 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:30 smithi022 ceph-mon[40250]: pgmap v1951: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:30.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:30 smithi022 ceph-mon[31360]: pgmap v1951: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:30.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:30.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:30 smithi122 ceph-mon[32255]: pgmap v1951: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:30] "GET /metrics HTTP/1.1" 200 202266 "" "Prometheus/2.18.1" 2021-12-15T19:03:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2812840270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2812840270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2812840270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:32 smithi022 ceph-mon[31360]: pgmap v1952: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:32 smithi022 ceph-mon[40250]: pgmap v1952: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:32 smithi122 ceph-mon[32255]: pgmap v1952: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:03:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:03:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:03:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:03:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:03:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:03:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:03:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:03:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:03:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:03:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:03:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:03:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:03:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:03:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:34 smithi022 ceph-mon[31360]: pgmap v1953: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:34 smithi022 ceph-mon[40250]: pgmap v1953: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:34 smithi122 ceph-mon[32255]: pgmap v1953: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:35.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:36 smithi022 ceph-mon[31360]: pgmap v1954: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2441839492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:36.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:36 smithi022 ceph-mon[40250]: pgmap v1954: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:36.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2441839492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:36 smithi122 ceph-mon[32255]: pgmap v1954: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2441839492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:03:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:03:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:03:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:38 smithi122 ceph-mon[32255]: pgmap v1955: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:38 smithi022 ceph-mon[31360]: pgmap v1955: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:38 smithi022 ceph-mon[40250]: pgmap v1955: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:40 smithi122 ceph-mon[32255]: pgmap v1956: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:40] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:03:40.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:40.687 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:40 smithi022 ceph-mon[31360]: pgmap v1956: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:40.687 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:40 smithi022 ceph-mon[40250]: pgmap v1956: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:41.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:41.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:41.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4003546880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:41.724 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4003546880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4003546880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:03:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:03:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:43 smithi022 ceph-mon[31360]: pgmap v1957: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:43 smithi022 ceph-mon[40250]: pgmap v1957: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:43 smithi122 ceph-mon[32255]: pgmap v1957: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:45 smithi022 ceph-mon[31360]: pgmap v1958: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:45 smithi022 ceph-mon[40250]: pgmap v1958: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:45 smithi122 ceph-mon[32255]: pgmap v1958: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:46.053 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:46.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:46.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:46.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:03:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:03:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:03:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:47 smithi022 ceph-mon[31360]: pgmap v1959: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2466104360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:47 smithi022 ceph-mon[40250]: pgmap v1959: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2466104360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:47 smithi122 ceph-mon[32255]: pgmap v1959: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2466104360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:48 smithi022 ceph-mon[31360]: pgmap v1960: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:48 smithi022 ceph-mon[40250]: pgmap v1960: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:48 smithi122 ceph-mon[32255]: pgmap v1960: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:50 smithi122 ceph-mon[32255]: pgmap v1961: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:50] "GET /metrics HTTP/1.1" 200 202258 "" "Prometheus/2.18.1" 2021-12-15T19:03:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:50 smithi022 ceph-mon[31360]: pgmap v1961: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:50 smithi022 ceph-mon[40250]: pgmap v1961: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:51.426 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:51.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:51.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:51.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:03:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:03:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3696974825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:03:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:03:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:03:52.493 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3696974825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:52.493 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:03:52.493 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:03:52.494 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:03:52.494 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3696974825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:52.494 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:03:52.494 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:03:52.495 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:03:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:53 smithi022 ceph-mon[31360]: pgmap v1962: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:03:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:53 smithi022 ceph-mon[40250]: pgmap v1962: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:03:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:53 smithi122 ceph-mon[32255]: pgmap v1962: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:03:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:54 smithi022 ceph-mon[31360]: pgmap v1963: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:54 smithi022 ceph-mon[40250]: pgmap v1963: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:54 smithi122 ceph-mon[32255]: pgmap v1963: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:03:56.801 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:03:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:03:57.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:03:57.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:03:57.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:03:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:03:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:03:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:57 smithi022 ceph-mon[31360]: pgmap v1964: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1973381458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:57 smithi022 ceph-mon[40250]: pgmap v1964: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1973381458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:57 smithi122 ceph-mon[32255]: pgmap v1964: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1973381458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:03:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:03:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:03:59 smithi022 ceph-mon[31360]: pgmap v1965: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:03:59 smithi022 ceph-mon[40250]: pgmap v1965: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:03:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:03:59 smithi122 ceph-mon[32255]: pgmap v1965: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:00] "GET /metrics HTTP/1.1" 200 202258 "" "Prometheus/2.18.1" 2021-12-15T19:04:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:00 smithi122 ceph-mon[32255]: pgmap v1966: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:00 smithi022 ceph-mon[31360]: pgmap v1966: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:00 smithi022 ceph-mon[40250]: pgmap v1966: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:02.171 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:02.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:02.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:02.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:04:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:04:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:03 smithi022 ceph-mon[31360]: pgmap v1967: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2431955731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:03 smithi022 ceph-mon[40250]: pgmap v1967: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2431955731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:03 smithi122 ceph-mon[32255]: pgmap v1967: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2431955731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:05 smithi022 ceph-mon[31360]: pgmap v1968: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:05 smithi022 ceph-mon[40250]: pgmap v1968: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:05 smithi122 ceph-mon[32255]: pgmap v1968: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:07.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:04:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:04:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:07 smithi022 ceph-mon[31360]: pgmap v1969: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:04:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:07 smithi022 ceph-mon[40250]: pgmap v1969: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:07.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:07 smithi122 ceph-mon[32255]: pgmap v1969: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:07.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2306820457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2306820457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2306820457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:09 smithi022 ceph-mon[31360]: pgmap v1970: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:09 smithi022 ceph-mon[40250]: pgmap v1970: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:09 smithi122 ceph-mon[32255]: pgmap v1970: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:10] "GET /metrics HTTP/1.1" 200 202258 "" "Prometheus/2.18.1" 2021-12-15T19:04:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:10 smithi122 ceph-mon[32255]: pgmap v1971: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:10 smithi022 ceph-mon[31360]: pgmap v1971: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:10 smithi022 ceph-mon[40250]: pgmap v1971: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:04:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:04:12.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:13.193 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:13 smithi022 ceph-mon[31360]: pgmap v1972: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:13.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:13.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:13.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:13 smithi022 ceph-mon[40250]: pgmap v1972: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:13 smithi122 ceph-mon[32255]: pgmap v1972: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1075682310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1075682310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1075682310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:15 smithi022 ceph-mon[31360]: pgmap v1973: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:15 smithi022 ceph-mon[40250]: pgmap v1973: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:15 smithi122 ceph-mon[32255]: pgmap v1973: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:17.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:04:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:04:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:17 smithi022 ceph-mon[31360]: pgmap v1974: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:04:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:17 smithi022 ceph-mon[40250]: pgmap v1974: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:17 smithi122 ceph-mon[32255]: pgmap v1974: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:18.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:18.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:18.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:18.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:19 smithi022 ceph-mon[31360]: pgmap v1975: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3596838991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:19 smithi022 ceph-mon[40250]: pgmap v1975: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3596838991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:19 smithi122 ceph-mon[32255]: pgmap v1975: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3596838991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:20 smithi122 ceph-mon[32255]: pgmap v1976: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:20] "GET /metrics HTTP/1.1" 200 202259 "" "Prometheus/2.18.1" 2021-12-15T19:04:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:20 smithi022 ceph-mon[31360]: pgmap v1976: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:20 smithi022 ceph-mon[40250]: pgmap v1976: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:22 smithi022 ceph-mon[31360]: pgmap v1977: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:22 smithi022 ceph-mon[40250]: pgmap v1977: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:22 smithi122 ceph-mon[32255]: pgmap v1977: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:04:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:04:23.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:24.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:24.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:24.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:24 smithi022 ceph-mon[31360]: pgmap v1978: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2650236245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:24 smithi022 ceph-mon[40250]: pgmap v1978: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2650236245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:24 smithi122 ceph-mon[32255]: pgmap v1978: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2650236245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:04:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:04:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:27 smithi022 ceph-mon[31360]: pgmap v1979: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:04:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:27 smithi022 ceph-mon[40250]: pgmap v1979: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:27 smithi122 ceph-mon[32255]: pgmap v1979: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:29.023 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:29 smithi022 ceph-mon[31360]: pgmap v1980: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:29 smithi022 ceph-mon[40250]: pgmap v1980: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:29 smithi122 ceph-mon[32255]: pgmap v1980: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:30] "GET /metrics HTTP/1.1" 200 202259 "" "Prometheus/2.18.1" 2021-12-15T19:04:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:30 smithi122 ceph-mon[32255]: pgmap v1981: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2197066726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:30 smithi022 ceph-mon[31360]: pgmap v1981: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2197066726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:30 smithi022 ceph-mon[40250]: pgmap v1981: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2197066726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:04:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:04:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[31360]: pgmap v1982: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:04:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:04:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:04:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:04:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[40250]: pgmap v1982: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:04:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:04:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:04:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:04:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:33 smithi122 ceph-mon[32255]: pgmap v1982: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:04:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:04:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:04:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:04:34.397 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:34 smithi022 ceph-mon[31360]: pgmap v1983: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:34 smithi022 ceph-mon[40250]: pgmap v1983: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:34 smithi122 ceph-mon[32255]: pgmap v1983: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3517421795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3517421795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3517421795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:04:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:04:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:37 smithi022 ceph-mon[31360]: pgmap v1984: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:04:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:37 smithi022 ceph-mon[40250]: pgmap v1984: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:37 smithi122 ceph-mon[32255]: pgmap v1984: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:39 smithi022 ceph-mon[31360]: pgmap v1985: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:39 smithi022 ceph-mon[40250]: pgmap v1985: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:39 smithi122 ceph-mon[32255]: pgmap v1985: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:39.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:40.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:40.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:40.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:40 smithi122 ceph-mon[32255]: pgmap v1986: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2161501106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:40] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:04:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:40 smithi022 ceph-mon[31360]: pgmap v1986: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2161501106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:40 smithi022 ceph-mon[40250]: pgmap v1986: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2161501106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:04:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:04:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:43 smithi022 ceph-mon[31360]: pgmap v1987: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:43 smithi022 ceph-mon[40250]: pgmap v1987: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:43 smithi122 ceph-mon[32255]: pgmap v1987: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:44 smithi022 ceph-mon[31360]: pgmap v1988: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:44 smithi022 ceph-mon[40250]: pgmap v1988: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:44 smithi122 ceph-mon[32255]: pgmap v1988: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:45.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:45.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:45.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:45.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:46 smithi022 ceph-mon[31360]: pgmap v1989: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3594130454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:46 smithi022 ceph-mon[40250]: pgmap v1989: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3594130454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:46 smithi122 ceph-mon[32255]: pgmap v1989: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3594130454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:04:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:04:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:04:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:48 smithi022 ceph-mon[31360]: pgmap v1990: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:48 smithi022 ceph-mon[40250]: pgmap v1990: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:48 smithi122 ceph-mon[32255]: pgmap v1990: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:50 smithi122 ceph-mon[32255]: pgmap v1991: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:50] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T19:04:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:50 smithi022 ceph-mon[31360]: pgmap v1991: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:50 smithi022 ceph-mon[40250]: pgmap v1991: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:50.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:50.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:50.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2430692709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2430692709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2430692709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:52.573 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:04:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:04:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[31360]: pgmap v1992: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:04:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:04:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:04:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:04:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[40250]: pgmap v1992: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:04:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:04:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:04:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:04:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:53 smithi122 ceph-mon[32255]: pgmap v1992: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:04:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:04:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:04:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:04:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:54 smithi022 ceph-mon[31360]: pgmap v1993: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:54 smithi022 ceph-mon[40250]: pgmap v1993: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:54 smithi122 ceph-mon[32255]: pgmap v1993: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:04:55.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:04:56.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:04:56.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:04:56.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:04:57.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:04:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:04:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:04:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:57 smithi022 ceph-mon[31360]: pgmap v1994: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1855016358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:57 smithi022 ceph-mon[40250]: pgmap v1994: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1855016358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:57 smithi122 ceph-mon[32255]: pgmap v1994: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1855016358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:04:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:04:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:04:59 smithi022 ceph-mon[31360]: pgmap v1995: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:04:59 smithi022 ceph-mon[40250]: pgmap v1995: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:04:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:04:59 smithi122 ceph-mon[32255]: pgmap v1995: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:00] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T19:05:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:00 smithi122 ceph-mon[32255]: pgmap v1996: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:00 smithi022 ceph-mon[31360]: pgmap v1996: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:00 smithi022 ceph-mon[40250]: pgmap v1996: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:01.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:01.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:01.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:01.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2246613902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2246613902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2246613902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:05:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:05:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:03 smithi022 ceph-mon[31360]: pgmap v1997: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:03 smithi022 ceph-mon[40250]: pgmap v1997: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:03 smithi122 ceph-mon[32255]: pgmap v1997: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:04 smithi022 ceph-mon[31360]: pgmap v1998: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:04 smithi022 ceph-mon[40250]: pgmap v1998: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:04 smithi122 ceph-mon[32255]: pgmap v1998: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:06.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:06.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:06.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:06.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:07.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:05:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:05:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:07 smithi022 ceph-mon[31360]: pgmap v1999: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1318826227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:07 smithi022 ceph-mon[40250]: pgmap v1999: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1318826227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:07 smithi122 ceph-mon[32255]: pgmap v1999: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1318826227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:05:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:08.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:09 smithi022 ceph-mon[31360]: pgmap v2000: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:09 smithi022 ceph-mon[40250]: pgmap v2000: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:09 smithi122 ceph-mon[32255]: pgmap v2000: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:10] "GET /metrics HTTP/1.1" 200 202259 "" "Prometheus/2.18.1" 2021-12-15T19:05:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:10 smithi122 ceph-mon[32255]: pgmap v2001: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:10 smithi022 ceph-mon[31360]: pgmap v2001: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:10 smithi022 ceph-mon[40250]: pgmap v2001: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:11.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:12.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:12.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:12.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:05:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:05:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:13 smithi022 ceph-mon[31360]: pgmap v2002: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/198230811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:13 smithi022 ceph-mon[40250]: pgmap v2002: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/198230811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:13 smithi122 ceph-mon[32255]: pgmap v2002: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/198230811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:14 smithi022 ceph-mon[31360]: pgmap v2003: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:14 smithi022 ceph-mon[40250]: pgmap v2003: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:14 smithi122 ceph-mon[32255]: pgmap v2003: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:17.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:05:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:05:17.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:17 smithi022 ceph-mon[31360]: pgmap v2004: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:17.531 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:05:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:17 smithi022 ceph-mon[40250]: pgmap v2004: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:17 smithi122 ceph-mon[32255]: pgmap v2004: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:17.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:17.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:17.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:18 smithi022 ceph-mon[31360]: pgmap v2005: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1347032080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:18 smithi022 ceph-mon[40250]: pgmap v2005: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1347032080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:18 smithi122 ceph-mon[32255]: pgmap v2005: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1347032080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:20] "GET /metrics HTTP/1.1" 200 202261 "" "Prometheus/2.18.1" 2021-12-15T19:05:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:20 smithi122 ceph-mon[32255]: pgmap v2006: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:20 smithi022 ceph-mon[31360]: pgmap v2006: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:20 smithi022 ceph-mon[40250]: pgmap v2006: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:22 smithi022 ceph-mon[31360]: pgmap v2007: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:22 smithi022 ceph-mon[40250]: pgmap v2007: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:22 smithi122 ceph-mon[32255]: pgmap v2007: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:05:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:05:22.730 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:23.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:23.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:23.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2901560284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2901560284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2901560284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:24 smithi022 ceph-mon[31360]: pgmap v2008: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:24 smithi022 ceph-mon[40250]: pgmap v2008: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:24 smithi122 ceph-mon[32255]: pgmap v2008: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:05:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:05:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:05:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:27 smithi022 ceph-mon[31360]: pgmap v2009: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:27 smithi022 ceph-mon[40250]: pgmap v2009: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:27 smithi122 ceph-mon[32255]: pgmap v2009: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:28.099 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:28.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:28.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:28.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:28.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:28.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:28 smithi022 ceph-mon[31360]: pgmap v2010: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:28.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:28.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:28 smithi022 ceph-mon[40250]: pgmap v2010: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:28 smithi122 ceph-mon[32255]: pgmap v2010: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3635492745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3635492745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3635492745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:30] "GET /metrics HTTP/1.1" 200 202261 "" "Prometheus/2.18.1" 2021-12-15T19:05:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:30 smithi122 ceph-mon[32255]: pgmap v2011: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:30 smithi022 ceph-mon[31360]: pgmap v2011: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:30 smithi022 ceph-mon[40250]: pgmap v2011: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:32 smithi022 ceph-mon[31360]: pgmap v2012: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:32 smithi022 ceph-mon[40250]: pgmap v2012: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:32 smithi122 ceph-mon[32255]: pgmap v2012: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:05:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:05:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:05:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:05:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:05:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:05:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:05:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:05:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:05:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:05:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:05:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:05:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:05:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:05:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:33.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:33.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:34 smithi122 ceph-mon[32255]: pgmap v2013: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2819463989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:34 smithi022 ceph-mon[31360]: pgmap v2013: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2819463989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:34 smithi022 ceph-mon[40250]: pgmap v2013: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2819463989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:36 smithi122 ceph-mon[32255]: pgmap v2014: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:36 smithi022 ceph-mon[31360]: pgmap v2014: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:36 smithi022 ceph-mon[40250]: pgmap v2014: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:05:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:05:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:05:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:38 smithi122 ceph-mon[32255]: pgmap v2015: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:38 smithi022 ceph-mon[31360]: pgmap v2015: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:38 smithi022 ceph-mon[40250]: pgmap v2015: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:39.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:39.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:39.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/843433887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/843433887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/843433887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:40 smithi122 ceph-mon[32255]: pgmap v2016: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:40] "GET /metrics HTTP/1.1" 200 202260 "" "Prometheus/2.18.1" 2021-12-15T19:05:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:40 smithi022 ceph-mon[31360]: pgmap v2016: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:40 smithi022 ceph-mon[40250]: pgmap v2016: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:42 smithi022 ceph-mon[31360]: pgmap v2017: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:42 smithi022 ceph-mon[40250]: pgmap v2017: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:42 smithi122 ceph-mon[32255]: pgmap v2017: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:05:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:05:44.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:44 smithi022 ceph-mon[31360]: pgmap v2018: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:44 smithi022 ceph-mon[40250]: pgmap v2018: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:44.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:44.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:44.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:44 smithi122 ceph-mon[32255]: pgmap v2018: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1988403434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1988403434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1988403434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:46 smithi122 ceph-mon[32255]: pgmap v2019: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:46 smithi022 ceph-mon[31360]: pgmap v2019: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:46 smithi022 ceph-mon[40250]: pgmap v2019: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:47.329 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:05:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:05:47.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:05:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:48 smithi122 ceph-mon[32255]: pgmap v2020: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:48 smithi022 ceph-mon[31360]: pgmap v2020: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:48 smithi022 ceph-mon[40250]: pgmap v2020: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:49.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:49.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:49.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:49.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:50] "GET /metrics HTTP/1.1" 200 202257 "" "Prometheus/2.18.1" 2021-12-15T19:05:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:50 smithi122 ceph-mon[32255]: pgmap v2021: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2854418078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:50 smithi022 ceph-mon[31360]: pgmap v2021: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2854418078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:50 smithi022 ceph-mon[40250]: pgmap v2021: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2854418078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:52 smithi022 ceph-mon[31360]: pgmap v2022: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:52 smithi022 ceph-mon[40250]: pgmap v2022: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:52 smithi122 ceph-mon[32255]: pgmap v2022: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:05:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:05:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:05:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:05:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:05:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:05:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:05:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:05:53.359 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:05:53.359 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:05:53.359 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:05:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:54 smithi122 ceph-mon[32255]: pgmap v2023: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[31360]: pgmap v2023: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[40250]: pgmap v2023: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:05:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:05:54.954 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:05:55.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:05:55.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:05:55.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:05:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3621620271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3621620271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3621620271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:05:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:56 smithi122 ceph-mon[32255]: pgmap v2024: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:56 smithi022 ceph-mon[31360]: pgmap v2024: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:56 smithi022 ceph-mon[40250]: pgmap v2024: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:57.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:05:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:05:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:05:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:05:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:05:58 smithi122 ceph-mon[32255]: pgmap v2025: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:05:58 smithi022 ceph-mon[31360]: pgmap v2025: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:05:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:05:58 smithi022 ceph-mon[40250]: pgmap v2025: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:00.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:00 smithi122 ceph-mon[32255]: pgmap v2026: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:00] "GET /metrics HTTP/1.1" 200 202257 "" "Prometheus/2.18.1" 2021-12-15T19:06:00.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:00.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:00.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:00.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:00 smithi022 ceph-mon[40250]: pgmap v2026: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:00.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:00 smithi022 ceph-mon[31360]: pgmap v2026: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2190396722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2190396722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2190396722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:02 smithi122 ceph-mon[32255]: pgmap v2027: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:06:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:06:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:02 smithi022 ceph-mon[31360]: pgmap v2027: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:02 smithi022 ceph-mon[40250]: pgmap v2027: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:04 smithi122 ceph-mon[32255]: pgmap v2028: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:04 smithi022 ceph-mon[31360]: pgmap v2028: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:04 smithi022 ceph-mon[40250]: pgmap v2028: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:05.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:06.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:06.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:06.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:06 smithi122 ceph-mon[32255]: pgmap v2029: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1141887654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:06 smithi022 ceph-mon[31360]: pgmap v2029: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1141887654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:06 smithi022 ceph-mon[40250]: pgmap v2029: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1141887654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:07.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:06:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:06:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:06:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:08 smithi122 ceph-mon[32255]: pgmap v2030: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:08 smithi022 ceph-mon[31360]: pgmap v2030: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:08 smithi022 ceph-mon[40250]: pgmap v2030: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:10 smithi122 ceph-mon[32255]: pgmap v2031: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:10] "GET /metrics HTTP/1.1" 200 202262 "" "Prometheus/2.18.1" 2021-12-15T19:06:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:10 smithi022 ceph-mon[31360]: pgmap v2031: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:10 smithi022 ceph-mon[40250]: pgmap v2031: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:11.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:11.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:11.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:11.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:12 smithi022 ceph-mon[31360]: pgmap v2032: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3206568146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:12 smithi022 ceph-mon[40250]: pgmap v2032: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3206568146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:06:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:06:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:12 smithi122 ceph-mon[32255]: pgmap v2032: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3206568146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:14 smithi122 ceph-mon[32255]: pgmap v2033: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:14 smithi022 ceph-mon[31360]: pgmap v2033: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:14 smithi022 ceph-mon[40250]: pgmap v2033: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:16.443 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:16 smithi122 ceph-mon[32255]: pgmap v2034: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:16 smithi022 ceph-mon[31360]: pgmap v2034: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:16 smithi022 ceph-mon[40250]: pgmap v2034: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:16.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:16.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:16.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:17.331 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:06:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:06:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2871715006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:06:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2871715006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2871715006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:18 smithi122 ceph-mon[32255]: pgmap v2035: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:18 smithi022 ceph-mon[31360]: pgmap v2035: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:18 smithi022 ceph-mon[40250]: pgmap v2035: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:20] "GET /metrics HTTP/1.1" 200 202261 "" "Prometheus/2.18.1" 2021-12-15T19:06:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:20 smithi122 ceph-mon[32255]: pgmap v2036: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:20 smithi022 ceph-mon[31360]: pgmap v2036: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:20 smithi022 ceph-mon[40250]: pgmap v2036: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:21.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:22.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:22.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:22.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:22 smithi022 ceph-mon[31360]: pgmap v2037: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1448411220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:22 smithi022 ceph-mon[40250]: pgmap v2037: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1448411220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:22 smithi122 ceph-mon[32255]: pgmap v2037: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1448411220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:06:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:06:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:24 smithi122 ceph-mon[32255]: pgmap v2038: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:24 smithi022 ceph-mon[31360]: pgmap v2038: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:24 smithi022 ceph-mon[40250]: pgmap v2038: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:26 smithi122 ceph-mon[32255]: pgmap v2039: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:26 smithi022 ceph-mon[31360]: pgmap v2039: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:26 smithi022 ceph-mon[40250]: pgmap v2039: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:27.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:27.334 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:06:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:06:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:06:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:27.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:27.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:27.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:28 smithi122 ceph-mon[32255]: pgmap v2040: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1511599706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:28 smithi022 ceph-mon[31360]: pgmap v2040: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1511599706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:28 smithi022 ceph-mon[40250]: pgmap v2040: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1511599706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:30 smithi122 ceph-mon[32255]: pgmap v2041: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:30] "GET /metrics HTTP/1.1" 200 202261 "" "Prometheus/2.18.1" 2021-12-15T19:06:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:30 smithi022 ceph-mon[31360]: pgmap v2041: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:30 smithi022 ceph-mon[40250]: pgmap v2041: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:32 smithi022 ceph-mon[31360]: pgmap v2042: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:32 smithi022 ceph-mon[40250]: pgmap v2042: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:32.556 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:32 smithi122 ceph-mon[32255]: pgmap v2042: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:06:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:06:32.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:32.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:32.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:06:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:06:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:06:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:06:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1360807091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:06:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:06:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:06:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:06:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1360807091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:06:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:06:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:06:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:06:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1360807091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:34 smithi022 ceph-mon[31360]: pgmap v2043: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:34 smithi022 ceph-mon[40250]: pgmap v2043: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:34 smithi122 ceph-mon[32255]: pgmap v2043: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:36 smithi122 ceph-mon[32255]: pgmap v2044: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:36 smithi022 ceph-mon[31360]: pgmap v2044: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:36 smithi022 ceph-mon[40250]: pgmap v2044: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:37.334 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:06:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:06:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:06:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:37.926 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:38.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:38.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:38.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:38 smithi122 ceph-mon[32255]: pgmap v2045: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1095788723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:38 smithi022 ceph-mon[31360]: pgmap v2045: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1095788723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:38 smithi022 ceph-mon[40250]: pgmap v2045: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1095788723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:40 smithi122 ceph-mon[32255]: pgmap v2046: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:40] "GET /metrics HTTP/1.1" 200 202263 "" "Prometheus/2.18.1" 2021-12-15T19:06:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:40 smithi022 ceph-mon[31360]: pgmap v2046: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:40 smithi022 ceph-mon[40250]: pgmap v2046: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:42 smithi022 ceph-mon[31360]: pgmap v2047: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:42 smithi022 ceph-mon[40250]: pgmap v2047: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:42 smithi122 ceph-mon[32255]: pgmap v2047: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:06:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:06:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:44 smithi122 ceph-mon[32255]: pgmap v2048: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3091284791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:44 smithi022 ceph-mon[31360]: pgmap v2048: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3091284791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:44 smithi022 ceph-mon[40250]: pgmap v2048: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3091284791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:46 smithi122 ceph-mon[32255]: pgmap v2049: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:46 smithi022 ceph-mon[31360]: pgmap v2049: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:46 smithi022 ceph-mon[40250]: pgmap v2049: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:47.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:06:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:06:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:06:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:48.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:48.669 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:48 smithi022 ceph-mon[31360]: pgmap v2050: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:48.669 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:48 smithi022 ceph-mon[40250]: pgmap v2050: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:48 smithi122 ceph-mon[32255]: pgmap v2050: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:49.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:49.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:49.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/514317288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/514317288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/514317288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:50] "GET /metrics HTTP/1.1" 200 202259 "" "Prometheus/2.18.1" 2021-12-15T19:06:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:50 smithi122 ceph-mon[32255]: pgmap v2051: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:50 smithi022 ceph-mon[31360]: pgmap v2051: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:50 smithi022 ceph-mon[40250]: pgmap v2051: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:52 smithi022 ceph-mon[31360]: pgmap v2052: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:52 smithi022 ceph-mon[40250]: pgmap v2052: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:52 smithi122 ceph-mon[32255]: pgmap v2052: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:06:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:06:54.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:54.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:06:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[31360]: pgmap v2053: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:06:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:06:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:06:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:06:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[40250]: pgmap v2053: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:06:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:06:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:06:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:06:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:54 smithi122 ceph-mon[32255]: pgmap v2053: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:06:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:06:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:06:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:06:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:06:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1543509944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1543509944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1543509944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:06:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:56 smithi122 ceph-mon[32255]: pgmap v2054: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:56 smithi022 ceph-mon[31360]: pgmap v2054: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:56 smithi022 ceph-mon[40250]: pgmap v2054: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:57.347 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:06:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:06:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:06:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:06:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:06:58 smithi122 ceph-mon[32255]: pgmap v2055: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:06:58 smithi022 ceph-mon[31360]: pgmap v2055: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:06:58 smithi022 ceph-mon[40250]: pgmap v2055: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:06:59.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:06:59.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:06:59.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:06:59.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:00] "GET /metrics HTTP/1.1" 200 202259 "" "Prometheus/2.18.1" 2021-12-15T19:07:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:00 smithi122 ceph-mon[32255]: pgmap v2056: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/992463416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:00 smithi022 ceph-mon[31360]: pgmap v2056: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/992463416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:00 smithi022 ceph-mon[40250]: pgmap v2056: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/992463416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:02 smithi022 ceph-mon[31360]: pgmap v2057: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:02 smithi022 ceph-mon[40250]: pgmap v2057: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:02 smithi122 ceph-mon[32255]: pgmap v2057: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:07:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:07:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:04 smithi122 ceph-mon[32255]: pgmap v2058: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:04 smithi022 ceph-mon[31360]: pgmap v2058: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:04 smithi022 ceph-mon[40250]: pgmap v2058: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:04.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:05.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:05.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:05.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2949089559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2949089559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2949089559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:06 smithi122 ceph-mon[32255]: pgmap v2059: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:06 smithi022 ceph-mon[31360]: pgmap v2059: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:06 smithi022 ceph-mon[40250]: pgmap v2059: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:07.348 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:07:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:07:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:07:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:08 smithi122 ceph-mon[32255]: pgmap v2060: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:08 smithi022 ceph-mon[31360]: pgmap v2060: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:08 smithi022 ceph-mon[40250]: pgmap v2060: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:10.151 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:10 smithi122 ceph-mon[32255]: pgmap v2061: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:10] "GET /metrics HTTP/1.1" 200 202257 "" "Prometheus/2.18.1" 2021-12-15T19:07:10.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:10.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:10.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:10.521 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:10 smithi022 ceph-mon[31360]: pgmap v2061: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:10.522 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:10 smithi022 ceph-mon[40250]: pgmap v2061: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2462714988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2462714988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2462714988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:12 smithi122 ceph-mon[32255]: pgmap v2062: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:07:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:07:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:12 smithi022 ceph-mon[31360]: pgmap v2062: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:12 smithi022 ceph-mon[40250]: pgmap v2062: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:14 smithi122 ceph-mon[32255]: pgmap v2063: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:14 smithi022 ceph-mon[31360]: pgmap v2063: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:14 smithi022 ceph-mon[40250]: pgmap v2063: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:15.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:15.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:15.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:16 smithi122 ceph-mon[32255]: pgmap v2064: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4128580905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:16 smithi022 ceph-mon[31360]: pgmap v2064: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4128580905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:16 smithi022 ceph-mon[40250]: pgmap v2064: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4128580905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:17.364 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:07:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:07:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:07:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:18 smithi122 ceph-mon[32255]: pgmap v2065: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:18 smithi022 ceph-mon[31360]: pgmap v2065: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:18 smithi022 ceph-mon[40250]: pgmap v2065: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:20 smithi122 ceph-mon[32255]: pgmap v2066: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:20] "GET /metrics HTTP/1.1" 200 202266 "" "Prometheus/2.18.1" 2021-12-15T19:07:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:20 smithi022 ceph-mon[31360]: pgmap v2066: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:20 smithi022 ceph-mon[40250]: pgmap v2066: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:20.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:21.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:21.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:21.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1967472576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:22 smithi022 ceph-mon[31360]: pgmap v2067: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1967472576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:22 smithi022 ceph-mon[40250]: pgmap v2067: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1967472576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:22 smithi122 ceph-mon[32255]: pgmap v2067: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:07:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:07:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:24 smithi122 ceph-mon[32255]: pgmap v2068: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:24 smithi022 ceph-mon[31360]: pgmap v2068: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:24 smithi022 ceph-mon[40250]: pgmap v2068: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:26.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:26 smithi022 ceph-mon[31360]: pgmap v2069: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:26 smithi022 ceph-mon[40250]: pgmap v2069: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:26.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:26 smithi122 ceph-mon[32255]: pgmap v2069: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:27.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:07:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:07:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3949284664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:07:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3949284664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3949284664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:28 smithi122 ceph-mon[32255]: pgmap v2070: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:28 smithi022 ceph-mon[31360]: pgmap v2070: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:28 smithi022 ceph-mon[40250]: pgmap v2070: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:07:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:30 smithi122 ceph-mon[32255]: pgmap v2071: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:30] "GET /metrics HTTP/1.1" 200 202266 "" "Prometheus/2.18.1" 2021-12-15T19:07:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:30 smithi022 ceph-mon[31360]: pgmap v2071: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:30 smithi022 ceph-mon[40250]: pgmap v2071: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:32.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:32.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:32.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:32 smithi022 ceph-mon[31360]: pgmap v2072: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/485894736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:32 smithi022 ceph-mon[40250]: pgmap v2072: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/485894736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:32 smithi122 ceph-mon[32255]: pgmap v2072: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/485894736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:32.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:07:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:07:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:07:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:07:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:07:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:07:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:07:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:07:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:07:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:07:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:07:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:07:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:07:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:07:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:34 smithi022 ceph-mon[31360]: pgmap v2073: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:34 smithi022 ceph-mon[40250]: pgmap v2073: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:34 smithi122 ceph-mon[32255]: pgmap v2073: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:36 smithi122 ceph-mon[32255]: pgmap v2074: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:36 smithi022 ceph-mon[31360]: pgmap v2074: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:36 smithi022 ceph-mon[40250]: pgmap v2074: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:37.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:37.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:07:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:07:37.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:37.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:37.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:07:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:38 smithi122 ceph-mon[32255]: pgmap v2075: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2706758703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:38 smithi022 ceph-mon[31360]: pgmap v2075: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2706758703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:38 smithi022 ceph-mon[40250]: pgmap v2075: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2706758703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:40 smithi122 ceph-mon[32255]: pgmap v2076: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:40] "GET /metrics HTTP/1.1" 200 202263 "" "Prometheus/2.18.1" 2021-12-15T19:07:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:40 smithi022 ceph-mon[31360]: pgmap v2076: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:40 smithi022 ceph-mon[40250]: pgmap v2076: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:42 smithi022 ceph-mon[31360]: pgmap v2077: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:42 smithi022 ceph-mon[40250]: pgmap v2077: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:07:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:07:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:42 smithi122 ceph-mon[32255]: pgmap v2077: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:42.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:42.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:42.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3358731701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3358731701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3358731701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:44 smithi022 ceph-mon[31360]: pgmap v2078: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:44 smithi022 ceph-mon[40250]: pgmap v2078: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:44 smithi122 ceph-mon[32255]: pgmap v2078: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:46 smithi122 ceph-mon[32255]: pgmap v2079: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:46 smithi022 ceph-mon[31360]: pgmap v2079: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:46 smithi022 ceph-mon[40250]: pgmap v2079: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:47.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:07:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:07:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:47.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:47.750 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:47.750 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:07:47.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:48.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:48.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:48.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:48 smithi022 ceph-mon[31360]: pgmap v2080: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3968350091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:48 smithi022 ceph-mon[40250]: pgmap v2080: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3968350091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:48 smithi122 ceph-mon[32255]: pgmap v2080: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3968350091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:50 smithi122 ceph-mon[32255]: pgmap v2081: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:50] "GET /metrics HTTP/1.1" 200 202261 "" "Prometheus/2.18.1" 2021-12-15T19:07:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:50 smithi022 ceph-mon[31360]: pgmap v2081: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:50 smithi022 ceph-mon[40250]: pgmap v2081: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:07:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:52 smithi022 ceph-mon[31360]: pgmap v2082: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:52 smithi022 ceph-mon[40250]: pgmap v2082: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:52 smithi122 ceph-mon[32255]: pgmap v2082: 132 pgs: 132 active+clean; 582 KiB data, 109 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:07:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:07:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:07:53.119 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:53.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:53.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:53.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:54 smithi122 ceph-mon[32255]: pgmap v2083: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2706821473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:07:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:07:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:07:54.731 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[31360]: pgmap v2083: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:54.732 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2706821473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:54.732 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:07:54.732 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:07:54.732 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:07:54.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[40250]: pgmap v2083: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:54.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2706821473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:07:54.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:07:54.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:07:54.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:07:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:07:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:07:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:07:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:56 smithi022 ceph-mon[31360]: pgmap v2084: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:56 smithi022 ceph-mon[40250]: pgmap v2084: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:57.057 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:56 smithi122 ceph-mon[32255]: pgmap v2084: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:07:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:07:57.748 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:07:58.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:58.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:07:58.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:07:58.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:07:58.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:07:58.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:07:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:58 smithi022 ceph-mon[31360]: pgmap v2085: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:58 smithi022 ceph-mon[40250]: pgmap v2085: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:07:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:58 smithi122 ceph-mon[32255]: pgmap v2085: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:08:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:07:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1162132956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:07:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1162132956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:07:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1162132956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:00] "GET /metrics HTTP/1.1" 200 202261 "" "Prometheus/2.18.1" 2021-12-15T19:08:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:00 smithi022 ceph-mon[31360]: pgmap v2086: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:00 smithi022 ceph-mon[40250]: pgmap v2086: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:00 smithi122 ceph-mon[32255]: pgmap v2086: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:02 smithi022 ceph-mon[31360]: pgmap v2087: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:02 smithi022 ceph-mon[40250]: pgmap v2087: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:02 smithi122 ceph-mon[32255]: pgmap v2087: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:08:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:08:03.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:04.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:04.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:04.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:04 smithi022 ceph-mon[31360]: pgmap v2088: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1824861631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:04 smithi022 ceph-mon[40250]: pgmap v2088: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1824861631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:04 smithi122 ceph-mon[32255]: pgmap v2088: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:08:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1824861631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:06 smithi122 ceph-mon[32255]: pgmap v2089: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:08:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:06 smithi022 ceph-mon[31360]: pgmap v2089: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:08:06.836 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:06 smithi022 ceph-mon[40250]: pgmap v2089: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:08:07.370 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:08:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:08:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:08:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:08 smithi122 ceph-mon[32255]: pgmap v2090: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:08:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:08 smithi022 ceph-mon[31360]: pgmap v2090: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:08:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:08 smithi022 ceph-mon[40250]: pgmap v2090: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:08:09.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:09.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:09.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:09.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:10] "GET /metrics HTTP/1.1" 200 202279 "" "Prometheus/2.18.1" 2021-12-15T19:08:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:10 smithi122 ceph-mon[32255]: pgmap v2091: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:08:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/512690452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:10 smithi022 ceph-mon[31360]: pgmap v2091: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:08:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/512690452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:10 smithi022 ceph-mon[40250]: pgmap v2091: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:08:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/512690452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:12 smithi022 ceph-mon[31360]: pgmap v2092: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:12 smithi022 ceph-mon[40250]: pgmap v2092: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:12 smithi122 ceph-mon[32255]: pgmap v2092: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:08:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:08:14.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:14 smithi122 ceph-mon[32255]: pgmap v2093: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:14 smithi022 ceph-mon[31360]: pgmap v2093: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:14 smithi022 ceph-mon[40250]: pgmap v2093: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:14.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:14.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:14.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/152819765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/152819765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/152819765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:16 smithi122 ceph-mon[32255]: pgmap v2094: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:16 smithi022 ceph-mon[31360]: pgmap v2094: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:16 smithi022 ceph-mon[40250]: pgmap v2094: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:17.369 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:08:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:08:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:08:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:18 smithi122 ceph-mon[32255]: pgmap v2095: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:18 smithi022 ceph-mon[31360]: pgmap v2095: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:18 smithi022 ceph-mon[40250]: pgmap v2095: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:19.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:20.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:20.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:20.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:20] "GET /metrics HTTP/1.1" 200 202276 "" "Prometheus/2.18.1" 2021-12-15T19:08:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:20 smithi122 ceph-mon[32255]: pgmap v2096: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4288511877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:20 smithi022 ceph-mon[31360]: pgmap v2096: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4288511877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:20 smithi022 ceph-mon[40250]: pgmap v2096: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4288511877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:22 smithi022 ceph-mon[31360]: pgmap v2097: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:22 smithi022 ceph-mon[40250]: pgmap v2097: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:22 smithi122 ceph-mon[32255]: pgmap v2097: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T19:08:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:08:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:08:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:24 smithi122 ceph-mon[32255]: pgmap v2098: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:24 smithi022 ceph-mon[31360]: pgmap v2098: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:24 smithi022 ceph-mon[40250]: pgmap v2098: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T19:08:25.341 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:25.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:25.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:25.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:26 smithi122 ceph-mon[32255]: pgmap v2099: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/436244594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:26 smithi022 ceph-mon[31360]: pgmap v2099: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/436244594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:26 smithi022 ceph-mon[40250]: pgmap v2099: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/436244594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:27.368 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:08:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:08:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:08:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:28 smithi122 ceph-mon[32255]: pgmap v2100: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:28 smithi022 ceph-mon[31360]: pgmap v2100: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:28 smithi022 ceph-mon[40250]: pgmap v2100: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:30] "GET /metrics HTTP/1.1" 200 202276 "" "Prometheus/2.18.1" 2021-12-15T19:08:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:30 smithi122 ceph-mon[32255]: pgmap v2101: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:30.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:30.715 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:30 smithi022 ceph-mon[31360]: pgmap v2101: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:30.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:30 smithi022 ceph-mon[40250]: pgmap v2101: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:31.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:31.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:31.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3533561080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3533561080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3533561080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:32 smithi122 ceph-mon[32255]: pgmap v2102: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:08:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:08:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:32 smithi022 ceph-mon[31360]: pgmap v2102: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:32 smithi022 ceph-mon[40250]: pgmap v2102: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:08:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:08:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:08:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:08:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:08:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:08:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:08:33.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:08:33.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:08:33.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:08:33.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:08:33.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:08:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:34 smithi122 ceph-mon[32255]: pgmap v2103: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:34 smithi022 ceph-mon[31360]: pgmap v2103: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:34 smithi022 ceph-mon[40250]: pgmap v2103: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:36.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:36 smithi122 ceph-mon[32255]: pgmap v2104: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:36 smithi022 ceph-mon[31360]: pgmap v2104: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:36 smithi022 ceph-mon[40250]: pgmap v2104: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:37.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:08:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:08:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1500686293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:08:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1500686293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1500686293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:38 smithi122 ceph-mon[32255]: pgmap v2105: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:38 smithi022 ceph-mon[31360]: pgmap v2105: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:38 smithi022 ceph-mon[40250]: pgmap v2105: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:40 smithi122 ceph-mon[32255]: pgmap v2106: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:40] "GET /metrics HTTP/1.1" 200 202279 "" "Prometheus/2.18.1" 2021-12-15T19:08:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:40 smithi022 ceph-mon[40250]: pgmap v2106: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:40 smithi022 ceph-mon[31360]: pgmap v2106: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:41.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:41.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:41.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:41.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:42 smithi022 ceph-mon[31360]: pgmap v2107: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1151022820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:42 smithi022 ceph-mon[40250]: pgmap v2107: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1151022820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:42 smithi122 ceph-mon[32255]: pgmap v2107: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1151022820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:08:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:08:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:44 smithi122 ceph-mon[32255]: pgmap v2108: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:44 smithi022 ceph-mon[31360]: pgmap v2108: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:44 smithi022 ceph-mon[40250]: pgmap v2108: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:46 smithi122 ceph-mon[32255]: pgmap v2109: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:46 smithi022 ceph-mon[31360]: pgmap v2109: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:46 smithi022 ceph-mon[40250]: pgmap v2109: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:46.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:47.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:47.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:47.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:47.375 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:08:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:08:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/196927615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:08:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/196927615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/196927615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:48 smithi122 ceph-mon[32255]: pgmap v2110: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:48 smithi022 ceph-mon[31360]: pgmap v2110: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:48 smithi022 ceph-mon[40250]: pgmap v2110: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:50] "GET /metrics HTTP/1.1" 200 202277 "" "Prometheus/2.18.1" 2021-12-15T19:08:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:50 smithi122 ceph-mon[32255]: pgmap v2111: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:50 smithi022 ceph-mon[31360]: pgmap v2111: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:50 smithi022 ceph-mon[40250]: pgmap v2111: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:52.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:52 smithi022 ceph-mon[31360]: pgmap v2112: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:52 smithi022 ceph-mon[40250]: pgmap v2112: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:52.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:52.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:52.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:52 smithi122 ceph-mon[32255]: pgmap v2112: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:08:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:08:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3323447429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3323447429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3323447429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:54 smithi022 ceph-mon[31360]: pgmap v2113: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:54 smithi022 ceph-mon[40250]: pgmap v2113: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:54 smithi122 ceph-mon[32255]: pgmap v2113: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:08:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:08:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:08:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:08:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:08:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:08:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:08:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:08:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:08:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:08:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:08:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:08:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:08:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:56 smithi022 ceph-mon[31360]: pgmap v2114: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:56 smithi022 ceph-mon[40250]: pgmap v2114: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:56 smithi122 ceph-mon[32255]: pgmap v2114: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:08:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:08:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:08:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:08:57.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:08:57.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:08:57.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:08:57.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:08:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:58 smithi022 ceph-mon[31360]: pgmap v2115: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:08:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/884298573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:58 smithi022 ceph-mon[40250]: pgmap v2115: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:08:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/884298573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:08:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:58 smithi122 ceph-mon[32255]: pgmap v2115: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:08:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:08:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/884298573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:00 smithi122 ceph-mon[32255]: pgmap v2116: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:00] "GET /metrics HTTP/1.1" 200 202277 "" "Prometheus/2.18.1" 2021-12-15T19:09:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:00 smithi022 ceph-mon[31360]: pgmap v2116: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:00 smithi022 ceph-mon[40250]: pgmap v2116: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:02 smithi022 ceph-mon[31360]: pgmap v2117: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:02 smithi022 ceph-mon[40250]: pgmap v2117: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:02 smithi122 ceph-mon[32255]: pgmap v2117: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:09:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:09:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:03.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/412846272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/412846272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/412846272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:04 smithi122 ceph-mon[32255]: pgmap v2118: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:04 smithi022 ceph-mon[31360]: pgmap v2118: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:04 smithi022 ceph-mon[40250]: pgmap v2118: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:06 smithi122 ceph-mon[32255]: pgmap v2119: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:06 smithi022 ceph-mon[31360]: pgmap v2119: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:06 smithi022 ceph-mon[40250]: pgmap v2119: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:07.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:09:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:09:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:09:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:08.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:08.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:08.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:08.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:08.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:08 smithi022 ceph-mon[40250]: pgmap v2120: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:08.675 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:08 smithi022 ceph-mon[31360]: pgmap v2120: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:08 smithi122 ceph-mon[32255]: pgmap v2120: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1955467838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1955467838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1955467838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:10] "GET /metrics HTTP/1.1" 200 202281 "" "Prometheus/2.18.1" 2021-12-15T19:09:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:10 smithi122 ceph-mon[32255]: pgmap v2121: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:10 smithi022 ceph-mon[31360]: pgmap v2121: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:10 smithi022 ceph-mon[40250]: pgmap v2121: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:12 smithi022 ceph-mon[31360]: pgmap v2122: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:12 smithi022 ceph-mon[40250]: pgmap v2122: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:12 smithi122 ceph-mon[32255]: pgmap v2122: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:09:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:09:13.676 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:14.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:14.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:14.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:14 smithi122 ceph-mon[32255]: pgmap v2123: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2951359832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:14 smithi022 ceph-mon[31360]: pgmap v2123: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2951359832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:14 smithi022 ceph-mon[40250]: pgmap v2123: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2951359832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:16 smithi122 ceph-mon[32255]: pgmap v2124: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:16 smithi022 ceph-mon[31360]: pgmap v2124: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:16 smithi022 ceph-mon[40250]: pgmap v2124: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:17.384 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:09:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:09:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:09:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:18 smithi122 ceph-mon[32255]: pgmap v2125: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:18 smithi022 ceph-mon[31360]: pgmap v2125: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:18 smithi022 ceph-mon[40250]: pgmap v2125: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:19.049 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:19.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:19.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:19.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:20 smithi122 ceph-mon[32255]: pgmap v2126: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/471420631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:20] "GET /metrics HTTP/1.1" 200 202282 "" "Prometheus/2.18.1" 2021-12-15T19:09:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:20 smithi022 ceph-mon[31360]: pgmap v2126: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/471420631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:20 smithi022 ceph-mon[40250]: pgmap v2126: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/471420631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:22 smithi022 ceph-mon[31360]: pgmap v2127: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:22 smithi022 ceph-mon[40250]: pgmap v2127: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:22 smithi122 ceph-mon[32255]: pgmap v2127: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:09:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:09:24.419 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:24 smithi122 ceph-mon[32255]: pgmap v2128: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:24 smithi022 ceph-mon[31360]: pgmap v2128: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:24 smithi022 ceph-mon[40250]: pgmap v2128: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:24.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:24.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:24.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/322611014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/322611014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/322611014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:26 smithi122 ceph-mon[32255]: pgmap v2129: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:26 smithi022 ceph-mon[31360]: pgmap v2129: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:26 smithi022 ceph-mon[40250]: pgmap v2129: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:27.390 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:09:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:09:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:09:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:28 smithi122 ceph-mon[32255]: pgmap v2130: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:28 smithi022 ceph-mon[31360]: pgmap v2130: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:28 smithi022 ceph-mon[40250]: pgmap v2130: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:29.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:30.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:30.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:30.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:30] "GET /metrics HTTP/1.1" 200 202282 "" "Prometheus/2.18.1" 2021-12-15T19:09:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:30 smithi122 ceph-mon[32255]: pgmap v2131: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3685337229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:30 smithi022 ceph-mon[31360]: pgmap v2131: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:30.601 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3685337229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:30.602 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:30 smithi022 ceph-mon[40250]: pgmap v2131: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:30.602 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3685337229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:32 smithi022 ceph-mon[31360]: pgmap v2132: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:32 smithi022 ceph-mon[40250]: pgmap v2132: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:32 smithi122 ceph-mon[32255]: pgmap v2132: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:09:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:09:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:09:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:09:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:09:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:09:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:09:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:09:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:09:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:09:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:34 smithi022 ceph-mon[31360]: pgmap v2133: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:34 smithi022 ceph-mon[40250]: pgmap v2133: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:34 smithi122 ceph-mon[32255]: pgmap v2133: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:35.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:35.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:35.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:35.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:36 smithi122 ceph-mon[32255]: pgmap v2134: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2876213534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:36 smithi022 ceph-mon[31360]: pgmap v2134: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2876213534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:36 smithi022 ceph-mon[40250]: pgmap v2134: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2876213534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:37.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:09:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:09:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:09:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:38 smithi122 ceph-mon[32255]: pgmap v2135: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:38 smithi022 ceph-mon[31360]: pgmap v2135: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:38 smithi022 ceph-mon[40250]: pgmap v2135: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:40 smithi122 ceph-mon[32255]: pgmap v2136: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:40] "GET /metrics HTTP/1.1" 200 202299 "" "Prometheus/2.18.1" 2021-12-15T19:09:40.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:40 smithi022 ceph-mon[31360]: pgmap v2136: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:40 smithi022 ceph-mon[40250]: pgmap v2136: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:40.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:40.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:40.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/748097794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/748097794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/748097794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:42 smithi122 ceph-mon[32255]: pgmap v2137: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:09:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:09:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:42 smithi022 ceph-mon[31360]: pgmap v2137: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:42 smithi022 ceph-mon[40250]: pgmap v2137: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:44 smithi122 ceph-mon[32255]: pgmap v2138: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:44 smithi022 ceph-mon[31360]: pgmap v2138: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:44 smithi022 ceph-mon[40250]: pgmap v2138: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:45.900 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:46.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:46.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:46.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:46 smithi022 ceph-mon[31360]: pgmap v2139: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2426432036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:46 smithi022 ceph-mon[40250]: pgmap v2139: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2426432036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:46 smithi122 ceph-mon[32255]: pgmap v2139: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2426432036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:47.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:09:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:09:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:09:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:48 smithi122 ceph-mon[32255]: pgmap v2140: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:48 smithi022 ceph-mon[31360]: pgmap v2140: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:48 smithi022 ceph-mon[40250]: pgmap v2140: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:50 smithi122 ceph-mon[32255]: pgmap v2141: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:50] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:09:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:50 smithi022 ceph-mon[31360]: pgmap v2141: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:50 smithi022 ceph-mon[40250]: pgmap v2141: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:51.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:51.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:51.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:51.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:52 smithi022 ceph-mon[31360]: pgmap v2142: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/8418658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:52 smithi022 ceph-mon[40250]: pgmap v2142: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/8418658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:52 smithi122 ceph-mon[32255]: pgmap v2142: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/8418658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:09:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:09:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:54 smithi122 ceph-mon[32255]: pgmap v2143: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:54 smithi022 ceph-mon[31360]: pgmap v2143: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:54 smithi022 ceph-mon[40250]: pgmap v2143: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:09:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:09:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:09:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:09:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:09:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:09:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:09:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:09:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:09:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:09:56.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:09:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:56 smithi122 ceph-mon[32255]: pgmap v2144: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:56 smithi022 ceph-mon[31360]: pgmap v2144: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:56 smithi022 ceph-mon[40250]: pgmap v2144: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:57.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:09:57.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:09:57.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:09:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:09:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:09:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2670736412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:09:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2670736412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2670736412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:09:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:09:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:58 smithi122 ceph-mon[32255]: pgmap v2145: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:58.759 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:58 smithi022 ceph-mon[31360]: pgmap v2145: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:09:58.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:58 smithi022 ceph-mon[40250]: pgmap v2145: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:09:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:09:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:00.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:00.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:09:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:00] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:10:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:00 smithi022 ceph-mon[31360]: pgmap v2146: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T19:10:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:00 smithi022 ceph-mon[40250]: pgmap v2146: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T19:10:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:00 smithi122 ceph-mon[32255]: pgmap v2146: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T19:10:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:10:02.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:02.270 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:02 smithi022 ceph-mon[31360]: pgmap v2147: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:02.271 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:02 smithi022 ceph-mon[40250]: pgmap v2147: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:02.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:02.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:02.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:02 smithi122 ceph-mon[32255]: pgmap v2147: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:10:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:10:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3331363256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3331363256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3331363256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:04 smithi022 ceph-mon[31360]: pgmap v2148: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:04 smithi022 ceph-mon[40250]: pgmap v2148: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:04 smithi122 ceph-mon[32255]: pgmap v2148: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:06 smithi122 ceph-mon[32255]: pgmap v2149: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:06 smithi022 ceph-mon[31360]: pgmap v2149: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:06 smithi022 ceph-mon[40250]: pgmap v2149: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:07.386 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:07.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:10:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:10:07.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:07.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:07.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:07.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:10:07.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:07.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:08 smithi122 ceph-mon[32255]: pgmap v2150: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1068430754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:08 smithi022 ceph-mon[31360]: pgmap v2150: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1068430754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:08 smithi022 ceph-mon[40250]: pgmap v2150: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1068430754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:10 smithi122 ceph-mon[32255]: pgmap v2151: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:10] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:10:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:10 smithi022 ceph-mon[31360]: pgmap v2151: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:10 smithi022 ceph-mon[40250]: pgmap v2151: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:12 smithi022 ceph-mon[31360]: pgmap v2152: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:12 smithi022 ceph-mon[40250]: pgmap v2152: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:12 smithi122 ceph-mon[32255]: pgmap v2152: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:10:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:10:12.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:13.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:13.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:13.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3788840941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3788840941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3788840941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:14 smithi122 ceph-mon[32255]: pgmap v2153: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:14 smithi022 ceph-mon[31360]: pgmap v2153: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:14 smithi022 ceph-mon[40250]: pgmap v2153: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:16 smithi122 ceph-mon[32255]: pgmap v2154: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:16 smithi022 ceph-mon[31360]: pgmap v2154: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:16 smithi022 ceph-mon[40250]: pgmap v2154: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:17.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:10:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:10:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:10:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:18.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:18.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:18 smithi022 ceph-mon[31360]: pgmap v2155: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:18.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:18.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:18.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:18 smithi122 ceph-mon[32255]: pgmap v2155: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:18 smithi022 ceph-mon[40250]: pgmap v2155: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3985956717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3985956717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3985956717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:20 smithi122 ceph-mon[32255]: pgmap v2156: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:20] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:10:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:20 smithi022 ceph-mon[31360]: pgmap v2156: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:20 smithi022 ceph-mon[40250]: pgmap v2156: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:22 smithi022 ceph-mon[31360]: pgmap v2157: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:22 smithi022 ceph-mon[40250]: pgmap v2157: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:22 smithi122 ceph-mon[32255]: pgmap v2157: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:10:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:10:23.504 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:23.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:23.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:23.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:24 smithi122 ceph-mon[32255]: pgmap v2158: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2475400854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:24 smithi022 ceph-mon[31360]: pgmap v2158: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2475400854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:24 smithi022 ceph-mon[40250]: pgmap v2158: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2475400854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:26 smithi122 ceph-mon[32255]: pgmap v2159: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:26 smithi022 ceph-mon[31360]: pgmap v2159: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:26 smithi022 ceph-mon[40250]: pgmap v2159: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:27.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:10:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:10:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:10:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:28 smithi122 ceph-mon[32255]: pgmap v2160: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:28 smithi022 ceph-mon[31360]: pgmap v2160: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:28 smithi022 ceph-mon[40250]: pgmap v2160: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:28.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:29.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:29.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:29.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2712446380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2712446380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2712446380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:30] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:10:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:30 smithi122 ceph-mon[32255]: pgmap v2161: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:30 smithi022 ceph-mon[31360]: pgmap v2161: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:30 smithi022 ceph-mon[40250]: pgmap v2161: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:32 smithi022 ceph-mon[31360]: pgmap v2162: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:32.610 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:32 smithi022 ceph-mon[40250]: pgmap v2162: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:32 smithi122 ceph-mon[32255]: pgmap v2162: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:10:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:10:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:10:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:10:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:10:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:10:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:10:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:10:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:10:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:10:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:10:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:10:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:10:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:10:34.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:34 smithi022 ceph-mon[31360]: pgmap v2163: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:34 smithi022 ceph-mon[40250]: pgmap v2163: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:34.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:34.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:34.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:34 smithi122 ceph-mon[32255]: pgmap v2163: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4142038706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4142038706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4142038706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:36 smithi122 ceph-mon[32255]: pgmap v2164: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:36 smithi022 ceph-mon[31360]: pgmap v2164: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:36 smithi022 ceph-mon[40250]: pgmap v2164: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:10:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:10:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:10:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:38 smithi122 ceph-mon[32255]: pgmap v2165: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:38 smithi022 ceph-mon[31360]: pgmap v2165: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:38 smithi022 ceph-mon[40250]: pgmap v2165: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:39.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:39.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:39.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:39.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:40] "GET /metrics HTTP/1.1" 200 202297 "" "Prometheus/2.18.1" 2021-12-15T19:10:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:40 smithi122 ceph-mon[32255]: pgmap v2166: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3242921273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:40 smithi022 ceph-mon[31360]: pgmap v2166: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3242921273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:40 smithi022 ceph-mon[40250]: pgmap v2166: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3242921273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:42 smithi122 ceph-mon[32255]: pgmap v2167: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:10:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:10:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:42 smithi022 ceph-mon[31360]: pgmap v2167: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:42 smithi022 ceph-mon[40250]: pgmap v2167: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:44 smithi122 ceph-mon[32255]: pgmap v2168: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:44 smithi022 ceph-mon[31360]: pgmap v2168: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:44 smithi022 ceph-mon[40250]: pgmap v2168: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:44.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:45.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:45.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:45.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1406518107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1406518107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1406518107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:46 smithi122 ceph-mon[32255]: pgmap v2169: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:46 smithi022 ceph-mon[31360]: pgmap v2169: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:46 smithi022 ceph-mon[40250]: pgmap v2169: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:47.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:10:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:10:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:10:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:48 smithi122 ceph-mon[32255]: pgmap v2170: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:48 smithi022 ceph-mon[31360]: pgmap v2170: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:48 smithi022 ceph-mon[40250]: pgmap v2170: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:50.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:50] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:10:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:50 smithi122 ceph-mon[32255]: pgmap v2171: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:50.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:50.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:50.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:50.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:50 smithi022 ceph-mon[40250]: pgmap v2171: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:50.739 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:50 smithi022 ceph-mon[31360]: pgmap v2171: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3893441794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3893441794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3893441794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:52 smithi122 ceph-mon[32255]: pgmap v2172: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:10:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:10:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:52 smithi022 ceph-mon[31360]: pgmap v2172: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:52 smithi022 ceph-mon[40250]: pgmap v2172: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:54 smithi122 ceph-mon[32255]: pgmap v2173: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:54 smithi022 ceph-mon[31360]: pgmap v2173: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:54 smithi022 ceph-mon[40250]: pgmap v2173: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:10:55.739 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:10:56.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:10:56.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:10:56.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:10:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:56 smithi122 ceph-mon[32255]: pgmap v2174: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1086254161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:56 smithi022 ceph-mon[31360]: pgmap v2174: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1086254161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:56 smithi022 ceph-mon[40250]: pgmap v2174: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1086254161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:10:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:57.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:10:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:10:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:10:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:10:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:10:58 smithi122 ceph-mon[32255]: pgmap v2175: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:10:58 smithi022 ceph-mon[31360]: pgmap v2175: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:10:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:10:58 smithi022 ceph-mon[40250]: pgmap v2175: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:00] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:11:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[31360]: pgmap v2176: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:11:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:11:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:11:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[40250]: pgmap v2176: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:11:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:11:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:11:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:00 smithi122 ceph-mon[32255]: pgmap v2176: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:11:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:11:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:11:01.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:01.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:01.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:01.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:11:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/749652114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:11:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/749652114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:11:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/749652114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:02.617 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:11:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:11:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:02 smithi122 ceph-mon[32255]: pgmap v2177: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:02 smithi022 ceph-mon[31360]: pgmap v2177: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:02 smithi022 ceph-mon[40250]: pgmap v2177: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:04 smithi122 ceph-mon[32255]: pgmap v2178: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:04 smithi022 ceph-mon[31360]: pgmap v2178: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:04 smithi022 ceph-mon[40250]: pgmap v2178: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:06.480 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:06 smithi022 ceph-mon[31360]: pgmap v2179: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:06 smithi022 ceph-mon[40250]: pgmap v2179: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:06 smithi122 ceph-mon[32255]: pgmap v2179: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:07.628 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:11:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:11:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:11:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/620863295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/620863295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/620863295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:08 smithi122 ceph-mon[32255]: pgmap v2180: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:08 smithi022 ceph-mon[31360]: pgmap v2180: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:08 smithi022 ceph-mon[40250]: pgmap v2180: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:10] "GET /metrics HTTP/1.1" 200 202306 "" "Prometheus/2.18.1" 2021-12-15T19:11:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:10 smithi122 ceph-mon[32255]: pgmap v2181: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:10 smithi022 ceph-mon[31360]: pgmap v2181: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:10 smithi022 ceph-mon[40250]: pgmap v2181: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:11.851 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:12.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:12.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:12.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:12 smithi022 ceph-mon[31360]: pgmap v2182: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2336141436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:12 smithi022 ceph-mon[40250]: pgmap v2182: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2336141436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:12 smithi122 ceph-mon[32255]: pgmap v2182: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2336141436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:11:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:11:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:14 smithi122 ceph-mon[32255]: pgmap v2183: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:14 smithi022 ceph-mon[31360]: pgmap v2183: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:14 smithi022 ceph-mon[40250]: pgmap v2183: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:16 smithi122 ceph-mon[32255]: pgmap v2184: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:16 smithi022 ceph-mon[31360]: pgmap v2184: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:16 smithi022 ceph-mon[40250]: pgmap v2184: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:17.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:11:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:17.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:17.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:17.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:11:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:11:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:18 smithi122 ceph-mon[32255]: pgmap v2185: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3597405815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:18 smithi022 ceph-mon[31360]: pgmap v2185: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3597405815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:18 smithi022 ceph-mon[40250]: pgmap v2185: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3597405815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:20.424 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:20] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:11:20.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:20 smithi122 ceph-mon[32255]: pgmap v2186: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:20 smithi022 ceph-mon[31360]: pgmap v2186: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:20 smithi022 ceph-mon[40250]: pgmap v2186: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:22.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:22.594 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:22 smithi022 ceph-mon[31360]: pgmap v2187: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:22.595 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:22 smithi022 ceph-mon[40250]: pgmap v2187: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:22 smithi122 ceph-mon[32255]: pgmap v2187: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:11:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:11:22.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:22.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:22.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1181613037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1181613037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1181613037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:24 smithi122 ceph-mon[32255]: pgmap v2188: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:24 smithi022 ceph-mon[40250]: pgmap v2188: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:24 smithi022 ceph-mon[31360]: pgmap v2188: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:26 smithi122 ceph-mon[32255]: pgmap v2189: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:26 smithi022 ceph-mon[31360]: pgmap v2189: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:26 smithi022 ceph-mon[40250]: pgmap v2189: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:11:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:11:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:11:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:27.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:28.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:28.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:28.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:28 smithi122 ceph-mon[32255]: pgmap v2190: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1337707553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:28 smithi022 ceph-mon[31360]: pgmap v2190: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1337707553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:28 smithi022 ceph-mon[40250]: pgmap v2190: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1337707553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:30.425 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:30] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:11:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:30 smithi122 ceph-mon[32255]: pgmap v2191: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:30 smithi022 ceph-mon[31360]: pgmap v2191: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:30 smithi022 ceph-mon[40250]: pgmap v2191: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:32 smithi122 ceph-mon[32255]: pgmap v2192: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:11:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:11:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:32 smithi022 ceph-mon[31360]: pgmap v2192: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:32 smithi022 ceph-mon[40250]: pgmap v2192: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:33.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:11:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:11:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:11:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:11:33.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:33.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:33.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:33.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:11:33.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:11:33.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:11:33.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:11:33.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:11:33.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:11:33.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:11:33.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:11:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:34 smithi122 ceph-mon[32255]: pgmap v2193: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2657094659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:34 smithi022 ceph-mon[31360]: pgmap v2193: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2657094659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:34 smithi022 ceph-mon[40250]: pgmap v2193: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2657094659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:36 smithi122 ceph-mon[32255]: pgmap v2194: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:36 smithi022 ceph-mon[31360]: pgmap v2194: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:36 smithi022 ceph-mon[40250]: pgmap v2194: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:37.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:11:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:11:37.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:11:37.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:38 smithi122 ceph-mon[32255]: pgmap v2195: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:38.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:38.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:38 smithi022 ceph-mon[40250]: pgmap v2195: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:38.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:38 smithi022 ceph-mon[31360]: pgmap v2195: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:39.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:39.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:39.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2230025549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2230025549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2230025549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:40.431 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:40] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:11:40.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:40 smithi122 ceph-mon[32255]: pgmap v2196: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:40 smithi022 ceph-mon[31360]: pgmap v2196: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:40 smithi022 ceph-mon[40250]: pgmap v2196: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:42 smithi122 ceph-mon[32255]: pgmap v2197: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:11:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:11:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:42 smithi022 ceph-mon[31360]: pgmap v2197: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:42 smithi022 ceph-mon[40250]: pgmap v2197: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:44.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:44.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:44.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:44.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:44 smithi122 ceph-mon[32255]: pgmap v2198: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:44 smithi022 ceph-mon[31360]: pgmap v2198: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:44 smithi022 ceph-mon[40250]: pgmap v2198: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1695238090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1695238090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1695238090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:46 smithi122 ceph-mon[32255]: pgmap v2199: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:46 smithi022 ceph-mon[31360]: pgmap v2199: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:46 smithi022 ceph-mon[40250]: pgmap v2199: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:47.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:11:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:11:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:11:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:48 smithi122 ceph-mon[32255]: pgmap v2200: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:48 smithi022 ceph-mon[31360]: pgmap v2200: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:48 smithi022 ceph-mon[40250]: pgmap v2200: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:49.438 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:49.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:49.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:49.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:50] "GET /metrics HTTP/1.1" 200 202310 "" "Prometheus/2.18.1" 2021-12-15T19:11:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:50 smithi022 ceph-mon[31360]: pgmap v2201: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3381640984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:50 smithi022 ceph-mon[40250]: pgmap v2201: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3381640984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:50 smithi122 ceph-mon[32255]: pgmap v2201: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3381640984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:52 smithi122 ceph-mon[32255]: pgmap v2202: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:11:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:11:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:52 smithi022 ceph-mon[31360]: pgmap v2202: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:52 smithi022 ceph-mon[40250]: pgmap v2202: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:54 smithi122 ceph-mon[32255]: pgmap v2203: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:54 smithi022 ceph-mon[31360]: pgmap v2203: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:54 smithi022 ceph-mon[40250]: pgmap v2203: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:11:54.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:11:55.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:11:55.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:11:55.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:11:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/972562570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/972562570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/972562570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:11:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:56 smithi122 ceph-mon[32255]: pgmap v2204: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:56 smithi022 ceph-mon[31360]: pgmap v2204: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:56 smithi022 ceph-mon[40250]: pgmap v2204: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:57.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:11:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:11:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:11:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:11:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:11:58 smithi022 ceph-mon[31360]: pgmap v2205: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:11:58 smithi022 ceph-mon[40250]: pgmap v2205: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:11:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:11:58 smithi122 ceph-mon[32255]: pgmap v2205: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:00.176 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:00] "GET /metrics HTTP/1.1" 200 202310 "" "Prometheus/2.18.1" 2021-12-15T19:12:00.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:00.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:00.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:00 smithi022 ceph-mon[31360]: pgmap v2206: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:00 smithi022 ceph-mon[40250]: pgmap v2206: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:00 smithi122 ceph-mon[32255]: pgmap v2206: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2141144224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:12:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:12:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:12:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:12:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2141144224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:12:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:12:01.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:12:01.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:12:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2141144224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:12:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:12:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:12:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:12:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:02 smithi122 ceph-mon[32255]: pgmap v2207: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:12:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:12:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:02 smithi022 ceph-mon[31360]: pgmap v2207: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:02 smithi022 ceph-mon[40250]: pgmap v2207: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:04 smithi022 ceph-mon[31360]: pgmap v2208: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:04 smithi022 ceph-mon[40250]: pgmap v2208: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:04 smithi122 ceph-mon[32255]: pgmap v2208: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:05.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:05.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:05.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:05.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:06 smithi022 ceph-mon[31360]: pgmap v2209: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/579418306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:06 smithi022 ceph-mon[40250]: pgmap v2209: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/579418306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:06 smithi122 ceph-mon[32255]: pgmap v2209: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/579418306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:07.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:12:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:12:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:12:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:08 smithi022 ceph-mon[31360]: pgmap v2210: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:08 smithi022 ceph-mon[40250]: pgmap v2210: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:08 smithi122 ceph-mon[32255]: pgmap v2210: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:10] "GET /metrics HTTP/1.1" 200 202299 "" "Prometheus/2.18.1" 2021-12-15T19:12:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:10 smithi022 ceph-mon[31360]: pgmap v2211: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:10 smithi022 ceph-mon[40250]: pgmap v2211: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:10.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:10 smithi122 ceph-mon[32255]: pgmap v2211: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:11.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:11.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:11.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2874492636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2874492636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2874492636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:12 smithi122 ceph-mon[32255]: pgmap v2212: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:12:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:12:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:12 smithi022 ceph-mon[31360]: pgmap v2212: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:12 smithi022 ceph-mon[40250]: pgmap v2212: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:14 smithi022 ceph-mon[31360]: pgmap v2213: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:14 smithi022 ceph-mon[40250]: pgmap v2213: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:14 smithi122 ceph-mon[32255]: pgmap v2213: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:16.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:16.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:16.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:16.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:16.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:16 smithi022 ceph-mon[31360]: pgmap v2214: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:16.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:16 smithi022 ceph-mon[40250]: pgmap v2214: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:16 smithi122 ceph-mon[32255]: pgmap v2214: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2322722646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:17.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:12:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:12:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:12:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2322722646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2322722646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:18 smithi022 ceph-mon[31360]: pgmap v2215: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:18 smithi022 ceph-mon[40250]: pgmap v2215: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:18 smithi122 ceph-mon[32255]: pgmap v2215: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:20] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:20 smithi022 ceph-mon[31360]: pgmap v2216: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:20 smithi022 ceph-mon[40250]: pgmap v2216: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:20 smithi122 ceph-mon[32255]: pgmap v2216: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:21.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:22.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:22.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:22.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:12:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:12:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:22 smithi122 ceph-mon[32255]: pgmap v2217: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1912386873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:22 smithi022 ceph-mon[31360]: pgmap v2217: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1912386873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:22 smithi022 ceph-mon[40250]: pgmap v2217: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1912386873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:24 smithi122 ceph-mon[32255]: pgmap v2218: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:24 smithi022 ceph-mon[31360]: pgmap v2218: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:24 smithi022 ceph-mon[40250]: pgmap v2218: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:26 smithi022 ceph-mon[31360]: pgmap v2219: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:26 smithi022 ceph-mon[40250]: pgmap v2219: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:26 smithi122 ceph-mon[32255]: pgmap v2219: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:27.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:27.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:27.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:27.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:12:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:12:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1514269696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:12:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1514269696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1514269696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:28 smithi022 ceph-mon[31360]: pgmap v2220: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:28 smithi022 ceph-mon[40250]: pgmap v2220: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:28 smithi122 ceph-mon[32255]: pgmap v2220: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:30] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:12:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:30 smithi022 ceph-mon[31360]: pgmap v2221: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:30 smithi022 ceph-mon[40250]: pgmap v2221: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:30 smithi122 ceph-mon[32255]: pgmap v2221: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:32.394 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:32 smithi122 ceph-mon[32255]: pgmap v2222: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:12:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:12:32.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:32.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:32.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:32.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:32 smithi022 ceph-mon[31360]: pgmap v2222: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:32.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:32 smithi022 ceph-mon[40250]: pgmap v2222: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:12:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:12:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:12:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:12:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2188101889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:12:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:12:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:12:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:12:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2188101889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:12:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:12:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:12:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:12:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2188101889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:34 smithi122 ceph-mon[32255]: pgmap v2223: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:34 smithi022 ceph-mon[31360]: pgmap v2223: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:34 smithi022 ceph-mon[40250]: pgmap v2223: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:36 smithi022 ceph-mon[31360]: pgmap v2224: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:36 smithi022 ceph-mon[40250]: pgmap v2224: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:36 smithi122 ceph-mon[32255]: pgmap v2224: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:37.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:12:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:12:37.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:37.765 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:37.765 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:12:37.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:38.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:38.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:38.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:38 smithi022 ceph-mon[31360]: pgmap v2225: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3607238764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:38 smithi022 ceph-mon[40250]: pgmap v2225: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3607238764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:38 smithi122 ceph-mon[32255]: pgmap v2225: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3607238764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:40] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:12:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:40 smithi022 ceph-mon[31360]: pgmap v2226: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:40 smithi022 ceph-mon[40250]: pgmap v2226: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:40 smithi122 ceph-mon[32255]: pgmap v2226: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:42 smithi122 ceph-mon[32255]: pgmap v2227: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:12:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:12:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:42 smithi022 ceph-mon[31360]: pgmap v2227: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:42 smithi022 ceph-mon[40250]: pgmap v2227: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:43.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:43.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:43.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:43.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:44 smithi022 ceph-mon[31360]: pgmap v2228: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1165556522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:44 smithi022 ceph-mon[40250]: pgmap v2228: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1165556522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:44 smithi122 ceph-mon[32255]: pgmap v2228: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1165556522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:46 smithi022 ceph-mon[31360]: pgmap v2229: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:46 smithi022 ceph-mon[40250]: pgmap v2229: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:46 smithi122 ceph-mon[32255]: pgmap v2229: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:47.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:12:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:12:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:12:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:48.508 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:48 smithi022 ceph-mon[31360]: pgmap v2230: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:48 smithi022 ceph-mon[40250]: pgmap v2230: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:48.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:48.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:48.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:48 smithi122 ceph-mon[32255]: pgmap v2230: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1910015940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1910015940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1910015940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:50] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:12:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:50 smithi022 ceph-mon[31360]: pgmap v2231: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:50 smithi022 ceph-mon[40250]: pgmap v2231: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:50 smithi122 ceph-mon[32255]: pgmap v2231: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:52 smithi122 ceph-mon[32255]: pgmap v2232: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:12:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:12:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:52 smithi022 ceph-mon[31360]: pgmap v2232: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:52 smithi022 ceph-mon[40250]: pgmap v2232: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:53.882 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:54.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:54.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:54.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:12:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:54 smithi022 ceph-mon[40250]: pgmap v2233: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/538216668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:54 smithi022 ceph-mon[31360]: pgmap v2233: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/538216668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:54 smithi122 ceph-mon[32255]: pgmap v2233: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:12:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/538216668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:12:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:56 smithi022 ceph-mon[31360]: pgmap v2234: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:56 smithi022 ceph-mon[40250]: pgmap v2234: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:56 smithi122 ceph-mon[32255]: pgmap v2234: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:12:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:12:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:12:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:12:58 smithi022 ceph-mon[31360]: pgmap v2235: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:12:58 smithi022 ceph-mon[40250]: pgmap v2235: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:12:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:12:58 smithi122 ceph-mon[32255]: pgmap v2235: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:12:59.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:12:59.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:12:59.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:12:59.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:00] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:13:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:00 smithi022 ceph-mon[31360]: pgmap v2236: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4104052709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:00 smithi022 ceph-mon[40250]: pgmap v2236: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4104052709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:00 smithi122 ceph-mon[32255]: pgmap v2236: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4104052709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:13:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:13:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:13:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:13:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:13:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:13:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:13:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:13:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:13:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:02 smithi122 ceph-mon[32255]: pgmap v2237: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:13:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:13:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:13:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:02 smithi022 ceph-mon[31360]: pgmap v2237: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:13:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:02 smithi022 ceph-mon[40250]: pgmap v2237: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:13:04.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:04 smithi122 ceph-mon[32255]: pgmap v2238: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:04.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:04.990 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:04 smithi022 ceph-mon[31360]: pgmap v2238: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:04.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:04 smithi022 ceph-mon[40250]: pgmap v2238: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3742549790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3742549790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3742549790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:06 smithi122 ceph-mon[32255]: pgmap v2239: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:06 smithi022 ceph-mon[31360]: pgmap v2239: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:06 smithi022 ceph-mon[40250]: pgmap v2239: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:13:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:13:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:13:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:08 smithi122 ceph-mon[32255]: pgmap v2240: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:08 smithi022 ceph-mon[31360]: pgmap v2240: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:08 smithi022 ceph-mon[40250]: pgmap v2240: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:09.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:10.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:10.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:10.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:10] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:13:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:10 smithi022 ceph-mon[31360]: pgmap v2241: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1700216581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:10 smithi022 ceph-mon[40250]: pgmap v2241: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1700216581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:10 smithi122 ceph-mon[32255]: pgmap v2241: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1700216581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:12 smithi122 ceph-mon[32255]: pgmap v2242: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:13:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:13:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:12 smithi022 ceph-mon[31360]: pgmap v2242: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:12 smithi022 ceph-mon[40250]: pgmap v2242: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:14 smithi022 ceph-mon[31360]: pgmap v2243: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:14 smithi022 ceph-mon[40250]: pgmap v2243: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:14 smithi122 ceph-mon[32255]: pgmap v2243: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:15.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:15.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:15.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:16 smithi022 ceph-mon[31360]: pgmap v2244: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/318969738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:16 smithi022 ceph-mon[40250]: pgmap v2244: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/318969738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:16 smithi122 ceph-mon[32255]: pgmap v2244: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/318969738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:13:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:13:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:13:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:18 smithi022 ceph-mon[31360]: pgmap v2245: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:18 smithi022 ceph-mon[40250]: pgmap v2245: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:18 smithi122 ceph-mon[32255]: pgmap v2245: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:20] "GET /metrics HTTP/1.1" 200 202302 "" "Prometheus/2.18.1" 2021-12-15T19:13:20.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:20.733 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:20 smithi022 ceph-mon[31360]: pgmap v2246: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:20.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:20 smithi022 ceph-mon[40250]: pgmap v2246: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:20 smithi122 ceph-mon[32255]: pgmap v2246: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:21.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:21.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:21.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1937146106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1937146106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1937146106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:22 smithi122 ceph-mon[32255]: pgmap v2247: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:13:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:13:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:22 smithi022 ceph-mon[31360]: pgmap v2247: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:22 smithi022 ceph-mon[40250]: pgmap v2247: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:24 smithi022 ceph-mon[31360]: pgmap v2248: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:24 smithi022 ceph-mon[40250]: pgmap v2248: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:24 smithi122 ceph-mon[32255]: pgmap v2248: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:26.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:26.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:26.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:26.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:26 smithi022 ceph-mon[31360]: pgmap v2249: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3219877178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:26 smithi022 ceph-mon[40250]: pgmap v2249: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3219877178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:26 smithi122 ceph-mon[32255]: pgmap v2249: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3219877178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:13:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:13:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:13:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:28 smithi022 ceph-mon[31360]: pgmap v2250: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:28 smithi022 ceph-mon[40250]: pgmap v2250: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:28 smithi122 ceph-mon[32255]: pgmap v2250: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:30] "GET /metrics HTTP/1.1" 200 202302 "" "Prometheus/2.18.1" 2021-12-15T19:13:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:30 smithi022 ceph-mon[31360]: pgmap v2251: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:30 smithi022 ceph-mon[40250]: pgmap v2251: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:30 smithi122 ceph-mon[32255]: pgmap v2251: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:31.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:31.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:31.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:31.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:32 smithi122 ceph-mon[32255]: pgmap v2252: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2755233090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:32.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:13:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:13:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:32 smithi022 ceph-mon[31360]: pgmap v2252: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2755233090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:32 smithi022 ceph-mon[40250]: pgmap v2252: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2755233090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:13:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:13:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:13:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:13:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:13:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:13:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:13:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:13:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:13:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:13:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:13:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:13:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:34 smithi122 ceph-mon[32255]: pgmap v2253: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:34 smithi022 ceph-mon[31360]: pgmap v2253: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:34 smithi022 ceph-mon[40250]: pgmap v2253: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:36 smithi022 ceph-mon[31360]: pgmap v2254: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:36 smithi022 ceph-mon[40250]: pgmap v2254: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:36.846 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:36 smithi122 ceph-mon[32255]: pgmap v2254: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:37.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:37.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:37.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3484798147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:13:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3484798147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3484798147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:37.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:13:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:13:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:38 smithi022 ceph-mon[31360]: pgmap v2255: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:38 smithi022 ceph-mon[40250]: pgmap v2255: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:38 smithi122 ceph-mon[32255]: pgmap v2255: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:40] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:13:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:40 smithi022 ceph-mon[31360]: pgmap v2256: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:40 smithi022 ceph-mon[40250]: pgmap v2256: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:40 smithi122 ceph-mon[32255]: pgmap v2256: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:42.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:42 smithi022 ceph-mon[31360]: pgmap v2257: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:42 smithi022 ceph-mon[40250]: pgmap v2257: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:42.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:42.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:42.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:42 smithi122 ceph-mon[32255]: pgmap v2257: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:13:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:13:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3963806031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3963806031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3963806031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:44 smithi122 ceph-mon[32255]: pgmap v2258: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:44 smithi022 ceph-mon[31360]: pgmap v2258: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:44 smithi022 ceph-mon[40250]: pgmap v2258: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:46 smithi022 ceph-mon[31360]: pgmap v2259: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:46 smithi022 ceph-mon[40250]: pgmap v2259: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:46 smithi122 ceph-mon[32255]: pgmap v2259: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:47.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:47.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:13:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:13:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:13:47.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:47.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:47.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:48.967 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:48 smithi022 ceph-mon[31360]: pgmap v2260: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:48.967 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2381986179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:48.969 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:48.969 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:48 smithi022 ceph-mon[40250]: pgmap v2260: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:48.969 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2381986179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:48.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:48.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:48 smithi122 ceph-mon[32255]: pgmap v2260: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:48.971 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2381986179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:50] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:13:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:50 smithi022 ceph-mon[31360]: pgmap v2261: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:50 smithi022 ceph-mon[40250]: pgmap v2261: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:50 smithi122 ceph-mon[32255]: pgmap v2261: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:52 smithi122 ceph-mon[32255]: pgmap v2262: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:13:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:13:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:52 smithi022 ceph-mon[31360]: pgmap v2262: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:52 smithi022 ceph-mon[40250]: pgmap v2262: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:52.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3078935955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3078935955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3078935955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:54 smithi122 ceph-mon[32255]: pgmap v2263: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:54 smithi022 ceph-mon[31360]: pgmap v2263: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:54 smithi022 ceph-mon[40250]: pgmap v2263: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:13:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:56 smithi022 ceph-mon[31360]: pgmap v2264: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:56 smithi022 ceph-mon[40250]: pgmap v2264: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:56 smithi122 ceph-mon[32255]: pgmap v2264: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:13:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:13:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:13:58.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:13:58.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:13:58.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:13:58.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:13:58.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:58.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:58 smithi022 ceph-mon[40250]: pgmap v2265: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:58.698 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:58.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:58 smithi022 ceph-mon[31360]: pgmap v2265: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:13:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:58 smithi122 ceph-mon[32255]: pgmap v2265: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:13:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:13:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1509087011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:13:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1509087011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:13:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:13:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1509087011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:00] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:14:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:00 smithi022 ceph-mon[31360]: pgmap v2266: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:00 smithi022 ceph-mon[40250]: pgmap v2266: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:00 smithi122 ceph-mon[32255]: pgmap v2266: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:14:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:14:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:02 smithi122 ceph-mon[32255]: pgmap v2267: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:14:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:14:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:14:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:14:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[31360]: pgmap v2267: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:14:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:14:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:14:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:14:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[40250]: pgmap v2267: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:14:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:14:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:14:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:14:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:04.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:04.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:04.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:04 smithi022 ceph-mon[31360]: pgmap v2268: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2860987866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:04 smithi022 ceph-mon[40250]: pgmap v2268: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2860987866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:04 smithi122 ceph-mon[32255]: pgmap v2268: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2860987866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:06 smithi022 ceph-mon[31360]: pgmap v2269: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:06 smithi022 ceph-mon[40250]: pgmap v2269: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:06 smithi122 ceph-mon[32255]: pgmap v2269: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:14:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:14:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:14:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:08 smithi022 ceph-mon[31360]: pgmap v2270: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:08 smithi022 ceph-mon[40250]: pgmap v2270: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:08 smithi122 ceph-mon[32255]: pgmap v2270: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:09.074 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:09.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:09.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:09.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1436486601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1436486601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1436486601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:10] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:14:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:10 smithi022 ceph-mon[31360]: pgmap v2271: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:10 smithi022 ceph-mon[40250]: pgmap v2271: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:10 smithi122 ceph-mon[32255]: pgmap v2271: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:12 smithi122 ceph-mon[32255]: pgmap v2272: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:14:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:14:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:12 smithi022 ceph-mon[31360]: pgmap v2272: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:12 smithi022 ceph-mon[40250]: pgmap v2272: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:14.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:14 smithi022 ceph-mon[31360]: pgmap v2273: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:14 smithi022 ceph-mon[40250]: pgmap v2273: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:14.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:14.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:14.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:14 smithi122 ceph-mon[32255]: pgmap v2273: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1377199688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1377199688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1377199688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:16 smithi022 ceph-mon[31360]: pgmap v2274: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:16 smithi022 ceph-mon[40250]: pgmap v2274: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:16 smithi122 ceph-mon[32255]: pgmap v2274: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:14:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:14:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:14:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:18 smithi022 ceph-mon[31360]: pgmap v2275: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:18 smithi022 ceph-mon[40250]: pgmap v2275: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:18 smithi122 ceph-mon[32255]: pgmap v2275: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:19.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:20.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:20.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:20.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:20] "GET /metrics HTTP/1.1" 200 202299 "" "Prometheus/2.18.1" 2021-12-15T19:14:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:20 smithi022 ceph-mon[31360]: pgmap v2276: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/233378778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:20 smithi022 ceph-mon[40250]: pgmap v2276: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/233378778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:20 smithi122 ceph-mon[32255]: pgmap v2276: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/233378778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:22 smithi122 ceph-mon[32255]: pgmap v2277: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:14:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:14:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:22 smithi022 ceph-mon[31360]: pgmap v2277: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:22 smithi022 ceph-mon[40250]: pgmap v2277: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:24 smithi022 ceph-mon[31360]: pgmap v2278: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:24 smithi022 ceph-mon[40250]: pgmap v2278: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:24 smithi122 ceph-mon[32255]: pgmap v2278: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:25.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:25.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:25.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:25.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:26.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:26 smithi022 ceph-mon[31360]: pgmap v2279: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:26.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/998140432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:26.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:26 smithi022 ceph-mon[40250]: pgmap v2279: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:26.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/998140432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:26 smithi122 ceph-mon[32255]: pgmap v2279: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/998140432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:14:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:14:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:14:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:28 smithi022 ceph-mon[31360]: pgmap v2280: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:28 smithi022 ceph-mon[40250]: pgmap v2280: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:28 smithi122 ceph-mon[32255]: pgmap v2280: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:30] "GET /metrics HTTP/1.1" 200 202299 "" "Prometheus/2.18.1" 2021-12-15T19:14:30.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:30 smithi022 ceph-mon[31360]: pgmap v2281: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:30 smithi022 ceph-mon[40250]: pgmap v2281: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:30.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:30.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:30.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:30 smithi122 ceph-mon[32255]: pgmap v2281: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1857277936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1857277936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1857277936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:14:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:14:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:32 smithi122 ceph-mon[32255]: pgmap v2282: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:14:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:14:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:14:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:14:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[31360]: pgmap v2282: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:14:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:14:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:14:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[40250]: pgmap v2282: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:14:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:14:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:14:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:14:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:34 smithi022 ceph-mon[40250]: pgmap v2283: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:34 smithi022 ceph-mon[31360]: pgmap v2283: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:34 smithi122 ceph-mon[32255]: pgmap v2283: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:36.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:36.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:36.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:36 smithi022 ceph-mon[31360]: pgmap v2284: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/492605527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:36 smithi022 ceph-mon[40250]: pgmap v2284: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/492605527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:36 smithi122 ceph-mon[32255]: pgmap v2284: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/492605527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:14:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:14:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:14:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:38 smithi022 ceph-mon[31360]: pgmap v2285: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:38 smithi022 ceph-mon[40250]: pgmap v2285: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:38 smithi122 ceph-mon[32255]: pgmap v2285: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:40] "GET /metrics HTTP/1.1" 200 202299 "" "Prometheus/2.18.1" 2021-12-15T19:14:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:40 smithi022 ceph-mon[40250]: pgmap v2286: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:40 smithi022 ceph-mon[31360]: pgmap v2286: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:40 smithi122 ceph-mon[32255]: pgmap v2286: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:41.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:41.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:41.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:41.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:14:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:14:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:42 smithi122 ceph-mon[32255]: pgmap v2287: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/865077611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:42 smithi022 ceph-mon[31360]: pgmap v2287: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/865077611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:42 smithi022 ceph-mon[40250]: pgmap v2287: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/865077611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:44 smithi022 ceph-mon[31360]: pgmap v2288: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:44 smithi022 ceph-mon[40250]: pgmap v2288: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:44 smithi122 ceph-mon[32255]: pgmap v2288: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:46.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:46 smithi022 ceph-mon[31360]: pgmap v2289: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:46 smithi022 ceph-mon[40250]: pgmap v2289: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:46 smithi122 ceph-mon[32255]: pgmap v2289: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:47.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:47.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:47.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3428907060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:47.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:14:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:14:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3428907060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:14:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3428907060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:48 smithi022 ceph-mon[31360]: pgmap v2290: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:48 smithi022 ceph-mon[40250]: pgmap v2290: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:48 smithi122 ceph-mon[32255]: pgmap v2290: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:50] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:14:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:50 smithi022 ceph-mon[31360]: pgmap v2291: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:50 smithi022 ceph-mon[40250]: pgmap v2291: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:50 smithi122 ceph-mon[32255]: pgmap v2291: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:52.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:52.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:52.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:52.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:52 smithi122 ceph-mon[32255]: pgmap v2292: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:14:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:14:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:52 smithi022 ceph-mon[31360]: pgmap v2292: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:52 smithi022 ceph-mon[40250]: pgmap v2292: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2455225998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2455225998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2455225998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:54 smithi122 ceph-mon[32255]: pgmap v2293: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:54 smithi022 ceph-mon[31360]: pgmap v2293: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:54 smithi022 ceph-mon[40250]: pgmap v2293: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:14:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:56 smithi022 ceph-mon[31360]: pgmap v2294: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:56 smithi022 ceph-mon[40250]: pgmap v2294: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:56 smithi122 ceph-mon[32255]: pgmap v2294: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:57.403 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:14:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:14:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:14:57.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:14:57.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:14:57.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:14:57.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:14:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:58 smithi022 ceph-mon[31360]: pgmap v2295: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:14:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2888183678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:58 smithi022 ceph-mon[40250]: pgmap v2295: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:14:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2888183678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:14:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:14:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:58 smithi122 ceph-mon[32255]: pgmap v2295: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:14:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:14:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2888183678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:00] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:15:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:00 smithi022 ceph-mon[31360]: pgmap v2296: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:00 smithi022 ceph-mon[40250]: pgmap v2296: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:00 smithi122 ceph-mon[32255]: pgmap v2296: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:15:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:15:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:02 smithi122 ceph-mon[32255]: pgmap v2297: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:15:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:15:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:15:02.774 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:02.774 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[40250]: pgmap v2297: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:02.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:15:02.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:15:02.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:15:02.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[31360]: pgmap v2297: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:02.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:15:02.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:15:02.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:15:03.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:03.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:03.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:15:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3614662759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:15:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3614662759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:15:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3614662759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:04 smithi022 ceph-mon[31360]: pgmap v2298: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:04 smithi022 ceph-mon[40250]: pgmap v2298: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:04 smithi122 ceph-mon[32255]: pgmap v2298: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:06 smithi022 ceph-mon[31360]: pgmap v2299: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:06 smithi022 ceph-mon[40250]: pgmap v2299: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:06 smithi122 ceph-mon[32255]: pgmap v2299: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:07.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:15:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:15:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:15:08.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:08.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:08.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:08.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:08 smithi022 ceph-mon[31360]: pgmap v2300: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:08 smithi022 ceph-mon[40250]: pgmap v2300: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:08 smithi122 ceph-mon[32255]: pgmap v2300: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/410459892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/410459892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/410459892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:10] "GET /metrics HTTP/1.1" 200 202297 "" "Prometheus/2.18.1" 2021-12-15T19:15:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:10 smithi022 ceph-mon[31360]: pgmap v2301: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:10 smithi022 ceph-mon[40250]: pgmap v2301: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:10 smithi122 ceph-mon[32255]: pgmap v2301: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:12 smithi122 ceph-mon[32255]: pgmap v2302: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:15:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:15:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:12 smithi022 ceph-mon[31360]: pgmap v2302: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:12 smithi022 ceph-mon[40250]: pgmap v2302: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:13.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:14 smithi022 ceph-mon[31360]: pgmap v2303: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1685834588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:14 smithi022 ceph-mon[40250]: pgmap v2303: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1685834588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:14 smithi122 ceph-mon[32255]: pgmap v2303: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1685834588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:16 smithi022 ceph-mon[31360]: pgmap v2304: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:16 smithi022 ceph-mon[40250]: pgmap v2304: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:16 smithi122 ceph-mon[32255]: pgmap v2304: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:15:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:15:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:15:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:18 smithi022 ceph-mon[31360]: pgmap v2305: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:18 smithi022 ceph-mon[40250]: pgmap v2305: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:18.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:18 smithi122 ceph-mon[32255]: pgmap v2305: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:19.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:19.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:19.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1456097739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1456097739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1456097739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:20] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:15:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:20 smithi022 ceph-mon[31360]: pgmap v2306: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:20 smithi022 ceph-mon[40250]: pgmap v2306: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:20 smithi122 ceph-mon[32255]: pgmap v2306: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:22 smithi122 ceph-mon[32255]: pgmap v2307: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:15:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:15:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:22 smithi022 ceph-mon[31360]: pgmap v2307: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:22 smithi022 ceph-mon[40250]: pgmap v2307: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:24.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:24 smithi022 ceph-mon[31360]: pgmap v2308: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:24 smithi022 ceph-mon[40250]: pgmap v2308: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:24.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:24.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:24.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:24 smithi122 ceph-mon[32255]: pgmap v2308: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3804720999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3804720999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3804720999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:26 smithi022 ceph-mon[31360]: pgmap v2309: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:26 smithi022 ceph-mon[40250]: pgmap v2309: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:26 smithi122 ceph-mon[32255]: pgmap v2309: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:15:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:15:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:15:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:28 smithi022 ceph-mon[31360]: pgmap v2310: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:28 smithi022 ceph-mon[40250]: pgmap v2310: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:28 smithi122 ceph-mon[32255]: pgmap v2310: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:29.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:29.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:29.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:29.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:30] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:15:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:30 smithi022 ceph-mon[31360]: pgmap v2311: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2327873237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:30 smithi022 ceph-mon[40250]: pgmap v2311: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2327873237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:30 smithi122 ceph-mon[32255]: pgmap v2311: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2327873237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:32 smithi122 ceph-mon[32255]: pgmap v2312: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:15:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:15:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:32 smithi022 ceph-mon[31360]: pgmap v2312: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:32 smithi022 ceph-mon[40250]: pgmap v2312: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:15:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:15:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:15:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:15:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:15:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:15:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:15:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:15:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:15:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:15:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:15:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:15:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:34 smithi122 ceph-mon[32255]: pgmap v2313: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:34 smithi022 ceph-mon[31360]: pgmap v2313: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:34 smithi022 ceph-mon[40250]: pgmap v2313: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:34.991 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:35.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:35.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:35.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3994050015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3994050015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3994050015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:36 smithi122 ceph-mon[32255]: pgmap v2314: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:36 smithi022 ceph-mon[31360]: pgmap v2314: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:36 smithi022 ceph-mon[40250]: pgmap v2314: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:15:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:15:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:15:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:38 smithi022 ceph-mon[31360]: pgmap v2315: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:38 smithi022 ceph-mon[40250]: pgmap v2315: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:38 smithi122 ceph-mon[32255]: pgmap v2315: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:40.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:40] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:15:40.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:40.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:40.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:40.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:40 smithi022 ceph-mon[40250]: pgmap v2316: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:40.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:40 smithi022 ceph-mon[31360]: pgmap v2316: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:40 smithi122 ceph-mon[32255]: pgmap v2316: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1534619532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1534619532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1534619532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:15:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:15:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:42 smithi122 ceph-mon[32255]: pgmap v2317: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:42 smithi022 ceph-mon[31360]: pgmap v2317: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:42 smithi022 ceph-mon[40250]: pgmap v2317: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:44 smithi022 ceph-mon[31360]: pgmap v2318: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:44 smithi022 ceph-mon[40250]: pgmap v2318: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:44 smithi122 ceph-mon[32255]: pgmap v2318: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:45.735 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:46.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:46.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:46.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:46 smithi022 ceph-mon[31360]: pgmap v2319: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1810667123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:46 smithi022 ceph-mon[40250]: pgmap v2319: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1810667123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:46 smithi122 ceph-mon[32255]: pgmap v2319: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1810667123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:47.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:15:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:15:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:15:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:48 smithi022 ceph-mon[31360]: pgmap v2320: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:48 smithi022 ceph-mon[40250]: pgmap v2320: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:48 smithi122 ceph-mon[32255]: pgmap v2320: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:50] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:15:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:50 smithi022 ceph-mon[31360]: pgmap v2321: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:50 smithi022 ceph-mon[40250]: pgmap v2321: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:50 smithi122 ceph-mon[32255]: pgmap v2321: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:51.106 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:51.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:51.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:51.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:52 smithi122 ceph-mon[32255]: pgmap v2322: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187868901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:15:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:15:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:52 smithi022 ceph-mon[31360]: pgmap v2322: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187868901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:52 smithi022 ceph-mon[40250]: pgmap v2322: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187868901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:54 smithi022 ceph-mon[31360]: pgmap v2323: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:54 smithi022 ceph-mon[40250]: pgmap v2323: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:54 smithi122 ceph-mon[32255]: pgmap v2323: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:15:56.478 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:15:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:56 smithi022 ceph-mon[31360]: pgmap v2324: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:56 smithi022 ceph-mon[40250]: pgmap v2324: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:56.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:15:56.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:15:56.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:15:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:56 smithi122 ceph-mon[32255]: pgmap v2324: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:57.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:15:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:15:57.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2102381537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2102381537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:15:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2102381537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:15:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:15:58 smithi022 ceph-mon[31360]: pgmap v2325: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:15:58 smithi022 ceph-mon[40250]: pgmap v2325: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:15:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:15:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:15:58 smithi122 ceph-mon[32255]: pgmap v2325: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:00] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:16:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:00 smithi022 ceph-mon[31360]: pgmap v2326: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:00 smithi022 ceph-mon[40250]: pgmap v2326: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:00 smithi122 ceph-mon[32255]: pgmap v2326: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:01.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:02.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:02.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:02.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:02.497 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:02 smithi022 ceph-mon[31360]: pgmap v2327: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:02.498 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3129430936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:02.498 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:02 smithi022 ceph-mon[40250]: pgmap v2327: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:02.498 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3129430936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:02 smithi122 ceph-mon[32255]: pgmap v2327: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3129430936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:16:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:16:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:16:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:16:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:16:03.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:16:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:16:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:16:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:16:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:16:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:16:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:16:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:04 smithi122 ceph-mon[32255]: pgmap v2328: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:04 smithi022 ceph-mon[31360]: pgmap v2328: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:04 smithi022 ceph-mon[40250]: pgmap v2328: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:06 smithi022 ceph-mon[31360]: pgmap v2329: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:06 smithi022 ceph-mon[40250]: pgmap v2329: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:06 smithi122 ceph-mon[32255]: pgmap v2329: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:07.220 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:16:07.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:07.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:07.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:16:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:16:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:08 smithi022 ceph-mon[31360]: pgmap v2330: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/22334714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:08 smithi022 ceph-mon[40250]: pgmap v2330: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/22334714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:08 smithi122 ceph-mon[32255]: pgmap v2330: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/22334714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:10] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:16:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:10 smithi022 ceph-mon[31360]: pgmap v2331: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:10 smithi022 ceph-mon[40250]: pgmap v2331: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:10 smithi122 ceph-mon[32255]: pgmap v2331: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:12.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:12 smithi122 ceph-mon[32255]: pgmap v2332: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:16:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:16:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:12 smithi022 ceph-mon[31360]: pgmap v2332: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:12 smithi022 ceph-mon[40250]: pgmap v2332: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/15197571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/15197571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/15197571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:14 smithi122 ceph-mon[32255]: pgmap v2333: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:14 smithi022 ceph-mon[31360]: pgmap v2333: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:14 smithi022 ceph-mon[40250]: pgmap v2333: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:16 smithi022 ceph-mon[31360]: pgmap v2334: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:16 smithi022 ceph-mon[40250]: pgmap v2334: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:16 smithi122 ceph-mon[32255]: pgmap v2334: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:16:17.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:16:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:16:17.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:18.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:18.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:18.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:18 smithi022 ceph-mon[31360]: pgmap v2335: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2135875793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:18 smithi022 ceph-mon[40250]: pgmap v2335: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2135875793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:18 smithi122 ceph-mon[32255]: pgmap v2335: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2135875793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:20] "GET /metrics HTTP/1.1" 200 202295 "" "Prometheus/2.18.1" 2021-12-15T19:16:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:20 smithi022 ceph-mon[31360]: pgmap v2336: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:20 smithi022 ceph-mon[40250]: pgmap v2336: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:20 smithi122 ceph-mon[32255]: pgmap v2336: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:16:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:16:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:22 smithi122 ceph-mon[32255]: pgmap v2337: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:22 smithi022 ceph-mon[31360]: pgmap v2337: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:22 smithi022 ceph-mon[40250]: pgmap v2337: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:23.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:24 smithi022 ceph-mon[31360]: pgmap v2338: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2928228155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:24 smithi022 ceph-mon[40250]: pgmap v2338: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2928228155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:24 smithi122 ceph-mon[32255]: pgmap v2338: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2928228155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:26 smithi022 ceph-mon[31360]: pgmap v2339: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:26 smithi022 ceph-mon[40250]: pgmap v2339: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:26 smithi122 ceph-mon[32255]: pgmap v2339: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:16:27.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:16:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:16:28.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:28 smithi022 ceph-mon[31360]: pgmap v2340: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:28 smithi022 ceph-mon[40250]: pgmap v2340: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:28.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:28 smithi122 ceph-mon[32255]: pgmap v2340: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:29.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:29.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:29.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:29.515 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4275238976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4275238976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4275238976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:30] "GET /metrics HTTP/1.1" 200 202295 "" "Prometheus/2.18.1" 2021-12-15T19:16:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:30 smithi022 ceph-mon[31360]: pgmap v2341: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:30 smithi022 ceph-mon[40250]: pgmap v2341: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:30 smithi122 ceph-mon[32255]: pgmap v2341: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:32 smithi122 ceph-mon[32255]: pgmap v2342: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:16:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:16:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:32 smithi022 ceph-mon[31360]: pgmap v2342: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:32 smithi022 ceph-mon[40250]: pgmap v2342: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:16:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:16:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:16:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:16:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:16:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:16:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:16:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:16:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:16:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:16:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:16:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:16:34.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:34.366 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:34 smithi022 ceph-mon[40250]: pgmap v2343: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:34.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:34.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:34.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:34 smithi122 ceph-mon[32255]: pgmap v2343: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:34 smithi022 ceph-mon[31360]: pgmap v2343: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/411105298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/411105298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/411105298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:36 smithi122 ceph-mon[32255]: pgmap v2344: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:36 smithi022 ceph-mon[31360]: pgmap v2344: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:36 smithi022 ceph-mon[40250]: pgmap v2344: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:16:37.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:16:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:16:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:38 smithi022 ceph-mon[31360]: pgmap v2345: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:38 smithi022 ceph-mon[40250]: pgmap v2345: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:38 smithi122 ceph-mon[32255]: pgmap v2345: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:39.444 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:39.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:39.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:39.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:40] "GET /metrics HTTP/1.1" 200 202297 "" "Prometheus/2.18.1" 2021-12-15T19:16:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:40 smithi022 ceph-mon[31360]: pgmap v2346: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2004176379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:40 smithi022 ceph-mon[40250]: pgmap v2346: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2004176379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:40 smithi122 ceph-mon[32255]: pgmap v2346: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2004176379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:42 smithi122 ceph-mon[32255]: pgmap v2347: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:16:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:16:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:42 smithi022 ceph-mon[31360]: pgmap v2347: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:42 smithi022 ceph-mon[40250]: pgmap v2347: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:44 smithi022 ceph-mon[31360]: pgmap v2348: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:44 smithi022 ceph-mon[40250]: pgmap v2348: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:44.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:44 smithi122 ceph-mon[32255]: pgmap v2348: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/525675991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/525675991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/525675991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:46 smithi022 ceph-mon[31360]: pgmap v2349: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:46 smithi022 ceph-mon[40250]: pgmap v2349: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:46 smithi122 ceph-mon[32255]: pgmap v2349: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:16:47.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:16:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:16:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:48 smithi022 ceph-mon[31360]: pgmap v2350: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:48 smithi022 ceph-mon[40250]: pgmap v2350: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:48 smithi122 ceph-mon[32255]: pgmap v2350: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:50.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:50] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:16:50.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:50.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:50.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:50 smithi022 ceph-mon[31360]: pgmap v2351: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:50 smithi022 ceph-mon[40250]: pgmap v2351: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:50 smithi122 ceph-mon[32255]: pgmap v2351: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3681914155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3681914155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3681914155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:52 smithi122 ceph-mon[32255]: pgmap v2352: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:16:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:16:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:52 smithi022 ceph-mon[31360]: pgmap v2352: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:52 smithi022 ceph-mon[40250]: pgmap v2352: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:54 smithi022 ceph-mon[31360]: pgmap v2353: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:54 smithi022 ceph-mon[40250]: pgmap v2353: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:54 smithi122 ceph-mon[32255]: pgmap v2353: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:16:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:16:55.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:16:55.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:16:55.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:16:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:56 smithi122 ceph-mon[32255]: pgmap v2354: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1005240406' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:56 smithi022 ceph-mon[31360]: pgmap v2354: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1005240406' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:56 smithi022 ceph-mon[40250]: pgmap v2354: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1005240406' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:16:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:16:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:16:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:16:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:58 smithi122 ceph-mon[32255]: pgmap v2355: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:16:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:58 smithi022 ceph-mon[31360]: pgmap v2355: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:16:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:16:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:58 smithi022 ceph-mon[40250]: pgmap v2355: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:16:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:16:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:00] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:17:00.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:00.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:00 smithi022 ceph-mon[31360]: pgmap v2356: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:00.930 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:00 smithi022 ceph-mon[40250]: pgmap v2356: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:00 smithi122 ceph-mon[32255]: pgmap v2356: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:01.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:01.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:01.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3967697835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3967697835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3967697835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:02 smithi122 ceph-mon[32255]: pgmap v2357: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:17:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:17:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:02 smithi022 ceph-mon[31360]: pgmap v2357: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:02 smithi022 ceph-mon[40250]: pgmap v2357: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:17:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:17:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:17:03.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:17:03.756 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:17:03.756 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:17:03.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:17:03.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:17:03.757 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:17:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:04 smithi022 ceph-mon[31360]: pgmap v2358: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:17:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:04 smithi022 ceph-mon[40250]: pgmap v2358: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:17:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:04 smithi122 ceph-mon[32255]: pgmap v2358: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:17:06.302 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:06.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:06.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:06.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:06 smithi022 ceph-mon[31360]: pgmap v2359: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4290364235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:06 smithi022 ceph-mon[40250]: pgmap v2359: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4290364235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:06 smithi122 ceph-mon[32255]: pgmap v2359: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4290364235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:17:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:17:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:17:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:08 smithi022 ceph-mon[31360]: pgmap v2360: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:08 smithi022 ceph-mon[40250]: pgmap v2360: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:09.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:08 smithi122 ceph-mon[32255]: pgmap v2360: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:10] "GET /metrics HTTP/1.1" 200 202302 "" "Prometheus/2.18.1" 2021-12-15T19:17:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:10 smithi022 ceph-mon[31360]: pgmap v2361: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:10 smithi022 ceph-mon[40250]: pgmap v2361: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:10 smithi122 ceph-mon[32255]: pgmap v2361: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:11.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:12.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:12.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:12.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:12 smithi122 ceph-mon[32255]: pgmap v2362: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2861241950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:17:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:17:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:12 smithi022 ceph-mon[31360]: pgmap v2362: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2861241950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:12 smithi022 ceph-mon[40250]: pgmap v2362: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2861241950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:14 smithi022 ceph-mon[31360]: pgmap v2363: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:14 smithi022 ceph-mon[40250]: pgmap v2363: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:14 smithi122 ceph-mon[32255]: pgmap v2363: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:16 smithi022 ceph-mon[31360]: pgmap v2364: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:16 smithi022 ceph-mon[40250]: pgmap v2364: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:16 smithi122 ceph-mon[32255]: pgmap v2364: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:17.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:17.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:17.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:17.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/972900430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:17:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/972900430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:17.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:17:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:17:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/972900430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:18 smithi022 ceph-mon[31360]: pgmap v2365: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:18 smithi022 ceph-mon[40250]: pgmap v2365: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:18 smithi122 ceph-mon[32255]: pgmap v2365: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:20] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:17:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:20 smithi122 ceph-mon[32255]: pgmap v2366: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:20 smithi022 ceph-mon[31360]: pgmap v2366: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:20 smithi022 ceph-mon[40250]: pgmap v2366: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:22.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:22 smithi122 ceph-mon[32255]: pgmap v2367: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:17:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:17:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:22 smithi022 ceph-mon[31360]: pgmap v2367: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:22 smithi022 ceph-mon[40250]: pgmap v2367: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:22.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:22.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:22.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4080079670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4080079670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4080079670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:24 smithi122 ceph-mon[32255]: pgmap v2368: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:24 smithi022 ceph-mon[31360]: pgmap v2368: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:24 smithi022 ceph-mon[40250]: pgmap v2368: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:26 smithi022 ceph-mon[31360]: pgmap v2369: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:26 smithi022 ceph-mon[40250]: pgmap v2369: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:26 smithi122 ceph-mon[32255]: pgmap v2369: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:17:27.783 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:27.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:17:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:17:28.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:28.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:28.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:28 smithi022 ceph-mon[40250]: pgmap v2370: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1117357516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:28 smithi022 ceph-mon[31360]: pgmap v2370: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1117357516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:28 smithi122 ceph-mon[32255]: pgmap v2370: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1117357516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:30] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:17:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:30 smithi122 ceph-mon[32255]: pgmap v2371: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:30 smithi022 ceph-mon[31360]: pgmap v2371: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:30 smithi022 ceph-mon[40250]: pgmap v2371: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:32 smithi122 ceph-mon[32255]: pgmap v2372: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:17:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:17:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:32 smithi022 ceph-mon[31360]: pgmap v2372: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:32 smithi022 ceph-mon[40250]: pgmap v2372: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:33.154 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:33.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:33.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:33.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:33.524 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:17:33.525 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:17:33.525 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:17:33.525 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:17:33.526 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:17:33.526 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:17:33.526 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:17:33.526 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:17:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:17:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:17:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:17:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:34 smithi122 ceph-mon[32255]: pgmap v2373: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/789486404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:34 smithi022 ceph-mon[31360]: pgmap v2373: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/789486404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:34 smithi022 ceph-mon[40250]: pgmap v2373: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/789486404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:36 smithi022 ceph-mon[31360]: pgmap v2374: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:36 smithi022 ceph-mon[40250]: pgmap v2374: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:36 smithi122 ceph-mon[32255]: pgmap v2374: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:17:37.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:17:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:17:38.525 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:38 smithi022 ceph-mon[31360]: pgmap v2375: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:38 smithi022 ceph-mon[40250]: pgmap v2375: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:38.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:38.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:38.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:38 smithi122 ceph-mon[32255]: pgmap v2375: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:17:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3550212098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3550212098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3550212098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:40] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:17:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:40 smithi122 ceph-mon[32255]: pgmap v2376: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:40 smithi022 ceph-mon[31360]: pgmap v2376: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:40 smithi022 ceph-mon[40250]: pgmap v2376: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:42 smithi122 ceph-mon[32255]: pgmap v2377: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:17:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:17:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:42 smithi022 ceph-mon[31360]: pgmap v2377: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:42 smithi022 ceph-mon[40250]: pgmap v2377: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:43.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:44.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:44.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:44.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:44 smithi122 ceph-mon[32255]: pgmap v2378: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/860396720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:44 smithi022 ceph-mon[31360]: pgmap v2378: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/860396720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:44 smithi022 ceph-mon[40250]: pgmap v2378: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/860396720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:46 smithi122 ceph-mon[32255]: pgmap v2379: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:46 smithi022 ceph-mon[31360]: pgmap v2379: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:46 smithi022 ceph-mon[40250]: pgmap v2379: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:17:47.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:17:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:17:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:48 smithi122 ceph-mon[32255]: pgmap v2380: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:48 smithi022 ceph-mon[31360]: pgmap v2380: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:48 smithi022 ceph-mon[40250]: pgmap v2380: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:49.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:49.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:49.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:49.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:50] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:17:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:50 smithi122 ceph-mon[32255]: pgmap v2381: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4027345428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:50 smithi022 ceph-mon[31360]: pgmap v2381: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4027345428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:50 smithi022 ceph-mon[40250]: pgmap v2381: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:17:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4027345428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:52 smithi122 ceph-mon[32255]: pgmap v2382: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:17:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:17:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:52 smithi022 ceph-mon[31360]: pgmap v2382: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:52 smithi022 ceph-mon[40250]: pgmap v2382: 132 pgs: 132 active+clean; 582 KiB data, 110 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:17:54.641 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:17:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:54 smithi122 ceph-mon[32255]: pgmap v2383: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T19:17:55.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:17:55.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:17:55.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:17:55.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:54 smithi022 ceph-mon[31360]: pgmap v2383: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T19:17:55.013 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:54 smithi022 ceph-mon[40250]: pgmap v2383: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T19:17:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/803511692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/803511692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/803511692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:17:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:56 smithi122 ceph-mon[32255]: pgmap v2384: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T19:17:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:56 smithi022 ceph-mon[31360]: pgmap v2384: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T19:17:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:56 smithi022 ceph-mon[40250]: pgmap v2384: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T19:17:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:17:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:17:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:17:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:58 smithi122 ceph-mon[32255]: pgmap v2385: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T19:17:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:17:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:58 smithi022 ceph-mon[31360]: pgmap v2385: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T19:17:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:17:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:17:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:58 smithi022 ceph-mon[40250]: pgmap v2385: 132 pgs: 132 active+clean; 582 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T19:17:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:17:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:00.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:00.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:00.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:00.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:00] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:18:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:00 smithi022 ceph-mon[31360]: pgmap v2386: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T19:18:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1061579423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:00 smithi022 ceph-mon[40250]: pgmap v2386: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T19:18:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1061579423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:00 smithi122 ceph-mon[32255]: pgmap v2386: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T19:18:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1061579423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:02 smithi122 ceph-mon[32255]: pgmap v2387: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T19:18:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:18:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:18:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:02 smithi022 ceph-mon[31360]: pgmap v2387: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T19:18:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:02 smithi022 ceph-mon[40250]: pgmap v2387: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T19:18:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:04 smithi122 ceph-mon[32255]: pgmap v2388: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T19:18:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:18:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:18:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:18:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:18:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[31360]: pgmap v2388: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T19:18:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:18:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:18:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:18:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:18:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[40250]: pgmap v2388: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T19:18:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:18:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:18:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:18:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:18:05.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:05.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:05.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:05.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:06.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:06 smithi122 ceph-mon[32255]: pgmap v2389: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:18:06.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1284618574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:06 smithi022 ceph-mon[31360]: pgmap v2389: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:18:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1284618574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:06 smithi022 ceph-mon[40250]: pgmap v2389: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:18:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1284618574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:18:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:18:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:18:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:08 smithi122 ceph-mon[32255]: pgmap v2390: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:18:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:08 smithi022 ceph-mon[31360]: pgmap v2390: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:18:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:08 smithi022 ceph-mon[40250]: pgmap v2390: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:18:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:10] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:18:10.759 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:10 smithi122 ceph-mon[32255]: pgmap v2391: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:18:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:10 smithi022 ceph-mon[31360]: pgmap v2391: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:18:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:10 smithi022 ceph-mon[40250]: pgmap v2391: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:18:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:11.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3618614535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:12.128 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3618614535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:12.128 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3618614535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:12 smithi122 ceph-mon[32255]: pgmap v2392: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:18:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:18:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:12 smithi022 ceph-mon[31360]: pgmap v2392: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:12 smithi022 ceph-mon[40250]: pgmap v2392: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:14 smithi122 ceph-mon[32255]: pgmap v2393: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:14 smithi022 ceph-mon[31360]: pgmap v2393: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:14 smithi022 ceph-mon[40250]: pgmap v2393: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:16.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:16.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:16.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:16 smithi022 ceph-mon[31360]: pgmap v2394: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3244096836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:16 smithi022 ceph-mon[40250]: pgmap v2394: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3244096836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:16 smithi122 ceph-mon[32255]: pgmap v2394: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3244096836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:18:17.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:18:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:18:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:18 smithi122 ceph-mon[32255]: pgmap v2395: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:18 smithi022 ceph-mon[31360]: pgmap v2395: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:18 smithi022 ceph-mon[40250]: pgmap v2395: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:20] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:18:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:20 smithi122 ceph-mon[32255]: pgmap v2396: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:20 smithi022 ceph-mon[31360]: pgmap v2396: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:20 smithi022 ceph-mon[40250]: pgmap v2396: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:21.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:21.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:21.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:21.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:22 smithi122 ceph-mon[32255]: pgmap v2397: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3825646737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:18:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:18:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:22 smithi022 ceph-mon[31360]: pgmap v2397: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3825646737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:22 smithi022 ceph-mon[40250]: pgmap v2397: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3825646737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:24 smithi122 ceph-mon[32255]: pgmap v2398: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:24 smithi022 ceph-mon[31360]: pgmap v2398: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:24 smithi022 ceph-mon[40250]: pgmap v2398: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:26.872 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:26.873 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:26 smithi022 ceph-mon[40250]: pgmap v2399: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:26.873 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:26 smithi022 ceph-mon[31360]: pgmap v2399: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:26 smithi122 ceph-mon[32255]: pgmap v2399: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:27.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:27.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:27.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:18:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3409102558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:27.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:18:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:18:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3409102558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3409102558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:28 smithi122 ceph-mon[32255]: pgmap v2400: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:28 smithi022 ceph-mon[31360]: pgmap v2400: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:28 smithi022 ceph-mon[40250]: pgmap v2400: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:30] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:18:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:30 smithi122 ceph-mon[32255]: pgmap v2401: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:30 smithi022 ceph-mon[31360]: pgmap v2401: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:30 smithi022 ceph-mon[40250]: pgmap v2401: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:32.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:32 smithi022 ceph-mon[31360]: pgmap v2402: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:32 smithi022 ceph-mon[40250]: pgmap v2402: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:32.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:32.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:32.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:32 smithi122 ceph-mon[32255]: pgmap v2402: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:18:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:18:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:18:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:18:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:18:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:18:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3188527423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:18:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:18:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:18:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:18:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3188527423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:18:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:18:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:18:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:18:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3188527423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:34 smithi122 ceph-mon[32255]: pgmap v2403: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:34 smithi022 ceph-mon[31360]: pgmap v2403: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:34 smithi022 ceph-mon[40250]: pgmap v2403: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:36 smithi122 ceph-mon[32255]: pgmap v2404: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:36 smithi022 ceph-mon[31360]: pgmap v2404: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:36 smithi022 ceph-mon[40250]: pgmap v2404: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:37.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:18:37.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:18:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:18:37.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:37.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:37.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:38 smithi122 ceph-mon[32255]: pgmap v2405: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187238583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:38 smithi022 ceph-mon[31360]: pgmap v2405: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187238583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:38 smithi022 ceph-mon[40250]: pgmap v2405: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187238583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:40] "GET /metrics HTTP/1.1" 200 202295 "" "Prometheus/2.18.1" 2021-12-15T19:18:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:40 smithi122 ceph-mon[32255]: pgmap v2406: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:40 smithi022 ceph-mon[31360]: pgmap v2406: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:40 smithi022 ceph-mon[40250]: pgmap v2406: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:42 smithi122 ceph-mon[32255]: pgmap v2407: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:18:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:18:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:42 smithi022 ceph-mon[31360]: pgmap v2407: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:42 smithi022 ceph-mon[40250]: pgmap v2407: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:42.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:43.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:43.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:43.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/786004773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/786004773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/786004773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:44 smithi122 ceph-mon[32255]: pgmap v2408: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:44 smithi022 ceph-mon[31360]: pgmap v2408: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:44 smithi022 ceph-mon[40250]: pgmap v2408: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:46 smithi122 ceph-mon[32255]: pgmap v2409: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:46 smithi022 ceph-mon[31360]: pgmap v2409: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:46 smithi022 ceph-mon[40250]: pgmap v2409: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:18:47.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:18:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:18:48.363 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:48.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:48.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:48.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:48.731 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:48 smithi022 ceph-mon[31360]: pgmap v2410: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:48.731 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:48.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:48 smithi022 ceph-mon[40250]: pgmap v2410: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:48.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:48 smithi122 ceph-mon[32255]: pgmap v2410: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/970125083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/970125083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/970125083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:50] "GET /metrics HTTP/1.1" 200 202297 "" "Prometheus/2.18.1" 2021-12-15T19:18:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:50 smithi122 ceph-mon[32255]: pgmap v2411: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:50 smithi022 ceph-mon[31360]: pgmap v2411: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:50 smithi022 ceph-mon[40250]: pgmap v2411: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:52 smithi122 ceph-mon[32255]: pgmap v2412: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:18:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:18:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:52 smithi022 ceph-mon[31360]: pgmap v2412: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:52 smithi022 ceph-mon[40250]: pgmap v2412: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:53.731 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:54.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:54 smithi122 ceph-mon[32255]: pgmap v2413: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3109180384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:54 smithi022 ceph-mon[31360]: pgmap v2413: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3109180384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:54 smithi022 ceph-mon[40250]: pgmap v2413: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:18:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3109180384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:56 smithi122 ceph-mon[32255]: pgmap v2414: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:56 smithi022 ceph-mon[31360]: pgmap v2414: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:56 smithi022 ceph-mon[40250]: pgmap v2414: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:18:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:18:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:18:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:58 smithi122 ceph-mon[32255]: pgmap v2415: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:58 smithi022 ceph-mon[31360]: pgmap v2415: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:58 smithi022 ceph-mon[40250]: pgmap v2415: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:18:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:18:59.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:18:59.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:18:59.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:18:59.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:18:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:18:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3104406553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:18:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3104406553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:18:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:18:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3104406553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:00] "GET /metrics HTTP/1.1" 200 202297 "" "Prometheus/2.18.1" 2021-12-15T19:19:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:00 smithi122 ceph-mon[32255]: pgmap v2416: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:00 smithi022 ceph-mon[31360]: pgmap v2416: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:00 smithi022 ceph-mon[40250]: pgmap v2416: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:02 smithi122 ceph-mon[32255]: pgmap v2417: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:19:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:19:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:02 smithi022 ceph-mon[31360]: pgmap v2417: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:02 smithi022 ceph-mon[40250]: pgmap v2417: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:04.469 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:04.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:04.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:04.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:04.829 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[40250]: pgmap v2418: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:04.830 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:19:04.830 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:19:04.830 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:19:04.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[31360]: pgmap v2418: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:04.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:19:04.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:19:04.831 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:19:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:04 smithi122 ceph-mon[32255]: pgmap v2418: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:19:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:19:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:19:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:19:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3357409989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:19:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3357409989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:19:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3357409989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:06 smithi022 ceph-mon[31360]: pgmap v2419: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:06 smithi022 ceph-mon[40250]: pgmap v2419: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:06 smithi122 ceph-mon[32255]: pgmap v2419: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:19:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:19:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:19:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:08 smithi022 ceph-mon[31360]: pgmap v2420: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:08 smithi022 ceph-mon[40250]: pgmap v2420: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:08 smithi122 ceph-mon[32255]: pgmap v2420: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:09.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:10] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:19:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:10 smithi022 ceph-mon[31360]: pgmap v2421: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4050923390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:10 smithi022 ceph-mon[40250]: pgmap v2421: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4050923390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:10 smithi122 ceph-mon[32255]: pgmap v2421: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4050923390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:12 smithi122 ceph-mon[32255]: pgmap v2422: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:12.870 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:19:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:19:12.871 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:12 smithi022 ceph-mon[31360]: pgmap v2422: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:12.872 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:12 smithi022 ceph-mon[40250]: pgmap v2422: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:14 smithi122 ceph-mon[32255]: pgmap v2423: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:14 smithi022 ceph-mon[31360]: pgmap v2423: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:14 smithi022 ceph-mon[40250]: pgmap v2423: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:15.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:15.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:15.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:15.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/614172095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/614172095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/614172095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:16 smithi122 ceph-mon[32255]: pgmap v2424: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:16 smithi022 ceph-mon[31360]: pgmap v2424: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:16 smithi022 ceph-mon[40250]: pgmap v2424: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:19:17.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:19:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:19:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:18 smithi122 ceph-mon[32255]: pgmap v2425: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:18 smithi022 ceph-mon[31360]: pgmap v2425: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:18 smithi022 ceph-mon[40250]: pgmap v2425: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:20] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:19:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:20 smithi122 ceph-mon[32255]: pgmap v2426: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:20.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:20.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:20.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:20.944 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:20 smithi022 ceph-mon[40250]: pgmap v2426: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:20.944 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:20 smithi022 ceph-mon[31360]: pgmap v2426: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1751227529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1751227529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1751227529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:22.571 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:19:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:19:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:22 smithi122 ceph-mon[32255]: pgmap v2427: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:22 smithi022 ceph-mon[31360]: pgmap v2427: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:22 smithi022 ceph-mon[40250]: pgmap v2427: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:24 smithi122 ceph-mon[32255]: pgmap v2428: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:24 smithi022 ceph-mon[31360]: pgmap v2428: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:24 smithi022 ceph-mon[40250]: pgmap v2428: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:25.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:26.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:26.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:26.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:26 smithi122 ceph-mon[32255]: pgmap v2429: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1874973121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:26 smithi022 ceph-mon[31360]: pgmap v2429: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1874973121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:26 smithi022 ceph-mon[40250]: pgmap v2429: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1874973121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:19:27.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:19:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:19:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:28 smithi122 ceph-mon[32255]: pgmap v2430: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:28 smithi022 ceph-mon[31360]: pgmap v2430: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:28 smithi022 ceph-mon[40250]: pgmap v2430: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:30] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:19:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:30 smithi122 ceph-mon[32255]: pgmap v2431: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:31.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:30 smithi022 ceph-mon[31360]: pgmap v2431: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:31.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:30 smithi022 ceph-mon[40250]: pgmap v2431: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:31.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:31.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:31.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:31.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:32 smithi122 ceph-mon[32255]: pgmap v2432: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3916695731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:19:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:19:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:32 smithi022 ceph-mon[31360]: pgmap v2432: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3916695731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:32 smithi022 ceph-mon[40250]: pgmap v2432: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3916695731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:19:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:19:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:19:33.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:19:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:19:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:19:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:19:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:19:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:19:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:19:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:19:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:34 smithi122 ceph-mon[32255]: pgmap v2433: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:34 smithi022 ceph-mon[31360]: pgmap v2433: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:34 smithi022 ceph-mon[40250]: pgmap v2433: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:36.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:36 smithi122 ceph-mon[32255]: pgmap v2434: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:36 smithi022 ceph-mon[31360]: pgmap v2434: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:36 smithi022 ceph-mon[40250]: pgmap v2434: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:37.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:37.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:37.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1714611077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:19:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1714611077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1714611077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:37.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:19:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:19:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:38 smithi122 ceph-mon[32255]: pgmap v2435: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:38 smithi022 ceph-mon[31360]: pgmap v2435: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:38 smithi022 ceph-mon[40250]: pgmap v2435: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:40] "GET /metrics HTTP/1.1" 200 202297 "" "Prometheus/2.18.1" 2021-12-15T19:19:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:40 smithi122 ceph-mon[32255]: pgmap v2436: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:40 smithi022 ceph-mon[31360]: pgmap v2436: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:40 smithi022 ceph-mon[40250]: pgmap v2436: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:42.403 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:42 smithi022 ceph-mon[40250]: pgmap v2437: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:42.771 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:19:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:19:42.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:42.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:42.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:42.772 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:42 smithi122 ceph-mon[32255]: pgmap v2437: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:42 smithi022 ceph-mon[31360]: pgmap v2437: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/173194034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/173194034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/173194034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:44 smithi122 ceph-mon[32255]: pgmap v2438: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:44 smithi022 ceph-mon[31360]: pgmap v2438: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:44 smithi022 ceph-mon[40250]: pgmap v2438: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:46 smithi122 ceph-mon[32255]: pgmap v2439: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:46 smithi022 ceph-mon[31360]: pgmap v2439: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:46 smithi022 ceph-mon[40250]: pgmap v2439: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:47.426 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:19:47.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:47.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:47.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:47.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:19:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:19:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:48 smithi122 ceph-mon[32255]: pgmap v2440: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3796909046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:48 smithi022 ceph-mon[31360]: pgmap v2440: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3796909046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:48 smithi022 ceph-mon[40250]: pgmap v2440: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3796909046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:50] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:19:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:50 smithi122 ceph-mon[32255]: pgmap v2441: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:50 smithi022 ceph-mon[31360]: pgmap v2441: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:50 smithi022 ceph-mon[40250]: pgmap v2441: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:52 smithi122 ceph-mon[32255]: pgmap v2442: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:19:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:19:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:52 smithi022 ceph-mon[31360]: pgmap v2442: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:52 smithi022 ceph-mon[40250]: pgmap v2442: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:52.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:53.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:53.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:53.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/925832736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/925832736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/925832736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:54 smithi122 ceph-mon[32255]: pgmap v2443: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:54 smithi022 ceph-mon[31360]: pgmap v2443: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:54 smithi022 ceph-mon[40250]: pgmap v2443: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:19:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:56 smithi122 ceph-mon[32255]: pgmap v2444: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:56 smithi022 ceph-mon[31360]: pgmap v2444: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:56 smithi022 ceph-mon[40250]: pgmap v2444: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:19:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:19:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:19:58.168 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:19:58.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:19:58.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:19:58.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:19:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:58 smithi122 ceph-mon[32255]: pgmap v2445: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:19:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806276775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:58 smithi022 ceph-mon[31360]: pgmap v2445: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:19:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806276775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:19:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:58 smithi022 ceph-mon[40250]: pgmap v2445: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:19:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:19:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:19:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806276775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:00] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:20:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:00 smithi122 ceph-mon[32255]: pgmap v2446: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T19:20:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:00 smithi022 ceph-mon[31360]: pgmap v2446: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T19:20:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:00 smithi022 ceph-mon[40250]: pgmap v2446: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T19:20:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:02 smithi122 ceph-mon[32255]: pgmap v2447: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:20:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:20:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:02 smithi022 ceph-mon[31360]: pgmap v2447: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:02 smithi022 ceph-mon[40250]: pgmap v2447: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:03.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:03.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:03.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:03.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:04 smithi122 ceph-mon[32255]: pgmap v2448: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1613611875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:04 smithi022 ceph-mon[31360]: pgmap v2448: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1613611875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:04 smithi022 ceph-mon[40250]: pgmap v2448: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1613611875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:20:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:20:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:20:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:20:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:20:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:20:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:20:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:20:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:20:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:06 smithi122 ceph-mon[32255]: pgmap v2449: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:06 smithi022 ceph-mon[31360]: pgmap v2449: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:06 smithi022 ceph-mon[40250]: pgmap v2449: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:20:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:20:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:20:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:08 smithi022 ceph-mon[31360]: pgmap v2450: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:08 smithi022 ceph-mon[40250]: pgmap v2450: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:08.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:08 smithi122 ceph-mon[32255]: pgmap v2450: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:09.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:09.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:09.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:10] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:20:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2818367630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2818367630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2818367630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:20:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:11 smithi022 ceph-mon[31360]: pgmap v2451: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:11 smithi022 ceph-mon[40250]: pgmap v2451: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:11 smithi122 ceph-mon[32255]: pgmap v2451: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:12 smithi122 ceph-mon[32255]: pgmap v2452: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:20:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:20:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:12 smithi022 ceph-mon[31360]: pgmap v2452: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:12 smithi022 ceph-mon[40250]: pgmap v2452: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:14.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:14.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:14.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:14 smithi122 ceph-mon[32255]: pgmap v2453: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:14 smithi022 ceph-mon[31360]: pgmap v2453: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:14 smithi022 ceph-mon[40250]: pgmap v2453: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1673716623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1673716623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1673716623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:16 smithi122 ceph-mon[32255]: pgmap v2454: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:16 smithi022 ceph-mon[31360]: pgmap v2454: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:16 smithi022 ceph-mon[40250]: pgmap v2454: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:20:17.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:20:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:20:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:18 smithi122 ceph-mon[32255]: pgmap v2455: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:18 smithi022 ceph-mon[31360]: pgmap v2455: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:18 smithi022 ceph-mon[40250]: pgmap v2455: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:19.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:20.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:20.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:20.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:20] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:20:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:20 smithi122 ceph-mon[32255]: pgmap v2456: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3488337280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:20 smithi022 ceph-mon[31360]: pgmap v2456: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3488337280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:20 smithi022 ceph-mon[40250]: pgmap v2456: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3488337280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:22 smithi122 ceph-mon[32255]: pgmap v2457: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:20:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:20:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:22 smithi022 ceph-mon[31360]: pgmap v2457: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:22 smithi022 ceph-mon[40250]: pgmap v2457: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:24 smithi122 ceph-mon[32255]: pgmap v2458: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:25.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:25.011 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:24 smithi022 ceph-mon[31360]: pgmap v2458: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:25.011 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:24 smithi022 ceph-mon[40250]: pgmap v2458: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:25.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:25.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:25.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/271001737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/271001737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/271001737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:26 smithi122 ceph-mon[32255]: pgmap v2459: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:26 smithi022 ceph-mon[31360]: pgmap v2459: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:26 smithi022 ceph-mon[40250]: pgmap v2459: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:20:27.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:20:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:20:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:28 smithi122 ceph-mon[32255]: pgmap v2460: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:28 smithi022 ceph-mon[31360]: pgmap v2460: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:28 smithi022 ceph-mon[40250]: pgmap v2460: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:29.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:30.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:30] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:20:30.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:30.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:30.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:30.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:30 smithi022 ceph-mon[31360]: pgmap v2461: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:30.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:30 smithi022 ceph-mon[40250]: pgmap v2461: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:30 smithi122 ceph-mon[32255]: pgmap v2461: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/966722384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/966722384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/966722384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:32.592 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:20:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:20:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:32 smithi122 ceph-mon[32255]: pgmap v2462: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:20:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:20:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[31360]: pgmap v2462: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:20:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:20:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:20:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:20:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[40250]: pgmap v2462: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:20:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:20:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:20:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:20:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:34 smithi122 ceph-mon[32255]: pgmap v2463: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:34 smithi022 ceph-mon[31360]: pgmap v2463: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:34 smithi022 ceph-mon[40250]: pgmap v2463: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:35.756 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:36 smithi122 ceph-mon[32255]: pgmap v2464: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2198829429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:36 smithi022 ceph-mon[31360]: pgmap v2464: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2198829429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:36 smithi022 ceph-mon[40250]: pgmap v2464: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2198829429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:20:37.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:20:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:20:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:38 smithi122 ceph-mon[32255]: pgmap v2465: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:38 smithi022 ceph-mon[31360]: pgmap v2465: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:38 smithi022 ceph-mon[40250]: pgmap v2465: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:40] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:20:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:40 smithi122 ceph-mon[32255]: pgmap v2466: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:40 smithi022 ceph-mon[31360]: pgmap v2466: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:40 smithi022 ceph-mon[40250]: pgmap v2466: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:41.125 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:41.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:41.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:41.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1507937044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:42 smithi122 ceph-mon[32255]: pgmap v2467: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:20:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:20:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1507937044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:42 smithi022 ceph-mon[31360]: pgmap v2467: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1507937044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:42 smithi022 ceph-mon[40250]: pgmap v2467: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:44 smithi122 ceph-mon[32255]: pgmap v2468: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:44 smithi022 ceph-mon[31360]: pgmap v2468: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:44 smithi022 ceph-mon[40250]: pgmap v2468: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:46.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:46 smithi022 ceph-mon[31360]: pgmap v2469: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:46 smithi022 ceph-mon[40250]: pgmap v2469: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:46.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:46.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:46.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:46 smithi122 ceph-mon[32255]: pgmap v2469: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1891389605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:20:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1891389605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1891389605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:47.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:20:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:20:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:48 smithi122 ceph-mon[32255]: pgmap v2470: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:48 smithi022 ceph-mon[40250]: pgmap v2470: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:48 smithi022 ceph-mon[31360]: pgmap v2470: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:50] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:20:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:50 smithi122 ceph-mon[32255]: pgmap v2471: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:50 smithi022 ceph-mon[31360]: pgmap v2471: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:50 smithi022 ceph-mon[40250]: pgmap v2471: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:51.866 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:52.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:52.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:52.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:52 smithi022 ceph-mon[31360]: pgmap v2472: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1670631830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:52 smithi022 ceph-mon[40250]: pgmap v2472: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1670631830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:52 smithi122 ceph-mon[32255]: pgmap v2472: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1670631830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:20:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:20:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:54 smithi122 ceph-mon[32255]: pgmap v2473: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:54 smithi022 ceph-mon[31360]: pgmap v2473: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:54 smithi022 ceph-mon[40250]: pgmap v2473: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:20:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:56 smithi122 ceph-mon[32255]: pgmap v2474: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:56 smithi022 ceph-mon[31360]: pgmap v2474: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:56 smithi022 ceph-mon[40250]: pgmap v2474: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:57.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:20:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:20:57.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:20:57.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:20:57.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:20:58.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:20:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:20:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:58 smithi122 ceph-mon[32255]: pgmap v2475: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1842905994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:20:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:58 smithi022 ceph-mon[31360]: pgmap v2475: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1842905994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:20:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:20:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:58 smithi022 ceph-mon[40250]: pgmap v2475: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:20:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1842905994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:20:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:20:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:00] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:21:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:00 smithi122 ceph-mon[32255]: pgmap v2476: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:00 smithi022 ceph-mon[31360]: pgmap v2476: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:00 smithi022 ceph-mon[40250]: pgmap v2476: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:02 smithi122 ceph-mon[32255]: pgmap v2477: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:21:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:21:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:02 smithi022 ceph-mon[31360]: pgmap v2477: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:02 smithi022 ceph-mon[40250]: pgmap v2477: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:02.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:02.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:02.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1149846894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1149846894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1149846894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:04 smithi122 ceph-mon[32255]: pgmap v2478: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:04 smithi022 ceph-mon[31360]: pgmap v2478: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:04 smithi022 ceph-mon[40250]: pgmap v2478: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:06 smithi122 ceph-mon[32255]: pgmap v2479: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:06 smithi022 ceph-mon[31360]: pgmap v2479: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:06 smithi022 ceph-mon[40250]: pgmap v2479: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:21:07.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:21:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:21:08.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:08.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:08.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:08 smithi022 ceph-mon[31360]: pgmap v2480: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/878080054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:08 smithi022 ceph-mon[40250]: pgmap v2480: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/878080054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:08 smithi122 ceph-mon[32255]: pgmap v2480: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/878080054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:10] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:21:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:10 smithi122 ceph-mon[32255]: pgmap v2481: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:21:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:21:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:21:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:21:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[31360]: pgmap v2481: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:21:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:21:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:21:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:21:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[40250]: pgmap v2481: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:21:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:21:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:21:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:21:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:12 smithi122 ceph-mon[32255]: pgmap v2482: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:21:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:21:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:12 smithi022 ceph-mon[31360]: pgmap v2482: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:12 smithi022 ceph-mon[40250]: pgmap v2482: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:13.349 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:13.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:13.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:13.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:14 smithi122 ceph-mon[32255]: pgmap v2483: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1620385433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:14 smithi022 ceph-mon[31360]: pgmap v2483: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1620385433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:14 smithi022 ceph-mon[40250]: pgmap v2483: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1620385433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:16 smithi122 ceph-mon[32255]: pgmap v2484: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:16 smithi022 ceph-mon[31360]: pgmap v2484: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:16 smithi022 ceph-mon[40250]: pgmap v2484: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:21:18.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:21:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:21:18.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:18 smithi122 ceph-mon[32255]: pgmap v2485: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:18 smithi022 ceph-mon[31360]: pgmap v2485: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:18 smithi022 ceph-mon[40250]: pgmap v2485: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:19.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:19.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:19.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4120382371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4120382371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4120382371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:20] "GET /metrics HTTP/1.1" 200 202292 "" "Prometheus/2.18.1" 2021-12-15T19:21:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:20 smithi122 ceph-mon[32255]: pgmap v2486: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:20 smithi022 ceph-mon[31360]: pgmap v2486: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:20 smithi022 ceph-mon[40250]: pgmap v2486: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:22 smithi122 ceph-mon[32255]: pgmap v2487: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:21:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:21:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:22 smithi022 ceph-mon[31360]: pgmap v2487: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:22 smithi022 ceph-mon[40250]: pgmap v2487: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:24.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:24.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:24.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:24.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:24 smithi022 ceph-mon[31360]: pgmap v2488: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/946445855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:24 smithi022 ceph-mon[40250]: pgmap v2488: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/946445855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:24 smithi122 ceph-mon[32255]: pgmap v2488: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/946445855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:26 smithi122 ceph-mon[32255]: pgmap v2489: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:26 smithi022 ceph-mon[40250]: pgmap v2489: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:26 smithi022 ceph-mon[31360]: pgmap v2489: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:21:28.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:21:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:21:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:28 smithi122 ceph-mon[32255]: pgmap v2490: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:28 smithi022 ceph-mon[40250]: pgmap v2490: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:28 smithi022 ceph-mon[31360]: pgmap v2490: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:29.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:29.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:29.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:29.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:30] "GET /metrics HTTP/1.1" 200 202292 "" "Prometheus/2.18.1" 2021-12-15T19:21:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:30 smithi122 ceph-mon[32255]: pgmap v2491: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3643489069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:30 smithi022 ceph-mon[40250]: pgmap v2491: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3643489069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:30 smithi022 ceph-mon[31360]: pgmap v2491: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3643489069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:32 smithi122 ceph-mon[32255]: pgmap v2492: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:21:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:21:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:32 smithi022 ceph-mon[31360]: pgmap v2492: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:32 smithi022 ceph-mon[40250]: pgmap v2492: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:21:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:21:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:21:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:21:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:21:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:21:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:21:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:21:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:21:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:21:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:21:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:21:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:34 smithi122 ceph-mon[32255]: pgmap v2493: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:34 smithi022 ceph-mon[31360]: pgmap v2493: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:34 smithi022 ceph-mon[40250]: pgmap v2493: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:34.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:35.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:35.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:35.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1483102851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1483102851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1483102851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:36 smithi022 ceph-mon[31360]: pgmap v2494: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:36 smithi022 ceph-mon[40250]: pgmap v2494: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:36 smithi122 ceph-mon[32255]: pgmap v2494: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:21:38.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:21:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:21:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:38 smithi122 ceph-mon[32255]: pgmap v2495: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:38 smithi022 ceph-mon[31360]: pgmap v2495: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:38 smithi022 ceph-mon[40250]: pgmap v2495: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:40.209 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:40] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:21:40.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:40.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:40.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:40 smithi122 ceph-mon[32255]: pgmap v2496: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3228570798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:40 smithi022 ceph-mon[31360]: pgmap v2496: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3228570798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:40 smithi022 ceph-mon[40250]: pgmap v2496: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3228570798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:42 smithi122 ceph-mon[32255]: pgmap v2497: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:21:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:21:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:42 smithi022 ceph-mon[31360]: pgmap v2497: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:42 smithi022 ceph-mon[40250]: pgmap v2497: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:44 smithi122 ceph-mon[32255]: pgmap v2498: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:44 smithi022 ceph-mon[31360]: pgmap v2498: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:44 smithi022 ceph-mon[40250]: pgmap v2498: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:45.582 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:45.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:45.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:45.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:46 smithi122 ceph-mon[32255]: pgmap v2499: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2832431121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:46 smithi022 ceph-mon[31360]: pgmap v2499: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2832431121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:46 smithi022 ceph-mon[40250]: pgmap v2499: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2832431121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:21:48.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:21:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:21:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:48 smithi122 ceph-mon[32255]: pgmap v2500: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:48 smithi022 ceph-mon[31360]: pgmap v2500: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:48 smithi022 ceph-mon[40250]: pgmap v2500: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:50] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:21:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:50 smithi122 ceph-mon[32255]: pgmap v2501: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:50.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:50.957 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:50 smithi022 ceph-mon[31360]: pgmap v2501: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:50.957 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:50 smithi022 ceph-mon[40250]: pgmap v2501: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:51.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:51.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:51.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2313605994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2313605994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2313605994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:52.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:21:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:21:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:52 smithi122 ceph-mon[32255]: pgmap v2502: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:52 smithi022 ceph-mon[31360]: pgmap v2502: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:52 smithi022 ceph-mon[40250]: pgmap v2502: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:54 smithi122 ceph-mon[32255]: pgmap v2503: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:54 smithi022 ceph-mon[31360]: pgmap v2503: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:54 smithi022 ceph-mon[40250]: pgmap v2503: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:21:56.327 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:21:56.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:21:56.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:21:56.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:21:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:56 smithi122 ceph-mon[32255]: pgmap v2504: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:56 smithi022 ceph-mon[31360]: pgmap v2504: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:56 smithi022 ceph-mon[40250]: pgmap v2504: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4085214774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:21:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4085214774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4085214774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:21:57.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:21:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:21:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:58 smithi122 ceph-mon[32255]: pgmap v2505: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:21:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:58 smithi022 ceph-mon[31360]: pgmap v2505: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:21:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:21:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:58 smithi022 ceph-mon[40250]: pgmap v2505: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:21:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:21:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:00] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:22:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:00 smithi122 ceph-mon[32255]: pgmap v2506: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:00 smithi022 ceph-mon[31360]: pgmap v2506: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:00 smithi022 ceph-mon[40250]: pgmap v2506: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:01.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:02.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:02.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:02.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:02 smithi122 ceph-mon[32255]: pgmap v2507: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1070783932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:22:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:22:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:02 smithi022 ceph-mon[31360]: pgmap v2507: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1070783932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:02 smithi022 ceph-mon[40250]: pgmap v2507: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1070783932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:04 smithi122 ceph-mon[32255]: pgmap v2508: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:04 smithi022 ceph-mon[31360]: pgmap v2508: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:04 smithi022 ceph-mon[40250]: pgmap v2508: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:06 smithi122 ceph-mon[32255]: pgmap v2509: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:06 smithi022 ceph-mon[31360]: pgmap v2509: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:06 smithi022 ceph-mon[40250]: pgmap v2509: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:07.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:07.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:07.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:07.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/501028374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:22:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/501028374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/501028374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:07.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:22:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:22:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:08 smithi122 ceph-mon[32255]: pgmap v2510: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:08 smithi022 ceph-mon[40250]: pgmap v2510: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:08 smithi022 ceph-mon[31360]: pgmap v2510: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:10] "GET /metrics HTTP/1.1" 200 202292 "" "Prometheus/2.18.1" 2021-12-15T19:22:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:10 smithi122 ceph-mon[32255]: pgmap v2511: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:22:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:22:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:22:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:22:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[40250]: pgmap v2511: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:22:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:22:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:22:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:22:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[31360]: pgmap v2511: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:22:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:22:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:22:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:22:12.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:12 smithi122 ceph-mon[32255]: pgmap v2512: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:22:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:22:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:12 smithi022 ceph-mon[31360]: pgmap v2512: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:12 smithi022 ceph-mon[40250]: pgmap v2512: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:12.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:12.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:12.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/698115991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/698115991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/698115991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:14 smithi022 ceph-mon[31360]: pgmap v2513: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:14 smithi022 ceph-mon[40250]: pgmap v2513: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:14 smithi122 ceph-mon[32255]: pgmap v2513: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:16 smithi122 ceph-mon[32255]: pgmap v2514: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:16 smithi022 ceph-mon[31360]: pgmap v2514: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:16 smithi022 ceph-mon[40250]: pgmap v2514: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:22:17.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:18.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:22:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:22:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:18 smithi122 ceph-mon[32255]: pgmap v2515: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3261869556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:18 smithi022 ceph-mon[31360]: pgmap v2515: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3261869556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:18 smithi022 ceph-mon[40250]: pgmap v2515: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3261869556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:20] "GET /metrics HTTP/1.1" 200 202287 "" "Prometheus/2.18.1" 2021-12-15T19:22:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:20 smithi122 ceph-mon[32255]: pgmap v2516: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:20 smithi022 ceph-mon[31360]: pgmap v2516: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:20 smithi022 ceph-mon[40250]: pgmap v2516: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:22 smithi122 ceph-mon[32255]: pgmap v2517: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:22:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:22:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:22 smithi022 ceph-mon[31360]: pgmap v2517: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:22 smithi022 ceph-mon[40250]: pgmap v2517: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:23.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:23.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:23.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:23.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/764852074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/764852074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/764852074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:24.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:24 smithi122 ceph-mon[32255]: pgmap v2518: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:24 smithi022 ceph-mon[31360]: pgmap v2518: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:25.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:24 smithi022 ceph-mon[40250]: pgmap v2518: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:26 smithi122 ceph-mon[32255]: pgmap v2519: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:26 smithi022 ceph-mon[31360]: pgmap v2519: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:26 smithi022 ceph-mon[40250]: pgmap v2519: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:22:28.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:22:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:22:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:28.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:28.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:28.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:28 smithi022 ceph-mon[31360]: pgmap v2520: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:28.931 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:28.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:28 smithi022 ceph-mon[40250]: pgmap v2520: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:28.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:28 smithi122 ceph-mon[32255]: pgmap v2520: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4258328100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4258328100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4258328100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:30] "GET /metrics HTTP/1.1" 200 202287 "" "Prometheus/2.18.1" 2021-12-15T19:22:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:30 smithi122 ceph-mon[32255]: pgmap v2521: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:30 smithi022 ceph-mon[31360]: pgmap v2521: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:30 smithi022 ceph-mon[40250]: pgmap v2521: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:32 smithi122 ceph-mon[32255]: pgmap v2522: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:22:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:22:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:32 smithi022 ceph-mon[31360]: pgmap v2522: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:32 smithi022 ceph-mon[40250]: pgmap v2522: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:22:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:22:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:22:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:22:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:22:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:22:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:22:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:22:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:22:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:22:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:22:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:22:34.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:34.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:34.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:34 smithi022 ceph-mon[31360]: pgmap v2523: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/852463586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:34 smithi022 ceph-mon[40250]: pgmap v2523: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/852463586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:34 smithi122 ceph-mon[32255]: pgmap v2523: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/852463586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:36 smithi122 ceph-mon[32255]: pgmap v2524: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:36 smithi022 ceph-mon[31360]: pgmap v2524: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:36 smithi022 ceph-mon[40250]: pgmap v2524: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:22:38.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:22:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:22:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:38 smithi122 ceph-mon[32255]: pgmap v2525: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:38 smithi022 ceph-mon[31360]: pgmap v2525: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:38 smithi022 ceph-mon[40250]: pgmap v2525: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:39.301 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:39.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:39.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:39.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:40] "GET /metrics HTTP/1.1" 200 202287 "" "Prometheus/2.18.1" 2021-12-15T19:22:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:40 smithi122 ceph-mon[32255]: pgmap v2526: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1460752397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:40 smithi022 ceph-mon[31360]: pgmap v2526: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1460752397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:40 smithi022 ceph-mon[40250]: pgmap v2526: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1460752397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:42 smithi122 ceph-mon[32255]: pgmap v2527: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:22:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:22:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:42 smithi022 ceph-mon[31360]: pgmap v2527: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:42 smithi022 ceph-mon[40250]: pgmap v2527: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:44.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:44 smithi122 ceph-mon[32255]: pgmap v2528: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:44 smithi022 ceph-mon[31360]: pgmap v2528: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:44 smithi022 ceph-mon[40250]: pgmap v2528: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:45.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:45.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:45.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/360439231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/360439231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/360439231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:46 smithi122 ceph-mon[32255]: pgmap v2529: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:46 smithi022 ceph-mon[31360]: pgmap v2529: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:46 smithi022 ceph-mon[40250]: pgmap v2529: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:22:48.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:22:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:22:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:48 smithi122 ceph-mon[32255]: pgmap v2530: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:48 smithi022 ceph-mon[31360]: pgmap v2530: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:48 smithi022 ceph-mon[40250]: pgmap v2530: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:50.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:50.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:50.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:50.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:50] "GET /metrics HTTP/1.1" 200 202299 "" "Prometheus/2.18.1" 2021-12-15T19:22:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:50 smithi022 ceph-mon[31360]: pgmap v2531: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1654938666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:50 smithi022 ceph-mon[40250]: pgmap v2531: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1654938666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:50 smithi122 ceph-mon[32255]: pgmap v2531: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1654938666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:52 smithi122 ceph-mon[32255]: pgmap v2532: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:22:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:22:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:52 smithi022 ceph-mon[31360]: pgmap v2532: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:52 smithi022 ceph-mon[40250]: pgmap v2532: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:54 smithi122 ceph-mon[32255]: pgmap v2533: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:54 smithi022 ceph-mon[31360]: pgmap v2533: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:54 smithi022 ceph-mon[40250]: pgmap v2533: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:22:55.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:22:55.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:22:55.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:22:55.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:22:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:56 smithi122 ceph-mon[32255]: pgmap v2534: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3084939211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:56 smithi022 ceph-mon[31360]: pgmap v2534: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3084939211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:56 smithi022 ceph-mon[40250]: pgmap v2534: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3084939211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:22:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:22:58.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:22:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:22:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:58 smithi122 ceph-mon[32255]: pgmap v2535: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:22:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:58 smithi022 ceph-mon[31360]: pgmap v2535: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:22:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:22:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:58 smithi022 ceph-mon[40250]: pgmap v2535: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:22:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:22:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:00] "GET /metrics HTTP/1.1" 200 202299 "" "Prometheus/2.18.1" 2021-12-15T19:23:00.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:00 smithi122 ceph-mon[32255]: pgmap v2536: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:00 smithi022 ceph-mon[31360]: pgmap v2536: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:00 smithi022 ceph-mon[40250]: pgmap v2536: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:01.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:01.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:01.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2608580327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2608580327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2608580327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:02.636 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:23:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:23:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:02 smithi122 ceph-mon[32255]: pgmap v2537: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:02 smithi022 ceph-mon[31360]: pgmap v2537: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:02 smithi022 ceph-mon[40250]: pgmap v2537: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:04 smithi122 ceph-mon[32255]: pgmap v2538: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:04 smithi022 ceph-mon[31360]: pgmap v2538: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:04 smithi022 ceph-mon[40250]: pgmap v2538: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:06.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:06.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:06.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:06.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:06 smithi022 ceph-mon[31360]: pgmap v2539: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3780549999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:06 smithi022 ceph-mon[40250]: pgmap v2539: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3780549999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:06 smithi122 ceph-mon[32255]: pgmap v2539: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3780549999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:23:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:23:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:23:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:08 smithi122 ceph-mon[32255]: pgmap v2540: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:08 smithi022 ceph-mon[31360]: pgmap v2540: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:08 smithi022 ceph-mon[40250]: pgmap v2540: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:10] "GET /metrics HTTP/1.1" 200 202292 "" "Prometheus/2.18.1" 2021-12-15T19:23:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:10 smithi122 ceph-mon[32255]: pgmap v2541: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:23:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:23:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:23:11.014 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[31360]: pgmap v2541: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:11.014 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:23:11.014 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:23:11.014 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:23:11.015 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[40250]: pgmap v2541: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:11.015 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:23:11.015 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:23:11.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:23:11.529 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:11.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:11.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:11.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:23:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/808403083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:23:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/808403083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:23:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/808403083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:23:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:23:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:13 smithi022 ceph-mon[31360]: pgmap v2542: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:13 smithi022 ceph-mon[40250]: pgmap v2542: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:13 smithi122 ceph-mon[32255]: pgmap v2542: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:15 smithi022 ceph-mon[31360]: pgmap v2543: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:15 smithi022 ceph-mon[40250]: pgmap v2543: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:15 smithi122 ceph-mon[32255]: pgmap v2543: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:16.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:17.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:17.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:17.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:17.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:17 smithi022 ceph-mon[31360]: pgmap v2544: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:17.274 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:17 smithi022 ceph-mon[40250]: pgmap v2544: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:17 smithi122 ceph-mon[32255]: pgmap v2544: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:23:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2686757009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:18.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:23:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:23:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2686757009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2686757009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:19 smithi122 ceph-mon[32255]: pgmap v2545: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:19 smithi022 ceph-mon[31360]: pgmap v2545: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:19 smithi022 ceph-mon[40250]: pgmap v2545: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:20] "GET /metrics HTTP/1.1" 200 202292 "" "Prometheus/2.18.1" 2021-12-15T19:23:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:21 smithi122 ceph-mon[32255]: pgmap v2546: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:21 smithi022 ceph-mon[31360]: pgmap v2546: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:21 smithi022 ceph-mon[40250]: pgmap v2546: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:22.274 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:22 smithi022 ceph-mon[31360]: pgmap v2547: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:22 smithi022 ceph-mon[40250]: pgmap v2547: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:22.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:22.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:22.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:22 smithi122 ceph-mon[32255]: pgmap v2547: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:23:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:23:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2103521075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2103521075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2103521075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:24 smithi022 ceph-mon[31360]: pgmap v2548: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:24 smithi022 ceph-mon[40250]: pgmap v2548: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:24 smithi122 ceph-mon[32255]: pgmap v2548: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:26 smithi122 ceph-mon[32255]: pgmap v2549: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:26 smithi022 ceph-mon[31360]: pgmap v2549: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:26 smithi022 ceph-mon[40250]: pgmap v2549: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:27.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:27.647 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:23:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:28.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:23:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:23:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:28 smithi122 ceph-mon[32255]: pgmap v2550: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3314298428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:28 smithi022 ceph-mon[31360]: pgmap v2550: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3314298428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:28 smithi022 ceph-mon[40250]: pgmap v2550: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3314298428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:30] "GET /metrics HTTP/1.1" 200 202292 "" "Prometheus/2.18.1" 2021-12-15T19:23:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:30 smithi122 ceph-mon[32255]: pgmap v2551: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:30 smithi022 ceph-mon[31360]: pgmap v2551: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:30 smithi022 ceph-mon[40250]: pgmap v2551: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:23:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:23:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:32 smithi122 ceph-mon[32255]: pgmap v2552: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:32 smithi022 ceph-mon[31360]: pgmap v2552: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:32 smithi022 ceph-mon[40250]: pgmap v2552: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:33.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:33.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:33.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:33.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:23:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:23:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:23:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:23:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4059021239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:23:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:23:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:23:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:23:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4059021239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:23:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:23:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:23:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:23:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4059021239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:34 smithi022 ceph-mon[31360]: pgmap v2553: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:34 smithi022 ceph-mon[40250]: pgmap v2553: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:34 smithi122 ceph-mon[32255]: pgmap v2553: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:36 smithi122 ceph-mon[32255]: pgmap v2554: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:36 smithi022 ceph-mon[31360]: pgmap v2554: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:36 smithi022 ceph-mon[40250]: pgmap v2554: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:23:38.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:23:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:23:38.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:38.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:38.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:38.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:38.765 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:38 smithi022 ceph-mon[31360]: pgmap v2555: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:38.765 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:38.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:38 smithi022 ceph-mon[40250]: pgmap v2555: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:38.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:38 smithi122 ceph-mon[32255]: pgmap v2555: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1836495529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1836495529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1836495529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:40] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:23:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:40 smithi122 ceph-mon[32255]: pgmap v2556: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:40 smithi022 ceph-mon[31360]: pgmap v2556: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:40 smithi022 ceph-mon[40250]: pgmap v2556: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:42 smithi122 ceph-mon[32255]: pgmap v2557: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:23:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:23:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:42 smithi022 ceph-mon[31360]: pgmap v2557: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:42 smithi022 ceph-mon[40250]: pgmap v2557: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:43.765 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:44.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:44.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:44.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:44 smithi122 ceph-mon[32255]: pgmap v2558: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1782842599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:44 smithi022 ceph-mon[31360]: pgmap v2558: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1782842599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:44 smithi022 ceph-mon[40250]: pgmap v2558: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1782842599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:46 smithi122 ceph-mon[32255]: pgmap v2559: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:46 smithi022 ceph-mon[31360]: pgmap v2559: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:46 smithi022 ceph-mon[40250]: pgmap v2559: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:23:48.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:23:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:23:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:48 smithi122 ceph-mon[32255]: pgmap v2560: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:48 smithi022 ceph-mon[31360]: pgmap v2560: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:48 smithi022 ceph-mon[40250]: pgmap v2560: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:49.137 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:49.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:49.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:49.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/115414349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/115414349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/115414349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:50] "GET /metrics HTTP/1.1" 200 202290 "" "Prometheus/2.18.1" 2021-12-15T19:23:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:50 smithi122 ceph-mon[32255]: pgmap v2561: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:50 smithi022 ceph-mon[31360]: pgmap v2561: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:50 smithi022 ceph-mon[40250]: pgmap v2561: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:52 smithi122 ceph-mon[32255]: pgmap v2562: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:23:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:23:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:52 smithi022 ceph-mon[31360]: pgmap v2562: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:52 smithi022 ceph-mon[40250]: pgmap v2562: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:54.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:23:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:54 smithi022 ceph-mon[31360]: pgmap v2563: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:54 smithi022 ceph-mon[40250]: pgmap v2563: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:54.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:23:54.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:23:54.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:23:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:54 smithi122 ceph-mon[32255]: pgmap v2563: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:23:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1384736238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1384736238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1384736238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:23:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:56 smithi122 ceph-mon[32255]: pgmap v2564: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:56 smithi022 ceph-mon[31360]: pgmap v2564: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:56 smithi022 ceph-mon[40250]: pgmap v2564: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:23:58.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:23:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:23:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:58 smithi122 ceph-mon[32255]: pgmap v2565: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:23:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:58 smithi022 ceph-mon[31360]: pgmap v2565: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:23:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:58 smithi022 ceph-mon[40250]: pgmap v2565: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:23:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:23:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:23:59.878 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:00.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:00.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:00.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:00] "GET /metrics HTTP/1.1" 200 202290 "" "Prometheus/2.18.1" 2021-12-15T19:24:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:00 smithi122 ceph-mon[32255]: pgmap v2566: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3121256351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:00 smithi022 ceph-mon[31360]: pgmap v2566: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3121256351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:00 smithi022 ceph-mon[40250]: pgmap v2566: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3121256351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:02 smithi122 ceph-mon[32255]: pgmap v2567: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:24:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:24:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:02 smithi022 ceph-mon[31360]: pgmap v2567: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:02 smithi022 ceph-mon[40250]: pgmap v2567: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:04 smithi122 ceph-mon[32255]: pgmap v2568: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:04 smithi022 ceph-mon[31360]: pgmap v2568: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:04 smithi022 ceph-mon[40250]: pgmap v2568: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:05.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:05.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:05.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:05.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/606908626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/606908626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/606908626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:06 smithi122 ceph-mon[32255]: pgmap v2569: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:06 smithi022 ceph-mon[31360]: pgmap v2569: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:06 smithi022 ceph-mon[40250]: pgmap v2569: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:24:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:24:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:24:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:08 smithi122 ceph-mon[32255]: pgmap v2570: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:08 smithi022 ceph-mon[31360]: pgmap v2570: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:08 smithi022 ceph-mon[40250]: pgmap v2570: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:10] "GET /metrics HTTP/1.1" 200 202292 "" "Prometheus/2.18.1" 2021-12-15T19:24:10.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:10 smithi122 ceph-mon[32255]: pgmap v2571: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:10.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:10.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:10.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:10.985 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:10 smithi022 ceph-mon[31360]: pgmap v2571: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:10.986 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:10 smithi022 ceph-mon[40250]: pgmap v2571: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3691359735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:24:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:24:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:24:11.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:24:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3691359735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:24:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:24:11.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:24:11.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:24:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3691359735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:24:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:24:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:24:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:24:12.658 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:24:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:24:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:12 smithi122 ceph-mon[32255]: pgmap v2572: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:12 smithi022 ceph-mon[31360]: pgmap v2572: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:12 smithi022 ceph-mon[40250]: pgmap v2572: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:14 smithi122 ceph-mon[32255]: pgmap v2573: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:14 smithi022 ceph-mon[31360]: pgmap v2573: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:14 smithi022 ceph-mon[40250]: pgmap v2573: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:15.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:16 smithi122 ceph-mon[32255]: pgmap v2574: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/851605104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:16 smithi022 ceph-mon[31360]: pgmap v2574: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/851605104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:16 smithi022 ceph-mon[40250]: pgmap v2574: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/851605104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:24:18.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:24:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:24:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:18 smithi122 ceph-mon[32255]: pgmap v2575: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:18 smithi022 ceph-mon[31360]: pgmap v2575: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:18 smithi022 ceph-mon[40250]: pgmap v2575: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:20] "GET /metrics HTTP/1.1" 200 202290 "" "Prometheus/2.18.1" 2021-12-15T19:24:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:20 smithi122 ceph-mon[32255]: pgmap v2576: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:20 smithi022 ceph-mon[31360]: pgmap v2576: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:20 smithi022 ceph-mon[40250]: pgmap v2576: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:21.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:21.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:21.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:21.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:22 smithi122 ceph-mon[32255]: pgmap v2577: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/844054258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:24:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:24:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:22 smithi022 ceph-mon[31360]: pgmap v2577: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/844054258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:22 smithi022 ceph-mon[40250]: pgmap v2577: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:22.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/844054258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:24 smithi122 ceph-mon[32255]: pgmap v2578: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:24 smithi022 ceph-mon[31360]: pgmap v2578: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:24 smithi022 ceph-mon[40250]: pgmap v2578: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:26 smithi122 ceph-mon[32255]: pgmap v2579: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:26 smithi022 ceph-mon[31360]: pgmap v2579: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:26 smithi022 ceph-mon[40250]: pgmap v2579: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:27.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:27.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:27.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:27.658 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:24:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2625342370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:27.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:24:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:24:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2625342370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2625342370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:28 smithi122 ceph-mon[32255]: pgmap v2580: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:28 smithi022 ceph-mon[31360]: pgmap v2580: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:28 smithi022 ceph-mon[40250]: pgmap v2580: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:30] "GET /metrics HTTP/1.1" 200 202290 "" "Prometheus/2.18.1" 2021-12-15T19:24:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:30 smithi122 ceph-mon[32255]: pgmap v2581: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:30 smithi022 ceph-mon[31360]: pgmap v2581: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:30 smithi022 ceph-mon[40250]: pgmap v2581: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:32.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:32.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:32.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:32.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:24:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:24:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:32 smithi122 ceph-mon[32255]: pgmap v2582: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3639112694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:24:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:24:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:24:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:24:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[31360]: pgmap v2582: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3639112694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:24:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:24:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:24:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:24:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[40250]: pgmap v2582: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3639112694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:24:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:24:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:24:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:24:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:34 smithi122 ceph-mon[32255]: pgmap v2583: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:34 smithi022 ceph-mon[31360]: pgmap v2583: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:34 smithi022 ceph-mon[40250]: pgmap v2583: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:36 smithi122 ceph-mon[32255]: pgmap v2584: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:36 smithi022 ceph-mon[31360]: pgmap v2584: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:36 smithi022 ceph-mon[40250]: pgmap v2584: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:37.465 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:24:37.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:37.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:37.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:38.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:24:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:24:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:38 smithi122 ceph-mon[32255]: pgmap v2585: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/679824956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:38 smithi022 ceph-mon[31360]: pgmap v2585: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/679824956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:38 smithi022 ceph-mon[40250]: pgmap v2585: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/679824956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:40] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:24:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:40 smithi122 ceph-mon[32255]: pgmap v2586: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:40 smithi022 ceph-mon[31360]: pgmap v2586: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:40 smithi022 ceph-mon[40250]: pgmap v2586: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:42 smithi122 ceph-mon[32255]: pgmap v2587: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:24:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:24:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:42 smithi022 ceph-mon[31360]: pgmap v2587: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:42 smithi022 ceph-mon[40250]: pgmap v2587: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:42.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:43.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:43.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:43.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1183375274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1183375274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1183375274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:44 smithi022 ceph-mon[31360]: pgmap v2588: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:44 smithi022 ceph-mon[40250]: pgmap v2588: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:44 smithi122 ceph-mon[32255]: pgmap v2588: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:46 smithi122 ceph-mon[32255]: pgmap v2589: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:46 smithi022 ceph-mon[31360]: pgmap v2589: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:46 smithi022 ceph-mon[40250]: pgmap v2589: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:24:48.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:24:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:24:48.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:48.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:48.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:48.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:48 smithi122 ceph-mon[32255]: pgmap v2590: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1834068807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:48 smithi022 ceph-mon[31360]: pgmap v2590: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1834068807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:48 smithi022 ceph-mon[40250]: pgmap v2590: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1834068807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:50] "GET /metrics HTTP/1.1" 200 202291 "" "Prometheus/2.18.1" 2021-12-15T19:24:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:50 smithi122 ceph-mon[32255]: pgmap v2591: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:50 smithi022 ceph-mon[31360]: pgmap v2591: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:50 smithi022 ceph-mon[40250]: pgmap v2591: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:24:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:24:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:52 smithi122 ceph-mon[32255]: pgmap v2592: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:52 smithi022 ceph-mon[31360]: pgmap v2592: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:52 smithi022 ceph-mon[40250]: pgmap v2592: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:53.577 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:53.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:53.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:53.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:54 smithi122 ceph-mon[32255]: pgmap v2593: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2993887110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:54 smithi022 ceph-mon[31360]: pgmap v2593: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2993887110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:54 smithi022 ceph-mon[40250]: pgmap v2593: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:24:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2993887110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:24:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:56 smithi122 ceph-mon[32255]: pgmap v2594: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:56 smithi022 ceph-mon[31360]: pgmap v2594: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:56 smithi022 ceph-mon[40250]: pgmap v2594: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:24:58.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:24:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:24:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:58 smithi122 ceph-mon[32255]: pgmap v2595: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:58.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:24:58.949 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:58 smithi022 ceph-mon[40250]: pgmap v2595: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:58.949 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:58.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:58 smithi022 ceph-mon[31360]: pgmap v2595: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:24:58.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:24:59.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:24:59.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:24:59.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:24:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:24:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2460043134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:24:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2460043134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:24:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2460043134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:00] "GET /metrics HTTP/1.1" 200 202291 "" "Prometheus/2.18.1" 2021-12-15T19:25:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:00 smithi122 ceph-mon[32255]: pgmap v2596: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:00 smithi022 ceph-mon[31360]: pgmap v2596: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:00 smithi022 ceph-mon[40250]: pgmap v2596: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:02 smithi122 ceph-mon[32255]: pgmap v2597: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:25:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:25:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:02 smithi022 ceph-mon[31360]: pgmap v2597: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:02 smithi022 ceph-mon[40250]: pgmap v2597: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:04.321 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:04.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:04.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:04.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:04 smithi122 ceph-mon[32255]: pgmap v2598: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:04 smithi022 ceph-mon[31360]: pgmap v2598: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:04 smithi022 ceph-mon[40250]: pgmap v2598: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4226233449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4226233449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4226233449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:06 smithi122 ceph-mon[32255]: pgmap v2599: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:06 smithi022 ceph-mon[31360]: pgmap v2599: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:06 smithi022 ceph-mon[40250]: pgmap v2599: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:25:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:25:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:25:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:08 smithi122 ceph-mon[32255]: pgmap v2600: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:08 smithi022 ceph-mon[31360]: pgmap v2600: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:08 smithi022 ceph-mon[40250]: pgmap v2600: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:09.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:10.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:10.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:10.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:10] "GET /metrics HTTP/1.1" 200 202301 "" "Prometheus/2.18.1" 2021-12-15T19:25:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:10 smithi122 ceph-mon[32255]: pgmap v2601: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3632725547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:10 smithi022 ceph-mon[31360]: pgmap v2601: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3632725547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:10 smithi022 ceph-mon[40250]: pgmap v2601: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3632725547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:25:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:25:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:25:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:25:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:25:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:12 smithi122 ceph-mon[32255]: pgmap v2602: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:25:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:25:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:25:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:25:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[31360]: pgmap v2602: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:25:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:25:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:25:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:25:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[40250]: pgmap v2602: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:25:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:14 smithi122 ceph-mon[32255]: pgmap v2603: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:14 smithi022 ceph-mon[31360]: pgmap v2603: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:14 smithi022 ceph-mon[40250]: pgmap v2603: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:15.064 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:15.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:15.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:15.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4102586572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4102586572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4102586572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:16 smithi122 ceph-mon[32255]: pgmap v2604: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:16 smithi022 ceph-mon[31360]: pgmap v2604: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:16 smithi022 ceph-mon[40250]: pgmap v2604: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:25:18.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:25:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:25:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:18 smithi122 ceph-mon[32255]: pgmap v2605: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:18 smithi022 ceph-mon[31360]: pgmap v2605: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:19.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:18 smithi022 ceph-mon[40250]: pgmap v2605: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:20.434 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:20] "GET /metrics HTTP/1.1" 200 202288 "" "Prometheus/2.18.1" 2021-12-15T19:25:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:20 smithi022 ceph-mon[31360]: pgmap v2606: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:20 smithi022 ceph-mon[40250]: pgmap v2606: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:20.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:20.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:20.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:20 smithi122 ceph-mon[32255]: pgmap v2606: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2129771562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2129771562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2129771562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:22.680 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:25:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:25:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:22 smithi122 ceph-mon[32255]: pgmap v2607: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:22 smithi022 ceph-mon[31360]: pgmap v2607: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:22 smithi022 ceph-mon[40250]: pgmap v2607: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:24 smithi122 ceph-mon[32255]: pgmap v2608: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:24 smithi022 ceph-mon[31360]: pgmap v2608: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:24 smithi022 ceph-mon[40250]: pgmap v2608: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:25.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:26 smithi022 ceph-mon[31360]: pgmap v2609: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/819214230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:26 smithi022 ceph-mon[40250]: pgmap v2609: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/819214230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:26 smithi122 ceph-mon[32255]: pgmap v2609: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/819214230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:25:28.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:25:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:25:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:28 smithi022 ceph-mon[31360]: pgmap v2610: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:28 smithi022 ceph-mon[40250]: pgmap v2610: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:28 smithi122 ceph-mon[32255]: pgmap v2610: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:30] "GET /metrics HTTP/1.1" 200 202288 "" "Prometheus/2.18.1" 2021-12-15T19:25:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:30 smithi022 ceph-mon[31360]: pgmap v2611: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:30 smithi022 ceph-mon[40250]: pgmap v2611: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:31.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:30 smithi122 ceph-mon[32255]: pgmap v2611: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:31.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:31.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:31.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:25:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:25:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3319930612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:32 smithi122 ceph-mon[32255]: pgmap v2612: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:25:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:25:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:25:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:25:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3319930612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[31360]: pgmap v2612: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:25:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:25:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:25:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:25:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3319930612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[40250]: pgmap v2612: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:25:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:25:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:25:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:25:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:34 smithi122 ceph-mon[32255]: pgmap v2613: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:34 smithi022 ceph-mon[31360]: pgmap v2613: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:34 smithi022 ceph-mon[40250]: pgmap v2613: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:36.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:36.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:36.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:36.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:36.910 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:36 smithi022 ceph-mon[40250]: pgmap v2614: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:36.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:36 smithi022 ceph-mon[31360]: pgmap v2614: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:36 smithi122 ceph-mon[32255]: pgmap v2614: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:37.679 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:25:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/32524133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:37.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:25:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:25:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/32524133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/32524133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:38 smithi122 ceph-mon[32255]: pgmap v2615: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:38 smithi022 ceph-mon[31360]: pgmap v2615: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:38 smithi022 ceph-mon[40250]: pgmap v2615: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:40] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:25:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:40 smithi122 ceph-mon[32255]: pgmap v2616: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:40 smithi022 ceph-mon[31360]: pgmap v2616: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:40 smithi022 ceph-mon[40250]: pgmap v2616: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:41.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:42.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:42.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:42.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:42 smithi122 ceph-mon[32255]: pgmap v2617: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3130097916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:25:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:25:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:42 smithi022 ceph-mon[31360]: pgmap v2617: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3130097916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:42 smithi022 ceph-mon[40250]: pgmap v2617: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3130097916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:44 smithi122 ceph-mon[32255]: pgmap v2618: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:44 smithi022 ceph-mon[31360]: pgmap v2618: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:44 smithi022 ceph-mon[40250]: pgmap v2618: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:46 smithi122 ceph-mon[32255]: pgmap v2619: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:46 smithi022 ceph-mon[31360]: pgmap v2619: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:46 smithi022 ceph-mon[40250]: pgmap v2619: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:47.281 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:47.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:47.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:47.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:47.650 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:25:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2550751583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:47.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:25:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:25:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2550751583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2550751583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:48 smithi122 ceph-mon[32255]: pgmap v2620: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:48 smithi022 ceph-mon[31360]: pgmap v2620: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:48 smithi022 ceph-mon[40250]: pgmap v2620: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:50] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:25:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:50 smithi022 ceph-mon[31360]: pgmap v2621: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:50 smithi022 ceph-mon[40250]: pgmap v2621: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:50 smithi122 ceph-mon[32255]: pgmap v2621: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:25:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:25:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:52 smithi122 ceph-mon[32255]: pgmap v2622: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:52 smithi022 ceph-mon[31360]: pgmap v2622: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:52 smithi022 ceph-mon[40250]: pgmap v2622: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:53.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:53.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:53.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4109512497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4109512497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4109512497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:54 smithi022 ceph-mon[31360]: pgmap v2623: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:54 smithi022 ceph-mon[40250]: pgmap v2623: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:54 smithi122 ceph-mon[32255]: pgmap v2623: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:25:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:56 smithi122 ceph-mon[32255]: pgmap v2624: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:56 smithi022 ceph-mon[31360]: pgmap v2624: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:56 smithi022 ceph-mon[40250]: pgmap v2624: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:25:58.020 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:25:58.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:25:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:25:58.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:25:58.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:25:58.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:25:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:58 smithi122 ceph-mon[32255]: pgmap v2625: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:25:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2498485523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:58 smithi022 ceph-mon[31360]: pgmap v2625: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:25:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2498485523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:25:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:58 smithi022 ceph-mon[40250]: pgmap v2625: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:25:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:25:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:25:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2498485523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:00] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:26:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:00 smithi122 ceph-mon[32255]: pgmap v2626: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:00 smithi022 ceph-mon[31360]: pgmap v2626: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:00 smithi022 ceph-mon[40250]: pgmap v2626: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:02 smithi122 ceph-mon[32255]: pgmap v2627: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:26:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:26:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:02 smithi022 ceph-mon[31360]: pgmap v2627: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:02 smithi022 ceph-mon[40250]: pgmap v2627: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:03.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:03.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:03.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:03.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:04 smithi122 ceph-mon[32255]: pgmap v2628: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3078927367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:04 smithi022 ceph-mon[31360]: pgmap v2628: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3078927367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:04 smithi022 ceph-mon[40250]: pgmap v2628: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3078927367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:06 smithi122 ceph-mon[32255]: pgmap v2629: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:06 smithi022 ceph-mon[31360]: pgmap v2629: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:06 smithi022 ceph-mon[40250]: pgmap v2629: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:26:08.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:26:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:26:08.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:08 smithi022 ceph-mon[31360]: pgmap v2630: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:08 smithi022 ceph-mon[40250]: pgmap v2630: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:09.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:09.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:09.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:08 smithi122 ceph-mon[32255]: pgmap v2630: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2077889565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2077889565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2077889565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:10] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:26:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:10 smithi022 ceph-mon[31360]: pgmap v2631: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:10 smithi022 ceph-mon[40250]: pgmap v2631: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:10 smithi122 ceph-mon[32255]: pgmap v2631: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:26:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:26:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:12 smithi122 ceph-mon[32255]: pgmap v2632: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:26:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:26:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:26:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:12.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[40250]: pgmap v2632: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:12.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:26:12.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:26:12.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:26:12.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:12.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:12.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[31360]: pgmap v2632: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:12.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:26:12.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:26:12.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:26:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:14.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:14.130 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:14.130 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:26:14.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:14.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:14.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:14 smithi122 ceph-mon[32255]: pgmap v2633: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2011539087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:14 smithi022 ceph-mon[31360]: pgmap v2633: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2011539087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:14 smithi022 ceph-mon[40250]: pgmap v2633: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2011539087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:16 smithi122 ceph-mon[32255]: pgmap v2634: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:16 smithi022 ceph-mon[31360]: pgmap v2634: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:16 smithi022 ceph-mon[40250]: pgmap v2634: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:26:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:26:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:26:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:18 smithi122 ceph-mon[32255]: pgmap v2635: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:18 smithi022 ceph-mon[31360]: pgmap v2635: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:18 smithi022 ceph-mon[40250]: pgmap v2635: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:19.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:19.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:19.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:19.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:20] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:26:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:20 smithi122 ceph-mon[32255]: pgmap v2636: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2906204429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:20 smithi022 ceph-mon[31360]: pgmap v2636: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2906204429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:20 smithi022 ceph-mon[40250]: pgmap v2636: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2906204429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:22 smithi122 ceph-mon[32255]: pgmap v2637: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:26:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:26:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:22 smithi022 ceph-mon[31360]: pgmap v2637: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:22 smithi022 ceph-mon[40250]: pgmap v2637: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:24.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:24 smithi022 ceph-mon[31360]: pgmap v2638: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:24 smithi022 ceph-mon[40250]: pgmap v2638: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:24 smithi122 ceph-mon[32255]: pgmap v2638: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:25.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:25.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:25.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1575950599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1575950599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1575950599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:26 smithi022 ceph-mon[31360]: pgmap v2639: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:26 smithi022 ceph-mon[40250]: pgmap v2639: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:26 smithi122 ceph-mon[32255]: pgmap v2639: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:26:28.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:26:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:26:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:28 smithi022 ceph-mon[31360]: pgmap v2640: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:28 smithi022 ceph-mon[40250]: pgmap v2640: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:28 smithi122 ceph-mon[32255]: pgmap v2640: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:30.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:30] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:26:30.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:30.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:30.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:30 smithi022 ceph-mon[31360]: pgmap v2641: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1056207284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:30 smithi022 ceph-mon[40250]: pgmap v2641: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:31.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1056207284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:30 smithi122 ceph-mon[32255]: pgmap v2641: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1056207284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:26:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:26:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:32 smithi122 ceph-mon[32255]: pgmap v2642: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:26:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:26:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:26:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:26:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[31360]: pgmap v2642: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:26:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:26:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:26:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:26:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[40250]: pgmap v2642: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:26:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:26:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:26:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:26:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:34 smithi022 ceph-mon[31360]: pgmap v2643: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:34 smithi022 ceph-mon[40250]: pgmap v2643: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:34 smithi122 ceph-mon[32255]: pgmap v2643: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:35.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:35.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:35.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:35.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:36 smithi022 ceph-mon[31360]: pgmap v2644: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3965209722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:36 smithi022 ceph-mon[40250]: pgmap v2644: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3965209722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:36 smithi122 ceph-mon[32255]: pgmap v2644: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3965209722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:26:38.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:26:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:26:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:38 smithi022 ceph-mon[31360]: pgmap v2645: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:38 smithi022 ceph-mon[40250]: pgmap v2645: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:38 smithi122 ceph-mon[32255]: pgmap v2645: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:40] "GET /metrics HTTP/1.1" 200 202288 "" "Prometheus/2.18.1" 2021-12-15T19:26:40.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:40.978 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:40 smithi022 ceph-mon[31360]: pgmap v2646: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:40.978 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:40 smithi022 ceph-mon[40250]: pgmap v2646: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:40 smithi122 ceph-mon[32255]: pgmap v2646: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:41.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:41.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:41.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/675066422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/675066422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/675066422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:26:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:26:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:42 smithi022 ceph-mon[31360]: pgmap v2647: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:42 smithi022 ceph-mon[40250]: pgmap v2647: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:42 smithi122 ceph-mon[32255]: pgmap v2647: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:44 smithi022 ceph-mon[31360]: pgmap v2648: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:44 smithi022 ceph-mon[40250]: pgmap v2648: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:44 smithi122 ceph-mon[32255]: pgmap v2648: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:46.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:46.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:46.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:46.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:46 smithi022 ceph-mon[31360]: pgmap v2649: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1366424890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:46 smithi022 ceph-mon[40250]: pgmap v2649: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1366424890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:46 smithi122 ceph-mon[32255]: pgmap v2649: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1366424890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:26:48.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:26:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:26:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:48 smithi022 ceph-mon[31360]: pgmap v2650: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:48 smithi022 ceph-mon[40250]: pgmap v2650: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:48 smithi122 ceph-mon[32255]: pgmap v2650: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:50] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:26:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:50 smithi022 ceph-mon[31360]: pgmap v2651: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:50 smithi022 ceph-mon[40250]: pgmap v2651: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:50 smithi122 ceph-mon[32255]: pgmap v2651: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:51.711 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:52.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:52.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:52.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:52.497 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:52 smithi022 ceph-mon[40250]: pgmap v2652: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:52.497 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1914424533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:52 smithi122 ceph-mon[32255]: pgmap v2652: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1914424533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:26:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:26:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:52 smithi022 ceph-mon[31360]: pgmap v2652: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1914424533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:54 smithi022 ceph-mon[31360]: pgmap v2653: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:54 smithi022 ceph-mon[40250]: pgmap v2653: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:54 smithi122 ceph-mon[32255]: pgmap v2653: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:26:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:56 smithi022 ceph-mon[31360]: pgmap v2654: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:56 smithi022 ceph-mon[40250]: pgmap v2654: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:57.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:26:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:56 smithi122 ceph-mon[32255]: pgmap v2654: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:57.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:26:57.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:26:57.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:26:57.704 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:26:57.991 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4028471999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4028471999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4028471999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:26:58.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:26:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:26:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:58 smithi022 ceph-mon[31360]: pgmap v2655: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:26:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:58 smithi022 ceph-mon[40250]: pgmap v2655: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:26:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:26:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:58 smithi122 ceph-mon[32255]: pgmap v2655: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:26:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:26:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:00] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:27:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:00 smithi022 ceph-mon[31360]: pgmap v2656: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:00 smithi022 ceph-mon[40250]: pgmap v2656: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:00 smithi122 ceph-mon[32255]: pgmap v2656: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:02.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:02 smithi122 ceph-mon[32255]: pgmap v2657: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:27:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:27:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:02 smithi022 ceph-mon[31360]: pgmap v2657: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:02 smithi022 ceph-mon[40250]: pgmap v2657: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:02.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:02.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:02.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4093997440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4093997440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4093997440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:04 smithi022 ceph-mon[31360]: pgmap v2658: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:04 smithi022 ceph-mon[40250]: pgmap v2658: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:04 smithi122 ceph-mon[32255]: pgmap v2658: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:06 smithi022 ceph-mon[31360]: pgmap v2659: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:06 smithi022 ceph-mon[40250]: pgmap v2659: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:07.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:06 smithi122 ceph-mon[32255]: pgmap v2659: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:27:07.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:08.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:08.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:08.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:27:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:27:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:08 smithi022 ceph-mon[31360]: pgmap v2660: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2399914053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:08 smithi022 ceph-mon[40250]: pgmap v2660: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2399914053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:08 smithi122 ceph-mon[32255]: pgmap v2660: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2399914053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:10] "GET /metrics HTTP/1.1" 200 202289 "" "Prometheus/2.18.1" 2021-12-15T19:27:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:10 smithi022 ceph-mon[31360]: pgmap v2661: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:10 smithi022 ceph-mon[40250]: pgmap v2661: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:10 smithi122 ceph-mon[32255]: pgmap v2661: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:12 smithi122 ceph-mon[32255]: pgmap v2662: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:27:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:27:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:12 smithi022 ceph-mon[31360]: pgmap v2662: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:12 smithi022 ceph-mon[40250]: pgmap v2662: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:13.190 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:27:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:27:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:27:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:27:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:27:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:27:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:27:13.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:27:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:27:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:27:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:27:13.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:27:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1389133851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:14 smithi022 ceph-mon[31360]: pgmap v2663: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1389133851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:14 smithi022 ceph-mon[40250]: pgmap v2663: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1389133851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:14 smithi122 ceph-mon[32255]: pgmap v2663: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:16 smithi022 ceph-mon[31360]: pgmap v2664: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:16 smithi022 ceph-mon[40250]: pgmap v2664: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:16 smithi122 ceph-mon[32255]: pgmap v2664: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:27:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:27:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:27:18.557 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:18.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:18.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:18.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:18.923 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:18 smithi022 ceph-mon[31360]: pgmap v2665: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:18.923 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:18.923 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:18 smithi022 ceph-mon[40250]: pgmap v2665: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:18.924 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:18 smithi122 ceph-mon[32255]: pgmap v2665: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1327345711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1327345711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1327345711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:20] "GET /metrics HTTP/1.1" 200 202295 "" "Prometheus/2.18.1" 2021-12-15T19:27:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:20 smithi022 ceph-mon[31360]: pgmap v2666: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:20 smithi022 ceph-mon[40250]: pgmap v2666: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:20 smithi122 ceph-mon[32255]: pgmap v2666: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:22 smithi122 ceph-mon[32255]: pgmap v2667: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:27:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:27:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:22 smithi022 ceph-mon[31360]: pgmap v2667: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:22 smithi022 ceph-mon[40250]: pgmap v2667: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:23.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:24.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:24.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:24.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:24 smithi022 ceph-mon[31360]: pgmap v2668: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1638961902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:24 smithi022 ceph-mon[40250]: pgmap v2668: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1638961902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:24 smithi122 ceph-mon[32255]: pgmap v2668: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1638961902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:26 smithi022 ceph-mon[31360]: pgmap v2669: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:26 smithi022 ceph-mon[40250]: pgmap v2669: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:26 smithi122 ceph-mon[32255]: pgmap v2669: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:27:28.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:27:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:27:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:28 smithi022 ceph-mon[31360]: pgmap v2670: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:28 smithi022 ceph-mon[40250]: pgmap v2670: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:28 smithi122 ceph-mon[32255]: pgmap v2670: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:29.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:29.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:29.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:29.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/198652573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/198652573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/198652573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:30] "GET /metrics HTTP/1.1" 200 202295 "" "Prometheus/2.18.1" 2021-12-15T19:27:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:30 smithi022 ceph-mon[31360]: pgmap v2671: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:30 smithi022 ceph-mon[40250]: pgmap v2671: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:30 smithi122 ceph-mon[32255]: pgmap v2671: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:27:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:27:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:32 smithi122 ceph-mon[32255]: pgmap v2672: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:27:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:27:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:27:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:27:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[31360]: pgmap v2672: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:27:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:27:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:27:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:27:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[40250]: pgmap v2672: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:27:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:27:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:27:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:27:34.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:34 smithi022 ceph-mon[31360]: pgmap v2673: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:34 smithi022 ceph-mon[40250]: pgmap v2673: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:35.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:35.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:35.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:34 smithi122 ceph-mon[32255]: pgmap v2673: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/900498229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/900498229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/900498229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:36 smithi022 ceph-mon[31360]: pgmap v2674: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:36 smithi022 ceph-mon[40250]: pgmap v2674: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:36 smithi122 ceph-mon[32255]: pgmap v2674: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:27:38.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:27:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:27:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:38 smithi022 ceph-mon[31360]: pgmap v2675: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:38 smithi022 ceph-mon[40250]: pgmap v2675: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:38 smithi122 ceph-mon[32255]: pgmap v2675: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:40.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:40.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:40.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:40.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:40] "GET /metrics HTTP/1.1" 200 202296 "" "Prometheus/2.18.1" 2021-12-15T19:27:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:40 smithi022 ceph-mon[31360]: pgmap v2676: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/603362766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:40 smithi022 ceph-mon[40250]: pgmap v2676: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:41.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/603362766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:40 smithi122 ceph-mon[32255]: pgmap v2676: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/603362766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:42 smithi122 ceph-mon[32255]: pgmap v2677: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:27:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:27:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:42 smithi022 ceph-mon[31360]: pgmap v2677: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:42 smithi022 ceph-mon[40250]: pgmap v2677: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:44 smithi022 ceph-mon[31360]: pgmap v2678: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:44 smithi022 ceph-mon[40250]: pgmap v2678: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:44 smithi122 ceph-mon[32255]: pgmap v2678: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:45.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:45.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:45.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:45.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:46 smithi022 ceph-mon[31360]: pgmap v2679: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806930828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:46 smithi022 ceph-mon[40250]: pgmap v2679: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806930828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:46 smithi122 ceph-mon[32255]: pgmap v2679: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806930828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:27:48.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:27:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:27:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:48 smithi022 ceph-mon[31360]: pgmap v2680: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:48 smithi022 ceph-mon[40250]: pgmap v2680: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:48 smithi122 ceph-mon[32255]: pgmap v2680: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:50] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:27:50.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:50 smithi022 ceph-mon[31360]: pgmap v2681: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:50 smithi022 ceph-mon[40250]: pgmap v2681: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:51.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:51.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:51.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:50 smithi122 ceph-mon[32255]: pgmap v2681: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:27:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1016412660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1016412660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1016412660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:27:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:27:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:52 smithi022 ceph-mon[31360]: pgmap v2682: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:52 smithi022 ceph-mon[40250]: pgmap v2682: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:52 smithi122 ceph-mon[32255]: pgmap v2682: 132 pgs: 132 active+clean; 582 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:27:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:54 smithi022 ceph-mon[31360]: pgmap v2683: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:54 smithi022 ceph-mon[40250]: pgmap v2683: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:54 smithi122 ceph-mon[32255]: pgmap v2683: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:56.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:27:56.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:27:56.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:27:56.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:27:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:56 smithi022 ceph-mon[31360]: pgmap v2684: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2772958462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:56 smithi022 ceph-mon[40250]: pgmap v2684: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2772958462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:56 smithi122 ceph-mon[32255]: pgmap v2684: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2772958462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:27:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:27:58.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:27:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:27:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:58 smithi022 ceph-mon[31360]: pgmap v2685: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:27:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:58 smithi022 ceph-mon[40250]: pgmap v2685: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:27:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:27:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:58 smithi122 ceph-mon[32255]: pgmap v2685: 132 pgs: 132 active+clean; 582 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:27:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:27:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:00] "GET /metrics HTTP/1.1" 200 202294 "" "Prometheus/2.18.1" 2021-12-15T19:28:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:00 smithi022 ceph-mon[31360]: pgmap v2686: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:00 smithi022 ceph-mon[40250]: pgmap v2686: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:00 smithi122 ceph-mon[32255]: pgmap v2686: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:01.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:01.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:01.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:02 smithi122 ceph-mon[32255]: pgmap v2687: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3488931290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:28:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:28:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:02 smithi022 ceph-mon[31360]: pgmap v2687: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3488931290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:02 smithi022 ceph-mon[40250]: pgmap v2687: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3488931290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:04 smithi022 ceph-mon[31360]: pgmap v2688: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:04 smithi022 ceph-mon[40250]: pgmap v2688: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:04 smithi122 ceph-mon[32255]: pgmap v2688: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:28:06.868 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:06 smithi022 ceph-mon[31360]: pgmap v2689: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:28:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:06 smithi022 ceph-mon[40250]: pgmap v2689: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:28:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:06 smithi122 ceph-mon[32255]: pgmap v2689: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:28:07.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:07.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:07.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:28:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/569780287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/569780287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:08.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/569780287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:28:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:28:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:08 smithi022 ceph-mon[31360]: pgmap v2690: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:28:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:08 smithi022 ceph-mon[40250]: pgmap v2690: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:28:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:08 smithi122 ceph-mon[32255]: pgmap v2690: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:28:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:10] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:28:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:10 smithi022 ceph-mon[31360]: pgmap v2691: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:28:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:10 smithi022 ceph-mon[40250]: pgmap v2691: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:28:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:10 smithi122 ceph-mon[32255]: pgmap v2691: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:28:12.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:12.514 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:12 smithi022 ceph-mon[40250]: pgmap v2692: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:12.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:12.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:12.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:12 smithi122 ceph-mon[32255]: pgmap v2692: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:28:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:28:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:12 smithi022 ceph-mon[31360]: pgmap v2692: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2075915303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:28:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:28:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:28:13.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2075915303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:13.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:28:13.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:28:13.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:28:13.698 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2075915303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:13.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:28:13.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:28:13.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:28:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:14 smithi022 ceph-mon[40250]: pgmap v2693: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:28:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:14 smithi022 ceph-mon[31360]: pgmap v2693: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:28:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:14 smithi122 ceph-mon[32255]: pgmap v2693: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:28:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:16 smithi022 ceph-mon[40250]: pgmap v2694: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:16 smithi022 ceph-mon[31360]: pgmap v2694: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:16 smithi122 ceph-mon[32255]: pgmap v2694: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:17.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:28:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:28:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:28:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:18 smithi022 ceph-mon[31360]: pgmap v2695: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1116899047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:18 smithi022 ceph-mon[40250]: pgmap v2695: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1116899047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:18 smithi122 ceph-mon[32255]: pgmap v2695: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1116899047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:20] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:28:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:20 smithi022 ceph-mon[31360]: pgmap v2696: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:20 smithi022 ceph-mon[40250]: pgmap v2696: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:20 smithi122 ceph-mon[32255]: pgmap v2696: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:22 smithi122 ceph-mon[32255]: pgmap v2697: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:28:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:28:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:22 smithi022 ceph-mon[31360]: pgmap v2697: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:22 smithi022 ceph-mon[40250]: pgmap v2697: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:22.986 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:23.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:23.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:23.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/940278894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/940278894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/940278894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:24 smithi022 ceph-mon[31360]: pgmap v2698: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:24 smithi022 ceph-mon[40250]: pgmap v2698: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:24 smithi122 ceph-mon[32255]: pgmap v2698: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:26 smithi022 ceph-mon[31360]: pgmap v2699: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:26 smithi022 ceph-mon[40250]: pgmap v2699: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:26 smithi122 ceph-mon[32255]: pgmap v2699: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:28:28.363 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:28.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:28:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:28:28.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:28.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:28.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:28 smithi022 ceph-mon[31360]: pgmap v2700: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/867336987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:28 smithi022 ceph-mon[40250]: pgmap v2700: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/867336987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:28 smithi122 ceph-mon[32255]: pgmap v2700: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/867336987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:30] "GET /metrics HTTP/1.1" 200 202293 "" "Prometheus/2.18.1" 2021-12-15T19:28:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:30 smithi022 ceph-mon[31360]: pgmap v2701: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:30 smithi022 ceph-mon[40250]: pgmap v2701: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:30 smithi122 ceph-mon[32255]: pgmap v2701: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:32 smithi122 ceph-mon[32255]: pgmap v2702: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:28:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:28:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:28:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:28:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[31360]: pgmap v2702: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:28:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:28:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:28:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:28:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[40250]: pgmap v2702: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:28:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:28:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:28:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:28:33.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:34.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:34.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:34.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:34 smithi022 ceph-mon[31360]: pgmap v2703: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/64268024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:34 smithi022 ceph-mon[40250]: pgmap v2703: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/64268024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:34 smithi122 ceph-mon[32255]: pgmap v2703: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/64268024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:36 smithi022 ceph-mon[31360]: pgmap v2704: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:36 smithi022 ceph-mon[40250]: pgmap v2704: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:36 smithi122 ceph-mon[32255]: pgmap v2704: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:28:38.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:28:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:28:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:38 smithi022 ceph-mon[31360]: pgmap v2705: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:38 smithi022 ceph-mon[40250]: pgmap v2705: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:39.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:38 smithi122 ceph-mon[32255]: pgmap v2705: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:39.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:39.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:39.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/192717990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/192717990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/192717990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:40] "GET /metrics HTTP/1.1" 200 202308 "" "Prometheus/2.18.1" 2021-12-15T19:28:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:40 smithi022 ceph-mon[31360]: pgmap v2706: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:40 smithi022 ceph-mon[40250]: pgmap v2706: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:40 smithi122 ceph-mon[32255]: pgmap v2706: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:42 smithi122 ceph-mon[32255]: pgmap v2707: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:28:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:28:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:42 smithi022 ceph-mon[31360]: pgmap v2707: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:42 smithi022 ceph-mon[40250]: pgmap v2707: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:28:44.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:44.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:44.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:44.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:44 smithi022 ceph-mon[31360]: pgmap v2708: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:44 smithi022 ceph-mon[40250]: pgmap v2708: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:44 smithi122 ceph-mon[32255]: pgmap v2708: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2417408259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2417408259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2417408259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:46 smithi022 ceph-mon[31360]: pgmap v2709: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:46 smithi022 ceph-mon[40250]: pgmap v2709: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:46 smithi122 ceph-mon[32255]: pgmap v2709: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:28:48.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:28:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:28:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:48 smithi022 ceph-mon[31360]: pgmap v2710: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:48 smithi022 ceph-mon[40250]: pgmap v2710: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:48 smithi122 ceph-mon[32255]: pgmap v2710: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:49.841 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:50.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:50.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:50.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:50] "GET /metrics HTTP/1.1" 200 202302 "" "Prometheus/2.18.1" 2021-12-15T19:28:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:50 smithi022 ceph-mon[31360]: pgmap v2711: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/451581561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:50 smithi022 ceph-mon[40250]: pgmap v2711: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/451581561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:50 smithi122 ceph-mon[32255]: pgmap v2711: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:51.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/451581561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:52 smithi122 ceph-mon[32255]: pgmap v2712: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:28:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:28:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:52 smithi022 ceph-mon[31360]: pgmap v2712: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:52 smithi022 ceph-mon[40250]: pgmap v2712: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:54 smithi022 ceph-mon[31360]: pgmap v2713: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:54 smithi022 ceph-mon[40250]: pgmap v2713: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:54 smithi122 ceph-mon[32255]: pgmap v2713: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:28:55.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:28:55.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:28:55.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:28:55.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:28:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4080031538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4080031538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4080031538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:28:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:56 smithi022 ceph-mon[31360]: pgmap v2714: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:56 smithi022 ceph-mon[40250]: pgmap v2714: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:56 smithi122 ceph-mon[32255]: pgmap v2714: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:28:58.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:28:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:28:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:58 smithi022 ceph-mon[31360]: pgmap v2715: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:28:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:58 smithi022 ceph-mon[40250]: pgmap v2715: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:28:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:28:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:58 smithi122 ceph-mon[32255]: pgmap v2715: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:28:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:28:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:00] "GET /metrics HTTP/1.1" 200 202302 "" "Prometheus/2.18.1" 2021-12-15T19:29:00.583 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:00.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:00.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:00.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:00.950 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:00 smithi022 ceph-mon[40250]: pgmap v2716: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:00.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:00 smithi022 ceph-mon[31360]: pgmap v2716: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:00 smithi122 ceph-mon[32255]: pgmap v2716: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3421243390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3421243390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3421243390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:29:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:29:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:02 smithi022 ceph-mon[31360]: pgmap v2717: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:02 smithi022 ceph-mon[40250]: pgmap v2717: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:02 smithi122 ceph-mon[32255]: pgmap v2717: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:04 smithi022 ceph-mon[31360]: pgmap v2718: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:04 smithi022 ceph-mon[40250]: pgmap v2718: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:04 smithi122 ceph-mon[32255]: pgmap v2718: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:05.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:06.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:06.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:06.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:06 smithi022 ceph-mon[31360]: pgmap v2719: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/504956882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:06 smithi022 ceph-mon[40250]: pgmap v2719: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/504956882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:06 smithi122 ceph-mon[32255]: pgmap v2719: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/504956882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:29:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:29:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:29:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:08 smithi022 ceph-mon[31360]: pgmap v2720: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:08 smithi022 ceph-mon[40250]: pgmap v2720: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:08 smithi122 ceph-mon[32255]: pgmap v2720: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:10] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:29:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:10 smithi022 ceph-mon[31360]: pgmap v2721: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:10 smithi022 ceph-mon[40250]: pgmap v2721: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:10 smithi122 ceph-mon[32255]: pgmap v2721: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:11.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:11.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:11.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:11.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:29:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:29:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/403074804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:12 smithi122 ceph-mon[32255]: pgmap v2722: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/403074804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:12 smithi022 ceph-mon[31360]: pgmap v2722: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/403074804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:12 smithi022 ceph-mon[40250]: pgmap v2722: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:29:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:29:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:29:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:29:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:29:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:29:14.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:29:14.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:29:14.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:29:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:15 smithi122 ceph-mon[32255]: pgmap v2723: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:29:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:15 smithi022 ceph-mon[31360]: pgmap v2723: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:29:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:15 smithi022 ceph-mon[40250]: pgmap v2723: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:29:16.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:17 smithi022 ceph-mon[31360]: pgmap v2724: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3630460497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:17 smithi022 ceph-mon[40250]: pgmap v2724: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3630460497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:17 smithi122 ceph-mon[32255]: pgmap v2724: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3630460497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:29:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:29:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:29:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:19 smithi122 ceph-mon[32255]: pgmap v2725: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:19 smithi022 ceph-mon[31360]: pgmap v2725: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:19 smithi022 ceph-mon[40250]: pgmap v2725: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:20] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:29:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:21 smithi122 ceph-mon[32255]: pgmap v2726: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:21 smithi022 ceph-mon[31360]: pgmap v2726: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:21 smithi022 ceph-mon[40250]: pgmap v2726: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:22.066 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:22.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:22.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:22.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:22 smithi122 ceph-mon[32255]: pgmap v2727: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1304341586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:29:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:29:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:22 smithi022 ceph-mon[31360]: pgmap v2727: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1304341586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:22 smithi022 ceph-mon[40250]: pgmap v2727: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1304341586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:24 smithi022 ceph-mon[31360]: pgmap v2728: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:24 smithi022 ceph-mon[40250]: pgmap v2728: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:24 smithi122 ceph-mon[32255]: pgmap v2728: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:26 smithi022 ceph-mon[31360]: pgmap v2729: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:26 smithi022 ceph-mon[40250]: pgmap v2729: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:26 smithi122 ceph-mon[32255]: pgmap v2729: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:27.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:29:27.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:27.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:27.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:28.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:29:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:29:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:28 smithi022 ceph-mon[31360]: pgmap v2730: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4070649130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:28 smithi022 ceph-mon[40250]: pgmap v2730: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4070649130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:28 smithi122 ceph-mon[32255]: pgmap v2730: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4070649130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:30] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:29:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:30 smithi022 ceph-mon[31360]: pgmap v2731: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:30 smithi022 ceph-mon[40250]: pgmap v2731: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:30 smithi122 ceph-mon[32255]: pgmap v2731: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:29:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:29:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:32 smithi122 ceph-mon[32255]: pgmap v2732: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:29:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:29:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:29:32.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:29:32.804 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:32.805 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[40250]: pgmap v2732: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:32.806 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:29:32.806 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:29:32.807 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:29:32.807 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:29:32.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[31360]: pgmap v2732: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:32.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:29:32.808 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:29:32.809 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:29:32.809 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:29:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/333075356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/333075356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/333075356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:34 smithi122 ceph-mon[32255]: pgmap v2733: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:34 smithi022 ceph-mon[31360]: pgmap v2733: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:34 smithi022 ceph-mon[40250]: pgmap v2733: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:36 smithi022 ceph-mon[31360]: pgmap v2734: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:36 smithi022 ceph-mon[40250]: pgmap v2734: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:36 smithi122 ceph-mon[32255]: pgmap v2734: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:29:38.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:38.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:29:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:29:38.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:38.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:38.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:38 smithi022 ceph-mon[31360]: pgmap v2735: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1326825766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:38 smithi022 ceph-mon[40250]: pgmap v2735: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1326825766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:38 smithi122 ceph-mon[32255]: pgmap v2735: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1326825766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:40] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:29:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:40 smithi022 ceph-mon[31360]: pgmap v2736: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:40 smithi022 ceph-mon[40250]: pgmap v2736: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:40 smithi122 ceph-mon[32255]: pgmap v2736: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:42 smithi122 ceph-mon[32255]: pgmap v2737: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:29:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:29:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:42 smithi022 ceph-mon[31360]: pgmap v2737: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:42 smithi022 ceph-mon[40250]: pgmap v2737: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:43.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:43.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:43.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:43.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:44 smithi022 ceph-mon[31360]: pgmap v2738: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3458383914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:45.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:44 smithi022 ceph-mon[40250]: pgmap v2738: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:45.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3458383914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:44 smithi122 ceph-mon[32255]: pgmap v2738: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3458383914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:46 smithi022 ceph-mon[31360]: pgmap v2739: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:46 smithi022 ceph-mon[40250]: pgmap v2739: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:46 smithi122 ceph-mon[32255]: pgmap v2739: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:29:48.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:29:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:29:48.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:48 smithi022 ceph-mon[31360]: pgmap v2740: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:48 smithi022 ceph-mon[40250]: pgmap v2740: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:48 smithi122 ceph-mon[32255]: pgmap v2740: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:49.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:49.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:49.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3125864303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3125864303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3125864303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:50] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:29:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:50 smithi022 ceph-mon[31360]: pgmap v2741: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:50 smithi022 ceph-mon[40250]: pgmap v2741: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:50 smithi122 ceph-mon[32255]: pgmap v2741: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:52 smithi122 ceph-mon[32255]: pgmap v2742: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:29:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:29:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:52 smithi022 ceph-mon[31360]: pgmap v2742: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:52 smithi022 ceph-mon[40250]: pgmap v2742: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:54.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:29:54.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:29:54.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:29:54.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:29:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:54 smithi022 ceph-mon[31360]: pgmap v2743: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2511937612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:54 smithi022 ceph-mon[40250]: pgmap v2743: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2511937612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:54 smithi122 ceph-mon[32255]: pgmap v2743: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:29:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2511937612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:29:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:56 smithi022 ceph-mon[31360]: pgmap v2744: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:56 smithi022 ceph-mon[40250]: pgmap v2744: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:56 smithi122 ceph-mon[32255]: pgmap v2744: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:29:58.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:29:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:29:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:58 smithi022 ceph-mon[31360]: pgmap v2745: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:29:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:58 smithi022 ceph-mon[40250]: pgmap v2745: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:29:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:58 smithi122 ceph-mon[32255]: pgmap v2745: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:29:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:29:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:29:59.663 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:00.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:00.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:00.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:00] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:30:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:00 smithi022 ceph-mon[31360]: pgmap v2746: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T19:30:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3889381996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:00 smithi022 ceph-mon[40250]: pgmap v2746: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T19:30:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3889381996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:00 smithi122 ceph-mon[32255]: pgmap v2746: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T19:30:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3889381996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:02 smithi122 ceph-mon[32255]: pgmap v2747: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:30:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:30:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:02 smithi022 ceph-mon[31360]: pgmap v2747: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:02 smithi022 ceph-mon[40250]: pgmap v2747: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:04 smithi022 ceph-mon[31360]: pgmap v2748: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:04 smithi022 ceph-mon[40250]: pgmap v2748: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:05.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:04 smithi122 ceph-mon[32255]: pgmap v2748: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:05.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:05.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:05.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2222757621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2222757621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2222757621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:06 smithi022 ceph-mon[31360]: pgmap v2749: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:06 smithi022 ceph-mon[40250]: pgmap v2749: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:06 smithi122 ceph-mon[32255]: pgmap v2749: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:30:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:30:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:30:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:08 smithi022 ceph-mon[31360]: pgmap v2750: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:08 smithi022 ceph-mon[40250]: pgmap v2750: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:08 smithi122 ceph-mon[32255]: pgmap v2750: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:10.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:10] "GET /metrics HTTP/1.1" 200 202295 "" "Prometheus/2.18.1" 2021-12-15T19:30:10.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:10 smithi022 ceph-mon[40250]: pgmap v2751: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:10.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1437440413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:10.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:10 smithi022 ceph-mon[31360]: pgmap v2751: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:10.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1437440413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:10.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:10.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:10.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:10 smithi122 ceph-mon[32255]: pgmap v2751: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1437440413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:12 smithi122 ceph-mon[32255]: pgmap v2752: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:30:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:30:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:12 smithi022 ceph-mon[31360]: pgmap v2752: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:12 smithi022 ceph-mon[40250]: pgmap v2752: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[31360]: pgmap v2753: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:30:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:30:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:30:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[40250]: pgmap v2753: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:30:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:30:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:30:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:14 smithi122 ceph-mon[32255]: pgmap v2753: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:30:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:30:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:30:15.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:16.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:16.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:16.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:16 smithi022 ceph-mon[31360]: pgmap v2754: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1087278166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:16 smithi022 ceph-mon[40250]: pgmap v2754: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1087278166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:16 smithi122 ceph-mon[32255]: pgmap v2754: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1087278166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:17.657 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:30:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:30:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:30:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[31360]: pgmap v2755: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[40250]: pgmap v2755: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:18 smithi122 ceph-mon[32255]: pgmap v2755: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:19.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:19.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:19.899 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:19.899 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:20] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:30:20.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:20 smithi122 ceph-mon[32255]: pgmap v2756: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:20.908 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:20 smithi022 ceph-mon[31360]: pgmap v2756: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:20.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:20 smithi022 ceph-mon[40250]: pgmap v2756: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:21.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:21.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:21.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:21.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:22.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:22.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/21213607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/21213607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/21213607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:30:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:30:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:23 smithi122 ceph-mon[32255]: pgmap v2757: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:23 smithi022 ceph-mon[31360]: pgmap v2757: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:23 smithi022 ceph-mon[40250]: pgmap v2757: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:30:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:25 smithi122 ceph-mon[32255]: pgmap v2758: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:25 smithi022 ceph-mon[31360]: pgmap v2758: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:25 smithi022 ceph-mon[40250]: pgmap v2758: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:26.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:26.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:26.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:26.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:27 smithi022 ceph-mon[31360]: pgmap v2759: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3558258473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:27 smithi022 ceph-mon[40250]: pgmap v2759: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3558258473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:27 smithi122 ceph-mon[32255]: pgmap v2759: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3558258473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:30:28.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:30:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:30:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:29 smithi122 ceph-mon[32255]: pgmap v2760: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:29 smithi022 ceph-mon[31360]: pgmap v2760: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:29 smithi022 ceph-mon[40250]: pgmap v2760: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:30] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:30:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:31 smithi122 ceph-mon[32255]: pgmap v2761: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:31 smithi022 ceph-mon[31360]: pgmap v2761: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:31 smithi022 ceph-mon[40250]: pgmap v2761: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:31.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:32.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:32.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:32.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:30:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:30:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:32 smithi122 ceph-mon[32255]: pgmap v2762: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4176696787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:30:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:30:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:30:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[31360]: pgmap v2762: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4176696787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:30:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:30:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:30:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[40250]: pgmap v2762: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4176696787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:30:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:30:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:30:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:30:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:34 smithi022 ceph-mon[31360]: pgmap v2763: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:34 smithi022 ceph-mon[40250]: pgmap v2763: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:34 smithi122 ceph-mon[32255]: pgmap v2763: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:36 smithi022 ceph-mon[31360]: pgmap v2764: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:36 smithi022 ceph-mon[40250]: pgmap v2764: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:36 smithi122 ceph-mon[32255]: pgmap v2764: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:37.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:30:37.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:37.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:37.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/573214407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/573214407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:38.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/573214407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:38.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:30:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:30:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:38 smithi022 ceph-mon[31360]: pgmap v2765: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:38 smithi022 ceph-mon[40250]: pgmap v2765: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:38 smithi122 ceph-mon[32255]: pgmap v2765: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:40] "GET /metrics HTTP/1.1" 200 202302 "" "Prometheus/2.18.1" 2021-12-15T19:30:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:40 smithi022 ceph-mon[31360]: pgmap v2766: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:40 smithi022 ceph-mon[40250]: pgmap v2766: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:40 smithi122 ceph-mon[32255]: pgmap v2766: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:42.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:42 smithi122 ceph-mon[32255]: pgmap v2767: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:30:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:30:43.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:43.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:43.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:43.003 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:42 smithi022 ceph-mon[31360]: pgmap v2767: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:43.003 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:42 smithi022 ceph-mon[40250]: pgmap v2767: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/505735353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/505735353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/505735353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:44 smithi122 ceph-mon[32255]: pgmap v2768: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:44 smithi022 ceph-mon[31360]: pgmap v2768: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:44 smithi022 ceph-mon[40250]: pgmap v2768: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:46 smithi022 ceph-mon[31360]: pgmap v2769: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:46 smithi022 ceph-mon[40250]: pgmap v2769: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:46 smithi122 ceph-mon[32255]: pgmap v2769: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:30:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:48.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:48.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:48.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:48.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:30:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:30:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:48 smithi022 ceph-mon[31360]: pgmap v2770: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3275092929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:48 smithi022 ceph-mon[40250]: pgmap v2770: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3275092929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:48 smithi122 ceph-mon[32255]: pgmap v2770: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3275092929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:50] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:30:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:50 smithi022 ceph-mon[31360]: pgmap v2771: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:50 smithi022 ceph-mon[40250]: pgmap v2771: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:50 smithi122 ceph-mon[32255]: pgmap v2771: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:52 smithi122 ceph-mon[32255]: pgmap v2772: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:30:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:30:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:52 smithi022 ceph-mon[31360]: pgmap v2772: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:52 smithi022 ceph-mon[40250]: pgmap v2772: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:53.377 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:53.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:53.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:53.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3688669138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3688669138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3688669138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:30:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:54 smithi122 ceph-mon[32255]: pgmap v2773: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:54 smithi022 ceph-mon[31360]: pgmap v2773: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:54 smithi022 ceph-mon[40250]: pgmap v2773: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:30:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:56 smithi122 ceph-mon[32255]: pgmap v2774: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:56 smithi022 ceph-mon[31360]: pgmap v2774: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:56 smithi022 ceph-mon[40250]: pgmap v2774: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:30:58.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:30:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:30:58.746 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:30:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:58 smithi022 ceph-mon[31360]: pgmap v2775: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:58 smithi022 ceph-mon[40250]: pgmap v2775: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:30:59.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:30:59.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:30:59.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:30:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:58 smithi122 ceph-mon[32255]: pgmap v2775: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:30:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:30:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1735581794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:30:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1735581794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:30:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1735581794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:00] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:31:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:00 smithi122 ceph-mon[32255]: pgmap v2776: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:00 smithi022 ceph-mon[31360]: pgmap v2776: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:00 smithi022 ceph-mon[40250]: pgmap v2776: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:02 smithi122 ceph-mon[32255]: pgmap v2777: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:31:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:31:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:02 smithi022 ceph-mon[31360]: pgmap v2777: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:02 smithi022 ceph-mon[40250]: pgmap v2777: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:04.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:04.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:04.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:04.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:04 smithi022 ceph-mon[31360]: pgmap v2778: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2228815543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:04 smithi022 ceph-mon[40250]: pgmap v2778: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2228815543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:04 smithi122 ceph-mon[32255]: pgmap v2778: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2228815543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:06 smithi022 ceph-mon[31360]: pgmap v2779: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:06 smithi022 ceph-mon[40250]: pgmap v2779: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:06 smithi122 ceph-mon[32255]: pgmap v2779: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:31:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:31:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:31:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:08 smithi022 ceph-mon[31360]: pgmap v2780: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:08 smithi022 ceph-mon[40250]: pgmap v2780: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:08 smithi122 ceph-mon[32255]: pgmap v2780: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:09.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:09.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:09.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:09.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:10] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:31:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:10 smithi022 ceph-mon[31360]: pgmap v2781: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2850726666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:10 smithi022 ceph-mon[40250]: pgmap v2781: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2850726666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:10 smithi122 ceph-mon[32255]: pgmap v2781: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2850726666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:12 smithi122 ceph-mon[32255]: pgmap v2782: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:31:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:31:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:12 smithi022 ceph-mon[31360]: pgmap v2782: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:12 smithi022 ceph-mon[40250]: pgmap v2782: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:14.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:14 smithi022 ceph-mon[31360]: pgmap v2783: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:14 smithi022 ceph-mon[40250]: pgmap v2783: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:14 smithi122 ceph-mon[32255]: pgmap v2783: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:15.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:15.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:15.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:16.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/779285307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:16.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/779285307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/779285307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:16 smithi122 ceph-mon[32255]: pgmap v2784: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:16 smithi022 ceph-mon[31360]: pgmap v2784: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:16 smithi022 ceph-mon[40250]: pgmap v2784: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:31:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:31:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:31:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:18 smithi122 ceph-mon[32255]: pgmap v2785: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:18 smithi022 ceph-mon[31360]: pgmap v2785: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:18 smithi022 ceph-mon[40250]: pgmap v2785: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:20.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:20] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:31:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:20.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:20.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:20 smithi022 ceph-mon[31360]: pgmap v2786: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1246987424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:20 smithi022 ceph-mon[40250]: pgmap v2786: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1246987424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:20 smithi122 ceph-mon[32255]: pgmap v2786: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1246987424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:31:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:31:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:22 smithi122 ceph-mon[32255]: pgmap v2787: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:31:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:31:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:31:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[31360]: pgmap v2787: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:31:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:31:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:31:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[40250]: pgmap v2787: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:31:23.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:31:23.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:31:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:31:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:31:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:31:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:24 smithi122 ceph-mon[32255]: pgmap v2788: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:24 smithi022 ceph-mon[31360]: pgmap v2788: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:24 smithi022 ceph-mon[40250]: pgmap v2788: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:26 smithi022 ceph-mon[31360]: pgmap v2789: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2878051570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:26 smithi022 ceph-mon[40250]: pgmap v2789: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2878051570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:26 smithi122 ceph-mon[32255]: pgmap v2789: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2878051570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:31:28.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:31:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:31:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:28 smithi122 ceph-mon[32255]: pgmap v2790: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:28 smithi022 ceph-mon[31360]: pgmap v2790: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:28 smithi022 ceph-mon[40250]: pgmap v2790: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:30] "GET /metrics HTTP/1.1" 200 202305 "" "Prometheus/2.18.1" 2021-12-15T19:31:30.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:30 smithi122 ceph-mon[32255]: pgmap v2791: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:31.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:30 smithi022 ceph-mon[31360]: pgmap v2791: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:31.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:30 smithi022 ceph-mon[40250]: pgmap v2791: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:31.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:31.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:31.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1214895229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1214895229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1214895229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:31:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:31:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:32 smithi122 ceph-mon[32255]: pgmap v2792: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:31:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:31:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:31:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:31:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[31360]: pgmap v2792: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:31:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:31:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:31:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:31:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[40250]: pgmap v2792: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:31:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:31:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:31:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:31:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:34 smithi122 ceph-mon[32255]: pgmap v2793: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:34 smithi022 ceph-mon[31360]: pgmap v2793: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:34 smithi022 ceph-mon[40250]: pgmap v2793: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:36.345 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:36.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:36.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:36.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:36 smithi022 ceph-mon[31360]: pgmap v2794: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2381537337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:36 smithi022 ceph-mon[40250]: pgmap v2794: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2381537337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:36 smithi122 ceph-mon[32255]: pgmap v2794: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2381537337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:31:38.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:31:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:31:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:38 smithi122 ceph-mon[32255]: pgmap v2795: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:38 smithi022 ceph-mon[31360]: pgmap v2795: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:38 smithi022 ceph-mon[40250]: pgmap v2795: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:40] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:31:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:40 smithi122 ceph-mon[32255]: pgmap v2796: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:40 smithi022 ceph-mon[31360]: pgmap v2796: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:40 smithi022 ceph-mon[40250]: pgmap v2796: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:42.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:42.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:42.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:42 smithi122 ceph-mon[32255]: pgmap v2797: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1380903440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:31:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:31:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:42 smithi022 ceph-mon[31360]: pgmap v2797: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1380903440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:42 smithi022 ceph-mon[40250]: pgmap v2797: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1380903440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:44 smithi122 ceph-mon[32255]: pgmap v2798: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:44 smithi022 ceph-mon[31360]: pgmap v2798: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:44 smithi022 ceph-mon[40250]: pgmap v2798: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:47.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:47.086 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:46 smithi022 ceph-mon[31360]: pgmap v2799: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:47.087 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:46 smithi022 ceph-mon[40250]: pgmap v2799: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:46 smithi122 ceph-mon[32255]: pgmap v2799: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:47.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:47.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:47.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:31:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3075700197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3075700197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3075700197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:48.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:31:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:31:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:48 smithi122 ceph-mon[32255]: pgmap v2800: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:48 smithi022 ceph-mon[31360]: pgmap v2800: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:48 smithi022 ceph-mon[40250]: pgmap v2800: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:50] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:31:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:50 smithi122 ceph-mon[32255]: pgmap v2801: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:50 smithi022 ceph-mon[31360]: pgmap v2801: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:50 smithi022 ceph-mon[40250]: pgmap v2801: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:52.453 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:52 smithi122 ceph-mon[32255]: pgmap v2802: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:31:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:31:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:52 smithi022 ceph-mon[31360]: pgmap v2802: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:52 smithi022 ceph-mon[40250]: pgmap v2802: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:52.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:52.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:52.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3164521782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3164521782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3164521782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:54 smithi122 ceph-mon[32255]: pgmap v2803: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:54 smithi022 ceph-mon[31360]: pgmap v2803: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:54 smithi022 ceph-mon[40250]: pgmap v2803: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:31:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:56 smithi122 ceph-mon[32255]: pgmap v2804: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:56 smithi022 ceph-mon[31360]: pgmap v2804: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:56 smithi022 ceph-mon[40250]: pgmap v2804: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:31:57.826 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:31:58.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:31:58.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:31:58.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:31:58.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:31:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:31:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:58 smithi122 ceph-mon[32255]: pgmap v2805: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2691968680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:31:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:58 smithi022 ceph-mon[31360]: pgmap v2805: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2691968680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:31:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:31:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:58 smithi022 ceph-mon[40250]: pgmap v2805: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:31:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2691968680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:31:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:31:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:00] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:32:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:00 smithi122 ceph-mon[32255]: pgmap v2806: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:00 smithi022 ceph-mon[31360]: pgmap v2806: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:00 smithi022 ceph-mon[40250]: pgmap v2806: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:02 smithi122 ceph-mon[32255]: pgmap v2807: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:02.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:32:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:32:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:02 smithi022 ceph-mon[31360]: pgmap v2807: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:02 smithi022 ceph-mon[40250]: pgmap v2807: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:03.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:03.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:03.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1246378666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1246378666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1246378666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:04 smithi122 ceph-mon[32255]: pgmap v2808: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:04 smithi022 ceph-mon[31360]: pgmap v2808: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:04 smithi022 ceph-mon[40250]: pgmap v2808: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:06 smithi122 ceph-mon[32255]: pgmap v2809: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:06 smithi022 ceph-mon[31360]: pgmap v2809: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:06 smithi022 ceph-mon[40250]: pgmap v2809: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:32:08.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:08.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:32:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:32:08.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:08.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:08.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:08.938 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:08 smithi022 ceph-mon[40250]: pgmap v2810: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:08.938 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:08.938 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:08 smithi022 ceph-mon[31360]: pgmap v2810: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:08.939 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:08 smithi122 ceph-mon[32255]: pgmap v2810: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4173865121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4173865121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4173865121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:10] "GET /metrics HTTP/1.1" 200 202302 "" "Prometheus/2.18.1" 2021-12-15T19:32:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:10 smithi122 ceph-mon[32255]: pgmap v2811: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:10 smithi022 ceph-mon[31360]: pgmap v2811: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:10 smithi022 ceph-mon[40250]: pgmap v2811: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:12.565 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:32:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:32:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:12 smithi122 ceph-mon[32255]: pgmap v2812: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:12 smithi022 ceph-mon[31360]: pgmap v2812: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:12 smithi022 ceph-mon[40250]: pgmap v2812: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:13.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:14.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:14.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:14.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:14 smithi122 ceph-mon[32255]: pgmap v2813: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3761355674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:14 smithi022 ceph-mon[31360]: pgmap v2813: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3761355674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:14 smithi022 ceph-mon[40250]: pgmap v2813: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3761355674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:16 smithi122 ceph-mon[32255]: pgmap v2814: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:16 smithi022 ceph-mon[31360]: pgmap v2814: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:16 smithi022 ceph-mon[40250]: pgmap v2814: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:32:18.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:32:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:32:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:18 smithi122 ceph-mon[32255]: pgmap v2815: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:18 smithi022 ceph-mon[31360]: pgmap v2815: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:18 smithi022 ceph-mon[40250]: pgmap v2815: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:19.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:19.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:19.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:19.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3059433157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3059433157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3059433157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:20] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:32:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:20 smithi122 ceph-mon[32255]: pgmap v2816: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:20 smithi022 ceph-mon[31360]: pgmap v2816: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:20 smithi022 ceph-mon[40250]: pgmap v2816: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:22.571 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:32:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:32:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:22 smithi122 ceph-mon[32255]: pgmap v2817: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:22 smithi022 ceph-mon[31360]: pgmap v2817: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:22 smithi022 ceph-mon[40250]: pgmap v2817: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:32:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:32:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:32:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:32:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:32:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:32:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:32:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:32:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:32:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:32:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:32:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:32:24.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:24 smithi122 ceph-mon[32255]: pgmap v2818: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:24 smithi022 ceph-mon[31360]: pgmap v2818: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:24 smithi022 ceph-mon[40250]: pgmap v2818: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:25.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:25.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:25.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1844838681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1844838681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1844838681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:26 smithi122 ceph-mon[32255]: pgmap v2819: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:26 smithi022 ceph-mon[31360]: pgmap v2819: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:26 smithi022 ceph-mon[40250]: pgmap v2819: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:32:28.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:32:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:32:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:28 smithi122 ceph-mon[32255]: pgmap v2820: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:28 smithi022 ceph-mon[31360]: pgmap v2820: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:28 smithi022 ceph-mon[40250]: pgmap v2820: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:30.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:30.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:30.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:30.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:30] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:32:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:30 smithi122 ceph-mon[32255]: pgmap v2821: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3821043593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:30 smithi022 ceph-mon[31360]: pgmap v2821: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3821043593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:30 smithi022 ceph-mon[40250]: pgmap v2821: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3821043593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:32.572 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:32:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:32:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:32 smithi122 ceph-mon[32255]: pgmap v2822: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:32:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:32:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:32:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:32:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[40250]: pgmap v2822: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:32:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:32:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:32:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:32:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[31360]: pgmap v2822: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:32:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:32:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:32:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:32:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:34 smithi122 ceph-mon[32255]: pgmap v2823: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:34 smithi022 ceph-mon[31360]: pgmap v2823: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:34 smithi022 ceph-mon[40250]: pgmap v2823: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:35.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:35.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:35.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:35.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1168914960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1168914960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1168914960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:36 smithi122 ceph-mon[32255]: pgmap v2824: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:36 smithi022 ceph-mon[31360]: pgmap v2824: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:36 smithi022 ceph-mon[40250]: pgmap v2824: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:32:38.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:32:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:32:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:38 smithi122 ceph-mon[32255]: pgmap v2825: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:38 smithi022 ceph-mon[31360]: pgmap v2825: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:38 smithi022 ceph-mon[40250]: pgmap v2825: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:40] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:32:40.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:41.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:41.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:41.159 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:40 smithi022 ceph-mon[40250]: pgmap v2826: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:41.160 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:40 smithi022 ceph-mon[31360]: pgmap v2826: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:40 smithi122 ceph-mon[32255]: pgmap v2826: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3331010926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3331010926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3331010926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:32:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:32:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:42 smithi122 ceph-mon[32255]: pgmap v2827: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:42 smithi022 ceph-mon[31360]: pgmap v2827: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:42 smithi022 ceph-mon[40250]: pgmap v2827: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:44 smithi122 ceph-mon[32255]: pgmap v2828: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:44 smithi022 ceph-mon[31360]: pgmap v2828: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:44 smithi022 ceph-mon[40250]: pgmap v2828: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:46.160 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:46.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:46.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:46 smithi122 ceph-mon[32255]: pgmap v2829: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/112533603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:46 smithi022 ceph-mon[31360]: pgmap v2829: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/112533603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:46 smithi022 ceph-mon[40250]: pgmap v2829: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/112533603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:32:48.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:32:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:32:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:48 smithi122 ceph-mon[32255]: pgmap v2830: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:48 smithi022 ceph-mon[31360]: pgmap v2830: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:48 smithi022 ceph-mon[40250]: pgmap v2830: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:50] "GET /metrics HTTP/1.1" 200 202306 "" "Prometheus/2.18.1" 2021-12-15T19:32:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:50 smithi122 ceph-mon[32255]: pgmap v2831: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:50 smithi022 ceph-mon[31360]: pgmap v2831: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:50 smithi022 ceph-mon[40250]: pgmap v2831: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:51.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:51.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:51.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:51.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:52.577 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:32:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:32:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:52 smithi122 ceph-mon[32255]: pgmap v2832: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3528325100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:52 smithi022 ceph-mon[31360]: pgmap v2832: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3528325100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:52 smithi022 ceph-mon[40250]: pgmap v2832: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3528325100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:54 smithi122 ceph-mon[32255]: pgmap v2833: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:54 smithi022 ceph-mon[31360]: pgmap v2833: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:54 smithi022 ceph-mon[40250]: pgmap v2833: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:32:56.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:32:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:56 smithi122 ceph-mon[32255]: pgmap v2834: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:56 smithi022 ceph-mon[31360]: pgmap v2834: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:56 smithi022 ceph-mon[40250]: pgmap v2834: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:57.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:32:57.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:32:57.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:32:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:32:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/568791278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/568791278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:58.291 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/568791278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:32:58.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:32:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:32:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:58 smithi122 ceph-mon[32255]: pgmap v2835: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:32:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:58 smithi022 ceph-mon[31360]: pgmap v2835: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:32:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:32:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:58 smithi022 ceph-mon[40250]: pgmap v2835: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:32:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:32:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:00] "GET /metrics HTTP/1.1" 200 202306 "" "Prometheus/2.18.1" 2021-12-15T19:33:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:00 smithi122 ceph-mon[32255]: pgmap v2836: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:00 smithi022 ceph-mon[31360]: pgmap v2836: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:00 smithi022 ceph-mon[40250]: pgmap v2836: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:02.282 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:02.578 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:33:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:33:02.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:02.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:02.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:02 smithi122 ceph-mon[32255]: pgmap v2837: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:02 smithi022 ceph-mon[31360]: pgmap v2837: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:02 smithi022 ceph-mon[40250]: pgmap v2837: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4067731698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4067731698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4067731698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:04 smithi122 ceph-mon[32255]: pgmap v2838: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:04 smithi022 ceph-mon[40250]: pgmap v2838: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:04 smithi022 ceph-mon[31360]: pgmap v2838: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:06 smithi122 ceph-mon[32255]: pgmap v2839: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:06 smithi022 ceph-mon[31360]: pgmap v2839: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:06 smithi022 ceph-mon[40250]: pgmap v2839: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:07.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:07.657 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:33:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:08.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:08.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:08.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:33:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:33:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:08 smithi122 ceph-mon[32255]: pgmap v2840: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:09.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1831908567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:09.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:08 smithi022 ceph-mon[31360]: pgmap v2840: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1831908567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:08 smithi022 ceph-mon[40250]: pgmap v2840: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1831908567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:10] "GET /metrics HTTP/1.1" 200 202307 "" "Prometheus/2.18.1" 2021-12-15T19:33:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:10 smithi122 ceph-mon[32255]: pgmap v2841: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:10 smithi022 ceph-mon[31360]: pgmap v2841: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:10 smithi022 ceph-mon[40250]: pgmap v2841: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:12.582 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:33:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:33:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:12 smithi122 ceph-mon[32255]: pgmap v2842: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:12 smithi022 ceph-mon[31360]: pgmap v2842: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:12 smithi022 ceph-mon[40250]: pgmap v2842: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:13.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:13.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:13.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3285781607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3285781607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3285781607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:14 smithi122 ceph-mon[32255]: pgmap v2843: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:14 smithi022 ceph-mon[31360]: pgmap v2843: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:14 smithi022 ceph-mon[40250]: pgmap v2843: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:16 smithi122 ceph-mon[32255]: pgmap v2844: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:16 smithi022 ceph-mon[31360]: pgmap v2844: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:16 smithi022 ceph-mon[40250]: pgmap v2844: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:33:18.405 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:18.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:33:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:33:18.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:18.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:18.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:18 smithi022 ceph-mon[31360]: pgmap v2845: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1287570147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:18 smithi022 ceph-mon[40250]: pgmap v2845: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1287570147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:18 smithi122 ceph-mon[32255]: pgmap v2845: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1287570147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:20] "GET /metrics HTTP/1.1" 200 202308 "" "Prometheus/2.18.1" 2021-12-15T19:33:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:20 smithi122 ceph-mon[32255]: pgmap v2846: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:20 smithi022 ceph-mon[31360]: pgmap v2846: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:20 smithi022 ceph-mon[40250]: pgmap v2846: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:22.584 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:33:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:33:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:22 smithi122 ceph-mon[32255]: pgmap v2847: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:22 smithi022 ceph-mon[31360]: pgmap v2847: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:22 smithi022 ceph-mon[40250]: pgmap v2847: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:23.776 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:33:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:33:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:33:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:33:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:33:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:33:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:33:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:33:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:33:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:33:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:33:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:33:24.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:24.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:24.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:24 smithi122 ceph-mon[32255]: pgmap v2848: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/562955147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:24 smithi022 ceph-mon[31360]: pgmap v2848: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/562955147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:24 smithi022 ceph-mon[40250]: pgmap v2848: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/562955147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:26 smithi122 ceph-mon[32255]: pgmap v2849: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:26 smithi022 ceph-mon[31360]: pgmap v2849: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:26 smithi022 ceph-mon[40250]: pgmap v2849: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:33:28.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:33:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:33:29.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:29.136 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:28 smithi022 ceph-mon[31360]: pgmap v2850: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:29.136 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:29.137 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:28 smithi022 ceph-mon[40250]: pgmap v2850: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:29.137 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:28 smithi122 ceph-mon[32255]: pgmap v2850: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:29.253 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:29.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:29.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:29.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1666781918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1666781918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1666781918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:30] "GET /metrics HTTP/1.1" 200 202308 "" "Prometheus/2.18.1" 2021-12-15T19:33:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:30 smithi122 ceph-mon[32255]: pgmap v2851: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:30 smithi022 ceph-mon[31360]: pgmap v2851: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:30 smithi022 ceph-mon[40250]: pgmap v2851: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:32.586 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:33:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:33:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:32 smithi122 ceph-mon[32255]: pgmap v2852: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:33:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:33:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:33:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:33:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[31360]: pgmap v2852: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:33:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:33:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:33:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:33:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[40250]: pgmap v2852: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:33:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:33:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:33:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:33:34.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:34.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:34.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:34.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:34 smithi122 ceph-mon[32255]: pgmap v2853: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:34 smithi022 ceph-mon[31360]: pgmap v2853: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:34 smithi022 ceph-mon[40250]: pgmap v2853: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1881867923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1881867923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1881867923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:36 smithi122 ceph-mon[32255]: pgmap v2854: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:36 smithi022 ceph-mon[31360]: pgmap v2854: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:36 smithi022 ceph-mon[40250]: pgmap v2854: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:33:38.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:33:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:33:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:38 smithi122 ceph-mon[32255]: pgmap v2855: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:38 smithi022 ceph-mon[31360]: pgmap v2855: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:38 smithi022 ceph-mon[40250]: pgmap v2855: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:39.874 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:40.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:40.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:40.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:40] "GET /metrics HTTP/1.1" 200 202308 "" "Prometheus/2.18.1" 2021-12-15T19:33:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:40 smithi122 ceph-mon[32255]: pgmap v2856: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1765122115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:40 smithi022 ceph-mon[31360]: pgmap v2856: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1765122115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:40 smithi022 ceph-mon[40250]: pgmap v2856: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1765122115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:42.590 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:33:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:33:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:42 smithi122 ceph-mon[32255]: pgmap v2857: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:42 smithi022 ceph-mon[31360]: pgmap v2857: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:42 smithi022 ceph-mon[40250]: pgmap v2857: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:44 smithi122 ceph-mon[32255]: pgmap v2858: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:45.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:45.242 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:44 smithi022 ceph-mon[31360]: pgmap v2858: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:45.243 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:44 smithi022 ceph-mon[40250]: pgmap v2858: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:45.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:45.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:45.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/57201960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/57201960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/57201960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:46 smithi122 ceph-mon[32255]: pgmap v2859: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:46 smithi022 ceph-mon[31360]: pgmap v2859: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:46 smithi022 ceph-mon[40250]: pgmap v2859: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:33:48.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:33:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:33:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:48 smithi122 ceph-mon[32255]: pgmap v2860: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:48 smithi022 ceph-mon[31360]: pgmap v2860: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:48 smithi022 ceph-mon[40250]: pgmap v2860: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:50] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:33:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:50.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:50.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:50.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:50.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:50 smithi022 ceph-mon[31360]: pgmap v2861: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:50.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:50 smithi022 ceph-mon[40250]: pgmap v2861: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:50 smithi122 ceph-mon[32255]: pgmap v2861: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3567675772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3567675772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3567675772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:33:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:33:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:52 smithi122 ceph-mon[32255]: pgmap v2862: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:52 smithi022 ceph-mon[31360]: pgmap v2862: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:52 smithi022 ceph-mon[40250]: pgmap v2862: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:54 smithi122 ceph-mon[32255]: pgmap v2863: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:54 smithi022 ceph-mon[31360]: pgmap v2863: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:54 smithi022 ceph-mon[40250]: pgmap v2863: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:33:55.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:33:56.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:33:56.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:33:56.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:33:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:56 smithi122 ceph-mon[32255]: pgmap v2864: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3511616200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:56 smithi022 ceph-mon[31360]: pgmap v2864: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3511616200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:56 smithi022 ceph-mon[40250]: pgmap v2864: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3511616200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:33:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:33:58.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:33:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:33:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:58 smithi122 ceph-mon[32255]: pgmap v2865: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:33:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:58 smithi022 ceph-mon[31360]: pgmap v2865: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:33:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:33:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:58 smithi022 ceph-mon[40250]: pgmap v2865: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:33:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:33:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:00] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:34:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:00 smithi122 ceph-mon[32255]: pgmap v2866: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:00 smithi022 ceph-mon[31360]: pgmap v2866: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:00 smithi022 ceph-mon[40250]: pgmap v2866: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:01.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:01.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:01.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:01.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:02.592 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:34:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:34:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3080209920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:02 smithi122 ceph-mon[32255]: pgmap v2867: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3080209920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:02 smithi022 ceph-mon[31360]: pgmap v2867: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3080209920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:02 smithi022 ceph-mon[40250]: pgmap v2867: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:04 smithi122 ceph-mon[32255]: pgmap v2868: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:04 smithi022 ceph-mon[31360]: pgmap v2868: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:04 smithi022 ceph-mon[40250]: pgmap v2868: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:06.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:06 smithi022 ceph-mon[31360]: pgmap v2869: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:06 smithi022 ceph-mon[40250]: pgmap v2869: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:07.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:07.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:07.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:06 smithi122 ceph-mon[32255]: pgmap v2869: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:34:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1880038476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1880038476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1880038476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:08.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:34:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:34:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:08 smithi122 ceph-mon[32255]: pgmap v2870: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:08 smithi022 ceph-mon[31360]: pgmap v2870: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:08 smithi022 ceph-mon[40250]: pgmap v2870: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:10] "GET /metrics HTTP/1.1" 200 202307 "" "Prometheus/2.18.1" 2021-12-15T19:34:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:10 smithi122 ceph-mon[32255]: pgmap v2871: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:10 smithi022 ceph-mon[31360]: pgmap v2871: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:10 smithi022 ceph-mon[40250]: pgmap v2871: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:12.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:12.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:12.593 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:34:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:34:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:12 smithi022 ceph-mon[31360]: pgmap v2872: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/364708617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:12 smithi022 ceph-mon[40250]: pgmap v2872: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/364708617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:12 smithi122 ceph-mon[32255]: pgmap v2872: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/364708617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:14 smithi122 ceph-mon[32255]: pgmap v2873: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:14 smithi022 ceph-mon[31360]: pgmap v2873: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:14 smithi022 ceph-mon[40250]: pgmap v2873: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:16 smithi122 ceph-mon[32255]: pgmap v2874: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:16 smithi022 ceph-mon[31360]: pgmap v2874: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:16 smithi022 ceph-mon[40250]: pgmap v2874: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:17.462 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:34:17.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:17.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:17.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/898612972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/898612972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/898612972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:18.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:34:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:34:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:18 smithi122 ceph-mon[32255]: pgmap v2875: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:18 smithi022 ceph-mon[31360]: pgmap v2875: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:18 smithi022 ceph-mon[40250]: pgmap v2875: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:20] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:34:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:20 smithi122 ceph-mon[32255]: pgmap v2876: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:20 smithi022 ceph-mon[31360]: pgmap v2876: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:20 smithi022 ceph-mon[40250]: pgmap v2876: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:22.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:34:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:34:22.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:22 smithi122 ceph-mon[32255]: pgmap v2877: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:22 smithi022 ceph-mon[31360]: pgmap v2877: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:22 smithi022 ceph-mon[40250]: pgmap v2877: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:23.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:23.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:23.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:23.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/877522855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:23.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:34:23.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:34:23.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:34:23.930 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/877522855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:23.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:34:23.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:34:23.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:34:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/877522855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:34:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:34:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:34:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:24 smithi022 ceph-mon[31360]: pgmap v2878: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:34:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:24 smithi022 ceph-mon[40250]: pgmap v2878: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:34:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:24 smithi122 ceph-mon[32255]: pgmap v2878: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:34:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:26 smithi022 ceph-mon[31360]: pgmap v2879: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:26 smithi022 ceph-mon[40250]: pgmap v2879: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:26 smithi122 ceph-mon[32255]: pgmap v2879: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:34:28.197 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:28.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:28.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:28.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:28.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:34:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:34:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:28 smithi022 ceph-mon[31360]: pgmap v2880: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4027881690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:28 smithi022 ceph-mon[40250]: pgmap v2880: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4027881690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:28 smithi122 ceph-mon[32255]: pgmap v2880: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4027881690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:30] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:34:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:30 smithi022 ceph-mon[31360]: pgmap v2881: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:30 smithi022 ceph-mon[40250]: pgmap v2881: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:30 smithi122 ceph-mon[32255]: pgmap v2881: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:32.599 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:34:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:34:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:32 smithi122 ceph-mon[32255]: pgmap v2882: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:34:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:34:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:34:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:34:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[31360]: pgmap v2882: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:34:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:34:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:34:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:34:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[40250]: pgmap v2882: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:34:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:34:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:34:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:34:33.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:33.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:33.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:33.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:34 smithi122 ceph-mon[32255]: pgmap v2883: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/120209651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:34 smithi022 ceph-mon[31360]: pgmap v2883: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/120209651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:34 smithi022 ceph-mon[40250]: pgmap v2883: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/120209651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:36 smithi122 ceph-mon[32255]: pgmap v2884: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:36 smithi022 ceph-mon[31360]: pgmap v2884: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:36 smithi022 ceph-mon[40250]: pgmap v2884: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:34:38.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:34:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:34:38.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:38 smithi122 ceph-mon[32255]: pgmap v2885: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:38 smithi022 ceph-mon[31360]: pgmap v2885: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:38 smithi022 ceph-mon[40250]: pgmap v2885: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:39.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:39.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:39.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1644751802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1644751802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1644751802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:40] "GET /metrics HTTP/1.1" 200 202300 "" "Prometheus/2.18.1" 2021-12-15T19:34:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:40 smithi122 ceph-mon[32255]: pgmap v2886: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:40 smithi022 ceph-mon[31360]: pgmap v2886: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:40 smithi022 ceph-mon[40250]: pgmap v2886: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:42.601 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:34:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:34:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:42 smithi122 ceph-mon[32255]: pgmap v2887: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:42 smithi022 ceph-mon[31360]: pgmap v2887: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:42 smithi022 ceph-mon[40250]: pgmap v2887: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:44.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:44.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:44.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:44.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:44 smithi022 ceph-mon[31360]: pgmap v2888: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301696632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:44 smithi022 ceph-mon[40250]: pgmap v2888: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301696632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:44 smithi122 ceph-mon[32255]: pgmap v2888: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301696632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:46 smithi122 ceph-mon[32255]: pgmap v2889: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:46 smithi022 ceph-mon[31360]: pgmap v2889: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:46 smithi022 ceph-mon[40250]: pgmap v2889: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:34:48.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:34:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:34:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:48 smithi122 ceph-mon[32255]: pgmap v2890: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:48 smithi022 ceph-mon[31360]: pgmap v2890: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:48 smithi022 ceph-mon[40250]: pgmap v2890: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:49.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:50.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:50.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:50.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:50] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:34:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:50 smithi122 ceph-mon[32255]: pgmap v2891: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/479148892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:50 smithi022 ceph-mon[31360]: pgmap v2891: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/479148892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:50 smithi022 ceph-mon[40250]: pgmap v2891: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/479148892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:52.602 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:34:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:34:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:52 smithi122 ceph-mon[32255]: pgmap v2892: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:52 smithi022 ceph-mon[31360]: pgmap v2892: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:52 smithi022 ceph-mon[40250]: pgmap v2892: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:55.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:34:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:54 smithi122 ceph-mon[32255]: pgmap v2893: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:54 smithi022 ceph-mon[31360]: pgmap v2893: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:54 smithi022 ceph-mon[40250]: pgmap v2893: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:34:55.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:34:55.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:34:55.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:34:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3670528329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3670528329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3670528329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:34:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:56 smithi122 ceph-mon[32255]: pgmap v2894: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:56 smithi022 ceph-mon[31360]: pgmap v2894: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:56 smithi022 ceph-mon[40250]: pgmap v2894: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:34:58.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:34:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:34:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:58 smithi122 ceph-mon[32255]: pgmap v2895: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:34:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:58 smithi022 ceph-mon[31360]: pgmap v2895: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:34:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:34:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:58 smithi022 ceph-mon[40250]: pgmap v2895: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:34:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:34:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:00.430 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:00] "GET /metrics HTTP/1.1" 200 202304 "" "Prometheus/2.18.1" 2021-12-15T19:35:00.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:00.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:00.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:00 smithi122 ceph-mon[32255]: pgmap v2896: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/932111249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:00 smithi022 ceph-mon[31360]: pgmap v2896: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/932111249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:00 smithi022 ceph-mon[40250]: pgmap v2896: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/932111249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:02.605 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:35:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:35:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:02 smithi122 ceph-mon[32255]: pgmap v2897: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:02 smithi022 ceph-mon[31360]: pgmap v2897: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:02 smithi022 ceph-mon[40250]: pgmap v2897: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:04 smithi122 ceph-mon[32255]: pgmap v2898: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:04 smithi022 ceph-mon[31360]: pgmap v2898: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:04 smithi022 ceph-mon[40250]: pgmap v2898: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:05.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:06.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:06.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:06.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:06 smithi122 ceph-mon[32255]: pgmap v2899: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3181335568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:06 smithi022 ceph-mon[31360]: pgmap v2899: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3181335568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:06 smithi022 ceph-mon[40250]: pgmap v2899: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3181335568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:35:08.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:35:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:35:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:08 smithi122 ceph-mon[32255]: pgmap v2900: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:08 smithi022 ceph-mon[31360]: pgmap v2900: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:08 smithi022 ceph-mon[40250]: pgmap v2900: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:10] "GET /metrics HTTP/1.1" 200 202303 "" "Prometheus/2.18.1" 2021-12-15T19:35:11.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:11.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:10 smithi022 ceph-mon[40250]: pgmap v2901: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:11.176 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:10 smithi022 ceph-mon[31360]: pgmap v2901: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:10 smithi122 ceph-mon[32255]: pgmap v2901: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:11.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:11.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:11.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4242485193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4242485193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4242485193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:35:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:35:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:12 smithi122 ceph-mon[32255]: pgmap v2902: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:12 smithi022 ceph-mon[31360]: pgmap v2902: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:12 smithi022 ceph-mon[40250]: pgmap v2902: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:14 smithi122 ceph-mon[32255]: pgmap v2903: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:14 smithi022 ceph-mon[31360]: pgmap v2903: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:14 smithi022 ceph-mon[40250]: pgmap v2903: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:16.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:16.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:16.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:16.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:16 smithi122 ceph-mon[32255]: pgmap v2904: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:16 smithi022 ceph-mon[31360]: pgmap v2904: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:16 smithi022 ceph-mon[40250]: pgmap v2904: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:35:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/654182084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/654182084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/654182084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:18.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:35:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:35:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:18 smithi122 ceph-mon[32255]: pgmap v2905: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:18 smithi022 ceph-mon[31360]: pgmap v2905: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:18 smithi022 ceph-mon[40250]: pgmap v2905: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:20] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:35:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:20 smithi122 ceph-mon[32255]: pgmap v2906: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:20 smithi022 ceph-mon[31360]: pgmap v2906: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:20 smithi022 ceph-mon[40250]: pgmap v2906: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:21.921 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:22.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:22.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:22.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:22.609 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:35:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:35:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:22 smithi122 ceph-mon[32255]: pgmap v2907: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1201266465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:22 smithi022 ceph-mon[31360]: pgmap v2907: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1201266465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:22 smithi022 ceph-mon[40250]: pgmap v2907: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1201266465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:24 smithi122 ceph-mon[32255]: pgmap v2908: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:35:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:35:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:35:25.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:35:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[31360]: pgmap v2908: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:35:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:35:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:35:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:35:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[40250]: pgmap v2908: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:35:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:35:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:35:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:35:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:26 smithi122 ceph-mon[32255]: pgmap v2909: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:26 smithi022 ceph-mon[31360]: pgmap v2909: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:26 smithi022 ceph-mon[40250]: pgmap v2909: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:27.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:27.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:27.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:27.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:27.658 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:35:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2636020640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2636020640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2636020640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:28.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:35:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:35:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:28 smithi122 ceph-mon[32255]: pgmap v2910: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:28 smithi022 ceph-mon[31360]: pgmap v2910: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:28 smithi022 ceph-mon[40250]: pgmap v2910: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:30] "GET /metrics HTTP/1.1" 200 202298 "" "Prometheus/2.18.1" 2021-12-15T19:35:31.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:30 smithi122 ceph-mon[32255]: pgmap v2911: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:30 smithi022 ceph-mon[31360]: pgmap v2911: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:30 smithi022 ceph-mon[40250]: pgmap v2911: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:32.614 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:35:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:35:32.659 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:32 smithi122 ceph-mon[32255]: pgmap v2912: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:35:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:35:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:35:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:35:33.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:33.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:33.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[31360]: pgmap v2912: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:35:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:35:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:35:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:35:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[40250]: pgmap v2912: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:35:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:35:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:35:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:35:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/783470584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/783470584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/783470584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:34 smithi122 ceph-mon[32255]: pgmap v2913: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:34 smithi022 ceph-mon[31360]: pgmap v2913: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:34 smithi022 ceph-mon[40250]: pgmap v2913: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:36 smithi122 ceph-mon[32255]: pgmap v2914: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:36 smithi022 ceph-mon[31360]: pgmap v2914: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:36 smithi022 ceph-mon[40250]: pgmap v2914: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:35:38.029 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:38.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:38.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:38.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:38.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:35:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:35:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:38 smithi122 ceph-mon[32255]: pgmap v2915: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3481234468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:38 smithi022 ceph-mon[31360]: pgmap v2915: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3481234468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:38 smithi022 ceph-mon[40250]: pgmap v2915: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3481234468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:40] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:35:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:40 smithi122 ceph-mon[32255]: pgmap v2916: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:40 smithi022 ceph-mon[31360]: pgmap v2916: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:40 smithi022 ceph-mon[40250]: pgmap v2916: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:42.621 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:35:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:35:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:42 smithi122 ceph-mon[32255]: pgmap v2917: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:42 smithi022 ceph-mon[31360]: pgmap v2917: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:42 smithi022 ceph-mon[40250]: pgmap v2917: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:43.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:43.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:43.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:43.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1199739818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1199739818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1199739818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:44 smithi122 ceph-mon[32255]: pgmap v2918: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:44 smithi022 ceph-mon[31360]: pgmap v2918: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:44 smithi022 ceph-mon[40250]: pgmap v2918: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:46 smithi122 ceph-mon[32255]: pgmap v2919: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:46 smithi022 ceph-mon[31360]: pgmap v2919: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:46 smithi022 ceph-mon[40250]: pgmap v2919: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:35:48.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:35:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:35:48.770 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:48 smithi022 ceph-mon[31360]: pgmap v2920: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:48 smithi022 ceph-mon[40250]: pgmap v2920: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:49.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:49.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:49.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:48 smithi122 ceph-mon[32255]: pgmap v2920: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1255307095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1255307095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1255307095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:50] "GET /metrics HTTP/1.1" 200 202311 "" "Prometheus/2.18.1" 2021-12-15T19:35:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:50 smithi122 ceph-mon[32255]: pgmap v2921: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:50 smithi022 ceph-mon[31360]: pgmap v2921: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:50 smithi022 ceph-mon[40250]: pgmap v2921: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:52.618 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:35:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:35:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:52 smithi122 ceph-mon[32255]: pgmap v2922: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:52 smithi022 ceph-mon[31360]: pgmap v2922: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:52 smithi022 ceph-mon[40250]: pgmap v2922: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:54.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:54.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:54.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:54.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:35:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:54 smithi122 ceph-mon[32255]: pgmap v2923: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3535436879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:54 smithi022 ceph-mon[31360]: pgmap v2923: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3535436879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:54 smithi022 ceph-mon[40250]: pgmap v2923: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:35:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3535436879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:35:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:56 smithi122 ceph-mon[32255]: pgmap v2924: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:56 smithi022 ceph-mon[31360]: pgmap v2924: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:56 smithi022 ceph-mon[40250]: pgmap v2924: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:35:58.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:35:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:35:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:58 smithi122 ceph-mon[32255]: pgmap v2925: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:35:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:58 smithi022 ceph-mon[31360]: pgmap v2925: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:35:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:58 smithi022 ceph-mon[40250]: pgmap v2925: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:35:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:35:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:35:59.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:35:59.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:35:59.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:35:59.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:00] "GET /metrics HTTP/1.1" 200 202311 "" "Prometheus/2.18.1" 2021-12-15T19:36:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:00 smithi122 ceph-mon[32255]: pgmap v2926: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1261710851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:00 smithi022 ceph-mon[31360]: pgmap v2926: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1261710851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:00 smithi022 ceph-mon[40250]: pgmap v2926: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1261710851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:02.621 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:36:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:36:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:02 smithi122 ceph-mon[32255]: pgmap v2927: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:02 smithi022 ceph-mon[31360]: pgmap v2927: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:02 smithi022 ceph-mon[40250]: pgmap v2927: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:04.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:04 smithi122 ceph-mon[32255]: pgmap v2928: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:05.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:05.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:05.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:05.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:04 smithi022 ceph-mon[31360]: pgmap v2928: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:05.255 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:04 smithi022 ceph-mon[40250]: pgmap v2928: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2876783599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2876783599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2876783599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:06 smithi122 ceph-mon[32255]: pgmap v2929: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:06 smithi022 ceph-mon[31360]: pgmap v2929: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:06 smithi022 ceph-mon[40250]: pgmap v2929: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:36:08.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:36:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:36:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:08 smithi122 ceph-mon[32255]: pgmap v2930: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:08 smithi022 ceph-mon[31360]: pgmap v2930: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:08 smithi022 ceph-mon[40250]: pgmap v2930: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:10.255 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:10] "GET /metrics HTTP/1.1" 200 202315 "" "Prometheus/2.18.1" 2021-12-15T19:36:10.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:10.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:10.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:10 smithi022 ceph-mon[31360]: pgmap v2931: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2916407579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:10 smithi022 ceph-mon[40250]: pgmap v2931: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2916407579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:10 smithi122 ceph-mon[32255]: pgmap v2931: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2916407579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:12.625 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:36:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:36:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:12 smithi122 ceph-mon[32255]: pgmap v2932: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:12 smithi022 ceph-mon[31360]: pgmap v2932: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:12 smithi022 ceph-mon[40250]: pgmap v2932: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:14 smithi122 ceph-mon[32255]: pgmap v2933: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:14 smithi022 ceph-mon[31360]: pgmap v2933: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:14 smithi022 ceph-mon[40250]: pgmap v2933: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:15.625 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:15.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:15.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:15.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:16 smithi122 ceph-mon[32255]: pgmap v2934: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/991944359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:16 smithi022 ceph-mon[31360]: pgmap v2934: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/991944359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:16 smithi022 ceph-mon[40250]: pgmap v2934: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/991944359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:36:18.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:36:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:36:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:18 smithi122 ceph-mon[32255]: pgmap v2935: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:18 smithi022 ceph-mon[31360]: pgmap v2935: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:18 smithi022 ceph-mon[40250]: pgmap v2935: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:20] "GET /metrics HTTP/1.1" 200 202314 "" "Prometheus/2.18.1" 2021-12-15T19:36:20.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:20 smithi122 ceph-mon[32255]: pgmap v2936: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:20 smithi022 ceph-mon[31360]: pgmap v2936: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:20 smithi022 ceph-mon[40250]: pgmap v2936: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:21.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:21.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:21.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/957831823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/957831823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/957831823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:36:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:36:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:22 smithi122 ceph-mon[32255]: pgmap v2937: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:22 smithi022 ceph-mon[31360]: pgmap v2937: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:22 smithi022 ceph-mon[40250]: pgmap v2937: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:24 smithi122 ceph-mon[32255]: pgmap v2938: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:25.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:36:25.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:36:25.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:36:25.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:25.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:25.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[31360]: pgmap v2938: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:25.263 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:36:25.263 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:36:25.263 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:36:25.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:25.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:25.264 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[40250]: pgmap v2938: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:25.264 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:36:25.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:36:25.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:36:25.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:25.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:26.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:36:26.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:26.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:26.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:36:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:27 smithi022 ceph-mon[31360]: pgmap v2939: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:27.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1360480194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:27 smithi022 ceph-mon[40250]: pgmap v2939: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1360480194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:27 smithi122 ceph-mon[32255]: pgmap v2939: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1360480194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:28.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:36:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:36:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:29 smithi022 ceph-mon[31360]: pgmap v2940: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:29 smithi022 ceph-mon[40250]: pgmap v2940: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:29 smithi122 ceph-mon[32255]: pgmap v2940: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:30] "GET /metrics HTTP/1.1" 200 202314 "" "Prometheus/2.18.1" 2021-12-15T19:36:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:31 smithi122 ceph-mon[32255]: pgmap v2941: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:31.743 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:31.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:31 smithi022 ceph-mon[40250]: pgmap v2941: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:31.745 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:31 smithi022 ceph-mon[31360]: pgmap v2941: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:32.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:32.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:32.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:32.628 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:36:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:36:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:32 smithi122 ceph-mon[32255]: pgmap v2942: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3995921143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:36:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:36:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:36:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:36:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[31360]: pgmap v2942: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3995921143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:36:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:36:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:36:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:36:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[40250]: pgmap v2942: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3995921143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:36:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:36:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:36:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:36:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:34 smithi122 ceph-mon[32255]: pgmap v2943: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:34 smithi022 ceph-mon[31360]: pgmap v2943: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:34 smithi022 ceph-mon[40250]: pgmap v2943: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:37.110 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:36 smithi122 ceph-mon[32255]: pgmap v2944: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:36 smithi022 ceph-mon[31360]: pgmap v2944: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:36 smithi022 ceph-mon[40250]: pgmap v2944: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:37.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:37.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:37.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:36:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1413885386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1413885386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1413885386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:38.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:36:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:36:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:38 smithi122 ceph-mon[32255]: pgmap v2945: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:38 smithi022 ceph-mon[31360]: pgmap v2945: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:38 smithi022 ceph-mon[40250]: pgmap v2945: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:40] "GET /metrics HTTP/1.1" 200 202313 "" "Prometheus/2.18.1" 2021-12-15T19:36:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:40 smithi122 ceph-mon[32255]: pgmap v2946: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:40 smithi022 ceph-mon[31360]: pgmap v2946: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:40 smithi022 ceph-mon[40250]: pgmap v2946: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:42.485 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:42.629 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:36:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:36:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:42 smithi022 ceph-mon[31360]: pgmap v2947: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:42 smithi022 ceph-mon[40250]: pgmap v2947: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:42.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:42.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:42.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:42 smithi122 ceph-mon[32255]: pgmap v2947: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/766220897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/766220897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/766220897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:44 smithi122 ceph-mon[32255]: pgmap v2948: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:44 smithi022 ceph-mon[31360]: pgmap v2948: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:44 smithi022 ceph-mon[40250]: pgmap v2948: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:46 smithi122 ceph-mon[32255]: pgmap v2949: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:46 smithi022 ceph-mon[31360]: pgmap v2949: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:46 smithi022 ceph-mon[40250]: pgmap v2949: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:36:47.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:48.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:48.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:48.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:48.876 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:36:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:36:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:48 smithi122 ceph-mon[32255]: pgmap v2950: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/426106287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:48 smithi022 ceph-mon[31360]: pgmap v2950: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/426106287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:48 smithi022 ceph-mon[40250]: pgmap v2950: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/426106287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:50] "GET /metrics HTTP/1.1" 200 202306 "" "Prometheus/2.18.1" 2021-12-15T19:36:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:50 smithi122 ceph-mon[32255]: pgmap v2951: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:50 smithi022 ceph-mon[31360]: pgmap v2951: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:50 smithi022 ceph-mon[40250]: pgmap v2951: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:52.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:36:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:36:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:52 smithi122 ceph-mon[32255]: pgmap v2952: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:52 smithi022 ceph-mon[31360]: pgmap v2952: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:52 smithi022 ceph-mon[40250]: pgmap v2952: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:53.228 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:53.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:53.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:53.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4166731360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4166731360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4166731360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:36:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:54 smithi122 ceph-mon[32255]: pgmap v2953: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:54 smithi022 ceph-mon[31360]: pgmap v2953: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:54 smithi022 ceph-mon[40250]: pgmap v2953: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:36:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:56 smithi122 ceph-mon[32255]: pgmap v2954: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:56 smithi022 ceph-mon[31360]: pgmap v2954: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:56 smithi022 ceph-mon[40250]: pgmap v2954: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:36:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:36:58.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:36:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:36:58.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:36:58.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:36:58.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:36:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:58 smithi122 ceph-mon[32255]: pgmap v2955: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:58 smithi022 ceph-mon[31360]: pgmap v2955: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:36:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:58 smithi022 ceph-mon[40250]: pgmap v2955: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:36:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:36:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2144870205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:36:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2144870205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:36:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2144870205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:00] "GET /metrics HTTP/1.1" 200 202306 "" "Prometheus/2.18.1" 2021-12-15T19:37:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:00 smithi122 ceph-mon[32255]: pgmap v2956: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:00 smithi022 ceph-mon[31360]: pgmap v2956: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:00 smithi022 ceph-mon[40250]: pgmap v2956: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:02.634 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:37:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:37:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:02 smithi122 ceph-mon[32255]: pgmap v2957: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:02 smithi022 ceph-mon[31360]: pgmap v2957: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:02 smithi022 ceph-mon[40250]: pgmap v2957: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:04.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:04.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:04.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:04 smithi122 ceph-mon[32255]: pgmap v2958: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2619905988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:04 smithi022 ceph-mon[31360]: pgmap v2958: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2619905988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:04 smithi022 ceph-mon[40250]: pgmap v2958: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2619905988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:06 smithi122 ceph-mon[32255]: pgmap v2959: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:06 smithi022 ceph-mon[31360]: pgmap v2959: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:06 smithi022 ceph-mon[40250]: pgmap v2959: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:37:08.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:37:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:37:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:08 smithi122 ceph-mon[32255]: pgmap v2960: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:08 smithi022 ceph-mon[31360]: pgmap v2960: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:08 smithi022 ceph-mon[40250]: pgmap v2960: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:09.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:09.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:09.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:09.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4057334210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4057334210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4057334210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:10] "GET /metrics HTTP/1.1" 200 202309 "" "Prometheus/2.18.1" 2021-12-15T19:37:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:10 smithi122 ceph-mon[32255]: pgmap v2961: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:10 smithi022 ceph-mon[31360]: pgmap v2961: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:10 smithi022 ceph-mon[40250]: pgmap v2961: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:12.635 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:37:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:37:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:12 smithi122 ceph-mon[32255]: pgmap v2962: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:12 smithi022 ceph-mon[31360]: pgmap v2962: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:12 smithi022 ceph-mon[40250]: pgmap v2962: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:14.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:14 smithi022 ceph-mon[31360]: pgmap v2963: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:14 smithi022 ceph-mon[40250]: pgmap v2963: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:15.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:15.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:15.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:14 smithi122 ceph-mon[32255]: pgmap v2963: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/918389613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/918389613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/918389613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:16 smithi122 ceph-mon[32255]: pgmap v2964: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:16 smithi022 ceph-mon[31360]: pgmap v2964: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:16 smithi022 ceph-mon[40250]: pgmap v2964: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:37:18.890 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:37:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:37:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:18 smithi122 ceph-mon[32255]: pgmap v2965: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:18 smithi022 ceph-mon[31360]: pgmap v2965: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:18 smithi022 ceph-mon[40250]: pgmap v2965: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:20.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:20] "GET /metrics HTTP/1.1" 200 202310 "" "Prometheus/2.18.1" 2021-12-15T19:37:20.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:20.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:20.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:20 smithi122 ceph-mon[32255]: pgmap v2966: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3587355178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:20 smithi022 ceph-mon[31360]: pgmap v2966: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3587355178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:20 smithi022 ceph-mon[40250]: pgmap v2966: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3587355178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:22.640 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:37:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:37:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:22 smithi122 ceph-mon[32255]: pgmap v2967: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:22 smithi022 ceph-mon[31360]: pgmap v2967: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:22 smithi022 ceph-mon[40250]: pgmap v2967: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:24 smithi122 ceph-mon[32255]: pgmap v2968: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:24 smithi022 ceph-mon[31360]: pgmap v2968: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:24 smithi022 ceph-mon[40250]: pgmap v2968: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:25.459 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:25.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:25.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:25.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:37:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:37:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:37:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:37:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3234112224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:37:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:37:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:37:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:37:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3234112224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:37:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:37:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:37:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:37:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3234112224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:26 smithi122 ceph-mon[32255]: pgmap v2969: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:26 smithi022 ceph-mon[31360]: pgmap v2969: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:26 smithi022 ceph-mon[40250]: pgmap v2969: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:37:28.890 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:37:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:37:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:28 smithi122 ceph-mon[32255]: pgmap v2970: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:28 smithi022 ceph-mon[31360]: pgmap v2970: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:28 smithi022 ceph-mon[40250]: pgmap v2970: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:30] "GET /metrics HTTP/1.1" 200 202310 "" "Prometheus/2.18.1" 2021-12-15T19:37:30.834 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:30 smithi122 ceph-mon[32255]: pgmap v2971: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:31.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:31.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:31.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:31.205 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:30 smithi022 ceph-mon[31360]: pgmap v2971: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:31.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:30 smithi022 ceph-mon[40250]: pgmap v2971: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4119875330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4119875330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4119875330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:37:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:37:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:32 smithi122 ceph-mon[32255]: pgmap v2972: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:37:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:37:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:37:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:37:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[31360]: pgmap v2972: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:37:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:37:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:37:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:37:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[40250]: pgmap v2972: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:37:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:37:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:37:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:37:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:34 smithi122 ceph-mon[32255]: pgmap v2973: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:34 smithi022 ceph-mon[31360]: pgmap v2973: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:34 smithi022 ceph-mon[40250]: pgmap v2973: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:36.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:36.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:36.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:36.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:36 smithi122 ceph-mon[32255]: pgmap v2974: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2366525938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:36 smithi022 ceph-mon[31360]: pgmap v2974: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2366525938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:36 smithi022 ceph-mon[40250]: pgmap v2974: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2366525938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:37:38.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:37:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:37:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:38 smithi122 ceph-mon[32255]: pgmap v2975: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:38 smithi022 ceph-mon[31360]: pgmap v2975: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:38 smithi022 ceph-mon[40250]: pgmap v2975: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:40] "GET /metrics HTTP/1.1" 200 202308 "" "Prometheus/2.18.1" 2021-12-15T19:37:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:40 smithi122 ceph-mon[32255]: pgmap v2976: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:40 smithi022 ceph-mon[31360]: pgmap v2976: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:40 smithi022 ceph-mon[40250]: pgmap v2976: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:41.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:41.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:41.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:41.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:42.643 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:37:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:37:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:42 smithi122 ceph-mon[32255]: pgmap v2977: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/852213377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:42 smithi022 ceph-mon[31360]: pgmap v2977: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/852213377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:42 smithi022 ceph-mon[40250]: pgmap v2977: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/852213377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:44 smithi122 ceph-mon[32255]: pgmap v2978: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:44 smithi022 ceph-mon[31360]: pgmap v2978: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:44 smithi022 ceph-mon[40250]: pgmap v2978: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:46.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:46 smithi122 ceph-mon[32255]: pgmap v2979: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:46 smithi022 ceph-mon[31360]: pgmap v2979: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:46 smithi022 ceph-mon[40250]: pgmap v2979: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:47.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:47.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:47.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:37:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/944327502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/944327502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/944327502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:48.900 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:37:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:37:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:48 smithi122 ceph-mon[32255]: pgmap v2980: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:48 smithi022 ceph-mon[31360]: pgmap v2980: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:48 smithi022 ceph-mon[40250]: pgmap v2980: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:50] "GET /metrics HTTP/1.1" 200 202315 "" "Prometheus/2.18.1" 2021-12-15T19:37:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:50 smithi122 ceph-mon[32255]: pgmap v2981: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:50 smithi022 ceph-mon[31360]: pgmap v2981: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:50 smithi022 ceph-mon[40250]: pgmap v2981: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:37:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:52.644 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:37:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:37:52.645 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:52 smithi022 ceph-mon[40250]: pgmap v2982: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:52.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:52.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:52.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:52 smithi122 ceph-mon[32255]: pgmap v2982: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:53.089 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:52 smithi022 ceph-mon[31360]: pgmap v2982: 132 pgs: 132 active+clean; 582 KiB data, 124 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:37:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2445243778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2445243778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2445243778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:54 smithi122 ceph-mon[32255]: pgmap v2983: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:54 smithi022 ceph-mon[31360]: pgmap v2983: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:54 smithi022 ceph-mon[40250]: pgmap v2983: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:56 smithi122 ceph-mon[32255]: pgmap v2984: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:56 smithi022 ceph-mon[31360]: pgmap v2984: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:56 smithi022 ceph-mon[40250]: pgmap v2984: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:57.695 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:37:57.696 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:37:58.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:37:58.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:37:58.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:37:58.897 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:37:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:37:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:58 smithi122 ceph-mon[32255]: pgmap v2985: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3489088733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:37:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:58 smithi022 ceph-mon[31360]: pgmap v2985: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3489088733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:37:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:37:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:58 smithi022 ceph-mon[40250]: pgmap v2985: 132 pgs: 132 active+clean; 582 KiB data, 128 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:37:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3489088733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:37:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:37:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:00] "GET /metrics HTTP/1.1" 200 202315 "" "Prometheus/2.18.1" 2021-12-15T19:38:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:00 smithi122 ceph-mon[32255]: pgmap v2986: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:00 smithi022 ceph-mon[31360]: pgmap v2986: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:00 smithi022 ceph-mon[40250]: pgmap v2986: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:02.647 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:38:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:38:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:02 smithi122 ceph-mon[32255]: pgmap v2987: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:02 smithi022 ceph-mon[31360]: pgmap v2987: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:02 smithi022 ceph-mon[40250]: pgmap v2987: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:03.066 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:03.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:03.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:03.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/617119945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/617119945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/617119945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:04 smithi122 ceph-mon[32255]: pgmap v2988: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:04 smithi022 ceph-mon[31360]: pgmap v2988: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:04 smithi022 ceph-mon[40250]: pgmap v2988: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:38:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:06 smithi122 ceph-mon[32255]: pgmap v2989: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:38:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:06 smithi022 ceph-mon[31360]: pgmap v2989: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:38:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:06 smithi022 ceph-mon[40250]: pgmap v2989: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:38:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:38:08.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:08.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:08.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:08.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:08.900 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:38:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:38:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:08 smithi122 ceph-mon[32255]: pgmap v2990: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:38:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/434659284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:08 smithi022 ceph-mon[31360]: pgmap v2990: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:38:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/434659284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:08 smithi022 ceph-mon[40250]: pgmap v2990: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:38:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/434659284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:10] "GET /metrics HTTP/1.1" 200 202310 "" "Prometheus/2.18.1" 2021-12-15T19:38:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:10 smithi122 ceph-mon[32255]: pgmap v2991: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:38:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:10 smithi022 ceph-mon[31360]: pgmap v2991: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:38:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:10 smithi022 ceph-mon[40250]: pgmap v2991: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:38:12.648 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:38:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:38:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:12 smithi122 ceph-mon[32255]: pgmap v2992: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:12 smithi022 ceph-mon[31360]: pgmap v2992: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:12 smithi022 ceph-mon[40250]: pgmap v2992: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:13.812 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:14.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:14.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:14.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:14 smithi122 ceph-mon[32255]: pgmap v2993: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1413124242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:14 smithi022 ceph-mon[31360]: pgmap v2993: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1413124242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:14 smithi022 ceph-mon[40250]: pgmap v2993: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1413124242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:16 smithi122 ceph-mon[32255]: pgmap v2994: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:16 smithi022 ceph-mon[31360]: pgmap v2994: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:16 smithi022 ceph-mon[40250]: pgmap v2994: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:38:18.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:38:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:38:19.184 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:19.185 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:18 smithi022 ceph-mon[31360]: pgmap v2995: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:19.185 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:19.185 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:18 smithi022 ceph-mon[40250]: pgmap v2995: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:19.186 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:18 smithi122 ceph-mon[32255]: pgmap v2995: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:19.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:19.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:19.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:19.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/792136869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/792136869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/792136869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:20] "GET /metrics HTTP/1.1" 200 202319 "" "Prometheus/2.18.1" 2021-12-15T19:38:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:20 smithi122 ceph-mon[32255]: pgmap v2996: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:20 smithi022 ceph-mon[31360]: pgmap v2996: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:20 smithi022 ceph-mon[40250]: pgmap v2996: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:22.652 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:38:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:38:22.653 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:22 smithi122 ceph-mon[32255]: pgmap v2997: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:22 smithi022 ceph-mon[31360]: pgmap v2997: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:22 smithi022 ceph-mon[40250]: pgmap v2997: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:24.552 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:24.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:24 smithi022 ceph-mon[31360]: pgmap v2998: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:24.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:24.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:24.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:24 smithi122 ceph-mon[32255]: pgmap v2998: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:24 smithi022 ceph-mon[40250]: pgmap v2998: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1419947543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:38:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:38:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:38:26.201 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1419947543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:26.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:38:26.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:38:26.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:38:26.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1419947543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:26.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:38:26.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:38:26.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:38:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:27 smithi022 ceph-mon[31360]: pgmap v2999: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:38:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:38:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:27 smithi022 ceph-mon[40250]: pgmap v2999: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:38:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:27 smithi122 ceph-mon[32255]: pgmap v2999: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:38:28.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:38:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:38:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:29 smithi022 ceph-mon[31360]: pgmap v3000: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:29 smithi022 ceph-mon[40250]: pgmap v3000: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:29 smithi122 ceph-mon[32255]: pgmap v3000: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:29.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:30.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:30.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:30.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:30] "GET /metrics HTTP/1.1" 200 202319 "" "Prometheus/2.18.1" 2021-12-15T19:38:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:31 smithi022 ceph-mon[31360]: pgmap v3001: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1369267785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:31 smithi022 ceph-mon[40250]: pgmap v3001: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1369267785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:31 smithi122 ceph-mon[32255]: pgmap v3001: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1369267785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:32.655 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:38:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:38:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:32 smithi122 ceph-mon[32255]: pgmap v3002: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:38:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:38:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:38:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:38:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[31360]: pgmap v3002: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:38:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:38:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:38:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:38:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[40250]: pgmap v3002: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:38:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:38:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:38:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:38:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:34 smithi122 ceph-mon[32255]: pgmap v3003: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:34 smithi022 ceph-mon[31360]: pgmap v3003: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:34 smithi022 ceph-mon[40250]: pgmap v3003: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:35.294 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:35.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:35.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:35.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1810873982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1810873982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1810873982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:36 smithi122 ceph-mon[32255]: pgmap v3004: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:36 smithi022 ceph-mon[31360]: pgmap v3004: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:36 smithi022 ceph-mon[40250]: pgmap v3004: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:38:38.915 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:38:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:38:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:38 smithi122 ceph-mon[32255]: pgmap v3005: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:38 smithi022 ceph-mon[31360]: pgmap v3005: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:38 smithi022 ceph-mon[40250]: pgmap v3005: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:39.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:40] "GET /metrics HTTP/1.1" 200 202313 "" "Prometheus/2.18.1" 2021-12-15T19:38:40.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:41.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:41.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:41.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:40 smithi122 ceph-mon[32255]: pgmap v3006: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:40 smithi022 ceph-mon[31360]: pgmap v3006: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:40 smithi022 ceph-mon[40250]: pgmap v3006: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2066758288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2066758288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2066758288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:38:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:38:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:42 smithi122 ceph-mon[32255]: pgmap v3007: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:42 smithi022 ceph-mon[31360]: pgmap v3007: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:42 smithi022 ceph-mon[40250]: pgmap v3007: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:44 smithi122 ceph-mon[32255]: pgmap v3008: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:44 smithi022 ceph-mon[31360]: pgmap v3008: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:44 smithi022 ceph-mon[40250]: pgmap v3008: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:46.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:46.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:46.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:46.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:46 smithi122 ceph-mon[32255]: pgmap v3009: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2406167203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:46 smithi022 ceph-mon[40250]: pgmap v3009: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2406167203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:46 smithi022 ceph-mon[31360]: pgmap v3009: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2406167203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:38:48.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:38:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:38:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:48 smithi122 ceph-mon[32255]: pgmap v3010: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:48 smithi022 ceph-mon[40250]: pgmap v3010: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:48 smithi022 ceph-mon[31360]: pgmap v3010: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:38:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:50] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:38:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:50 smithi122 ceph-mon[32255]: pgmap v3011: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:50 smithi022 ceph-mon[40250]: pgmap v3011: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:50 smithi022 ceph-mon[31360]: pgmap v3011: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:51.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:51.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:51.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:51.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:52.657 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:38:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:38:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4184548321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:52 smithi122 ceph-mon[32255]: pgmap v3012: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4184548321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:52 smithi022 ceph-mon[31360]: pgmap v3012: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4184548321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:52 smithi022 ceph-mon[40250]: pgmap v3012: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:38:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:54 smithi122 ceph-mon[32255]: pgmap v3013: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:54 smithi022 ceph-mon[31360]: pgmap v3013: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:54 smithi022 ceph-mon[40250]: pgmap v3013: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:38:56.778 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:38:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:56 smithi022 ceph-mon[31360]: pgmap v3014: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:38:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:56 smithi022 ceph-mon[40250]: pgmap v3014: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:38:57.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:38:57.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:38:57.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:38:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:56 smithi122 ceph-mon[32255]: pgmap v3014: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:38:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:38:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/825577030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/825577030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/825577030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:38:58.925 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:38:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:38:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:58 smithi122 ceph-mon[32255]: pgmap v3015: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:38:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:58 smithi022 ceph-mon[31360]: pgmap v3015: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:38:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:58 smithi022 ceph-mon[40250]: pgmap v3015: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:39:00.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:00] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:39:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:38:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:38:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:38:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:00 smithi122 ceph-mon[32255]: pgmap v3016: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:00 smithi022 ceph-mon[31360]: pgmap v3016: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:00 smithi022 ceph-mon[40250]: pgmap v3016: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:02.150 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:02.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:02.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:02.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:02.659 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:39:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:39:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:02 smithi022 ceph-mon[31360]: pgmap v3017: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:39:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/139534869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:02 smithi022 ceph-mon[40250]: pgmap v3017: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:39:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/139534869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:02 smithi122 ceph-mon[32255]: pgmap v3017: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:39:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/139534869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:04 smithi122 ceph-mon[32255]: pgmap v3018: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:04 smithi022 ceph-mon[31360]: pgmap v3018: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:04 smithi022 ceph-mon[40250]: pgmap v3018: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:06 smithi122 ceph-mon[32255]: pgmap v3019: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:06 smithi022 ceph-mon[31360]: pgmap v3019: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:06 smithi022 ceph-mon[40250]: pgmap v3019: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:07.522 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:39:07.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:07.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:07.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/398230569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/398230569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/398230569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:08.924 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:39:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:39:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:08 smithi122 ceph-mon[32255]: pgmap v3020: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:08 smithi022 ceph-mon[31360]: pgmap v3020: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:08 smithi022 ceph-mon[40250]: pgmap v3020: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:10.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:10] "GET /metrics HTTP/1.1" 200 202315 "" "Prometheus/2.18.1" 2021-12-15T19:39:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:10 smithi122 ceph-mon[32255]: pgmap v3021: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:10 smithi022 ceph-mon[31360]: pgmap v3021: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:10 smithi022 ceph-mon[40250]: pgmap v3021: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:12.661 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:39:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:39:12.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:12 smithi122 ceph-mon[32255]: pgmap v3022: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:12 smithi022 ceph-mon[31360]: pgmap v3022: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:12 smithi022 ceph-mon[40250]: pgmap v3022: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:13.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:13.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:13.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2589067924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2589067924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2589067924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:14 smithi122 ceph-mon[32255]: pgmap v3023: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:14 smithi022 ceph-mon[31360]: pgmap v3023: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:14 smithi022 ceph-mon[40250]: pgmap v3023: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:16 smithi122 ceph-mon[32255]: pgmap v3024: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:16 smithi022 ceph-mon[31360]: pgmap v3024: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:16 smithi022 ceph-mon[40250]: pgmap v3024: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:39:18.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:18.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:18.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:18.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:18.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:39:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:39:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:18 smithi122 ceph-mon[32255]: pgmap v3025: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3581451101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:18 smithi022 ceph-mon[31360]: pgmap v3025: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3581451101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:18 smithi022 ceph-mon[40250]: pgmap v3025: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3581451101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:20.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:20] "GET /metrics HTTP/1.1" 200 202313 "" "Prometheus/2.18.1" 2021-12-15T19:39:20.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:20 smithi122 ceph-mon[32255]: pgmap v3026: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:20 smithi022 ceph-mon[31360]: pgmap v3026: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:20 smithi022 ceph-mon[40250]: pgmap v3026: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:22.663 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:39:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:39:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:22 smithi122 ceph-mon[32255]: pgmap v3027: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:22 smithi022 ceph-mon[31360]: pgmap v3027: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:22 smithi022 ceph-mon[40250]: pgmap v3027: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:23.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:24.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:24.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:24.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:24 smithi122 ceph-mon[32255]: pgmap v3028: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4043325198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:24 smithi022 ceph-mon[31360]: pgmap v3028: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4043325198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:24 smithi022 ceph-mon[40250]: pgmap v3028: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4043325198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:26 smithi122 ceph-mon[32255]: pgmap v3029: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:39:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:39:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:39:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:39:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[31360]: pgmap v3029: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:39:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:39:27.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:39:27.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:39:27.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[40250]: pgmap v3029: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:27.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:39:27.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:39:27.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:39:27.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:39:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:39:28.926 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:39:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:39:29.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:28 smithi122 ceph-mon[32255]: pgmap v3030: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:28 smithi022 ceph-mon[31360]: pgmap v3030: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:28 smithi022 ceph-mon[40250]: pgmap v3030: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:29.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:29.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:29.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:30.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:30] "GET /metrics HTTP/1.1" 200 202313 "" "Prometheus/2.18.1" 2021-12-15T19:39:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:30.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/647458791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/647458791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/647458791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:30 smithi122 ceph-mon[32255]: pgmap v3031: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:30 smithi022 ceph-mon[31360]: pgmap v3031: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:30 smithi022 ceph-mon[40250]: pgmap v3031: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:32.664 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:39:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:39:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:32 smithi122 ceph-mon[32255]: pgmap v3032: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:39:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:39:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:39:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:39:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[31360]: pgmap v3032: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:39:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:39:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:39:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:39:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[40250]: pgmap v3032: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:39:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:39:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:39:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:39:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:34.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:34.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:34.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:34 smithi022 ceph-mon[31360]: pgmap v3033: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1602807581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:34 smithi022 ceph-mon[40250]: pgmap v3033: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1602807581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:34 smithi122 ceph-mon[32255]: pgmap v3033: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1602807581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:36 smithi122 ceph-mon[32255]: pgmap v3034: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:36 smithi022 ceph-mon[31360]: pgmap v3034: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:36 smithi022 ceph-mon[40250]: pgmap v3034: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:39:38.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:39:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:39:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:38 smithi122 ceph-mon[32255]: pgmap v3035: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:38 smithi022 ceph-mon[31360]: pgmap v3035: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:38 smithi022 ceph-mon[40250]: pgmap v3035: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:39.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:40.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:40.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:40.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:40.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:40.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:40] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:39:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:40 smithi022 ceph-mon[31360]: pgmap v3036: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/405681813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:40 smithi022 ceph-mon[40250]: pgmap v3036: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/405681813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:40 smithi122 ceph-mon[32255]: pgmap v3036: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/405681813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:42.667 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:39:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:39:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:42 smithi122 ceph-mon[32255]: pgmap v3037: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:42 smithi022 ceph-mon[31360]: pgmap v3037: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:42 smithi022 ceph-mon[40250]: pgmap v3037: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:44 smithi122 ceph-mon[32255]: pgmap v3038: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:44 smithi022 ceph-mon[31360]: pgmap v3038: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:44 smithi022 ceph-mon[40250]: pgmap v3038: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:45.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:45.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:45.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1836795959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1836795959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1836795959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:46 smithi122 ceph-mon[32255]: pgmap v3039: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:46 smithi022 ceph-mon[31360]: pgmap v3039: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:46 smithi022 ceph-mon[40250]: pgmap v3039: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:39:48.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:39:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:39:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:48 smithi022 ceph-mon[31360]: pgmap v3040: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:48 smithi022 ceph-mon[40250]: pgmap v3040: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:48 smithi122 ceph-mon[32255]: pgmap v3040: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:50] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:39:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:39:50.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:50.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:50 smithi022 ceph-mon[31360]: pgmap v3041: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1580942481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:50 smithi022 ceph-mon[40250]: pgmap v3041: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1580942481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:50 smithi122 ceph-mon[32255]: pgmap v3041: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:51.621 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1580942481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:52.668 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:39:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:39:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:52 smithi122 ceph-mon[32255]: pgmap v3042: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:52 smithi022 ceph-mon[31360]: pgmap v3042: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:52 smithi022 ceph-mon[40250]: pgmap v3042: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:54 smithi122 ceph-mon[32255]: pgmap v3043: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:54 smithi022 ceph-mon[31360]: pgmap v3043: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:55.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:54 smithi022 ceph-mon[40250]: pgmap v3043: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:39:55.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:39:56.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:39:56.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:39:56.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:39:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:56 smithi122 ceph-mon[32255]: pgmap v3044: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3950607283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:56 smithi022 ceph-mon[31360]: pgmap v3044: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3950607283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:56 smithi022 ceph-mon[40250]: pgmap v3044: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3950607283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:39:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:39:58.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:39:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:39:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:58 smithi022 ceph-mon[31360]: pgmap v3045: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:58 smithi022 ceph-mon[40250]: pgmap v3045: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:39:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:58 smithi122 ceph-mon[32255]: pgmap v3045: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:39:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:39:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:00] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:40:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:39:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:01.234 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:01.234 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:00 smithi022 ceph-mon[40250]: pgmap v3046: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:01.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T19:40:01.235 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:00 smithi022 ceph-mon[31360]: pgmap v3046: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:01.235 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T19:40:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:00 smithi122 ceph-mon[32255]: pgmap v3046: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T19:40:01.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:01.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:01.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3014265887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3014265887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3014265887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:40:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:40:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:02 smithi022 ceph-mon[31360]: pgmap v3047: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:02 smithi022 ceph-mon[40250]: pgmap v3047: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:02 smithi122 ceph-mon[32255]: pgmap v3047: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:04 smithi022 ceph-mon[31360]: pgmap v3048: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:04 smithi022 ceph-mon[40250]: pgmap v3048: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:04 smithi122 ceph-mon[32255]: pgmap v3048: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:06.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:06.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:06.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:06.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:06 smithi022 ceph-mon[31360]: pgmap v3049: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:06 smithi022 ceph-mon[40250]: pgmap v3049: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:06 smithi122 ceph-mon[32255]: pgmap v3049: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:40:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2228460020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2228460020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2228460020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:08.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:40:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:40:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:08 smithi022 ceph-mon[31360]: pgmap v3050: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:08 smithi022 ceph-mon[40250]: pgmap v3050: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:08 smithi122 ceph-mon[32255]: pgmap v3050: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:10] "GET /metrics HTTP/1.1" 200 202313 "" "Prometheus/2.18.1" 2021-12-15T19:40:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:10 smithi022 ceph-mon[31360]: pgmap v3051: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:10 smithi022 ceph-mon[40250]: pgmap v3051: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:10 smithi122 ceph-mon[32255]: pgmap v3051: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:11.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:12.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:12.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:12.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:12.673 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:40:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:40:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:12 smithi122 ceph-mon[32255]: pgmap v3052: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1375692974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:12 smithi022 ceph-mon[31360]: pgmap v3052: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1375692974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:12 smithi022 ceph-mon[40250]: pgmap v3052: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1375692974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:14 smithi022 ceph-mon[31360]: pgmap v3053: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:14 smithi022 ceph-mon[40250]: pgmap v3053: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:14 smithi122 ceph-mon[32255]: pgmap v3053: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:16 smithi022 ceph-mon[31360]: pgmap v3054: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:17.448 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:16 smithi022 ceph-mon[40250]: pgmap v3054: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:17.450 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:17.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:16 smithi122 ceph-mon[32255]: pgmap v3054: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:17.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:17.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:17.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:17.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:40:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1183171415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1183171415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1183171415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:18.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:40:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:40:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:18 smithi022 ceph-mon[31360]: pgmap v3055: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:18 smithi022 ceph-mon[40250]: pgmap v3055: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:18 smithi122 ceph-mon[32255]: pgmap v3055: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:20] "GET /metrics HTTP/1.1" 200 202321 "" "Prometheus/2.18.1" 2021-12-15T19:40:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:20 smithi022 ceph-mon[31360]: pgmap v3056: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:20 smithi022 ceph-mon[40250]: pgmap v3056: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:20 smithi122 ceph-mon[32255]: pgmap v3056: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:22.676 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:40:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:40:22.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:22 smithi122 ceph-mon[32255]: pgmap v3057: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:22 smithi022 ceph-mon[31360]: pgmap v3057: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:22 smithi022 ceph-mon[40250]: pgmap v3057: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:23.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:23.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:23.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2839110454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2839110454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2839110454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:24 smithi022 ceph-mon[31360]: pgmap v3058: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:24 smithi022 ceph-mon[40250]: pgmap v3058: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:24 smithi122 ceph-mon[32255]: pgmap v3058: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[31360]: pgmap v3059: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:40:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:40:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:40:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[40250]: pgmap v3059: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:40:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:40:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:40:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:26 smithi122 ceph-mon[32255]: pgmap v3059: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:40:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:40:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:40:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:40:28.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:28.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:28.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:28.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:28.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:40:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:40:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:28 smithi022 ceph-mon[31360]: pgmap v3060: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2076493851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:28 smithi022 ceph-mon[40250]: pgmap v3060: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2076493851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:28 smithi122 ceph-mon[32255]: pgmap v3060: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2076493851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:30] "GET /metrics HTTP/1.1" 200 202321 "" "Prometheus/2.18.1" 2021-12-15T19:40:31.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:31 smithi122 ceph-mon[32255]: pgmap v3061: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:31.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:31.604 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:31 smithi022 ceph-mon[31360]: pgmap v3061: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:31 smithi022 ceph-mon[40250]: pgmap v3061: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:32.609 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:40:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:40:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:32 smithi122 ceph-mon[32255]: pgmap v3062: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:40:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:40:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:40:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:40:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[40250]: pgmap v3062: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:40:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:40:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:40:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:40:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:40:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[31360]: pgmap v3062: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:40:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:40:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:40:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:40:33.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:33.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:33.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:33.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3887400649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3887400649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3887400649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:34 smithi122 ceph-mon[32255]: pgmap v3063: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:34 smithi022 ceph-mon[31360]: pgmap v3063: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:34 smithi022 ceph-mon[40250]: pgmap v3063: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:36 smithi022 ceph-mon[31360]: pgmap v3064: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:36 smithi022 ceph-mon[40250]: pgmap v3064: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:36 smithi122 ceph-mon[32255]: pgmap v3064: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:40:38.832 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:38.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:40:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:40:39.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:39.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:39.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:39.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:38 smithi022 ceph-mon[31360]: pgmap v3065: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:39.204 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:38 smithi022 ceph-mon[40250]: pgmap v3065: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:38 smithi122 ceph-mon[32255]: pgmap v3065: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1817655900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1817655900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:40] "GET /metrics HTTP/1.1" 200 202315 "" "Prometheus/2.18.1" 2021-12-15T19:40:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1817655900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:40 smithi022 ceph-mon[31360]: pgmap v3066: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:40 smithi022 ceph-mon[40250]: pgmap v3066: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:40 smithi122 ceph-mon[32255]: pgmap v3066: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:42.683 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:40:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:40:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:42 smithi122 ceph-mon[32255]: pgmap v3067: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:42 smithi022 ceph-mon[31360]: pgmap v3067: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:42 smithi022 ceph-mon[40250]: pgmap v3067: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:44.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:44.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:44.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:44.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:44 smithi022 ceph-mon[31360]: pgmap v3068: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/203398137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:44 smithi022 ceph-mon[40250]: pgmap v3068: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/203398137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:44 smithi122 ceph-mon[32255]: pgmap v3068: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/203398137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:46 smithi022 ceph-mon[31360]: pgmap v3069: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:46 smithi022 ceph-mon[40250]: pgmap v3069: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:46 smithi122 ceph-mon[32255]: pgmap v3069: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:40:48.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:40:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:40:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:48 smithi022 ceph-mon[31360]: pgmap v3070: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:48 smithi022 ceph-mon[40250]: pgmap v3070: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:48 smithi122 ceph-mon[32255]: pgmap v3070: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:49.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:49.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:49.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:49.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/656509478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/656509478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:40:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/656509478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:50] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:40:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:50 smithi022 ceph-mon[31360]: pgmap v3071: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:50 smithi022 ceph-mon[40250]: pgmap v3071: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:50 smithi122 ceph-mon[32255]: pgmap v3071: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:52.683 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:40:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:40:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:52 smithi122 ceph-mon[32255]: pgmap v3072: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:52 smithi022 ceph-mon[31360]: pgmap v3072: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:52 smithi022 ceph-mon[40250]: pgmap v3072: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:40:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:54 smithi022 ceph-mon[31360]: pgmap v3073: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:54 smithi022 ceph-mon[40250]: pgmap v3073: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:55.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:40:55.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:40:55.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:40:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:54 smithi122 ceph-mon[32255]: pgmap v3073: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:40:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1682579046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1682579046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1682579046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:40:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:56 smithi022 ceph-mon[31360]: pgmap v3074: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:56 smithi022 ceph-mon[40250]: pgmap v3074: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:56 smithi122 ceph-mon[32255]: pgmap v3074: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:40:58.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:40:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:40:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:58 smithi022 ceph-mon[31360]: pgmap v3075: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:58 smithi022 ceph-mon[40250]: pgmap v3075: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:40:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:58 smithi122 ceph-mon[32255]: pgmap v3075: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:40:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:40:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:00.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:00] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:41:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:40:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:00.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:00.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:00.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:00 smithi022 ceph-mon[31360]: pgmap v3076: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1045183016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:00 smithi022 ceph-mon[40250]: pgmap v3076: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1045183016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:00 smithi122 ceph-mon[32255]: pgmap v3076: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1045183016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:41:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:41:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:02 smithi122 ceph-mon[32255]: pgmap v3077: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:02 smithi022 ceph-mon[31360]: pgmap v3077: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:02 smithi022 ceph-mon[40250]: pgmap v3077: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:04 smithi022 ceph-mon[31360]: pgmap v3078: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:04 smithi022 ceph-mon[40250]: pgmap v3078: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:04 smithi122 ceph-mon[32255]: pgmap v3078: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:05.692 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:06.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:06.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:06.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:06 smithi022 ceph-mon[31360]: pgmap v3079: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/984785804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:06 smithi022 ceph-mon[40250]: pgmap v3079: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/984785804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:06 smithi122 ceph-mon[32255]: pgmap v3079: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:07.532 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/984785804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:41:08.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:41:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:41:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:08 smithi022 ceph-mon[31360]: pgmap v3080: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:08 smithi022 ceph-mon[40250]: pgmap v3080: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:08 smithi122 ceph-mon[32255]: pgmap v3080: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:10] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:41:11.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:10 smithi022 ceph-mon[31360]: pgmap v3081: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:10 smithi022 ceph-mon[40250]: pgmap v3081: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:11.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:11.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:11.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:10 smithi122 ceph-mon[32255]: pgmap v3081: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1539142433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1539142433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1539142433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:41:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:41:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:12 smithi022 ceph-mon[31360]: pgmap v3082: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:12 smithi022 ceph-mon[40250]: pgmap v3082: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:12 smithi122 ceph-mon[32255]: pgmap v3082: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:14 smithi022 ceph-mon[31360]: pgmap v3083: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:14 smithi022 ceph-mon[40250]: pgmap v3083: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:14 smithi122 ceph-mon[32255]: pgmap v3083: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:16.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:16.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:16.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:16.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:16 smithi022 ceph-mon[31360]: pgmap v3084: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/524485808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:16 smithi022 ceph-mon[40250]: pgmap v3084: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/524485808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:16 smithi122 ceph-mon[32255]: pgmap v3084: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/524485808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:41:18.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:41:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:41:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:18 smithi022 ceph-mon[31360]: pgmap v3085: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:18 smithi022 ceph-mon[40250]: pgmap v3085: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:18 smithi122 ceph-mon[32255]: pgmap v3085: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:20] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T19:41:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:20 smithi022 ceph-mon[31360]: pgmap v3086: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:20 smithi022 ceph-mon[40250]: pgmap v3086: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:20 smithi122 ceph-mon[32255]: pgmap v3086: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:21.801 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:22.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:22.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:22.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:41:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:41:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:22 smithi022 ceph-mon[31360]: pgmap v3087: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/837025812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:22 smithi022 ceph-mon[40250]: pgmap v3087: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/837025812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:22 smithi122 ceph-mon[32255]: pgmap v3087: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/837025812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:24 smithi022 ceph-mon[31360]: pgmap v3088: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:24 smithi022 ceph-mon[40250]: pgmap v3088: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:24 smithi122 ceph-mon[32255]: pgmap v3088: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:27.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:26 smithi022 ceph-mon[40250]: pgmap v3089: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:26 smithi022 ceph-mon[31360]: pgmap v3089: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:26 smithi122 ceph-mon[32255]: pgmap v3089: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:27.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:27.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:27.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:41:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3061070467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3061070467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3061070467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:28.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:41:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:41:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:28 smithi022 ceph-mon[31360]: pgmap v3090: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:28 smithi022 ceph-mon[40250]: pgmap v3090: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:28 smithi122 ceph-mon[32255]: pgmap v3090: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:30] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T19:41:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:30 smithi022 ceph-mon[31360]: pgmap v3091: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:30 smithi022 ceph-mon[40250]: pgmap v3091: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:30 smithi122 ceph-mon[32255]: pgmap v3091: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:41:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:41:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:41:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:41:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:41:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:41:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:41:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:41:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:41:32.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:41:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:41:32.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:32.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:32.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[31360]: pgmap v3092: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:41:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:41:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:41:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/825067810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[40250]: pgmap v3092: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:41:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:41:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:41:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:41:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:41:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/825067810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:33 smithi122 ceph-mon[32255]: pgmap v3092: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:41:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:41:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:41:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:41:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:41:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/825067810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:35 smithi122 ceph-mon[32255]: pgmap v3093: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:35 smithi022 ceph-mon[31360]: pgmap v3093: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:35 smithi022 ceph-mon[40250]: pgmap v3093: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:37 smithi022 ceph-mon[31360]: pgmap v3094: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:37 smithi022 ceph-mon[40250]: pgmap v3094: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:37 smithi122 ceph-mon[32255]: pgmap v3094: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:41:37.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:38.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:38.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:38.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:38.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:41:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:41:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:39 smithi122 ceph-mon[32255]: pgmap v3095: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2793539975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:39 smithi022 ceph-mon[40250]: pgmap v3095: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2793539975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:39 smithi022 ceph-mon[31360]: pgmap v3095: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2793539975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:40] "GET /metrics HTTP/1.1" 200 202314 "" "Prometheus/2.18.1" 2021-12-15T19:41:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:41 smithi122 ceph-mon[32255]: pgmap v3096: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:41 smithi022 ceph-mon[31360]: pgmap v3096: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:41 smithi022 ceph-mon[40250]: pgmap v3096: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:41:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:41:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:42 smithi022 ceph-mon[31360]: pgmap v3097: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:42 smithi022 ceph-mon[40250]: pgmap v3097: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:42 smithi122 ceph-mon[32255]: pgmap v3097: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:43.285 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:43.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2610233952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2610233952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2610233952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:44 smithi022 ceph-mon[31360]: pgmap v3098: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:44 smithi022 ceph-mon[40250]: pgmap v3098: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:44 smithi122 ceph-mon[32255]: pgmap v3098: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:46 smithi022 ceph-mon[31360]: pgmap v3099: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:46 smithi022 ceph-mon[40250]: pgmap v3099: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:46 smithi122 ceph-mon[32255]: pgmap v3099: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:41:48.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:48.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:41:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:41:49.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:49.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:49.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:48 smithi022 ceph-mon[31360]: pgmap v3100: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:48 smithi022 ceph-mon[40250]: pgmap v3100: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:48 smithi122 ceph-mon[32255]: pgmap v3100: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3232843953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3232843953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:41:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3232843953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:50] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:41:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:50 smithi022 ceph-mon[31360]: pgmap v3101: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:50 smithi022 ceph-mon[40250]: pgmap v3101: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:50 smithi122 ceph-mon[32255]: pgmap v3101: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:41:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:41:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:52 smithi022 ceph-mon[31360]: pgmap v3102: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:52 smithi022 ceph-mon[40250]: pgmap v3102: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:52 smithi122 ceph-mon[32255]: pgmap v3102: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:54.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:54.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:54.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:54.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:41:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:54 smithi022 ceph-mon[40250]: pgmap v3103: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2384292879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:54 smithi022 ceph-mon[31360]: pgmap v3103: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2384292879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:54 smithi122 ceph-mon[32255]: pgmap v3103: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:41:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2384292879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:41:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:56 smithi022 ceph-mon[31360]: pgmap v3104: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:56 smithi022 ceph-mon[40250]: pgmap v3104: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:56 smithi122 ceph-mon[32255]: pgmap v3104: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:41:58.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:41:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:41:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:58 smithi022 ceph-mon[31360]: pgmap v3105: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:58 smithi022 ceph-mon[40250]: pgmap v3105: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:59.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:41:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:58 smithi122 ceph-mon[32255]: pgmap v3105: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:41:59.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:41:59.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:41:59.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:41:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/613421315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:41:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/613421315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:00] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:42:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:41:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/613421315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:00 smithi022 ceph-mon[31360]: pgmap v3106: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:00 smithi022 ceph-mon[40250]: pgmap v3106: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:00 smithi122 ceph-mon[32255]: pgmap v3106: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:42:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:42:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:02 smithi022 ceph-mon[31360]: pgmap v3107: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:02 smithi022 ceph-mon[40250]: pgmap v3107: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:02 smithi122 ceph-mon[32255]: pgmap v3107: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:04.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:04 smithi022 ceph-mon[31360]: pgmap v3108: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:04 smithi022 ceph-mon[40250]: pgmap v3108: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:05.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:05.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:05.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:04 smithi122 ceph-mon[32255]: pgmap v3108: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2413734807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2413734807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2413734807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:06 smithi022 ceph-mon[31360]: pgmap v3109: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:06 smithi022 ceph-mon[40250]: pgmap v3109: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:06 smithi122 ceph-mon[32255]: pgmap v3109: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:42:08.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:42:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:42:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:08 smithi022 ceph-mon[31360]: pgmap v3110: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:08 smithi022 ceph-mon[40250]: pgmap v3110: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:08 smithi122 ceph-mon[32255]: pgmap v3110: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:10.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:10] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:42:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:10.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:10.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:10.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:10 smithi022 ceph-mon[31360]: pgmap v3111: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4050014258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:10 smithi022 ceph-mon[40250]: pgmap v3111: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4050014258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:10 smithi122 ceph-mon[32255]: pgmap v3111: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4050014258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:42:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:42:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:12 smithi022 ceph-mon[31360]: pgmap v3112: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:12 smithi022 ceph-mon[40250]: pgmap v3112: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:12 smithi122 ceph-mon[32255]: pgmap v3112: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:14 smithi022 ceph-mon[31360]: pgmap v3113: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:14 smithi022 ceph-mon[40250]: pgmap v3113: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:14 smithi122 ceph-mon[32255]: pgmap v3113: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:15.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:15.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:15.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/682173653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/682173653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/682173653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:16 smithi022 ceph-mon[31360]: pgmap v3114: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:16 smithi022 ceph-mon[40250]: pgmap v3114: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:16 smithi122 ceph-mon[32255]: pgmap v3114: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:42:18.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:42:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:42:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:18 smithi022 ceph-mon[31360]: pgmap v3115: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:18 smithi022 ceph-mon[40250]: pgmap v3115: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:18 smithi122 ceph-mon[32255]: pgmap v3115: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:20.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:20.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:20] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:42:20.872 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:21.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:21.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:21.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:21.239 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:20 smithi022 ceph-mon[31360]: pgmap v3116: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:21.239 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:20 smithi022 ceph-mon[40250]: pgmap v3116: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:20 smithi122 ceph-mon[32255]: pgmap v3116: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3598614238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3598614238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3598614238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:42:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:42:23.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:22 smithi022 ceph-mon[31360]: pgmap v3117: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:23.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:22 smithi022 ceph-mon[40250]: pgmap v3117: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:22 smithi122 ceph-mon[32255]: pgmap v3117: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:24 smithi022 ceph-mon[31360]: pgmap v3118: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:24 smithi022 ceph-mon[40250]: pgmap v3118: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:24 smithi122 ceph-mon[32255]: pgmap v3118: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:26.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:26.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:26.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:26.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:26 smithi022 ceph-mon[31360]: pgmap v3119: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2403853310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:26 smithi022 ceph-mon[40250]: pgmap v3119: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2403853310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:26 smithi122 ceph-mon[32255]: pgmap v3119: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2403853310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:42:28.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:42:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:42:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:28 smithi022 ceph-mon[31360]: pgmap v3120: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:28 smithi022 ceph-mon[40250]: pgmap v3120: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:28 smithi122 ceph-mon[32255]: pgmap v3120: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:30] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:42:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:31 smithi022 ceph-mon[31360]: pgmap v3121: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:31 smithi022 ceph-mon[40250]: pgmap v3121: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:31 smithi122 ceph-mon[32255]: pgmap v3121: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:31.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:31.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:31.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:31.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:42:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:42:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: pgmap v3122: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/44188833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:42:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:42:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:42:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:42:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:42:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:42:32.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:42:32.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:42:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: pgmap v3122: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/44188833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:42:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:42:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:42:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:42:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:42:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:42:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:42:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:42:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: pgmap v3122: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/44188833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:42:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:42:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:42:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:42:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:42:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:42:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:42:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:42:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:34 smithi022 ceph-mon[31360]: pgmap v3123: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:34 smithi022 ceph-mon[40250]: pgmap v3123: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:34 smithi122 ceph-mon[32255]: pgmap v3123: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:36.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:36 smithi022 ceph-mon[31360]: pgmap v3124: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:36 smithi022 ceph-mon[40250]: pgmap v3124: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:37.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:37.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:37.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:36 smithi122 ceph-mon[32255]: pgmap v3124: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:42:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3725714399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3725714399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3725714399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:38.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:42:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:42:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:38 smithi022 ceph-mon[31360]: pgmap v3125: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:38 smithi022 ceph-mon[40250]: pgmap v3125: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:38 smithi122 ceph-mon[32255]: pgmap v3125: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:40.471 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:40] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T19:42:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:40 smithi022 ceph-mon[31360]: pgmap v3126: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:40 smithi022 ceph-mon[40250]: pgmap v3126: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:41.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:40 smithi122 ceph-mon[32255]: pgmap v3126: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:42:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:42:42.706 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:42 smithi022 ceph-mon[40250]: pgmap v3127: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:42.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:42.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:42.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:42 smithi022 ceph-mon[31360]: pgmap v3127: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:42 smithi122 ceph-mon[32255]: pgmap v3127: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/625532974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/625532974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/625532974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:44 smithi022 ceph-mon[31360]: pgmap v3128: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:44 smithi022 ceph-mon[40250]: pgmap v3128: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:44 smithi122 ceph-mon[32255]: pgmap v3128: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:46 smithi022 ceph-mon[31360]: pgmap v3129: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:46 smithi022 ceph-mon[40250]: pgmap v3129: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:46 smithi122 ceph-mon[32255]: pgmap v3129: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:47.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:47.718 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:42:48.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:48.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:48.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:48 smithi122 ceph-mon[32255]: pgmap v3130: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/27394022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:49.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:42:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:42:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:48 smithi022 ceph-mon[31360]: pgmap v3130: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/27394022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:48 smithi022 ceph-mon[40250]: pgmap v3130: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/27394022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:42:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:50] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:42:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:50 smithi022 ceph-mon[31360]: pgmap v3131: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:50 smithi022 ceph-mon[40250]: pgmap v3131: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:50 smithi122 ceph-mon[32255]: pgmap v3131: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:42:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:42:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:52 smithi022 ceph-mon[31360]: pgmap v3132: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:52 smithi022 ceph-mon[40250]: pgmap v3132: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:53.089 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:52 smithi122 ceph-mon[32255]: pgmap v3132: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:53.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:53.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:53.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4251482313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4251482313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4251482313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:54 smithi022 ceph-mon[31360]: pgmap v3133: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:54 smithi022 ceph-mon[40250]: pgmap v3133: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:54 smithi122 ceph-mon[32255]: pgmap v3133: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:42:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:56 smithi022 ceph-mon[31360]: pgmap v3134: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:56 smithi022 ceph-mon[40250]: pgmap v3134: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:56 smithi122 ceph-mon[32255]: pgmap v3134: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:42:58.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:42:58.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:42:58.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:42:58.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:42:59.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:42:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:42:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:58 smithi122 ceph-mon[32255]: pgmap v3135: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:59.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3324069113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:58 smithi022 ceph-mon[31360]: pgmap v3135: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3324069113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:42:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:58 smithi022 ceph-mon[40250]: pgmap v3135: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:42:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3324069113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:42:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:42:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:42:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:00] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:43:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:00 smithi022 ceph-mon[31360]: pgmap v3136: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:00 smithi022 ceph-mon[40250]: pgmap v3136: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:00 smithi122 ceph-mon[32255]: pgmap v3136: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:43:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:43:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:02 smithi022 ceph-mon[31360]: pgmap v3137: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:02 smithi022 ceph-mon[40250]: pgmap v3137: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:02 smithi122 ceph-mon[32255]: pgmap v3137: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:03.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:04.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:04.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:04.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:04 smithi022 ceph-mon[31360]: pgmap v3138: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3757861370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:04 smithi022 ceph-mon[40250]: pgmap v3138: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3757861370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:04 smithi122 ceph-mon[32255]: pgmap v3138: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3757861370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:06 smithi022 ceph-mon[31360]: pgmap v3139: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:06 smithi022 ceph-mon[40250]: pgmap v3139: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:06 smithi122 ceph-mon[32255]: pgmap v3139: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:43:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:43:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:43:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:08 smithi122 ceph-mon[32255]: pgmap v3140: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:09.199 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:08 smithi022 ceph-mon[31360]: pgmap v3140: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:08 smithi022 ceph-mon[40250]: pgmap v3140: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:09.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:09.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:09.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:09.998 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:09.999 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3048107817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:09.999 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:09.999 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3048107817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:10] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:43:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3048107817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:10 smithi022 ceph-mon[31360]: pgmap v3141: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:11 smithi022 ceph-mon[40250]: pgmap v3141: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:11 smithi122 ceph-mon[32255]: pgmap v3141: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:43:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:43:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:12 smithi022 ceph-mon[31360]: pgmap v3142: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:12 smithi022 ceph-mon[40250]: pgmap v3142: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:12 smithi122 ceph-mon[32255]: pgmap v3142: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:14.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:14.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:14.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:14.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:14 smithi022 ceph-mon[31360]: pgmap v3143: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/977645997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:14 smithi022 ceph-mon[40250]: pgmap v3143: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/977645997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:14 smithi122 ceph-mon[32255]: pgmap v3143: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/977645997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:16 smithi022 ceph-mon[31360]: pgmap v3144: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:16 smithi022 ceph-mon[40250]: pgmap v3144: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:16 smithi122 ceph-mon[32255]: pgmap v3144: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:43:19.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:43:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:43:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:18 smithi122 ceph-mon[32255]: pgmap v3145: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:18 smithi022 ceph-mon[31360]: pgmap v3145: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:18 smithi022 ceph-mon[40250]: pgmap v3145: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:19.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:20.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:20.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:20.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:20] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:43:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:21 smithi022 ceph-mon[31360]: pgmap v3146: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1796842371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:21 smithi022 ceph-mon[40250]: pgmap v3146: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1796842371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:21 smithi122 ceph-mon[32255]: pgmap v3146: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1796842371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:43:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:43:23.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:22 smithi022 ceph-mon[31360]: pgmap v3147: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:23.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:22 smithi022 ceph-mon[40250]: pgmap v3147: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:22 smithi122 ceph-mon[32255]: pgmap v3147: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:24 smithi022 ceph-mon[31360]: pgmap v3148: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:24 smithi022 ceph-mon[40250]: pgmap v3148: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:25.307 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:24 smithi122 ceph-mon[32255]: pgmap v3148: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:25.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:25.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:25.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3079600928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3079600928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3079600928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:27 smithi022 ceph-mon[31360]: pgmap v3149: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:27 smithi022 ceph-mon[40250]: pgmap v3149: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:27 smithi122 ceph-mon[32255]: pgmap v3149: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:43:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:29 smithi122 ceph-mon[32255]: pgmap v3150: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:43:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:43:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:29 smithi022 ceph-mon[31360]: pgmap v3150: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:29 smithi022 ceph-mon[40250]: pgmap v3150: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:30] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:43:30.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:31.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:31.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:31.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:31 smithi022 ceph-mon[31360]: pgmap v3151: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:31 smithi022 ceph-mon[40250]: pgmap v3151: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:31 smithi122 ceph-mon[32255]: pgmap v3151: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/279057680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/279057680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/279057680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:32.580 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:43:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:43:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: pgmap v3152: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:43:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:43:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:43:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:43:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:43:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:43:33.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:43:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: pgmap v3152: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:33.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:43:33.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:43:33.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:43:33.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:43:33.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:43:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:43:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:43:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: pgmap v3152: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:43:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:43:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:43:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:43:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:43:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:43:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:43:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:43:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:43:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:43:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:35 smithi022 ceph-mon[31360]: pgmap v3153: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:35 smithi022 ceph-mon[40250]: pgmap v3153: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:35 smithi122 ceph-mon[32255]: pgmap v3153: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:36.042 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:36.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:36.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:36.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:37 smithi022 ceph-mon[31360]: pgmap v3154: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3628628193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:37 smithi022 ceph-mon[40250]: pgmap v3154: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3628628193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:37 smithi122 ceph-mon[32255]: pgmap v3154: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3628628193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:43:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:39 smithi122 ceph-mon[32255]: pgmap v3155: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:43:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:43:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:39 smithi022 ceph-mon[31360]: pgmap v3155: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:39 smithi022 ceph-mon[40250]: pgmap v3155: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:40] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:43:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:41.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:41.410 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:41 smithi022 ceph-mon[40250]: pgmap v3156: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:41.411 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:41 smithi022 ceph-mon[31360]: pgmap v3156: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:41 smithi122 ceph-mon[32255]: pgmap v3156: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:41.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:41.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:41.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:43:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:43:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2362305333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:42 smithi022 ceph-mon[31360]: pgmap v3157: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2362305333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:42 smithi022 ceph-mon[40250]: pgmap v3157: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2362305333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:42 smithi122 ceph-mon[32255]: pgmap v3157: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:45 smithi022 ceph-mon[31360]: pgmap v3158: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:45 smithi022 ceph-mon[40250]: pgmap v3158: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:45 smithi122 ceph-mon[32255]: pgmap v3158: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:46.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:47.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:47.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:47.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:47.151 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:47 smithi022 ceph-mon[40250]: pgmap v3159: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:47.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:47 smithi022 ceph-mon[31360]: pgmap v3159: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:47 smithi122 ceph-mon[32255]: pgmap v3159: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:43:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3058180777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3058180777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:48.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3058180777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:49 smithi122 ceph-mon[32255]: pgmap v3160: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:43:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:43:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:49 smithi022 ceph-mon[31360]: pgmap v3160: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:49 smithi022 ceph-mon[40250]: pgmap v3160: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:43:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:50] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:43:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:51 smithi022 ceph-mon[31360]: pgmap v3161: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:51 smithi022 ceph-mon[40250]: pgmap v3161: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:51 smithi122 ceph-mon[32255]: pgmap v3161: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:52.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:52.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:52.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:52.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:43:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:43:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:52 smithi022 ceph-mon[31360]: pgmap v3162: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3449872423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:52 smithi022 ceph-mon[40250]: pgmap v3162: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3449872423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:52 smithi122 ceph-mon[32255]: pgmap v3162: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3449872423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:55 smithi022 ceph-mon[31360]: pgmap v3163: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:55 smithi022 ceph-mon[40250]: pgmap v3163: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:55 smithi122 ceph-mon[32255]: pgmap v3163: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:43:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:57 smithi022 ceph-mon[31360]: pgmap v3164: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:57 smithi022 ceph-mon[40250]: pgmap v3164: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:57 smithi122 ceph-mon[32255]: pgmap v3164: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:57.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:43:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:43:57.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:43:57.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:43:57.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:43:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/887370119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/887370119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/887370119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:43:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:43:59 smithi122 ceph-mon[32255]: pgmap v3165: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:59.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:43:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:43:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:43:59 smithi022 ceph-mon[31360]: pgmap v3165: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:43:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:43:59 smithi022 ceph-mon[40250]: pgmap v3165: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:00] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:44:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:01 smithi022 ceph-mon[31360]: pgmap v3166: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:01 smithi022 ceph-mon[40250]: pgmap v3166: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:01 smithi122 ceph-mon[32255]: pgmap v3166: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:44:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:44:02.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:02 smithi022 ceph-mon[31360]: pgmap v3167: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:02 smithi022 ceph-mon[40250]: pgmap v3167: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:02 smithi122 ceph-mon[32255]: pgmap v3167: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:03.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:03.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:03.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4165734776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4165734776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4165734776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:04 smithi022 ceph-mon[31360]: pgmap v3168: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:04 smithi022 ceph-mon[40250]: pgmap v3168: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:04 smithi122 ceph-mon[32255]: pgmap v3168: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:07 smithi022 ceph-mon[31360]: pgmap v3169: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:07 smithi022 ceph-mon[40250]: pgmap v3169: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:07 smithi122 ceph-mon[32255]: pgmap v3169: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:44:08.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:08.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:08.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:08.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:09 smithi122 ceph-mon[32255]: pgmap v3170: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1238014274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:44:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:44:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:09 smithi022 ceph-mon[40250]: pgmap v3170: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1238014274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:09 smithi022 ceph-mon[31360]: pgmap v3170: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1238014274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:10] "GET /metrics HTTP/1.1" 200 202315 "" "Prometheus/2.18.1" 2021-12-15T19:44:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:11 smithi022 ceph-mon[31360]: pgmap v3171: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:11 smithi022 ceph-mon[40250]: pgmap v3171: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:11 smithi122 ceph-mon[32255]: pgmap v3171: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:44:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:44:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:12 smithi022 ceph-mon[31360]: pgmap v3172: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:12 smithi022 ceph-mon[40250]: pgmap v3172: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:12 smithi122 ceph-mon[32255]: pgmap v3172: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:13.633 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2053749010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2053749010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2053749010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:15 smithi122 ceph-mon[32255]: pgmap v3173: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:15 smithi022 ceph-mon[31360]: pgmap v3173: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:15 smithi022 ceph-mon[40250]: pgmap v3173: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:17 smithi022 ceph-mon[31360]: pgmap v3174: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:17 smithi022 ceph-mon[40250]: pgmap v3174: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:17 smithi122 ceph-mon[32255]: pgmap v3174: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:44:19.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:19 smithi122 ceph-mon[32255]: pgmap v3175: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:19.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:44:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:44:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:19 smithi022 ceph-mon[31360]: pgmap v3175: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:19 smithi022 ceph-mon[40250]: pgmap v3175: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:19.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:19.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:19.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:20] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T19:44:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/956079002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/956079002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/956079002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:21 smithi122 ceph-mon[32255]: pgmap v3176: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:21 smithi022 ceph-mon[31360]: pgmap v3176: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:21 smithi022 ceph-mon[40250]: pgmap v3176: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:44:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:44:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:22 smithi022 ceph-mon[31360]: pgmap v3177: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:22 smithi022 ceph-mon[40250]: pgmap v3177: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:22 smithi122 ceph-mon[32255]: pgmap v3177: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:24.376 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:25 smithi022 ceph-mon[31360]: pgmap v3178: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1082665396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:25 smithi022 ceph-mon[40250]: pgmap v3178: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1082665396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:25 smithi122 ceph-mon[32255]: pgmap v3178: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1082665396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:27 smithi022 ceph-mon[31360]: pgmap v3179: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:27 smithi022 ceph-mon[40250]: pgmap v3179: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:27.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:27 smithi122 ceph-mon[32255]: pgmap v3179: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:44:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:29 smithi122 ceph-mon[32255]: pgmap v3180: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:44:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:44:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:29 smithi022 ceph-mon[31360]: pgmap v3180: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:29 smithi022 ceph-mon[40250]: pgmap v3180: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:29.748 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:30.023 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:30.023 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:30.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:30.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:30.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:30] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T19:44:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:31 smithi022 ceph-mon[31360]: pgmap v3181: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2657606333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:31 smithi022 ceph-mon[40250]: pgmap v3181: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2657606333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:31 smithi122 ceph-mon[32255]: pgmap v3181: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2657606333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:44:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:44:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[31360]: pgmap v3182: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:44:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:44:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:44:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:44:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[40250]: pgmap v3182: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:44:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:44:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:44:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:44:33.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:32 smithi122 ceph-mon[32255]: pgmap v3182: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:33.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:44:33.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:44:33.061 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:44:33.062 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:44:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:44:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:44:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:44:34.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:44:34.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:44:34.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:44:34.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:44:34.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:44:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:44:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:44:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:44:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:44:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:34 smithi022 ceph-mon[31360]: pgmap v3183: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:34 smithi022 ceph-mon[40250]: pgmap v3183: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:35.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:34 smithi122 ceph-mon[32255]: pgmap v3183: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:35.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:35.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:35.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2141494734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2141494734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2141494734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:36 smithi022 ceph-mon[31360]: pgmap v3184: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:36 smithi022 ceph-mon[40250]: pgmap v3184: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:36 smithi122 ceph-mon[32255]: pgmap v3184: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:44:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:39 smithi122 ceph-mon[32255]: pgmap v3185: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:44:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:44:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:39 smithi022 ceph-mon[31360]: pgmap v3185: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:39 smithi022 ceph-mon[40250]: pgmap v3185: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:40] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T19:44:40.485 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:40.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:40.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:40.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:41 smithi022 ceph-mon[31360]: pgmap v3186: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2595557407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:41.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:41 smithi022 ceph-mon[40250]: pgmap v3186: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:41.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2595557407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:41 smithi122 ceph-mon[32255]: pgmap v3186: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:41.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2595557407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:44:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:44:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:42 smithi022 ceph-mon[31360]: pgmap v3187: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:42 smithi022 ceph-mon[40250]: pgmap v3187: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:42 smithi122 ceph-mon[32255]: pgmap v3187: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:45 smithi022 ceph-mon[31360]: pgmap v3188: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:45 smithi022 ceph-mon[40250]: pgmap v3188: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:45 smithi122 ceph-mon[32255]: pgmap v3188: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:45.856 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:46.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:46.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:46.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:47 smithi022 ceph-mon[31360]: pgmap v3189: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1369809798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:47 smithi022 ceph-mon[40250]: pgmap v3189: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1369809798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:47 smithi122 ceph-mon[32255]: pgmap v3189: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1369809798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:44:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:49 smithi122 ceph-mon[32255]: pgmap v3190: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:49.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:44:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:44:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:49 smithi022 ceph-mon[31360]: pgmap v3190: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:49 smithi022 ceph-mon[40250]: pgmap v3190: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:50] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T19:44:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:44:51.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:51 smithi122 ceph-mon[32255]: pgmap v3191: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:51 smithi022 ceph-mon[31360]: pgmap v3191: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:51 smithi022 ceph-mon[40250]: pgmap v3191: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:51.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:51.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:51.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1924532251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1924532251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1924532251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:44:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:44:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:53 smithi122 ceph-mon[32255]: pgmap v3192: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:53 smithi022 ceph-mon[31360]: pgmap v3192: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:53 smithi022 ceph-mon[40250]: pgmap v3192: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:55 smithi122 ceph-mon[32255]: pgmap v3193: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:55 smithi022 ceph-mon[31360]: pgmap v3193: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:55 smithi022 ceph-mon[40250]: pgmap v3193: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:44:56.599 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:44:56.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:44:56.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:44:56.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:44:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:57 smithi022 ceph-mon[31360]: pgmap v3194: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/477911189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:57 smithi022 ceph-mon[40250]: pgmap v3194: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/477911189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:57 smithi122 ceph-mon[32255]: pgmap v3194: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/477911189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:44:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:44:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:44:59 smithi122 ceph-mon[32255]: pgmap v3195: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:59.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:44:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:44:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:44:59 smithi022 ceph-mon[31360]: pgmap v3195: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:44:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:44:59 smithi022 ceph-mon[40250]: pgmap v3195: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:00] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T19:45:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:01 smithi122 ceph-mon[32255]: pgmap v3196: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:01 smithi022 ceph-mon[31360]: pgmap v3196: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:01 smithi022 ceph-mon[40250]: pgmap v3196: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:01.969 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:02.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:02.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:02.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:45:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:45:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:02 smithi022 ceph-mon[31360]: pgmap v3197: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2700614462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:02 smithi022 ceph-mon[40250]: pgmap v3197: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2700614462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:02 smithi122 ceph-mon[32255]: pgmap v3197: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2700614462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:05 smithi022 ceph-mon[31360]: pgmap v3198: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:05 smithi022 ceph-mon[40250]: pgmap v3198: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:05 smithi122 ceph-mon[32255]: pgmap v3198: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:07 smithi022 ceph-mon[31360]: pgmap v3199: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:07 smithi022 ceph-mon[40250]: pgmap v3199: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:07.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:07 smithi122 ceph-mon[32255]: pgmap v3199: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:07.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:07.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:07.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:07.712 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:45:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3407458392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3407458392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3407458392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:09 smithi122 ceph-mon[32255]: pgmap v3200: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:45:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:45:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:09 smithi022 ceph-mon[31360]: pgmap v3200: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:09 smithi022 ceph-mon[40250]: pgmap v3200: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:10] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:45:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:11 smithi122 ceph-mon[32255]: pgmap v3201: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:11 smithi022 ceph-mon[31360]: pgmap v3201: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:11 smithi022 ceph-mon[40250]: pgmap v3201: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:45:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:45:12.713 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:12 smithi022 ceph-mon[31360]: pgmap v3202: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:12 smithi022 ceph-mon[40250]: pgmap v3202: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:13.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:13.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:13.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:12 smithi122 ceph-mon[32255]: pgmap v3202: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2345220029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2345220029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2345220029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:14 smithi022 ceph-mon[31360]: pgmap v3203: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:14 smithi022 ceph-mon[40250]: pgmap v3203: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:14 smithi122 ceph-mon[32255]: pgmap v3203: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:17 smithi022 ceph-mon[31360]: pgmap v3204: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:17 smithi022 ceph-mon[40250]: pgmap v3204: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:17 smithi122 ceph-mon[32255]: pgmap v3204: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:45:18.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:19.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:45:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:45:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:19 smithi122 ceph-mon[32255]: pgmap v3205: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1175542523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:19 smithi022 ceph-mon[31360]: pgmap v3205: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1175542523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:19 smithi022 ceph-mon[40250]: pgmap v3205: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1175542523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:20] "GET /metrics HTTP/1.1" 200 202319 "" "Prometheus/2.18.1" 2021-12-15T19:45:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:21 smithi122 ceph-mon[32255]: pgmap v3206: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:21 smithi022 ceph-mon[31360]: pgmap v3206: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:21 smithi022 ceph-mon[40250]: pgmap v3206: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:45:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:45:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:22 smithi022 ceph-mon[31360]: pgmap v3207: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:22 smithi022 ceph-mon[40250]: pgmap v3207: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:22 smithi122 ceph-mon[32255]: pgmap v3207: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:23.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:23.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:23.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:23.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/39864673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/39864673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/39864673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:24 smithi122 ceph-mon[32255]: pgmap v3208: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:24 smithi022 ceph-mon[31360]: pgmap v3208: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:24 smithi022 ceph-mon[40250]: pgmap v3208: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:27 smithi022 ceph-mon[31360]: pgmap v3209: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:27 smithi022 ceph-mon[40250]: pgmap v3209: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:27 smithi122 ceph-mon[32255]: pgmap v3209: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:45:28.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:29 smithi122 ceph-mon[32255]: pgmap v3210: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:45:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:45:29.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:29.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:29.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:29.200 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:29 smithi022 ceph-mon[31360]: pgmap v3210: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:29.200 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:29 smithi022 ceph-mon[40250]: pgmap v3210: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:30] "GET /metrics HTTP/1.1" 200 202319 "" "Prometheus/2.18.1" 2021-12-15T19:45:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4009987130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4009987130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4009987130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:31 smithi122 ceph-mon[32255]: pgmap v3211: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:31 smithi022 ceph-mon[31360]: pgmap v3211: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:31 smithi022 ceph-mon[40250]: pgmap v3211: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:45:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:45:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[31360]: pgmap v3212: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:45:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:45:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:45:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:45:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[40250]: pgmap v3212: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:45:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:45:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:45:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:45:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:32 smithi122 ceph-mon[32255]: pgmap v3212: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:45:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:45:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:45:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:45:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:45:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:45:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:45:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:45:33.903 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:45:33.904 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:45:33.904 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:45:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:34.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:34.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:34.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:34.920 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:45:34.920 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:34 smithi022 ceph-mon[40250]: pgmap v3213: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:34.920 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4074292921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:45:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:34 smithi122 ceph-mon[32255]: pgmap v3213: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4074292921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:45:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:34 smithi022 ceph-mon[31360]: pgmap v3213: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4074292921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:37 smithi022 ceph-mon[31360]: pgmap v3214: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:37 smithi022 ceph-mon[40250]: pgmap v3214: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:37 smithi122 ceph-mon[32255]: pgmap v3214: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:45:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:39 smithi122 ceph-mon[32255]: pgmap v3215: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:45:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:45:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:39 smithi022 ceph-mon[31360]: pgmap v3215: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:39 smithi022 ceph-mon[40250]: pgmap v3215: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:39.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:39.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:39.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:39.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/306766835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/306766835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:40] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:45:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/306766835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:41 smithi122 ceph-mon[32255]: pgmap v3216: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:41 smithi022 ceph-mon[31360]: pgmap v3216: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:41 smithi022 ceph-mon[40250]: pgmap v3216: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:45:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:45:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:42 smithi022 ceph-mon[31360]: pgmap v3217: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:42 smithi022 ceph-mon[40250]: pgmap v3217: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:42 smithi122 ceph-mon[32255]: pgmap v3217: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:44.932 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:45 smithi022 ceph-mon[31360]: pgmap v3218: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:45 smithi022 ceph-mon[40250]: pgmap v3218: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:45.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:45.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:45.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:45 smithi122 ceph-mon[32255]: pgmap v3218: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3992542541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3992542541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3992542541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:47 smithi022 ceph-mon[31360]: pgmap v3219: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:47 smithi022 ceph-mon[40250]: pgmap v3219: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:47 smithi122 ceph-mon[32255]: pgmap v3219: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:45:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:49 smithi122 ceph-mon[32255]: pgmap v3220: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:45:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:45:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:49 smithi022 ceph-mon[31360]: pgmap v3220: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:49 smithi022 ceph-mon[40250]: pgmap v3220: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:50.299 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:50] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:45:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:45:50.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:50.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:50.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:51 smithi122 ceph-mon[32255]: pgmap v3221: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1031524339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:51 smithi022 ceph-mon[31360]: pgmap v3221: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1031524339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:51 smithi022 ceph-mon[40250]: pgmap v3221: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1031524339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:45:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:45:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:52 smithi022 ceph-mon[31360]: pgmap v3222: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:52 smithi022 ceph-mon[40250]: pgmap v3222: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:52 smithi122 ceph-mon[32255]: pgmap v3222: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:55 smithi122 ceph-mon[32255]: pgmap v3223: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:55 smithi022 ceph-mon[31360]: pgmap v3223: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:55 smithi022 ceph-mon[40250]: pgmap v3223: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:45:55.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:45:56.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:45:56.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:45:56.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:45:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/772477201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/772477201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/772477201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:45:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:57 smithi022 ceph-mon[31360]: pgmap v3224: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:57 smithi022 ceph-mon[40250]: pgmap v3224: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:57 smithi122 ceph-mon[32255]: pgmap v3224: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:45:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:45:59 smithi122 ceph-mon[32255]: pgmap v3225: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:59.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:45:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:45:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:45:59 smithi022 ceph-mon[31360]: pgmap v3225: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:45:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:45:59 smithi022 ceph-mon[40250]: pgmap v3225: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:00] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:46:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:01.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:01.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:01.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:01.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:01.409 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:01 smithi022 ceph-mon[40250]: pgmap v3226: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:01.410 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:01 smithi022 ceph-mon[31360]: pgmap v3226: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:01 smithi122 ceph-mon[32255]: pgmap v3226: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3453197948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3453197948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3453197948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:46:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:46:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:03 smithi122 ceph-mon[32255]: pgmap v3227: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:03 smithi022 ceph-mon[31360]: pgmap v3227: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:03 smithi022 ceph-mon[40250]: pgmap v3227: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:05 smithi122 ceph-mon[32255]: pgmap v3228: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:05 smithi022 ceph-mon[31360]: pgmap v3228: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:05 smithi022 ceph-mon[40250]: pgmap v3228: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:06.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:06.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:06.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:06.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:07 smithi022 ceph-mon[31360]: pgmap v3229: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2859258995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:07 smithi022 ceph-mon[40250]: pgmap v3229: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2859258995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:07 smithi122 ceph-mon[32255]: pgmap v3229: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2859258995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:46:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:46:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:46:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:09 smithi122 ceph-mon[32255]: pgmap v3230: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:09 smithi022 ceph-mon[31360]: pgmap v3230: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:09 smithi022 ceph-mon[40250]: pgmap v3230: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:10] "GET /metrics HTTP/1.1" 200 202314 "" "Prometheus/2.18.1" 2021-12-15T19:46:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:11 smithi122 ceph-mon[32255]: pgmap v3231: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:11 smithi022 ceph-mon[31360]: pgmap v3231: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:11 smithi022 ceph-mon[40250]: pgmap v3231: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:11.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:12.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:12.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:12.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:46:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:46:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:12 smithi022 ceph-mon[31360]: pgmap v3232: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2175232767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:12 smithi022 ceph-mon[40250]: pgmap v3232: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2175232767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:12 smithi122 ceph-mon[32255]: pgmap v3232: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2175232767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:15 smithi122 ceph-mon[32255]: pgmap v3233: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:15 smithi022 ceph-mon[31360]: pgmap v3233: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:15 smithi022 ceph-mon[40250]: pgmap v3233: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:17.154 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:17.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:17 smithi022 ceph-mon[40250]: pgmap v3234: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:17 smithi022 ceph-mon[31360]: pgmap v3234: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:17 smithi122 ceph-mon[32255]: pgmap v3234: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:17.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:17.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:17.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:46:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1189856250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1189856250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1189856250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:19.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:46:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:46:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:19 smithi122 ceph-mon[32255]: pgmap v3235: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:19 smithi022 ceph-mon[31360]: pgmap v3235: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:19 smithi022 ceph-mon[40250]: pgmap v3235: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:20] "GET /metrics HTTP/1.1" 200 202319 "" "Prometheus/2.18.1" 2021-12-15T19:46:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:20.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:20.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:21.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:21 smithi122 ceph-mon[32255]: pgmap v3236: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:21 smithi022 ceph-mon[31360]: pgmap v3236: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:21 smithi022 ceph-mon[40250]: pgmap v3236: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:22.527 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:46:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:46:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:22 smithi022 ceph-mon[31360]: pgmap v3237: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:22 smithi022 ceph-mon[40250]: pgmap v3237: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:22.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:22.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:22.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:22 smithi122 ceph-mon[32255]: pgmap v3237: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3699686977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3699686977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3699686977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:24 smithi022 ceph-mon[31360]: pgmap v3238: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:24 smithi022 ceph-mon[40250]: pgmap v3238: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:24 smithi122 ceph-mon[32255]: pgmap v3238: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:27 smithi022 ceph-mon[31360]: pgmap v3239: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:27 smithi022 ceph-mon[40250]: pgmap v3239: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:27 smithi122 ceph-mon[32255]: pgmap v3239: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:46:27.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:28.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:46:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:46:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:29 smithi122 ceph-mon[32255]: pgmap v3240: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2611780950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:29 smithi022 ceph-mon[31360]: pgmap v3240: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2611780950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:29 smithi022 ceph-mon[40250]: pgmap v3240: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2611780950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:30] "GET /metrics HTTP/1.1" 200 202319 "" "Prometheus/2.18.1" 2021-12-15T19:46:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:31 smithi122 ceph-mon[32255]: pgmap v3241: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:31 smithi022 ceph-mon[31360]: pgmap v3241: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:31 smithi022 ceph-mon[40250]: pgmap v3241: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:46:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:46:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[31360]: pgmap v3242: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:46:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:46:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:46:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:46:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[40250]: pgmap v3242: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:46:33.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:46:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:46:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:46:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:32 smithi122 ceph-mon[32255]: pgmap v3242: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:46:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:46:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:46:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:46:33.266 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:33.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:33.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:33.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:33.960 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1633406497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:33.960 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1633406497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1633406497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:46:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:46:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:46:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[31360]: pgmap v3243: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:46:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:46:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:46:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[40250]: pgmap v3243: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:46:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:46:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:46:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:34 smithi122 ceph-mon[32255]: pgmap v3243: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:46:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:36 smithi022 ceph-mon[31360]: pgmap v3244: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:36 smithi022 ceph-mon[40250]: pgmap v3244: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:36 smithi122 ceph-mon[32255]: pgmap v3244: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:46:38.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:39.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:39.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:39.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:46:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:46:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:39 smithi122 ceph-mon[32255]: pgmap v3245: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2491637917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:39 smithi022 ceph-mon[31360]: pgmap v3245: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2491637917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:39 smithi022 ceph-mon[40250]: pgmap v3245: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2491637917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:40] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T19:46:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:41 smithi122 ceph-mon[32255]: pgmap v3246: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:41 smithi022 ceph-mon[31360]: pgmap v3246: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:41 smithi022 ceph-mon[40250]: pgmap v3246: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:46:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:46:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:42 smithi022 ceph-mon[31360]: pgmap v3247: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:43.102 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:42 smithi022 ceph-mon[40250]: pgmap v3247: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:42 smithi122 ceph-mon[32255]: pgmap v3247: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:44.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:44.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:44.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:44.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:45 smithi122 ceph-mon[32255]: pgmap v3248: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2852900954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:45 smithi022 ceph-mon[31360]: pgmap v3248: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2852900954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:45 smithi022 ceph-mon[40250]: pgmap v3248: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2852900954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:47 smithi022 ceph-mon[31360]: pgmap v3249: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:47 smithi022 ceph-mon[40250]: pgmap v3249: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:47 smithi122 ceph-mon[32255]: pgmap v3249: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:46:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:49 smithi122 ceph-mon[32255]: pgmap v3250: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:46:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:46:49.382 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:49.383 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:49 smithi022 ceph-mon[31360]: pgmap v3250: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:49.383 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:49 smithi022 ceph-mon[40250]: pgmap v3250: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:49.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:49.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:49.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:50] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:46:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2820335142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2820335142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:46:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2820335142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:51 smithi122 ceph-mon[32255]: pgmap v3251: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:51 smithi022 ceph-mon[31360]: pgmap v3251: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:51 smithi022 ceph-mon[40250]: pgmap v3251: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:46:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:46:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:52 smithi022 ceph-mon[31360]: pgmap v3252: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:52 smithi022 ceph-mon[40250]: pgmap v3252: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:52 smithi122 ceph-mon[32255]: pgmap v3252: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:54.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:46:55.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:46:55.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:46:55.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:46:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:55 smithi122 ceph-mon[32255]: pgmap v3253: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:55 smithi022 ceph-mon[31360]: pgmap v3253: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:55 smithi022 ceph-mon[40250]: pgmap v3253: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:46:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2023779458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2023779458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2023779458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:46:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:57 smithi022 ceph-mon[31360]: pgmap v3254: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:57 smithi022 ceph-mon[40250]: pgmap v3254: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:57 smithi122 ceph-mon[32255]: pgmap v3254: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:46:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:46:59 smithi122 ceph-mon[32255]: pgmap v3255: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:59.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:46:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:46:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:46:59 smithi022 ceph-mon[31360]: pgmap v3255: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:46:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:46:59 smithi022 ceph-mon[40250]: pgmap v3255: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:00.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:00] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T19:47:00.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:00.495 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:00.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:01 smithi122 ceph-mon[32255]: pgmap v3256: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/274153233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:01 smithi022 ceph-mon[31360]: pgmap v3256: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/274153233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:01 smithi022 ceph-mon[40250]: pgmap v3256: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/274153233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:47:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:47:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:02 smithi022 ceph-mon[31360]: pgmap v3257: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:02 smithi022 ceph-mon[40250]: pgmap v3257: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:02 smithi122 ceph-mon[32255]: pgmap v3257: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:05 smithi122 ceph-mon[32255]: pgmap v3258: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:05.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:05.497 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:05 smithi022 ceph-mon[31360]: pgmap v3258: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:05.497 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:05 smithi022 ceph-mon[40250]: pgmap v3258: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:05.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:05.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:05.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3785250099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3785250099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3785250099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:07 smithi022 ceph-mon[31360]: pgmap v3259: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:07 smithi022 ceph-mon[40250]: pgmap v3259: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:07 smithi122 ceph-mon[32255]: pgmap v3259: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:47:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:47:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:47:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:09 smithi122 ceph-mon[32255]: pgmap v3260: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:09 smithi022 ceph-mon[31360]: pgmap v3260: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:09 smithi022 ceph-mon[40250]: pgmap v3260: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:10] "GET /metrics HTTP/1.1" 200 202319 "" "Prometheus/2.18.1" 2021-12-15T19:47:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:10.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:11.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:11.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:11.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:11.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:11 smithi022 ceph-mon[31360]: pgmap v3261: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:11.243 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:11 smithi022 ceph-mon[40250]: pgmap v3261: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:11 smithi122 ceph-mon[32255]: pgmap v3261: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3349588900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:47:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:47:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3349588900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3349588900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:13 smithi122 ceph-mon[32255]: pgmap v3262: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:13 smithi022 ceph-mon[31360]: pgmap v3262: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:13 smithi022 ceph-mon[40250]: pgmap v3262: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:15 smithi122 ceph-mon[32255]: pgmap v3263: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:15 smithi022 ceph-mon[31360]: pgmap v3263: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:15 smithi022 ceph-mon[40250]: pgmap v3263: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:16.244 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:16.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:16.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:16.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:17 smithi022 ceph-mon[31360]: pgmap v3264: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4130220053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:17 smithi022 ceph-mon[40250]: pgmap v3264: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4130220053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:17 smithi122 ceph-mon[32255]: pgmap v3264: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4130220053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:47:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:19 smithi122 ceph-mon[32255]: pgmap v3265: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:19.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:47:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:47:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:19 smithi022 ceph-mon[31360]: pgmap v3265: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:19 smithi022 ceph-mon[40250]: pgmap v3265: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:20] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:47:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:21 smithi122 ceph-mon[32255]: pgmap v3266: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:21 smithi022 ceph-mon[31360]: pgmap v3266: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:21 smithi022 ceph-mon[40250]: pgmap v3266: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:21.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:21.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:21.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:21.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:47:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:47:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/101819711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:22 smithi022 ceph-mon[31360]: pgmap v3267: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/101819711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:22 smithi022 ceph-mon[40250]: pgmap v3267: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/101819711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:22 smithi122 ceph-mon[32255]: pgmap v3267: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:25 smithi122 ceph-mon[32255]: pgmap v3268: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:25 smithi022 ceph-mon[31360]: pgmap v3268: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:25 smithi022 ceph-mon[40250]: pgmap v3268: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:27 smithi022 ceph-mon[31360]: pgmap v3269: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:27 smithi022 ceph-mon[40250]: pgmap v3269: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:27.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:27.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:27.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:27 smithi122 ceph-mon[32255]: pgmap v3269: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:47:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1347585537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1347585537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1347585537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:29 smithi122 ceph-mon[32255]: pgmap v3270: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:47:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:47:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:29 smithi022 ceph-mon[31360]: pgmap v3270: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:29 smithi022 ceph-mon[40250]: pgmap v3270: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:30] "GET /metrics HTTP/1.1" 200 202317 "" "Prometheus/2.18.1" 2021-12-15T19:47:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:31 smithi122 ceph-mon[32255]: pgmap v3271: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:31 smithi022 ceph-mon[31360]: pgmap v3271: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:31 smithi022 ceph-mon[40250]: pgmap v3271: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:32.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:47:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:47:32.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:32.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:32.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[31360]: pgmap v3272: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:47:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:47:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:47:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:47:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/707439356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[40250]: pgmap v3272: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:47:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:47:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:47:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:47:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/707439356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:32 smithi122 ceph-mon[32255]: pgmap v3272: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:47:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:47:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:47:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:47:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/707439356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:35 smithi122 ceph-mon[32255]: pgmap v3273: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:47:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:47:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:47:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[31360]: pgmap v3273: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:47:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:47:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:47:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[40250]: pgmap v3273: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:47:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:47:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:47:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:47:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:47:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:47:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:37 smithi022 ceph-mon[31360]: pgmap v3274: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:47:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:37 smithi022 ceph-mon[40250]: pgmap v3274: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:37 smithi122 ceph-mon[32255]: pgmap v3274: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:37.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:38.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:38.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:38.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1072455921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1072455921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1072455921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:39.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:47:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:47:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:39 smithi022 ceph-mon[31360]: pgmap v3275: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:39 smithi022 ceph-mon[40250]: pgmap v3275: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:39 smithi122 ceph-mon[32255]: pgmap v3275: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:40] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:47:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:41 smithi022 ceph-mon[31360]: pgmap v3276: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:41 smithi022 ceph-mon[40250]: pgmap v3276: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:41 smithi122 ceph-mon[32255]: pgmap v3276: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:47:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:47:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:42 smithi022 ceph-mon[31360]: pgmap v3277: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:42 smithi022 ceph-mon[40250]: pgmap v3277: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:43.097 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:42 smithi122 ceph-mon[32255]: pgmap v3277: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:43.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:43.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:43.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3507989713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3507989713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3507989713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:44 smithi122 ceph-mon[32255]: pgmap v3278: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:44 smithi022 ceph-mon[31360]: pgmap v3278: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:44 smithi022 ceph-mon[40250]: pgmap v3278: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:47 smithi022 ceph-mon[31360]: pgmap v3279: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:47 smithi022 ceph-mon[40250]: pgmap v3279: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:47 smithi122 ceph-mon[32255]: pgmap v3279: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:47:48.471 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:48.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:48.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:48.838 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:49 smithi022 ceph-mon[31360]: pgmap v3280: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/184121962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:49 smithi022 ceph-mon[40250]: pgmap v3280: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/184121962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:49.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:47:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:47:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:49 smithi122 ceph-mon[32255]: pgmap v3280: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/184121962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:50] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:47:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:50.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:47:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:51 smithi122 ceph-mon[32255]: pgmap v3281: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:51 smithi022 ceph-mon[31360]: pgmap v3281: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:51 smithi022 ceph-mon[40250]: pgmap v3281: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:47:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:47:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:47:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:52 smithi022 ceph-mon[31360]: pgmap v3282: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:52 smithi022 ceph-mon[40250]: pgmap v3282: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:52 smithi122 ceph-mon[32255]: pgmap v3282: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:47:53.839 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:54.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:54.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:54.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:47:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:55 smithi122 ceph-mon[32255]: pgmap v3283: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3177598864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:55 smithi022 ceph-mon[31360]: pgmap v3283: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3177598864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:55 smithi022 ceph-mon[40250]: pgmap v3283: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3177598864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:47:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:57 smithi022 ceph-mon[31360]: pgmap v3284: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:57 smithi022 ceph-mon[40250]: pgmap v3284: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:57 smithi122 ceph-mon[32255]: pgmap v3284: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:47:59.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:47:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:47:59 smithi122 ceph-mon[32255]: pgmap v3285: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:47:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:47:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:47:59 smithi022 ceph-mon[31360]: pgmap v3285: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:47:59 smithi022 ceph-mon[40250]: pgmap v3285: 132 pgs: 132 active+clean; 582 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:47:59.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:47:59.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:47:59.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:00] "GET /metrics HTTP/1.1" 200 202318 "" "Prometheus/2.18.1" 2021-12-15T19:48:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2232923899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2232923899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2232923899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:01 smithi122 ceph-mon[32255]: pgmap v3286: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:01 smithi022 ceph-mon[31360]: pgmap v3286: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:01 smithi022 ceph-mon[40250]: pgmap v3286: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:48:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:48:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:02 smithi022 ceph-mon[31360]: pgmap v3287: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:02 smithi022 ceph-mon[40250]: pgmap v3287: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:02 smithi122 ceph-mon[32255]: pgmap v3287: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:04.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:04.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:04.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:04.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:05 smithi022 ceph-mon[31360]: pgmap v3288: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2826933680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:05 smithi022 ceph-mon[40250]: pgmap v3288: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2826933680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:05 smithi122 ceph-mon[32255]: pgmap v3288: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:48:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2826933680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:07 smithi022 ceph-mon[31360]: pgmap v3289: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:48:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:07 smithi022 ceph-mon[40250]: pgmap v3289: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:48:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:07 smithi122 ceph-mon[32255]: pgmap v3289: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:48:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:48:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:09 smithi122 ceph-mon[32255]: pgmap v3290: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:48:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:48:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:48:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:09 smithi022 ceph-mon[31360]: pgmap v3290: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:48:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:09 smithi022 ceph-mon[40250]: pgmap v3290: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:48:09.957 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:10.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:10.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:10.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:10] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T19:48:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:11 smithi122 ceph-mon[32255]: pgmap v3291: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:48:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1175148539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:11 smithi022 ceph-mon[31360]: pgmap v3291: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:48:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1175148539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:11 smithi022 ceph-mon[40250]: pgmap v3291: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:48:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1175148539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:48:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:48:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:12 smithi122 ceph-mon[32255]: pgmap v3292: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:12 smithi022 ceph-mon[31360]: pgmap v3292: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:12 smithi022 ceph-mon[40250]: pgmap v3292: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:15.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:15 smithi122 ceph-mon[32255]: pgmap v3293: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:15 smithi022 ceph-mon[31360]: pgmap v3293: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:15 smithi022 ceph-mon[40250]: pgmap v3293: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:15.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:15.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:15.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1197152169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1197152169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1197152169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:17 smithi022 ceph-mon[31360]: pgmap v3294: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:17 smithi022 ceph-mon[40250]: pgmap v3294: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:17 smithi122 ceph-mon[32255]: pgmap v3294: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:48:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:19 smithi122 ceph-mon[32255]: pgmap v3295: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:48:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:48:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:19 smithi022 ceph-mon[31360]: pgmap v3295: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:19 smithi022 ceph-mon[40250]: pgmap v3295: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:48:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:20.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:21.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:21.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:21.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:21 smithi122 ceph-mon[32255]: pgmap v3296: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1941473253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:21 smithi022 ceph-mon[31360]: pgmap v3296: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1941473253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:21 smithi022 ceph-mon[40250]: pgmap v3296: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1941473253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:48:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:48:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:22 smithi122 ceph-mon[32255]: pgmap v3297: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:22 smithi022 ceph-mon[31360]: pgmap v3297: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:22 smithi022 ceph-mon[40250]: pgmap v3297: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:25 smithi122 ceph-mon[32255]: pgmap v3298: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:25 smithi022 ceph-mon[31360]: pgmap v3298: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:25 smithi022 ceph-mon[40250]: pgmap v3298: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:26.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:26.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:26.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:26.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:27 smithi022 ceph-mon[31360]: pgmap v3299: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2294500860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:27 smithi022 ceph-mon[40250]: pgmap v3299: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2294500860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:27 smithi122 ceph-mon[32255]: pgmap v3299: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2294500860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:48:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:48:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:48:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:29 smithi122 ceph-mon[32255]: pgmap v3300: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:29 smithi022 ceph-mon[31360]: pgmap v3300: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:29 smithi022 ceph-mon[40250]: pgmap v3300: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:48:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:31 smithi122 ceph-mon[32255]: pgmap v3301: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:31.443 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:31.443 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:31 smithi022 ceph-mon[40250]: pgmap v3301: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:31.444 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:31 smithi022 ceph-mon[31360]: pgmap v3301: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:31.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:31.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:31.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:48:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:48:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3208580061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:32 smithi122 ceph-mon[32255]: pgmap v3302: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:48:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:48:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:48:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:48:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3208580061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[31360]: pgmap v3302: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:48:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:48:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:48:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3208580061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[40250]: pgmap v3302: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:48:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:48:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:48:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:35 smithi122 ceph-mon[32255]: pgmap v3303: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:35 smithi022 ceph-mon[31360]: pgmap v3303: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:35 smithi022 ceph-mon[40250]: pgmap v3303: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:48:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:48:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:48:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:48:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:48:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:48:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:48:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:48:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:48:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:48:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:48:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:48:36.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:37.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:37.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:37.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:37 smithi022 ceph-mon[31360]: pgmap v3304: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:37 smithi022 ceph-mon[40250]: pgmap v3304: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:37 smithi122 ceph-mon[32255]: pgmap v3304: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:48:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2011987953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2011987953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2011987953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:39 smithi122 ceph-mon[32255]: pgmap v3305: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:39.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:48:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:48:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:39 smithi022 ceph-mon[31360]: pgmap v3305: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:39 smithi022 ceph-mon[40250]: pgmap v3305: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:40] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T19:48:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:41 smithi122 ceph-mon[32255]: pgmap v3306: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:41 smithi022 ceph-mon[31360]: pgmap v3306: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:41 smithi022 ceph-mon[40250]: pgmap v3306: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:42.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:42.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:42.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:42.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:48:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:48:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:42 smithi022 ceph-mon[31360]: pgmap v3307: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3945290917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:42 smithi022 ceph-mon[40250]: pgmap v3307: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3945290917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:42 smithi122 ceph-mon[32255]: pgmap v3307: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3945290917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:45 smithi122 ceph-mon[32255]: pgmap v3308: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:45 smithi022 ceph-mon[31360]: pgmap v3308: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:45 smithi022 ceph-mon[40250]: pgmap v3308: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:47 smithi022 ceph-mon[31360]: pgmap v3309: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:47 smithi022 ceph-mon[40250]: pgmap v3309: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:47 smithi122 ceph-mon[32255]: pgmap v3309: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:47.553 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:48:47.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:47.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:47.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2904063354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2904063354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2904063354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:49 smithi122 ceph-mon[32255]: pgmap v3310: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:49.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:48:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:48:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:49 smithi022 ceph-mon[31360]: pgmap v3310: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:49 smithi022 ceph-mon[40250]: pgmap v3310: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T19:48:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:48:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:51 smithi122 ceph-mon[32255]: pgmap v3311: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:51.665 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:51 smithi022 ceph-mon[31360]: pgmap v3311: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:51.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:51 smithi022 ceph-mon[40250]: pgmap v3311: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:48:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:48:52.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:52 smithi122 ceph-mon[32255]: pgmap v3312: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:52 smithi022 ceph-mon[31360]: pgmap v3312: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:52 smithi022 ceph-mon[40250]: pgmap v3312: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:53.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:53.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:53.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2731578613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2731578613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2731578613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:54 smithi122 ceph-mon[32255]: pgmap v3313: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:54 smithi022 ceph-mon[31360]: pgmap v3313: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:54 smithi022 ceph-mon[40250]: pgmap v3313: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:48:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:57 smithi022 ceph-mon[31360]: pgmap v3314: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:57 smithi022 ceph-mon[40250]: pgmap v3314: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:57 smithi122 ceph-mon[32255]: pgmap v3314: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:48:58.292 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:48:58.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:48:58.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:48:58.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:48:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:48:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:48:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:59 smithi122 ceph-mon[32255]: pgmap v3315: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:48:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2796132599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:59 smithi022 ceph-mon[31360]: pgmap v3315: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:48:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2796132599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:48:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:59 smithi022 ceph-mon[40250]: pgmap v3315: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:48:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:48:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2796132599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T19:49:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:01 smithi122 ceph-mon[32255]: pgmap v3316: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:01 smithi022 ceph-mon[31360]: pgmap v3316: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:01 smithi022 ceph-mon[40250]: pgmap v3316: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:49:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:49:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:02 smithi122 ceph-mon[32255]: pgmap v3317: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:02 smithi022 ceph-mon[31360]: pgmap v3317: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:02 smithi022 ceph-mon[40250]: pgmap v3317: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:03.666 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3327262943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3327262943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3327262943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:05 smithi122 ceph-mon[32255]: pgmap v3318: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:05 smithi022 ceph-mon[31360]: pgmap v3318: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:05 smithi022 ceph-mon[40250]: pgmap v3318: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:07 smithi022 ceph-mon[31360]: pgmap v3319: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:07 smithi022 ceph-mon[40250]: pgmap v3319: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:07 smithi122 ceph-mon[32255]: pgmap v3319: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:49:09.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:09.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:09.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:09.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:09.406 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:09 smithi022 ceph-mon[31360]: pgmap v3320: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:09.406 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:09 smithi022 ceph-mon[40250]: pgmap v3320: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:09.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:49:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:49:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:09 smithi122 ceph-mon[32255]: pgmap v3320: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:49:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:10] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:49:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1351398089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1351398089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1351398089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:11 smithi122 ceph-mon[32255]: pgmap v3321: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:11 smithi022 ceph-mon[31360]: pgmap v3321: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:11 smithi022 ceph-mon[40250]: pgmap v3321: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:49:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:49:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:12 smithi122 ceph-mon[32255]: pgmap v3322: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:12 smithi022 ceph-mon[31360]: pgmap v3322: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:12 smithi022 ceph-mon[40250]: pgmap v3322: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:14.407 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:14.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:14.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:14.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:15 smithi122 ceph-mon[32255]: pgmap v3323: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/143577763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:15 smithi022 ceph-mon[31360]: pgmap v3323: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/143577763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:15 smithi022 ceph-mon[40250]: pgmap v3323: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/143577763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:17 smithi022 ceph-mon[31360]: pgmap v3324: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:17.633 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:17 smithi022 ceph-mon[40250]: pgmap v3324: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:17.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:17 smithi122 ceph-mon[32255]: pgmap v3324: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:49:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:19 smithi122 ceph-mon[32255]: pgmap v3325: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:49:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:49:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:19 smithi022 ceph-mon[31360]: pgmap v3325: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:19 smithi022 ceph-mon[40250]: pgmap v3325: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:19.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:20.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:20.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:20.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:20] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T19:49:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:21 smithi122 ceph-mon[32255]: pgmap v3326: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/360962031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:21 smithi022 ceph-mon[31360]: pgmap v3326: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/360962031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:21 smithi022 ceph-mon[40250]: pgmap v3326: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/360962031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:49:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:49:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:22 smithi122 ceph-mon[32255]: pgmap v3327: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:22 smithi022 ceph-mon[31360]: pgmap v3327: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:22 smithi022 ceph-mon[40250]: pgmap v3327: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:25.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:25 smithi122 ceph-mon[32255]: pgmap v3328: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:25.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:25.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:25.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:25.515 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:25 smithi022 ceph-mon[40250]: pgmap v3328: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:25.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:25 smithi022 ceph-mon[31360]: pgmap v3328: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4276692401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4276692401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4276692401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:27 smithi022 ceph-mon[31360]: pgmap v3329: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:27 smithi022 ceph-mon[40250]: pgmap v3329: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:27 smithi122 ceph-mon[32255]: pgmap v3329: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:49:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:29 smithi122 ceph-mon[32255]: pgmap v3330: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:49:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:49:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:29 smithi022 ceph-mon[31360]: pgmap v3330: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:29 smithi022 ceph-mon[40250]: pgmap v3330: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:30] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T19:49:30.516 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:30.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:30.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:30.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:30.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:30.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:31 smithi022 ceph-mon[31360]: pgmap v3331: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3217486572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:31 smithi022 ceph-mon[40250]: pgmap v3331: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3217486572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:31 smithi122 ceph-mon[32255]: pgmap v3331: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3217486572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:49:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:49:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:32 smithi122 ceph-mon[32255]: pgmap v3332: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:49:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:49:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:49:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:49:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[31360]: pgmap v3332: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:49:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:49:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:49:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:49:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[40250]: pgmap v3332: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:49:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:49:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:49:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:49:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:35 smithi122 ceph-mon[32255]: pgmap v3333: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:35 smithi022 ceph-mon[31360]: pgmap v3333: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:35 smithi022 ceph-mon[40250]: pgmap v3333: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:35.889 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:36.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:36.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:36.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:49:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:49:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:49:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:49:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:49:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:49:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:49:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:49:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:49:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:49:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:37 smithi022 ceph-mon[31360]: pgmap v3334: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:49:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2133124269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:37 smithi022 ceph-mon[40250]: pgmap v3334: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:49:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2133124269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:37 smithi122 ceph-mon[32255]: pgmap v3334: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:49:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2133124269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:39 smithi122 ceph-mon[32255]: pgmap v3335: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:49:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:49:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:39 smithi022 ceph-mon[31360]: pgmap v3335: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:39 smithi022 ceph-mon[40250]: pgmap v3335: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:40] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T19:49:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:41.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:41 smithi022 ceph-mon[31360]: pgmap v3336: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:41 smithi022 ceph-mon[40250]: pgmap v3336: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:41.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:41.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:41.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:41 smithi122 ceph-mon[32255]: pgmap v3336: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4165175756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4165175756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4165175756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:49:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:49:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:43 smithi022 ceph-mon[31360]: pgmap v3337: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:43 smithi022 ceph-mon[40250]: pgmap v3337: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:43 smithi122 ceph-mon[32255]: pgmap v3337: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:45 smithi022 ceph-mon[31360]: pgmap v3338: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:45 smithi022 ceph-mon[40250]: pgmap v3338: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:45 smithi122 ceph-mon[32255]: pgmap v3338: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:46.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:46.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:46.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:46.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:47 smithi022 ceph-mon[31360]: pgmap v3339: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3003790780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:47 smithi022 ceph-mon[40250]: pgmap v3339: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:47.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3003790780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:47 smithi122 ceph-mon[32255]: pgmap v3339: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3003790780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:49:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:49 smithi122 ceph-mon[32255]: pgmap v3340: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:49.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:49:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:49:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:49 smithi022 ceph-mon[31360]: pgmap v3340: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:49 smithi022 ceph-mon[40250]: pgmap v3340: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:49:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:49:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:51 smithi022 ceph-mon[31360]: pgmap v3341: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:51 smithi022 ceph-mon[40250]: pgmap v3341: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:51 smithi122 ceph-mon[32255]: pgmap v3341: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:52.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:52.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:52.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:49:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:49:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:52 smithi122 ceph-mon[32255]: pgmap v3342: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/942292271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:52 smithi022 ceph-mon[31360]: pgmap v3342: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/942292271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:52 smithi022 ceph-mon[40250]: pgmap v3342: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/942292271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:55 smithi122 ceph-mon[32255]: pgmap v3343: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:55 smithi022 ceph-mon[31360]: pgmap v3343: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:55 smithi022 ceph-mon[40250]: pgmap v3343: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:49:57.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:49:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:57 smithi022 ceph-mon[31360]: pgmap v3344: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:57 smithi022 ceph-mon[40250]: pgmap v3344: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:57 smithi122 ceph-mon[32255]: pgmap v3344: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:57.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:49:57.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:49:57.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:49:57.734 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:49:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2958929784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2958929784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2958929784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:49:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:49:59 smithi122 ceph-mon[32255]: pgmap v3345: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:49:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:49:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:49:59 smithi022 ceph-mon[31360]: pgmap v3345: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:49:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:49:59 smithi022 ceph-mon[40250]: pgmap v3345: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T19:50:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T19:50:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T19:50:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:01 smithi122 ceph-mon[32255]: pgmap v3346: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:01 smithi022 ceph-mon[31360]: pgmap v3346: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:01 smithi022 ceph-mon[40250]: pgmap v3346: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:50:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:50:02.735 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:02 smithi022 ceph-mon[31360]: pgmap v3347: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:02 smithi022 ceph-mon[40250]: pgmap v3347: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:03.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:03.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:03.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:02 smithi122 ceph-mon[32255]: pgmap v3347: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3300983849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3300983849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3300983849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:04 smithi122 ceph-mon[32255]: pgmap v3348: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:04 smithi022 ceph-mon[31360]: pgmap v3348: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:04 smithi022 ceph-mon[40250]: pgmap v3348: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:07 smithi022 ceph-mon[31360]: pgmap v3349: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:07 smithi022 ceph-mon[40250]: pgmap v3349: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:07 smithi122 ceph-mon[32255]: pgmap v3349: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:50:08.107 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:08.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:08.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:08.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:09.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:50:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:50:09.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:09 smithi122 ceph-mon[32255]: pgmap v3350: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:09.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3540475157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:09 smithi022 ceph-mon[31360]: pgmap v3350: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3540475157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:09 smithi022 ceph-mon[40250]: pgmap v3350: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3540475157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:10] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T19:50:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:11 smithi122 ceph-mon[32255]: pgmap v3351: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:11 smithi022 ceph-mon[31360]: pgmap v3351: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:11 smithi022 ceph-mon[40250]: pgmap v3351: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:50:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:50:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:12 smithi122 ceph-mon[32255]: pgmap v3352: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:12 smithi022 ceph-mon[31360]: pgmap v3352: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:12 smithi022 ceph-mon[40250]: pgmap v3352: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:13.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:13.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:13.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:13.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2705092492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2705092492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2705092492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:14 smithi122 ceph-mon[32255]: pgmap v3353: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:14 smithi022 ceph-mon[31360]: pgmap v3353: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:15.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:14 smithi022 ceph-mon[40250]: pgmap v3353: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:17 smithi022 ceph-mon[31360]: pgmap v3354: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:17 smithi022 ceph-mon[40250]: pgmap v3354: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:17 smithi122 ceph-mon[32255]: pgmap v3354: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:50:18.847 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:19.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:19.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:19.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:19 smithi122 ceph-mon[32255]: pgmap v3355: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:50:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:50:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:19 smithi022 ceph-mon[31360]: pgmap v3355: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:19 smithi022 ceph-mon[40250]: pgmap v3355: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3685489592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:50:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3685489592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3685489592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:21 smithi122 ceph-mon[32255]: pgmap v3356: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:21 smithi022 ceph-mon[31360]: pgmap v3356: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:21.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:21 smithi022 ceph-mon[40250]: pgmap v3356: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:50:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:50:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:22 smithi122 ceph-mon[32255]: pgmap v3357: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:22 smithi022 ceph-mon[31360]: pgmap v3357: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:22 smithi022 ceph-mon[40250]: pgmap v3357: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:24.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:24.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:24.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:24.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:25 smithi122 ceph-mon[32255]: pgmap v3358: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1444965637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:25 smithi022 ceph-mon[31360]: pgmap v3358: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1444965637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:25 smithi022 ceph-mon[40250]: pgmap v3358: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1444965637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:27 smithi022 ceph-mon[31360]: pgmap v3359: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:27 smithi022 ceph-mon[40250]: pgmap v3359: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:27 smithi122 ceph-mon[32255]: pgmap v3359: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:50:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:29 smithi122 ceph-mon[32255]: pgmap v3360: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:50:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:50:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:29 smithi022 ceph-mon[31360]: pgmap v3360: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:29 smithi022 ceph-mon[40250]: pgmap v3360: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:29.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:29.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:29.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:29.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3605952742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3605952742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:50:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3605952742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:31 smithi122 ceph-mon[32255]: pgmap v3361: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:31 smithi022 ceph-mon[31360]: pgmap v3361: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:31 smithi022 ceph-mon[40250]: pgmap v3361: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:50:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:50:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:32 smithi122 ceph-mon[32255]: pgmap v3362: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:50:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:50:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:50:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:50:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[31360]: pgmap v3362: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:50:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:50:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:50:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:50:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[40250]: pgmap v3362: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:50:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:50:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:50:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:50:34.966 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:35 smithi022 ceph-mon[31360]: pgmap v3363: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:35 smithi022 ceph-mon[40250]: pgmap v3363: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:35.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:35.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:35.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:35 smithi122 ceph-mon[32255]: pgmap v3363: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3192170455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3192170455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3192170455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[31360]: pgmap v3364: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:50:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:50:37.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:50:37.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[40250]: pgmap v3364: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:37.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:50:37.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:50:37.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:50:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:37 smithi122 ceph-mon[32255]: pgmap v3364: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:50:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:50:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:50:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:50:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:50:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:50:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:39 smithi122 ceph-mon[32255]: pgmap v3365: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:39 smithi022 ceph-mon[31360]: pgmap v3365: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:39 smithi022 ceph-mon[40250]: pgmap v3365: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:40.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:40] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:50:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:40.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:40.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:40.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:41 smithi122 ceph-mon[32255]: pgmap v3366: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4224740062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:41 smithi022 ceph-mon[31360]: pgmap v3366: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4224740062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:41 smithi022 ceph-mon[40250]: pgmap v3366: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4224740062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:50:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:50:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:50:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:43 smithi122 ceph-mon[32255]: pgmap v3367: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:43 smithi022 ceph-mon[31360]: pgmap v3367: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:43 smithi022 ceph-mon[40250]: pgmap v3367: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:45 smithi122 ceph-mon[32255]: pgmap v3368: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:45.693 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:45.693 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:45 smithi022 ceph-mon[31360]: pgmap v3368: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:45.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:45 smithi022 ceph-mon[40250]: pgmap v3368: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:46.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:46.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:46.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:46.349 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4167449220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4167449220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4167449220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:47 smithi122 ceph-mon[32255]: pgmap v3369: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:47 smithi022 ceph-mon[31360]: pgmap v3369: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:50:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:47 smithi022 ceph-mon[40250]: pgmap v3369: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:49 smithi122 ceph-mon[32255]: pgmap v3370: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:49.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:50:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:50:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:49 smithi022 ceph-mon[31360]: pgmap v3370: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:49 smithi022 ceph-mon[40250]: pgmap v3370: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:50] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T19:50:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:50:51.066 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:51.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:51.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:51.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:51 smithi122 ceph-mon[32255]: pgmap v3371: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:51 smithi022 ceph-mon[31360]: pgmap v3371: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:51 smithi022 ceph-mon[40250]: pgmap v3371: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3178054537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:50:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:50:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3178054537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3178054537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:53 smithi122 ceph-mon[32255]: pgmap v3372: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:53 smithi022 ceph-mon[31360]: pgmap v3372: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:53 smithi022 ceph-mon[40250]: pgmap v3372: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:55 smithi122 ceph-mon[32255]: pgmap v3373: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:55 smithi022 ceph-mon[31360]: pgmap v3373: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:55 smithi022 ceph-mon[40250]: pgmap v3373: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:50:56.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:50:56.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:50:56.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:50:56.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:50:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:57 smithi122 ceph-mon[32255]: pgmap v3374: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4051012604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:50:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:57 smithi022 ceph-mon[31360]: pgmap v3374: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4051012604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:57 smithi022 ceph-mon[40250]: pgmap v3374: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4051012604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:50:59.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:50:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:50:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:50:59 smithi122 ceph-mon[32255]: pgmap v3375: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:50:59 smithi022 ceph-mon[31360]: pgmap v3375: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:50:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:50:59 smithi022 ceph-mon[40250]: pgmap v3375: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:00] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T19:51:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:01 smithi122 ceph-mon[32255]: pgmap v3376: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:01 smithi022 ceph-mon[31360]: pgmap v3376: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:01 smithi022 ceph-mon[40250]: pgmap v3376: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:01.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:02.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:02.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:02.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:51:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:51:03.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:02 smithi122 ceph-mon[32255]: pgmap v3377: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:03.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3780547568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:02 smithi022 ceph-mon[31360]: pgmap v3377: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3780547568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:02 smithi022 ceph-mon[40250]: pgmap v3377: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3780547568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:05 smithi122 ceph-mon[32255]: pgmap v3378: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:05 smithi022 ceph-mon[31360]: pgmap v3378: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:05 smithi022 ceph-mon[40250]: pgmap v3378: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:07.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:07 smithi022 ceph-mon[31360]: pgmap v3379: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:07 smithi022 ceph-mon[40250]: pgmap v3379: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:07 smithi122 ceph-mon[32255]: pgmap v3379: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:07.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:07.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:07.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:51:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/604217755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/604217755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/604217755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:09 smithi122 ceph-mon[32255]: pgmap v3380: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:51:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:51:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:09 smithi022 ceph-mon[31360]: pgmap v3380: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:09 smithi022 ceph-mon[40250]: pgmap v3380: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:10] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T19:51:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:11 smithi122 ceph-mon[32255]: pgmap v3381: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:11 smithi022 ceph-mon[31360]: pgmap v3381: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:11 smithi022 ceph-mon[40250]: pgmap v3381: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:12.556 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:51:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:51:12.817 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:12 smithi022 ceph-mon[40250]: pgmap v3382: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:12.818 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:12 smithi022 ceph-mon[31360]: pgmap v3382: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:12.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:12.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:12.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:12 smithi122 ceph-mon[32255]: pgmap v3382: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3773346025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3773346025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3773346025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:14 smithi122 ceph-mon[32255]: pgmap v3383: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:14 smithi022 ceph-mon[31360]: pgmap v3383: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:14 smithi022 ceph-mon[40250]: pgmap v3383: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:17 smithi022 ceph-mon[31360]: pgmap v3384: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:17 smithi022 ceph-mon[40250]: pgmap v3384: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:17 smithi122 ceph-mon[32255]: pgmap v3384: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:51:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:18.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:18.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:18.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:51:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:51:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:19 smithi122 ceph-mon[32255]: pgmap v3385: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3220768854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:19 smithi022 ceph-mon[31360]: pgmap v3385: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3220768854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:19 smithi022 ceph-mon[40250]: pgmap v3385: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3220768854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:20] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T19:51:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:21 smithi122 ceph-mon[32255]: pgmap v3386: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:21 smithi022 ceph-mon[31360]: pgmap v3386: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:21 smithi022 ceph-mon[40250]: pgmap v3386: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:51:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:51:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:22 smithi122 ceph-mon[32255]: pgmap v3387: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:22 smithi022 ceph-mon[31360]: pgmap v3387: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:22 smithi022 ceph-mon[40250]: pgmap v3387: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:23.303 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:23.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:23.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:23.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3895878520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3895878520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3895878520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:24 smithi122 ceph-mon[32255]: pgmap v3388: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:24 smithi022 ceph-mon[31360]: pgmap v3388: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:24 smithi022 ceph-mon[40250]: pgmap v3388: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:27 smithi122 ceph-mon[32255]: pgmap v3389: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:27 smithi022 ceph-mon[31360]: pgmap v3389: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:51:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:27 smithi022 ceph-mon[40250]: pgmap v3389: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:28.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:51:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:51:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:29 smithi122 ceph-mon[32255]: pgmap v3390: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1858717334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:29 smithi022 ceph-mon[31360]: pgmap v3390: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1858717334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:29 smithi022 ceph-mon[40250]: pgmap v3390: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1858717334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:30] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T19:51:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:31 smithi122 ceph-mon[32255]: pgmap v3391: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:31 smithi022 ceph-mon[31360]: pgmap v3391: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:31 smithi022 ceph-mon[40250]: pgmap v3391: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:51:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:51:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:32 smithi122 ceph-mon[32255]: pgmap v3392: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:51:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:51:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:51:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:51:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[31360]: pgmap v3392: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:51:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:51:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:51:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:51:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[40250]: pgmap v3392: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:51:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:51:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:51:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:51:34.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:34.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:34.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:35 smithi122 ceph-mon[32255]: pgmap v3393: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1652867953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:35 smithi022 ceph-mon[31360]: pgmap v3393: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1652867953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:35 smithi022 ceph-mon[40250]: pgmap v3393: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1652867953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:37 smithi122 ceph-mon[32255]: pgmap v3394: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:37 smithi022 ceph-mon[31360]: pgmap v3394: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:51:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:37 smithi022 ceph-mon[40250]: pgmap v3394: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:39.414 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:39 smithi022 ceph-mon[31360]: pgmap v3395: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:39.414 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:39 smithi022 ceph-mon[40250]: pgmap v3395: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:39 smithi122 ceph-mon[32255]: pgmap v3395: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:39.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:51:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:51:39.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:39.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:39.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:40.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:40.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1150117515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:40] "GET /metrics HTTP/1.1" 200 202337 "" "Prometheus/2.18.1" 2021-12-15T19:51:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1150117515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1150117515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:41.410 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:41 smithi122 ceph-mon[32255]: pgmap v3396: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:41.420 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:41 smithi022 ceph-mon[31360]: pgmap v3396: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:41.421 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:41 smithi022 ceph-mon[40250]: pgmap v3396: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:51:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:51:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:51:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:51:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:51:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:51:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:51:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:51:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:51:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:51:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:51:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:51:42.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:51:42.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:51:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:43 smithi122 ceph-mon[32255]: pgmap v3397: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:43 smithi022 ceph-mon[31360]: pgmap v3397: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:43 smithi022 ceph-mon[40250]: pgmap v3397: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:44.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:45 smithi122 ceph-mon[32255]: pgmap v3398: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3424717681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:45 smithi022 ceph-mon[31360]: pgmap v3398: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3424717681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:45 smithi022 ceph-mon[40250]: pgmap v3398: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3424717681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:47 smithi122 ceph-mon[32255]: pgmap v3399: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:47 smithi022 ceph-mon[31360]: pgmap v3399: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:51:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:47 smithi022 ceph-mon[40250]: pgmap v3399: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:49 smithi122 ceph-mon[32255]: pgmap v3400: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:49.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:51:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:51:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:49 smithi022 ceph-mon[31360]: pgmap v3400: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:49 smithi022 ceph-mon[40250]: pgmap v3400: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:50.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:51:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:51:50.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:50.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:50.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:51 smithi122 ceph-mon[32255]: pgmap v3401: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3544307339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:51 smithi022 ceph-mon[31360]: pgmap v3401: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3544307339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:51 smithi022 ceph-mon[40250]: pgmap v3401: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3544307339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:51:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:51:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:52 smithi122 ceph-mon[32255]: pgmap v3402: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:52 smithi022 ceph-mon[31360]: pgmap v3402: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:52 smithi022 ceph-mon[40250]: pgmap v3402: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:55 smithi122 ceph-mon[32255]: pgmap v3403: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:55 smithi022 ceph-mon[31360]: pgmap v3403: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:55 smithi022 ceph-mon[40250]: pgmap v3403: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:51:55.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:51:55.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:51:55.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:51:55.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:51:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1627556916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1627556916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1627556916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:51:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:57 smithi122 ceph-mon[32255]: pgmap v3404: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:57 smithi022 ceph-mon[31360]: pgmap v3404: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:51:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:57 smithi022 ceph-mon[40250]: pgmap v3404: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:51:59 smithi122 ceph-mon[32255]: pgmap v3405: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:51:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:51:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:51:59 smithi022 ceph-mon[31360]: pgmap v3405: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:51:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:51:59 smithi022 ceph-mon[40250]: pgmap v3405: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:52:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:00.904 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:01.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:01.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:01.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:01 smithi122 ceph-mon[32255]: pgmap v3406: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:01 smithi022 ceph-mon[31360]: pgmap v3406: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:01 smithi022 ceph-mon[40250]: pgmap v3406: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/464238679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:52:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:52:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/464238679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/464238679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:03 smithi122 ceph-mon[32255]: pgmap v3407: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:03 smithi022 ceph-mon[31360]: pgmap v3407: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:03 smithi022 ceph-mon[40250]: pgmap v3407: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:05 smithi122 ceph-mon[32255]: pgmap v3408: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:05 smithi022 ceph-mon[31360]: pgmap v3408: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:05 smithi022 ceph-mon[40250]: pgmap v3408: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:06.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:06.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:06.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:06.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:07 smithi122 ceph-mon[32255]: pgmap v3409: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1915848679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:52:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:07 smithi022 ceph-mon[31360]: pgmap v3409: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1915848679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:07 smithi022 ceph-mon[40250]: pgmap v3409: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1915848679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:52:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:52:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:09 smithi122 ceph-mon[32255]: pgmap v3410: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:09 smithi022 ceph-mon[31360]: pgmap v3410: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:09 smithi022 ceph-mon[40250]: pgmap v3410: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:10] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T19:52:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:11 smithi122 ceph-mon[32255]: pgmap v3411: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:11 smithi022 ceph-mon[31360]: pgmap v3411: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:11 smithi022 ceph-mon[40250]: pgmap v3411: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:11.643 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:12.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:12.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:12.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:52:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:52:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/583067871' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:12 smithi122 ceph-mon[32255]: pgmap v3412: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/583067871' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:12 smithi022 ceph-mon[31360]: pgmap v3412: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/583067871' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:12 smithi022 ceph-mon[40250]: pgmap v3412: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:15 smithi122 ceph-mon[32255]: pgmap v3413: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:15 smithi022 ceph-mon[31360]: pgmap v3413: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:15 smithi022 ceph-mon[40250]: pgmap v3413: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:17 smithi022 ceph-mon[31360]: pgmap v3414: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:17 smithi022 ceph-mon[40250]: pgmap v3414: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:17.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:17.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:17.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:17 smithi122 ceph-mon[32255]: pgmap v3414: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:52:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/216653440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/216653440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/216653440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:19 smithi122 ceph-mon[32255]: pgmap v3415: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:52:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:52:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:19 smithi022 ceph-mon[31360]: pgmap v3415: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:19 smithi022 ceph-mon[40250]: pgmap v3415: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:20] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:52:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:21 smithi122 ceph-mon[32255]: pgmap v3416: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:21 smithi022 ceph-mon[31360]: pgmap v3416: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:21 smithi022 ceph-mon[40250]: pgmap v3416: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:22.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:52:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:52:22.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:22.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:22.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:22 smithi022 ceph-mon[31360]: pgmap v3417: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/208010993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:22 smithi022 ceph-mon[40250]: pgmap v3417: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/208010993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:22 smithi122 ceph-mon[32255]: pgmap v3417: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/208010993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:25 smithi122 ceph-mon[32255]: pgmap v3418: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:25 smithi022 ceph-mon[31360]: pgmap v3418: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:25 smithi022 ceph-mon[40250]: pgmap v3418: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:27 smithi122 ceph-mon[32255]: pgmap v3419: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:27 smithi022 ceph-mon[31360]: pgmap v3419: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:52:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:27 smithi022 ceph-mon[40250]: pgmap v3419: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:27.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:28.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:28.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:28.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2414848109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2414848109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2414848109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:29 smithi122 ceph-mon[32255]: pgmap v3420: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:52:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:52:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:29 smithi022 ceph-mon[31360]: pgmap v3420: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:29 smithi022 ceph-mon[40250]: pgmap v3420: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:30] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:52:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:31 smithi122 ceph-mon[32255]: pgmap v3421: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:31 smithi022 ceph-mon[31360]: pgmap v3421: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:31 smithi022 ceph-mon[40250]: pgmap v3421: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:52:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:52:33.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:33.136 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[40250]: pgmap v3422: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:33.137 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:52:33.137 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:52:33.137 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:52:33.137 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:52:33.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[31360]: pgmap v3422: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:33.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:52:33.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:52:33.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:52:33.139 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:52:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:32 smithi122 ceph-mon[32255]: pgmap v3422: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:52:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:52:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:52:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:52:33.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:33.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:33.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3990326257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3990326257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3990326257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:34 smithi122 ceph-mon[32255]: pgmap v3423: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:34 smithi022 ceph-mon[31360]: pgmap v3423: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:34 smithi022 ceph-mon[40250]: pgmap v3423: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:37 smithi122 ceph-mon[32255]: pgmap v3424: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:37 smithi022 ceph-mon[31360]: pgmap v3424: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:52:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:37 smithi022 ceph-mon[40250]: pgmap v3424: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:38.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:38.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:38.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:38.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:52:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:52:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:39 smithi122 ceph-mon[32255]: pgmap v3425: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2099071605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:39 smithi022 ceph-mon[40250]: pgmap v3425: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2099071605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:39 smithi022 ceph-mon[31360]: pgmap v3425: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2099071605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:40] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T19:52:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:41 smithi122 ceph-mon[32255]: pgmap v3426: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:41 smithi022 ceph-mon[31360]: pgmap v3426: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:41 smithi022 ceph-mon[40250]: pgmap v3426: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:42.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:52:42.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:52:42.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:52:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:52:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:52:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:52:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:52:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:52:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:52:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:52:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:43 smithi022 ceph-mon[31360]: pgmap v3427: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:52:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:43 smithi022 ceph-mon[40250]: pgmap v3427: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:52:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:43 smithi122 ceph-mon[32255]: pgmap v3427: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:52:43.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:44.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:44.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:44.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3383005514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3383005514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3383005514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:45 smithi122 ceph-mon[32255]: pgmap v3428: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:45 smithi022 ceph-mon[31360]: pgmap v3428: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:45 smithi022 ceph-mon[40250]: pgmap v3428: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:47 smithi122 ceph-mon[32255]: pgmap v3429: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:47 smithi022 ceph-mon[31360]: pgmap v3429: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:52:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:47 smithi022 ceph-mon[40250]: pgmap v3429: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:49.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:49 smithi022 ceph-mon[31360]: pgmap v3430: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:49 smithi022 ceph-mon[40250]: pgmap v3430: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:49.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:49.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:49.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:49 smithi122 ceph-mon[32255]: pgmap v3430: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:52:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:52:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T19:52:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3470334576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3470334576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:52:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3470334576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:51 smithi122 ceph-mon[32255]: pgmap v3431: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:51 smithi022 ceph-mon[31360]: pgmap v3431: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:51 smithi022 ceph-mon[40250]: pgmap v3431: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:52:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:52:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:52 smithi122 ceph-mon[32255]: pgmap v3432: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:52 smithi022 ceph-mon[31360]: pgmap v3432: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:52 smithi022 ceph-mon[40250]: pgmap v3432: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:54.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:52:54.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:52:54.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:52:54.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:52:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:55 smithi022 ceph-mon[31360]: pgmap v3433: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/760759694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:55 smithi022 ceph-mon[40250]: pgmap v3433: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/760759694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:55 smithi122 ceph-mon[32255]: pgmap v3433: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/760759694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:52:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:57 smithi122 ceph-mon[32255]: pgmap v3434: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:57 smithi022 ceph-mon[31360]: pgmap v3434: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:57 smithi022 ceph-mon[40250]: pgmap v3434: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:52:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:52:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:52:59 smithi122 ceph-mon[32255]: pgmap v3435: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:52:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:52:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:52:59 smithi022 ceph-mon[31360]: pgmap v3435: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:52:59 smithi022 ceph-mon[40250]: pgmap v3435: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:52:59.991 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:00.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:00.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:00.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:00.457 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T19:53:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:01 smithi122 ceph-mon[32255]: pgmap v3436: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1958293663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:01 smithi022 ceph-mon[31360]: pgmap v3436: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:01.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1958293663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:01 smithi022 ceph-mon[40250]: pgmap v3436: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1958293663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:53:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:53:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:02 smithi122 ceph-mon[32255]: pgmap v3437: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:02 smithi022 ceph-mon[31360]: pgmap v3437: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:02 smithi022 ceph-mon[40250]: pgmap v3437: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:05.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:05 smithi122 ceph-mon[32255]: pgmap v3438: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:05 smithi022 ceph-mon[31360]: pgmap v3438: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:05 smithi022 ceph-mon[40250]: pgmap v3438: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:05.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:05.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:05.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3912757257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3912757257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3912757257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:07.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:07 smithi122 ceph-mon[32255]: pgmap v3439: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:07 smithi022 ceph-mon[40250]: pgmap v3439: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:07 smithi022 ceph-mon[31360]: pgmap v3439: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:53:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:53:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:53:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:09 smithi122 ceph-mon[32255]: pgmap v3440: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:09 smithi022 ceph-mon[31360]: pgmap v3440: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:09 smithi022 ceph-mon[40250]: pgmap v3440: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:10] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T19:53:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:10.735 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:11 smithi022 ceph-mon[40250]: pgmap v3441: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1292781162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:11 smithi022 ceph-mon[31360]: pgmap v3441: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1292781162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:11 smithi122 ceph-mon[32255]: pgmap v3441: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1292781162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:53:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:53:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:12 smithi122 ceph-mon[32255]: pgmap v3442: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:12 smithi022 ceph-mon[31360]: pgmap v3442: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:12 smithi022 ceph-mon[40250]: pgmap v3442: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:15 smithi122 ceph-mon[32255]: pgmap v3443: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:15 smithi022 ceph-mon[31360]: pgmap v3443: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:15 smithi022 ceph-mon[40250]: pgmap v3443: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:16.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:16.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:16.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:17 smithi122 ceph-mon[32255]: pgmap v3444: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2527109524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:53:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:17 smithi022 ceph-mon[31360]: pgmap v3444: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:17.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2527109524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:17 smithi022 ceph-mon[40250]: pgmap v3444: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:17.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2527109524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:19 smithi122 ceph-mon[32255]: pgmap v3445: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:53:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:53:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:19 smithi022 ceph-mon[31360]: pgmap v3445: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:19 smithi022 ceph-mon[40250]: pgmap v3445: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:53:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:21.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:21.474 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:21 smithi022 ceph-mon[40250]: pgmap v3446: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:21.474 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:21 smithi022 ceph-mon[31360]: pgmap v3446: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:21 smithi122 ceph-mon[32255]: pgmap v3446: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:21.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:21.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:21.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2823350563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2823350563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2823350563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:53:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:53:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:23 smithi022 ceph-mon[31360]: pgmap v3447: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:23 smithi022 ceph-mon[40250]: pgmap v3447: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:23 smithi122 ceph-mon[32255]: pgmap v3447: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:25 smithi022 ceph-mon[31360]: pgmap v3448: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:25 smithi022 ceph-mon[40250]: pgmap v3448: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:25 smithi122 ceph-mon[32255]: pgmap v3448: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:26.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:27.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:27 smithi022 ceph-mon[31360]: pgmap v3449: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:27.196 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:27 smithi022 ceph-mon[40250]: pgmap v3449: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:27.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:27.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:27.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:53:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:27 smithi122 ceph-mon[32255]: pgmap v3449: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2201045772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2201045772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2201045772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:29 smithi022 ceph-mon[31360]: pgmap v3450: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:29 smithi022 ceph-mon[40250]: pgmap v3450: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:53:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:53:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:29 smithi122 ceph-mon[32255]: pgmap v3450: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:53:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:31 smithi022 ceph-mon[31360]: pgmap v3451: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:31 smithi022 ceph-mon[40250]: pgmap v3451: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:31 smithi122 ceph-mon[32255]: pgmap v3451: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:32.214 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:32.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:32.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:32.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:53:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:53:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:32 smithi122 ceph-mon[32255]: pgmap v3452: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1606796019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:53:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:53:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:53:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:53:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[31360]: pgmap v3452: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1606796019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:53:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:53:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:53:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:53:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[40250]: pgmap v3452: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1606796019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:53:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:53:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:53:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:53:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:35 smithi022 ceph-mon[31360]: pgmap v3453: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:35 smithi022 ceph-mon[40250]: pgmap v3453: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:35 smithi122 ceph-mon[32255]: pgmap v3453: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:37 smithi022 ceph-mon[31360]: pgmap v3454: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:53:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:37 smithi022 ceph-mon[40250]: pgmap v3454: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:37.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:37 smithi122 ceph-mon[32255]: pgmap v3454: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:37.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:37.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:37.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2376928884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2376928884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2376928884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:39 smithi022 ceph-mon[31360]: pgmap v3455: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:39 smithi022 ceph-mon[40250]: pgmap v3455: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:39 smithi122 ceph-mon[32255]: pgmap v3455: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:53:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:53:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:40] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T19:53:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:41 smithi022 ceph-mon[31360]: pgmap v3456: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:41 smithi022 ceph-mon[40250]: pgmap v3456: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:41 smithi122 ceph-mon[32255]: pgmap v3456: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:53:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:53:42.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[31360]: pgmap v3457: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:53:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:53:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:53:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:53:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[40250]: pgmap v3457: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:53:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:53:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:53:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:53:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:42 smithi122 ceph-mon[32255]: pgmap v3457: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:53:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:53:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:53:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:53:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2390739417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2390739417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2390739417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:44 smithi122 ceph-mon[32255]: pgmap v3458: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:44 smithi022 ceph-mon[40250]: pgmap v3458: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:44 smithi022 ceph-mon[31360]: pgmap v3458: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:47 smithi022 ceph-mon[31360]: pgmap v3459: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:53:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:47 smithi022 ceph-mon[40250]: pgmap v3459: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:47 smithi122 ceph-mon[32255]: pgmap v3459: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:48.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:48.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:48.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:48.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:49 smithi022 ceph-mon[31360]: pgmap v3460: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1910031489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:49 smithi022 ceph-mon[40250]: pgmap v3460: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1910031489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:53:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:53:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:49 smithi122 ceph-mon[32255]: pgmap v3460: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1910031489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:50] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T19:53:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:53:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:51 smithi022 ceph-mon[31360]: pgmap v3461: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:51 smithi022 ceph-mon[40250]: pgmap v3461: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:51 smithi122 ceph-mon[32255]: pgmap v3461: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:53:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:53:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:52 smithi122 ceph-mon[32255]: pgmap v3462: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:52 smithi022 ceph-mon[31360]: pgmap v3462: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:52 smithi022 ceph-mon[40250]: pgmap v3462: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:53.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:54.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:54.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:54.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3281308721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3281308721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3281308721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:53:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:55 smithi122 ceph-mon[32255]: pgmap v3463: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:55 smithi022 ceph-mon[31360]: pgmap v3463: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:55 smithi022 ceph-mon[40250]: pgmap v3463: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:57 smithi022 ceph-mon[31360]: pgmap v3464: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:53:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:57 smithi022 ceph-mon[40250]: pgmap v3464: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:57 smithi122 ceph-mon[32255]: pgmap v3464: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:53:59.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:53:59.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:53:59.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:53:59.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:53:59.429 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:53:59 smithi022 ceph-mon[31360]: pgmap v3465: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:59.429 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:53:59 smithi022 ceph-mon[40250]: pgmap v3465: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:53:59 smithi122 ceph-mon[32255]: pgmap v3465: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:53:59.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:53:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:54:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:00] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T19:54:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2146278021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2146278021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2146278021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:01 smithi022 ceph-mon[31360]: pgmap v3466: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:01 smithi022 ceph-mon[40250]: pgmap v3466: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:01 smithi122 ceph-mon[32255]: pgmap v3466: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:54:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:54:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:02 smithi122 ceph-mon[32255]: pgmap v3467: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:02 smithi022 ceph-mon[31360]: pgmap v3467: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:02 smithi022 ceph-mon[40250]: pgmap v3467: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:04.430 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:04.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:04.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:04.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:05 smithi022 ceph-mon[31360]: pgmap v3468: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4150650430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:05 smithi022 ceph-mon[40250]: pgmap v3468: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4150650430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:05 smithi122 ceph-mon[32255]: pgmap v3468: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4150650430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:07 smithi022 ceph-mon[31360]: pgmap v3469: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:54:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:07 smithi022 ceph-mon[40250]: pgmap v3469: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:07 smithi122 ceph-mon[32255]: pgmap v3469: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:09 smithi022 ceph-mon[31360]: pgmap v3470: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:09 smithi022 ceph-mon[40250]: pgmap v3470: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:54:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:54:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:09 smithi122 ceph-mon[32255]: pgmap v3470: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:09.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:10.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:10.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:10.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:10] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T19:54:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4006811139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4006811139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4006811139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:11 smithi022 ceph-mon[31360]: pgmap v3471: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:11 smithi022 ceph-mon[40250]: pgmap v3471: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:11 smithi122 ceph-mon[32255]: pgmap v3471: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:54:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:54:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:12 smithi122 ceph-mon[32255]: pgmap v3472: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:12 smithi022 ceph-mon[31360]: pgmap v3472: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:12 smithi022 ceph-mon[40250]: pgmap v3472: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:15.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:15 smithi022 ceph-mon[31360]: pgmap v3473: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:15 smithi022 ceph-mon[40250]: pgmap v3473: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:15.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:15.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:15.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:15 smithi122 ceph-mon[32255]: pgmap v3473: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1656429111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1656429111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1656429111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:17 smithi022 ceph-mon[31360]: pgmap v3474: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:54:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:17 smithi022 ceph-mon[40250]: pgmap v3474: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:17 smithi122 ceph-mon[32255]: pgmap v3474: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:19 smithi022 ceph-mon[31360]: pgmap v3475: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:19 smithi022 ceph-mon[40250]: pgmap v3475: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:54:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:54:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:19 smithi122 ceph-mon[32255]: pgmap v3475: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:20] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T19:54:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:20.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:20.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:20.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:20.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:21 smithi022 ceph-mon[31360]: pgmap v3476: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1287742461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:21 smithi022 ceph-mon[40250]: pgmap v3476: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1287742461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:21 smithi122 ceph-mon[32255]: pgmap v3476: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1287742461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:54:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:54:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:22 smithi122 ceph-mon[32255]: pgmap v3477: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:22 smithi022 ceph-mon[31360]: pgmap v3477: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:22 smithi022 ceph-mon[40250]: pgmap v3477: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:25 smithi022 ceph-mon[31360]: pgmap v3478: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:25 smithi022 ceph-mon[40250]: pgmap v3478: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:25 smithi122 ceph-mon[32255]: pgmap v3478: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:25.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:26.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:26.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:26.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:54:27.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:27 smithi022 ceph-mon[31360]: pgmap v3479: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:27.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1606917242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:27 smithi022 ceph-mon[40250]: pgmap v3479: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1606917242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:27 smithi122 ceph-mon[32255]: pgmap v3479: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1606917242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:29 smithi022 ceph-mon[31360]: pgmap v3480: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:29 smithi022 ceph-mon[40250]: pgmap v3480: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:29 smithi122 ceph-mon[32255]: pgmap v3480: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:54:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:54:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:30] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T19:54:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:31.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:31 smithi022 ceph-mon[31360]: pgmap v3481: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:31 smithi022 ceph-mon[40250]: pgmap v3481: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:31.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:31.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:31.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:31 smithi122 ceph-mon[32255]: pgmap v3481: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:32.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/991864707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:32.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/991864707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/991864707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:54:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:54:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[31360]: pgmap v3482: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:33.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:54:33.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:54:33.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:54:33.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:54:33.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[40250]: pgmap v3482: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:33.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:54:33.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:54:33.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:54:33.757 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:54:33.757 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:33 smithi122 ceph-mon[32255]: pgmap v3482: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:33.758 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:54:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:35 smithi022 ceph-mon[31360]: pgmap v3483: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:35 smithi022 ceph-mon[40250]: pgmap v3483: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:35 smithi122 ceph-mon[32255]: pgmap v3483: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:36.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:37.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:37.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:37.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:37 smithi022 ceph-mon[31360]: pgmap v3484: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/192758356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:37 smithi022 ceph-mon[40250]: pgmap v3484: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/192758356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:37 smithi122 ceph-mon[32255]: pgmap v3484: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/192758356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:54:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:39 smithi022 ceph-mon[31360]: pgmap v3485: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:39 smithi022 ceph-mon[40250]: pgmap v3485: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:39 smithi122 ceph-mon[32255]: pgmap v3485: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:54:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:54:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:40] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T19:54:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:41 smithi022 ceph-mon[31360]: pgmap v3486: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:41 smithi022 ceph-mon[40250]: pgmap v3486: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:41 smithi122 ceph-mon[32255]: pgmap v3486: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:42.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:42.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:54:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:54:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[31360]: pgmap v3487: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3695692865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:54:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:54:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:54:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[40250]: pgmap v3487: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3695692865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:54:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:54:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:54:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:42 smithi122 ceph-mon[32255]: pgmap v3487: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3695692865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:54:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:54:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:54:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:54:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:54:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:54:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:45 smithi022 ceph-mon[31360]: pgmap v3488: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:45 smithi022 ceph-mon[40250]: pgmap v3488: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:45 smithi122 ceph-mon[32255]: pgmap v3488: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:47.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:47 smithi022 ceph-mon[31360]: pgmap v3489: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:54:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:47 smithi022 ceph-mon[40250]: pgmap v3489: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:47 smithi122 ceph-mon[32255]: pgmap v3489: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:47.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:47.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:47.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301535107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301535107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301535107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:49 smithi022 ceph-mon[31360]: pgmap v3490: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:49 smithi022 ceph-mon[40250]: pgmap v3490: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:49 smithi122 ceph-mon[32255]: pgmap v3490: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:49.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:54:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:54:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:54:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:54:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:51 smithi022 ceph-mon[31360]: pgmap v3491: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:51 smithi022 ceph-mon[40250]: pgmap v3491: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:51 smithi122 ceph-mon[32255]: pgmap v3491: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:54:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:54:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:52 smithi022 ceph-mon[31360]: pgmap v3492: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:52 smithi022 ceph-mon[40250]: pgmap v3492: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:53.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:53.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:53.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:52 smithi122 ceph-mon[32255]: pgmap v3492: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4144781131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4144781131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4144781131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:54 smithi122 ceph-mon[32255]: pgmap v3493: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:54 smithi022 ceph-mon[31360]: pgmap v3493: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:54 smithi022 ceph-mon[40250]: pgmap v3493: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:54:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:57 smithi022 ceph-mon[31360]: pgmap v3494: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:57 smithi022 ceph-mon[40250]: pgmap v3494: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:57 smithi122 ceph-mon[32255]: pgmap v3494: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:54:58.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:54:58.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:54:58.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:54:58.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:54:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:59 smithi022 ceph-mon[31360]: pgmap v3495: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:54:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/734875498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:59 smithi022 ceph-mon[40250]: pgmap v3495: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:54:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/734875498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:54:59.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:54:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:54:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:59 smithi122 ceph-mon[32255]: pgmap v3495: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:54:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:54:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/734875498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:55:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:01 smithi022 ceph-mon[31360]: pgmap v3496: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:01 smithi022 ceph-mon[40250]: pgmap v3496: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:01 smithi122 ceph-mon[32255]: pgmap v3496: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:55:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:55:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:02 smithi122 ceph-mon[32255]: pgmap v3497: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:02 smithi022 ceph-mon[31360]: pgmap v3497: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:02 smithi022 ceph-mon[40250]: pgmap v3497: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:03.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:03.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:03.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:03.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1696691895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1696691895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1696691895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:04 smithi122 ceph-mon[32255]: pgmap v3498: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:04 smithi022 ceph-mon[40250]: pgmap v3498: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:04 smithi022 ceph-mon[31360]: pgmap v3498: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:07 smithi022 ceph-mon[31360]: pgmap v3499: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:55:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:07 smithi022 ceph-mon[40250]: pgmap v3499: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:07 smithi122 ceph-mon[32255]: pgmap v3499: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:08.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:09.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:09.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:09.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:09 smithi022 ceph-mon[31360]: pgmap v3500: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:09 smithi022 ceph-mon[40250]: pgmap v3500: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:09 smithi122 ceph-mon[32255]: pgmap v3500: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:55:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:55:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:55:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/380662700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/380662700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/380662700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:11 smithi022 ceph-mon[31360]: pgmap v3501: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:11 smithi022 ceph-mon[40250]: pgmap v3501: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:11 smithi122 ceph-mon[32255]: pgmap v3501: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:55:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:55:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:12 smithi122 ceph-mon[32255]: pgmap v3502: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:12 smithi022 ceph-mon[31360]: pgmap v3502: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:12 smithi022 ceph-mon[40250]: pgmap v3502: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:14.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:14.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:14.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:14.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:15 smithi022 ceph-mon[40250]: pgmap v3503: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2333811376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:15 smithi022 ceph-mon[31360]: pgmap v3503: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2333811376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:15 smithi122 ceph-mon[32255]: pgmap v3503: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2333811376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:55:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:17 smithi022 ceph-mon[40250]: pgmap v3504: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:17 smithi022 ceph-mon[31360]: pgmap v3504: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:17 smithi122 ceph-mon[32255]: pgmap v3504: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:19 smithi022 ceph-mon[31360]: pgmap v3505: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:19 smithi022 ceph-mon[40250]: pgmap v3505: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:19.616 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:55:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:55:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:19 smithi122 ceph-mon[32255]: pgmap v3505: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:19.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:19.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:19.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1875639272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1875639272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:20] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:55:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1875639272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:21 smithi022 ceph-mon[31360]: pgmap v3506: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:21 smithi022 ceph-mon[40250]: pgmap v3506: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:21 smithi122 ceph-mon[32255]: pgmap v3506: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:55:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:55:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:22 smithi122 ceph-mon[32255]: pgmap v3507: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:22 smithi022 ceph-mon[31360]: pgmap v3507: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:22 smithi022 ceph-mon[40250]: pgmap v3507: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:24.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:25 smithi022 ceph-mon[31360]: pgmap v3508: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:25 smithi022 ceph-mon[40250]: pgmap v3508: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:25.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:25.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:25.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:25 smithi122 ceph-mon[32255]: pgmap v3508: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3926102681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3926102681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3926102681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:27 smithi022 ceph-mon[31360]: pgmap v3509: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:55:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:27 smithi022 ceph-mon[40250]: pgmap v3509: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:27 smithi122 ceph-mon[32255]: pgmap v3509: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:29 smithi022 ceph-mon[31360]: pgmap v3510: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:29 smithi022 ceph-mon[40250]: pgmap v3510: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:55:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:55:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:29 smithi122 ceph-mon[32255]: pgmap v3510: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:30.366 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:30] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:55:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:30.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:30.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:30.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:31 smithi022 ceph-mon[31360]: pgmap v3511: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3612435624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:31 smithi022 ceph-mon[40250]: pgmap v3511: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3612435624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:31 smithi122 ceph-mon[32255]: pgmap v3511: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3612435624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:55:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:55:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:32 smithi122 ceph-mon[32255]: pgmap v3512: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:55:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:55:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:55:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:55:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[31360]: pgmap v3512: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:55:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:55:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:55:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:55:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[40250]: pgmap v3512: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:55:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:55:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:55:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:55:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:35 smithi022 ceph-mon[31360]: pgmap v3513: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:35 smithi022 ceph-mon[40250]: pgmap v3513: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:35 smithi122 ceph-mon[32255]: pgmap v3513: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:35.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:36.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:36.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:36.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3436757517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3436757517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3436757517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:37 smithi022 ceph-mon[31360]: pgmap v3514: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:55:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:37 smithi022 ceph-mon[40250]: pgmap v3514: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:37 smithi122 ceph-mon[32255]: pgmap v3514: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:39 smithi022 ceph-mon[31360]: pgmap v3515: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:39 smithi022 ceph-mon[40250]: pgmap v3515: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:55:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:55:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:39 smithi122 ceph-mon[32255]: pgmap v3515: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:40] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:55:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:41.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:41.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:41.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:41.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:41 smithi022 ceph-mon[31360]: pgmap v3516: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:41.478 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:41 smithi022 ceph-mon[40250]: pgmap v3516: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:41 smithi122 ceph-mon[32255]: pgmap v3516: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3445760536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3445760536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3445760536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:55:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:55:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[31360]: pgmap v3517: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:55:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:55:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:55:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[40250]: pgmap v3517: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:55:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:55:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:55:43.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:43 smithi122 ceph-mon[32255]: pgmap v3517: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:43.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:55:43.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:55:43.663 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:55:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:55:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:55:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:55:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:45 smithi122 ceph-mon[32255]: pgmap v3518: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:45 smithi022 ceph-mon[31360]: pgmap v3518: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:45 smithi022 ceph-mon[40250]: pgmap v3518: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:46.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:46.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:46.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:46.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:47.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:55:47.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:47 smithi022 ceph-mon[31360]: pgmap v3519: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:47.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4293716393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:47 smithi122 ceph-mon[32255]: pgmap v3519: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4293716393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:47 smithi022 ceph-mon[40250]: pgmap v3519: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4293716393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:49.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:55:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:55:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:49 smithi122 ceph-mon[32255]: pgmap v3520: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:49 smithi022 ceph-mon[31360]: pgmap v3520: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:49 smithi022 ceph-mon[40250]: pgmap v3520: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:50] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T19:55:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:55:51.847 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:51 smithi022 ceph-mon[31360]: pgmap v3521: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:51 smithi022 ceph-mon[40250]: pgmap v3521: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:51 smithi122 ceph-mon[32255]: pgmap v3521: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:52.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:52.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:52.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:55:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:55:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:52 smithi122 ceph-mon[32255]: pgmap v3522: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/684217598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:52 smithi022 ceph-mon[31360]: pgmap v3522: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/684217598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:52 smithi022 ceph-mon[40250]: pgmap v3522: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/684217598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:55 smithi022 ceph-mon[31360]: pgmap v3523: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:55 smithi022 ceph-mon[40250]: pgmap v3523: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:55 smithi122 ceph-mon[32255]: pgmap v3523: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:57.219 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:55:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:57 smithi022 ceph-mon[31360]: pgmap v3524: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:55:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:57 smithi022 ceph-mon[40250]: pgmap v3524: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:57.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:55:57.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:55:57.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:55:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:57 smithi122 ceph-mon[32255]: pgmap v3524: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:55:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/317760920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/317760920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/317760920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:55:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:55:59 smithi022 ceph-mon[31360]: pgmap v3525: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:55:59 smithi022 ceph-mon[40250]: pgmap v3525: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:55:59 smithi122 ceph-mon[32255]: pgmap v3525: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:55:59.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:55:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:56:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:00] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T19:56:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:01 smithi022 ceph-mon[31360]: pgmap v3526: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:01 smithi022 ceph-mon[40250]: pgmap v3526: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:01 smithi122 ceph-mon[32255]: pgmap v3526: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:02.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:56:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:56:02.885 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:02 smithi022 ceph-mon[40250]: pgmap v3527: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:02.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:02.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:02.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:02 smithi122 ceph-mon[32255]: pgmap v3527: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:02 smithi022 ceph-mon[31360]: pgmap v3527: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/251766325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/251766325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/251766325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:04 smithi122 ceph-mon[32255]: pgmap v3528: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:04 smithi022 ceph-mon[31360]: pgmap v3528: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:04 smithi022 ceph-mon[40250]: pgmap v3528: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:07 smithi022 ceph-mon[31360]: pgmap v3529: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:56:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:07 smithi022 ceph-mon[40250]: pgmap v3529: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:07 smithi122 ceph-mon[32255]: pgmap v3529: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:07.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:08.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:08.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:08.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:09 smithi022 ceph-mon[31360]: pgmap v3530: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1481111311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:09 smithi022 ceph-mon[40250]: pgmap v3530: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1481111311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:56:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:56:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:09 smithi122 ceph-mon[32255]: pgmap v3530: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1481111311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:10] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T19:56:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:11 smithi022 ceph-mon[31360]: pgmap v3531: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:11 smithi022 ceph-mon[40250]: pgmap v3531: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:11 smithi122 ceph-mon[32255]: pgmap v3531: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:56:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:56:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:12 smithi122 ceph-mon[32255]: pgmap v3532: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:12 smithi022 ceph-mon[31360]: pgmap v3532: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:12 smithi022 ceph-mon[40250]: pgmap v3532: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:13.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:13.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:13.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:13.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3690514264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3690514264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3690514264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:14 smithi122 ceph-mon[32255]: pgmap v3533: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:14 smithi022 ceph-mon[31360]: pgmap v3533: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:14 smithi022 ceph-mon[40250]: pgmap v3533: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:17 smithi022 ceph-mon[31360]: pgmap v3534: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:56:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:17 smithi022 ceph-mon[40250]: pgmap v3534: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:17 smithi122 ceph-mon[32255]: pgmap v3534: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:18.704 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:19.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:19.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:19.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:19 smithi022 ceph-mon[31360]: pgmap v3535: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/925580428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:19 smithi022 ceph-mon[40250]: pgmap v3535: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/925580428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:56:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:56:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:19 smithi122 ceph-mon[32255]: pgmap v3535: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/925580428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:20] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T19:56:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:21 smithi022 ceph-mon[31360]: pgmap v3536: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:21 smithi022 ceph-mon[40250]: pgmap v3536: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:21 smithi122 ceph-mon[32255]: pgmap v3536: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:56:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:56:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:22 smithi122 ceph-mon[32255]: pgmap v3537: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:22 smithi022 ceph-mon[31360]: pgmap v3537: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:22 smithi022 ceph-mon[40250]: pgmap v3537: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:24.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:24.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:24.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:24.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:25 smithi022 ceph-mon[40250]: pgmap v3538: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3015807955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:25 smithi022 ceph-mon[31360]: pgmap v3538: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3015807955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:25 smithi122 ceph-mon[32255]: pgmap v3538: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3015807955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:56:27.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:27 smithi022 ceph-mon[31360]: pgmap v3539: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:27.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:27 smithi022 ceph-mon[40250]: pgmap v3539: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:27 smithi122 ceph-mon[32255]: pgmap v3539: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:29.447 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:29 smithi022 ceph-mon[31360]: pgmap v3540: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:29 smithi022 ceph-mon[40250]: pgmap v3540: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:56:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:56:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:29 smithi122 ceph-mon[32255]: pgmap v3540: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:29.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:29.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:29.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:30] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T19:56:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4232307706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4232307706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4232307706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:31 smithi022 ceph-mon[31360]: pgmap v3541: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:31 smithi022 ceph-mon[40250]: pgmap v3541: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:31 smithi122 ceph-mon[32255]: pgmap v3541: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:56:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:56:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:32 smithi122 ceph-mon[32255]: pgmap v3542: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:56:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:56:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:56:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:56:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[31360]: pgmap v3542: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:56:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:56:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:56:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:56:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[40250]: pgmap v3542: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:56:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:56:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:56:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:56:34.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:35.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:35.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:35.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:35 smithi022 ceph-mon[31360]: pgmap v3543: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1269974735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:35 smithi022 ceph-mon[40250]: pgmap v3543: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1269974735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:35 smithi122 ceph-mon[32255]: pgmap v3543: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1269974735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:37.531 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:56:37.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:37 smithi022 ceph-mon[31360]: pgmap v3544: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:37.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:37 smithi022 ceph-mon[40250]: pgmap v3544: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:37.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:37 smithi122 ceph-mon[32255]: pgmap v3544: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:39 smithi022 ceph-mon[31360]: pgmap v3545: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:39 smithi022 ceph-mon[40250]: pgmap v3545: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:39 smithi122 ceph-mon[32255]: pgmap v3545: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:56:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:56:40.184 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:40] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T19:56:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:40.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:40.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:40.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:41.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:41 smithi022 ceph-mon[40250]: pgmap v3546: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:41.253 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1874169962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:41.253 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:41 smithi022 ceph-mon[31360]: pgmap v3546: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:41.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1874169962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:41 smithi122 ceph-mon[32255]: pgmap v3546: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1874169962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:56:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:56:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:42 smithi122 ceph-mon[32255]: pgmap v3547: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:43.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:42 smithi022 ceph-mon[40250]: pgmap v3547: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:42 smithi022 ceph-mon[31360]: pgmap v3547: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:56:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:56:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:56:44.058 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:56:44.058 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:56:44.059 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:56:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:45 smithi122 ceph-mon[32255]: pgmap v3548: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[31360]: pgmap v3548: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[40250]: pgmap v3548: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:56:45.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:45.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:45.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:45.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/293095126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/293095126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/293095126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:47 smithi022 ceph-mon[31360]: pgmap v3549: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:47 smithi022 ceph-mon[40250]: pgmap v3549: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:47 smithi122 ceph-mon[32255]: pgmap v3549: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:56:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:49 smithi022 ceph-mon[31360]: pgmap v3550: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:49 smithi022 ceph-mon[40250]: pgmap v3550: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:49 smithi122 ceph-mon[32255]: pgmap v3550: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:49.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:56:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:56:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:56:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:56:50.928 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:51.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:51.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:51.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:51 smithi022 ceph-mon[31360]: pgmap v3551: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:51 smithi022 ceph-mon[40250]: pgmap v3551: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:51 smithi122 ceph-mon[32255]: pgmap v3551: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3328877492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3328877492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3328877492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:56:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:56:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:53 smithi022 ceph-mon[31360]: pgmap v3552: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:53 smithi022 ceph-mon[40250]: pgmap v3552: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:53 smithi122 ceph-mon[32255]: pgmap v3552: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:55 smithi022 ceph-mon[31360]: pgmap v3553: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:55 smithi022 ceph-mon[40250]: pgmap v3553: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:55 smithi122 ceph-mon[32255]: pgmap v3553: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:56.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:56:56.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:56:56.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:56:56.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:56:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:56:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:57 smithi022 ceph-mon[31360]: pgmap v3554: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1577658612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:57 smithi022 ceph-mon[40250]: pgmap v3554: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1577658612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:57 smithi122 ceph-mon[32255]: pgmap v3554: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:56:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1577658612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:56:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:56:59 smithi022 ceph-mon[31360]: pgmap v3555: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:56:59 smithi022 ceph-mon[40250]: pgmap v3555: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:56:59 smithi122 ceph-mon[32255]: pgmap v3555: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:56:59.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:56:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:57:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T19:57:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:01 smithi022 ceph-mon[31360]: pgmap v3556: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:01 smithi022 ceph-mon[40250]: pgmap v3556: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:01.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:01 smithi122 ceph-mon[32255]: pgmap v3556: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:02.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:02.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:02.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:57:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:57:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2823668704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:02 smithi122 ceph-mon[32255]: pgmap v3557: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2823668704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:02 smithi022 ceph-mon[31360]: pgmap v3557: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2823668704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:02 smithi022 ceph-mon[40250]: pgmap v3557: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:05 smithi022 ceph-mon[31360]: pgmap v3558: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:05 smithi022 ceph-mon[40250]: pgmap v3558: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:05 smithi122 ceph-mon[32255]: pgmap v3558: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:07.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:07 smithi022 ceph-mon[31360]: pgmap v3559: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:07 smithi022 ceph-mon[40250]: pgmap v3559: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:07.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:07.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:07.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:07.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:07 smithi122 ceph-mon[32255]: pgmap v3559: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:57:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/951098312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:08.581 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/951098312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/951098312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:09 smithi022 ceph-mon[31360]: pgmap v3560: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:09 smithi022 ceph-mon[40250]: pgmap v3560: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:09 smithi122 ceph-mon[32255]: pgmap v3560: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:57:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:57:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:10] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T19:57:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:11 smithi022 ceph-mon[31360]: pgmap v3561: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:11 smithi022 ceph-mon[40250]: pgmap v3561: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:11 smithi122 ceph-mon[32255]: pgmap v3561: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:12.417 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:57:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:57:12.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:12.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:12.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:12 smithi122 ceph-mon[32255]: pgmap v3562: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1612647610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:12 smithi022 ceph-mon[31360]: pgmap v3562: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1612647610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:12 smithi022 ceph-mon[40250]: pgmap v3562: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1612647610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:15 smithi022 ceph-mon[31360]: pgmap v3563: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:15 smithi022 ceph-mon[40250]: pgmap v3563: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:15 smithi122 ceph-mon[32255]: pgmap v3563: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:17 smithi022 ceph-mon[31360]: pgmap v3564: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:57:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:17 smithi022 ceph-mon[40250]: pgmap v3564: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:17 smithi122 ceph-mon[32255]: pgmap v3564: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:17.785 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:18.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:18.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:18.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/997519148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/997519148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/997519148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:19 smithi022 ceph-mon[31360]: pgmap v3565: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:19 smithi022 ceph-mon[40250]: pgmap v3565: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:57:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:57:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:19 smithi122 ceph-mon[32255]: pgmap v3565: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:57:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:21 smithi022 ceph-mon[31360]: pgmap v3566: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:21 smithi022 ceph-mon[40250]: pgmap v3566: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:21 smithi122 ceph-mon[32255]: pgmap v3566: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:57:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:57:23.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:23.159 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:22 smithi022 ceph-mon[31360]: pgmap v3567: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:23.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:22 smithi022 ceph-mon[40250]: pgmap v3567: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:22 smithi122 ceph-mon[32255]: pgmap v3567: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:23.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:23.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:23.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1035456759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1035456759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1035456759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:24 smithi122 ceph-mon[32255]: pgmap v3568: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:24 smithi022 ceph-mon[31360]: pgmap v3568: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:24 smithi022 ceph-mon[40250]: pgmap v3568: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:27 smithi022 ceph-mon[31360]: pgmap v3569: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:57:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:27 smithi022 ceph-mon[40250]: pgmap v3569: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:27 smithi122 ceph-mon[32255]: pgmap v3569: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:28.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:28.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:28.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:29 smithi022 ceph-mon[31360]: pgmap v3570: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3178995189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:29 smithi022 ceph-mon[40250]: pgmap v3570: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3178995189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:29 smithi122 ceph-mon[32255]: pgmap v3570: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3178995189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:29.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:57:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:57:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:57:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:31 smithi022 ceph-mon[31360]: pgmap v3571: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:31 smithi022 ceph-mon[40250]: pgmap v3571: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:31 smithi122 ceph-mon[32255]: pgmap v3571: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:57:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:57:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:32 smithi122 ceph-mon[32255]: pgmap v3572: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:57:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:57:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:57:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:57:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[31360]: pgmap v3572: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:57:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:57:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:57:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:57:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[40250]: pgmap v3572: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:57:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:57:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:57:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:57:33.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:34.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:34.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:34.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:35 smithi022 ceph-mon[31360]: pgmap v3573: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/232215757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:35 smithi022 ceph-mon[40250]: pgmap v3573: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/232215757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:35 smithi122 ceph-mon[32255]: pgmap v3573: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/232215757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:37 smithi022 ceph-mon[31360]: pgmap v3574: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:57:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:37 smithi022 ceph-mon[40250]: pgmap v3574: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:37 smithi122 ceph-mon[32255]: pgmap v3574: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:39.274 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:39 smithi022 ceph-mon[31360]: pgmap v3575: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:39 smithi022 ceph-mon[40250]: pgmap v3575: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:39.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:39.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:39.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:57:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:57:39.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:39 smithi122 ceph-mon[32255]: pgmap v3575: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1064073144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:40] "GET /metrics HTTP/1.1" 200 202337 "" "Prometheus/2.18.1" 2021-12-15T19:57:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1064073144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1064073144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:41 smithi022 ceph-mon[31360]: pgmap v3576: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:41 smithi022 ceph-mon[40250]: pgmap v3576: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:41 smithi122 ceph-mon[32255]: pgmap v3576: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:57:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:57:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:42 smithi122 ceph-mon[32255]: pgmap v3577: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:42 smithi022 ceph-mon[31360]: pgmap v3577: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:42 smithi022 ceph-mon[40250]: pgmap v3577: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:44.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:45.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:45.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:45.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[31360]: pgmap v3578: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:57:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:57:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:57:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:57:45.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2574281730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[40250]: pgmap v3578: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:57:45.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:57:45.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:57:45.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:57:45.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2574281730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:45 smithi122 ceph-mon[32255]: pgmap v3578: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:57:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:57:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:57:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:57:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2574281730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:47 smithi022 ceph-mon[31360]: pgmap v3579: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:57:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:47 smithi022 ceph-mon[40250]: pgmap v3579: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:47 smithi122 ceph-mon[32255]: pgmap v3579: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:49 smithi022 ceph-mon[31360]: pgmap v3580: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:49 smithi022 ceph-mon[40250]: pgmap v3580: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:49 smithi122 ceph-mon[32255]: pgmap v3580: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:57:49.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:57:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:57:50.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:50.269 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:50.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:50.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:50.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:57:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:57:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:51 smithi022 ceph-mon[31360]: pgmap v3581: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1381357990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:51 smithi022 ceph-mon[40250]: pgmap v3581: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1381357990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:51 smithi122 ceph-mon[32255]: pgmap v3581: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1381357990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:57:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:57:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:52 smithi122 ceph-mon[32255]: pgmap v3582: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:52 smithi022 ceph-mon[31360]: pgmap v3582: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:52 smithi022 ceph-mon[40250]: pgmap v3582: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:57:55.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:57:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:55 smithi022 ceph-mon[31360]: pgmap v3583: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:57:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:55 smithi022 ceph-mon[40250]: pgmap v3583: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:57:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:55 smithi122 ceph-mon[32255]: pgmap v3583: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:57:55.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:57:55.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:57:55.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:57:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/300077985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/300077985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/300077985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:57:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:57 smithi022 ceph-mon[31360]: pgmap v3584: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:57:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:57:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:57 smithi022 ceph-mon[40250]: pgmap v3584: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:57:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:57 smithi122 ceph-mon[32255]: pgmap v3584: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T19:57:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:57:59 smithi022 ceph-mon[31360]: pgmap v3585: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T19:57:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:57:59 smithi022 ceph-mon[40250]: pgmap v3585: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T19:57:59.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:57:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:57:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:57:59 smithi122 ceph-mon[32255]: pgmap v3585: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T19:58:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:58:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:00.756 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:01.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:01.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:01.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:01 smithi022 ceph-mon[31360]: pgmap v3586: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3528854482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:01 smithi022 ceph-mon[40250]: pgmap v3586: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3528854482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:01 smithi122 ceph-mon[32255]: pgmap v3586: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3528854482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:58:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:58:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:02 smithi122 ceph-mon[32255]: pgmap v3587: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:02 smithi022 ceph-mon[31360]: pgmap v3587: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:02 smithi022 ceph-mon[40250]: pgmap v3587: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:05 smithi022 ceph-mon[31360]: pgmap v3588: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:05 smithi022 ceph-mon[40250]: pgmap v3588: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:05 smithi122 ceph-mon[32255]: pgmap v3588: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T19:58:06.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:06.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:06.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:06.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:07 smithi022 ceph-mon[31360]: pgmap v3589: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:58:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2424549146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:58:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:07 smithi022 ceph-mon[40250]: pgmap v3589: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:58:07.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2424549146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:07 smithi122 ceph-mon[32255]: pgmap v3589: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T19:58:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2424549146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:09 smithi022 ceph-mon[31360]: pgmap v3590: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:58:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:09 smithi022 ceph-mon[40250]: pgmap v3590: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:58:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:09 smithi122 ceph-mon[32255]: pgmap v3590: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T19:58:09.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:58:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:58:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:10] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T19:58:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:11.498 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:11 smithi022 ceph-mon[31360]: pgmap v3591: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T19:58:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:11 smithi022 ceph-mon[40250]: pgmap v3591: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T19:58:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:11 smithi122 ceph-mon[32255]: pgmap v3591: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T19:58:11.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:11.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:11.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2861716280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2861716280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2861716280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:58:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:58:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:13 smithi022 ceph-mon[31360]: pgmap v3592: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:13 smithi022 ceph-mon[40250]: pgmap v3592: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:13 smithi122 ceph-mon[32255]: pgmap v3592: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:15 smithi122 ceph-mon[32255]: pgmap v3593: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:15 smithi022 ceph-mon[31360]: pgmap v3593: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:15 smithi022 ceph-mon[40250]: pgmap v3593: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:16.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:17.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:17.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:17.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:17 smithi022 ceph-mon[31360]: pgmap v3594: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3121398032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:58:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:17 smithi022 ceph-mon[40250]: pgmap v3594: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3121398032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:17 smithi122 ceph-mon[32255]: pgmap v3594: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3121398032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:58:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:58:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:19 smithi122 ceph-mon[32255]: pgmap v3595: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:19 smithi022 ceph-mon[31360]: pgmap v3595: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:19 smithi022 ceph-mon[40250]: pgmap v3595: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:20] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T19:58:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:21 smithi122 ceph-mon[32255]: pgmap v3596: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:21 smithi022 ceph-mon[31360]: pgmap v3596: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:21 smithi022 ceph-mon[40250]: pgmap v3596: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:22.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:22.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:22.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:22.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:58:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:58:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:22 smithi122 ceph-mon[32255]: pgmap v3597: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/283063256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:22 smithi022 ceph-mon[31360]: pgmap v3597: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/283063256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:22 smithi022 ceph-mon[40250]: pgmap v3597: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/283063256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:25 smithi022 ceph-mon[31360]: pgmap v3598: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:25 smithi022 ceph-mon[40250]: pgmap v3598: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:25 smithi122 ceph-mon[32255]: pgmap v3598: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:27 smithi022 ceph-mon[31360]: pgmap v3599: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:58:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:27 smithi022 ceph-mon[40250]: pgmap v3599: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:27.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:27 smithi122 ceph-mon[32255]: pgmap v3599: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:27.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:27.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:27.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3637325029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3637325029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3637325029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:29 smithi022 ceph-mon[31360]: pgmap v3600: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:29 smithi022 ceph-mon[40250]: pgmap v3600: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:58:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:58:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:29 smithi122 ceph-mon[32255]: pgmap v3600: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:30] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T19:58:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:31 smithi122 ceph-mon[32255]: pgmap v3601: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:31 smithi022 ceph-mon[31360]: pgmap v3601: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:31 smithi022 ceph-mon[40250]: pgmap v3601: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:58:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:58:32.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:32 smithi122 ceph-mon[32255]: pgmap v3602: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:58:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:58:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:58:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:58:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[40250]: pgmap v3602: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:58:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:58:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:58:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:58:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[31360]: pgmap v3602: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:58:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:58:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:58:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:58:33.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:33.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:33.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2954351454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2954351454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2954351454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:34 smithi122 ceph-mon[32255]: pgmap v3603: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:34 smithi022 ceph-mon[31360]: pgmap v3603: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:34 smithi022 ceph-mon[40250]: pgmap v3603: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:37 smithi022 ceph-mon[31360]: pgmap v3604: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:58:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:37 smithi022 ceph-mon[40250]: pgmap v3604: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:37 smithi122 ceph-mon[32255]: pgmap v3604: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:38.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:38.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:38.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:38.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:39 smithi022 ceph-mon[31360]: pgmap v3605: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1971545322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:39 smithi022 ceph-mon[40250]: pgmap v3605: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1971545322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:58:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:58:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:39 smithi122 ceph-mon[32255]: pgmap v3605: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1971545322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:40] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:58:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:41 smithi122 ceph-mon[32255]: pgmap v3606: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:41 smithi022 ceph-mon[31360]: pgmap v3606: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:42.135 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:41 smithi022 ceph-mon[40250]: pgmap v3606: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:58:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:58:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:42 smithi122 ceph-mon[32255]: pgmap v3607: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:42 smithi022 ceph-mon[31360]: pgmap v3607: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:42 smithi022 ceph-mon[40250]: pgmap v3607: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:43.725 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3771910814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3771910814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3771910814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:45 smithi122 ceph-mon[32255]: pgmap v3608: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:58:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:58:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:58:45.469 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[40250]: pgmap v3608: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:45.469 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:58:45.469 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:58:45.470 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:58:45.470 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[31360]: pgmap v3608: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:45.470 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:58:45.470 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:58:45.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:58:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:58:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:58:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:58:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:58:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:47 smithi022 ceph-mon[31360]: pgmap v3609: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:47 smithi022 ceph-mon[40250]: pgmap v3609: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:47 smithi122 ceph-mon[32255]: pgmap v3609: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:49.095 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:49 smithi022 ceph-mon[31360]: pgmap v3610: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1384570650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:49 smithi022 ceph-mon[40250]: pgmap v3610: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1384570650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:58:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:58:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:49 smithi122 ceph-mon[32255]: pgmap v3610: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1384570650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:58:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:58:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:51 smithi022 ceph-mon[31360]: pgmap v3611: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:51 smithi022 ceph-mon[40250]: pgmap v3611: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:51 smithi122 ceph-mon[32255]: pgmap v3611: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:58:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:58:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:52 smithi122 ceph-mon[32255]: pgmap v3612: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:52 smithi022 ceph-mon[31360]: pgmap v3612: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:52 smithi022 ceph-mon[40250]: pgmap v3612: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:54.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:58:54.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:58:54.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:58:54.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:58:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:55 smithi122 ceph-mon[32255]: pgmap v3613: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3740114078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:55 smithi022 ceph-mon[31360]: pgmap v3613: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3740114078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:55 smithi022 ceph-mon[40250]: pgmap v3613: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3740114078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:58:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:57 smithi122 ceph-mon[32255]: pgmap v3614: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:57 smithi022 ceph-mon[31360]: pgmap v3614: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:58:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:57 smithi022 ceph-mon[40250]: pgmap v3614: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:58:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:58:59 smithi122 ceph-mon[32255]: pgmap v3615: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:59.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:58:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:58:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:58:59 smithi022 ceph-mon[31360]: pgmap v3615: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:58:59 smithi022 ceph-mon[40250]: pgmap v3615: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:58:59.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:59:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/817980443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/817980443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/817980443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:01 smithi122 ceph-mon[32255]: pgmap v3616: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:01 smithi022 ceph-mon[31360]: pgmap v3616: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:01 smithi022 ceph-mon[40250]: pgmap v3616: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:59:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:59:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:02 smithi122 ceph-mon[32255]: pgmap v3617: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:02 smithi022 ceph-mon[31360]: pgmap v3617: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:02 smithi022 ceph-mon[40250]: pgmap v3617: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:05.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:05 smithi022 ceph-mon[31360]: pgmap v3618: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:05 smithi022 ceph-mon[40250]: pgmap v3618: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:05.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:05.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:05.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:05 smithi122 ceph-mon[32255]: pgmap v3618: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2008348200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2008348200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2008348200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:07 smithi122 ceph-mon[32255]: pgmap v3619: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:07 smithi022 ceph-mon[31360]: pgmap v3619: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:59:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:07 smithi022 ceph-mon[40250]: pgmap v3619: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:09 smithi122 ceph-mon[32255]: pgmap v3620: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:59:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:59:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:09 smithi022 ceph-mon[31360]: pgmap v3620: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:09 smithi022 ceph-mon[40250]: pgmap v3620: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:10] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T19:59:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:10.577 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:10.578 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:10.578 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:10.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:10.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:10.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:11 smithi122 ceph-mon[32255]: pgmap v3621: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/112574286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:11 smithi022 ceph-mon[31360]: pgmap v3621: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/112574286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:11 smithi022 ceph-mon[40250]: pgmap v3621: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/112574286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:59:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:59:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:12 smithi122 ceph-mon[32255]: pgmap v3622: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:12 smithi022 ceph-mon[31360]: pgmap v3622: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:12 smithi022 ceph-mon[40250]: pgmap v3622: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:15 smithi122 ceph-mon[32255]: pgmap v3623: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:15 smithi022 ceph-mon[31360]: pgmap v3623: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:15 smithi022 ceph-mon[40250]: pgmap v3623: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:15.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:16.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:16.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:16.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:17 smithi122 ceph-mon[32255]: pgmap v3624: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1655009027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:59:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:17 smithi022 ceph-mon[31360]: pgmap v3624: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1655009027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:17 smithi022 ceph-mon[40250]: pgmap v3624: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1655009027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:19 smithi122 ceph-mon[32255]: pgmap v3625: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:19.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:59:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:59:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:19 smithi022 ceph-mon[31360]: pgmap v3625: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:19 smithi022 ceph-mon[40250]: pgmap v3625: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:20] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:59:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:21.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:21.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:21.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:21.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:21.681 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:21 smithi022 ceph-mon[31360]: pgmap v3626: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:21.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:21 smithi022 ceph-mon[40250]: pgmap v3626: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:21 smithi122 ceph-mon[32255]: pgmap v3626: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3754723179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:59:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:59:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3754723179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3754723179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:23 smithi122 ceph-mon[32255]: pgmap v3627: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:23 smithi022 ceph-mon[31360]: pgmap v3627: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:23 smithi022 ceph-mon[40250]: pgmap v3627: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:25 smithi122 ceph-mon[32255]: pgmap v3628: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:25 smithi022 ceph-mon[31360]: pgmap v3628: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:25 smithi022 ceph-mon[40250]: pgmap v3628: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:26.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:27.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:27.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:27.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:27 smithi122 ceph-mon[32255]: pgmap v3629: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1094978076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:59:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:27 smithi022 ceph-mon[31360]: pgmap v3629: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:27.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1094978076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:27.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:27 smithi022 ceph-mon[40250]: pgmap v3629: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:27.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1094978076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:29 smithi122 ceph-mon[32255]: pgmap v3630: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:59:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:59:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:29 smithi022 ceph-mon[31360]: pgmap v3630: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:29 smithi022 ceph-mon[40250]: pgmap v3630: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:30] "GET /metrics HTTP/1.1" 200 202322 "" "Prometheus/2.18.1" 2021-12-15T19:59:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:31 smithi122 ceph-mon[32255]: pgmap v3631: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:31 smithi022 ceph-mon[31360]: pgmap v3631: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:31 smithi022 ceph-mon[40250]: pgmap v3631: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:32.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:32.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:59:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:59:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:32 smithi122 ceph-mon[32255]: pgmap v3632: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1972230476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:59:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:59:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:59:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:59:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[31360]: pgmap v3632: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1972230476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:59:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:59:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:59:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:59:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[40250]: pgmap v3632: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1972230476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:59:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:59:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T19:59:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T19:59:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:35 smithi122 ceph-mon[32255]: pgmap v3633: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:35 smithi022 ceph-mon[31360]: pgmap v3633: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:35 smithi022 ceph-mon[40250]: pgmap v3633: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:37.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:37 smithi122 ceph-mon[32255]: pgmap v3634: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:37 smithi022 ceph-mon[31360]: pgmap v3634: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:59:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:37 smithi022 ceph-mon[40250]: pgmap v3634: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T19:59:37.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:37.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:37.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2483848181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2483848181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2483848181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:39.555 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:39 smithi122 ceph-mon[32255]: pgmap v3635: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:39 smithi022 ceph-mon[31360]: pgmap v3635: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:39 smithi022 ceph-mon[40250]: pgmap v3635: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:39.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:59:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:59:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:40] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:59:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:41 smithi122 ceph-mon[32255]: pgmap v3636: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:41 smithi022 ceph-mon[31360]: pgmap v3636: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:41 smithi022 ceph-mon[40250]: pgmap v3636: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:59:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:59:42.794 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:43.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:42 smithi022 ceph-mon[40250]: pgmap v3637: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:43.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:42 smithi022 ceph-mon[31360]: pgmap v3637: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:42 smithi122 ceph-mon[32255]: pgmap v3637: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4221065931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4221065931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4221065931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:44 smithi022 ceph-mon[31360]: pgmap v3638: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:44 smithi022 ceph-mon[40250]: pgmap v3638: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:44 smithi122 ceph-mon[32255]: pgmap v3638: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:59:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:59:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:59:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:59:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:59:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:59:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:59:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:59:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T19:59:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T19:59:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T19:59:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T19:59:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:46 smithi022 ceph-mon[31360]: pgmap v3639: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:46 smithi022 ceph-mon[40250]: pgmap v3639: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:46 smithi122 ceph-mon[32255]: pgmap v3639: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:59:48.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:49.561 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:49 smithi122 ceph-mon[32255]: pgmap v3640: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:49.562 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1700891850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:49 smithi022 ceph-mon[31360]: pgmap v3640: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1700891850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:49 smithi022 ceph-mon[40250]: pgmap v3640: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1700891850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:59:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T19:59:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 19:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T19:59:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T19:59:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:51 smithi122 ceph-mon[32255]: pgmap v3641: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:51 smithi022 ceph-mon[31360]: pgmap v3641: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:51 smithi022 ceph-mon[40250]: pgmap v3641: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 19:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T19:59:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T19:59:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:52 smithi022 ceph-mon[31360]: pgmap v3642: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:52 smithi022 ceph-mon[40250]: pgmap v3642: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:52 smithi122 ceph-mon[32255]: pgmap v3642: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:53.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:53.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:53.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:53.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2186566653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2186566653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2186566653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:54 smithi022 ceph-mon[31360]: pgmap v3643: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:54 smithi022 ceph-mon[40250]: pgmap v3643: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:54 smithi122 ceph-mon[32255]: pgmap v3643: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:57 smithi122 ceph-mon[32255]: pgmap v3644: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:57 smithi022 ceph-mon[31360]: pgmap v3644: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 19:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:19:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T19:59:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:57 smithi022 ceph-mon[40250]: pgmap v3644: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T19:59:58.907 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T19:59:59.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T19:59:59.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T19:59:59.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T19:59:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:59 smithi022 ceph-mon[31360]: pgmap v3645: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 19:59:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1578220702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:59 smithi022 ceph-mon[40250]: pgmap v3645: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 19:59:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1578220702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:59.568 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:59 smithi122 ceph-mon[32255]: pgmap v3645: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T19:59:59.569 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 19:59:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1578220702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T19:59:59.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 19:59:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:00:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:00:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T20:00:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T20:00:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T20:00:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:01 smithi122 ceph-mon[32255]: pgmap v3646: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:01 smithi022 ceph-mon[31360]: pgmap v3646: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:01 smithi022 ceph-mon[40250]: pgmap v3646: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:00:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:00:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:02 smithi022 ceph-mon[31360]: pgmap v3647: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:02 smithi022 ceph-mon[40250]: pgmap v3647: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:02 smithi122 ceph-mon[32255]: pgmap v3647: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:04.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:04.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:04.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:04.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:05 smithi122 ceph-mon[32255]: pgmap v3648: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3165733565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:05 smithi022 ceph-mon[31360]: pgmap v3648: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3165733565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:05 smithi022 ceph-mon[40250]: pgmap v3648: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3165733565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:07 smithi122 ceph-mon[32255]: pgmap v3649: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:07 smithi022 ceph-mon[31360]: pgmap v3649: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:00:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:07 smithi022 ceph-mon[40250]: pgmap v3649: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:09.570 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:09 smithi122 ceph-mon[32255]: pgmap v3650: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:09.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:09.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:09 smithi022 ceph-mon[31360]: pgmap v3650: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:09.648 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:09 smithi022 ceph-mon[40250]: pgmap v3650: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:09.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:00:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:00:10.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:10.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:10.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:10] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:00:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2074424748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2074424748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2074424748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:11 smithi122 ceph-mon[32255]: pgmap v3651: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:11 smithi022 ceph-mon[31360]: pgmap v3651: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:11 smithi022 ceph-mon[40250]: pgmap v3651: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:00:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:00:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:12 smithi022 ceph-mon[31360]: pgmap v3652: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:12 smithi022 ceph-mon[40250]: pgmap v3652: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:12 smithi122 ceph-mon[32255]: pgmap v3652: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:15.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:15.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:15.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:15.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:15 smithi122 ceph-mon[32255]: pgmap v3653: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:15 smithi022 ceph-mon[40250]: pgmap v3653: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:15 smithi022 ceph-mon[31360]: pgmap v3653: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2244944543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2244944543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2244944543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:17.590 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:17 smithi022 ceph-mon[31360]: pgmap v3654: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:17.591 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:00:17.591 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:17 smithi022 ceph-mon[40250]: pgmap v3654: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:17 smithi122 ceph-mon[32255]: pgmap v3654: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:19.581 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:19 smithi122 ceph-mon[32255]: pgmap v3655: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:19 smithi022 ceph-mon[31360]: pgmap v3655: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:19 smithi022 ceph-mon[40250]: pgmap v3655: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:19.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:00:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:00:20.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:20] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:00:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:20.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:20.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:20.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:20.757 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:20.757 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:21 smithi122 ceph-mon[32255]: pgmap v3656: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4090690005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:21 smithi022 ceph-mon[31360]: pgmap v3656: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4090690005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:21 smithi022 ceph-mon[40250]: pgmap v3656: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4090690005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:00:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:00:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:22 smithi022 ceph-mon[31360]: pgmap v3657: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:22 smithi022 ceph-mon[40250]: pgmap v3657: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:22 smithi122 ceph-mon[32255]: pgmap v3657: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:25 smithi122 ceph-mon[32255]: pgmap v3658: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:25.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:25.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:25 smithi022 ceph-mon[40250]: pgmap v3658: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:25.759 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:25 smithi022 ceph-mon[31360]: pgmap v3658: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:26.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:26.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:26.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3672745118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3672745118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3672745118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:27 smithi122 ceph-mon[32255]: pgmap v3659: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:27 smithi022 ceph-mon[31360]: pgmap v3659: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:00:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:27 smithi022 ceph-mon[40250]: pgmap v3659: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:29.592 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:29 smithi122 ceph-mon[32255]: pgmap v3660: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:29 smithi022 ceph-mon[31360]: pgmap v3660: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:29 smithi022 ceph-mon[40250]: pgmap v3660: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:29.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:00:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:00:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:30] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:00:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:31.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:31.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:31.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:31.499 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:31 smithi022 ceph-mon[31360]: pgmap v3661: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:31.500 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:31 smithi022 ceph-mon[40250]: pgmap v3661: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:31 smithi122 ceph-mon[32255]: pgmap v3661: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3348730567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:00:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:00:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3348730567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3348730567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:33 smithi122 ceph-mon[32255]: pgmap v3662: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:00:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:00:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:00:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:00:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[31360]: pgmap v3662: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:00:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:00:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:00:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:00:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[40250]: pgmap v3662: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:00:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:00:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:00:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:00:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:35 smithi122 ceph-mon[32255]: pgmap v3663: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:35 smithi022 ceph-mon[31360]: pgmap v3663: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:35 smithi022 ceph-mon[40250]: pgmap v3663: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:36.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:36.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:36.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:36.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:37 smithi122 ceph-mon[32255]: pgmap v3664: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3630627670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:00:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:37 smithi022 ceph-mon[31360]: pgmap v3664: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3630627670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:37 smithi022 ceph-mon[40250]: pgmap v3664: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3630627670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:39.602 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:39 smithi122 ceph-mon[32255]: pgmap v3665: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:39 smithi022 ceph-mon[31360]: pgmap v3665: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:39 smithi022 ceph-mon[40250]: pgmap v3665: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:39.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:00:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:00:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:40] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:00:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:41 smithi122 ceph-mon[32255]: pgmap v3666: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:41 smithi022 ceph-mon[31360]: pgmap v3666: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:41 smithi022 ceph-mon[40250]: pgmap v3666: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:41.873 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:42.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:42.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:42.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:00:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:00:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2277765747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:42 smithi022 ceph-mon[31360]: pgmap v3667: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2277765747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:42 smithi022 ceph-mon[40250]: pgmap v3667: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2277765747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:42 smithi122 ceph-mon[32255]: pgmap v3667: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:45 smithi122 ceph-mon[32255]: pgmap v3668: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:45 smithi022 ceph-mon[31360]: pgmap v3668: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:45 smithi022 ceph-mon[40250]: pgmap v3668: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:00:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:00:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:00:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:00:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:00:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:00:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:00:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:00:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:00:47.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:00:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:47 smithi022 ceph-mon[31360]: pgmap v3669: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:47 smithi022 ceph-mon[40250]: pgmap v3669: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:47.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:47.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:47.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:47 smithi122 ceph-mon[32255]: pgmap v3669: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2931421287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2931421287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2931421287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:49.611 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:49 smithi122 ceph-mon[32255]: pgmap v3670: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:49 smithi022 ceph-mon[31360]: pgmap v3670: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:49 smithi022 ceph-mon[40250]: pgmap v3670: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:00:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:00:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:50] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:00:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:50.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.874 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.874 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:00:50.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:50.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:51.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:51 smithi022 ceph-mon[40250]: pgmap v3671: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:51.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:51.740 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:51 smithi022 ceph-mon[31360]: pgmap v3671: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:51.741 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:51.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:51 smithi122 ceph-mon[32255]: pgmap v3671: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:51.815 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:00:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:00:52.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:52.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:52.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:52.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:52 smithi022 ceph-mon[31360]: pgmap v3672: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:52 smithi022 ceph-mon[40250]: pgmap v3672: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:52 smithi122 ceph-mon[32255]: pgmap v3672: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1541134675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1541134675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1541134675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:00:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:54 smithi022 ceph-mon[31360]: pgmap v3673: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:54 smithi022 ceph-mon[40250]: pgmap v3673: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:54 smithi122 ceph-mon[32255]: pgmap v3673: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:57 smithi122 ceph-mon[32255]: pgmap v3674: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:57 smithi022 ceph-mon[31360]: pgmap v3674: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:00:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:57 smithi022 ceph-mon[40250]: pgmap v3674: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:00:57.998 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:00:58.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:00:58.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:00:58.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:00:59.618 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:59 smithi122 ceph-mon[32255]: pgmap v3675: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:59.618 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:00:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2460300580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:59 smithi022 ceph-mon[31360]: pgmap v3675: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:00:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2460300580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:59 smithi022 ceph-mon[40250]: pgmap v3675: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:00:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:00:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2460300580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:00:59.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:00:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:01:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:00] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:01:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:01.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:01 smithi022 ceph-mon[31360]: pgmap v3676: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:01.677 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:01 smithi022 ceph-mon[40250]: pgmap v3676: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:01 smithi122 ceph-mon[32255]: pgmap v3676: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:01:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:01:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:02 smithi022 ceph-mon[31360]: pgmap v3677: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:02 smithi022 ceph-mon[40250]: pgmap v3677: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:03.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:02 smithi122 ceph-mon[32255]: pgmap v3677: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:03.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:03.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:03.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1114765803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1114765803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1114765803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:04 smithi022 ceph-mon[31360]: pgmap v3678: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:04 smithi022 ceph-mon[40250]: pgmap v3678: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:04 smithi122 ceph-mon[32255]: pgmap v3678: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:07 smithi122 ceph-mon[32255]: pgmap v3679: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:07 smithi022 ceph-mon[31360]: pgmap v3679: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:01:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:07 smithi022 ceph-mon[40250]: pgmap v3679: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:08.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:09.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:09.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:09.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:09 smithi022 ceph-mon[31360]: pgmap v3680: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3358338562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:09 smithi022 ceph-mon[40250]: pgmap v3680: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3358338562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:09.628 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:09 smithi122 ceph-mon[32255]: pgmap v3680: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:09.628 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3358338562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:09.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:01:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:01:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:10] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T20:01:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:11 smithi122 ceph-mon[32255]: pgmap v3681: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:11 smithi022 ceph-mon[31360]: pgmap v3681: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:11 smithi022 ceph-mon[40250]: pgmap v3681: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:01:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:01:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:12 smithi022 ceph-mon[31360]: pgmap v3682: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:12 smithi022 ceph-mon[40250]: pgmap v3682: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:12 smithi122 ceph-mon[32255]: pgmap v3682: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:14.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:14.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:14.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:14.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:15 smithi122 ceph-mon[32255]: pgmap v3683: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1897973460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:15 smithi022 ceph-mon[40250]: pgmap v3683: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:15.886 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1897973460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:15.886 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:15 smithi022 ceph-mon[31360]: pgmap v3683: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:15.887 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1897973460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:17 smithi122 ceph-mon[32255]: pgmap v3684: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:01:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:17 smithi022 ceph-mon[31360]: pgmap v3684: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:17 smithi022 ceph-mon[40250]: pgmap v3684: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:19.486 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:19.638 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:19 smithi122 ceph-mon[32255]: pgmap v3685: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:19 smithi022 ceph-mon[31360]: pgmap v3685: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:19 smithi022 ceph-mon[40250]: pgmap v3685: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:19.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:19.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:19.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:19.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:01:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:01:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:20] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T20:01:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2985539324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2985539324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2985539324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:21 smithi122 ceph-mon[32255]: pgmap v3686: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:21 smithi022 ceph-mon[31360]: pgmap v3686: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:21 smithi022 ceph-mon[40250]: pgmap v3686: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:01:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:01:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:22 smithi022 ceph-mon[31360]: pgmap v3687: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:22 smithi022 ceph-mon[40250]: pgmap v3687: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:22 smithi122 ceph-mon[32255]: pgmap v3687: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:24.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:25.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:25.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:25.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:25 smithi022 ceph-mon[31360]: pgmap v3688: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3953135766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:25 smithi022 ceph-mon[40250]: pgmap v3688: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3953135766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:25 smithi122 ceph-mon[32255]: pgmap v3688: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3953135766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:27 smithi122 ceph-mon[32255]: pgmap v3689: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:27 smithi022 ceph-mon[31360]: pgmap v3689: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:01:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:27 smithi022 ceph-mon[40250]: pgmap v3689: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:29.649 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:29 smithi122 ceph-mon[32255]: pgmap v3690: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:29 smithi022 ceph-mon[31360]: pgmap v3690: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:29 smithi022 ceph-mon[40250]: pgmap v3690: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:29.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:01:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:01:30.228 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:30] "GET /metrics HTTP/1.1" 200 202316 "" "Prometheus/2.18.1" 2021-12-15T20:01:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:30.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:30.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:30.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:31 smithi122 ceph-mon[32255]: pgmap v3691: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/317655306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:31 smithi022 ceph-mon[31360]: pgmap v3691: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/317655306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:31 smithi022 ceph-mon[40250]: pgmap v3691: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/317655306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:01:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:01:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[31360]: pgmap v3692: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:01:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:01:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:01:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:01:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[40250]: pgmap v3692: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:01:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:01:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:01:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:01:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:32 smithi122 ceph-mon[32255]: pgmap v3692: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:01:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:01:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:01:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:01:35.604 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:35.605 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:35 smithi022 ceph-mon[40250]: pgmap v3693: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:35.605 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:35 smithi022 ceph-mon[31360]: pgmap v3693: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:35 smithi122 ceph-mon[32255]: pgmap v3693: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:35.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:35.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:35.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1839791836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1839791836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:36.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1839791836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:37 smithi122 ceph-mon[32255]: pgmap v3694: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:37 smithi022 ceph-mon[31360]: pgmap v3694: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:01:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:37 smithi022 ceph-mon[40250]: pgmap v3694: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:39.660 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:39 smithi122 ceph-mon[32255]: pgmap v3695: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:39 smithi022 ceph-mon[31360]: pgmap v3695: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:39 smithi022 ceph-mon[40250]: pgmap v3695: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:39.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:01:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:01:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:40] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:01:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:40.969 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:41.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:41.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:41.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:41 smithi122 ceph-mon[32255]: pgmap v3696: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/678074798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:41 smithi022 ceph-mon[31360]: pgmap v3696: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/678074798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:41 smithi022 ceph-mon[40250]: pgmap v3696: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/678074798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:01:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:01:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:42 smithi022 ceph-mon[31360]: pgmap v3697: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:42 smithi022 ceph-mon[40250]: pgmap v3697: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:42 smithi122 ceph-mon[32255]: pgmap v3697: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:01:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:45 smithi122 ceph-mon[32255]: pgmap v3698: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:45 smithi022 ceph-mon[31360]: pgmap v3698: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:45 smithi022 ceph-mon[40250]: pgmap v3698: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:46.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:46.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:46.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:46.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:47 smithi122 ceph-mon[32255]: pgmap v3699: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2328564843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:01:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:47 smithi022 ceph-mon[31360]: pgmap v3699: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2328564843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:47 smithi022 ceph-mon[40250]: pgmap v3699: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2328564843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:49.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:49 smithi122 ceph-mon[32255]: pgmap v3700: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:49 smithi022 ceph-mon[31360]: pgmap v3700: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:49 smithi022 ceph-mon[40250]: pgmap v3700: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:01:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:01:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:01:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:01:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:51 smithi122 ceph-mon[32255]: pgmap v3701: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:51.711 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:51.712 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:51 smithi022 ceph-mon[40250]: pgmap v3701: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:51.712 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:51 smithi022 ceph-mon[31360]: pgmap v3701: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:52.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:52.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:52.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:01:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:01:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4164088031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:52 smithi022 ceph-mon[31360]: pgmap v3702: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4164088031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:52 smithi022 ceph-mon[40250]: pgmap v3702: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4164088031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:52 smithi122 ceph-mon[32255]: pgmap v3702: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:01:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:01:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:01:54.276 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:01:54.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:01:54.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:01:54.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:01:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:01:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:01:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:55 smithi122 ceph-mon[32255]: pgmap v3703: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:01:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:55 smithi022 ceph-mon[31360]: pgmap v3703: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:01:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:55 smithi022 ceph-mon[40250]: pgmap v3703: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:01:57.083 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:01:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:57 smithi022 ceph-mon[31360]: pgmap v3704: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:57 smithi022 ceph-mon[40250]: pgmap v3704: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:57.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:01:57.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:01:57.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:01:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:57 smithi122 ceph-mon[32255]: pgmap v3704: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:01:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:01:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1270120710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1270120710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1270120710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:01:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:01:59 smithi122 ceph-mon[32255]: pgmap v3705: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:01:59 smithi022 ceph-mon[31360]: pgmap v3705: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:01:59 smithi022 ceph-mon[40250]: pgmap v3705: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:01:59.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:01:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:02:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:02:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:01 smithi122 ceph-mon[32255]: pgmap v3706: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:01 smithi022 ceph-mon[31360]: pgmap v3706: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:01 smithi022 ceph-mon[40250]: pgmap v3706: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:02.455 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:02:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:02:02.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:02.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:02.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:02 smithi022 ceph-mon[31360]: pgmap v3707: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/226222909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:02 smithi022 ceph-mon[40250]: pgmap v3707: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/226222909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:02 smithi122 ceph-mon[32255]: pgmap v3707: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/226222909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:05 smithi122 ceph-mon[32255]: pgmap v3708: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:05 smithi022 ceph-mon[31360]: pgmap v3708: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:05 smithi022 ceph-mon[40250]: pgmap v3708: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:07.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:07 smithi122 ceph-mon[32255]: pgmap v3709: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:07 smithi022 ceph-mon[31360]: pgmap v3709: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:02:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:07 smithi022 ceph-mon[40250]: pgmap v3709: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:07.826 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:08.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:08.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:08.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1175487560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1175487560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1175487560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:09.678 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:09 smithi122 ceph-mon[32255]: pgmap v3710: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:09 smithi022 ceph-mon[31360]: pgmap v3710: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:09 smithi022 ceph-mon[40250]: pgmap v3710: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:09.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:02:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:02:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:10] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:02:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:11 smithi122 ceph-mon[32255]: pgmap v3711: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:11 smithi022 ceph-mon[31360]: pgmap v3711: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:11 smithi022 ceph-mon[40250]: pgmap v3711: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:02:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:02:13.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:12 smithi022 ceph-mon[31360]: pgmap v3712: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:12 smithi022 ceph-mon[40250]: pgmap v3712: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:12 smithi122 ceph-mon[32255]: pgmap v3712: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3627388207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3627388207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3627388207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:14 smithi022 ceph-mon[31360]: pgmap v3713: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:14 smithi022 ceph-mon[40250]: pgmap v3713: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:14 smithi122 ceph-mon[32255]: pgmap v3713: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:17 smithi122 ceph-mon[32255]: pgmap v3714: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:17 smithi022 ceph-mon[31360]: pgmap v3714: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:02:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:17 smithi022 ceph-mon[40250]: pgmap v3714: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:18.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:18.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:18.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:18.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:19 smithi122 ceph-mon[32255]: pgmap v3715: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1295782747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:19 smithi022 ceph-mon[31360]: pgmap v3715: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1295782747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:19 smithi022 ceph-mon[40250]: pgmap v3715: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1295782747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:20.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:02:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:02:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:02:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:21 smithi122 ceph-mon[32255]: pgmap v3716: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:21 smithi022 ceph-mon[31360]: pgmap v3716: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:21 smithi022 ceph-mon[40250]: pgmap v3716: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:02:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:02:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:22 smithi022 ceph-mon[31360]: pgmap v3717: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:22 smithi022 ceph-mon[40250]: pgmap v3717: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:22 smithi122 ceph-mon[32255]: pgmap v3717: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:23.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:24.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:24.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:24.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4155903206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4155903206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4155903206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:25 smithi122 ceph-mon[32255]: pgmap v3718: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:25 smithi022 ceph-mon[31360]: pgmap v3718: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:25 smithi022 ceph-mon[40250]: pgmap v3718: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:27 smithi122 ceph-mon[32255]: pgmap v3719: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:27 smithi022 ceph-mon[31360]: pgmap v3719: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:02:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:27 smithi022 ceph-mon[40250]: pgmap v3719: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:29.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:29.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:29.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:29.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:29.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:29 smithi022 ceph-mon[31360]: pgmap v3720: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:29.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:29 smithi022 ceph-mon[40250]: pgmap v3720: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:29 smithi122 ceph-mon[32255]: pgmap v3720: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:02:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:02:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2339417075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:02:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2339417075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2339417075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:31 smithi122 ceph-mon[32255]: pgmap v3721: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:31 smithi022 ceph-mon[31360]: pgmap v3721: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:31 smithi022 ceph-mon[40250]: pgmap v3721: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:02:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:02:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[31360]: pgmap v3722: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:02:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:02:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:02:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:02:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[40250]: pgmap v3722: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:02:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:02:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:02:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:02:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:32 smithi122 ceph-mon[32255]: pgmap v3722: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:02:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:02:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:02:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:02:34.681 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:35.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:35.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:35.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:35 smithi122 ceph-mon[32255]: pgmap v3723: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/963373694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:35 smithi022 ceph-mon[31360]: pgmap v3723: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/963373694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:35 smithi022 ceph-mon[40250]: pgmap v3723: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/963373694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:37 smithi122 ceph-mon[32255]: pgmap v3724: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:37 smithi022 ceph-mon[31360]: pgmap v3724: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:02:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:37 smithi022 ceph-mon[40250]: pgmap v3724: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:39 smithi122 ceph-mon[32255]: pgmap v3725: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:39 smithi022 ceph-mon[31360]: pgmap v3725: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:39 smithi022 ceph-mon[40250]: pgmap v3725: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:40.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:02:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:02:40.355 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:40.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:40.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:40] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:02:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:41 smithi122 ceph-mon[32255]: pgmap v3726: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2551928395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:41 smithi022 ceph-mon[31360]: pgmap v3726: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2551928395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:41 smithi022 ceph-mon[40250]: pgmap v3726: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2551928395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:02:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:02:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:42 smithi022 ceph-mon[31360]: pgmap v3727: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:42 smithi022 ceph-mon[40250]: pgmap v3727: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:42 smithi122 ceph-mon[32255]: pgmap v3727: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:45.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:45 smithi122 ceph-mon[32255]: pgmap v3728: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:45 smithi022 ceph-mon[31360]: pgmap v3728: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:45 smithi022 ceph-mon[40250]: pgmap v3728: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:45.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:45.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:45.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1650903430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1650903430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1650903430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:47 smithi122 ceph-mon[32255]: pgmap v3729: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:47 smithi022 ceph-mon[31360]: pgmap v3729: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:02:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:47 smithi022 ceph-mon[40250]: pgmap v3729: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:49 smithi122 ceph-mon[32255]: pgmap v3730: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:49 smithi022 ceph-mon[31360]: pgmap v3730: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:49 smithi022 ceph-mon[40250]: pgmap v3730: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:02:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:02:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:50] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:02:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:02:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:51.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:51.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:51.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:51 smithi022 ceph-mon[31360]: pgmap v3731: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/808362313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:51 smithi022 ceph-mon[40250]: pgmap v3731: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/808362313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:51 smithi122 ceph-mon[32255]: pgmap v3731: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/808362313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:02:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:02:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:52 smithi022 ceph-mon[31360]: pgmap v3732: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:52 smithi022 ceph-mon[40250]: pgmap v3732: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:52 smithi122 ceph-mon[32255]: pgmap v3732: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:02:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:02:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:02:54.731 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:02:54.731 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:02:54.732 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:02:54.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:02:54.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:02:54.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:02:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:55 smithi022 ceph-mon[31360]: pgmap v3733: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:02:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:55 smithi022 ceph-mon[40250]: pgmap v3733: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:02:56.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:02:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:55 smithi122 ceph-mon[32255]: pgmap v3733: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:02:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:02:56.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:02:56.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:02:56.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:02:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1851234757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1851234757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1851234757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:02:57.744 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:02:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:57 smithi022 ceph-mon[31360]: pgmap v3734: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:57 smithi022 ceph-mon[40250]: pgmap v3734: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:02:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:57 smithi122 ceph-mon[32255]: pgmap v3734: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:02:59 smithi022 ceph-mon[31360]: pgmap v3735: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:02:59 smithi022 ceph-mon[40250]: pgmap v3735: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:00.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:02:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:03:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:02:59 smithi122 ceph-mon[32255]: pgmap v3735: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:00] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:03:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:01.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:01.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:01.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:01.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:01.905 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:01 smithi022 ceph-mon[31360]: pgmap v3736: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:01.905 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:01 smithi022 ceph-mon[40250]: pgmap v3736: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:01 smithi122 ceph-mon[32255]: pgmap v3736: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:03:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:03:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/627823302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/627823302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/627823302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:03 smithi022 ceph-mon[31360]: pgmap v3737: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:03 smithi022 ceph-mon[40250]: pgmap v3737: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:03 smithi122 ceph-mon[32255]: pgmap v3737: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:05 smithi022 ceph-mon[31360]: pgmap v3738: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:05 smithi022 ceph-mon[40250]: pgmap v3738: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:05 smithi122 ceph-mon[32255]: pgmap v3738: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:06.906 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:07.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:07.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:07.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:03:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:07 smithi022 ceph-mon[31360]: pgmap v3739: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/331854049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:07 smithi022 ceph-mon[40250]: pgmap v3739: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/331854049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:07 smithi122 ceph-mon[32255]: pgmap v3739: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/331854049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:09 smithi022 ceph-mon[31360]: pgmap v3740: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:09 smithi022 ceph-mon[40250]: pgmap v3740: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:10.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:03:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:03:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:09 smithi122 ceph-mon[32255]: pgmap v3740: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:10] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:03:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:11 smithi022 ceph-mon[31360]: pgmap v3741: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:11 smithi022 ceph-mon[40250]: pgmap v3741: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:11 smithi122 ceph-mon[32255]: pgmap v3741: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:12.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:12.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:12.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:12.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:03:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:03:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:12 smithi022 ceph-mon[31360]: pgmap v3742: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1073250825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:12 smithi022 ceph-mon[40250]: pgmap v3742: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1073250825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:12 smithi122 ceph-mon[32255]: pgmap v3742: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1073250825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:15 smithi122 ceph-mon[32255]: pgmap v3743: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:15 smithi022 ceph-mon[31360]: pgmap v3743: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:15 smithi022 ceph-mon[40250]: pgmap v3743: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:17.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:17.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:17 smithi022 ceph-mon[31360]: pgmap v3744: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:17.650 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:03:17.650 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:17 smithi022 ceph-mon[40250]: pgmap v3744: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:17 smithi122 ceph-mon[32255]: pgmap v3744: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:18.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:18.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:18.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3920407156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3920407156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3920407156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:19 smithi122 ceph-mon[32255]: pgmap v3745: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:19 smithi022 ceph-mon[31360]: pgmap v3745: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:19 smithi022 ceph-mon[40250]: pgmap v3745: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:20.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:03:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:03:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:20] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T20:03:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:21 smithi122 ceph-mon[32255]: pgmap v3746: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:21 smithi022 ceph-mon[31360]: pgmap v3746: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:21 smithi022 ceph-mon[40250]: pgmap v3746: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:03:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:03:23.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:22 smithi022 ceph-mon[31360]: pgmap v3747: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:22 smithi022 ceph-mon[40250]: pgmap v3747: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:23.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:23.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:23.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:22 smithi122 ceph-mon[32255]: pgmap v3747: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/72853684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/72853684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/72853684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:24 smithi022 ceph-mon[31360]: pgmap v3748: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:24 smithi022 ceph-mon[40250]: pgmap v3748: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:24 smithi122 ceph-mon[32255]: pgmap v3748: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:27 smithi122 ceph-mon[32255]: pgmap v3749: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:27 smithi022 ceph-mon[31360]: pgmap v3749: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:03:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:27 smithi022 ceph-mon[40250]: pgmap v3749: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:28.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:28.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:28.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:28.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:29 smithi122 ceph-mon[32255]: pgmap v3750: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1440745922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:29 smithi022 ceph-mon[31360]: pgmap v3750: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1440745922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:29 smithi022 ceph-mon[40250]: pgmap v3750: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1440745922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:03:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:03:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:30] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T20:03:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:31 smithi122 ceph-mon[32255]: pgmap v3751: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:31 smithi022 ceph-mon[31360]: pgmap v3751: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:31 smithi022 ceph-mon[40250]: pgmap v3751: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:03:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:03:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[31360]: pgmap v3752: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:03:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:03:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:03:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:03:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[40250]: pgmap v3752: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:03:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:03:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:03:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:03:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:32 smithi122 ceph-mon[32255]: pgmap v3752: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:03:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:03:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:03:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:03:33.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:34.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:34.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:34.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/386007695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/386007695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/386007695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:35 smithi122 ceph-mon[32255]: pgmap v3753: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:35 smithi022 ceph-mon[31360]: pgmap v3753: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:35 smithi022 ceph-mon[40250]: pgmap v3753: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:37 smithi122 ceph-mon[32255]: pgmap v3754: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:37 smithi022 ceph-mon[31360]: pgmap v3754: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:03:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:37 smithi022 ceph-mon[40250]: pgmap v3754: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:39.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:39.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:39.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:39.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:39.500 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:39 smithi022 ceph-mon[31360]: pgmap v3755: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:39.500 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:39 smithi022 ceph-mon[40250]: pgmap v3755: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:39 smithi122 ceph-mon[32255]: pgmap v3755: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:03:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:03:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:40] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:03:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2329270324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2329270324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2329270324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:41 smithi122 ceph-mon[32255]: pgmap v3756: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:41 smithi022 ceph-mon[31360]: pgmap v3756: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:41 smithi022 ceph-mon[40250]: pgmap v3756: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:03:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:03:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:42 smithi022 ceph-mon[31360]: pgmap v3757: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:42 smithi022 ceph-mon[40250]: pgmap v3757: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:42 smithi122 ceph-mon[32255]: pgmap v3757: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:03:44.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:44.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:44.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:44.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:45 smithi122 ceph-mon[32255]: pgmap v3758: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3283190802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:45 smithi022 ceph-mon[31360]: pgmap v3758: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3283190802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:45 smithi022 ceph-mon[40250]: pgmap v3758: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:03:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3283190802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:47 smithi122 ceph-mon[32255]: pgmap v3759: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:47 smithi022 ceph-mon[31360]: pgmap v3759: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:03:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:47 smithi022 ceph-mon[40250]: pgmap v3759: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:49 smithi122 ceph-mon[32255]: pgmap v3760: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:49 smithi022 ceph-mon[31360]: pgmap v3760: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:49 smithi022 ceph-mon[40250]: pgmap v3760: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:49.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:03:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:03:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:50] "GET /metrics HTTP/1.1" 200 202321 "" "Prometheus/2.18.1" 2021-12-15T20:03:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1339513730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1339513730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:03:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1339513730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:51 smithi122 ceph-mon[32255]: pgmap v3761: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:51 smithi022 ceph-mon[31360]: pgmap v3761: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:51 smithi022 ceph-mon[40250]: pgmap v3761: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:03:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:03:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:52 smithi022 ceph-mon[31360]: pgmap v3762: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:52 smithi022 ceph-mon[40250]: pgmap v3762: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:52 smithi122 ceph-mon[32255]: pgmap v3762: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:03:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[31360]: pgmap v3763: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:03:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:03:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:03:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:03:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[40250]: pgmap v3763: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:03:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:03:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:03:55.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:03:55.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:03:55.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:03:55.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:03:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:55 smithi122 ceph-mon[32255]: pgmap v3763: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:03:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:03:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:03:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:03:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4188439126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4188439126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4188439126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:03:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:57 smithi122 ceph-mon[32255]: pgmap v3764: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:57 smithi022 ceph-mon[31360]: pgmap v3764: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:03:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:57 smithi022 ceph-mon[40250]: pgmap v3764: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:03:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:03:59 smithi122 ceph-mon[32255]: pgmap v3765: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:03:59 smithi022 ceph-mon[31360]: pgmap v3765: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:03:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:03:59 smithi022 ceph-mon[40250]: pgmap v3765: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:00.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:03:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:04:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:00] "GET /metrics HTTP/1.1" 200 202321 "" "Prometheus/2.18.1" 2021-12-15T20:04:00.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:00.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:00.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:01 smithi122 ceph-mon[32255]: pgmap v3766: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/88433583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:01 smithi022 ceph-mon[31360]: pgmap v3766: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/88433583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:01 smithi022 ceph-mon[40250]: pgmap v3766: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/88433583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:04:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:04:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:02 smithi022 ceph-mon[31360]: pgmap v3767: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:02 smithi022 ceph-mon[40250]: pgmap v3767: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:02 smithi122 ceph-mon[32255]: pgmap v3767: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:05 smithi122 ceph-mon[32255]: pgmap v3768: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:05 smithi022 ceph-mon[31360]: pgmap v3768: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:05 smithi022 ceph-mon[40250]: pgmap v3768: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:05.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:06.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:06.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:06.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2377681478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2377681478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2377681478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:07 smithi122 ceph-mon[32255]: pgmap v3769: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:07 smithi022 ceph-mon[31360]: pgmap v3769: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:04:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:07 smithi022 ceph-mon[40250]: pgmap v3769: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:09 smithi122 ceph-mon[32255]: pgmap v3770: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:09 smithi022 ceph-mon[31360]: pgmap v3770: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:09 smithi022 ceph-mon[40250]: pgmap v3770: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:10.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:04:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:04:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:10] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T20:04:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:11.345 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:11 smithi122 ceph-mon[32255]: pgmap v3771: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:11.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:11.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:11.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:11.716 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:11 smithi022 ceph-mon[31360]: pgmap v3771: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:11.717 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:11 smithi022 ceph-mon[40250]: pgmap v3771: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2001344440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:04:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:04:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2001344440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2001344440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:13 smithi122 ceph-mon[32255]: pgmap v3772: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:13 smithi022 ceph-mon[31360]: pgmap v3772: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:13 smithi022 ceph-mon[40250]: pgmap v3772: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:15 smithi122 ceph-mon[32255]: pgmap v3773: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:15 smithi022 ceph-mon[31360]: pgmap v3773: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:15 smithi022 ceph-mon[40250]: pgmap v3773: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:16.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:17.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:17.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:17.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:17 smithi122 ceph-mon[32255]: pgmap v3774: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/423748299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:04:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:17 smithi022 ceph-mon[31360]: pgmap v3774: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/423748299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:17 smithi022 ceph-mon[40250]: pgmap v3774: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/423748299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:19 smithi122 ceph-mon[32255]: pgmap v3775: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:19 smithi022 ceph-mon[31360]: pgmap v3775: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:19 smithi022 ceph-mon[40250]: pgmap v3775: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:20.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:04:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:04:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:20] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:04:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:21 smithi122 ceph-mon[32255]: pgmap v3776: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:21 smithi022 ceph-mon[31360]: pgmap v3776: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:21 smithi022 ceph-mon[40250]: pgmap v3776: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:22.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:22.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:22.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:22.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:04:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:04:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:23 smithi022 ceph-mon[31360]: pgmap v3777: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2630508622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:23 smithi022 ceph-mon[40250]: pgmap v3777: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2630508622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:23 smithi122 ceph-mon[32255]: pgmap v3777: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2630508622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:25 smithi122 ceph-mon[32255]: pgmap v3778: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:25 smithi022 ceph-mon[31360]: pgmap v3778: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:25 smithi022 ceph-mon[40250]: pgmap v3778: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:27.462 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:27 smithi122 ceph-mon[32255]: pgmap v3779: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:27 smithi022 ceph-mon[31360]: pgmap v3779: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:04:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:27 smithi022 ceph-mon[40250]: pgmap v3779: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:27.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:27.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:27.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3122226324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3122226324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3122226324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:29 smithi122 ceph-mon[32255]: pgmap v3780: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:29 smithi022 ceph-mon[31360]: pgmap v3780: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:29 smithi022 ceph-mon[40250]: pgmap v3780: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:04:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:04:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:30] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:04:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:31 smithi122 ceph-mon[32255]: pgmap v3781: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:31 smithi022 ceph-mon[31360]: pgmap v3781: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:31 smithi022 ceph-mon[40250]: pgmap v3781: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:04:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:04:32.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:33.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:33.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:33.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:33.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[31360]: pgmap v3782: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:33.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:04:33.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:04:33.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:04:33.205 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:04:33.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[40250]: pgmap v3782: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:33.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:04:33.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:04:33.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:04:33.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:04:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:33 smithi122 ceph-mon[32255]: pgmap v3782: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:04:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:04:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:04:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:04:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3840522739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3840522739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3840522739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:35 smithi022 ceph-mon[31360]: pgmap v3783: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:35 smithi022 ceph-mon[40250]: pgmap v3783: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:35 smithi122 ceph-mon[32255]: pgmap v3783: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:37 smithi122 ceph-mon[32255]: pgmap v3784: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:37 smithi022 ceph-mon[31360]: pgmap v3784: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:04:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:37 smithi022 ceph-mon[40250]: pgmap v3784: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:38.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:38.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:38.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:38.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:39 smithi122 ceph-mon[32255]: pgmap v3785: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1986426804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:39 smithi022 ceph-mon[31360]: pgmap v3785: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1986426804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:39 smithi022 ceph-mon[40250]: pgmap v3785: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1986426804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:04:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:04:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:40] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:04:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:41 smithi122 ceph-mon[32255]: pgmap v3786: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:41 smithi022 ceph-mon[31360]: pgmap v3786: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:41 smithi022 ceph-mon[40250]: pgmap v3786: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:04:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:04:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:42 smithi022 ceph-mon[31360]: pgmap v3787: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:42 smithi022 ceph-mon[40250]: pgmap v3787: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:42 smithi122 ceph-mon[32255]: pgmap v3787: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:43.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:43.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:43.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:43.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1231167642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1231167642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1231167642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:45 smithi022 ceph-mon[31360]: pgmap v3788: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:45 smithi022 ceph-mon[40250]: pgmap v3788: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:45 smithi122 ceph-mon[32255]: pgmap v3788: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:47 smithi122 ceph-mon[32255]: pgmap v3789: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:47 smithi022 ceph-mon[31360]: pgmap v3789: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:04:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:47 smithi022 ceph-mon[40250]: pgmap v3789: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:48.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:49.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:49.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:49.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:49 smithi122 ceph-mon[32255]: pgmap v3790: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3862198451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:49 smithi022 ceph-mon[31360]: pgmap v3790: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3862198451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:49 smithi022 ceph-mon[40250]: pgmap v3790: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3862198451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:04:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:04:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:04:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:04:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:51 smithi122 ceph-mon[32255]: pgmap v3791: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:51 smithi022 ceph-mon[31360]: pgmap v3791: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:51 smithi022 ceph-mon[40250]: pgmap v3791: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:04:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:04:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:53 smithi022 ceph-mon[31360]: pgmap v3792: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:53 smithi022 ceph-mon[40250]: pgmap v3792: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:53 smithi122 ceph-mon[32255]: pgmap v3792: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:54.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:54.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:04:54.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:04:54.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:04:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[31360]: pgmap v3793: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1179426158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:04:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:04:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:04:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[40250]: pgmap v3793: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1179426158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:04:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:04:55.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:04:55.650 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:55 smithi122 ceph-mon[32255]: pgmap v3793: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:55.650 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1179426158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:04:55.650 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:04:55.651 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:04:55.651 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:04:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:04:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:04:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:04:57.661 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:04:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:57 smithi122 ceph-mon[32255]: pgmap v3794: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:57 smithi022 ceph-mon[31360]: pgmap v3794: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:57 smithi022 ceph-mon[40250]: pgmap v3794: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:04:59.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:04:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:04:59 smithi122 ceph-mon[32255]: pgmap v3795: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:04:59.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:04:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:05:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:04:59 smithi022 ceph-mon[31360]: pgmap v3795: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:04:59 smithi022 ceph-mon[40250]: pgmap v3795: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:05:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1978507868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1978507868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1978507868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:01 smithi122 ceph-mon[32255]: pgmap v3796: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:01 smithi022 ceph-mon[31360]: pgmap v3796: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:01 smithi022 ceph-mon[40250]: pgmap v3796: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:05:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:05:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:03 smithi022 ceph-mon[31360]: pgmap v3797: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:03 smithi022 ceph-mon[40250]: pgmap v3797: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:03 smithi122 ceph-mon[32255]: pgmap v3797: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:05.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:05.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:05.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:05.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:05 smithi122 ceph-mon[32255]: pgmap v3798: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:05 smithi022 ceph-mon[31360]: pgmap v3798: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:05 smithi022 ceph-mon[40250]: pgmap v3798: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1029429446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1029429446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1029429446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:07 smithi122 ceph-mon[32255]: pgmap v3799: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:07 smithi022 ceph-mon[31360]: pgmap v3799: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:05:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:07 smithi022 ceph-mon[40250]: pgmap v3799: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:09 smithi122 ceph-mon[32255]: pgmap v3800: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:09 smithi022 ceph-mon[31360]: pgmap v3800: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:09 smithi022 ceph-mon[40250]: pgmap v3800: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:10.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:05:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:05:10.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:10] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:05:10.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:10.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:10.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:11 smithi122 ceph-mon[32255]: pgmap v3801: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/692853590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:11 smithi022 ceph-mon[31360]: pgmap v3801: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:11.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/692853590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:11.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:11 smithi022 ceph-mon[40250]: pgmap v3801: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:11.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/692853590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:05:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:05:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:13 smithi022 ceph-mon[31360]: pgmap v3802: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:13 smithi022 ceph-mon[40250]: pgmap v3802: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:13 smithi122 ceph-mon[32255]: pgmap v3802: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:15 smithi122 ceph-mon[32255]: pgmap v3803: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:15 smithi022 ceph-mon[31360]: pgmap v3803: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:15 smithi022 ceph-mon[40250]: pgmap v3803: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:15.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:16.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:16.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:16.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3921072572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3921072572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3921072572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:17 smithi122 ceph-mon[32255]: pgmap v3804: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:17 smithi022 ceph-mon[31360]: pgmap v3804: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:05:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:17 smithi022 ceph-mon[40250]: pgmap v3804: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:19 smithi122 ceph-mon[32255]: pgmap v3805: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:19 smithi022 ceph-mon[31360]: pgmap v3805: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:19 smithi022 ceph-mon[40250]: pgmap v3805: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:20.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:05:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:05:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:20] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T20:05:21.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:21 smithi022 ceph-mon[31360]: pgmap v3806: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:21 smithi022 ceph-mon[40250]: pgmap v3806: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:21.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:21.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:21.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:21 smithi122 ceph-mon[32255]: pgmap v3806: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3622581819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:05:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:05:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3622581819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3622581819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:23 smithi122 ceph-mon[32255]: pgmap v3807: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:23 smithi022 ceph-mon[31360]: pgmap v3807: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:23 smithi022 ceph-mon[40250]: pgmap v3807: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:25 smithi122 ceph-mon[32255]: pgmap v3808: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:25 smithi022 ceph-mon[31360]: pgmap v3808: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:25 smithi022 ceph-mon[40250]: pgmap v3808: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:26.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:26.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:26.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:26.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:27 smithi122 ceph-mon[32255]: pgmap v3809: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/336101690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:05:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:27 smithi022 ceph-mon[31360]: pgmap v3809: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/336101690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:27 smithi022 ceph-mon[40250]: pgmap v3809: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/336101690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:29 smithi122 ceph-mon[32255]: pgmap v3810: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:29 smithi022 ceph-mon[31360]: pgmap v3810: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:29 smithi022 ceph-mon[40250]: pgmap v3810: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:30.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:05:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:05:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:30] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T20:05:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:31 smithi122 ceph-mon[32255]: pgmap v3811: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:31 smithi022 ceph-mon[40250]: pgmap v3811: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:31 smithi022 ceph-mon[31360]: pgmap v3811: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:31.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:32.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:32.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:32.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:05:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:05:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/670853304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[31360]: pgmap v3812: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:05:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:05:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:05:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:05:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/670853304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[40250]: pgmap v3812: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:05:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:05:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:05:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:05:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/670853304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:33 smithi122 ceph-mon[32255]: pgmap v3812: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:05:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:05:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:05:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:05:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:35 smithi122 ceph-mon[32255]: pgmap v3813: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:35 smithi022 ceph-mon[31360]: pgmap v3813: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:35 smithi022 ceph-mon[40250]: pgmap v3813: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:37.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:37 smithi022 ceph-mon[31360]: pgmap v3814: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:05:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:37 smithi022 ceph-mon[40250]: pgmap v3814: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:37.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:37.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:37.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:37 smithi122 ceph-mon[32255]: pgmap v3814: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3508571138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3508571138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3508571138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:39 smithi122 ceph-mon[32255]: pgmap v3815: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:39 smithi022 ceph-mon[31360]: pgmap v3815: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:39 smithi022 ceph-mon[40250]: pgmap v3815: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:05:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:05:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:40] "GET /metrics HTTP/1.1" 200 202321 "" "Prometheus/2.18.1" 2021-12-15T20:05:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:41 smithi122 ceph-mon[32255]: pgmap v3816: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:41 smithi022 ceph-mon[31360]: pgmap v3816: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:41 smithi022 ceph-mon[40250]: pgmap v3816: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:42.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:05:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:05:43.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:43.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:43.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:43 smithi022 ceph-mon[31360]: pgmap v3817: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3021293975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:43.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:43 smithi022 ceph-mon[40250]: pgmap v3817: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:43.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3021293975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:43 smithi122 ceph-mon[32255]: pgmap v3817: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3021293975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:45 smithi122 ceph-mon[32255]: pgmap v3818: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:45 smithi022 ceph-mon[31360]: pgmap v3818: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:45 smithi022 ceph-mon[40250]: pgmap v3818: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:47 smithi122 ceph-mon[32255]: pgmap v3819: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:47 smithi022 ceph-mon[31360]: pgmap v3819: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:05:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:47 smithi022 ceph-mon[40250]: pgmap v3819: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:48.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:48.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:48.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:48.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/384760520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/384760520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/384760520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:49 smithi122 ceph-mon[32255]: pgmap v3820: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:49 smithi022 ceph-mon[31360]: pgmap v3820: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:49 smithi022 ceph-mon[40250]: pgmap v3820: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:05:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:05:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:50] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T20:05:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:51 smithi122 ceph-mon[32255]: pgmap v3821: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:51 smithi022 ceph-mon[31360]: pgmap v3821: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:05:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:51 smithi022 ceph-mon[40250]: pgmap v3821: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:05:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:05:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:53 smithi022 ceph-mon[31360]: pgmap v3822: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:53 smithi022 ceph-mon[40250]: pgmap v3822: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:53.382 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:53 smithi122 ceph-mon[32255]: pgmap v3822: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:53.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:53.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:53.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3610357341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3610357341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3610357341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:55 smithi022 ceph-mon[31360]: pgmap v3823: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:55 smithi022 ceph-mon[40250]: pgmap v3823: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:55 smithi122 ceph-mon[32255]: pgmap v3823: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:05:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:05:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:05:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:05:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:05:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:05:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:05:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:05:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:05:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:05:57.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:57 smithi022 ceph-mon[31360]: pgmap v3824: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:05:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:57 smithi022 ceph-mon[40250]: pgmap v3824: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:05:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:57 smithi122 ceph-mon[32255]: pgmap v3824: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:05:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:05:58.756 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:05:59.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:05:59.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:05:59.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:05:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:59 smithi122 ceph-mon[32255]: pgmap v3825: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:05:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1100174473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:59 smithi022 ceph-mon[31360]: pgmap v3825: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:05:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1100174473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:05:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:59 smithi022 ceph-mon[40250]: pgmap v3825: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:05:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:05:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1100174473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:00.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:05:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:06:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:00] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T20:06:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:01 smithi122 ceph-mon[32255]: pgmap v3826: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:01 smithi022 ceph-mon[31360]: pgmap v3826: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:01 smithi022 ceph-mon[40250]: pgmap v3826: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:06:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:06:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:03 smithi022 ceph-mon[31360]: pgmap v3827: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:03 smithi022 ceph-mon[40250]: pgmap v3827: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:03 smithi122 ceph-mon[32255]: pgmap v3827: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:04.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:04.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:04.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:04.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:05 smithi122 ceph-mon[32255]: pgmap v3828: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/923469718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:05 smithi022 ceph-mon[31360]: pgmap v3828: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/923469718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:05 smithi022 ceph-mon[40250]: pgmap v3828: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/923469718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:07 smithi122 ceph-mon[32255]: pgmap v3829: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:07 smithi022 ceph-mon[31360]: pgmap v3829: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:06:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:07 smithi022 ceph-mon[40250]: pgmap v3829: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:09.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:09 smithi122 ceph-mon[32255]: pgmap v3830: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:09 smithi022 ceph-mon[31360]: pgmap v3830: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:09 smithi022 ceph-mon[40250]: pgmap v3830: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:10.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:06:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:06:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2195926177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:10.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:10] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:06:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2195926177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2195926177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:11 smithi122 ceph-mon[32255]: pgmap v3831: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:11 smithi022 ceph-mon[31360]: pgmap v3831: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:11 smithi022 ceph-mon[40250]: pgmap v3831: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:06:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:06:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:13 smithi022 ceph-mon[31360]: pgmap v3832: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:13 smithi022 ceph-mon[40250]: pgmap v3832: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:13 smithi122 ceph-mon[32255]: pgmap v3832: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:14.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:15 smithi022 ceph-mon[31360]: pgmap v3833: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1636341522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:15 smithi022 ceph-mon[40250]: pgmap v3833: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1636341522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:15 smithi122 ceph-mon[32255]: pgmap v3833: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1636341522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:17 smithi122 ceph-mon[32255]: pgmap v3834: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:17 smithi022 ceph-mon[31360]: pgmap v3834: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:06:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:17 smithi022 ceph-mon[40250]: pgmap v3834: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:19 smithi122 ceph-mon[32255]: pgmap v3835: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:19 smithi022 ceph-mon[31360]: pgmap v3835: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:19 smithi022 ceph-mon[40250]: pgmap v3835: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:20.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:06:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:06:20.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:20] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:06:20.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:21 smithi122 ceph-mon[32255]: pgmap v3836: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/29684244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:21 smithi022 ceph-mon[31360]: pgmap v3836: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/29684244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:21 smithi022 ceph-mon[40250]: pgmap v3836: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/29684244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:06:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:06:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:23 smithi022 ceph-mon[31360]: pgmap v3837: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:23 smithi022 ceph-mon[40250]: pgmap v3837: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:23 smithi122 ceph-mon[32255]: pgmap v3837: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:25.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:25 smithi122 ceph-mon[32255]: pgmap v3838: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:25 smithi022 ceph-mon[31360]: pgmap v3838: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:25 smithi022 ceph-mon[40250]: pgmap v3838: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:25.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:25.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:25.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4177091023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4177091023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4177091023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:27 smithi122 ceph-mon[32255]: pgmap v3839: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:27 smithi022 ceph-mon[31360]: pgmap v3839: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:06:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:27 smithi022 ceph-mon[40250]: pgmap v3839: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:29 smithi122 ceph-mon[32255]: pgmap v3840: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:29 smithi022 ceph-mon[31360]: pgmap v3840: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:29 smithi022 ceph-mon[40250]: pgmap v3840: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:06:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:06:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:30] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:06:30.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:31.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:31.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:31.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:31 smithi122 ceph-mon[32255]: pgmap v3841: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3355415569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:31 smithi022 ceph-mon[31360]: pgmap v3841: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3355415569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:31 smithi022 ceph-mon[40250]: pgmap v3841: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3355415569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:06:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:06:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[40250]: pgmap v3842: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:06:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:06:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:06:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:06:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[31360]: pgmap v3842: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:06:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:06:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:06:33.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:06:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:33 smithi122 ceph-mon[32255]: pgmap v3842: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:06:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:06:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:06:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:06:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:35 smithi122 ceph-mon[32255]: pgmap v3843: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:35 smithi022 ceph-mon[31360]: pgmap v3843: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:35 smithi022 ceph-mon[40250]: pgmap v3843: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:36.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:36.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:36.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:37 smithi122 ceph-mon[32255]: pgmap v3844: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2483405431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:06:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:37 smithi022 ceph-mon[31360]: pgmap v3844: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2483405431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:37 smithi022 ceph-mon[40250]: pgmap v3844: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2483405431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:39 smithi122 ceph-mon[32255]: pgmap v3845: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:39 smithi022 ceph-mon[31360]: pgmap v3845: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:39 smithi022 ceph-mon[40250]: pgmap v3845: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:40.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:06:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:06:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:40] "GET /metrics HTTP/1.1" 200 202323 "" "Prometheus/2.18.1" 2021-12-15T20:06:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:41 smithi122 ceph-mon[32255]: pgmap v3846: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:41.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:41.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:41.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:41 smithi022 ceph-mon[31360]: pgmap v3846: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:41.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:41.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:41 smithi022 ceph-mon[40250]: pgmap v3846: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:42.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:42.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:42.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:06:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:06:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1073875731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:43 smithi022 ceph-mon[31360]: pgmap v3847: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1073875731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:43 smithi022 ceph-mon[40250]: pgmap v3847: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1073875731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:43.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:43 smithi122 ceph-mon[32255]: pgmap v3847: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:45 smithi122 ceph-mon[32255]: pgmap v3848: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:45 smithi022 ceph-mon[31360]: pgmap v3848: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:45 smithi022 ceph-mon[40250]: pgmap v3848: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:47.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:47.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:47.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:47.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:47 smithi122 ceph-mon[32255]: pgmap v3849: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:47 smithi022 ceph-mon[31360]: pgmap v3849: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:06:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:47 smithi022 ceph-mon[40250]: pgmap v3849: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/846742168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/846742168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/846742168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:49 smithi122 ceph-mon[32255]: pgmap v3850: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:49 smithi022 ceph-mon[31360]: pgmap v3850: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:49 smithi022 ceph-mon[40250]: pgmap v3850: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:50.176 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:06:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:06:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:50] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:06:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:51 smithi122 ceph-mon[32255]: pgmap v3851: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:51 smithi022 ceph-mon[31360]: pgmap v3851: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:06:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:51 smithi022 ceph-mon[40250]: pgmap v3851: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:52.480 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:06:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:06:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:53 smithi022 ceph-mon[31360]: pgmap v3852: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2594991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:53 smithi022 ceph-mon[40250]: pgmap v3852: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2594991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:53 smithi122 ceph-mon[32255]: pgmap v3852: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2594991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:55 smithi122 ceph-mon[32255]: pgmap v3853: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:55 smithi022 ceph-mon[31360]: pgmap v3853: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:55 smithi022 ceph-mon[40250]: pgmap v3853: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:06:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:06:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:06:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:06:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:06:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:06:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:06:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:06:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:06:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:06:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[31360]: pgmap v3854: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[40250]: pgmap v3854: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.850 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:06:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:57 smithi122 ceph-mon[32255]: pgmap v3854: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:06:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:06:58.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:06:58.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:06:58.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:06:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2635529404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2635529404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2635529404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:06:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:06:59 smithi022 ceph-mon[31360]: pgmap v3855: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:06:59 smithi022 ceph-mon[40250]: pgmap v3855: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:06:59.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:06:59 smithi122 ceph-mon[32255]: pgmap v3855: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:00.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:00] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:07:00.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:06:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:07:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:01 smithi122 ceph-mon[32255]: pgmap v3856: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:01 smithi022 ceph-mon[31360]: pgmap v3856: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:01 smithi022 ceph-mon[40250]: pgmap v3856: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:07:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:07:03.223 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:03 smithi122 ceph-mon[32255]: pgmap v3857: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:03 smithi022 ceph-mon[31360]: pgmap v3857: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:03 smithi022 ceph-mon[40250]: pgmap v3857: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:03.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:03.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:03.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3378707289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:04.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3378707289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:04.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3378707289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:05 smithi122 ceph-mon[32255]: pgmap v3858: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:05 smithi022 ceph-mon[31360]: pgmap v3858: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:05 smithi022 ceph-mon[40250]: pgmap v3858: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:07 smithi122 ceph-mon[32255]: pgmap v3859: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:07 smithi022 ceph-mon[31360]: pgmap v3859: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:07:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:07 smithi022 ceph-mon[40250]: pgmap v3859: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:08.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:08.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:08.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:08.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:09 smithi122 ceph-mon[32255]: pgmap v3860: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2336276472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:09 smithi022 ceph-mon[31360]: pgmap v3860: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2336276472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:09 smithi022 ceph-mon[40250]: pgmap v3860: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2336276472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:07:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:07:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:10] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:07:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:11 smithi122 ceph-mon[32255]: pgmap v3861: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:11 smithi022 ceph-mon[31360]: pgmap v3861: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:11 smithi022 ceph-mon[40250]: pgmap v3861: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:07:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:07:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:13 smithi122 ceph-mon[32255]: pgmap v3862: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:13 smithi022 ceph-mon[31360]: pgmap v3862: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:13 smithi022 ceph-mon[40250]: pgmap v3862: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:13.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:14.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:14.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:14.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2333562268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2333562268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2333562268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:15 smithi122 ceph-mon[32255]: pgmap v3863: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:15 smithi022 ceph-mon[31360]: pgmap v3863: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:15 smithi022 ceph-mon[40250]: pgmap v3863: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:17 smithi122 ceph-mon[32255]: pgmap v3864: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:17 smithi022 ceph-mon[31360]: pgmap v3864: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:07:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:17 smithi022 ceph-mon[40250]: pgmap v3864: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:19.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:19 smithi122 ceph-mon[32255]: pgmap v3865: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:19.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:19.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:19.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:19.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:19 smithi022 ceph-mon[31360]: pgmap v3865: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:19.706 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:19 smithi022 ceph-mon[40250]: pgmap v3865: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:20.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:07:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:07:20.387 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:07:20.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1793802411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1793802411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1793802411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:21 smithi122 ceph-mon[32255]: pgmap v3866: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:21 smithi022 ceph-mon[31360]: pgmap v3866: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:21 smithi022 ceph-mon[40250]: pgmap v3866: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:07:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:07:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:23 smithi122 ceph-mon[32255]: pgmap v3867: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:23 smithi022 ceph-mon[31360]: pgmap v3867: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:23 smithi022 ceph-mon[40250]: pgmap v3867: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:24.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:25.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:25.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:25.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:25 smithi122 ceph-mon[32255]: pgmap v3868: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2388492732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:25 smithi022 ceph-mon[31360]: pgmap v3868: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2388492732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:25 smithi022 ceph-mon[40250]: pgmap v3868: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2388492732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:27 smithi122 ceph-mon[32255]: pgmap v3869: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:27 smithi022 ceph-mon[31360]: pgmap v3869: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:07:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:27 smithi022 ceph-mon[40250]: pgmap v3869: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:29 smithi022 ceph-mon[31360]: pgmap v3870: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:29 smithi022 ceph-mon[40250]: pgmap v3870: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:29 smithi122 ceph-mon[32255]: pgmap v3870: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:30.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:07:30.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:07:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:07:30.441 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3071977833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3071977833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3071977833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:31 smithi022 ceph-mon[31360]: pgmap v3871: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:31 smithi022 ceph-mon[40250]: pgmap v3871: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:31 smithi122 ceph-mon[32255]: pgmap v3871: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:07:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:07:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:33 smithi122 ceph-mon[32255]: pgmap v3872: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:07:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:07:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:07:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:07:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[31360]: pgmap v3872: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:07:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:07:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:07:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:07:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[40250]: pgmap v3872: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:07:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:07:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:07:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:07:35.443 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:35 smithi122 ceph-mon[32255]: pgmap v3873: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:35 smithi022 ceph-mon[31360]: pgmap v3873: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:35 smithi022 ceph-mon[40250]: pgmap v3873: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:35.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:35.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:35.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4272979493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4272979493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4272979493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:37 smithi022 ceph-mon[31360]: pgmap v3874: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:07:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:37 smithi022 ceph-mon[40250]: pgmap v3874: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:37 smithi122 ceph-mon[32255]: pgmap v3874: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:39 smithi022 ceph-mon[31360]: pgmap v3875: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:39 smithi022 ceph-mon[40250]: pgmap v3875: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:39 smithi122 ceph-mon[32255]: pgmap v3875: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:40] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:07:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:07:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:07:40.811 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:41.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:41.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:41.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:41 smithi022 ceph-mon[31360]: pgmap v3876: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:41.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3511888794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:41.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:41.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:41 smithi022 ceph-mon[40250]: pgmap v3876: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:41.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3511888794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:41 smithi122 ceph-mon[32255]: pgmap v3876: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3511888794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:07:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:07:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:43 smithi122 ceph-mon[32255]: pgmap v3877: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:43 smithi022 ceph-mon[31360]: pgmap v3877: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:43 smithi022 ceph-mon[40250]: pgmap v3877: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:45 smithi022 ceph-mon[31360]: pgmap v3878: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:45 smithi022 ceph-mon[40250]: pgmap v3878: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:45 smithi122 ceph-mon[32255]: pgmap v3878: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:46.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:46.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:46.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:46.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:07:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:47 smithi022 ceph-mon[31360]: pgmap v3879: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1259184531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:47 smithi022 ceph-mon[40250]: pgmap v3879: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1259184531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:47 smithi122 ceph-mon[32255]: pgmap v3879: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1259184531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:49 smithi022 ceph-mon[31360]: pgmap v3880: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:49 smithi022 ceph-mon[40250]: pgmap v3880: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:50] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:07:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:07:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:07:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:49 smithi122 ceph-mon[32255]: pgmap v3880: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:07:51.553 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:51 smithi022 ceph-mon[31360]: pgmap v3881: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:51 smithi022 ceph-mon[40250]: pgmap v3881: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:51.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:51.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:51.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:07:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:51 smithi122 ceph-mon[32255]: pgmap v3881: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1567015444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:07:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:07:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1567015444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1567015444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:53 smithi022 ceph-mon[31360]: pgmap v3882: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:53 smithi022 ceph-mon[40250]: pgmap v3882: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:53 smithi122 ceph-mon[32255]: pgmap v3882: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:07:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:55 smithi022 ceph-mon[31360]: pgmap v3883: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:07:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:55 smithi022 ceph-mon[40250]: pgmap v3883: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:07:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:55 smithi122 ceph-mon[32255]: pgmap v3883: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:07:56.926 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:07:57.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:07:57.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:07:57.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:07:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:57 smithi122 ceph-mon[32255]: pgmap v3884: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:07:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:07:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:07:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:07:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1445427449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:07:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:07:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[31360]: pgmap v3884: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:07:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:07:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:07:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:07:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1445427449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:07:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[40250]: pgmap v3884: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:07:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:07:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:07:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:07:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1445427449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:07:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:07:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:07:59 smithi022 ceph-mon[31360]: pgmap v3885: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:07:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:07:59 smithi022 ceph-mon[40250]: pgmap v3885: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:07:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:07:59 smithi122 ceph-mon[32255]: pgmap v3885: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:00] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:08:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:07:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:08:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:01.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:01 smithi022 ceph-mon[31360]: pgmap v3886: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:01.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:01.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:01 smithi022 ceph-mon[40250]: pgmap v3886: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:01 smithi122 ceph-mon[32255]: pgmap v3886: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:02.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:02.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:02.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:08:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:08:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:03 smithi122 ceph-mon[32255]: pgmap v3887: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2195124350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:03 smithi022 ceph-mon[31360]: pgmap v3887: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2195124350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:03 smithi022 ceph-mon[40250]: pgmap v3887: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:03.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2195124350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:05 smithi022 ceph-mon[31360]: pgmap v3888: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:05 smithi022 ceph-mon[40250]: pgmap v3888: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:05 smithi122 ceph-mon[32255]: pgmap v3888: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:08:07.676 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:07.677 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:08:07.677 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:07 smithi022 ceph-mon[31360]: pgmap v3889: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:08:07.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:07 smithi022 ceph-mon[40250]: pgmap v3889: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:08:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:07 smithi122 ceph-mon[32255]: pgmap v3889: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:08:08.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:08.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:08.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1593014895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1593014895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1593014895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:09 smithi022 ceph-mon[31360]: pgmap v3890: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:08:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:09 smithi022 ceph-mon[40250]: pgmap v3890: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:08:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:09 smithi122 ceph-mon[32255]: pgmap v3890: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:08:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:08:10.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:08:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:08:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:11 smithi022 ceph-mon[31360]: pgmap v3891: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:11 smithi022 ceph-mon[40250]: pgmap v3891: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:11 smithi122 ceph-mon[32255]: pgmap v3891: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:08:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:08:13.047 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:13.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:13.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:13.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:13.418 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:13 smithi022 ceph-mon[31360]: pgmap v3892: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:13.418 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:13 smithi022 ceph-mon[40250]: pgmap v3892: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:13.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:13 smithi122 ceph-mon[32255]: pgmap v3892: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2458875907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2458875907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2458875907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:15 smithi122 ceph-mon[32255]: pgmap v3893: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:15 smithi022 ceph-mon[31360]: pgmap v3893: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:15 smithi022 ceph-mon[40250]: pgmap v3893: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:17 smithi022 ceph-mon[31360]: pgmap v3894: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:08:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:17 smithi022 ceph-mon[40250]: pgmap v3894: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:17 smithi122 ceph-mon[32255]: pgmap v3894: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:18.419 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:18.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:18.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:18.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:19 smithi022 ceph-mon[31360]: pgmap v3895: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3911681127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:19 smithi022 ceph-mon[40250]: pgmap v3895: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3911681127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:19 smithi122 ceph-mon[32255]: pgmap v3895: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3911681127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:20] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:08:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:08:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:08:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:21 smithi022 ceph-mon[31360]: pgmap v3896: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:21 smithi022 ceph-mon[40250]: pgmap v3896: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:21 smithi122 ceph-mon[32255]: pgmap v3896: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:08:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:08:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:23 smithi122 ceph-mon[32255]: pgmap v3897: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:23 smithi022 ceph-mon[31360]: pgmap v3897: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:23 smithi022 ceph-mon[40250]: pgmap v3897: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:23.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:24.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:24.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:24.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1905228550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1905228550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1905228550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:25 smithi022 ceph-mon[31360]: pgmap v3898: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:25 smithi022 ceph-mon[40250]: pgmap v3898: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:25 smithi122 ceph-mon[32255]: pgmap v3898: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:08:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:27 smithi022 ceph-mon[31360]: pgmap v3899: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:27 smithi022 ceph-mon[40250]: pgmap v3899: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:27 smithi122 ceph-mon[32255]: pgmap v3899: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:29.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:29.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:29.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:29.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:29 smithi022 ceph-mon[31360]: pgmap v3900: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:29 smithi022 ceph-mon[40250]: pgmap v3900: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:29 smithi122 ceph-mon[32255]: pgmap v3900: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:30] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:08:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:08:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:08:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2120594466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2120594466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2120594466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:31 smithi022 ceph-mon[31360]: pgmap v3901: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:31 smithi022 ceph-mon[40250]: pgmap v3901: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:31 smithi122 ceph-mon[32255]: pgmap v3901: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:08:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:08:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:33 smithi122 ceph-mon[32255]: pgmap v3902: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:08:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:08:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:08:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:08:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[31360]: pgmap v3902: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:08:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:08:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:08:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:08:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[40250]: pgmap v3902: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:08:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:08:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:08:33.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:08:34.533 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:34.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:34.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:34.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:35 smithi022 ceph-mon[31360]: pgmap v3903: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3712528026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:35 smithi022 ceph-mon[40250]: pgmap v3903: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3712528026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:35 smithi122 ceph-mon[32255]: pgmap v3903: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3712528026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:37 smithi022 ceph-mon[31360]: pgmap v3904: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:08:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:37 smithi022 ceph-mon[40250]: pgmap v3904: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:37 smithi122 ceph-mon[32255]: pgmap v3904: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:39 smithi022 ceph-mon[31360]: pgmap v3905: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:39 smithi022 ceph-mon[40250]: pgmap v3905: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:39 smithi122 ceph-mon[32255]: pgmap v3905: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:40.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:40.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:40.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:40] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:08:40.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:08:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:08:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3510896395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3510896395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3510896395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:41 smithi022 ceph-mon[31360]: pgmap v3906: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:41 smithi022 ceph-mon[40250]: pgmap v3906: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:41 smithi122 ceph-mon[32255]: pgmap v3906: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:08:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:08:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:43 smithi122 ceph-mon[32255]: pgmap v3907: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:43 smithi022 ceph-mon[31360]: pgmap v3907: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:43 smithi022 ceph-mon[40250]: pgmap v3907: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:45.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:45 smithi022 ceph-mon[31360]: pgmap v3908: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:45 smithi022 ceph-mon[40250]: pgmap v3908: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:45.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:45.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:45.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:45 smithi122 ceph-mon[32255]: pgmap v3908: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1093152522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1093152522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1093152522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:47 smithi022 ceph-mon[31360]: pgmap v3909: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:08:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:47 smithi022 ceph-mon[40250]: pgmap v3909: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:47 smithi122 ceph-mon[32255]: pgmap v3909: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:49 smithi022 ceph-mon[31360]: pgmap v3910: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:49 smithi022 ceph-mon[40250]: pgmap v3910: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:49 smithi122 ceph-mon[32255]: pgmap v3910: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:08:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:08:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:08:50.645 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:51.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:51.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:51.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:51 smithi022 ceph-mon[31360]: pgmap v3911: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2534150089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:51 smithi022 ceph-mon[40250]: pgmap v3911: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2534150089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:08:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:51 smithi122 ceph-mon[32255]: pgmap v3911: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2534150089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:08:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:08:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:53 smithi122 ceph-mon[32255]: pgmap v3912: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:53 smithi022 ceph-mon[31360]: pgmap v3912: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:53 smithi022 ceph-mon[40250]: pgmap v3912: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:55 smithi022 ceph-mon[31360]: pgmap v3913: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:55 smithi022 ceph-mon[40250]: pgmap v3913: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:55 smithi122 ceph-mon[32255]: pgmap v3913: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:56.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:08:56.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:08:56.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:08:56.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:08:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1977816983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1977816983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1977816983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:08:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:08:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[31360]: pgmap v3914: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:08:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:08:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:08:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[40250]: pgmap v3914: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:08:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:08:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:08:57.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:57 smithi122 ceph-mon[32255]: pgmap v3914: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:08:57.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:08:57.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:08:57.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:08:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:08:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:08:58 smithi122 ceph-mon[32255]: pgmap v3915: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:08:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:08:58 smithi022 ceph-mon[31360]: pgmap v3915: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:08:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:08:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:08:58 smithi022 ceph-mon[40250]: pgmap v3915: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:09:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:09:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:09:01.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:01.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:01.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:01.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:01.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:01.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:01 smithi022 ceph-mon[40250]: pgmap v3916: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:01.758 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:01.759 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:01 smithi022 ceph-mon[31360]: pgmap v3916: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:01 smithi122 ceph-mon[32255]: pgmap v3916: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1997032573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:09:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:09:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1997032573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1997032573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:03 smithi022 ceph-mon[31360]: pgmap v3917: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:03 smithi022 ceph-mon[40250]: pgmap v3917: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:03 smithi122 ceph-mon[32255]: pgmap v3917: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:05 smithi022 ceph-mon[31360]: pgmap v3918: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:05 smithi022 ceph-mon[40250]: pgmap v3918: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:05 smithi122 ceph-mon[32255]: pgmap v3918: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:06.759 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:07.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:07.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:07.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:09:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:07 smithi022 ceph-mon[31360]: pgmap v3919: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/915118733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:07 smithi022 ceph-mon[40250]: pgmap v3919: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/915118733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:07 smithi122 ceph-mon[32255]: pgmap v3919: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/915118733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:09 smithi022 ceph-mon[31360]: pgmap v3920: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:09 smithi022 ceph-mon[40250]: pgmap v3920: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:09 smithi122 ceph-mon[32255]: pgmap v3920: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:10] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:09:10.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:09:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:09:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:11 smithi022 ceph-mon[31360]: pgmap v3921: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:11 smithi022 ceph-mon[40250]: pgmap v3921: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:11 smithi122 ceph-mon[32255]: pgmap v3921: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:12.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:12.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:12.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:12.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:09:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:09:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:13 smithi122 ceph-mon[32255]: pgmap v3922: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2177656484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:13 smithi022 ceph-mon[31360]: pgmap v3922: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2177656484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:13 smithi022 ceph-mon[40250]: pgmap v3922: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2177656484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:15 smithi022 ceph-mon[31360]: pgmap v3923: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:15 smithi022 ceph-mon[40250]: pgmap v3923: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:15 smithi122 ceph-mon[32255]: pgmap v3923: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:17.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:17 smithi022 ceph-mon[31360]: pgmap v3924: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:09:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:17 smithi022 ceph-mon[40250]: pgmap v3924: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:17.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:17.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:17.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:17 smithi122 ceph-mon[32255]: pgmap v3924: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/357243392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/357243392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/357243392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:19 smithi022 ceph-mon[31360]: pgmap v3925: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:19 smithi022 ceph-mon[40250]: pgmap v3925: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:19 smithi122 ceph-mon[32255]: pgmap v3925: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:20] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:09:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:09:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:09:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:21 smithi022 ceph-mon[31360]: pgmap v3926: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:21 smithi022 ceph-mon[40250]: pgmap v3926: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:21 smithi122 ceph-mon[32255]: pgmap v3926: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:09:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:09:22.874 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:23.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:23.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:23.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:23.250 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:23 smithi022 ceph-mon[31360]: pgmap v3927: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:23.250 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:23 smithi022 ceph-mon[40250]: pgmap v3927: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:23 smithi122 ceph-mon[32255]: pgmap v3927: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1247595403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1247595403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1247595403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:25 smithi122 ceph-mon[32255]: pgmap v3928: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:25 smithi022 ceph-mon[31360]: pgmap v3928: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:25 smithi022 ceph-mon[40250]: pgmap v3928: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:27 smithi022 ceph-mon[31360]: pgmap v3929: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:09:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:27 smithi022 ceph-mon[40250]: pgmap v3929: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:27 smithi122 ceph-mon[32255]: pgmap v3929: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:28.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:28.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:29 smithi022 ceph-mon[40250]: pgmap v3930: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2183286279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:29 smithi022 ceph-mon[31360]: pgmap v3930: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2183286279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:29 smithi122 ceph-mon[32255]: pgmap v3930: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2183286279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:09:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:09:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:30] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:09:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:31 smithi022 ceph-mon[40250]: pgmap v3931: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:31 smithi022 ceph-mon[31360]: pgmap v3931: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:31 smithi122 ceph-mon[32255]: pgmap v3931: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:09:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:09:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:33 smithi122 ceph-mon[32255]: pgmap v3932: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:09:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:09:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:09:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:09:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[40250]: pgmap v3932: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:09:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:09:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[31360]: pgmap v3932: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:09:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:09:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:09:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:09:33.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:33.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:33.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:33.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4251805764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4251805764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4251805764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:35 smithi122 ceph-mon[32255]: pgmap v3933: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:35 smithi022 ceph-mon[31360]: pgmap v3933: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:35 smithi022 ceph-mon[40250]: pgmap v3933: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:37 smithi022 ceph-mon[31360]: pgmap v3934: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:09:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:37 smithi022 ceph-mon[40250]: pgmap v3934: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:37 smithi122 ceph-mon[32255]: pgmap v3934: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:38.993 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:39.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:39.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:39.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:39 smithi022 ceph-mon[31360]: pgmap v3935: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2156459317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:39 smithi022 ceph-mon[40250]: pgmap v3935: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2156459317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:39 smithi122 ceph-mon[32255]: pgmap v3935: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2156459317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:09:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:09:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:40] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:09:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:41 smithi022 ceph-mon[31360]: pgmap v3936: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:41 smithi022 ceph-mon[40250]: pgmap v3936: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:41 smithi122 ceph-mon[32255]: pgmap v3936: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:09:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:09:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:43 smithi122 ceph-mon[32255]: pgmap v3937: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:43 smithi022 ceph-mon[31360]: pgmap v3937: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:43 smithi022 ceph-mon[40250]: pgmap v3937: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:44.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:44.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:44.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:45 smithi022 ceph-mon[31360]: pgmap v3938: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2654384854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:45 smithi022 ceph-mon[40250]: pgmap v3938: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2654384854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:45 smithi122 ceph-mon[32255]: pgmap v3938: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2654384854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:47 smithi022 ceph-mon[31360]: pgmap v3939: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:09:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:47 smithi022 ceph-mon[40250]: pgmap v3939: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:47 smithi122 ceph-mon[32255]: pgmap v3939: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:49.735 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:49.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:49 smithi022 ceph-mon[31360]: pgmap v3940: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:49.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:49 smithi022 ceph-mon[40250]: pgmap v3940: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:49 smithi122 ceph-mon[32255]: pgmap v3940: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:50.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:50.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:50.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:50] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:09:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:09:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:09:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4223046221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4223046221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4223046221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:51 smithi022 ceph-mon[31360]: pgmap v3941: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:51 smithi022 ceph-mon[40250]: pgmap v3941: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:09:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:51 smithi122 ceph-mon[32255]: pgmap v3941: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:09:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:09:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:53 smithi122 ceph-mon[32255]: pgmap v3942: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:53 smithi022 ceph-mon[31360]: pgmap v3942: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:53 smithi022 ceph-mon[40250]: pgmap v3942: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:55.107 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:09:55.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:09:55.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:09:55.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:09:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:55 smithi022 ceph-mon[40250]: pgmap v3943: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2909740434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:55 smithi022 ceph-mon[31360]: pgmap v3943: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2909740434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:55 smithi122 ceph-mon[32255]: pgmap v3943: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2909740434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:09:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:57 smithi022 ceph-mon[31360]: pgmap v3944: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:09:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:57 smithi022 ceph-mon[40250]: pgmap v3944: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:57.933 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:57 smithi122 ceph-mon[32255]: pgmap v3944: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:09:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:09:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:09:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:09:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:09:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:09:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:09:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:09:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:09:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:09:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:09:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:09:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:09:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:09:59 smithi022 ceph-mon[31360]: pgmap v3945: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:09:59 smithi022 ceph-mon[40250]: pgmap v3945: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:09:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:09:59 smithi122 ceph-mon[32255]: pgmap v3945: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:00] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:10:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:10:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:10:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T20:10:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T20:10:00.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:00.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:00.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T20:10:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:01 smithi022 ceph-mon[31360]: pgmap v3946: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1619291402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:01 smithi022 ceph-mon[40250]: pgmap v3946: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1619291402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:01.871 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:01.872 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:01 smithi122 ceph-mon[32255]: pgmap v3946: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:01.872 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1619291402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:10:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:10:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:03 smithi122 ceph-mon[32255]: pgmap v3947: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:03 smithi022 ceph-mon[31360]: pgmap v3947: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:03 smithi022 ceph-mon[40250]: pgmap v3947: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:05 smithi022 ceph-mon[31360]: pgmap v3948: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:05 smithi022 ceph-mon[40250]: pgmap v3948: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:05.846 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:05 smithi122 ceph-mon[32255]: pgmap v3948: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:06.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:06.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:06.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:06.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/860761192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:06.484 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/860761192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/860761192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:07 smithi022 ceph-mon[31360]: pgmap v3949: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:10:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:07 smithi022 ceph-mon[40250]: pgmap v3949: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:07 smithi122 ceph-mon[32255]: pgmap v3949: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:09 smithi022 ceph-mon[31360]: pgmap v3950: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:09 smithi022 ceph-mon[40250]: pgmap v3950: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:09 smithi122 ceph-mon[32255]: pgmap v3950: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:10] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:10:10.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:10:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:10:11.217 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:11.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:11.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:11.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:11 smithi022 ceph-mon[31360]: pgmap v3951: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:11 smithi022 ceph-mon[40250]: pgmap v3951: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:11 smithi122 ceph-mon[32255]: pgmap v3951: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3004117214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:10:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:10:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3004117214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3004117214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:13 smithi022 ceph-mon[31360]: pgmap v3952: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:13 smithi022 ceph-mon[40250]: pgmap v3952: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:13 smithi122 ceph-mon[32255]: pgmap v3952: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:15.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:15 smithi022 ceph-mon[31360]: pgmap v3953: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:15.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:15 smithi022 ceph-mon[40250]: pgmap v3953: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:15 smithi122 ceph-mon[32255]: pgmap v3953: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:16.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:16.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:16.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:16.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:10:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:17 smithi022 ceph-mon[31360]: pgmap v3954: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2461757326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:17 smithi022 ceph-mon[40250]: pgmap v3954: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2461757326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:17 smithi122 ceph-mon[32255]: pgmap v3954: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2461757326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:19 smithi022 ceph-mon[31360]: pgmap v3955: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:19 smithi022 ceph-mon[40250]: pgmap v3955: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:19 smithi122 ceph-mon[32255]: pgmap v3955: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:20] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:10:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:10:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:10:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:21 smithi022 ceph-mon[31360]: pgmap v3956: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:21 smithi022 ceph-mon[40250]: pgmap v3956: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:21 smithi122 ceph-mon[32255]: pgmap v3956: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:21.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:22.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:22.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:22.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:10:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:10:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2327355935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:23 smithi122 ceph-mon[32255]: pgmap v3957: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2327355935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:23 smithi022 ceph-mon[31360]: pgmap v3957: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2327355935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:23 smithi022 ceph-mon[40250]: pgmap v3957: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:25 smithi022 ceph-mon[31360]: pgmap v3958: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:25 smithi022 ceph-mon[40250]: pgmap v3958: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:25 smithi122 ceph-mon[32255]: pgmap v3958: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:27.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:27.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:27.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:27.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:27.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:10:27.709 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:27 smithi022 ceph-mon[31360]: pgmap v3959: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:27.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:27 smithi022 ceph-mon[40250]: pgmap v3959: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:27 smithi122 ceph-mon[32255]: pgmap v3959: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3213618042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3213618042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3213618042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:29 smithi022 ceph-mon[31360]: pgmap v3960: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:29 smithi022 ceph-mon[40250]: pgmap v3960: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:29 smithi122 ceph-mon[32255]: pgmap v3960: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:30] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:10:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:10:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:10:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:31 smithi022 ceph-mon[31360]: pgmap v3961: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:31 smithi022 ceph-mon[40250]: pgmap v3961: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:31 smithi122 ceph-mon[32255]: pgmap v3961: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:10:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:10:32.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:33.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:33.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:33.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:33 smithi122 ceph-mon[32255]: pgmap v3962: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:10:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:10:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:10:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:10:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/170090223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[31360]: pgmap v3962: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:10:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:10:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:10:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:10:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/170090223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[40250]: pgmap v3962: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:10:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:10:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:10:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:10:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/170090223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:35 smithi022 ceph-mon[31360]: pgmap v3963: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:35 smithi022 ceph-mon[40250]: pgmap v3963: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:35 smithi122 ceph-mon[32255]: pgmap v3963: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:37 smithi022 ceph-mon[31360]: pgmap v3964: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:10:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:37 smithi022 ceph-mon[40250]: pgmap v3964: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:37 smithi122 ceph-mon[32255]: pgmap v3964: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:38.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:38.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:38.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:38.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3963796070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3963796070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3963796070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:39 smithi022 ceph-mon[31360]: pgmap v3965: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:39.823 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:39 smithi022 ceph-mon[40250]: pgmap v3965: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:39 smithi122 ceph-mon[32255]: pgmap v3965: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:40] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:10:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:10:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:10:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:41 smithi022 ceph-mon[31360]: pgmap v3966: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:41 smithi022 ceph-mon[40250]: pgmap v3966: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:41 smithi122 ceph-mon[32255]: pgmap v3966: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:10:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:10:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:43 smithi122 ceph-mon[32255]: pgmap v3967: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:43.444 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:43.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:43 smithi022 ceph-mon[31360]: pgmap v3967: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:43.445 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:43 smithi022 ceph-mon[40250]: pgmap v3967: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:43.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:43.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:43.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1572077394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1572077394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1572077394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:45 smithi122 ceph-mon[32255]: pgmap v3968: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:45 smithi022 ceph-mon[31360]: pgmap v3968: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:45 smithi022 ceph-mon[40250]: pgmap v3968: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:47 smithi022 ceph-mon[31360]: pgmap v3969: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:10:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:47 smithi022 ceph-mon[40250]: pgmap v3969: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:47 smithi122 ceph-mon[32255]: pgmap v3969: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:10:48.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:49.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:49.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:49.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:49 smithi022 ceph-mon[31360]: pgmap v3970: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/135068116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:49 smithi022 ceph-mon[40250]: pgmap v3970: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/135068116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:49 smithi122 ceph-mon[32255]: pgmap v3970: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/135068116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:50] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:10:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:10:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:10:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:51 smithi022 ceph-mon[31360]: pgmap v3971: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:51 smithi022 ceph-mon[40250]: pgmap v3971: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:10:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:51 smithi122 ceph-mon[32255]: pgmap v3971: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:10:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:10:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:53 smithi122 ceph-mon[32255]: pgmap v3972: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:53 smithi022 ceph-mon[31360]: pgmap v3972: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:53 smithi022 ceph-mon[40250]: pgmap v3972: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:54.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:54.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:54.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:54.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:55 smithi022 ceph-mon[40250]: pgmap v3973: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3840411121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:55 smithi022 ceph-mon[31360]: pgmap v3973: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3840411121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:55 smithi122 ceph-mon[32255]: pgmap v3973: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3840411121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:10:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:57 smithi022 ceph-mon[31360]: pgmap v3974: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:10:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:57 smithi022 ceph-mon[40250]: pgmap v3974: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:57 smithi122 ceph-mon[32255]: pgmap v3974: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:10:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:10:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:10:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:10:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:10:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:10:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:10:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:10:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:10:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:10:59.554 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:10:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:10:59 smithi022 ceph-mon[31360]: pgmap v3975: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:10:59 smithi022 ceph-mon[40250]: pgmap v3975: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:10:59.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:10:59.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:10:59.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:10:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:10:59 smithi122 ceph-mon[32255]: pgmap v3975: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:00] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:11:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:11:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:11:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1565114318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1565114318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1565114318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:01 smithi022 ceph-mon[31360]: pgmap v3976: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:01 smithi022 ceph-mon[40250]: pgmap v3976: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:01 smithi122 ceph-mon[32255]: pgmap v3976: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:11:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:11:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:03 smithi022 ceph-mon[31360]: pgmap v3977: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:03 smithi022 ceph-mon[40250]: pgmap v3977: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:03.410 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:03.411 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:03 smithi122 ceph-mon[32255]: pgmap v3977: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:03.411 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:11:04.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:05 smithi122 ceph-mon[32255]: pgmap v3978: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/210420003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:05 smithi022 ceph-mon[31360]: pgmap v3978: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/210420003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:05 smithi022 ceph-mon[40250]: pgmap v3978: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/210420003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:07 smithi022 ceph-mon[31360]: pgmap v3979: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:11:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:07 smithi022 ceph-mon[40250]: pgmap v3979: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:07 smithi122 ceph-mon[32255]: pgmap v3979: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:09 smithi022 ceph-mon[31360]: pgmap v3980: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:09 smithi022 ceph-mon[40250]: pgmap v3980: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:09 smithi122 ceph-mon[32255]: pgmap v3980: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:10.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:10] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:11:10.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:11:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:11:10.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:10.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:10.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:11 smithi022 ceph-mon[31360]: pgmap v3981: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3616649625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:11 smithi022 ceph-mon[40250]: pgmap v3981: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3616649625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:11 smithi122 ceph-mon[32255]: pgmap v3981: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3616649625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:11:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:11:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:13 smithi122 ceph-mon[32255]: pgmap v3982: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:13 smithi022 ceph-mon[31360]: pgmap v3982: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:13 smithi022 ceph-mon[40250]: pgmap v3982: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:15.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:15 smithi022 ceph-mon[31360]: pgmap v3983: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:15 smithi022 ceph-mon[40250]: pgmap v3983: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:15 smithi122 ceph-mon[32255]: pgmap v3983: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:16.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:16.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:16.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:16.505 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3115035784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:16.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3115035784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3115035784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:17 smithi022 ceph-mon[31360]: pgmap v3984: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:11:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:17 smithi022 ceph-mon[40250]: pgmap v3984: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:17 smithi122 ceph-mon[32255]: pgmap v3984: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:19 smithi022 ceph-mon[31360]: pgmap v3985: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:19 smithi022 ceph-mon[40250]: pgmap v3985: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:19 smithi122 ceph-mon[32255]: pgmap v3985: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:20] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:11:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:11:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:11:21.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:21.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:21.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:21.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:21 smithi022 ceph-mon[31360]: pgmap v3986: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4025144548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:21 smithi022 ceph-mon[40250]: pgmap v3986: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4025144548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:21 smithi122 ceph-mon[32255]: pgmap v3986: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4025144548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:11:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:11:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:23 smithi122 ceph-mon[32255]: pgmap v3987: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:23 smithi022 ceph-mon[31360]: pgmap v3987: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:23 smithi022 ceph-mon[40250]: pgmap v3987: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:25 smithi022 ceph-mon[31360]: pgmap v3988: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:25 smithi022 ceph-mon[40250]: pgmap v3988: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:25 smithi122 ceph-mon[32255]: pgmap v3988: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:26.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:26.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:26.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:27 smithi022 ceph-mon[31360]: pgmap v3989: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/573975105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:11:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:27 smithi022 ceph-mon[40250]: pgmap v3989: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/573975105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:27 smithi122 ceph-mon[32255]: pgmap v3989: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/573975105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:29 smithi022 ceph-mon[31360]: pgmap v3990: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:29 smithi022 ceph-mon[40250]: pgmap v3990: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:29 smithi122 ceph-mon[32255]: pgmap v3990: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:30] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:11:30.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:11:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:11:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:31 smithi022 ceph-mon[31360]: pgmap v3991: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:31 smithi022 ceph-mon[40250]: pgmap v3991: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:31.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:31 smithi122 ceph-mon[32255]: pgmap v3991: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:32.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:32.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:32.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:11:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:11:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/788621535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:33 smithi122 ceph-mon[32255]: pgmap v3992: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:11:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:11:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:11:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:11:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/788621535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[31360]: pgmap v3992: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:11:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:11:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:11:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:11:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/788621535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[40250]: pgmap v3992: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:11:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:11:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:11:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:11:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:35 smithi022 ceph-mon[31360]: pgmap v3993: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:35 smithi022 ceph-mon[40250]: pgmap v3993: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:35 smithi122 ceph-mon[32255]: pgmap v3993: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:37.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:37.508 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:11:37.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:37.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:37.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:37 smithi022 ceph-mon[31360]: pgmap v3994: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:37 smithi022 ceph-mon[40250]: pgmap v3994: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:37 smithi122 ceph-mon[32255]: pgmap v3994: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/387261198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/387261198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/387261198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:39 smithi022 ceph-mon[31360]: pgmap v3995: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:39 smithi022 ceph-mon[40250]: pgmap v3995: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:39 smithi122 ceph-mon[32255]: pgmap v3995: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:40] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:11:40.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:11:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:11:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:42.098 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:41 smithi022 ceph-mon[31360]: pgmap v3996: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:42.099 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:42.099 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:41 smithi022 ceph-mon[40250]: pgmap v3996: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:42.100 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:42.100 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:41 smithi122 ceph-mon[32255]: pgmap v3996: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:42.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:11:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:11:42.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:42.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:42.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:43 smithi022 ceph-mon[31360]: pgmap v3997: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3186636222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:43 smithi022 ceph-mon[40250]: pgmap v3997: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3186636222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:43 smithi122 ceph-mon[32255]: pgmap v3997: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3186636222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:45 smithi022 ceph-mon[31360]: pgmap v3998: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:45 smithi022 ceph-mon[40250]: pgmap v3998: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:45 smithi122 ceph-mon[32255]: pgmap v3998: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:47 smithi022 ceph-mon[31360]: pgmap v3999: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:11:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:47 smithi022 ceph-mon[40250]: pgmap v3999: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:47.907 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:47 smithi122 ceph-mon[32255]: pgmap v3999: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:48.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:48.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/327923935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/327923935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/327923935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:49 smithi022 ceph-mon[31360]: pgmap v4000: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:49 smithi022 ceph-mon[40250]: pgmap v4000: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:49 smithi122 ceph-mon[32255]: pgmap v4000: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:11:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:11:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:11:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:51 smithi022 ceph-mon[31360]: pgmap v4001: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:51 smithi022 ceph-mon[40250]: pgmap v4001: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:11:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:51 smithi122 ceph-mon[32255]: pgmap v4001: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:11:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:11:53.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:53 smithi122 ceph-mon[32255]: pgmap v4002: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:53 smithi022 ceph-mon[31360]: pgmap v4002: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:53 smithi022 ceph-mon[40250]: pgmap v4002: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:53.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:53.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:53.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3187862877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3187862877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3187862877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:55 smithi122 ceph-mon[32255]: pgmap v4003: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:55 smithi022 ceph-mon[31360]: pgmap v4003: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:55 smithi022 ceph-mon[40250]: pgmap v4003: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:57 smithi022 ceph-mon[31360]: pgmap v4004: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:11:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:57 smithi022 ceph-mon[40250]: pgmap v4004: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:57 smithi122 ceph-mon[32255]: pgmap v4004: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:11:58.653 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:11:59.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:11:59.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:11:59.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:11:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:59 smithi022 ceph-mon[31360]: pgmap v4005: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:11:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3813753723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:59 smithi022 ceph-mon[40250]: pgmap v4005: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:11:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3813753723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:11:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:59 smithi122 ceph-mon[32255]: pgmap v4005: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:11:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:11:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3813753723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:12:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:12:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:12:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:01 smithi022 ceph-mon[31360]: pgmap v4006: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:01 smithi022 ceph-mon[40250]: pgmap v4006: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:01 smithi122 ceph-mon[32255]: pgmap v4006: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:12:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:12:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:03 smithi122 ceph-mon[32255]: pgmap v4007: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:03.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:03 smithi022 ceph-mon[31360]: pgmap v4007: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:03.469 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:03 smithi022 ceph-mon[40250]: pgmap v4007: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:04.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:12:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:12:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:12:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:12:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:12:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:12:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:12:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:12:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:12:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:12:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:12:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:12:04.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:04.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:04.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3616578036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:05 smithi122 ceph-mon[32255]: pgmap v4008: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3616578036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:05 smithi022 ceph-mon[31360]: pgmap v4008: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3616578036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:05 smithi022 ceph-mon[40250]: pgmap v4008: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:07 smithi022 ceph-mon[31360]: pgmap v4009: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:07 smithi022 ceph-mon[40250]: pgmap v4009: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:07 smithi122 ceph-mon[32255]: pgmap v4009: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:09.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:09.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:09.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:09.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:09.749 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:09 smithi022 ceph-mon[31360]: pgmap v4010: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:09.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:09 smithi022 ceph-mon[40250]: pgmap v4010: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:09 smithi122 ceph-mon[32255]: pgmap v4010: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:10] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:12:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:12:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:12:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1072757284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1072757284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1072757284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:11 smithi022 ceph-mon[31360]: pgmap v4011: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:11 smithi022 ceph-mon[40250]: pgmap v4011: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:11 smithi122 ceph-mon[32255]: pgmap v4011: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:12:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:12:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:13 smithi122 ceph-mon[32255]: pgmap v4012: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:13 smithi022 ceph-mon[31360]: pgmap v4012: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:13 smithi022 ceph-mon[40250]: pgmap v4012: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:14.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:15.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:15.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:15.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:15 smithi022 ceph-mon[31360]: pgmap v4013: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3100446166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:15 smithi022 ceph-mon[40250]: pgmap v4013: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3100446166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:15 smithi122 ceph-mon[32255]: pgmap v4013: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3100446166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:17 smithi022 ceph-mon[31360]: pgmap v4014: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:12:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:17 smithi022 ceph-mon[40250]: pgmap v4014: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:17 smithi122 ceph-mon[32255]: pgmap v4014: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:19 smithi022 ceph-mon[31360]: pgmap v4015: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:19 smithi022 ceph-mon[40250]: pgmap v4015: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:19 smithi122 ceph-mon[32255]: pgmap v4015: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:20.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:20] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:12:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:12:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:12:20.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:20.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:20.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2031070594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2031070594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2031070594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:21 smithi122 ceph-mon[32255]: pgmap v4016: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:21 smithi022 ceph-mon[31360]: pgmap v4016: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:21 smithi022 ceph-mon[40250]: pgmap v4016: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:12:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:12:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:23 smithi122 ceph-mon[32255]: pgmap v4017: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:23 smithi022 ceph-mon[31360]: pgmap v4017: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:23 smithi022 ceph-mon[40250]: pgmap v4017: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:25.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:25 smithi022 ceph-mon[31360]: pgmap v4018: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:25 smithi022 ceph-mon[40250]: pgmap v4018: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:25.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:25.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:25.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:25 smithi122 ceph-mon[32255]: pgmap v4018: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/71120875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/71120875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/71120875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:27 smithi022 ceph-mon[31360]: pgmap v4019: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:12:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:27 smithi022 ceph-mon[40250]: pgmap v4019: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:27 smithi122 ceph-mon[32255]: pgmap v4019: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:29 smithi122 ceph-mon[32255]: pgmap v4020: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:29 smithi022 ceph-mon[31360]: pgmap v4020: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:29 smithi022 ceph-mon[40250]: pgmap v4020: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:30] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:12:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:12:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:12:30.866 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:31.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:31.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:31.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:31 smithi122 ceph-mon[32255]: pgmap v4021: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2846316101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:31 smithi022 ceph-mon[31360]: pgmap v4021: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2846316101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:31 smithi022 ceph-mon[40250]: pgmap v4021: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2846316101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:12:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:12:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:33 smithi122 ceph-mon[32255]: pgmap v4022: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:12:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:12:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:12:33.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:12:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[31360]: pgmap v4022: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:12:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:12:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:12:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:12:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[40250]: pgmap v4022: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:12:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:12:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:12:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:12:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:35 smithi022 ceph-mon[31360]: pgmap v4023: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:35 smithi022 ceph-mon[40250]: pgmap v4023: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:35 smithi122 ceph-mon[32255]: pgmap v4023: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:36.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:36.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:36.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:36.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:12:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:37 smithi022 ceph-mon[31360]: pgmap v4024: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1433278114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:37 smithi022 ceph-mon[40250]: pgmap v4024: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1433278114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:37 smithi122 ceph-mon[32255]: pgmap v4024: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1433278114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:39 smithi122 ceph-mon[32255]: pgmap v4025: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:39 smithi022 ceph-mon[31360]: pgmap v4025: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:39 smithi022 ceph-mon[40250]: pgmap v4025: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:40] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:12:40.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:12:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:12:41.605 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:41 smithi122 ceph-mon[32255]: pgmap v4026: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:41.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:41.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:41.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:41.975 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:41.975 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:41 smithi022 ceph-mon[31360]: pgmap v4026: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:41.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:41.976 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:41 smithi022 ceph-mon[40250]: pgmap v4026: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/190346676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:12:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:12:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/190346676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/190346676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:43 smithi122 ceph-mon[32255]: pgmap v4027: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:43 smithi022 ceph-mon[31360]: pgmap v4027: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:43 smithi022 ceph-mon[40250]: pgmap v4027: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:45 smithi122 ceph-mon[32255]: pgmap v4028: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:45 smithi022 ceph-mon[31360]: pgmap v4028: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:45 smithi022 ceph-mon[40250]: pgmap v4028: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:46.976 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:47.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:47.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:47.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:12:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:47 smithi022 ceph-mon[31360]: pgmap v4029: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1894021449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:47 smithi022 ceph-mon[40250]: pgmap v4029: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1894021449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:47 smithi122 ceph-mon[32255]: pgmap v4029: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1894021449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:49 smithi122 ceph-mon[32255]: pgmap v4030: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:49 smithi022 ceph-mon[31360]: pgmap v4030: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:49 smithi022 ceph-mon[40250]: pgmap v4030: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:12:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:12:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:12:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:51 smithi122 ceph-mon[32255]: pgmap v4031: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:51 smithi022 ceph-mon[31360]: pgmap v4031: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:12:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:51 smithi022 ceph-mon[40250]: pgmap v4031: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:52.342 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:12:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:12:52.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:52.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:52.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:53 smithi122 ceph-mon[32255]: pgmap v4032: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3809649709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:53.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:53 smithi022 ceph-mon[31360]: pgmap v4032: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:53.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3809649709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:53.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:53 smithi022 ceph-mon[40250]: pgmap v4032: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:53.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3809649709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:55 smithi122 ceph-mon[32255]: pgmap v4033: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:55 smithi022 ceph-mon[31360]: pgmap v4033: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:55 smithi022 ceph-mon[40250]: pgmap v4033: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:12:57.710 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:12:57.711 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:12:57.712 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:57 smithi022 ceph-mon[31360]: pgmap v4034: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:57.712 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:57 smithi022 ceph-mon[40250]: pgmap v4034: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:57 smithi122 ceph-mon[32255]: pgmap v4034: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:12:58.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:12:58.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:12:58.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:12:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/921957686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/921957686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/921957686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:12:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:12:59 smithi122 ceph-mon[32255]: pgmap v4035: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:12:59 smithi022 ceph-mon[31360]: pgmap v4035: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:12:59 smithi022 ceph-mon[40250]: pgmap v4035: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:13:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:13:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:13:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:01 smithi122 ceph-mon[32255]: pgmap v4036: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:01 smithi022 ceph-mon[31360]: pgmap v4036: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:01 smithi022 ceph-mon[40250]: pgmap v4036: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:13:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:13:03.082 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:03 smithi122 ceph-mon[32255]: pgmap v4037: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:03.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:03.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:03.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:03.455 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:03 smithi022 ceph-mon[40250]: pgmap v4037: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:03.456 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:03 smithi022 ceph-mon[31360]: pgmap v4037: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2578081439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:13:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:13:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:13:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2578081439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:13:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:13:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:13:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2578081439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:13:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:13:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:13:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:13:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:05 smithi122 ceph-mon[32255]: pgmap v4038: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:13:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:05 smithi022 ceph-mon[31360]: pgmap v4038: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:13:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:05 smithi022 ceph-mon[40250]: pgmap v4038: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:07 smithi022 ceph-mon[31360]: pgmap v4039: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:13:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:07 smithi022 ceph-mon[40250]: pgmap v4039: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:07 smithi122 ceph-mon[32255]: pgmap v4039: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:08.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:08.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:08.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:08.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:09 smithi122 ceph-mon[32255]: pgmap v4040: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1804658680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:09 smithi022 ceph-mon[31360]: pgmap v4040: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1804658680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:09 smithi022 ceph-mon[40250]: pgmap v4040: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1804658680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:10] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:13:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:13:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:13:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:11 smithi122 ceph-mon[32255]: pgmap v4041: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:11 smithi022 ceph-mon[31360]: pgmap v4041: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:11 smithi022 ceph-mon[40250]: pgmap v4041: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:13:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:13:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:13 smithi122 ceph-mon[32255]: pgmap v4042: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:13 smithi022 ceph-mon[31360]: pgmap v4042: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:13 smithi022 ceph-mon[40250]: pgmap v4042: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:14.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:14.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:14.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4256653134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4256653134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4256653134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:15 smithi022 ceph-mon[31360]: pgmap v4043: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:15 smithi022 ceph-mon[40250]: pgmap v4043: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:15 smithi122 ceph-mon[32255]: pgmap v4043: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:13:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:17 smithi022 ceph-mon[31360]: pgmap v4044: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:17 smithi022 ceph-mon[40250]: pgmap v4044: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:17 smithi122 ceph-mon[32255]: pgmap v4044: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:19.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:19.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:19.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:19.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:19 smithi122 ceph-mon[32255]: pgmap v4045: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:19 smithi022 ceph-mon[31360]: pgmap v4045: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:19 smithi022 ceph-mon[40250]: pgmap v4045: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:20] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:13:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:13:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:13:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3170447311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3170447311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3170447311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:21 smithi122 ceph-mon[32255]: pgmap v4046: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:21 smithi022 ceph-mon[31360]: pgmap v4046: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:21 smithi022 ceph-mon[40250]: pgmap v4046: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:13:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:13:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:23 smithi122 ceph-mon[32255]: pgmap v4047: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:23 smithi022 ceph-mon[31360]: pgmap v4047: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:23 smithi022 ceph-mon[40250]: pgmap v4047: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:24.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:24.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:24.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:25 smithi122 ceph-mon[32255]: pgmap v4048: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/195997248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:25 smithi022 ceph-mon[31360]: pgmap v4048: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/195997248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:25 smithi022 ceph-mon[40250]: pgmap v4048: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/195997248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:27 smithi022 ceph-mon[31360]: pgmap v4049: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:13:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:27 smithi022 ceph-mon[40250]: pgmap v4049: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:27 smithi122 ceph-mon[32255]: pgmap v4049: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:29 smithi122 ceph-mon[32255]: pgmap v4050: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:29.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:29.940 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:29 smithi022 ceph-mon[31360]: pgmap v4050: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:29.941 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:29 smithi022 ceph-mon[40250]: pgmap v4050: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:30.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:30.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:30.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:30] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:13:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:13:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:13:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/78867583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/78867583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/78867583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:31 smithi122 ceph-mon[32255]: pgmap v4051: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:31 smithi022 ceph-mon[31360]: pgmap v4051: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:31 smithi022 ceph-mon[40250]: pgmap v4051: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:13:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:13:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:33 smithi122 ceph-mon[32255]: pgmap v4052: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:13:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:13:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:13:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:13:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[31360]: pgmap v4052: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:13:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:13:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:13:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:13:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[40250]: pgmap v4052: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:13:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:13:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:13:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:13:35.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:35.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:35.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:35 smithi022 ceph-mon[40250]: pgmap v4053: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:35.685 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:35 smithi022 ceph-mon[31360]: pgmap v4053: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:35 smithi122 ceph-mon[32255]: pgmap v4053: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744481047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744481047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744481047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:37 smithi022 ceph-mon[31360]: pgmap v4054: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:13:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:37 smithi022 ceph-mon[40250]: pgmap v4054: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:37 smithi122 ceph-mon[32255]: pgmap v4054: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:39 smithi122 ceph-mon[32255]: pgmap v4055: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:39 smithi022 ceph-mon[31360]: pgmap v4055: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:39 smithi022 ceph-mon[40250]: pgmap v4055: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:40] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:13:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:13:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:13:40.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:41.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:41.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:41.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:41 smithi122 ceph-mon[32255]: pgmap v4056: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3642012988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:41 smithi022 ceph-mon[31360]: pgmap v4056: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3642012988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:41 smithi022 ceph-mon[40250]: pgmap v4056: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3642012988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:13:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:13:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:43 smithi122 ceph-mon[32255]: pgmap v4057: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:43 smithi022 ceph-mon[31360]: pgmap v4057: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:43 smithi022 ceph-mon[40250]: pgmap v4057: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:45 smithi122 ceph-mon[32255]: pgmap v4058: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:45 smithi022 ceph-mon[31360]: pgmap v4058: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:45 smithi022 ceph-mon[40250]: pgmap v4058: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:46.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:46.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:46.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:46.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/988997970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/988997970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/988997970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:47 smithi022 ceph-mon[31360]: pgmap v4059: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:13:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:47 smithi022 ceph-mon[40250]: pgmap v4059: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:47 smithi122 ceph-mon[32255]: pgmap v4059: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:49 smithi122 ceph-mon[32255]: pgmap v4060: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:49 smithi022 ceph-mon[31360]: pgmap v4060: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:49 smithi022 ceph-mon[40250]: pgmap v4060: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:13:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:13:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:50] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T20:13:51.424 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:51 smithi022 ceph-mon[31360]: pgmap v4061: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:51 smithi022 ceph-mon[40250]: pgmap v4061: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:51.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:51.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:51.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:13:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:51 smithi122 ceph-mon[32255]: pgmap v4061: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2183938446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:13:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:13:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2183938446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2183938446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:53 smithi122 ceph-mon[32255]: pgmap v4062: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:53 smithi022 ceph-mon[31360]: pgmap v4062: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:53 smithi022 ceph-mon[40250]: pgmap v4062: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:55 smithi122 ceph-mon[32255]: pgmap v4063: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:55 smithi022 ceph-mon[31360]: pgmap v4063: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:55 smithi022 ceph-mon[40250]: pgmap v4063: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:13:56.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:13:57.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:13:57.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:13:57.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:13:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:13:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:57 smithi122 ceph-mon[32255]: pgmap v4064: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3329351359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:57 smithi022 ceph-mon[31360]: pgmap v4064: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3329351359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:57 smithi022 ceph-mon[40250]: pgmap v4064: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:13:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3329351359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:13:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:13:59 smithi122 ceph-mon[32255]: pgmap v4065: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:13:59 smithi022 ceph-mon[31360]: pgmap v4065: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:13:59 smithi022 ceph-mon[40250]: pgmap v4065: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:14:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:14:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:00] "GET /metrics HTTP/1.1" 200 202320 "" "Prometheus/2.18.1" 2021-12-15T20:14:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:01 smithi122 ceph-mon[32255]: pgmap v4066: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:01 smithi022 ceph-mon[31360]: pgmap v4066: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:01 smithi022 ceph-mon[40250]: pgmap v4066: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:02.163 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:02.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:02.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:02.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:14:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:14:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:03 smithi122 ceph-mon[32255]: pgmap v4067: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2121084370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:03 smithi022 ceph-mon[31360]: pgmap v4067: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2121084370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:03 smithi022 ceph-mon[40250]: pgmap v4067: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2121084370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:14:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:14:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:14:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:14:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:14:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:14:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:14:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:14:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:14:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:05 smithi122 ceph-mon[32255]: pgmap v4068: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:14:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:05 smithi022 ceph-mon[31360]: pgmap v4068: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:14:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:05 smithi022 ceph-mon[40250]: pgmap v4068: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:14:07.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:14:07.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:07.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:07.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:07 smithi122 ceph-mon[32255]: pgmap v4069: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:07 smithi022 ceph-mon[31360]: pgmap v4069: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:07 smithi022 ceph-mon[40250]: pgmap v4069: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/971170394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/971170394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/971170394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:09 smithi122 ceph-mon[32255]: pgmap v4070: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:09 smithi022 ceph-mon[31360]: pgmap v4070: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:09 smithi022 ceph-mon[40250]: pgmap v4070: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:10] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:14:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:14:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:14:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:11 smithi122 ceph-mon[32255]: pgmap v4071: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:11 smithi022 ceph-mon[31360]: pgmap v4071: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:11 smithi022 ceph-mon[40250]: pgmap v4071: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:14:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:14:12.906 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:13 smithi022 ceph-mon[31360]: pgmap v4072: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:13 smithi022 ceph-mon[40250]: pgmap v4072: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:13 smithi122 ceph-mon[32255]: pgmap v4072: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/467257297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/467257297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/467257297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:15 smithi122 ceph-mon[32255]: pgmap v4073: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:15 smithi022 ceph-mon[31360]: pgmap v4073: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:15 smithi022 ceph-mon[40250]: pgmap v4073: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:17 smithi022 ceph-mon[31360]: pgmap v4074: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:14:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:17 smithi022 ceph-mon[40250]: pgmap v4074: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:17 smithi122 ceph-mon[32255]: pgmap v4074: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:18.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:18.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:18.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:18.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:19 smithi122 ceph-mon[32255]: pgmap v4075: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1866293314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:19 smithi022 ceph-mon[31360]: pgmap v4075: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1866293314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:19 smithi022 ceph-mon[40250]: pgmap v4075: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1866293314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:20] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:14:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:14:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:14:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:21 smithi122 ceph-mon[32255]: pgmap v4076: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:21 smithi022 ceph-mon[31360]: pgmap v4076: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:21 smithi022 ceph-mon[40250]: pgmap v4076: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:14:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:14:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:23 smithi122 ceph-mon[32255]: pgmap v4077: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:23 smithi022 ceph-mon[31360]: pgmap v4077: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:23 smithi022 ceph-mon[40250]: pgmap v4077: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:23.650 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:24.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:24.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:24.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1150350232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1150350232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1150350232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:25 smithi122 ceph-mon[32255]: pgmap v4078: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:25 smithi022 ceph-mon[31360]: pgmap v4078: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:25 smithi022 ceph-mon[40250]: pgmap v4078: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:27 smithi022 ceph-mon[31360]: pgmap v4079: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:14:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:27 smithi022 ceph-mon[40250]: pgmap v4079: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:27 smithi122 ceph-mon[32255]: pgmap v4079: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:29.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:29.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:29.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:29.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:29 smithi022 ceph-mon[31360]: pgmap v4080: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2717067189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:29 smithi022 ceph-mon[40250]: pgmap v4080: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2717067189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:29 smithi122 ceph-mon[32255]: pgmap v4080: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2717067189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:30] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:14:30.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:14:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:14:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:31 smithi122 ceph-mon[32255]: pgmap v4081: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:31 smithi022 ceph-mon[31360]: pgmap v4081: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:31 smithi022 ceph-mon[40250]: pgmap v4081: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:14:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:14:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:33 smithi122 ceph-mon[32255]: pgmap v4082: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:14:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:14:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:14:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:14:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[31360]: pgmap v4082: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:14:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:14:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:14:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:14:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[40250]: pgmap v4082: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:14:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:14:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:14:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:14:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:34.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:34.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:34.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:35 smithi122 ceph-mon[32255]: pgmap v4083: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2548621446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:35 smithi022 ceph-mon[31360]: pgmap v4083: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2548621446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:35 smithi022 ceph-mon[40250]: pgmap v4083: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2548621446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:37 smithi022 ceph-mon[31360]: pgmap v4084: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:14:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:37 smithi022 ceph-mon[40250]: pgmap v4084: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:37 smithi122 ceph-mon[32255]: pgmap v4084: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:39.765 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:39 smithi122 ceph-mon[32255]: pgmap v4085: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:39 smithi022 ceph-mon[31360]: pgmap v4085: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:39 smithi022 ceph-mon[40250]: pgmap v4085: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:40.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:40.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:40.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:40] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:14:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:14:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:14:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2408502297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2408502297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2408502297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:41 smithi122 ceph-mon[32255]: pgmap v4086: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:41 smithi022 ceph-mon[31360]: pgmap v4086: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:41 smithi022 ceph-mon[40250]: pgmap v4086: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:14:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:14:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:43 smithi122 ceph-mon[32255]: pgmap v4087: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:43 smithi022 ceph-mon[31360]: pgmap v4087: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:43 smithi022 ceph-mon[40250]: pgmap v4087: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:45.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:45.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:45.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:45.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:45 smithi022 ceph-mon[31360]: pgmap v4088: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1525733840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:45 smithi022 ceph-mon[40250]: pgmap v4088: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1525733840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:45 smithi122 ceph-mon[32255]: pgmap v4088: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1525733840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:47 smithi022 ceph-mon[31360]: pgmap v4089: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:14:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:47 smithi022 ceph-mon[40250]: pgmap v4089: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:47 smithi122 ceph-mon[32255]: pgmap v4089: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:49 smithi122 ceph-mon[32255]: pgmap v4090: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:49 smithi022 ceph-mon[31360]: pgmap v4090: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:49 smithi022 ceph-mon[40250]: pgmap v4090: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:50] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:14:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:14:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:14:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:50.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:50.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:50.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:51 smithi122 ceph-mon[32255]: pgmap v4091: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/164829555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:51 smithi022 ceph-mon[31360]: pgmap v4091: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/164829555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:14:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:51 smithi022 ceph-mon[40250]: pgmap v4091: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/164829555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:14:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:14:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:53 smithi122 ceph-mon[32255]: pgmap v4092: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:53 smithi022 ceph-mon[31360]: pgmap v4092: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:53 smithi022 ceph-mon[40250]: pgmap v4092: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:55.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:14:55.878 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:55 smithi022 ceph-mon[31360]: pgmap v4093: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:55.878 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:55 smithi022 ceph-mon[40250]: pgmap v4093: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:55 smithi122 ceph-mon[32255]: pgmap v4093: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:14:56.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:14:56.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:14:56.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:14:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4024351588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4024351588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4024351588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:14:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:57 smithi022 ceph-mon[31360]: pgmap v4094: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:14:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:57 smithi022 ceph-mon[40250]: pgmap v4094: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:57 smithi122 ceph-mon[32255]: pgmap v4094: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:14:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:14:59 smithi122 ceph-mon[32255]: pgmap v4095: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:14:59 smithi022 ceph-mon[31360]: pgmap v4095: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:14:59 smithi022 ceph-mon[40250]: pgmap v4095: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:00] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:15:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:15:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:15:01.247 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:01.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:01.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:01.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:01 smithi122 ceph-mon[32255]: pgmap v4096: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:01 smithi022 ceph-mon[31360]: pgmap v4096: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:01 smithi022 ceph-mon[40250]: pgmap v4096: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:02.580 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:15:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:15:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221638190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221638190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221638190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:03 smithi122 ceph-mon[32255]: pgmap v4097: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:03 smithi022 ceph-mon[31360]: pgmap v4097: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:03 smithi022 ceph-mon[40250]: pgmap v4097: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:05 smithi122 ceph-mon[32255]: pgmap v4098: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:15:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:15:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:15:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:15:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[31360]: pgmap v4098: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:15:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:15:06.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:15:06.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:15:06.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[40250]: pgmap v4098: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:15:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:15:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:15:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:15:06.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:06.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:06.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:06.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:15:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:07 smithi022 ceph-mon[31360]: pgmap v4099: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3714454749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:07 smithi022 ceph-mon[40250]: pgmap v4099: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3714454749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:07 smithi122 ceph-mon[32255]: pgmap v4099: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3714454749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:09 smithi122 ceph-mon[32255]: pgmap v4100: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:09 smithi022 ceph-mon[31360]: pgmap v4100: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:09 smithi022 ceph-mon[40250]: pgmap v4100: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:10] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:15:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:15:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:15:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:11 smithi122 ceph-mon[32255]: pgmap v4101: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:11.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:11.991 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:11.991 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:11 smithi022 ceph-mon[40250]: pgmap v4101: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:11.991 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:11.992 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:11 smithi022 ceph-mon[31360]: pgmap v4101: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:12.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:12.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:12.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:15:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:15:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3667497526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:13 smithi122 ceph-mon[32255]: pgmap v4102: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3667497526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:13 smithi022 ceph-mon[31360]: pgmap v4102: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3667497526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:13.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:13 smithi022 ceph-mon[40250]: pgmap v4102: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:15 smithi122 ceph-mon[32255]: pgmap v4103: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:15 smithi022 ceph-mon[31360]: pgmap v4103: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:15 smithi022 ceph-mon[40250]: pgmap v4103: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:17.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:17.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:17.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:17.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:17.732 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:17 smithi022 ceph-mon[31360]: pgmap v4104: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:17.733 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:15:17.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:17 smithi022 ceph-mon[40250]: pgmap v4104: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:17 smithi122 ceph-mon[32255]: pgmap v4104: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2994710960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2994710960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2994710960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:19 smithi122 ceph-mon[32255]: pgmap v4105: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:19 smithi022 ceph-mon[31360]: pgmap v4105: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:19 smithi022 ceph-mon[40250]: pgmap v4105: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:20] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:15:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:15:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:15:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:21 smithi122 ceph-mon[32255]: pgmap v4106: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:21 smithi022 ceph-mon[40250]: pgmap v4106: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:21 smithi022 ceph-mon[31360]: pgmap v4106: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:15:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:15:22.733 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:23.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:23.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:23.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:23 smithi122 ceph-mon[32255]: pgmap v4107: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2286653158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:23 smithi022 ceph-mon[31360]: pgmap v4107: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2286653158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:23 smithi022 ceph-mon[40250]: pgmap v4107: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2286653158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:25 smithi122 ceph-mon[32255]: pgmap v4108: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:25 smithi022 ceph-mon[31360]: pgmap v4108: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:25 smithi022 ceph-mon[40250]: pgmap v4108: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:27 smithi022 ceph-mon[31360]: pgmap v4109: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:15:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:27 smithi022 ceph-mon[40250]: pgmap v4109: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:27 smithi122 ceph-mon[32255]: pgmap v4109: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2556473742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2556473742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2556473742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:29 smithi122 ceph-mon[32255]: pgmap v4110: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:29 smithi022 ceph-mon[31360]: pgmap v4110: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:29 smithi022 ceph-mon[40250]: pgmap v4110: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:30] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:15:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:15:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:15:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:31 smithi122 ceph-mon[32255]: pgmap v4111: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:31 smithi022 ceph-mon[31360]: pgmap v4111: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:31 smithi022 ceph-mon[40250]: pgmap v4111: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:15:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:15:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:33 smithi122 ceph-mon[32255]: pgmap v4112: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:15:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:15:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:15:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:15:33.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:33.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[31360]: pgmap v4112: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:33.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:15:33.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:15:33.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:15:33.472 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:15:33.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[40250]: pgmap v4112: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:33.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:15:33.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:15:33.473 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:15:33.474 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:15:33.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:33.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:33.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1704150611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1704150611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1704150611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:35 smithi122 ceph-mon[32255]: pgmap v4113: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:35 smithi022 ceph-mon[31360]: pgmap v4113: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:35 smithi022 ceph-mon[40250]: pgmap v4113: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:37 smithi022 ceph-mon[31360]: pgmap v4114: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:15:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:37 smithi022 ceph-mon[40250]: pgmap v4114: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:37 smithi122 ceph-mon[32255]: pgmap v4114: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:38.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:39.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:39.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:39.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:39 smithi122 ceph-mon[32255]: pgmap v4115: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1267471217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:39 smithi022 ceph-mon[31360]: pgmap v4115: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1267471217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:39 smithi022 ceph-mon[40250]: pgmap v4115: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1267471217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:40] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:15:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:15:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:15:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:41 smithi122 ceph-mon[32255]: pgmap v4116: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:41 smithi022 ceph-mon[31360]: pgmap v4116: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:41 smithi022 ceph-mon[40250]: pgmap v4116: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:15:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:15:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:43 smithi122 ceph-mon[32255]: pgmap v4117: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:43 smithi022 ceph-mon[31360]: pgmap v4117: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:43 smithi022 ceph-mon[40250]: pgmap v4117: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:44.205 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:44.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:44.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:44.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3226322569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3226322569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3226322569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:45 smithi122 ceph-mon[32255]: pgmap v4118: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:45 smithi022 ceph-mon[31360]: pgmap v4118: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:45 smithi022 ceph-mon[40250]: pgmap v4118: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:47 smithi022 ceph-mon[31360]: pgmap v4119: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:15:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:47 smithi022 ceph-mon[40250]: pgmap v4119: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:47 smithi122 ceph-mon[32255]: pgmap v4119: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:49.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:49 smithi122 ceph-mon[32255]: pgmap v4120: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:49.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:49.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:49.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:49.946 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:49 smithi022 ceph-mon[31360]: pgmap v4120: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:49.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:49 smithi022 ceph-mon[40250]: pgmap v4120: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:50] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:15:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:15:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:15:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1916424413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1916424413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1916424413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:51 smithi122 ceph-mon[32255]: pgmap v4121: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:51 smithi022 ceph-mon[31360]: pgmap v4121: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:15:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:51 smithi022 ceph-mon[40250]: pgmap v4121: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:15:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:15:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:53 smithi122 ceph-mon[32255]: pgmap v4122: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:53 smithi022 ceph-mon[31360]: pgmap v4122: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:53 smithi022 ceph-mon[40250]: pgmap v4122: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:15:55.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:15:55.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:15:55.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:55 smithi122 ceph-mon[32255]: pgmap v4123: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/231256742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:55 smithi022 ceph-mon[31360]: pgmap v4123: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/231256742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:55 smithi022 ceph-mon[40250]: pgmap v4123: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:15:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/231256742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:15:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:57 smithi022 ceph-mon[31360]: pgmap v4124: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:15:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:57 smithi022 ceph-mon[40250]: pgmap v4124: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:57 smithi122 ceph-mon[32255]: pgmap v4124: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:15:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:15:59 smithi122 ceph-mon[32255]: pgmap v4125: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:15:59 smithi022 ceph-mon[31360]: pgmap v4125: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:15:59 smithi022 ceph-mon[40250]: pgmap v4125: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:00.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:00] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:16:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:16:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:16:00.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:00.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:00.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:01 smithi122 ceph-mon[32255]: pgmap v4126: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/222498618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:01 smithi022 ceph-mon[31360]: pgmap v4126: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/222498618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:01 smithi022 ceph-mon[40250]: pgmap v4126: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/222498618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:16:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:16:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:03 smithi122 ceph-mon[32255]: pgmap v4127: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:03 smithi022 ceph-mon[31360]: pgmap v4127: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:03 smithi022 ceph-mon[40250]: pgmap v4127: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:05.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:05 smithi122 ceph-mon[32255]: pgmap v4128: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:16:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:16:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:16:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[31360]: pgmap v4128: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:16:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:16:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:16:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[40250]: pgmap v4128: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:16:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:16:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:16:06.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:06.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:06.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:16:07.646 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3999813810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:07.647 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:16:07.647 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3999813810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:07.648 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:16:07.648 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3999813810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:07.722 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:16:08.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:07 smithi022 ceph-mon[31360]: pgmap v4129: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:08.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:07 smithi022 ceph-mon[40250]: pgmap v4129: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:07 smithi122 ceph-mon[32255]: pgmap v4129: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:09 smithi022 ceph-mon[31360]: pgmap v4130: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:09 smithi022 ceph-mon[40250]: pgmap v4130: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:09 smithi122 ceph-mon[32255]: pgmap v4130: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:16:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:16:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:16:11.047 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:11.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:11 smithi022 ceph-mon[31360]: pgmap v4131: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2121591985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:11 smithi022 ceph-mon[40250]: pgmap v4131: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2121591985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:11 smithi122 ceph-mon[32255]: pgmap v4131: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2121591985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:16:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:16:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:13 smithi122 ceph-mon[32255]: pgmap v4132: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:13 smithi022 ceph-mon[31360]: pgmap v4132: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:13 smithi022 ceph-mon[40250]: pgmap v4132: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:15 smithi122 ceph-mon[32255]: pgmap v4133: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:15 smithi022 ceph-mon[31360]: pgmap v4133: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:15 smithi022 ceph-mon[40250]: pgmap v4133: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:16.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:16.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:16.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:16.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:16:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:17 smithi022 ceph-mon[31360]: pgmap v4134: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2290057041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:17 smithi022 ceph-mon[40250]: pgmap v4134: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2290057041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:17 smithi122 ceph-mon[32255]: pgmap v4134: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2290057041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:19 smithi122 ceph-mon[32255]: pgmap v4135: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:19 smithi022 ceph-mon[31360]: pgmap v4135: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:19 smithi022 ceph-mon[40250]: pgmap v4135: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:20] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:16:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:16:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:16:21.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:21 smithi122 ceph-mon[32255]: pgmap v4136: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:21 smithi022 ceph-mon[31360]: pgmap v4136: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:21 smithi022 ceph-mon[40250]: pgmap v4136: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:22.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:22.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:22.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:22.601 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:16:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:16:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/605294356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/605294356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/605294356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:23 smithi122 ceph-mon[32255]: pgmap v4137: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:23 smithi022 ceph-mon[31360]: pgmap v4137: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:23 smithi022 ceph-mon[40250]: pgmap v4137: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:25 smithi122 ceph-mon[32255]: pgmap v4138: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:25 smithi022 ceph-mon[31360]: pgmap v4138: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:25 smithi022 ceph-mon[40250]: pgmap v4138: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:27.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:16:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:27.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:27.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:27 smithi122 ceph-mon[32255]: pgmap v4139: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:28.335 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2434644510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:28.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:27 smithi022 ceph-mon[31360]: pgmap v4139: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:28.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2434644510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:28.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:27 smithi022 ceph-mon[40250]: pgmap v4139: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:28.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2434644510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:29 smithi122 ceph-mon[32255]: pgmap v4140: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:29 smithi022 ceph-mon[31360]: pgmap v4140: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:29 smithi022 ceph-mon[40250]: pgmap v4140: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:30] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:16:30.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:16:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:16:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:31 smithi122 ceph-mon[32255]: pgmap v4141: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:31 smithi022 ceph-mon[31360]: pgmap v4141: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:31 smithi022 ceph-mon[40250]: pgmap v4141: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:32.536 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:16:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:16:32.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:32.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:32.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:33.159 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[40250]: pgmap v4142: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:33.159 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:16:33.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:16:33.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:16:33.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:16:33.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/543423493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:33 smithi122 ceph-mon[32255]: pgmap v4142: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:16:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:16:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:16:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:16:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/543423493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[31360]: pgmap v4142: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:16:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:16:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:16:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:16:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/543423493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:35 smithi122 ceph-mon[32255]: pgmap v4143: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:35 smithi022 ceph-mon[31360]: pgmap v4143: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:35 smithi022 ceph-mon[40250]: pgmap v4143: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:37 smithi022 ceph-mon[31360]: pgmap v4144: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:16:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:37 smithi022 ceph-mon[40250]: pgmap v4144: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:37.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:37 smithi122 ceph-mon[32255]: pgmap v4144: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:38.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:38.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:38.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4102502014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4102502014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4102502014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:39 smithi122 ceph-mon[32255]: pgmap v4145: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:39 smithi022 ceph-mon[31360]: pgmap v4145: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:39 smithi022 ceph-mon[40250]: pgmap v4145: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:40] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:16:40.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:16:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:16:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:41 smithi122 ceph-mon[32255]: pgmap v4146: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:41 smithi022 ceph-mon[31360]: pgmap v4146: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:41 smithi022 ceph-mon[40250]: pgmap v4146: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:16:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:16:43.281 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:43 smithi122 ceph-mon[32255]: pgmap v4147: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:43 smithi022 ceph-mon[31360]: pgmap v4147: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:43 smithi022 ceph-mon[40250]: pgmap v4147: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:43.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:43.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:43.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1276301273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1276301273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1276301273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:45 smithi122 ceph-mon[32255]: pgmap v4148: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:45 smithi022 ceph-mon[31360]: pgmap v4148: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:45 smithi022 ceph-mon[40250]: pgmap v4148: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:47 smithi022 ceph-mon[31360]: pgmap v4149: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:16:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:47 smithi022 ceph-mon[40250]: pgmap v4149: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:47 smithi122 ceph-mon[32255]: pgmap v4149: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:48.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:49 smithi122 ceph-mon[32255]: pgmap v4150: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2062302430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:49 smithi022 ceph-mon[31360]: pgmap v4150: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2062302430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:49 smithi022 ceph-mon[40250]: pgmap v4150: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2062302430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:16:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:16:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:50] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:16:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:51 smithi122 ceph-mon[32255]: pgmap v4151: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:51 smithi022 ceph-mon[31360]: pgmap v4151: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:16:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:51 smithi022 ceph-mon[40250]: pgmap v4151: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:16:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:16:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:53 smithi122 ceph-mon[32255]: pgmap v4152: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:53 smithi022 ceph-mon[31360]: pgmap v4152: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:53 smithi022 ceph-mon[40250]: pgmap v4152: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:54.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:54.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:54.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:54.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1375748807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1375748807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1375748807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:16:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:55 smithi122 ceph-mon[32255]: pgmap v4153: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:55 smithi022 ceph-mon[31360]: pgmap v4153: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:55 smithi022 ceph-mon[40250]: pgmap v4153: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:57 smithi022 ceph-mon[31360]: pgmap v4154: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:16:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:57 smithi022 ceph-mon[40250]: pgmap v4154: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:57 smithi122 ceph-mon[32255]: pgmap v4154: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:16:59.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:16:59.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:16:59.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:16:59.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:16:59.758 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:16:59 smithi022 ceph-mon[31360]: pgmap v4155: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:59.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:16:59 smithi022 ceph-mon[40250]: pgmap v4155: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:16:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:16:59 smithi122 ceph-mon[32255]: pgmap v4155: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:00] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:17:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:17:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:17:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3659089275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3659089275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3659089275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:01 smithi122 ceph-mon[32255]: pgmap v4156: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:01 smithi022 ceph-mon[31360]: pgmap v4156: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:01 smithi022 ceph-mon[40250]: pgmap v4156: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:17:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:17:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:03 smithi122 ceph-mon[32255]: pgmap v4157: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:03 smithi022 ceph-mon[31360]: pgmap v4157: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:03 smithi022 ceph-mon[40250]: pgmap v4157: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:04.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:05.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:05.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:05.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:05 smithi122 ceph-mon[32255]: pgmap v4158: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1223636721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:05 smithi022 ceph-mon[31360]: pgmap v4158: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1223636721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:05 smithi022 ceph-mon[40250]: pgmap v4158: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1223636721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:17:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:17:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:17:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:06.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:17:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:17:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:17:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:07.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:17:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:17:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:17:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:17:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:07 smithi022 ceph-mon[31360]: pgmap v4159: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:17:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:07 smithi022 ceph-mon[40250]: pgmap v4159: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:07 smithi122 ceph-mon[32255]: pgmap v4159: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:09 smithi122 ceph-mon[32255]: pgmap v4160: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:09 smithi022 ceph-mon[31360]: pgmap v4160: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:09 smithi022 ceph-mon[40250]: pgmap v4160: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:10.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:10.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:10] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:17:10.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:10.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:10.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1770940848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:10.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:17:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:17:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1770940848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1770940848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:11 smithi122 ceph-mon[32255]: pgmap v4161: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:11 smithi022 ceph-mon[31360]: pgmap v4161: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:11 smithi022 ceph-mon[40250]: pgmap v4161: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:17:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:17:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:13 smithi122 ceph-mon[32255]: pgmap v4162: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:13 smithi022 ceph-mon[31360]: pgmap v4162: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:13 smithi022 ceph-mon[40250]: pgmap v4162: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:15.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:15 smithi022 ceph-mon[31360]: pgmap v4163: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:15 smithi022 ceph-mon[40250]: pgmap v4163: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:15 smithi122 ceph-mon[32255]: pgmap v4163: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/515695443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/515695443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/515695443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:17 smithi022 ceph-mon[31360]: pgmap v4164: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:17:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:17 smithi022 ceph-mon[40250]: pgmap v4164: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:17 smithi122 ceph-mon[32255]: pgmap v4164: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:19 smithi122 ceph-mon[32255]: pgmap v4165: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:19 smithi022 ceph-mon[31360]: pgmap v4165: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:19 smithi022 ceph-mon[40250]: pgmap v4165: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:17:20.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:17:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:17:20.868 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:21.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:21.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:21.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:21 smithi122 ceph-mon[32255]: pgmap v4166: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3999091404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:21 smithi022 ceph-mon[31360]: pgmap v4166: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3999091404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:22.121 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:22.121 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:21 smithi022 ceph-mon[40250]: pgmap v4166: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:22.122 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3999091404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:17:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:17:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:23 smithi122 ceph-mon[32255]: pgmap v4167: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:23 smithi022 ceph-mon[31360]: pgmap v4167: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:23 smithi022 ceph-mon[40250]: pgmap v4167: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:25 smithi122 ceph-mon[32255]: pgmap v4168: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:25 smithi022 ceph-mon[31360]: pgmap v4168: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:25 smithi022 ceph-mon[40250]: pgmap v4168: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:26.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:26.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:26.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:26.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3542190108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3542190108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3542190108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:27 smithi022 ceph-mon[31360]: pgmap v4169: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:17:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:27 smithi022 ceph-mon[40250]: pgmap v4169: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:27 smithi122 ceph-mon[32255]: pgmap v4169: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:29 smithi122 ceph-mon[32255]: pgmap v4170: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:29 smithi022 ceph-mon[31360]: pgmap v4170: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:29 smithi022 ceph-mon[40250]: pgmap v4170: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:17:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:17:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:17:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:31 smithi122 ceph-mon[32255]: pgmap v4171: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:31.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:31.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:31.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:31.990 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:31.990 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:31 smithi022 ceph-mon[31360]: pgmap v4171: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:31.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:31.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:31 smithi022 ceph-mon[40250]: pgmap v4171: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:32.626 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:17:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:17:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2300234547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2300234547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2300234547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:33 smithi122 ceph-mon[32255]: pgmap v4172: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:17:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[31360]: pgmap v4172: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:17:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:17:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:17:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:17:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[40250]: pgmap v4172: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:17:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:17:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:17:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:17:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:35 smithi122 ceph-mon[32255]: pgmap v4173: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:35 smithi022 ceph-mon[31360]: pgmap v4173: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:35 smithi022 ceph-mon[40250]: pgmap v4173: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:37.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:37.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:37.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:37.633 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:17:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:37 smithi122 ceph-mon[32255]: pgmap v4174: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4224530427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:37 smithi022 ceph-mon[31360]: pgmap v4174: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4224530427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:37 smithi022 ceph-mon[40250]: pgmap v4174: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:38.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4224530427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:39 smithi122 ceph-mon[32255]: pgmap v4175: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:39 smithi022 ceph-mon[31360]: pgmap v4175: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:39 smithi022 ceph-mon[40250]: pgmap v4175: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:40] "GET /metrics HTTP/1.1" 200 202324 "" "Prometheus/2.18.1" 2021-12-15T20:17:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:17:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:17:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:41 smithi122 ceph-mon[32255]: pgmap v4176: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:41 smithi022 ceph-mon[31360]: pgmap v4176: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:41 smithi022 ceph-mon[40250]: pgmap v4176: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:42.360 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:17:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:17:42.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:42.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:42.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:43 smithi122 ceph-mon[32255]: pgmap v4177: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2457321376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:43 smithi022 ceph-mon[31360]: pgmap v4177: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2457321376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:43 smithi022 ceph-mon[40250]: pgmap v4177: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2457321376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:45 smithi122 ceph-mon[32255]: pgmap v4178: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:45 smithi022 ceph-mon[31360]: pgmap v4178: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:45 smithi022 ceph-mon[40250]: pgmap v4178: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:47.728 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:47.729 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:17:47.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:47 smithi022 ceph-mon[31360]: pgmap v4179: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:47.730 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:47 smithi022 ceph-mon[40250]: pgmap v4179: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:47 smithi122 ceph-mon[32255]: pgmap v4179: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:48.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:48.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:48.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1974127028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1974127028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1974127028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:49 smithi122 ceph-mon[32255]: pgmap v4180: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:49 smithi022 ceph-mon[31360]: pgmap v4180: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:49 smithi022 ceph-mon[40250]: pgmap v4180: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:17:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:17:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:17:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:17:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:51 smithi122 ceph-mon[32255]: pgmap v4181: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:51 smithi022 ceph-mon[31360]: pgmap v4181: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:17:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:51 smithi022 ceph-mon[40250]: pgmap v4181: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:17:52.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:17:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:53 smithi122 ceph-mon[32255]: pgmap v4182: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:53.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:53.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:53.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:53.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:53 smithi022 ceph-mon[31360]: pgmap v4182: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:53.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:53 smithi022 ceph-mon[40250]: pgmap v4182: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:17:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1729594490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1729594490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1729594490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:17:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:55 smithi122 ceph-mon[32255]: pgmap v4183: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T20:17:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:55 smithi022 ceph-mon[31360]: pgmap v4183: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T20:17:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:55 smithi022 ceph-mon[40250]: pgmap v4183: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T20:17:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:57 smithi022 ceph-mon[31360]: pgmap v4184: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T20:17:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:17:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:57 smithi022 ceph-mon[40250]: pgmap v4184: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T20:17:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:57 smithi122 ceph-mon[32255]: pgmap v4184: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T20:17:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:17:58.838 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:17:58.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:17:58.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:17:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:59 smithi122 ceph-mon[32255]: pgmap v4185: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T20:17:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:17:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2754090425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:59 smithi022 ceph-mon[31360]: pgmap v4185: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T20:18:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:17:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2754090425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:59 smithi022 ceph-mon[40250]: pgmap v4185: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T20:18:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:17:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2754090425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:18:00.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:18:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:18:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:01 smithi122 ceph-mon[32255]: pgmap v4186: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T20:18:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:01 smithi022 ceph-mon[31360]: pgmap v4186: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T20:18:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:01 smithi022 ceph-mon[40250]: pgmap v4186: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T20:18:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:18:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:18:03.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:03 smithi122 ceph-mon[32255]: pgmap v4187: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T20:18:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:03 smithi022 ceph-mon[31360]: pgmap v4187: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T20:18:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:03 smithi022 ceph-mon[40250]: pgmap v4187: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2021-12-15T20:18:03.841 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:04.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:04.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:04.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1827918665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1827918665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1827918665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:05 smithi022 ceph-mon[31360]: pgmap v4188: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T20:18:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:05 smithi022 ceph-mon[40250]: pgmap v4188: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T20:18:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:05 smithi122 ceph-mon[32255]: pgmap v4188: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T20:18:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:18:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:18:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:18:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:18:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:18:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:18:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:18:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:18:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:18:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:18:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:07 smithi022 ceph-mon[31360]: pgmap v4189: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:18:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:18:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:07 smithi022 ceph-mon[40250]: pgmap v4189: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:18:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:18:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:07 smithi122 ceph-mon[32255]: pgmap v4189: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:18:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:18:09.210 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:09.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:09.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:09.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:09 smithi122 ceph-mon[32255]: pgmap v4190: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:18:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/668266353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:09 smithi022 ceph-mon[31360]: pgmap v4190: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:18:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/668266353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:09 smithi022 ceph-mon[40250]: pgmap v4190: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:18:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/668266353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:10] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:18:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:18:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:18:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:11 smithi122 ceph-mon[32255]: pgmap v4191: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:11 smithi022 ceph-mon[31360]: pgmap v4191: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:11 smithi022 ceph-mon[40250]: pgmap v4191: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:18:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:18:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:13 smithi122 ceph-mon[32255]: pgmap v4192: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:13 smithi022 ceph-mon[31360]: pgmap v4192: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:13 smithi022 ceph-mon[40250]: pgmap v4192: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:14.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:14.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:14.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:14.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:15 smithi122 ceph-mon[32255]: pgmap v4193: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2343925101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:15 smithi022 ceph-mon[31360]: pgmap v4193: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2343925101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:15 smithi022 ceph-mon[40250]: pgmap v4193: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2343925101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:17 smithi022 ceph-mon[31360]: pgmap v4194: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:18:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:17 smithi022 ceph-mon[40250]: pgmap v4194: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:17 smithi122 ceph-mon[32255]: pgmap v4194: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:19 smithi122 ceph-mon[32255]: pgmap v4195: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:19.953 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:19.953 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:19 smithi022 ceph-mon[31360]: pgmap v4195: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:19.954 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:19 smithi022 ceph-mon[40250]: pgmap v4195: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:20] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:18:20.638 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2207470052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2207470052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:20.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:18:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:18:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2207470052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:21 smithi122 ceph-mon[32255]: pgmap v4196: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:21 smithi022 ceph-mon[31360]: pgmap v4196: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:21 smithi022 ceph-mon[40250]: pgmap v4196: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:18:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:18:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:23 smithi122 ceph-mon[32255]: pgmap v4197: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:23 smithi022 ceph-mon[31360]: pgmap v4197: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:23 smithi022 ceph-mon[40250]: pgmap v4197: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:25.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:25.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:25.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:25.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:25 smithi122 ceph-mon[32255]: pgmap v4198: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:25 smithi022 ceph-mon[31360]: pgmap v4198: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:25 smithi022 ceph-mon[40250]: pgmap v4198: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2663782857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2663782857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2663782857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:27 smithi022 ceph-mon[31360]: pgmap v4199: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:18:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:27 smithi022 ceph-mon[40250]: pgmap v4199: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:27 smithi122 ceph-mon[32255]: pgmap v4199: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:29 smithi122 ceph-mon[32255]: pgmap v4200: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:29 smithi022 ceph-mon[31360]: pgmap v4200: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:29 smithi022 ceph-mon[40250]: pgmap v4200: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:30] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:18:30.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:18:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:18:31.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:31.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:31.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:31 smithi122 ceph-mon[32255]: pgmap v4201: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3540989219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:31 smithi022 ceph-mon[31360]: pgmap v4201: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3540989219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:31 smithi022 ceph-mon[40250]: pgmap v4201: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3540989219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:18:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:18:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[31360]: pgmap v4202: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:18:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:18:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:18:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:18:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[40250]: pgmap v4202: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:18:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:18:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:18:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:18:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:33 smithi122 ceph-mon[32255]: pgmap v4202: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:18:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:18:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:18:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:18:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:35 smithi122 ceph-mon[32255]: pgmap v4203: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:35 smithi022 ceph-mon[31360]: pgmap v4203: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:35 smithi022 ceph-mon[40250]: pgmap v4203: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:36.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1724969046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1724969046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1724969046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:37 smithi022 ceph-mon[31360]: pgmap v4204: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:18:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:37 smithi022 ceph-mon[40250]: pgmap v4204: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:37 smithi122 ceph-mon[32255]: pgmap v4204: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:39 smithi122 ceph-mon[32255]: pgmap v4205: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:39 smithi022 ceph-mon[31360]: pgmap v4205: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:39 smithi022 ceph-mon[40250]: pgmap v4205: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:40] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:18:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:18:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:18:41.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:41 smithi022 ceph-mon[31360]: pgmap v4206: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:41 smithi022 ceph-mon[40250]: pgmap v4206: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:41.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:41.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:41.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:41 smithi122 ceph-mon[32255]: pgmap v4206: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:42.646 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:18:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:18:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/437249200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/437249200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/437249200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:43 smithi122 ceph-mon[32255]: pgmap v4207: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:43 smithi022 ceph-mon[31360]: pgmap v4207: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:43 smithi022 ceph-mon[40250]: pgmap v4207: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:45 smithi122 ceph-mon[32255]: pgmap v4208: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:45 smithi022 ceph-mon[31360]: pgmap v4208: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:45 smithi022 ceph-mon[40250]: pgmap v4208: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:46.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:47.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:47.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:47.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:18:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:47 smithi122 ceph-mon[32255]: pgmap v4209: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2319286097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:47 smithi022 ceph-mon[31360]: pgmap v4209: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2319286097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:47 smithi022 ceph-mon[40250]: pgmap v4209: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2319286097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:49 smithi122 ceph-mon[32255]: pgmap v4210: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:49 smithi022 ceph-mon[31360]: pgmap v4210: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:49 smithi022 ceph-mon[40250]: pgmap v4210: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:50] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:18:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:18:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:18:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:51 smithi122 ceph-mon[32255]: pgmap v4211: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:51 smithi022 ceph-mon[31360]: pgmap v4211: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:18:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:51 smithi022 ceph-mon[40250]: pgmap v4211: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:52.171 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:52.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:52.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:52.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:18:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:18:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/437932995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:53 smithi022 ceph-mon[31360]: pgmap v4212: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/437932995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:53 smithi022 ceph-mon[40250]: pgmap v4212: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/437932995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:53 smithi122 ceph-mon[32255]: pgmap v4212: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:55 smithi122 ceph-mon[32255]: pgmap v4213: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:55 smithi022 ceph-mon[31360]: pgmap v4213: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:55 smithi022 ceph-mon[40250]: pgmap v4213: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:18:57.541 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:18:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:57 smithi022 ceph-mon[31360]: pgmap v4214: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:18:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:57 smithi022 ceph-mon[40250]: pgmap v4214: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:57.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:18:57.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:18:57.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:18:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:57 smithi122 ceph-mon[32255]: pgmap v4214: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:18:57.936 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:57 smithi122 conmon[74079]: level=info ts=2021-12-15T20:18:57.458Z caller=compact.go:495 component=tsdb msg="write block" mint=1639588733270 maxt=1639591200000 ulid=01FPZVYZ7JPFWBETAPBFAEJVR9 duration=64.510275ms 2021-12-15T20:18:57.937 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:18:57 smithi122 conmon[74079]: level=info ts=2021-12-15T20:18:57.471Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=5.515864ms 2021-12-15T20:18:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4261835563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4261835563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4261835563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:18:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:18:59 smithi122 ceph-mon[32255]: pgmap v4215: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:18:59 smithi022 ceph-mon[31360]: pgmap v4215: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:18:59 smithi022 ceph-mon[40250]: pgmap v4215: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:00] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:19:00.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:19:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:19:01.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:01.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:01 smithi122 ceph-mon[32255]: pgmap v4216: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:01 smithi022 ceph-mon[31360]: pgmap v4216: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:01 smithi022 ceph-mon[40250]: pgmap v4216: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:19:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:19:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:03.192 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:03 smithi022 ceph-mon[40250]: pgmap v4217: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:03.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:03.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:03.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:03 smithi022 ceph-mon[31360]: pgmap v4217: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:03 smithi122 ceph-mon[32255]: pgmap v4217: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3466970821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3466970821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3466970821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:05.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:05 smithi022 ceph-mon[31360]: pgmap v4218: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:05 smithi022 ceph-mon[40250]: pgmap v4218: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:05 smithi122 ceph-mon[32255]: pgmap v4218: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:19:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[40250]: pgmap v4219: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:19:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:19:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:19:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:19:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[31360]: pgmap v4219: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:19:07.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:19:07.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:19:07.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:19:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:07 smithi122 ceph-mon[32255]: pgmap v4219: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:19:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:19:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:19:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:19:08.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:08.647 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3475768570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:08.647 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3475768570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:08.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:08.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:08.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3475768570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:09 smithi122 ceph-mon[32255]: pgmap v4220: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:09 smithi022 ceph-mon[31360]: pgmap v4220: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:09 smithi022 ceph-mon[40250]: pgmap v4220: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:19:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:19:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:19:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:11 smithi122 ceph-mon[32255]: pgmap v4221: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:11 smithi022 ceph-mon[31360]: pgmap v4221: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:11 smithi022 ceph-mon[40250]: pgmap v4221: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:19:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:19:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:13 smithi022 ceph-mon[31360]: pgmap v4222: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:13 smithi022 ceph-mon[40250]: pgmap v4222: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:13.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:13 smithi122 ceph-mon[32255]: pgmap v4222: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:14.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:14.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:14.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3775117081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3775117081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3775117081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:15 smithi022 ceph-mon[31360]: pgmap v4223: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:15 smithi022 ceph-mon[40250]: pgmap v4223: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:15 smithi122 ceph-mon[32255]: pgmap v4223: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:17.646 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:19:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:17 smithi122 ceph-mon[32255]: pgmap v4224: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:17 smithi022 ceph-mon[31360]: pgmap v4224: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:17 smithi022 ceph-mon[40250]: pgmap v4224: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:19.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:19.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:19.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:19.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:19 smithi122 ceph-mon[32255]: pgmap v4225: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2261557987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:19 smithi022 ceph-mon[31360]: pgmap v4225: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2261557987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:19 smithi022 ceph-mon[40250]: pgmap v4225: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2261557987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:19:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:19:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:19:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:21 smithi122 ceph-mon[32255]: pgmap v4226: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:21 smithi022 ceph-mon[31360]: pgmap v4226: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:21 smithi022 ceph-mon[40250]: pgmap v4226: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:19:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:19:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:23 smithi022 ceph-mon[31360]: pgmap v4227: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:23 smithi022 ceph-mon[40250]: pgmap v4227: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:23 smithi122 ceph-mon[32255]: pgmap v4227: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:24.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:24.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:24.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:24.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:25 smithi122 ceph-mon[32255]: pgmap v4228: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3036271692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:25 smithi022 ceph-mon[31360]: pgmap v4228: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3036271692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:25 smithi022 ceph-mon[40250]: pgmap v4228: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3036271692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:27.651 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:19:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:27 smithi122 ceph-mon[32255]: pgmap v4229: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:27 smithi022 ceph-mon[31360]: pgmap v4229: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:27 smithi022 ceph-mon[40250]: pgmap v4229: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:29.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:29 smithi122 ceph-mon[32255]: pgmap v4230: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:29 smithi022 ceph-mon[31360]: pgmap v4230: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:29 smithi022 ceph-mon[40250]: pgmap v4230: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:30.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:30.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:30.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:19:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3907718452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:30.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:19:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:19:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3907718452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3907718452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:31 smithi122 ceph-mon[32255]: pgmap v4231: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:31 smithi022 ceph-mon[31360]: pgmap v4231: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:31 smithi022 ceph-mon[40250]: pgmap v4231: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:19:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:19:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[31360]: pgmap v4232: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:19:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:19:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:19:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:19:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[40250]: pgmap v4232: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:19:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:19:33.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:19:33.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:19:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:33 smithi122 ceph-mon[32255]: pgmap v4232: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:19:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:19:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:19:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:19:35.139 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:35.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:35.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:35.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:35 smithi022 ceph-mon[31360]: pgmap v4233: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2504189150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:35 smithi022 ceph-mon[40250]: pgmap v4233: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2504189150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:35 smithi122 ceph-mon[32255]: pgmap v4233: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2504189150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:37.654 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:19:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:37 smithi122 ceph-mon[32255]: pgmap v4234: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:37 smithi022 ceph-mon[31360]: pgmap v4234: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:37 smithi022 ceph-mon[40250]: pgmap v4234: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:39 smithi122 ceph-mon[32255]: pgmap v4235: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:39 smithi022 ceph-mon[31360]: pgmap v4235: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:39 smithi022 ceph-mon[40250]: pgmap v4235: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:40] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:19:40.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:40.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:40.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:40.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:19:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:19:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:41 smithi122 ceph-mon[32255]: pgmap v4236: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4281015460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:41 smithi022 ceph-mon[31360]: pgmap v4236: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4281015460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:41 smithi022 ceph-mon[40250]: pgmap v4236: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4281015460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:19:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:19:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:43 smithi022 ceph-mon[31360]: pgmap v4237: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:43 smithi022 ceph-mon[40250]: pgmap v4237: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:43 smithi122 ceph-mon[32255]: pgmap v4237: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:45.880 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:45 smithi122 ceph-mon[32255]: pgmap v4238: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:45 smithi022 ceph-mon[31360]: pgmap v4238: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:45 smithi022 ceph-mon[40250]: pgmap v4238: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:46.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:46.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:46.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3139761106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3139761106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3139761106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:47.661 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:19:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:47 smithi122 ceph-mon[32255]: pgmap v4239: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:47 smithi022 ceph-mon[31360]: pgmap v4239: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:47 smithi022 ceph-mon[40250]: pgmap v4239: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:49 smithi122 ceph-mon[32255]: pgmap v4240: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:49 smithi022 ceph-mon[31360]: pgmap v4240: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:49 smithi022 ceph-mon[40250]: pgmap v4240: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:19:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:19:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:19:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:51.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:51.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:51.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:51 smithi122 ceph-mon[32255]: pgmap v4241: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2573780739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:51 smithi022 ceph-mon[31360]: pgmap v4241: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2573780739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:19:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:51 smithi022 ceph-mon[40250]: pgmap v4241: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2573780739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:19:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:19:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:53 smithi022 ceph-mon[31360]: pgmap v4242: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:53 smithi022 ceph-mon[40250]: pgmap v4242: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:53 smithi122 ceph-mon[32255]: pgmap v4242: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:55 smithi122 ceph-mon[32255]: pgmap v4243: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:55 smithi022 ceph-mon[31360]: pgmap v4243: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:55 smithi022 ceph-mon[40250]: pgmap v4243: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:19:56.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:19:56.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:19:56.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:19:56.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:19:57.661 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:19:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:57 smithi122 ceph-mon[32255]: pgmap v4244: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2282492420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:57 smithi022 ceph-mon[31360]: pgmap v4244: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2282492420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:57 smithi022 ceph-mon[40250]: pgmap v4244: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:19:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2282492420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:19:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:19:59 smithi122 ceph-mon[32255]: pgmap v4245: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:19:59 smithi022 ceph-mon[31360]: pgmap v4245: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:19:59 smithi022 ceph-mon[40250]: pgmap v4245: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:20:00.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:20:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:20:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T20:20:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T20:20:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T20:20:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:01 smithi122 ceph-mon[32255]: pgmap v4246: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:01.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:01.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:01.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:01 smithi022 ceph-mon[31360]: pgmap v4246: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:01.989 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:01.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:01 smithi022 ceph-mon[40250]: pgmap v4246: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:02.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:02.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:02.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:20:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:20:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/898940996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:03 smithi022 ceph-mon[31360]: pgmap v4247: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/898940996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:03 smithi022 ceph-mon[40250]: pgmap v4247: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/898940996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:03 smithi122 ceph-mon[32255]: pgmap v4247: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:05 smithi122 ceph-mon[32255]: pgmap v4248: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:05 smithi022 ceph-mon[31360]: pgmap v4248: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:05 smithi022 ceph-mon[40250]: pgmap v4248: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:07.360 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:07.664 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:20:07.665 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[40250]: pgmap v4249: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:07.665 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:20:07.666 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:20:07.666 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:20:07.666 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[31360]: pgmap v4249: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:07.666 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:20:07.667 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:20:07.667 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:20:07.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:07.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:07.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:07 smithi122 ceph-mon[32255]: pgmap v4249: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:20:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:20:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/66143104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:20:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/66143104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:20:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/66143104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:20:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:09 smithi022 ceph-mon[31360]: pgmap v4250: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:09 smithi022 ceph-mon[40250]: pgmap v4250: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:10] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:20:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:09 smithi122 ceph-mon[32255]: pgmap v4250: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:20:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:20:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:11 smithi022 ceph-mon[31360]: pgmap v4251: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:11 smithi022 ceph-mon[40250]: pgmap v4251: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:11 smithi122 ceph-mon[32255]: pgmap v4251: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:20:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:20:12.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:13.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:13.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:13.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:13 smithi022 ceph-mon[31360]: pgmap v4252: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3368373966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:13 smithi022 ceph-mon[40250]: pgmap v4252: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3368373966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:13 smithi122 ceph-mon[32255]: pgmap v4252: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3368373966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:15 smithi122 ceph-mon[32255]: pgmap v4253: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:15 smithi022 ceph-mon[31360]: pgmap v4253: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:15 smithi022 ceph-mon[40250]: pgmap v4253: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:17.667 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:20:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:17 smithi122 ceph-mon[32255]: pgmap v4254: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:17 smithi022 ceph-mon[31360]: pgmap v4254: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:17 smithi022 ceph-mon[40250]: pgmap v4254: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:18.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:18.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:18.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:18.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2295406390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2295406390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2295406390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:19 smithi122 ceph-mon[32255]: pgmap v4255: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:19 smithi022 ceph-mon[31360]: pgmap v4255: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:19 smithi022 ceph-mon[40250]: pgmap v4255: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:20] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:20:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:20:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:20:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:21 smithi122 ceph-mon[32255]: pgmap v4256: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:21 smithi022 ceph-mon[31360]: pgmap v4256: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:21 smithi022 ceph-mon[40250]: pgmap v4256: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:20:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:20:23.481 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:23.482 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:23 smithi022 ceph-mon[31360]: pgmap v4257: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:23.482 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:23 smithi022 ceph-mon[40250]: pgmap v4257: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:23 smithi122 ceph-mon[32255]: pgmap v4257: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/983207666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/983207666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/983207666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:25 smithi022 ceph-mon[31360]: pgmap v4258: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:25 smithi022 ceph-mon[40250]: pgmap v4258: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:25 smithi122 ceph-mon[32255]: pgmap v4258: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:27.668 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:20:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:27 smithi122 ceph-mon[32255]: pgmap v4259: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:27 smithi022 ceph-mon[31360]: pgmap v4259: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:27 smithi022 ceph-mon[40250]: pgmap v4259: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:28.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:29.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:29.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:29.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:29 smithi122 ceph-mon[32255]: pgmap v4260: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2422437419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:29 smithi022 ceph-mon[31360]: pgmap v4260: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2422437419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:29 smithi022 ceph-mon[40250]: pgmap v4260: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2422437419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:30] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:20:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:20:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:20:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:31 smithi122 ceph-mon[32255]: pgmap v4261: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:31 smithi022 ceph-mon[31360]: pgmap v4261: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:31 smithi022 ceph-mon[40250]: pgmap v4261: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:20:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[31360]: pgmap v4262: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:20:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:20:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[40250]: pgmap v4262: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:20:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:20:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:20:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:20:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:33 smithi122 ceph-mon[32255]: pgmap v4262: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:20:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:20:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:20:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:20:34.219 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:34.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:34.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:34.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1792320316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1792320316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1792320316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:35 smithi122 ceph-mon[32255]: pgmap v4263: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:35 smithi022 ceph-mon[31360]: pgmap v4263: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:35 smithi022 ceph-mon[40250]: pgmap v4263: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:37.672 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:20:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:37 smithi122 ceph-mon[32255]: pgmap v4264: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:37 smithi022 ceph-mon[31360]: pgmap v4264: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:37 smithi022 ceph-mon[40250]: pgmap v4264: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:39.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:39 smithi122 ceph-mon[32255]: pgmap v4265: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:39.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:39.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:39.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:39.961 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:39 smithi022 ceph-mon[40250]: pgmap v4265: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:39.961 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:39 smithi022 ceph-mon[31360]: pgmap v4265: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:40] "GET /metrics HTTP/1.1" 200 202337 "" "Prometheus/2.18.1" 2021-12-15T20:20:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/765485712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:40.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:20:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:20:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/765485712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/765485712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:41 smithi122 ceph-mon[32255]: pgmap v4266: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:41 smithi022 ceph-mon[31360]: pgmap v4266: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:41 smithi022 ceph-mon[40250]: pgmap v4266: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:20:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:20:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:43 smithi022 ceph-mon[31360]: pgmap v4267: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:43 smithi022 ceph-mon[40250]: pgmap v4267: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:43 smithi122 ceph-mon[32255]: pgmap v4267: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:44.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:45.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:45.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:45.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:45 smithi122 ceph-mon[32255]: pgmap v4268: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/924663008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:45 smithi022 ceph-mon[31360]: pgmap v4268: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/924663008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:45 smithi022 ceph-mon[40250]: pgmap v4268: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/924663008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:47.677 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:20:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:47 smithi122 ceph-mon[32255]: pgmap v4269: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:47 smithi022 ceph-mon[31360]: pgmap v4269: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:47 smithi022 ceph-mon[40250]: pgmap v4269: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:49 smithi122 ceph-mon[32255]: pgmap v4270: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:49 smithi022 ceph-mon[31360]: pgmap v4270: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:49 smithi022 ceph-mon[40250]: pgmap v4270: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:50.332 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:50] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:20:50.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:50.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:50.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:20:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:20:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:51 smithi122 ceph-mon[32255]: pgmap v4271: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2786967860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:51 smithi022 ceph-mon[31360]: pgmap v4271: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2786967860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:20:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:51 smithi022 ceph-mon[40250]: pgmap v4271: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2786967860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:20:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:20:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:53 smithi022 ceph-mon[31360]: pgmap v4272: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:53 smithi022 ceph-mon[40250]: pgmap v4272: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:53 smithi122 ceph-mon[32255]: pgmap v4272: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:20:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:55 smithi122 ceph-mon[32255]: pgmap v4273: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:55 smithi022 ceph-mon[31360]: pgmap v4273: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:55 smithi022 ceph-mon[40250]: pgmap v4273: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:20:56.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:20:56.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:20:56.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:20:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2181080148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2181080148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2181080148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:20:57.684 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:20:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:57 smithi122 ceph-mon[32255]: pgmap v4274: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:57 smithi022 ceph-mon[31360]: pgmap v4274: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:20:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:57 smithi022 ceph-mon[40250]: pgmap v4274: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:20:59 smithi022 ceph-mon[31360]: pgmap v4275: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:20:59 smithi022 ceph-mon[40250]: pgmap v4275: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:20:59 smithi122 ceph-mon[32255]: pgmap v4275: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:00] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:21:00.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:21:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:21:01.078 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:01.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:01.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:01.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:01 smithi022 ceph-mon[31360]: pgmap v4276: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4258662955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:01 smithi022 ceph-mon[40250]: pgmap v4276: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4258662955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:01 smithi122 ceph-mon[32255]: pgmap v4276: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4258662955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:21:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:21:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:03 smithi022 ceph-mon[31360]: pgmap v4277: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:03 smithi022 ceph-mon[40250]: pgmap v4277: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:03 smithi122 ceph-mon[32255]: pgmap v4277: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:05 smithi122 ceph-mon[32255]: pgmap v4278: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:05 smithi022 ceph-mon[31360]: pgmap v4278: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:05 smithi022 ceph-mon[40250]: pgmap v4278: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:06.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:06.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:06.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:06.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:07.682 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:21:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:07 smithi122 ceph-mon[32255]: pgmap v4279: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/495531081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:08.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:07 smithi022 ceph-mon[31360]: pgmap v4279: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:08.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/495531081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:08.018 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:07 smithi022 ceph-mon[40250]: pgmap v4279: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:21:08.018 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/495531081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:21:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:21:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:21:09.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:21:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:21:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:21:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:21:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:21:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:21:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:09 smithi022 ceph-mon[31360]: pgmap v4280: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:09 smithi022 ceph-mon[40250]: pgmap v4280: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:09 smithi122 ceph-mon[32255]: pgmap v4280: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:10] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:21:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:21:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:21:11.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:11 smithi022 ceph-mon[31360]: pgmap v4281: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:11 smithi022 ceph-mon[40250]: pgmap v4281: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:12.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:12.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:12.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:11 smithi122 ceph-mon[32255]: pgmap v4281: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:21:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:21:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3773393666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3773393666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3773393666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:13 smithi122 ceph-mon[32255]: pgmap v4282: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:13 smithi022 ceph-mon[31360]: pgmap v4282: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:13 smithi022 ceph-mon[40250]: pgmap v4282: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:21:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:15 smithi122 ceph-mon[32255]: pgmap v4283: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:15 smithi022 ceph-mon[31360]: pgmap v4283: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:15 smithi022 ceph-mon[40250]: pgmap v4283: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:17.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:21:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:17.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:17.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:17 smithi122 ceph-mon[32255]: pgmap v4284: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/709859964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:17 smithi022 ceph-mon[31360]: pgmap v4284: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/709859964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:17 smithi022 ceph-mon[40250]: pgmap v4284: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/709859964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:19 smithi122 ceph-mon[32255]: pgmap v4285: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:19 smithi022 ceph-mon[31360]: pgmap v4285: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:19 smithi022 ceph-mon[40250]: pgmap v4285: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:21:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:21:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:21:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:21 smithi122 ceph-mon[32255]: pgmap v4286: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:21 smithi022 ceph-mon[31360]: pgmap v4286: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:21 smithi022 ceph-mon[40250]: pgmap v4286: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:22.557 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:21:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:21:22.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:22.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:22.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:23 smithi022 ceph-mon[31360]: pgmap v4287: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/597600853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:23 smithi022 ceph-mon[40250]: pgmap v4287: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/597600853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:23 smithi122 ceph-mon[32255]: pgmap v4287: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/597600853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:25 smithi022 ceph-mon[31360]: pgmap v4288: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:25 smithi022 ceph-mon[40250]: pgmap v4288: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:25 smithi122 ceph-mon[32255]: pgmap v4288: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:27.689 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:21:27.928 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:27 smithi022 ceph-mon[31360]: pgmap v4289: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:27 smithi022 ceph-mon[40250]: pgmap v4289: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:27 smithi122 ceph-mon[32255]: pgmap v4289: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:28.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:28.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:28.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:28.695 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1655996816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1655996816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1655996816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:29 smithi022 ceph-mon[31360]: pgmap v4290: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:29 smithi022 ceph-mon[40250]: pgmap v4290: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:29 smithi122 ceph-mon[32255]: pgmap v4290: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:21:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:21:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:21:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:31 smithi022 ceph-mon[31360]: pgmap v4291: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:31 smithi022 ceph-mon[40250]: pgmap v4291: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:31 smithi122 ceph-mon[32255]: pgmap v4291: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:21:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:21:33.299 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[40250]: pgmap v4292: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:21:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:21:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:21:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:21:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[31360]: pgmap v4292: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:21:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:21:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:21:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:21:33.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:33.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:33.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:33 smithi122 ceph-mon[32255]: pgmap v4292: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:21:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:21:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:21:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:21:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1820510963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1820510963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1820510963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:35 smithi022 ceph-mon[31360]: pgmap v4293: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:35 smithi022 ceph-mon[40250]: pgmap v4293: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:35 smithi122 ceph-mon[32255]: pgmap v4293: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:37.689 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:21:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:37 smithi022 ceph-mon[31360]: pgmap v4294: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:37 smithi022 ceph-mon[40250]: pgmap v4294: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:37 smithi122 ceph-mon[32255]: pgmap v4294: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:38.669 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:39.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:39.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:39.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:39 smithi022 ceph-mon[31360]: pgmap v4295: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2546528521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:39 smithi022 ceph-mon[40250]: pgmap v4295: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2546528521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:39 smithi122 ceph-mon[32255]: pgmap v4295: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2546528521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:40] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:21:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:21:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:21:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:41 smithi022 ceph-mon[31360]: pgmap v4296: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:41 smithi022 ceph-mon[40250]: pgmap v4296: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:41 smithi122 ceph-mon[32255]: pgmap v4296: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:21:42.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:21:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:43 smithi022 ceph-mon[31360]: pgmap v4297: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:43 smithi022 ceph-mon[40250]: pgmap v4297: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:43 smithi122 ceph-mon[32255]: pgmap v4297: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:44.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:44.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:44.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:44.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3550369388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3550369388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3550369388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:45 smithi022 ceph-mon[31360]: pgmap v4298: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:45 smithi022 ceph-mon[40250]: pgmap v4298: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:45 smithi122 ceph-mon[32255]: pgmap v4298: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:47.692 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:21:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:47 smithi022 ceph-mon[31360]: pgmap v4299: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:47 smithi022 ceph-mon[40250]: pgmap v4299: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:47 smithi122 ceph-mon[32255]: pgmap v4299: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:49.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:49.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:49.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:49.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:49 smithi022 ceph-mon[31360]: pgmap v4300: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:49 smithi022 ceph-mon[40250]: pgmap v4300: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:49 smithi122 ceph-mon[32255]: pgmap v4300: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:50] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:21:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1974240191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:21:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:21:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1974240191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1974240191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:51 smithi022 ceph-mon[31360]: pgmap v4301: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:51 smithi022 ceph-mon[40250]: pgmap v4301: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:51 smithi122 ceph-mon[32255]: pgmap v4301: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:21:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:21:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:21:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:53 smithi022 ceph-mon[31360]: pgmap v4302: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:53 smithi022 ceph-mon[40250]: pgmap v4302: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:53 smithi122 ceph-mon[32255]: pgmap v4302: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:54.782 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:21:55.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:21:55.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:21:55.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:21:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:55 smithi022 ceph-mon[31360]: pgmap v4303: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3699619705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:55 smithi022 ceph-mon[40250]: pgmap v4303: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3699619705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:55 smithi122 ceph-mon[32255]: pgmap v4303: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:21:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3699619705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:21:57.697 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:21:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:57 smithi022 ceph-mon[31360]: pgmap v4304: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:57 smithi022 ceph-mon[40250]: pgmap v4304: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:21:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:57 smithi122 ceph-mon[32255]: pgmap v4304: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:21:59 smithi022 ceph-mon[31360]: pgmap v4305: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:21:59 smithi022 ceph-mon[40250]: pgmap v4305: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:00.154 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:21:59 smithi122 ceph-mon[32255]: pgmap v4305: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:00] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:22:00.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:00.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:00.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/131525703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/131525703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/131525703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:00.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:22:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:22:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:01 smithi022 ceph-mon[31360]: pgmap v4306: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:01 smithi022 ceph-mon[40250]: pgmap v4306: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:01 smithi122 ceph-mon[32255]: pgmap v4306: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:22:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:03 smithi022 ceph-mon[31360]: pgmap v4307: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:03 smithi022 ceph-mon[40250]: pgmap v4307: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:03 smithi122 ceph-mon[32255]: pgmap v4307: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:05.527 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:05 smithi022 ceph-mon[31360]: pgmap v4308: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:05 smithi022 ceph-mon[40250]: pgmap v4308: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:05.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:05.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:05.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:05 smithi122 ceph-mon[32255]: pgmap v4308: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4151430510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4151430510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4151430510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:07.704 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:22:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:07 smithi022 ceph-mon[31360]: pgmap v4309: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:07 smithi022 ceph-mon[40250]: pgmap v4309: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:07 smithi122 ceph-mon[32255]: pgmap v4309: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:09 smithi022 ceph-mon[31360]: pgmap v4310: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:09 smithi022 ceph-mon[40250]: pgmap v4310: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:09 smithi122 ceph-mon[32255]: pgmap v4310: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:10] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:22:10.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:22:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:22:11.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:11.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:11.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:11 smithi022 ceph-mon[31360]: pgmap v4311: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1859690334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:11 smithi022 ceph-mon[40250]: pgmap v4311: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1859690334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:11 smithi122 ceph-mon[32255]: pgmap v4311: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1859690334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:22:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:22:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:13 smithi022 ceph-mon[31360]: pgmap v4312: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:13 smithi022 ceph-mon[40250]: pgmap v4312: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:13.663 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:13 smithi122 ceph-mon[32255]: pgmap v4312: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:22:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:22:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:22:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:22:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:22:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:22:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:22:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:22:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:22:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:22:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:22:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:22:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:15 smithi022 ceph-mon[31360]: pgmap v4313: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:15 smithi022 ceph-mon[40250]: pgmap v4313: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:15 smithi122 ceph-mon[32255]: pgmap v4313: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:16.269 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:16.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:16.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:16.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3209466214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3209466214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3209466214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:17.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:22:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:17 smithi022 ceph-mon[31360]: pgmap v4314: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:17 smithi022 ceph-mon[40250]: pgmap v4314: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:17 smithi122 ceph-mon[32255]: pgmap v4314: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:19 smithi022 ceph-mon[31360]: pgmap v4315: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:19 smithi022 ceph-mon[40250]: pgmap v4315: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:19 smithi122 ceph-mon[32255]: pgmap v4315: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:20] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:22:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:22:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:22:21.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:22.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:22.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:22.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:22.011 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:21 smithi022 ceph-mon[40250]: pgmap v4316: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:22.012 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:22.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:21 smithi022 ceph-mon[31360]: pgmap v4316: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:22.013 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:21 smithi122 ceph-mon[32255]: pgmap v4316: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:22:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:22:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2674755551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2674755551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2674755551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:23 smithi022 ceph-mon[31360]: pgmap v4317: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:23 smithi022 ceph-mon[40250]: pgmap v4317: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:23 smithi122 ceph-mon[32255]: pgmap v4317: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:25 smithi022 ceph-mon[31360]: pgmap v4318: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:25 smithi022 ceph-mon[40250]: pgmap v4318: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:25 smithi122 ceph-mon[32255]: pgmap v4318: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:27.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:27.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:27.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:27.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:27.714 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:22:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:27 smithi022 ceph-mon[31360]: pgmap v4319: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2847633054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:27 smithi022 ceph-mon[40250]: pgmap v4319: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2847633054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:27 smithi122 ceph-mon[32255]: pgmap v4319: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2847633054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:29 smithi022 ceph-mon[31360]: pgmap v4320: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:29 smithi022 ceph-mon[40250]: pgmap v4320: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:29 smithi122 ceph-mon[32255]: pgmap v4320: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:30] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:22:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:22:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:22:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:31 smithi022 ceph-mon[31360]: pgmap v4321: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:31 smithi022 ceph-mon[40250]: pgmap v4321: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:31 smithi122 ceph-mon[32255]: pgmap v4321: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:32.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:22:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:22:32.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:32.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:32.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[31360]: pgmap v4322: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:22:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:22:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:22:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:22:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/559811461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[40250]: pgmap v4322: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:22:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:22:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:22:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:22:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/559811461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:33 smithi122 ceph-mon[32255]: pgmap v4322: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:22:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:22:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:22:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:22:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/559811461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:35 smithi022 ceph-mon[31360]: pgmap v4323: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:35 smithi022 ceph-mon[40250]: pgmap v4323: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:35 smithi122 ceph-mon[32255]: pgmap v4323: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:37.709 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:22:37.751 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:37 smithi022 ceph-mon[31360]: pgmap v4324: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:37 smithi022 ceph-mon[40250]: pgmap v4324: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:38.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:38.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:38.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:37 smithi122 ceph-mon[32255]: pgmap v4324: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4212006950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4212006950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4212006950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:39 smithi022 ceph-mon[31360]: pgmap v4325: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:39 smithi022 ceph-mon[40250]: pgmap v4325: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:39 smithi122 ceph-mon[32255]: pgmap v4325: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:40] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:22:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:22:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:22:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:41 smithi022 ceph-mon[31360]: pgmap v4326: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:41 smithi022 ceph-mon[40250]: pgmap v4326: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:41 smithi122 ceph-mon[32255]: pgmap v4326: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:22:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:22:43.121 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:43.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:43.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:43.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:43.493 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:43 smithi022 ceph-mon[31360]: pgmap v4327: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:43.493 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:43 smithi022 ceph-mon[40250]: pgmap v4327: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:43 smithi122 ceph-mon[32255]: pgmap v4327: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/937939338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/937939338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/937939338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:45 smithi022 ceph-mon[31360]: pgmap v4328: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:45 smithi022 ceph-mon[40250]: pgmap v4328: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:45 smithi122 ceph-mon[32255]: pgmap v4328: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:47.710 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:22:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:47 smithi022 ceph-mon[31360]: pgmap v4329: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:47 smithi022 ceph-mon[40250]: pgmap v4329: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:47 smithi122 ceph-mon[32255]: pgmap v4329: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:48.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:48.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:48.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:48.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:49 smithi022 ceph-mon[31360]: pgmap v4330: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1824452350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:49 smithi022 ceph-mon[40250]: pgmap v4330: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1824452350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:49 smithi122 ceph-mon[32255]: pgmap v4330: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1824452350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:50] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:22:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:22:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:22:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:51 smithi022 ceph-mon[31360]: pgmap v4331: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:51 smithi022 ceph-mon[40250]: pgmap v4331: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:51 smithi122 ceph-mon[32255]: pgmap v4331: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:22:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:22:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:22:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:53 smithi022 ceph-mon[31360]: pgmap v4332: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:53 smithi022 ceph-mon[40250]: pgmap v4332: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:53 smithi122 ceph-mon[32255]: pgmap v4332: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:53.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:54.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:54.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:54.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:22:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4108291573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4108291573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4108291573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:22:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:55 smithi022 ceph-mon[31360]: pgmap v4333: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:55 smithi022 ceph-mon[40250]: pgmap v4333: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:55 smithi122 ceph-mon[32255]: pgmap v4333: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:22:57.713 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:22:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:57 smithi022 ceph-mon[40250]: pgmap v4334: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:57 smithi022 ceph-mon[31360]: pgmap v4334: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:57 smithi122 ceph-mon[32255]: pgmap v4334: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:22:59.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:22:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:22:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:22:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:59 smithi022 ceph-mon[31360]: pgmap v4335: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:22:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2399375405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:59 smithi022 ceph-mon[40250]: pgmap v4335: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:22:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2399375405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:59 smithi122 ceph-mon[32255]: pgmap v4335: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:22:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2399375405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:00] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:23:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:23:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:23:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:01 smithi022 ceph-mon[31360]: pgmap v4336: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:01 smithi022 ceph-mon[40250]: pgmap v4336: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:01 smithi122 ceph-mon[32255]: pgmap v4336: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:23:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:23:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:03 smithi022 ceph-mon[31360]: pgmap v4337: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:03 smithi022 ceph-mon[40250]: pgmap v4337: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:03 smithi122 ceph-mon[32255]: pgmap v4337: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:04.610 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:04.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:04.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:04.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:05 smithi022 ceph-mon[31360]: pgmap v4338: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1429039142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:05 smithi022 ceph-mon[40250]: pgmap v4338: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1429039142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:05 smithi122 ceph-mon[32255]: pgmap v4338: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1429039142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:07.722 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:23:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:07 smithi022 ceph-mon[31360]: pgmap v4339: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:07 smithi022 ceph-mon[40250]: pgmap v4339: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:07 smithi122 ceph-mon[32255]: pgmap v4339: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T20:23:09.982 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:09.983 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:09 smithi022 ceph-mon[40250]: pgmap v4340: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:09.983 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:09 smithi022 ceph-mon[31360]: pgmap v4340: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:09 smithi122 ceph-mon[32255]: pgmap v4340: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:10.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:10.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:10.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:10] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:23:10.729 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2075133788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2075133788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:23:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:23:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2075133788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:11 smithi022 ceph-mon[31360]: pgmap v4341: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:11 smithi022 ceph-mon[40250]: pgmap v4341: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:11 smithi122 ceph-mon[32255]: pgmap v4341: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:23:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:23:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:13 smithi022 ceph-mon[31360]: pgmap v4342: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:13 smithi022 ceph-mon[40250]: pgmap v4342: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:13 smithi122 ceph-mon[32255]: pgmap v4342: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:23:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:23:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:23:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:23:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:23:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:23:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:23:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:23:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:23:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:23:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:23:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:23:15.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:15 smithi022 ceph-mon[31360]: pgmap v4343: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:15 smithi022 ceph-mon[40250]: pgmap v4343: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:15 smithi122 ceph-mon[32255]: pgmap v4343: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:15.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:15.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1165462945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1165462945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1165462945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:17 smithi022 ceph-mon[31360]: pgmap v4344: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:23:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:17 smithi022 ceph-mon[40250]: pgmap v4344: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:17 smithi122 ceph-mon[32255]: pgmap v4344: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:19 smithi022 ceph-mon[31360]: pgmap v4345: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:19 smithi022 ceph-mon[40250]: pgmap v4345: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:19 smithi122 ceph-mon[32255]: pgmap v4345: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:20] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:23:20.725 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:21.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:21.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:21.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:21.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:23:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:23:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:21 smithi022 ceph-mon[31360]: pgmap v4346: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4157321469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:21 smithi022 ceph-mon[40250]: pgmap v4346: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4157321469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:21 smithi122 ceph-mon[32255]: pgmap v4346: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4157321469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:23:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:23:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:23 smithi022 ceph-mon[31360]: pgmap v4347: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:23 smithi022 ceph-mon[40250]: pgmap v4347: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:23 smithi122 ceph-mon[32255]: pgmap v4347: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:26.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:25 smithi022 ceph-mon[31360]: pgmap v4348: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:26.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:25 smithi022 ceph-mon[40250]: pgmap v4348: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:25 smithi122 ceph-mon[32255]: pgmap v4348: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:26.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:26.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:26.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2465429615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2465429615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2465429615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:27.731 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:23:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:27 smithi022 ceph-mon[31360]: pgmap v4349: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:28.122 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:27 smithi022 ceph-mon[40250]: pgmap v4349: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:27 smithi122 ceph-mon[32255]: pgmap v4349: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:29 smithi022 ceph-mon[31360]: pgmap v4350: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:29 smithi022 ceph-mon[40250]: pgmap v4350: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:29 smithi122 ceph-mon[32255]: pgmap v4350: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:30] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:23:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:23:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:23:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:31.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:31.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:31.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:31 smithi022 ceph-mon[31360]: pgmap v4351: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:31 smithi022 ceph-mon[40250]: pgmap v4351: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:31 smithi122 ceph-mon[32255]: pgmap v4351: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:23:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:23:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2303670760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:23:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:23:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:23:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:23:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2303670760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:23:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:23:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:23:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:23:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2303670760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:23:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:23:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:23:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:23:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:33 smithi022 ceph-mon[31360]: pgmap v4352: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:33 smithi022 ceph-mon[40250]: pgmap v4352: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:33 smithi122 ceph-mon[32255]: pgmap v4352: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:35 smithi022 ceph-mon[31360]: pgmap v4353: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:35 smithi022 ceph-mon[40250]: pgmap v4353: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:35 smithi122 ceph-mon[32255]: pgmap v4353: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:36.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:37.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:37.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:37.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:23:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:37 smithi022 ceph-mon[31360]: pgmap v4354: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1920435850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:37 smithi022 ceph-mon[40250]: pgmap v4354: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1920435850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:37 smithi122 ceph-mon[32255]: pgmap v4354: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1920435850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:39 smithi022 ceph-mon[31360]: pgmap v4355: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:39 smithi022 ceph-mon[40250]: pgmap v4355: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:39 smithi122 ceph-mon[32255]: pgmap v4355: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:40] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:23:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:23:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:23:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:41 smithi022 ceph-mon[31360]: pgmap v4356: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:41 smithi022 ceph-mon[40250]: pgmap v4356: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:41 smithi122 ceph-mon[32255]: pgmap v4356: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:42.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:42.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:42.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:42.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:23:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:23:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3632322167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:43 smithi022 ceph-mon[31360]: pgmap v4357: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3632322167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:43 smithi022 ceph-mon[40250]: pgmap v4357: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3632322167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:43 smithi122 ceph-mon[32255]: pgmap v4357: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:45 smithi022 ceph-mon[31360]: pgmap v4358: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:45 smithi022 ceph-mon[40250]: pgmap v4358: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:45 smithi122 ceph-mon[32255]: pgmap v4358: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:47.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:47.734 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:23:47.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:47.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:47.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:47 smithi022 ceph-mon[31360]: pgmap v4359: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:47 smithi022 ceph-mon[40250]: pgmap v4359: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:47 smithi122 ceph-mon[32255]: pgmap v4359: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3838192850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3838192850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3838192850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:49 smithi022 ceph-mon[31360]: pgmap v4360: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:49 smithi022 ceph-mon[40250]: pgmap v4360: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:49 smithi122 ceph-mon[32255]: pgmap v4360: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:23:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:23:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:23:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:51 smithi022 ceph-mon[31360]: pgmap v4361: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:51 smithi022 ceph-mon[40250]: pgmap v4361: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:51 smithi122 ceph-mon[32255]: pgmap v4361: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:23:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:23:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:23:52.943 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:53.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:53 smithi022 ceph-mon[31360]: pgmap v4362: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:53 smithi022 ceph-mon[40250]: pgmap v4362: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:53 smithi122 ceph-mon[32255]: pgmap v4362: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3475310632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3475310632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3475310632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:55 smithi022 ceph-mon[31360]: pgmap v4363: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:55 smithi022 ceph-mon[40250]: pgmap v4363: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:55 smithi122 ceph-mon[32255]: pgmap v4363: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:23:57.739 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:23:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:57 smithi022 ceph-mon[31360]: pgmap v4364: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:57 smithi022 ceph-mon[40250]: pgmap v4364: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:57 smithi122 ceph-mon[32255]: pgmap v4364: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:23:58.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:23:58.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:23:58.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:23:58.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:23:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/315719658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:59.102 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/315719658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:23:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/315719658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:23:59 smithi022 ceph-mon[31360]: pgmap v4365: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:23:59 smithi022 ceph-mon[40250]: pgmap v4365: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:23:59 smithi122 ceph-mon[32255]: pgmap v4365: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:24:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:24:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:24:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:01 smithi022 ceph-mon[31360]: pgmap v4366: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:01 smithi022 ceph-mon[40250]: pgmap v4366: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:01 smithi122 ceph-mon[32255]: pgmap v4366: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:24:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:24:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:03 smithi022 ceph-mon[31360]: pgmap v4367: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:03 smithi022 ceph-mon[40250]: pgmap v4367: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:03 smithi122 ceph-mon[32255]: pgmap v4367: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:04.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:04.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:04.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/732944822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/732944822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/732944822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:05 smithi022 ceph-mon[31360]: pgmap v4368: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:05 smithi022 ceph-mon[40250]: pgmap v4368: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:05 smithi122 ceph-mon[32255]: pgmap v4368: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:07.737 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:24:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:07 smithi022 ceph-mon[31360]: pgmap v4369: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:07 smithi022 ceph-mon[40250]: pgmap v4369: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:07 smithi122 ceph-mon[32255]: pgmap v4369: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:09.062 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:09 smithi022 ceph-mon[31360]: pgmap v4370: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:10.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2887653452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:09 smithi022 ceph-mon[40250]: pgmap v4370: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:10.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2887653452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:09 smithi122 ceph-mon[32255]: pgmap v4370: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2887653452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:24:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:24:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:24:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:11 smithi022 ceph-mon[31360]: pgmap v4371: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:11 smithi022 ceph-mon[40250]: pgmap v4371: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:11 smithi122 ceph-mon[32255]: pgmap v4371: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:24:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:24:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:13 smithi022 ceph-mon[31360]: pgmap v4372: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:13 smithi022 ceph-mon[40250]: pgmap v4372: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:13 smithi122 ceph-mon[32255]: pgmap v4372: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:24:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:24:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:24:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:24:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:24:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:24:14.602 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:24:14.602 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:24:14.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:24:14.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:14.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:14.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:24:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:15 smithi122 ceph-mon[32255]: pgmap v4373: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2061543718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:16.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:24:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:15 smithi022 ceph-mon[31360]: pgmap v4373: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2061543718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:24:16.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:15 smithi022 ceph-mon[40250]: pgmap v4373: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:16.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2061543718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:17.741 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:24:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:17 smithi022 ceph-mon[31360]: pgmap v4374: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:17 smithi022 ceph-mon[40250]: pgmap v4374: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:17 smithi122 ceph-mon[32255]: pgmap v4374: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:19.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:19 smithi022 ceph-mon[31360]: pgmap v4375: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:19 smithi022 ceph-mon[40250]: pgmap v4375: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:20.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:20.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:20.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:19 smithi122 ceph-mon[32255]: pgmap v4375: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:20] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:24:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3142748877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3142748877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3142748877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:21.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:24:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:24:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:21 smithi022 ceph-mon[31360]: pgmap v4376: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:21 smithi022 ceph-mon[40250]: pgmap v4376: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:21 smithi122 ceph-mon[32255]: pgmap v4376: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:24:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:24:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:23 smithi022 ceph-mon[31360]: pgmap v4377: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:23 smithi022 ceph-mon[40250]: pgmap v4377: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:23 smithi122 ceph-mon[32255]: pgmap v4377: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:25.166 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:25.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:25.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:25.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:25 smithi022 ceph-mon[31360]: pgmap v4378: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2333316691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:25 smithi022 ceph-mon[40250]: pgmap v4378: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2333316691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:25 smithi122 ceph-mon[32255]: pgmap v4378: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2333316691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:27.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:24:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:27 smithi022 ceph-mon[31360]: pgmap v4379: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:27 smithi022 ceph-mon[40250]: pgmap v4379: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:27 smithi122 ceph-mon[32255]: pgmap v4379: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:29 smithi022 ceph-mon[31360]: pgmap v4380: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:29 smithi022 ceph-mon[40250]: pgmap v4380: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:29 smithi122 ceph-mon[32255]: pgmap v4380: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:30] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:24:30.536 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:30.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:30.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:30.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:24:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:24:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:31 smithi022 ceph-mon[31360]: pgmap v4381: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/722591737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:31 smithi022 ceph-mon[40250]: pgmap v4381: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/722591737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:31 smithi122 ceph-mon[32255]: pgmap v4381: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/722591737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:24:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:24:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:24:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:24:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:24:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:24:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[31360]: pgmap v4382: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:24:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:24:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:24:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:24:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:33 smithi022 ceph-mon[40250]: pgmap v4382: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:33 smithi122 ceph-mon[32255]: pgmap v4382: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:35.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:35 smithi022 ceph-mon[31360]: pgmap v4383: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:35 smithi022 ceph-mon[40250]: pgmap v4383: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:35 smithi122 ceph-mon[32255]: pgmap v4383: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:36.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:36.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:36.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/123245803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/123245803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/123245803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:37.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:24:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:37 smithi022 ceph-mon[31360]: pgmap v4384: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:37 smithi022 ceph-mon[40250]: pgmap v4384: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:37 smithi122 ceph-mon[32255]: pgmap v4384: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:39 smithi022 ceph-mon[31360]: pgmap v4385: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:39 smithi022 ceph-mon[40250]: pgmap v4385: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:39 smithi122 ceph-mon[32255]: pgmap v4385: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:40] "GET /metrics HTTP/1.1" 200 202337 "" "Prometheus/2.18.1" 2021-12-15T20:24:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:24:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:24:41.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:41.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:41.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:41.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:41 smithi022 ceph-mon[31360]: pgmap v4386: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1493816009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:41 smithi022 ceph-mon[40250]: pgmap v4386: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1493816009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:41 smithi122 ceph-mon[32255]: pgmap v4386: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1493816009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:24:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:24:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:43 smithi022 ceph-mon[31360]: pgmap v4387: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:43 smithi022 ceph-mon[40250]: pgmap v4387: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:43 smithi122 ceph-mon[32255]: pgmap v4387: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:45 smithi022 ceph-mon[31360]: pgmap v4388: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:45 smithi022 ceph-mon[40250]: pgmap v4388: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:45 smithi122 ceph-mon[32255]: pgmap v4388: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:46.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:47.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:47.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:47.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:47.751 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:24:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:47 smithi022 ceph-mon[40250]: pgmap v4389: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2143723645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:47 smithi022 ceph-mon[31360]: pgmap v4389: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2143723645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:47 smithi122 ceph-mon[32255]: pgmap v4389: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2143723645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:49 smithi022 ceph-mon[31360]: pgmap v4390: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:49 smithi022 ceph-mon[40250]: pgmap v4390: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:49 smithi122 ceph-mon[32255]: pgmap v4390: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:24:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:24:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:24:52.026 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:52.026 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:51 smithi022 ceph-mon[31360]: pgmap v4391: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:52.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:52.027 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:51 smithi022 ceph-mon[40250]: pgmap v4391: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:52.027 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:51 smithi122 ceph-mon[32255]: pgmap v4391: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:24:52.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:52.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:52.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:24:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:24:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2296109822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:53 smithi022 ceph-mon[31360]: pgmap v4392: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2296109822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:53 smithi022 ceph-mon[40250]: pgmap v4392: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2296109822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:53 smithi122 ceph-mon[32255]: pgmap v4392: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:55 smithi022 ceph-mon[31360]: pgmap v4393: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:55 smithi022 ceph-mon[40250]: pgmap v4393: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:55 smithi122 ceph-mon[32255]: pgmap v4393: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:24:57.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:24:57.754 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:24:57.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:24:57.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:24:57.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:24:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:57 smithi022 ceph-mon[31360]: pgmap v4394: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1954621990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:57 smithi022 ceph-mon[40250]: pgmap v4394: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1954621990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:24:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:57 smithi122 ceph-mon[32255]: pgmap v4394: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:24:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1954621990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:24:59 smithi022 ceph-mon[31360]: pgmap v4395: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:24:59 smithi022 ceph-mon[40250]: pgmap v4395: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:24:59 smithi122 ceph-mon[32255]: pgmap v4395: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:25:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:25:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:25:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:01 smithi022 ceph-mon[31360]: pgmap v4396: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:01 smithi022 ceph-mon[40250]: pgmap v4396: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:01 smithi122 ceph-mon[32255]: pgmap v4396: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:25:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:25:02.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:03.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:03.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:03.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:03 smithi022 ceph-mon[31360]: pgmap v4397: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3184987589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:03 smithi022 ceph-mon[40250]: pgmap v4397: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3184987589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:03 smithi122 ceph-mon[32255]: pgmap v4397: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3184987589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:05 smithi022 ceph-mon[31360]: pgmap v4398: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:05 smithi022 ceph-mon[40250]: pgmap v4398: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:05 smithi122 ceph-mon[32255]: pgmap v4398: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:07.757 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:25:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:07 smithi022 ceph-mon[31360]: pgmap v4399: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:07 smithi022 ceph-mon[40250]: pgmap v4399: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:08.139 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:07 smithi122 ceph-mon[32255]: pgmap v4399: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:08.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:08.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:08.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:08.762 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2921290298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2921290298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2921290298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:09 smithi022 ceph-mon[31360]: pgmap v4400: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:09 smithi022 ceph-mon[40250]: pgmap v4400: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:10.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:09 smithi122 ceph-mon[32255]: pgmap v4400: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:25:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:25:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:25:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:11 smithi022 ceph-mon[31360]: pgmap v4401: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:11 smithi022 ceph-mon[40250]: pgmap v4401: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:11 smithi122 ceph-mon[32255]: pgmap v4401: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:25:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:25:13.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:13 smithi022 ceph-mon[31360]: pgmap v4402: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:13 smithi022 ceph-mon[40250]: pgmap v4402: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:13 smithi122 ceph-mon[32255]: pgmap v4402: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:13.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/755909747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/755909747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:14.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/755909747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:25:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:25:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:25:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:15 smithi122 ceph-mon[32255]: pgmap v4403: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:25:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:25:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:25:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:25:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[31360]: pgmap v4403: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:25:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:25:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:25:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:25:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[40250]: pgmap v4403: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:25:17.760 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:25:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:17 smithi022 ceph-mon[31360]: pgmap v4404: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:17 smithi022 ceph-mon[40250]: pgmap v4404: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:17 smithi122 ceph-mon[32255]: pgmap v4404: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:18.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:19.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:19.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:19.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:19 smithi022 ceph-mon[31360]: pgmap v4405: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1170449118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:19 smithi022 ceph-mon[40250]: pgmap v4405: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1170449118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:19 smithi122 ceph-mon[32255]: pgmap v4405: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1170449118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:20] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:25:21.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:25:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:25:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:21 smithi022 ceph-mon[31360]: pgmap v4406: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:21 smithi022 ceph-mon[40250]: pgmap v4406: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:21 smithi122 ceph-mon[32255]: pgmap v4406: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:25:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:25:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:23 smithi022 ceph-mon[31360]: pgmap v4407: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:23 smithi022 ceph-mon[40250]: pgmap v4407: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:23 smithi122 ceph-mon[32255]: pgmap v4407: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:24.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:24.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:24.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:24.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/414861572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/414861572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/414861572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:25 smithi122 ceph-mon[32255]: pgmap v4408: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:25 smithi022 ceph-mon[31360]: pgmap v4408: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:25 smithi022 ceph-mon[40250]: pgmap v4408: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:27.761 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:25:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:27 smithi022 ceph-mon[31360]: pgmap v4409: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:27 smithi022 ceph-mon[40250]: pgmap v4409: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:27 smithi122 ceph-mon[32255]: pgmap v4409: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:29.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:29.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:29.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:29.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:29.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:29 smithi022 ceph-mon[31360]: pgmap v4410: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:29.994 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:29 smithi022 ceph-mon[40250]: pgmap v4410: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:29 smithi122 ceph-mon[32255]: pgmap v4410: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:30] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:25:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1006948904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1006948904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1006948904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:25:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:25:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:31 smithi022 ceph-mon[31360]: pgmap v4411: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:31 smithi022 ceph-mon[40250]: pgmap v4411: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:31 smithi122 ceph-mon[32255]: pgmap v4411: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:25:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:25:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:25:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:25:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:25:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:25:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[40250]: pgmap v4412: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:25:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:25:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:25:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:25:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:33 smithi022 ceph-mon[31360]: pgmap v4412: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:25:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:25:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:25:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:25:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:33 smithi122 ceph-mon[32255]: pgmap v4412: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:34.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:35.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:35.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:35.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:35 smithi022 ceph-mon[31360]: pgmap v4413: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2724314532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:35 smithi022 ceph-mon[40250]: pgmap v4413: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2724314532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:35 smithi122 ceph-mon[32255]: pgmap v4413: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2724314532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:37.767 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:25:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:37 smithi022 ceph-mon[31360]: pgmap v4414: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:37 smithi022 ceph-mon[40250]: pgmap v4414: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:37 smithi122 ceph-mon[32255]: pgmap v4414: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:39 smithi022 ceph-mon[31360]: pgmap v4415: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:39 smithi022 ceph-mon[40250]: pgmap v4415: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:39 smithi122 ceph-mon[32255]: pgmap v4415: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:40.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:40] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:25:40.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:40.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:40.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1685578750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1685578750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:25:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:25:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1685578750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:41 smithi022 ceph-mon[31360]: pgmap v4416: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:41 smithi022 ceph-mon[40250]: pgmap v4416: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:41 smithi122 ceph-mon[32255]: pgmap v4416: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:25:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:25:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:43 smithi122 ceph-mon[32255]: pgmap v4417: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:43 smithi022 ceph-mon[40250]: pgmap v4417: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:43 smithi022 ceph-mon[31360]: pgmap v4417: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:45.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:45 smithi022 ceph-mon[31360]: pgmap v4418: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:45 smithi022 ceph-mon[40250]: pgmap v4418: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:46.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:46.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:46.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:45 smithi122 ceph-mon[32255]: pgmap v4418: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1983572676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1983572676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1983572676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:47.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:25:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:47 smithi022 ceph-mon[31360]: pgmap v4419: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:47 smithi022 ceph-mon[40250]: pgmap v4419: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:47 smithi122 ceph-mon[32255]: pgmap v4419: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:49 smithi022 ceph-mon[31360]: pgmap v4420: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:49 smithi022 ceph-mon[40250]: pgmap v4420: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:49 smithi122 ceph-mon[32255]: pgmap v4420: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:25:51.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:25:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:25:51.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:51.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:51.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:51 smithi022 ceph-mon[31360]: pgmap v4421: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/866744352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:51 smithi022 ceph-mon[40250]: pgmap v4421: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/866744352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:51 smithi122 ceph-mon[32255]: pgmap v4421: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:25:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/866744352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:25:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:25:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:53 smithi122 ceph-mon[32255]: pgmap v4422: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:53 smithi022 ceph-mon[31360]: pgmap v4422: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:53 smithi022 ceph-mon[40250]: pgmap v4422: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:55 smithi022 ceph-mon[31360]: pgmap v4423: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:55 smithi022 ceph-mon[40250]: pgmap v4423: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:55 smithi122 ceph-mon[32255]: pgmap v4423: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:25:56.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:25:56.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:25:56.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:25:56.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:25:57.772 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:25:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:57 smithi022 ceph-mon[31360]: pgmap v4424: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/471427655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:57 smithi022 ceph-mon[40250]: pgmap v4424: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/471427655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:25:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:57 smithi122 ceph-mon[32255]: pgmap v4424: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:25:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/471427655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:25:59 smithi022 ceph-mon[31360]: pgmap v4425: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:25:59 smithi022 ceph-mon[40250]: pgmap v4425: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:25:59 smithi122 ceph-mon[32255]: pgmap v4425: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:26:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:26:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:26:01.851 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:01 smithi022 ceph-mon[31360]: pgmap v4426: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:01 smithi022 ceph-mon[40250]: pgmap v4426: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:01 smithi122 ceph-mon[32255]: pgmap v4426: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:02.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:02.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:02.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:26:02.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:26:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1495407900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1495407900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1495407900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:03 smithi122 ceph-mon[32255]: pgmap v4427: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:03 smithi022 ceph-mon[31360]: pgmap v4427: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:03 smithi022 ceph-mon[40250]: pgmap v4427: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:05 smithi122 ceph-mon[32255]: pgmap v4428: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:05 smithi022 ceph-mon[31360]: pgmap v4428: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:05 smithi022 ceph-mon[40250]: pgmap v4428: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:07.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:26:07.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:07.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:07.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:07 smithi022 ceph-mon[31360]: pgmap v4429: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:08.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1367274127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:07 smithi022 ceph-mon[40250]: pgmap v4429: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:08.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1367274127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:07 smithi122 ceph-mon[32255]: pgmap v4429: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1367274127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:09 smithi122 ceph-mon[32255]: pgmap v4430: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:09 smithi022 ceph-mon[31360]: pgmap v4430: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:09 smithi022 ceph-mon[40250]: pgmap v4430: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:10] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:26:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:26:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:26:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:11 smithi122 ceph-mon[32255]: pgmap v4431: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:11 smithi022 ceph-mon[31360]: pgmap v4431: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:11 smithi022 ceph-mon[40250]: pgmap v4431: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:12.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:26:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:26:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:12.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:13 smithi122 ceph-mon[32255]: pgmap v4432: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2233294907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:13 smithi022 ceph-mon[40250]: pgmap v4432: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2233294907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:13 smithi022 ceph-mon[31360]: pgmap v4432: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2233294907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:15 smithi122 ceph-mon[32255]: pgmap v4433: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:26:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:26:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:26:15.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:26:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[31360]: pgmap v4433: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:26:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:26:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:26:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[40250]: pgmap v4433: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:26:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:26:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:26:17.966 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:17 smithi022 ceph-mon[31360]: pgmap v4434: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:17 smithi022 ceph-mon[40250]: pgmap v4434: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:17 smithi122 ceph-mon[32255]: pgmap v4434: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:18.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:18.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:18.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3768470751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3768470751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3768470751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:19 smithi122 ceph-mon[32255]: pgmap v4435: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:19 smithi022 ceph-mon[31360]: pgmap v4435: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:19 smithi022 ceph-mon[40250]: pgmap v4435: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:26:21.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:26:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:26:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:21 smithi122 ceph-mon[32255]: pgmap v4436: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:21 smithi022 ceph-mon[31360]: pgmap v4436: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:21 smithi022 ceph-mon[40250]: pgmap v4436: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:26:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:26:23.335 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:23 smithi122 ceph-mon[32255]: pgmap v4437: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:23.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:23 smithi022 ceph-mon[31360]: pgmap v4437: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:23.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:23 smithi022 ceph-mon[40250]: pgmap v4437: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1281366544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1281366544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1281366544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:25 smithi122 ceph-mon[32255]: pgmap v4438: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:25 smithi022 ceph-mon[31360]: pgmap v4438: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:25 smithi022 ceph-mon[40250]: pgmap v4438: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:26:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:27 smithi122 ceph-mon[32255]: pgmap v4439: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:27 smithi022 ceph-mon[31360]: pgmap v4439: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:27 smithi022 ceph-mon[40250]: pgmap v4439: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:28.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:29.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:29.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:29.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:29 smithi122 ceph-mon[32255]: pgmap v4440: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/54648584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:29 smithi022 ceph-mon[31360]: pgmap v4440: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/54648584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:29 smithi022 ceph-mon[40250]: pgmap v4440: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/54648584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:26:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:26:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:26:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:31 smithi122 ceph-mon[32255]: pgmap v4441: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:31 smithi022 ceph-mon[31360]: pgmap v4441: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:31 smithi022 ceph-mon[40250]: pgmap v4441: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:26:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:26:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:26:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:26:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:26:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:26:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:33 smithi122 ceph-mon[32255]: pgmap v4442: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:26:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:26:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:26:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:26:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[31360]: pgmap v4442: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:26:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:26:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:26:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:26:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:33 smithi022 ceph-mon[40250]: pgmap v4442: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:34.069 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:34.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:34.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:34.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1160536604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1160536604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1160536604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:35 smithi022 ceph-mon[31360]: pgmap v4443: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:35 smithi022 ceph-mon[40250]: pgmap v4443: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:35 smithi122 ceph-mon[32255]: pgmap v4443: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:26:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:37 smithi122 ceph-mon[32255]: pgmap v4444: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:37 smithi022 ceph-mon[31360]: pgmap v4444: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:37 smithi022 ceph-mon[40250]: pgmap v4444: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:39.439 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:39.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:39.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:39.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:39 smithi122 ceph-mon[32255]: pgmap v4445: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:39 smithi022 ceph-mon[31360]: pgmap v4445: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:39 smithi022 ceph-mon[40250]: pgmap v4445: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:40] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:26:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3542107832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:41.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:26:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:26:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3542107832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3542107832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:41 smithi122 ceph-mon[32255]: pgmap v4446: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:41 smithi022 ceph-mon[31360]: pgmap v4446: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:41 smithi022 ceph-mon[40250]: pgmap v4446: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:26:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:26:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:43 smithi122 ceph-mon[32255]: pgmap v4447: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:43.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:43 smithi022 ceph-mon[31360]: pgmap v4447: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:43.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:43 smithi022 ceph-mon[40250]: pgmap v4447: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:44.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:45.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:45.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:45 smithi122 ceph-mon[32255]: pgmap v4448: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3321482961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:45 smithi022 ceph-mon[31360]: pgmap v4448: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3321482961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:45 smithi022 ceph-mon[40250]: pgmap v4448: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3321482961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:26:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:47 smithi122 ceph-mon[32255]: pgmap v4449: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:47 smithi022 ceph-mon[31360]: pgmap v4449: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:47 smithi022 ceph-mon[40250]: pgmap v4449: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:49 smithi122 ceph-mon[32255]: pgmap v4450: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:50.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:50.179 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:49 smithi022 ceph-mon[40250]: pgmap v4450: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:50.179 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:49 smithi022 ceph-mon[31360]: pgmap v4450: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:50] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:26:50.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:50.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:50.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/267676890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/267676890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/267676890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:26:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:26:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:51 smithi122 ceph-mon[32255]: pgmap v4451: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:51 smithi022 ceph-mon[31360]: pgmap v4451: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:51 smithi022 ceph-mon[40250]: pgmap v4451: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:26:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:26:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:26:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:53 smithi122 ceph-mon[32255]: pgmap v4452: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:53 smithi022 ceph-mon[31360]: pgmap v4452: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:53 smithi022 ceph-mon[40250]: pgmap v4452: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:55.553 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:26:55.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:26:55.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:26:55.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:26:55.924 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:55 smithi022 ceph-mon[31360]: pgmap v4453: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:55.925 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:55 smithi022 ceph-mon[40250]: pgmap v4453: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:55 smithi122 ceph-mon[32255]: pgmap v4453: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:26:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2934810339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2934810339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2934810339' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:26:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:26:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:57 smithi122 ceph-mon[32255]: pgmap v4454: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:57 smithi022 ceph-mon[31360]: pgmap v4454: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:26:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:57 smithi022 ceph-mon[40250]: pgmap v4454: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:26:59 smithi122 ceph-mon[32255]: pgmap v4455: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:26:59 smithi022 ceph-mon[31360]: pgmap v4455: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:26:59 smithi022 ceph-mon[40250]: pgmap v4455: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:00] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:27:00.925 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:27:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:27:01.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:01.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:01.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:01 smithi122 ceph-mon[32255]: pgmap v4456: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3810964583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:01 smithi022 ceph-mon[31360]: pgmap v4456: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3810964583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:01 smithi022 ceph-mon[40250]: pgmap v4456: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3810964583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:27:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:27:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:03 smithi122 ceph-mon[32255]: pgmap v4457: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:03 smithi022 ceph-mon[31360]: pgmap v4457: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:03 smithi022 ceph-mon[40250]: pgmap v4457: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:05 smithi122 ceph-mon[32255]: pgmap v4458: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:05 smithi022 ceph-mon[31360]: pgmap v4458: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:05 smithi022 ceph-mon[40250]: pgmap v4458: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:06.294 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:06.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:06.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:06.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1986312138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1986312138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1986312138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:27:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:07 smithi122 ceph-mon[32255]: pgmap v4459: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:07 smithi022 ceph-mon[31360]: pgmap v4459: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:07 smithi022 ceph-mon[40250]: pgmap v4459: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:09 smithi122 ceph-mon[32255]: pgmap v4460: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:09 smithi022 ceph-mon[31360]: pgmap v4460: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:09 smithi022 ceph-mon[40250]: pgmap v4460: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:10] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:27:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:27:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:27:11.667 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:11 smithi022 ceph-mon[40250]: pgmap v4461: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:11 smithi022 ceph-mon[31360]: pgmap v4461: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:12.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:12.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:12.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:11 smithi122 ceph-mon[32255]: pgmap v4461: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:27:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:27:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2559953028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2559953028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2559953028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:13 smithi122 ceph-mon[32255]: pgmap v4462: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:13 smithi022 ceph-mon[31360]: pgmap v4462: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:13 smithi022 ceph-mon[40250]: pgmap v4462: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:15 smithi122 ceph-mon[32255]: pgmap v4463: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:27:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:27:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:27:15.969 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[40250]: pgmap v4463: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:15.970 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:27:15.971 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:27:15.971 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:27:15.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[31360]: pgmap v4463: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:15.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:27:15.973 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:27:15.973 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:27:17.042 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:17.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:17.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:27:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:27:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:17 smithi022 ceph-mon[31360]: pgmap v4464: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1363226556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:17 smithi022 ceph-mon[40250]: pgmap v4464: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1363226556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:17 smithi122 ceph-mon[32255]: pgmap v4464: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1363226556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:19 smithi022 ceph-mon[31360]: pgmap v4465: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:19 smithi022 ceph-mon[40250]: pgmap v4465: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:20] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:27:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:19 smithi122 ceph-mon[32255]: pgmap v4465: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:21.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:27:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:27:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:21 smithi022 ceph-mon[31360]: pgmap v4466: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:21 smithi022 ceph-mon[40250]: pgmap v4466: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:21 smithi122 ceph-mon[32255]: pgmap v4466: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:22.412 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:27:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:27:22.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:22.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:22.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:23 smithi122 ceph-mon[32255]: pgmap v4467: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2721275038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:23 smithi022 ceph-mon[31360]: pgmap v4467: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2721275038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:23 smithi022 ceph-mon[40250]: pgmap v4467: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2721275038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:25 smithi122 ceph-mon[32255]: pgmap v4468: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:25 smithi022 ceph-mon[31360]: pgmap v4468: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:25 smithi022 ceph-mon[40250]: pgmap v4468: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:27:27.780 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:28.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:28.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:28.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:28.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:27 smithi022 ceph-mon[31360]: pgmap v4469: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:28.152 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:27 smithi022 ceph-mon[40250]: pgmap v4469: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:27 smithi122 ceph-mon[32255]: pgmap v4469: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1524520236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1524520236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1524520236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:29 smithi122 ceph-mon[32255]: pgmap v4470: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:29 smithi022 ceph-mon[31360]: pgmap v4470: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:29 smithi022 ceph-mon[40250]: pgmap v4470: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:30] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:27:31.190 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:27:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:27:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:31 smithi122 ceph-mon[32255]: pgmap v4471: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:31 smithi022 ceph-mon[31360]: pgmap v4471: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:31 smithi022 ceph-mon[40250]: pgmap v4471: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:27:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:27:33.153 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:33.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:33.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:33.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:33.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:27:33.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:27:33.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:27:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:27:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[40250]: pgmap v4472: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:27:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:27:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:27:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:27:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:33 smithi022 ceph-mon[31360]: pgmap v4472: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:27:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:27:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:27:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:27:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:33 smithi122 ceph-mon[32255]: pgmap v4472: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2900342558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2900342558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2900342558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:35 smithi122 ceph-mon[32255]: pgmap v4473: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:35 smithi022 ceph-mon[31360]: pgmap v4473: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:35 smithi022 ceph-mon[40250]: pgmap v4473: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:27:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:37 smithi122 ceph-mon[32255]: pgmap v4474: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:37 smithi022 ceph-mon[31360]: pgmap v4474: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:37 smithi022 ceph-mon[40250]: pgmap v4474: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:38.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:38.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:38.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:38.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:39 smithi122 ceph-mon[32255]: pgmap v4475: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1150817310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:39 smithi022 ceph-mon[31360]: pgmap v4475: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1150817310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:39 smithi022 ceph-mon[40250]: pgmap v4475: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1150817310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:40] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:27:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:27:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:27:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:41 smithi122 ceph-mon[32255]: pgmap v4476: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:41 smithi022 ceph-mon[31360]: pgmap v4476: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:41 smithi022 ceph-mon[40250]: pgmap v4476: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:27:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:27:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:43 smithi122 ceph-mon[32255]: pgmap v4477: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:43 smithi022 ceph-mon[31360]: pgmap v4477: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:43 smithi022 ceph-mon[40250]: pgmap v4477: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:43.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:44.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:44.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:44.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/730380482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/730380482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/730380482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:45 smithi122 ceph-mon[32255]: pgmap v4478: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:45 smithi022 ceph-mon[31360]: pgmap v4478: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:45 smithi022 ceph-mon[40250]: pgmap v4478: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:27:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:47 smithi122 ceph-mon[32255]: pgmap v4479: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:47 smithi022 ceph-mon[31360]: pgmap v4479: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:47 smithi022 ceph-mon[40250]: pgmap v4479: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:49.268 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:49.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:49.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:49.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:49 smithi022 ceph-mon[31360]: pgmap v4480: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3108850609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:49 smithi022 ceph-mon[40250]: pgmap v4480: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3108850609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:49 smithi122 ceph-mon[32255]: pgmap v4480: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:27:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3108850609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:50] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:27:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:27:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:27:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:51 smithi122 ceph-mon[32255]: pgmap v4481: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:51 smithi022 ceph-mon[31360]: pgmap v4481: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:51 smithi022 ceph-mon[40250]: pgmap v4481: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:27:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:27:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:27:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:53 smithi122 ceph-mon[32255]: pgmap v4482: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:53.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:53 smithi022 ceph-mon[31360]: pgmap v4482: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:53.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:53 smithi022 ceph-mon[40250]: pgmap v4482: 132 pgs: 132 active+clean; 582 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:27:54.637 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:27:55.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:27:55.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:27:55.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:27:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:55 smithi122 ceph-mon[32255]: pgmap v4483: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:27:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1016478856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:55 smithi022 ceph-mon[40250]: pgmap v4483: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:27:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1016478856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:55 smithi022 ceph-mon[31360]: pgmap v4483: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:27:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1016478856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:27:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:27:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:57 smithi122 ceph-mon[32255]: pgmap v4484: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:27:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:57 smithi022 ceph-mon[40250]: pgmap v4484: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:27:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:57 smithi022 ceph-mon[31360]: pgmap v4484: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:28:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:27:59 smithi122 ceph-mon[32255]: pgmap v4485: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:27:59 smithi022 ceph-mon[31360]: pgmap v4485: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:27:59 smithi022 ceph-mon[40250]: pgmap v4485: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:00.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:00.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:00.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:00] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:28:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:28:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:28:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1297057892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1297057892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1297057892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:01 smithi122 ceph-mon[32255]: pgmap v4486: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:01 smithi022 ceph-mon[31360]: pgmap v4486: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:01 smithi022 ceph-mon[40250]: pgmap v4486: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:28:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:28:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:03 smithi122 ceph-mon[32255]: pgmap v4487: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:03 smithi022 ceph-mon[31360]: pgmap v4487: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:03 smithi022 ceph-mon[40250]: pgmap v4487: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:05.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:05.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:05.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:05.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:05 smithi022 ceph-mon[31360]: pgmap v4488: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1980193332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:05 smithi022 ceph-mon[40250]: pgmap v4488: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1980193332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:05 smithi122 ceph-mon[32255]: pgmap v4488: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:28:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1980193332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:28:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:07 smithi122 ceph-mon[32255]: pgmap v4489: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:28:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:07 smithi022 ceph-mon[31360]: pgmap v4489: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:28:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:07 smithi022 ceph-mon[40250]: pgmap v4489: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:28:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:09 smithi122 ceph-mon[32255]: pgmap v4490: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:28:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:09 smithi022 ceph-mon[31360]: pgmap v4490: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:28:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:09 smithi022 ceph-mon[40250]: pgmap v4490: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:28:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:10] "GET /metrics HTTP/1.1" 200 202337 "" "Prometheus/2.18.1" 2021-12-15T20:28:10.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:11.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:28:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:28:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:11 smithi122 ceph-mon[32255]: pgmap v4491: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1258428401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:11 smithi022 ceph-mon[31360]: pgmap v4491: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1258428401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:11 smithi022 ceph-mon[40250]: pgmap v4491: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1258428401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:28:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:28:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:13 smithi122 ceph-mon[32255]: pgmap v4492: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:13 smithi022 ceph-mon[40250]: pgmap v4492: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:13 smithi022 ceph-mon[31360]: pgmap v4492: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:16.116 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:16.117 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:15 smithi022 ceph-mon[40250]: pgmap v4493: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:16.118 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:15 smithi022 ceph-mon[31360]: pgmap v4493: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:15 smithi122 ceph-mon[32255]: pgmap v4493: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:16.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:16.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:16.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:28:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:28:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:28:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/672884897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:28:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:28:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:28:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/672884897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:28:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:28:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:28:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/672884897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:28:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:17 smithi122 ceph-mon[32255]: pgmap v4494: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:28:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:17 smithi022 ceph-mon[31360]: pgmap v4494: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:28:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:17 smithi022 ceph-mon[40250]: pgmap v4494: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:28:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:19 smithi122 ceph-mon[32255]: pgmap v4495: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:19 smithi022 ceph-mon[31360]: pgmap v4495: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:19 smithi022 ceph-mon[40250]: pgmap v4495: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:28:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:28:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:28:21.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:21.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:21.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:21.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:21 smithi122 ceph-mon[32255]: pgmap v4496: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2301811865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:21 smithi022 ceph-mon[31360]: pgmap v4496: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2301811865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:21 smithi022 ceph-mon[40250]: pgmap v4496: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2301811865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:28:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:28:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:23 smithi122 ceph-mon[32255]: pgmap v4497: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:23 smithi022 ceph-mon[31360]: pgmap v4497: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:23 smithi022 ceph-mon[40250]: pgmap v4497: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:25 smithi122 ceph-mon[32255]: pgmap v4498: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:25 smithi022 ceph-mon[31360]: pgmap v4498: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:25 smithi022 ceph-mon[40250]: pgmap v4498: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:26.860 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:27.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:27.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:27.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:28:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:27 smithi122 ceph-mon[32255]: pgmap v4499: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2945798349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:27 smithi022 ceph-mon[31360]: pgmap v4499: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2945798349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:27 smithi022 ceph-mon[40250]: pgmap v4499: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2945798349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:29 smithi122 ceph-mon[32255]: pgmap v4500: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:29 smithi022 ceph-mon[31360]: pgmap v4500: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:29 smithi022 ceph-mon[40250]: pgmap v4500: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:28:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:28:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:28:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:31 smithi122 ceph-mon[32255]: pgmap v4501: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:32.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:32.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:31 smithi022 ceph-mon[31360]: pgmap v4501: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:32.232 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:32.233 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:31 smithi022 ceph-mon[40250]: pgmap v4501: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:32.233 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:32.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:32.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:32.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:28:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:28:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1970231740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:28:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:28:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:28:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:28:33.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:33 smithi122 ceph-mon[32255]: pgmap v4502: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1970231740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:28:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:28:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:28:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:28:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[40250]: pgmap v4502: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1970231740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:28:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:28:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:28:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:28:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:33 smithi022 ceph-mon[31360]: pgmap v4502: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:35 smithi122 ceph-mon[32255]: pgmap v4503: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:35 smithi022 ceph-mon[31360]: pgmap v4503: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:35 smithi022 ceph-mon[40250]: pgmap v4503: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:37.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:28:37.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:37.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:37.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:37 smithi122 ceph-mon[32255]: pgmap v4504: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:37 smithi022 ceph-mon[31360]: pgmap v4504: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:37 smithi022 ceph-mon[40250]: pgmap v4504: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1147006833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1147006833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1147006833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:39 smithi122 ceph-mon[32255]: pgmap v4505: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:39 smithi022 ceph-mon[31360]: pgmap v4505: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:39 smithi022 ceph-mon[40250]: pgmap v4505: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:40] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:28:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:28:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:28:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:41 smithi122 ceph-mon[32255]: pgmap v4506: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:41 smithi022 ceph-mon[31360]: pgmap v4506: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:41 smithi022 ceph-mon[40250]: pgmap v4506: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:28:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:28:42.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:43.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:43.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:43.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:43 smithi122 ceph-mon[32255]: pgmap v4507: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:43 smithi022 ceph-mon[31360]: pgmap v4507: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:43 smithi022 ceph-mon[40250]: pgmap v4507: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3418964250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3418964250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3418964250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:45 smithi122 ceph-mon[32255]: pgmap v4508: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:45 smithi022 ceph-mon[31360]: pgmap v4508: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:45 smithi022 ceph-mon[40250]: pgmap v4508: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:28:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:47 smithi122 ceph-mon[32255]: pgmap v4509: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:47 smithi022 ceph-mon[31360]: pgmap v4509: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:47 smithi022 ceph-mon[40250]: pgmap v4509: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:48.347 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:48.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4029644791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4029644791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4029644791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:49 smithi122 ceph-mon[32255]: pgmap v4510: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:49 smithi022 ceph-mon[31360]: pgmap v4510: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:49 smithi022 ceph-mon[40250]: pgmap v4510: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:28:51.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:28:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:28:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:51 smithi122 ceph-mon[32255]: pgmap v4511: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:51 smithi022 ceph-mon[31360]: pgmap v4511: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:51 smithi022 ceph-mon[40250]: pgmap v4511: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:28:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:28:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:28:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:53 smithi122 ceph-mon[32255]: pgmap v4512: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:53.718 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:53.718 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:53 smithi022 ceph-mon[40250]: pgmap v4512: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:53.719 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:53 smithi022 ceph-mon[31360]: pgmap v4512: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:54.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:54.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:54.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:28:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221524942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221524942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221524942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:28:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:55 smithi122 ceph-mon[32255]: pgmap v4513: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:55 smithi022 ceph-mon[31360]: pgmap v4513: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:55 smithi022 ceph-mon[40250]: pgmap v4513: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:28:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:28:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:57 smithi122 ceph-mon[32255]: pgmap v4514: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:57 smithi022 ceph-mon[31360]: pgmap v4514: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:57 smithi022 ceph-mon[40250]: pgmap v4514: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:28:59.089 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:28:59.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:28:59.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:28:59.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:00.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:59 smithi122 ceph-mon[32255]: pgmap v4515: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:00.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:28:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3372749790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:59 smithi022 ceph-mon[31360]: pgmap v4515: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:28:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3372749790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:59 smithi022 ceph-mon[40250]: pgmap v4515: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:28:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3372749790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:29:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:29:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:29:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:01 smithi122 ceph-mon[32255]: pgmap v4516: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:01 smithi022 ceph-mon[31360]: pgmap v4516: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:01 smithi022 ceph-mon[40250]: pgmap v4516: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:29:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:29:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:03 smithi122 ceph-mon[32255]: pgmap v4517: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:03 smithi022 ceph-mon[31360]: pgmap v4517: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:03 smithi022 ceph-mon[40250]: pgmap v4517: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:04.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:04.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:04.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:04.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2177125388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2177125388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2177125388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:05 smithi122 ceph-mon[32255]: pgmap v4518: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:05 smithi022 ceph-mon[31360]: pgmap v4518: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:05 smithi022 ceph-mon[40250]: pgmap v4518: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:29:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:07 smithi122 ceph-mon[32255]: pgmap v4519: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:07 smithi022 ceph-mon[31360]: pgmap v4519: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:07 smithi022 ceph-mon[40250]: pgmap v4519: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:09.826 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:09 smithi122 ceph-mon[32255]: pgmap v4520: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:10.197 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:09 smithi022 ceph-mon[40250]: pgmap v4520: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:10.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:09 smithi022 ceph-mon[31360]: pgmap v4520: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:10.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:10] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:29:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/375237714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:11.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:29:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:29:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/375237714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/375237714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:12.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:11 smithi122 ceph-mon[32255]: pgmap v4521: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:12.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:11 smithi022 ceph-mon[31360]: pgmap v4521: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:11 smithi022 ceph-mon[40250]: pgmap v4521: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:29:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:29:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:13 smithi122 ceph-mon[32255]: pgmap v4522: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:13.999 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:13 smithi022 ceph-mon[31360]: pgmap v4522: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:13.999 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:13 smithi022 ceph-mon[40250]: pgmap v4522: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:15.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:15.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:15.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:15 smithi122 ceph-mon[32255]: pgmap v4523: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1098127519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:15 smithi022 ceph-mon[31360]: pgmap v4523: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1098127519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:15 smithi022 ceph-mon[40250]: pgmap v4523: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1098127519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:29:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:17 smithi122 ceph-mon[32255]: pgmap v4524: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:29:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:29:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:29:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:29:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[31360]: pgmap v4524: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:29:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:29:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:29:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:29:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[40250]: pgmap v4524: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:29:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:29:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:29:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:29:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:19 smithi122 ceph-mon[32255]: pgmap v4525: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:19 smithi022 ceph-mon[31360]: pgmap v4525: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:19 smithi022 ceph-mon[40250]: pgmap v4525: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:20] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:29:20.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:20.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:20.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:20.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:29:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:29:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:21 smithi122 ceph-mon[32255]: pgmap v4526: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:22.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1832755479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:22.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:21 smithi022 ceph-mon[31360]: pgmap v4526: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1832755479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:21 smithi022 ceph-mon[40250]: pgmap v4526: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1832755479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:29:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:29:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:23 smithi122 ceph-mon[32255]: pgmap v4527: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:23 smithi022 ceph-mon[31360]: pgmap v4527: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:23 smithi022 ceph-mon[40250]: pgmap v4527: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:25.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:25 smithi122 ceph-mon[32255]: pgmap v4528: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:25 smithi022 ceph-mon[31360]: pgmap v4528: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:25 smithi022 ceph-mon[40250]: pgmap v4528: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:26.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:26.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:26.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/853409993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/853409993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/853409993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:29:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:27 smithi122 ceph-mon[32255]: pgmap v4529: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:27 smithi022 ceph-mon[31360]: pgmap v4529: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:27 smithi022 ceph-mon[40250]: pgmap v4529: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:29 smithi122 ceph-mon[32255]: pgmap v4530: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:29 smithi022 ceph-mon[31360]: pgmap v4530: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:29 smithi022 ceph-mon[40250]: pgmap v4530: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:30] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:29:31.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:29:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:29:31.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:31.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:31.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:31 smithi022 ceph-mon[31360]: pgmap v4531: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1827982174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:31 smithi022 ceph-mon[40250]: pgmap v4531: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1827982174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:31 smithi122 ceph-mon[32255]: pgmap v4531: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1827982174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:29:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:29:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:29:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:29:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:29:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:29:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:33 smithi122 ceph-mon[32255]: pgmap v4532: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:29:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:29:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:29:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:29:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[31360]: pgmap v4532: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:29:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:29:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:29:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:29:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:33 smithi022 ceph-mon[40250]: pgmap v4532: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:35 smithi122 ceph-mon[32255]: pgmap v4533: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:35 smithi022 ceph-mon[31360]: pgmap v4533: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:35 smithi022 ceph-mon[40250]: pgmap v4533: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:36.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:37.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:37.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:37.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:29:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:37 smithi122 ceph-mon[32255]: pgmap v4534: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1438338434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:37 smithi022 ceph-mon[31360]: pgmap v4534: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1438338434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:37 smithi022 ceph-mon[40250]: pgmap v4534: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1438338434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:39 smithi122 ceph-mon[32255]: pgmap v4535: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:39 smithi022 ceph-mon[31360]: pgmap v4535: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:39 smithi022 ceph-mon[40250]: pgmap v4535: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:40] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:29:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:29:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:29:42.043 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:41 smithi122 ceph-mon[32255]: pgmap v4536: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:41 smithi022 ceph-mon[31360]: pgmap v4536: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:41 smithi022 ceph-mon[40250]: pgmap v4536: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:42.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:42.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:42.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:29:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:29:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1911102669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:43 smithi122 ceph-mon[32255]: pgmap v4537: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1911102669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:43 smithi022 ceph-mon[31360]: pgmap v4537: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1911102669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:43 smithi022 ceph-mon[40250]: pgmap v4537: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:45 smithi122 ceph-mon[32255]: pgmap v4538: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:45 smithi022 ceph-mon[31360]: pgmap v4538: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:45 smithi022 ceph-mon[40250]: pgmap v4538: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:47.415 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:29:47.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:47.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:47.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:47 smithi122 ceph-mon[32255]: pgmap v4539: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2674408912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:47 smithi022 ceph-mon[31360]: pgmap v4539: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2674408912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:47 smithi022 ceph-mon[40250]: pgmap v4539: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2674408912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:49 smithi122 ceph-mon[32255]: pgmap v4540: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:49 smithi022 ceph-mon[31360]: pgmap v4540: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:49 smithi022 ceph-mon[40250]: pgmap v4540: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:50] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:29:51.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:29:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:29:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:51 smithi122 ceph-mon[32255]: pgmap v4541: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:51 smithi022 ceph-mon[31360]: pgmap v4541: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:52.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:51 smithi022 ceph-mon[40250]: pgmap v4541: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:29:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:29:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:29:52.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:53.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:53.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:53.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:53 smithi022 ceph-mon[31360]: pgmap v4542: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2391098449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:53 smithi022 ceph-mon[40250]: pgmap v4542: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2391098449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:53 smithi122 ceph-mon[32255]: pgmap v4542: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2391098449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:55 smithi122 ceph-mon[32255]: pgmap v4543: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:55 smithi022 ceph-mon[31360]: pgmap v4543: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:55 smithi022 ceph-mon[40250]: pgmap v4543: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:29:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:29:58.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:29:58.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:57 smithi022 ceph-mon[31360]: pgmap v4544: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:58.156 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:57 smithi022 ceph-mon[40250]: pgmap v4544: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:57 smithi122 ceph-mon[32255]: pgmap v4544: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:29:58.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:29:58.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:29:58.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:29:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3323005105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3323005105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:29:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3323005105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:29:59 smithi122 ceph-mon[32255]: pgmap v4545: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:29:59 smithi022 ceph-mon[31360]: pgmap v4545: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:29:59 smithi022 ceph-mon[40250]: pgmap v4545: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:00] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:30:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T20:30:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:30:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:30:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T20:30:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T20:30:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:01 smithi122 ceph-mon[32255]: pgmap v4546: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:01 smithi022 ceph-mon[31360]: pgmap v4546: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:01 smithi022 ceph-mon[40250]: pgmap v4546: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:30:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:30:03.525 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:03 smithi122 ceph-mon[32255]: pgmap v4547: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:03 smithi022 ceph-mon[31360]: pgmap v4547: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:03 smithi022 ceph-mon[40250]: pgmap v4547: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:03.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:03.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:03.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3459739875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3459739875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3459739875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:05 smithi122 ceph-mon[32255]: pgmap v4548: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:05 smithi022 ceph-mon[31360]: pgmap v4548: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:05 smithi022 ceph-mon[40250]: pgmap v4548: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:30:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:07 smithi122 ceph-mon[32255]: pgmap v4549: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:07 smithi022 ceph-mon[31360]: pgmap v4549: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:07 smithi022 ceph-mon[40250]: pgmap v4549: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:08.894 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:09.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:09.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:09.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:09 smithi122 ceph-mon[32255]: pgmap v4550: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2176489783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:09 smithi022 ceph-mon[31360]: pgmap v4550: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2176489783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:09 smithi022 ceph-mon[40250]: pgmap v4550: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2176489783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:10] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:30:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:30:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:30:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:11 smithi122 ceph-mon[32255]: pgmap v4551: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:11 smithi022 ceph-mon[31360]: pgmap v4551: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:11 smithi022 ceph-mon[40250]: pgmap v4551: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:30:12.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:30:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:13 smithi122 ceph-mon[32255]: pgmap v4552: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:13 smithi022 ceph-mon[31360]: pgmap v4552: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:13 smithi022 ceph-mon[40250]: pgmap v4552: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:14.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:14.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:14.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:14.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/946824757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/946824757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/946824757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:15 smithi122 ceph-mon[32255]: pgmap v4553: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:15 smithi022 ceph-mon[31360]: pgmap v4553: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:15 smithi022 ceph-mon[40250]: pgmap v4553: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:30:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:17 smithi122 ceph-mon[32255]: pgmap v4554: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:30:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:30:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:30:18.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:30:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[31360]: pgmap v4554: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:30:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:30:18.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:30:18.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:30:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[40250]: pgmap v4554: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:30:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:30:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:30:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:30:19.633 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:20.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:19 smithi022 ceph-mon[31360]: pgmap v4555: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:20.007 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:19 smithi022 ceph-mon[40250]: pgmap v4555: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:20.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:20.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:20.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:19 smithi122 ceph-mon[32255]: pgmap v4555: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:20] "GET /metrics HTTP/1.1" 200 202341 "" "Prometheus/2.18.1" 2021-12-15T20:30:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4231596320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:21.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:30:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:30:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4231596320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4231596320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:21 smithi122 ceph-mon[32255]: pgmap v4556: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:21 smithi022 ceph-mon[31360]: pgmap v4556: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:21 smithi022 ceph-mon[40250]: pgmap v4556: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:30:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:30:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:23 smithi122 ceph-mon[32255]: pgmap v4557: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:23 smithi022 ceph-mon[31360]: pgmap v4557: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:23 smithi022 ceph-mon[40250]: pgmap v4557: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:25.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:25.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:25.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:25.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:25 smithi122 ceph-mon[32255]: pgmap v4558: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/212704833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:25 smithi022 ceph-mon[31360]: pgmap v4558: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/212704833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:25 smithi022 ceph-mon[40250]: pgmap v4558: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/212704833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:30:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:27 smithi122 ceph-mon[32255]: pgmap v4559: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:27 smithi022 ceph-mon[31360]: pgmap v4559: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:27 smithi022 ceph-mon[40250]: pgmap v4559: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:29 smithi122 ceph-mon[32255]: pgmap v4560: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:29 smithi022 ceph-mon[31360]: pgmap v4560: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:29 smithi022 ceph-mon[40250]: pgmap v4560: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:30.378 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:30] "GET /metrics HTTP/1.1" 200 202341 "" "Prometheus/2.18.1" 2021-12-15T20:30:30.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:30.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:30.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1814148730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1814148730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:30:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:30:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1814148730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:31 smithi122 ceph-mon[32255]: pgmap v4561: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:32.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:31 smithi022 ceph-mon[31360]: pgmap v4561: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:32.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:31 smithi022 ceph-mon[40250]: pgmap v4561: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:30:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:30:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:30:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:30:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:30:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:30:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:33 smithi122 ceph-mon[32255]: pgmap v4562: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:30:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:30:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:30:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:30:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[31360]: pgmap v4562: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:30:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:30:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:30:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:30:33.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:33 smithi022 ceph-mon[40250]: pgmap v4562: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:35.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:35 smithi022 ceph-mon[31360]: pgmap v4563: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:35 smithi022 ceph-mon[40250]: pgmap v4563: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:36.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:36.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:36.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:35 smithi122 ceph-mon[32255]: pgmap v4563: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3908590412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3908590412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3908590412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:30:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:37 smithi122 ceph-mon[32255]: pgmap v4564: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:37 smithi022 ceph-mon[31360]: pgmap v4564: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:37 smithi022 ceph-mon[40250]: pgmap v4564: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:39 smithi122 ceph-mon[32255]: pgmap v4565: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:39 smithi022 ceph-mon[31360]: pgmap v4565: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:39 smithi022 ceph-mon[40250]: pgmap v4565: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:40] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:30:41.121 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:30:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:30:41.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:41.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:41.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:41 smithi122 ceph-mon[32255]: pgmap v4566: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/619319131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:41 smithi022 ceph-mon[31360]: pgmap v4566: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/619319131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:41 smithi022 ceph-mon[40250]: pgmap v4566: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/619319131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:30:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:30:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:43 smithi122 ceph-mon[32255]: pgmap v4567: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:43 smithi022 ceph-mon[40250]: pgmap v4567: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:43 smithi022 ceph-mon[31360]: pgmap v4567: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:45 smithi122 ceph-mon[32255]: pgmap v4568: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:45 smithi022 ceph-mon[31360]: pgmap v4568: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:45 smithi022 ceph-mon[40250]: pgmap v4568: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:46.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3621784228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3621784228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3621784228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:30:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:47 smithi122 ceph-mon[32255]: pgmap v4569: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:47 smithi022 ceph-mon[31360]: pgmap v4569: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:47 smithi022 ceph-mon[40250]: pgmap v4569: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:49 smithi122 ceph-mon[32255]: pgmap v4570: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:49 smithi022 ceph-mon[31360]: pgmap v4570: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:49 smithi022 ceph-mon[40250]: pgmap v4570: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:30:51.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:30:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:30:51.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:51 smithi122 ceph-mon[32255]: pgmap v4571: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:52.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:52.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:52.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:52.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:51 smithi022 ceph-mon[40250]: pgmap v4571: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:52.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:52.238 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:51 smithi022 ceph-mon[31360]: pgmap v4571: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:52.238 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:30:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:30:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:30:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3042752439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3042752439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3042752439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:53 smithi122 ceph-mon[32255]: pgmap v4572: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:53 smithi022 ceph-mon[31360]: pgmap v4572: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:53 smithi022 ceph-mon[40250]: pgmap v4572: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:55 smithi122 ceph-mon[32255]: pgmap v4573: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:55 smithi022 ceph-mon[31360]: pgmap v4573: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:55 smithi022 ceph-mon[40250]: pgmap v4573: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:30:57.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:30:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:30:57.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:30:57.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:30:57.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:30:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:57 smithi122 ceph-mon[32255]: pgmap v4574: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3590735347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:57 smithi022 ceph-mon[31360]: pgmap v4574: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3590735347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:30:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:57 smithi022 ceph-mon[40250]: pgmap v4574: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:30:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3590735347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:30:59 smithi122 ceph-mon[32255]: pgmap v4575: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:30:59 smithi022 ceph-mon[31360]: pgmap v4575: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:30:59 smithi022 ceph-mon[40250]: pgmap v4575: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:31:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:31:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:31:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:01 smithi122 ceph-mon[32255]: pgmap v4576: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:01 smithi022 ceph-mon[40250]: pgmap v4576: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:01 smithi022 ceph-mon[31360]: pgmap v4576: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:31:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:31:02.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:02.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:02.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:03 smithi122 ceph-mon[32255]: pgmap v4577: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2778490854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:03 smithi022 ceph-mon[31360]: pgmap v4577: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2778490854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:03 smithi022 ceph-mon[40250]: pgmap v4577: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2778490854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:05 smithi122 ceph-mon[32255]: pgmap v4578: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:05 smithi022 ceph-mon[31360]: pgmap v4578: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:05 smithi022 ceph-mon[40250]: pgmap v4578: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:31:07.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:07 smithi122 ceph-mon[32255]: pgmap v4579: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:07 smithi022 ceph-mon[31360]: pgmap v4579: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:07 smithi022 ceph-mon[40250]: pgmap v4579: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:08.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:08.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:08.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4055860965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4055860965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4055860965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:09 smithi122 ceph-mon[32255]: pgmap v4580: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:09 smithi022 ceph-mon[31360]: pgmap v4580: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:09 smithi022 ceph-mon[40250]: pgmap v4580: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:10] "GET /metrics HTTP/1.1" 200 202337 "" "Prometheus/2.18.1" 2021-12-15T20:31:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:31:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:31:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:11 smithi122 ceph-mon[32255]: pgmap v4581: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:11 smithi022 ceph-mon[31360]: pgmap v4581: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:11 smithi022 ceph-mon[40250]: pgmap v4581: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:31:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:31:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:13.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:13.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:13.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:13.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:13 smithi022 ceph-mon[31360]: pgmap v4582: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2006894140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:13 smithi022 ceph-mon[40250]: pgmap v4582: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2006894140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:13 smithi122 ceph-mon[32255]: pgmap v4582: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2006894140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:15 smithi122 ceph-mon[32255]: pgmap v4583: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:15 smithi022 ceph-mon[31360]: pgmap v4583: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:15 smithi022 ceph-mon[40250]: pgmap v4583: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:31:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:17 smithi122 ceph-mon[32255]: pgmap v4584: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:31:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:31:18.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:31:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[31360]: pgmap v4584: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:31:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:31:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:31:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[40250]: pgmap v4584: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:31:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:31:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:31:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:31:18.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:19.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:19.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:19.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:19 smithi122 ceph-mon[32255]: pgmap v4585: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/436719032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:19 smithi022 ceph-mon[31360]: pgmap v4585: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/436719032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:19 smithi022 ceph-mon[40250]: pgmap v4585: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/436719032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:31:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:31:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:31:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:21 smithi022 ceph-mon[40250]: pgmap v4586: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:21 smithi022 ceph-mon[31360]: pgmap v4586: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:21 smithi122 ceph-mon[32255]: pgmap v4586: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:31:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:31:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:24.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:24.370 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:24 smithi022 ceph-mon[31360]: pgmap v4587: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:24.370 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:24.371 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:24 smithi022 ceph-mon[40250]: pgmap v4587: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:24.371 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:24 smithi122 ceph-mon[32255]: pgmap v4587: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:24.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:24.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:24.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/73355774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/73355774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/73355774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:26 smithi122 ceph-mon[32255]: pgmap v4588: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:26 smithi022 ceph-mon[31360]: pgmap v4588: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:26 smithi022 ceph-mon[40250]: pgmap v4588: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:31:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:27 smithi122 ceph-mon[32255]: pgmap v4589: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:27 smithi022 ceph-mon[31360]: pgmap v4589: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:31:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:27 smithi022 ceph-mon[40250]: pgmap v4589: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:29.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:29.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:29.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:29.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:30.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:29 smithi122 ceph-mon[32255]: pgmap v4590: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:30.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/116071681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:29 smithi022 ceph-mon[31360]: pgmap v4590: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/116071681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:29 smithi022 ceph-mon[40250]: pgmap v4590: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/116071681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:31:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:31:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:31:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:31 smithi122 ceph-mon[32255]: pgmap v4591: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:31 smithi022 ceph-mon[31360]: pgmap v4591: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:31 smithi022 ceph-mon[40250]: pgmap v4591: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:31:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:31:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:31:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:31:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:31:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:31:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:31:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:31:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:31:33.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:31:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:33.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:31:33.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:31:33.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:31:33.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:31:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:33 smithi122 ceph-mon[32255]: pgmap v4592: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:33 smithi022 ceph-mon[31360]: pgmap v4592: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:33 smithi022 ceph-mon[40250]: pgmap v4592: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:34.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:35.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:35.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:35.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:35 smithi122 ceph-mon[32255]: pgmap v4593: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/613812021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:35 smithi022 ceph-mon[31360]: pgmap v4593: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/613812021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:35 smithi022 ceph-mon[40250]: pgmap v4593: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/613812021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:31:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:37 smithi122 ceph-mon[32255]: pgmap v4594: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:37 smithi022 ceph-mon[31360]: pgmap v4594: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:37 smithi022 ceph-mon[40250]: pgmap v4594: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:39 smithi122 ceph-mon[32255]: pgmap v4595: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:40.199 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:40.200 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:39 smithi022 ceph-mon[40250]: pgmap v4595: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:40.200 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:39 smithi022 ceph-mon[31360]: pgmap v4595: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:40] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:31:40.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:40.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:40.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:40.890 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3007091320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:40.891 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3007091320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3007091320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:41.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:31:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:31:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:41 smithi122 ceph-mon[32255]: pgmap v4596: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:41 smithi022 ceph-mon[31360]: pgmap v4596: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:41 smithi022 ceph-mon[40250]: pgmap v4596: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:31:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:31:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:43 smithi122 ceph-mon[32255]: pgmap v4597: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:43 smithi022 ceph-mon[31360]: pgmap v4597: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:43 smithi022 ceph-mon[40250]: pgmap v4597: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:45.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:45.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:45.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:45.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:45 smithi122 ceph-mon[32255]: pgmap v4598: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:45 smithi022 ceph-mon[31360]: pgmap v4598: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:45 smithi022 ceph-mon[40250]: pgmap v4598: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/876326554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/876326554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/876326554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:31:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:47 smithi122 ceph-mon[32255]: pgmap v4599: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:47 smithi022 ceph-mon[31360]: pgmap v4599: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:47 smithi022 ceph-mon[40250]: pgmap v4599: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:49 smithi122 ceph-mon[32255]: pgmap v4600: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:49 smithi022 ceph-mon[31360]: pgmap v4600: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:49 smithi022 ceph-mon[40250]: pgmap v4600: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:31:50.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:51.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:51.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:51.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:51.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:31:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:31:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:51 smithi122 ceph-mon[32255]: pgmap v4601: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1850128583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:51 smithi022 ceph-mon[31360]: pgmap v4601: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1850128583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:51 smithi022 ceph-mon[40250]: pgmap v4601: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1850128583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:31:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:31:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:31:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:53 smithi122 ceph-mon[32255]: pgmap v4602: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:53 smithi022 ceph-mon[31360]: pgmap v4602: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:53 smithi022 ceph-mon[40250]: pgmap v4602: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:55 smithi122 ceph-mon[32255]: pgmap v4603: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:55 smithi022 ceph-mon[31360]: pgmap v4603: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:55 smithi022 ceph-mon[40250]: pgmap v4603: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:31:56.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:31:56.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:31:56.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:31:56.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:31:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3020363498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3020363498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3020363498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:31:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:31:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:57 smithi122 ceph-mon[32255]: pgmap v4604: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:57 smithi022 ceph-mon[31360]: pgmap v4604: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:31:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:57 smithi022 ceph-mon[40250]: pgmap v4604: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:31:59 smithi122 ceph-mon[32255]: pgmap v4605: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:31:59 smithi022 ceph-mon[31360]: pgmap v4605: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:31:59 smithi022 ceph-mon[40250]: pgmap v4605: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:32:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:32:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:32:01.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:01 smithi022 ceph-mon[31360]: pgmap v4606: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:01 smithi022 ceph-mon[40250]: pgmap v4606: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:01 smithi122 ceph-mon[32255]: pgmap v4606: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:32:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:32:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/758507468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/758507468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/758507468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:03 smithi122 ceph-mon[32255]: pgmap v4607: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:03 smithi022 ceph-mon[31360]: pgmap v4607: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:03 smithi022 ceph-mon[40250]: pgmap v4607: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:05 smithi122 ceph-mon[32255]: pgmap v4608: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:05 smithi022 ceph-mon[31360]: pgmap v4608: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:05 smithi022 ceph-mon[40250]: pgmap v4608: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:07.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:07.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:07.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:07.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:32:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:07 smithi122 ceph-mon[32255]: pgmap v4609: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2608079929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:07 smithi022 ceph-mon[31360]: pgmap v4609: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2608079929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:07 smithi022 ceph-mon[40250]: pgmap v4609: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2608079929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:09 smithi122 ceph-mon[32255]: pgmap v4610: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:10.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:10] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:32:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:09 smithi022 ceph-mon[31360]: pgmap v4610: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:09 smithi022 ceph-mon[40250]: pgmap v4610: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:32:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:32:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:11 smithi022 ceph-mon[31360]: pgmap v4611: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:11 smithi022 ceph-mon[40250]: pgmap v4611: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:11 smithi122 ceph-mon[32255]: pgmap v4611: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:12.426 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:32:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:32:12.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:12.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:12.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2825955836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2825955836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2825955836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:13 smithi022 ceph-mon[31360]: pgmap v4612: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:13 smithi022 ceph-mon[40250]: pgmap v4612: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:13 smithi122 ceph-mon[32255]: pgmap v4612: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:15 smithi022 ceph-mon[31360]: pgmap v4613: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:15 smithi022 ceph-mon[40250]: pgmap v4613: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:15 smithi122 ceph-mon[32255]: pgmap v4613: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:32:17.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:18.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:18.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:18.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:18.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:17 smithi022 ceph-mon[31360]: pgmap v4614: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:18.172 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:17 smithi022 ceph-mon[40250]: pgmap v4614: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:17 smithi122 ceph-mon[32255]: pgmap v4614: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3476519498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3476519498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3476519498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:19 smithi022 ceph-mon[31360]: pgmap v4615: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:19 smithi022 ceph-mon[40250]: pgmap v4615: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:32:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:19 smithi122 ceph-mon[32255]: pgmap v4615: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:32:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:32:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:21 smithi022 ceph-mon[31360]: pgmap v4616: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:21 smithi022 ceph-mon[40250]: pgmap v4616: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:21 smithi122 ceph-mon[32255]: pgmap v4616: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:32:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:32:23.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:23.356 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:23.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:23.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:23.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:23 smithi022 ceph-mon[31360]: pgmap v4617: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3935125676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:23 smithi022 ceph-mon[40250]: pgmap v4617: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3935125676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:23 smithi122 ceph-mon[32255]: pgmap v4617: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3935125676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:25 smithi122 ceph-mon[32255]: pgmap v4618: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:32:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:32:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:32:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[40250]: pgmap v4618: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:32:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:32:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:32:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[31360]: pgmap v4618: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:32:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:32:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:32:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:32:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:27 smithi122 ceph-mon[32255]: pgmap v4619: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:32:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:27 smithi022 ceph-mon[31360]: pgmap v4619: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:32:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:32:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:27 smithi022 ceph-mon[40250]: pgmap v4619: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:28.544 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:28.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:28.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:28.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:29 smithi122 ceph-mon[32255]: pgmap v4620: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2746415095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:29 smithi022 ceph-mon[31360]: pgmap v4620: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2746415095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:29 smithi022 ceph-mon[40250]: pgmap v4620: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2746415095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:32:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:32:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:32:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:31 smithi122 ceph-mon[32255]: pgmap v4621: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:31 smithi022 ceph-mon[31360]: pgmap v4621: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:31 smithi022 ceph-mon[40250]: pgmap v4621: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:32:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:32:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:32:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:32:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:32:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:32:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:32:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:32:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:32:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:32:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:32:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:32:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:32:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:32:33.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:33 smithi122 ceph-mon[32255]: pgmap v4622: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:33 smithi022 ceph-mon[31360]: pgmap v4622: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:33 smithi022 ceph-mon[40250]: pgmap v4622: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:34.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:34.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:34.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/656109805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/656109805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/656109805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:35 smithi122 ceph-mon[32255]: pgmap v4623: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:35 smithi022 ceph-mon[31360]: pgmap v4623: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:35 smithi022 ceph-mon[40250]: pgmap v4623: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:32:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:37 smithi122 ceph-mon[32255]: pgmap v4624: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:37 smithi022 ceph-mon[31360]: pgmap v4624: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:37 smithi022 ceph-mon[40250]: pgmap v4624: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:39.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:39.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:39.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:39.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:39 smithi122 ceph-mon[32255]: pgmap v4625: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3198866711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:39 smithi022 ceph-mon[31360]: pgmap v4625: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3198866711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:39 smithi022 ceph-mon[40250]: pgmap v4625: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3198866711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:40] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:32:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:32:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:32:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:41 smithi122 ceph-mon[32255]: pgmap v4626: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:41 smithi022 ceph-mon[31360]: pgmap v4626: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:41 smithi022 ceph-mon[40250]: pgmap v4626: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:32:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:32:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:43 smithi122 ceph-mon[32255]: pgmap v4627: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:43 smithi022 ceph-mon[31360]: pgmap v4627: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:43 smithi022 ceph-mon[40250]: pgmap v4627: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:45.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:45.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:45.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:45 smithi122 ceph-mon[32255]: pgmap v4628: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3058830872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:45 smithi022 ceph-mon[31360]: pgmap v4628: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3058830872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:45 smithi022 ceph-mon[40250]: pgmap v4628: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3058830872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:32:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:47 smithi122 ceph-mon[32255]: pgmap v4629: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:47 smithi022 ceph-mon[31360]: pgmap v4629: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:47 smithi022 ceph-mon[40250]: pgmap v4629: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:50.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:49 smithi122 ceph-mon[32255]: pgmap v4630: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:49 smithi022 ceph-mon[31360]: pgmap v4630: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:49 smithi022 ceph-mon[40250]: pgmap v4630: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:50.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:50.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:50.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:32:51.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2711663954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2711663954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2711663954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:51.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:32:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:32:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:51 smithi122 ceph-mon[32255]: pgmap v4631: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:51 smithi022 ceph-mon[31360]: pgmap v4631: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:51 smithi022 ceph-mon[40250]: pgmap v4631: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:32:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:32:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:32:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:53 smithi122 ceph-mon[32255]: pgmap v4632: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:53 smithi022 ceph-mon[31360]: pgmap v4632: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:53 smithi022 ceph-mon[40250]: pgmap v4632: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:55.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:32:55.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:32:55.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:32:55.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:32:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:55 smithi022 ceph-mon[31360]: pgmap v4633: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2618757818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:55 smithi022 ceph-mon[40250]: pgmap v4633: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2618757818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:55 smithi122 ceph-mon[32255]: pgmap v4633: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:32:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2618757818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:32:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:32:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:57 smithi122 ceph-mon[32255]: pgmap v4634: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:57 smithi022 ceph-mon[31360]: pgmap v4634: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:32:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:57 smithi022 ceph-mon[40250]: pgmap v4634: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:00.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:33:00.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:32:59 smithi122 ceph-mon[32255]: pgmap v4635: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:32:59 smithi022 ceph-mon[31360]: pgmap v4635: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:32:59 smithi022 ceph-mon[40250]: pgmap v4635: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:00.765 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:01.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:01.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:01.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:33:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:33:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:01 smithi022 ceph-mon[31360]: pgmap v4636: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2907080471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:01 smithi022 ceph-mon[40250]: pgmap v4636: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2907080471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:01 smithi122 ceph-mon[32255]: pgmap v4636: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2907080471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:33:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:33:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:04.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:03 smithi022 ceph-mon[31360]: pgmap v4637: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:04.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:03 smithi022 ceph-mon[40250]: pgmap v4637: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:03 smithi122 ceph-mon[32255]: pgmap v4637: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:06.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:05 smithi022 ceph-mon[31360]: pgmap v4638: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:05 smithi022 ceph-mon[40250]: pgmap v4638: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:05 smithi122 ceph-mon[32255]: pgmap v4638: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:06.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:06.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:06.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2970775239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2970775239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2970775239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:33:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:07 smithi022 ceph-mon[31360]: pgmap v4639: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:07 smithi022 ceph-mon[40250]: pgmap v4639: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:07 smithi122 ceph-mon[32255]: pgmap v4639: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:09 smithi022 ceph-mon[31360]: pgmap v4640: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:09 smithi022 ceph-mon[40250]: pgmap v4640: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:10] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:33:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:09 smithi122 ceph-mon[32255]: pgmap v4640: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:11.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:11.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:33:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:33:11.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:11.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:11.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:11 smithi022 ceph-mon[31360]: pgmap v4641: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/466464142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:11 smithi022 ceph-mon[40250]: pgmap v4641: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/466464142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:11 smithi122 ceph-mon[32255]: pgmap v4641: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/466464142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:33:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:33:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:13 smithi022 ceph-mon[31360]: pgmap v4642: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:13 smithi022 ceph-mon[40250]: pgmap v4642: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:13 smithi122 ceph-mon[32255]: pgmap v4642: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:15 smithi022 ceph-mon[31360]: pgmap v4643: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:15 smithi022 ceph-mon[40250]: pgmap v4643: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:15 smithi122 ceph-mon[32255]: pgmap v4643: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:16.878 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:17.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:17.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:17.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:33:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:17 smithi022 ceph-mon[31360]: pgmap v4644: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/220581711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:17 smithi022 ceph-mon[40250]: pgmap v4644: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/220581711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:17 smithi122 ceph-mon[32255]: pgmap v4644: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/220581711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:19 smithi022 ceph-mon[31360]: pgmap v4645: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:19 smithi022 ceph-mon[40250]: pgmap v4645: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:20] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:33:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:19 smithi122 ceph-mon[32255]: pgmap v4645: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:21.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:33:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:33:22.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:22.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:21 smithi022 ceph-mon[31360]: pgmap v4646: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:22.251 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:21 smithi022 ceph-mon[40250]: pgmap v4646: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:21 smithi122 ceph-mon[32255]: pgmap v4646: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:22.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:22.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:22.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:33:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:33:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2362897673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2362897673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2362897673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:23 smithi022 ceph-mon[31360]: pgmap v4647: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:23 smithi022 ceph-mon[40250]: pgmap v4647: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:23 smithi122 ceph-mon[32255]: pgmap v4647: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:25 smithi022 ceph-mon[31360]: pgmap v4648: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:25 smithi022 ceph-mon[40250]: pgmap v4648: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:26.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:25 smithi122 ceph-mon[32255]: pgmap v4648: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:33:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:33:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:33:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:33:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:33:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:33:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:33:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:33:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:33:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:33:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:33:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:33:27.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:33:27.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:27.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:27.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:27 smithi022 ceph-mon[31360]: pgmap v4649: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:27 smithi022 ceph-mon[40250]: pgmap v4649: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:27 smithi122 ceph-mon[32255]: pgmap v4649: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2624352998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2624352998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2624352998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:29 smithi022 ceph-mon[31360]: pgmap v4650: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:29 smithi022 ceph-mon[40250]: pgmap v4650: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:29 smithi122 ceph-mon[32255]: pgmap v4650: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:30] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:33:31.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:33:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:33:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:31 smithi022 ceph-mon[31360]: pgmap v4651: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:31 smithi022 ceph-mon[40250]: pgmap v4651: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:31 smithi122 ceph-mon[32255]: pgmap v4651: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:33:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:33:32.996 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:33:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:33:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:33:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:33:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:33:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:33:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:33:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:33:33.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:33.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:33.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:33:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:33:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:33:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:33:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:33 smithi022 ceph-mon[40250]: pgmap v4652: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3050530307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:33 smithi022 ceph-mon[31360]: pgmap v4652: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3050530307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:33 smithi122 ceph-mon[32255]: pgmap v4652: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3050530307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:35 smithi022 ceph-mon[31360]: pgmap v4653: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:35 smithi022 ceph-mon[40250]: pgmap v4653: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:35 smithi122 ceph-mon[32255]: pgmap v4653: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:33:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:37 smithi022 ceph-mon[31360]: pgmap v4654: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:37 smithi022 ceph-mon[40250]: pgmap v4654: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:38.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:37 smithi122 ceph-mon[32255]: pgmap v4654: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:38.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:38.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:38.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3352899486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3352899486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3352899486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:40 smithi022 ceph-mon[31360]: pgmap v4655: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:40 smithi022 ceph-mon[40250]: pgmap v4655: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:40 smithi122 ceph-mon[32255]: pgmap v4655: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:40] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:33:41.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:33:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:33:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:42 smithi022 ceph-mon[31360]: pgmap v4656: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:42 smithi022 ceph-mon[40250]: pgmap v4656: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:42 smithi122 ceph-mon[32255]: pgmap v4656: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:33:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:33:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:43.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:44.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:44.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:44.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:44 smithi022 ceph-mon[31360]: pgmap v4657: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:44 smithi022 ceph-mon[40250]: pgmap v4657: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:44 smithi122 ceph-mon[32255]: pgmap v4657: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3547947621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3547947621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3547947621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:46 smithi022 ceph-mon[31360]: pgmap v4658: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:46 smithi022 ceph-mon[40250]: pgmap v4658: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:46 smithi122 ceph-mon[32255]: pgmap v4658: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:33:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:48 smithi022 ceph-mon[31360]: pgmap v4659: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:48 smithi022 ceph-mon[40250]: pgmap v4659: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:48 smithi122 ceph-mon[32255]: pgmap v4659: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:49.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:49.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:49.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:50 smithi022 ceph-mon[31360]: pgmap v4660: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2990686787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:50 smithi022 ceph-mon[40250]: pgmap v4660: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2990686787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:50] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:33:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:50 smithi122 ceph-mon[32255]: pgmap v4660: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2990686787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:51.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:33:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:33:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:52 smithi022 ceph-mon[31360]: pgmap v4661: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:52 smithi022 ceph-mon[40250]: pgmap v4661: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:52 smithi122 ceph-mon[32255]: pgmap v4661: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:33:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:33:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:33:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:54 smithi022 ceph-mon[31360]: pgmap v4662: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:54 smithi022 ceph-mon[40250]: pgmap v4662: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:54 smithi122 ceph-mon[32255]: pgmap v4662: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:54.483 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:33:54.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:33:54.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:33:54.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:33:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1085655538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1085655538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1085655538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:33:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:56 smithi022 ceph-mon[31360]: pgmap v4663: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:56 smithi022 ceph-mon[40250]: pgmap v4663: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:56 smithi122 ceph-mon[32255]: pgmap v4663: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:33:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:33:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:33:58 smithi122 ceph-mon[32255]: pgmap v4664: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:33:58 smithi022 ceph-mon[31360]: pgmap v4664: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:33:58 smithi022 ceph-mon[40250]: pgmap v4664: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:33:59.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:00.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:00.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:00.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:00.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:00 smithi022 ceph-mon[31360]: pgmap v4665: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:00.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:00 smithi022 ceph-mon[40250]: pgmap v4665: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:00] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:34:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:00 smithi122 ceph-mon[32255]: pgmap v4665: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1984219176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:34:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:34:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1984219176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1984219176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:02 smithi122 ceph-mon[32255]: pgmap v4666: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:02 smithi022 ceph-mon[31360]: pgmap v4666: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:02 smithi022 ceph-mon[40250]: pgmap v4666: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:34:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:34:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:03.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:04 smithi122 ceph-mon[32255]: pgmap v4667: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:04 smithi022 ceph-mon[31360]: pgmap v4667: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:04 smithi022 ceph-mon[40250]: pgmap v4667: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:05.228 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:05.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:05.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:05.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:06 smithi122 ceph-mon[32255]: pgmap v4668: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1239733989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:06 smithi022 ceph-mon[31360]: pgmap v4668: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1239733989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:06 smithi022 ceph-mon[40250]: pgmap v4668: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1239733989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:34:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:08 smithi122 ceph-mon[32255]: pgmap v4669: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:08 smithi022 ceph-mon[31360]: pgmap v4669: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:08 smithi022 ceph-mon[40250]: pgmap v4669: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:10] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:34:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:10 smithi122 ceph-mon[32255]: pgmap v4670: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:10 smithi022 ceph-mon[31360]: pgmap v4670: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:10 smithi022 ceph-mon[40250]: pgmap v4670: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:10.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:10.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:10.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:10.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3007178862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3007178862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3007178862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:11.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:34:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:34:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:12 smithi122 ceph-mon[32255]: pgmap v4671: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:12 smithi022 ceph-mon[31360]: pgmap v4671: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:12 smithi022 ceph-mon[40250]: pgmap v4671: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:34:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:34:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:14 smithi122 ceph-mon[32255]: pgmap v4672: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:14 smithi022 ceph-mon[31360]: pgmap v4672: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:14 smithi022 ceph-mon[40250]: pgmap v4672: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:15.973 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:16 smithi022 ceph-mon[31360]: pgmap v4673: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:16 smithi022 ceph-mon[40250]: pgmap v4673: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:16.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:16.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:16.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:16 smithi122 ceph-mon[32255]: pgmap v4673: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2930148279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2930148279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2930148279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:34:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:18 smithi122 ceph-mon[32255]: pgmap v4674: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:18 smithi022 ceph-mon[31360]: pgmap v4674: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:18 smithi022 ceph-mon[40250]: pgmap v4674: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:34:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:20 smithi122 ceph-mon[32255]: pgmap v4675: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:20 smithi022 ceph-mon[31360]: pgmap v4675: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:20 smithi022 ceph-mon[40250]: pgmap v4675: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:21.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:21.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:34:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:34:21.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:21.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:21.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:22 smithi122 ceph-mon[32255]: pgmap v4676: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3966492186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:34:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:34:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:22 smithi022 ceph-mon[31360]: pgmap v4676: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3966492186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:22 smithi022 ceph-mon[40250]: pgmap v4676: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3966492186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:24 smithi122 ceph-mon[32255]: pgmap v4677: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:24 smithi022 ceph-mon[31360]: pgmap v4677: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:24 smithi022 ceph-mon[40250]: pgmap v4677: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:26 smithi122 ceph-mon[32255]: pgmap v4678: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:26 smithi022 ceph-mon[31360]: pgmap v4678: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:26 smithi022 ceph-mon[40250]: pgmap v4678: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:26.711 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:27.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:27.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:27.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:34:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:34:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:34:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2878007311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:34:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:34:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:34:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2878007311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:34:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:34:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:34:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2878007311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:34:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:28 smithi122 ceph-mon[32255]: pgmap v4679: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:34:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:28 smithi022 ceph-mon[31360]: pgmap v4679: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:34:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:28 smithi022 ceph-mon[40250]: pgmap v4679: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:34:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:34:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:30 smithi122 ceph-mon[32255]: pgmap v4680: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:30 smithi022 ceph-mon[31360]: pgmap v4680: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:30 smithi022 ceph-mon[40250]: pgmap v4680: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:31.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:34:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:34:32.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:32 smithi122 ceph-mon[32255]: pgmap v4681: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:34:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:34:32.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:32.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:32.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:32.463 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:32 smithi022 ceph-mon[31360]: pgmap v4681: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:32.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:32 smithi022 ceph-mon[40250]: pgmap v4681: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2454408456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:34:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:34:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:34:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:34:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2454408456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:34:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:34:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:34:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:34:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2454408456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:34:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:34:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:34:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:34:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:34 smithi122 ceph-mon[32255]: pgmap v4682: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:34 smithi022 ceph-mon[31360]: pgmap v4682: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:34 smithi022 ceph-mon[40250]: pgmap v4682: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:36 smithi122 ceph-mon[32255]: pgmap v4683: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:36.728 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:36 smithi022 ceph-mon[31360]: pgmap v4683: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:36.729 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:36 smithi022 ceph-mon[40250]: pgmap v4683: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:37.464 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:34:37.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:37.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:37.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:38 smithi122 ceph-mon[32255]: pgmap v4684: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4106332611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:38 smithi022 ceph-mon[31360]: pgmap v4684: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4106332611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:38 smithi022 ceph-mon[40250]: pgmap v4684: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4106332611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:40] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:34:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:40 smithi122 ceph-mon[32255]: pgmap v4685: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:40 smithi022 ceph-mon[31360]: pgmap v4685: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:40 smithi022 ceph-mon[40250]: pgmap v4685: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:41.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:34:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:34:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:42 smithi122 ceph-mon[32255]: pgmap v4686: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:34:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:34:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:42 smithi022 ceph-mon[31360]: pgmap v4686: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:42 smithi022 ceph-mon[40250]: pgmap v4686: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:42.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:43.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:43.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:43.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:44 smithi122 ceph-mon[32255]: pgmap v4687: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2526958470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:44 smithi022 ceph-mon[31360]: pgmap v4687: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2526958470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:44 smithi022 ceph-mon[40250]: pgmap v4687: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2526958470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:46 smithi122 ceph-mon[32255]: pgmap v4688: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:46 smithi022 ceph-mon[31360]: pgmap v4688: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:46 smithi022 ceph-mon[40250]: pgmap v4688: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:34:48.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:48 smithi122 ceph-mon[32255]: pgmap v4689: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:48 smithi022 ceph-mon[31360]: pgmap v4689: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:48 smithi022 ceph-mon[40250]: pgmap v4689: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:48.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:48.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:48.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4042271045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4042271045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4042271045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:50] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:34:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:50 smithi122 ceph-mon[32255]: pgmap v4690: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:50 smithi022 ceph-mon[31360]: pgmap v4690: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:50 smithi022 ceph-mon[40250]: pgmap v4690: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:51.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:34:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:34:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:52 smithi122 ceph-mon[32255]: pgmap v4691: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:34:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:34:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:52 smithi022 ceph-mon[31360]: pgmap v4691: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:52 smithi022 ceph-mon[40250]: pgmap v4691: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:34:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:53.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:53.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:53.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:34:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:54 smithi022 ceph-mon[31360]: pgmap v4692: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1914909004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:54 smithi022 ceph-mon[40250]: pgmap v4692: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1914909004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:54 smithi122 ceph-mon[32255]: pgmap v4692: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1914909004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:34:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:56 smithi122 ceph-mon[32255]: pgmap v4693: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:56 smithi022 ceph-mon[31360]: pgmap v4693: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:56 smithi022 ceph-mon[40250]: pgmap v4693: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:34:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:34:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:34:58 smithi122 ceph-mon[32255]: pgmap v4694: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:34:58 smithi022 ceph-mon[31360]: pgmap v4694: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:34:58 smithi022 ceph-mon[40250]: pgmap v4694: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:34:58.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:34:59.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:34:59.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:34:59.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:00 smithi122 ceph-mon[32255]: pgmap v4695: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2164032630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:00] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:35:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:00 smithi022 ceph-mon[31360]: pgmap v4695: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2164032630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:00 smithi022 ceph-mon[40250]: pgmap v4695: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2164032630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:01.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:35:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:35:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:02 smithi122 ceph-mon[32255]: pgmap v4696: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:35:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:35:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:02 smithi022 ceph-mon[31360]: pgmap v4696: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:02 smithi022 ceph-mon[40250]: pgmap v4696: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:04.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:04 smithi122 ceph-mon[32255]: pgmap v4697: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:04 smithi022 ceph-mon[31360]: pgmap v4697: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:04 smithi022 ceph-mon[40250]: pgmap v4697: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:04.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:04.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:04.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/93627674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/93627674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/93627674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:06 smithi122 ceph-mon[32255]: pgmap v4698: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:06 smithi022 ceph-mon[31360]: pgmap v4698: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:06 smithi022 ceph-mon[40250]: pgmap v4698: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:07 smithi122 ceph-mon[32255]: pgmap v4699: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:07 smithi022 ceph-mon[31360]: pgmap v4699: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:35:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:07 smithi022 ceph-mon[40250]: pgmap v4699: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:09.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:09 smithi022 ceph-mon[31360]: pgmap v4700: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:09 smithi022 ceph-mon[40250]: pgmap v4700: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:10.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:10.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:10.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:09 smithi122 ceph-mon[32255]: pgmap v4700: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:10] "GET /metrics HTTP/1.1" 200 202337 "" "Prometheus/2.18.1" 2021-12-15T20:35:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396254143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396254143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:11.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396254143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:11.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:35:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:35:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:11 smithi022 ceph-mon[31360]: pgmap v4701: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:11 smithi022 ceph-mon[40250]: pgmap v4701: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:11 smithi122 ceph-mon[32255]: pgmap v4701: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:35:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:35:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:13 smithi022 ceph-mon[31360]: pgmap v4702: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:13 smithi022 ceph-mon[40250]: pgmap v4702: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:13 smithi122 ceph-mon[32255]: pgmap v4702: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:15.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:15.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:15.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:15.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:15 smithi022 ceph-mon[31360]: pgmap v4703: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2627441028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:15 smithi022 ceph-mon[40250]: pgmap v4703: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2627441028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:15 smithi122 ceph-mon[32255]: pgmap v4703: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2627441028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:35:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:17 smithi022 ceph-mon[31360]: pgmap v4704: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:17 smithi022 ceph-mon[40250]: pgmap v4704: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:17 smithi122 ceph-mon[32255]: pgmap v4704: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:19 smithi022 ceph-mon[31360]: pgmap v4705: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:19 smithi022 ceph-mon[40250]: pgmap v4705: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:20.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:35:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:19 smithi122 ceph-mon[32255]: pgmap v4705: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:20.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:20.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:20.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1572029456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1572029456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:21.302 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1572029456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:21.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:35:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:35:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:21 smithi022 ceph-mon[31360]: pgmap v4706: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:21 smithi022 ceph-mon[40250]: pgmap v4706: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:21 smithi122 ceph-mon[32255]: pgmap v4706: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:35:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:35:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:23 smithi022 ceph-mon[31360]: pgmap v4707: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:23 smithi022 ceph-mon[40250]: pgmap v4707: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:23 smithi122 ceph-mon[32255]: pgmap v4707: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:25.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:26.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:25 smithi022 ceph-mon[31360]: pgmap v4708: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:26.168 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:25 smithi022 ceph-mon[40250]: pgmap v4708: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:25 smithi122 ceph-mon[32255]: pgmap v4708: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/321607423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/321607423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:27.330 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/321607423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:27.737 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:35:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[31360]: pgmap v4709: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:35:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:35:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:35:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:35:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[40250]: pgmap v4709: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:35:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:35:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:35:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:35:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:27 smithi122 ceph-mon[32255]: pgmap v4709: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:35:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:35:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:35:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:35:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:29 smithi022 ceph-mon[31360]: pgmap v4710: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:29 smithi022 ceph-mon[40250]: pgmap v4710: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:35:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:29 smithi122 ceph-mon[32255]: pgmap v4710: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:31.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:31.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:31.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:31.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:31.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:35:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:35:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:31 smithi022 ceph-mon[31360]: pgmap v4711: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:32.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/972931274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:31 smithi022 ceph-mon[40250]: pgmap v4711: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:32.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/972931274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:31 smithi122 ceph-mon[32255]: pgmap v4711: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/972931274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:35:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:35:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:35:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:35:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:35:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:35:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:35:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:35:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:35:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:35:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:35:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:33 smithi022 ceph-mon[31360]: pgmap v4712: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:33 smithi022 ceph-mon[40250]: pgmap v4712: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:33 smithi122 ceph-mon[32255]: pgmap v4712: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:35 smithi022 ceph-mon[31360]: pgmap v4713: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:35 smithi022 ceph-mon[40250]: pgmap v4713: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:35 smithi122 ceph-mon[32255]: pgmap v4713: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:36.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:36.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:36.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:36.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1042224395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1042224395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1042224395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:35:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:37 smithi022 ceph-mon[31360]: pgmap v4714: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:37 smithi022 ceph-mon[40250]: pgmap v4714: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:37 smithi122 ceph-mon[32255]: pgmap v4714: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:39 smithi022 ceph-mon[31360]: pgmap v4715: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:39 smithi022 ceph-mon[40250]: pgmap v4715: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:39 smithi122 ceph-mon[32255]: pgmap v4715: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:40] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:35:41.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:35:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:35:41.907 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:42.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:42.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:42.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:41 smithi022 ceph-mon[31360]: pgmap v4716: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:41 smithi022 ceph-mon[40250]: pgmap v4716: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:41 smithi122 ceph-mon[32255]: pgmap v4716: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:35:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:35:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1716479218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1716479218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1716479218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:43 smithi022 ceph-mon[31360]: pgmap v4717: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:43 smithi022 ceph-mon[40250]: pgmap v4717: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:43 smithi122 ceph-mon[32255]: pgmap v4717: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:45 smithi022 ceph-mon[31360]: pgmap v4718: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:45 smithi022 ceph-mon[40250]: pgmap v4718: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:45 smithi122 ceph-mon[32255]: pgmap v4718: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:47.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:47.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:47.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:47.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:47.652 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:35:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:48 smithi022 ceph-mon[31360]: pgmap v4719: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1650785398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:48 smithi022 ceph-mon[40250]: pgmap v4719: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1650785398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:48 smithi122 ceph-mon[32255]: pgmap v4719: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1650785398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:50 smithi022 ceph-mon[31360]: pgmap v4720: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:50 smithi022 ceph-mon[40250]: pgmap v4720: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:50] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:35:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:50 smithi122 ceph-mon[32255]: pgmap v4720: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:51.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:35:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:35:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:52 smithi022 ceph-mon[31360]: pgmap v4721: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:52 smithi022 ceph-mon[40250]: pgmap v4721: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:52 smithi122 ceph-mon[32255]: pgmap v4721: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:52.653 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:35:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:35:53.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:53.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:53.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:35:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:54 smithi022 ceph-mon[31360]: pgmap v4722: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/302707608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:54 smithi022 ceph-mon[40250]: pgmap v4722: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/302707608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:54 smithi122 ceph-mon[32255]: pgmap v4722: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/302707608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:56 smithi022 ceph-mon[31360]: pgmap v4723: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:56 smithi022 ceph-mon[40250]: pgmap v4723: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:56 smithi122 ceph-mon[32255]: pgmap v4723: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:35:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:35:58.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:35:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:58 smithi022 ceph-mon[31360]: pgmap v4724: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:58 smithi022 ceph-mon[40250]: pgmap v4724: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:58.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:35:58.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:35:58.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:35:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:58 smithi122 ceph-mon[32255]: pgmap v4724: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:35:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:35:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2647796979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:35:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2647796979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:35:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:35:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2647796979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:00 smithi022 ceph-mon[31360]: pgmap v4725: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:00 smithi022 ceph-mon[40250]: pgmap v4725: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:00] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:36:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:00 smithi122 ceph-mon[32255]: pgmap v4725: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:01.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:36:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:36:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:02 smithi022 ceph-mon[31360]: pgmap v4726: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:02 smithi022 ceph-mon[40250]: pgmap v4726: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:02 smithi122 ceph-mon[32255]: pgmap v4726: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:36:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:36:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:03.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:03.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:03.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:03.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:04 smithi022 ceph-mon[31360]: pgmap v4727: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2266511284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:04 smithi022 ceph-mon[40250]: pgmap v4727: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2266511284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:04 smithi122 ceph-mon[32255]: pgmap v4727: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2266511284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:06 smithi022 ceph-mon[31360]: pgmap v4728: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:06 smithi022 ceph-mon[40250]: pgmap v4728: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:06 smithi122 ceph-mon[32255]: pgmap v4728: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:36:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:08 smithi122 ceph-mon[32255]: pgmap v4729: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:08 smithi022 ceph-mon[31360]: pgmap v4729: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:08.643 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:08 smithi022 ceph-mon[40250]: pgmap v4729: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:08.769 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:09.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:09.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:09.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:36:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:10 smithi122 ceph-mon[32255]: pgmap v4730: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3839933697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:10 smithi022 ceph-mon[31360]: pgmap v4730: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3839933697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:10 smithi022 ceph-mon[40250]: pgmap v4730: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3839933697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:11.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:36:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:36:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:12 smithi122 ceph-mon[32255]: pgmap v4731: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:12 smithi022 ceph-mon[31360]: pgmap v4731: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:12 smithi022 ceph-mon[40250]: pgmap v4731: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:36:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:36:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:14.140 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:14 smithi122 ceph-mon[32255]: pgmap v4732: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:14.509 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:14 smithi022 ceph-mon[31360]: pgmap v4732: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:14.510 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:14 smithi022 ceph-mon[40250]: pgmap v4732: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/530824553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/530824553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/530824553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:16 smithi122 ceph-mon[32255]: pgmap v4733: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:16 smithi022 ceph-mon[31360]: pgmap v4733: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:16 smithi022 ceph-mon[40250]: pgmap v4733: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:36:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:18 smithi122 ceph-mon[32255]: pgmap v4734: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:18 smithi022 ceph-mon[31360]: pgmap v4734: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:18 smithi022 ceph-mon[40250]: pgmap v4734: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:19.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:19.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:19.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:19.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:20 smithi022 ceph-mon[31360]: pgmap v4735: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3770198123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:20 smithi022 ceph-mon[40250]: pgmap v4735: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3770198123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:20] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:36:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:20 smithi122 ceph-mon[32255]: pgmap v4735: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3770198123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:21.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:36:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:36:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:22 smithi122 ceph-mon[32255]: pgmap v4736: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:22 smithi022 ceph-mon[31360]: pgmap v4736: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:22 smithi022 ceph-mon[40250]: pgmap v4736: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:36:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:36:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:24 smithi122 ceph-mon[32255]: pgmap v4737: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:24 smithi022 ceph-mon[31360]: pgmap v4737: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:24 smithi022 ceph-mon[40250]: pgmap v4737: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:25.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:25.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:25.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:26 smithi122 ceph-mon[32255]: pgmap v4738: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/6227804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:26 smithi022 ceph-mon[31360]: pgmap v4738: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/6227804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:26 smithi022 ceph-mon[40250]: pgmap v4738: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/6227804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:36:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:28 smithi122 ceph-mon[32255]: pgmap v4739: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:36:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:36:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:36:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[31360]: pgmap v4739: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:36:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:36:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:36:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[40250]: pgmap v4739: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:36:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:36:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:36:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:36:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:29 smithi022 ceph-mon[31360]: pgmap v4740: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:36:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:29 smithi022 ceph-mon[40250]: pgmap v4740: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:36:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:29 smithi122 ceph-mon[32255]: pgmap v4740: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:30.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:30] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:36:30.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:30.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:30.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508308463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508308463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508308463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:31.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:36:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:36:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:31 smithi122 ceph-mon[32255]: pgmap v4741: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:31 smithi022 ceph-mon[31360]: pgmap v4741: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:31 smithi022 ceph-mon[40250]: pgmap v4741: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:32.665 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:36:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:36:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:36:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:36:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:36:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:36:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:33 smithi122 ceph-mon[32255]: pgmap v4742: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:36:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:36:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:36:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:36:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[31360]: pgmap v4742: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:36:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:36:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:36:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:36:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:33 smithi022 ceph-mon[40250]: pgmap v4742: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:35.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:35.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:35.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:35 smithi022 ceph-mon[31360]: pgmap v4743: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:35 smithi022 ceph-mon[40250]: pgmap v4743: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:35 smithi122 ceph-mon[32255]: pgmap v4743: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2699273545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2699273545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2699273545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:36:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:37 smithi022 ceph-mon[31360]: pgmap v4744: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:37 smithi022 ceph-mon[40250]: pgmap v4744: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:37 smithi122 ceph-mon[32255]: pgmap v4744: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:39 smithi022 ceph-mon[31360]: pgmap v4745: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:39 smithi022 ceph-mon[40250]: pgmap v4745: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:39 smithi122 ceph-mon[32255]: pgmap v4745: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:40] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:36:40.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:41.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:41.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:41.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:41.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:36:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:36:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:41 smithi022 ceph-mon[31360]: pgmap v4746: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/330977565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:41 smithi022 ceph-mon[40250]: pgmap v4746: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/330977565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:41 smithi122 ceph-mon[32255]: pgmap v4746: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/330977565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:36:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:36:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:43 smithi022 ceph-mon[31360]: pgmap v4747: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:43 smithi022 ceph-mon[40250]: pgmap v4747: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:43 smithi122 ceph-mon[32255]: pgmap v4747: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:45 smithi022 ceph-mon[31360]: pgmap v4748: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:45 smithi022 ceph-mon[40250]: pgmap v4748: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:46.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:45 smithi122 ceph-mon[32255]: pgmap v4748: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:46.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:46.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:46.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2568986492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2568986492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2568986492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:36:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:47 smithi022 ceph-mon[31360]: pgmap v4749: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:47 smithi022 ceph-mon[40250]: pgmap v4749: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:47 smithi122 ceph-mon[32255]: pgmap v4749: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:49 smithi022 ceph-mon[31360]: pgmap v4750: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:49 smithi022 ceph-mon[40250]: pgmap v4750: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:50] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:36:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:49 smithi122 ceph-mon[32255]: pgmap v4750: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:51.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:36:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:36:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:52.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:51 smithi022 ceph-mon[31360]: pgmap v4751: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:51 smithi022 ceph-mon[40250]: pgmap v4751: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:51 smithi122 ceph-mon[32255]: pgmap v4751: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:36:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:36:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/295794462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/295794462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:36:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/295794462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:53 smithi022 ceph-mon[31360]: pgmap v4752: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:53 smithi022 ceph-mon[40250]: pgmap v4752: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:53 smithi122 ceph-mon[32255]: pgmap v4752: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:55 smithi022 ceph-mon[31360]: pgmap v4753: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:55 smithi022 ceph-mon[40250]: pgmap v4753: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:55 smithi122 ceph-mon[32255]: pgmap v4753: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:36:57.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:36:57.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:36:57.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:36:57.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:36:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:36:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:57 smithi022 ceph-mon[31360]: pgmap v4754: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:36:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/514081425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:57 smithi022 ceph-mon[40250]: pgmap v4754: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:36:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/514081425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:36:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:57 smithi122 ceph-mon[32255]: pgmap v4754: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:36:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:36:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/514081425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:00 smithi022 ceph-mon[31360]: pgmap v4755: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:00 smithi022 ceph-mon[40250]: pgmap v4755: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:00 smithi122 ceph-mon[32255]: pgmap v4755: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:00] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:37:01.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:37:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:37:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:02 smithi022 ceph-mon[31360]: pgmap v4756: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:02 smithi022 ceph-mon[40250]: pgmap v4756: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:02 smithi122 ceph-mon[32255]: pgmap v4756: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:02.476 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:37:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:37:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:02.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/871474492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/871474492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/871474492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:04 smithi022 ceph-mon[31360]: pgmap v4757: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:04 smithi022 ceph-mon[40250]: pgmap v4757: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:04 smithi122 ceph-mon[32255]: pgmap v4757: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:06 smithi022 ceph-mon[31360]: pgmap v4758: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:06 smithi022 ceph-mon[40250]: pgmap v4758: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:06 smithi122 ceph-mon[32255]: pgmap v4758: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:37:07.844 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:08.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:08.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:08.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:08.218 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:08 smithi022 ceph-mon[40250]: pgmap v4759: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:08.219 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:08 smithi022 ceph-mon[31360]: pgmap v4759: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:08 smithi122 ceph-mon[32255]: pgmap v4759: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/952185911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/952185911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/952185911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:10 smithi022 ceph-mon[31360]: pgmap v4760: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:10 smithi022 ceph-mon[40250]: pgmap v4760: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:10 smithi122 ceph-mon[32255]: pgmap v4760: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:10] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:37:11.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:37:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:37:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:12 smithi022 ceph-mon[31360]: pgmap v4761: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:12 smithi022 ceph-mon[40250]: pgmap v4761: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:12 smithi122 ceph-mon[32255]: pgmap v4761: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:37:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:37:13.219 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:14 smithi022 ceph-mon[40250]: pgmap v4762: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1673646387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:14 smithi022 ceph-mon[31360]: pgmap v4762: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1673646387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:14 smithi122 ceph-mon[32255]: pgmap v4762: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1673646387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:16 smithi022 ceph-mon[31360]: pgmap v4763: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:16 smithi022 ceph-mon[40250]: pgmap v4763: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:16 smithi122 ceph-mon[32255]: pgmap v4763: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:37:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:18 smithi122 ceph-mon[32255]: pgmap v4764: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:18 smithi022 ceph-mon[31360]: pgmap v4764: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:18 smithi022 ceph-mon[40250]: pgmap v4764: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:18.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:18.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:18.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:18.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1023981954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1023981954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1023981954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:20] "GET /metrics HTTP/1.1" 200 202340 "" "Prometheus/2.18.1" 2021-12-15T20:37:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:20 smithi122 ceph-mon[32255]: pgmap v4765: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:20 smithi022 ceph-mon[31360]: pgmap v4765: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:20 smithi022 ceph-mon[40250]: pgmap v4765: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:21.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:37:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:37:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:22 smithi122 ceph-mon[32255]: pgmap v4766: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:22 smithi022 ceph-mon[31360]: pgmap v4766: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:22 smithi022 ceph-mon[40250]: pgmap v4766: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:37:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:37:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:23.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:24 smithi022 ceph-mon[31360]: pgmap v4767: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:24 smithi022 ceph-mon[40250]: pgmap v4767: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:24.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:24.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:24.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:24 smithi122 ceph-mon[32255]: pgmap v4767: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/382883573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/382883573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/382883573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:26 smithi122 ceph-mon[32255]: pgmap v4768: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:26 smithi022 ceph-mon[31360]: pgmap v4768: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:26 smithi022 ceph-mon[40250]: pgmap v4768: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:37:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:28 smithi122 ceph-mon[32255]: pgmap v4769: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:28 smithi022 ceph-mon[31360]: pgmap v4769: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:28 smithi022 ceph-mon[40250]: pgmap v4769: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:29.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:37:29.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:37:29.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:37:29.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:29.340 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:29.340 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:37:29.340 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:37:29.341 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:37:29.341 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:29.341 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:29.341 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:37:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:37:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:37:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:29.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:29.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:29.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:30] "GET /metrics HTTP/1.1" 200 202340 "" "Prometheus/2.18.1" 2021-12-15T20:37:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:30 smithi122 ceph-mon[32255]: pgmap v4770: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3235480793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:30 smithi022 ceph-mon[31360]: pgmap v4770: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3235480793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:30 smithi022 ceph-mon[40250]: pgmap v4770: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:37:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3235480793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:31.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:37:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:37:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:32 smithi122 ceph-mon[32255]: pgmap v4771: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:32 smithi022 ceph-mon[31360]: pgmap v4771: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:32 smithi022 ceph-mon[40250]: pgmap v4771: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:37:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:37:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:37:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:37:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:37:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:37:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:37:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:37:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:34 smithi122 ceph-mon[32255]: pgmap v4772: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:34 smithi022 ceph-mon[31360]: pgmap v4772: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:34 smithi022 ceph-mon[40250]: pgmap v4772: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:34.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:35.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:35.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:35.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2779092357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2779092357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2779092357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:36 smithi122 ceph-mon[32255]: pgmap v4773: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:36 smithi022 ceph-mon[31360]: pgmap v4773: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:36 smithi022 ceph-mon[40250]: pgmap v4773: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:37:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:38 smithi122 ceph-mon[32255]: pgmap v4774: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:38 smithi022 ceph-mon[31360]: pgmap v4774: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:38 smithi022 ceph-mon[40250]: pgmap v4774: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:40.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:40 smithi122 ceph-mon[32255]: pgmap v4775: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:40] "GET /metrics HTTP/1.1" 200 202339 "" "Prometheus/2.18.1" 2021-12-15T20:37:40.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:40.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:40.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:40.440 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:40 smithi022 ceph-mon[40250]: pgmap v4775: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:40.440 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:40 smithi022 ceph-mon[31360]: pgmap v4775: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:41.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1050868236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1050868236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1050868236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:41.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:37:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:37:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:37:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:37:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:42 smithi122 ceph-mon[32255]: pgmap v4776: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:42 smithi022 ceph-mon[31360]: pgmap v4776: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:42 smithi022 ceph-mon[40250]: pgmap v4776: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:44 smithi122 ceph-mon[32255]: pgmap v4777: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:44 smithi022 ceph-mon[31360]: pgmap v4777: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:44 smithi022 ceph-mon[40250]: pgmap v4777: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:45.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:45.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:45.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:45.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:46 smithi122 ceph-mon[32255]: pgmap v4778: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/268849324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:46 smithi022 ceph-mon[31360]: pgmap v4778: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/268849324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:46 smithi022 ceph-mon[40250]: pgmap v4778: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/268849324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:37:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:48 smithi122 ceph-mon[32255]: pgmap v4779: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:48 smithi022 ceph-mon[31360]: pgmap v4779: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:48 smithi022 ceph-mon[40250]: pgmap v4779: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:50] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:37:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:50 smithi122 ceph-mon[32255]: pgmap v4780: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:50 smithi022 ceph-mon[31360]: pgmap v4780: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:50 smithi022 ceph-mon[40250]: pgmap v4780: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:37:50.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:51.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:51.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:51.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:51.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:37:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:37:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:52 smithi122 ceph-mon[32255]: pgmap v4781: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3589278388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:37:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:37:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:52 smithi022 ceph-mon[31360]: pgmap v4781: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3589278388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:52 smithi022 ceph-mon[40250]: pgmap v4781: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3589278388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:37:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:54 smithi122 ceph-mon[32255]: pgmap v4782: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:54 smithi022 ceph-mon[31360]: pgmap v4782: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:54 smithi022 ceph-mon[40250]: pgmap v4782: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:37:56.180 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:37:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:56 smithi122 ceph-mon[32255]: pgmap v4783: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:37:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:56 smithi022 ceph-mon[31360]: pgmap v4783: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:37:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:56 smithi022 ceph-mon[40250]: pgmap v4783: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:37:56.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:37:56.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:37:56.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:37:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/743918595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/743918595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/743918595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:37:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:37:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:37:58 smithi122 ceph-mon[32255]: pgmap v4784: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:37:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:37:58 smithi022 ceph-mon[31360]: pgmap v4784: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:37:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:37:58 smithi022 ceph-mon[40250]: pgmap v4784: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:38:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:00 smithi122 ceph-mon[32255]: pgmap v4785: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:00] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:38:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:00 smithi022 ceph-mon[31360]: pgmap v4785: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:00 smithi022 ceph-mon[40250]: pgmap v4785: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:01.551 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:01.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:38:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:38:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:02 smithi022 ceph-mon[31360]: pgmap v4786: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2083151818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:02 smithi022 ceph-mon[40250]: pgmap v4786: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2083151818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:38:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:38:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:02 smithi122 ceph-mon[32255]: pgmap v4786: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2083151818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:04 smithi122 ceph-mon[32255]: pgmap v4787: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:04 smithi022 ceph-mon[31360]: pgmap v4787: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:04 smithi022 ceph-mon[40250]: pgmap v4787: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:06 smithi122 ceph-mon[32255]: pgmap v4788: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:06 smithi022 ceph-mon[31360]: pgmap v4788: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:06 smithi022 ceph-mon[40250]: pgmap v4788: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:38:06.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:07.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:07.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:07.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:38:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:08 smithi122 ceph-mon[32255]: pgmap v4789: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:38:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2314435341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:08 smithi022 ceph-mon[31360]: pgmap v4789: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:38:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2314435341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:08 smithi022 ceph-mon[40250]: pgmap v4789: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:38:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2314435341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:10] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:38:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:10 smithi122 ceph-mon[32255]: pgmap v4790: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:38:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:10 smithi022 ceph-mon[31360]: pgmap v4790: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:38:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:10 smithi022 ceph-mon[40250]: pgmap v4790: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:38:11.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:38:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:38:12.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:12 smithi122 ceph-mon[32255]: pgmap v4791: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:38:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:38:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:12 smithi022 ceph-mon[31360]: pgmap v4791: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:12 smithi022 ceph-mon[40250]: pgmap v4791: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:12.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:12.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:12.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/900339710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/900339710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/900339710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:14 smithi122 ceph-mon[32255]: pgmap v4792: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:14 smithi022 ceph-mon[31360]: pgmap v4792: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:14 smithi022 ceph-mon[40250]: pgmap v4792: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:16 smithi122 ceph-mon[32255]: pgmap v4793: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:16 smithi022 ceph-mon[31360]: pgmap v4793: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:16 smithi022 ceph-mon[40250]: pgmap v4793: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:17.662 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:38:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:18.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:18.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:18 smithi122 ceph-mon[32255]: pgmap v4794: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3196879821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:18 smithi022 ceph-mon[31360]: pgmap v4794: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3196879821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:18 smithi022 ceph-mon[40250]: pgmap v4794: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3196879821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:20 smithi122 ceph-mon[32255]: pgmap v4795: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:20] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:38:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:20 smithi022 ceph-mon[31360]: pgmap v4795: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:20 smithi022 ceph-mon[40250]: pgmap v4795: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:21.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:38:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:38:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:22 smithi122 ceph-mon[32255]: pgmap v4796: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:38:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:38:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:22 smithi022 ceph-mon[31360]: pgmap v4796: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:22 smithi022 ceph-mon[40250]: pgmap v4796: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:23.034 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:23.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:23.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:23.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:23.405 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:23.405 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:24 smithi122 ceph-mon[32255]: pgmap v4797: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1796742411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:24 smithi022 ceph-mon[31360]: pgmap v4797: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1796742411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:24 smithi022 ceph-mon[40250]: pgmap v4797: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1796742411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:26 smithi122 ceph-mon[32255]: pgmap v4798: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:26 smithi022 ceph-mon[31360]: pgmap v4798: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:26 smithi022 ceph-mon[40250]: pgmap v4798: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:38:28.406 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:28.406 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:28 smithi022 ceph-mon[31360]: pgmap v4799: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:28.407 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:28 smithi022 ceph-mon[40250]: pgmap v4799: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:28 smithi122 ceph-mon[32255]: pgmap v4799: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:28.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:28.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:28.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2002012986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:38:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:38:29.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:38:29.503 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2002012986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:29.503 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:38:29.504 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:38:29.504 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:38:29.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2002012986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:29.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:38:29.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:38:29.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:38:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:30] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:38:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:30 smithi122 ceph-mon[32255]: pgmap v4800: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:38:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:30 smithi022 ceph-mon[31360]: pgmap v4800: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:38:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:30 smithi022 ceph-mon[40250]: pgmap v4800: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:38:31.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:38:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:38:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:32 smithi122 ceph-mon[32255]: pgmap v4801: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:38:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:38:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:32 smithi022 ceph-mon[31360]: pgmap v4801: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:32 smithi022 ceph-mon[40250]: pgmap v4801: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:38:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:38:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:38:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:38:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:38:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:38:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:38:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:38:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:38:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:38:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:38:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:38:33.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:34.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:34.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:34.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:34 smithi122 ceph-mon[32255]: pgmap v4802: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3627378645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:34 smithi022 ceph-mon[31360]: pgmap v4802: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3627378645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:34 smithi022 ceph-mon[40250]: pgmap v4802: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3627378645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:36 smithi122 ceph-mon[32255]: pgmap v4803: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:36 smithi022 ceph-mon[31360]: pgmap v4803: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:36 smithi022 ceph-mon[40250]: pgmap v4803: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:38:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:38 smithi122 ceph-mon[32255]: pgmap v4804: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:38.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:38 smithi022 ceph-mon[31360]: pgmap v4804: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:38.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:38 smithi022 ceph-mon[40250]: pgmap v4804: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:39.144 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:39.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:39.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:39.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:40] "GET /metrics HTTP/1.1" 200 202325 "" "Prometheus/2.18.1" 2021-12-15T20:38:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:40 smithi122 ceph-mon[32255]: pgmap v4805: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3184485056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:40 smithi022 ceph-mon[40250]: pgmap v4805: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3184485056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:40 smithi022 ceph-mon[31360]: pgmap v4805: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3184485056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:41.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:38:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:38:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:42 smithi122 ceph-mon[32255]: pgmap v4806: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:38:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:38:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:42 smithi022 ceph-mon[31360]: pgmap v4806: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:42 smithi022 ceph-mon[40250]: pgmap v4806: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:44 smithi122 ceph-mon[32255]: pgmap v4807: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:44.514 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:44.515 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:44 smithi022 ceph-mon[31360]: pgmap v4807: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:44.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:44 smithi022 ceph-mon[40250]: pgmap v4807: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:44.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/849683882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/849683882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/849683882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:46 smithi122 ceph-mon[32255]: pgmap v4808: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:46 smithi022 ceph-mon[31360]: pgmap v4808: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:46 smithi022 ceph-mon[40250]: pgmap v4808: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:47 smithi122 ceph-mon[32255]: pgmap v4809: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:47 smithi022 ceph-mon[31360]: pgmap v4809: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:38:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:47 smithi022 ceph-mon[40250]: pgmap v4809: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:49.880 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:50.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:50.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:50.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:50.249 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:50 smithi022 ceph-mon[31360]: pgmap v4810: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:50.250 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:50 smithi022 ceph-mon[40250]: pgmap v4810: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:50 smithi122 ceph-mon[32255]: pgmap v4810: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:50] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:38:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4076825840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4076825840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4076825840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:51.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:38:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:38:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:52 smithi022 ceph-mon[31360]: pgmap v4811: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:52 smithi022 ceph-mon[40250]: pgmap v4811: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:52 smithi122 ceph-mon[32255]: pgmap v4811: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:38:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:38:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:38:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:54 smithi022 ceph-mon[31360]: pgmap v4812: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:54 smithi022 ceph-mon[40250]: pgmap v4812: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:54 smithi122 ceph-mon[32255]: pgmap v4812: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:55.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:38:55.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:38:55.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:38:55.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:38:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:56 smithi022 ceph-mon[31360]: pgmap v4813: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/397263516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:56 smithi022 ceph-mon[40250]: pgmap v4813: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/397263516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:56 smithi122 ceph-mon[32255]: pgmap v4813: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:38:56.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/397263516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:38:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:38:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:38:58 smithi022 ceph-mon[31360]: pgmap v4814: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:38:58 smithi022 ceph-mon[40250]: pgmap v4814: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:38:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:38:58 smithi122 ceph-mon[32255]: pgmap v4814: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:00 smithi022 ceph-mon[31360]: pgmap v4815: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:00 smithi022 ceph-mon[40250]: pgmap v4815: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:00] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:00 smithi122 ceph-mon[32255]: pgmap v4815: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:00.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:00.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:00.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:00.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4212672553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4212672553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4212672553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:01.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:39:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:39:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:02 smithi022 ceph-mon[31360]: pgmap v4816: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:02 smithi022 ceph-mon[40250]: pgmap v4816: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:02 smithi122 ceph-mon[32255]: pgmap v4816: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:39:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:39:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:04 smithi122 ceph-mon[32255]: pgmap v4817: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:04 smithi022 ceph-mon[31360]: pgmap v4817: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:04 smithi022 ceph-mon[40250]: pgmap v4817: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:05.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:06 smithi022 ceph-mon[31360]: pgmap v4818: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:06 smithi022 ceph-mon[40250]: pgmap v4818: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:06.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:06.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:06.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:06 smithi122 ceph-mon[32255]: pgmap v4818: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4047075719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4047075719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4047075719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:39:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:08 smithi122 ceph-mon[32255]: pgmap v4819: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:08 smithi022 ceph-mon[31360]: pgmap v4819: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:08 smithi022 ceph-mon[40250]: pgmap v4819: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:10] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:39:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:10 smithi122 ceph-mon[32255]: pgmap v4820: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:10 smithi022 ceph-mon[31360]: pgmap v4820: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:10 smithi022 ceph-mon[40250]: pgmap v4820: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:11.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:11.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:11.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:11.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:39:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:39:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:12 smithi122 ceph-mon[32255]: pgmap v4821: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2556381505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:12 smithi022 ceph-mon[31360]: pgmap v4821: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2556381505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:12 smithi022 ceph-mon[40250]: pgmap v4821: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2556381505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:39:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:39:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:14 smithi122 ceph-mon[32255]: pgmap v4822: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:14 smithi022 ceph-mon[31360]: pgmap v4822: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:14 smithi022 ceph-mon[40250]: pgmap v4822: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:16 smithi122 ceph-mon[32255]: pgmap v4823: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:16 smithi022 ceph-mon[31360]: pgmap v4823: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:16 smithi022 ceph-mon[40250]: pgmap v4823: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:16.728 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:17.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:17.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:17.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:39:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:18 smithi122 ceph-mon[32255]: pgmap v4824: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1937047479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:18 smithi022 ceph-mon[31360]: pgmap v4824: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1937047479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:18 smithi022 ceph-mon[40250]: pgmap v4824: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1937047479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:20 smithi122 ceph-mon[32255]: pgmap v4825: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:20] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:39:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:20 smithi022 ceph-mon[31360]: pgmap v4825: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:20 smithi022 ceph-mon[40250]: pgmap v4825: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:21.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:39:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:39:22.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:22 smithi122 ceph-mon[32255]: pgmap v4826: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:22.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:22.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:22.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:22.479 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:22 smithi022 ceph-mon[31360]: pgmap v4826: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:22.480 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:22 smithi022 ceph-mon[40250]: pgmap v4826: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:39:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:39:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/409478731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/409478731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/409478731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:24 smithi122 ceph-mon[32255]: pgmap v4827: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:24 smithi022 ceph-mon[31360]: pgmap v4827: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:24 smithi022 ceph-mon[40250]: pgmap v4827: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:26 smithi122 ceph-mon[32255]: pgmap v4828: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:26 smithi022 ceph-mon[31360]: pgmap v4828: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:26 smithi022 ceph-mon[40250]: pgmap v4828: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:27.480 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:39:27.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:27.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:27.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:28 smithi022 ceph-mon[31360]: pgmap v4829: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/992180862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:28 smithi022 ceph-mon[40250]: pgmap v4829: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/992180862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:28 smithi122 ceph-mon[32255]: pgmap v4829: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/992180862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[31360]: pgmap v4830: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:39:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:39:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:39:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:39:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[40250]: pgmap v4830: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:39:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:39:30.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:39:30.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:39:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:30] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:39:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:30 smithi122 ceph-mon[32255]: pgmap v4830: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:39:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:39:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:39:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:39:31.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:39:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:39:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:32 smithi122 ceph-mon[32255]: pgmap v4831: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:39:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:39:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:32 smithi022 ceph-mon[31360]: pgmap v4831: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:32 smithi022 ceph-mon[40250]: pgmap v4831: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:32.847 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:33.218 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:33.218 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:39:33.218 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:39:33.218 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:39:33.219 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:39:33.219 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:33.219 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:39:33.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:39:33.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:39:33.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:39:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:39:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:39:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:39:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:39:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:34 smithi122 ceph-mon[32255]: pgmap v4832: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3458609012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:34 smithi022 ceph-mon[31360]: pgmap v4832: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3458609012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:34 smithi022 ceph-mon[40250]: pgmap v4832: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3458609012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:36 smithi122 ceph-mon[32255]: pgmap v4833: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:36 smithi022 ceph-mon[31360]: pgmap v4833: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:36 smithi022 ceph-mon[40250]: pgmap v4833: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:39:38.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:38 smithi122 ceph-mon[32255]: pgmap v4834: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:38 smithi022 ceph-mon[31360]: pgmap v4834: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:38 smithi022 ceph-mon[40250]: pgmap v4834: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:38.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:38.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:38.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1560297541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1560297541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1560297541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:40 smithi122 ceph-mon[32255]: pgmap v4835: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:40] "GET /metrics HTTP/1.1" 200 202326 "" "Prometheus/2.18.1" 2021-12-15T20:39:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:40 smithi022 ceph-mon[31360]: pgmap v4835: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:40 smithi022 ceph-mon[40250]: pgmap v4835: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:41.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:39:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:39:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:42 smithi122 ceph-mon[32255]: pgmap v4836: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:39:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:39:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:42 smithi022 ceph-mon[31360]: pgmap v4836: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:42 smithi022 ceph-mon[40250]: pgmap v4836: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:43.587 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:43.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:43.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:43.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:44 smithi022 ceph-mon[31360]: pgmap v4837: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/656440868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:44 smithi022 ceph-mon[40250]: pgmap v4837: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/656440868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:44 smithi122 ceph-mon[32255]: pgmap v4837: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/656440868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:46 smithi122 ceph-mon[32255]: pgmap v4838: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:46 smithi022 ceph-mon[31360]: pgmap v4838: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:46 smithi022 ceph-mon[40250]: pgmap v4838: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:39:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:48 smithi122 ceph-mon[32255]: pgmap v4839: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:48 smithi022 ceph-mon[31360]: pgmap v4839: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:48 smithi022 ceph-mon[40250]: pgmap v4839: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:48.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:49.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:49.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:49.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:50] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:39:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:50 smithi122 ceph-mon[32255]: pgmap v4840: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4145451160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:50 smithi022 ceph-mon[31360]: pgmap v4840: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4145451160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:50 smithi022 ceph-mon[40250]: pgmap v4840: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4145451160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:51.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:39:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:39:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:52 smithi122 ceph-mon[32255]: pgmap v4841: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:39:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:39:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:52 smithi022 ceph-mon[31360]: pgmap v4841: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:52 smithi022 ceph-mon[40250]: pgmap v4841: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:39:54.335 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:39:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:54 smithi122 ceph-mon[32255]: pgmap v4842: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:54 smithi022 ceph-mon[31360]: pgmap v4842: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:54 smithi022 ceph-mon[40250]: pgmap v4842: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:54.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:39:54.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:39:54.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:39:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2817288915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2817288915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2817288915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:39:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:56 smithi122 ceph-mon[32255]: pgmap v4843: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:56 smithi022 ceph-mon[31360]: pgmap v4843: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:56 smithi022 ceph-mon[40250]: pgmap v4843: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:39:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:39:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:39:58 smithi122 ceph-mon[32255]: pgmap v4844: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:39:58 smithi022 ceph-mon[31360]: pgmap v4844: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:39:58 smithi022 ceph-mon[40250]: pgmap v4844: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:39:59.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:00.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:00.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:00.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:00 smithi022 ceph-mon[31360]: pgmap v4845: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T20:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2006939458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:00.361 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:00 smithi022 ceph-mon[40250]: pgmap v4845: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:00.361 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T20:40:00.362 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2006939458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:00] "GET /metrics HTTP/1.1" 200 202328 "" "Prometheus/2.18.1" 2021-12-15T20:40:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:00 smithi122 ceph-mon[32255]: pgmap v4845: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T20:40:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2006939458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:01.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:40:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:40:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:02 smithi122 ceph-mon[32255]: pgmap v4846: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:02.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:02 smithi022 ceph-mon[31360]: pgmap v4846: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:02.232 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:02 smithi022 ceph-mon[40250]: pgmap v4846: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:40:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:40:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:04 smithi122 ceph-mon[32255]: pgmap v4847: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:04 smithi022 ceph-mon[31360]: pgmap v4847: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:04 smithi022 ceph-mon[40250]: pgmap v4847: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:05.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:05.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:05.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:06 smithi122 ceph-mon[32255]: pgmap v4848: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/267064622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:06 smithi022 ceph-mon[31360]: pgmap v4848: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/267064622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:06 smithi022 ceph-mon[40250]: pgmap v4848: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/267064622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:40:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:08 smithi022 ceph-mon[31360]: pgmap v4849: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:08 smithi022 ceph-mon[40250]: pgmap v4849: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:08 smithi122 ceph-mon[32255]: pgmap v4849: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:40:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:10 smithi122 ceph-mon[32255]: pgmap v4850: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:10.468 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:10.468 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:10 smithi022 ceph-mon[40250]: pgmap v4850: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:10.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:10 smithi022 ceph-mon[31360]: pgmap v4850: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1800027692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1800027692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1800027692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:11.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:40:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:40:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:12 smithi122 ceph-mon[32255]: pgmap v4851: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:40:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:40:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:12 smithi022 ceph-mon[31360]: pgmap v4851: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:12 smithi022 ceph-mon[40250]: pgmap v4851: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:14 smithi122 ceph-mon[32255]: pgmap v4852: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:14 smithi022 ceph-mon[31360]: pgmap v4852: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:14 smithi022 ceph-mon[40250]: pgmap v4852: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:15.842 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:16.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:16.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:16.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:16 smithi122 ceph-mon[32255]: pgmap v4853: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:16 smithi022 ceph-mon[31360]: pgmap v4853: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:16 smithi022 ceph-mon[40250]: pgmap v4853: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/706989096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/706989096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:40:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/706989096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:18 smithi122 ceph-mon[32255]: pgmap v4854: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:18 smithi022 ceph-mon[31360]: pgmap v4854: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:18 smithi022 ceph-mon[40250]: pgmap v4854: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:20 smithi022 ceph-mon[31360]: pgmap v4855: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:20 smithi022 ceph-mon[40250]: pgmap v4855: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:20] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:40:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:20 smithi122 ceph-mon[32255]: pgmap v4855: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:21.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:21.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:21.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:21.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:21.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:40:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:40:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:22 smithi122 ceph-mon[32255]: pgmap v4856: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3887663487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:40:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:40:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:22 smithi022 ceph-mon[40250]: pgmap v4856: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3887663487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:22 smithi022 ceph-mon[31360]: pgmap v4856: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3887663487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:24 smithi122 ceph-mon[32255]: pgmap v4857: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:24 smithi022 ceph-mon[31360]: pgmap v4857: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:24.634 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:24 smithi022 ceph-mon[40250]: pgmap v4857: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:26 smithi122 ceph-mon[32255]: pgmap v4858: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:26 smithi022 ceph-mon[31360]: pgmap v4858: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:26 smithi022 ceph-mon[40250]: pgmap v4858: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:26.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:26.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:26.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4213744245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4213744245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4213744245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:40:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:28 smithi122 ceph-mon[32255]: pgmap v4859: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:28 smithi022 ceph-mon[31360]: pgmap v4859: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:28 smithi022 ceph-mon[40250]: pgmap v4859: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[31360]: pgmap v4860: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:40:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:40:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:40:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[40250]: pgmap v4860: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:40:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:40:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:40:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:30 smithi122 ceph-mon[32255]: pgmap v4860: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:40:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:40:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:40:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:30] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:40:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:40:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:31 smithi022 ceph-mon[31360]: pgmap v4861: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:40:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:31 smithi022 ceph-mon[40250]: pgmap v4861: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:31.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:40:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:40:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:40:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:31 smithi122 ceph-mon[32255]: pgmap v4861: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:31.955 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:32.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:32.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:32.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2317331573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:40:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:40:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2317331573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2317331573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:40:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:40:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:40:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:40:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[31360]: pgmap v4862: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:40:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:40:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:40:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:40:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:33 smithi022 ceph-mon[40250]: pgmap v4862: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:40:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:40:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:40:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:40:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:33 smithi122 ceph-mon[32255]: pgmap v4862: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:36 smithi122 ceph-mon[32255]: pgmap v4863: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:36 smithi022 ceph-mon[31360]: pgmap v4863: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:36 smithi022 ceph-mon[40250]: pgmap v4863: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:37.327 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:37.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:37.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:37.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:37.699 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:40:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:38 smithi122 ceph-mon[32255]: pgmap v4864: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1276873876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:38 smithi022 ceph-mon[31360]: pgmap v4864: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1276873876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:38 smithi022 ceph-mon[40250]: pgmap v4864: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1276873876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:40 smithi122 ceph-mon[32255]: pgmap v4865: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:40] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:40:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:40 smithi022 ceph-mon[31360]: pgmap v4865: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:40 smithi022 ceph-mon[40250]: pgmap v4865: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:41.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:40:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:40:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:42 smithi122 ceph-mon[32255]: pgmap v4866: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:42 smithi022 ceph-mon[31360]: pgmap v4866: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:42 smithi022 ceph-mon[40250]: pgmap v4866: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:40:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:40:42.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:43.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:43.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:43.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:44 smithi122 ceph-mon[32255]: pgmap v4867: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/334621823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:44 smithi022 ceph-mon[31360]: pgmap v4867: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/334621823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:44 smithi022 ceph-mon[40250]: pgmap v4867: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/334621823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:46 smithi122 ceph-mon[32255]: pgmap v4868: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:46 smithi022 ceph-mon[31360]: pgmap v4868: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:46 smithi022 ceph-mon[40250]: pgmap v4868: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:40:48.074 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:48 smithi122 ceph-mon[32255]: pgmap v4869: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:48.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:48.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:48.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:48.439 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:48 smithi022 ceph-mon[31360]: pgmap v4869: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:48.440 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:48 smithi022 ceph-mon[40250]: pgmap v4869: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/700329972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/700329972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/700329972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:50 smithi122 ceph-mon[32255]: pgmap v4870: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:50] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:40:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:50 smithi022 ceph-mon[31360]: pgmap v4870: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:50 smithi022 ceph-mon[40250]: pgmap v4870: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:51.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:40:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:40:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:52 smithi122 ceph-mon[32255]: pgmap v4871: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:52 smithi022 ceph-mon[31360]: pgmap v4871: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:52 smithi022 ceph-mon[40250]: pgmap v4871: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:40:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:40:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:53.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:53.441 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:53.441 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:40:53.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:53.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:53.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:40:54.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:54 smithi022 ceph-mon[31360]: pgmap v4872: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:54.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2510878046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:54 smithi022 ceph-mon[40250]: pgmap v4872: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2510878046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:54 smithi122 ceph-mon[32255]: pgmap v4872: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2510878046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:40:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:56 smithi122 ceph-mon[32255]: pgmap v4873: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:56 smithi022 ceph-mon[31360]: pgmap v4873: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:56 smithi022 ceph-mon[40250]: pgmap v4873: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:40:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:40:58.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:40:58 smithi122 ceph-mon[32255]: pgmap v4874: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:40:58 smithi022 ceph-mon[31360]: pgmap v4874: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:40:58 smithi022 ceph-mon[40250]: pgmap v4874: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:40:58.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:40:59.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:40:59.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:40:59.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:00 smithi122 ceph-mon[32255]: pgmap v4875: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/489058441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:00] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:41:00.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:00 smithi022 ceph-mon[31360]: pgmap v4875: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:00.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/489058441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:00.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:00 smithi022 ceph-mon[40250]: pgmap v4875: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:00.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/489058441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:01.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:41:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:41:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:02 smithi122 ceph-mon[32255]: pgmap v4876: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:41:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:41:02.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:02 smithi022 ceph-mon[31360]: pgmap v4876: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:02.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:02 smithi022 ceph-mon[40250]: pgmap v4876: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:04.188 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:04 smithi122 ceph-mon[32255]: pgmap v4877: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:04 smithi022 ceph-mon[31360]: pgmap v4877: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:04 smithi022 ceph-mon[40250]: pgmap v4877: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:04.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:04.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:04.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/648654642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/648654642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/648654642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:06 smithi122 ceph-mon[32255]: pgmap v4878: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:06.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:06 smithi022 ceph-mon[31360]: pgmap v4878: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:06.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:06 smithi022 ceph-mon[40250]: pgmap v4878: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:41:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:08 smithi122 ceph-mon[32255]: pgmap v4879: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:08 smithi022 ceph-mon[31360]: pgmap v4879: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:08 smithi022 ceph-mon[40250]: pgmap v4879: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:09.558 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:09.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:09.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:09.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:10 smithi022 ceph-mon[31360]: pgmap v4880: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2249030680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:10 smithi022 ceph-mon[40250]: pgmap v4880: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2249030680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:41:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:10 smithi122 ceph-mon[32255]: pgmap v4880: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2249030680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:11.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:41:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:41:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:12 smithi122 ceph-mon[32255]: pgmap v4881: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:41:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:41:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:12 smithi022 ceph-mon[31360]: pgmap v4881: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:12 smithi022 ceph-mon[40250]: pgmap v4881: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:14 smithi122 ceph-mon[32255]: pgmap v4882: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:14 smithi022 ceph-mon[31360]: pgmap v4882: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:14 smithi022 ceph-mon[40250]: pgmap v4882: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:14.925 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:15.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:15.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:15.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:16 smithi122 ceph-mon[32255]: pgmap v4883: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2278729460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:16 smithi022 ceph-mon[31360]: pgmap v4883: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2278729460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:16 smithi022 ceph-mon[40250]: pgmap v4883: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2278729460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:41:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:18 smithi122 ceph-mon[32255]: pgmap v4884: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:18 smithi022 ceph-mon[31360]: pgmap v4884: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:18 smithi022 ceph-mon[40250]: pgmap v4884: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:20] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:41:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:20 smithi122 ceph-mon[32255]: pgmap v4885: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:20 smithi022 ceph-mon[31360]: pgmap v4885: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:20 smithi022 ceph-mon[40250]: pgmap v4885: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:20.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:20.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:20.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2202975141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2202975141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2202975141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:21.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:41:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:41:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:22 smithi122 ceph-mon[32255]: pgmap v4886: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:41:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:41:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:22 smithi022 ceph-mon[31360]: pgmap v4886: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:22 smithi022 ceph-mon[40250]: pgmap v4886: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:24 smithi122 ceph-mon[32255]: pgmap v4887: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:24 smithi022 ceph-mon[31360]: pgmap v4887: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:24 smithi022 ceph-mon[40250]: pgmap v4887: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:25.665 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:26.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:26.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:26.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:26 smithi122 ceph-mon[32255]: pgmap v4888: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1877253927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:26 smithi022 ceph-mon[31360]: pgmap v4888: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1877253927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:26 smithi022 ceph-mon[40250]: pgmap v4888: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1877253927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:41:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:28 smithi122 ceph-mon[32255]: pgmap v4889: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:28 smithi022 ceph-mon[31360]: pgmap v4889: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:28 smithi022 ceph-mon[40250]: pgmap v4889: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:30 smithi122 ceph-mon[32255]: pgmap v4890: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:30] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:41:30.491 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:30 smithi022 ceph-mon[31360]: pgmap v4890: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:30.492 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:30 smithi022 ceph-mon[40250]: pgmap v4890: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:31.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:31.411 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:41:31.411 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:41:31.411 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:41:31.411 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:41:31.412 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:41:31.412 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:41:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:41:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:41:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:41:31.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:41:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:41:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:32 smithi122 ceph-mon[32255]: pgmap v4891: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1711332074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:41:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:41:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:32 smithi022 ceph-mon[31360]: pgmap v4891: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1711332074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:32 smithi022 ceph-mon[40250]: pgmap v4891: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1711332074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:41:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:41:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:41:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:41:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:41:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:41:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:41:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:34 smithi022 ceph-mon[31360]: pgmap v4892: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:34 smithi022 ceph-mon[40250]: pgmap v4892: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:34 smithi122 ceph-mon[32255]: pgmap v4892: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:35 smithi122 ceph-mon[32255]: pgmap v4893: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:35 smithi022 ceph-mon[40250]: pgmap v4893: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:35 smithi022 ceph-mon[31360]: pgmap v4893: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:36.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:41:36.412 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:36.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:36.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:36.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2435508240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2435508240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2435508240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:41:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:37 smithi122 ceph-mon[32255]: pgmap v4894: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:37 smithi022 ceph-mon[31360]: pgmap v4894: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:37 smithi022 ceph-mon[40250]: pgmap v4894: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:40] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:41:40.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:40 smithi122 ceph-mon[32255]: pgmap v4895: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:40 smithi022 ceph-mon[31360]: pgmap v4895: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:40 smithi022 ceph-mon[40250]: pgmap v4895: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:41.782 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:41.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:41:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:41:42.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:42.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:42.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:42 smithi122 ceph-mon[32255]: pgmap v4896: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:42 smithi022 ceph-mon[31360]: pgmap v4896: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:42 smithi022 ceph-mon[40250]: pgmap v4896: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:41:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:41:43.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:43.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4044934688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4044934688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4044934688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:44 smithi122 ceph-mon[32255]: pgmap v4897: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:44 smithi022 ceph-mon[31360]: pgmap v4897: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:44 smithi022 ceph-mon[40250]: pgmap v4897: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:46 smithi122 ceph-mon[32255]: pgmap v4898: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:46 smithi022 ceph-mon[31360]: pgmap v4898: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:46 smithi022 ceph-mon[40250]: pgmap v4898: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:47.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:47.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:47.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:47.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:47.526 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:41:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:48 smithi122 ceph-mon[32255]: pgmap v4899: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2546392215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:48 smithi022 ceph-mon[31360]: pgmap v4899: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:48.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2546392215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:48 smithi022 ceph-mon[40250]: pgmap v4899: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2546392215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:41:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:50 smithi122 ceph-mon[32255]: pgmap v4900: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:50 smithi022 ceph-mon[31360]: pgmap v4900: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:50 smithi022 ceph-mon[40250]: pgmap v4900: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:51.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:41:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:41:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:52 smithi122 ceph-mon[32255]: pgmap v4901: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:41:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:41:52.527 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:52 smithi022 ceph-mon[40250]: pgmap v4901: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:52 smithi022 ceph-mon[31360]: pgmap v4901: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:52.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:52.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:52.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4073015067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4073015067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:41:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4073015067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:54 smithi122 ceph-mon[32255]: pgmap v4902: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:54 smithi022 ceph-mon[31360]: pgmap v4902: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:54 smithi022 ceph-mon[40250]: pgmap v4902: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:56 smithi122 ceph-mon[32255]: pgmap v4903: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:56 smithi022 ceph-mon[31360]: pgmap v4903: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:56 smithi022 ceph-mon[40250]: pgmap v4903: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:41:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:41:57.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:41:58.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:41:58.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:41:58.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:41:58.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:58 smithi022 ceph-mon[40250]: pgmap v4904: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:58.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:58 smithi022 ceph-mon[31360]: pgmap v4904: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:58.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:58 smithi122 ceph-mon[32255]: pgmap v4904: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:41:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:41:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1874857692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:41:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1874857692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:41:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:41:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1874857692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:00 smithi122 ceph-mon[32255]: pgmap v4905: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:42:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:00 smithi022 ceph-mon[31360]: pgmap v4905: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:00 smithi022 ceph-mon[40250]: pgmap v4905: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:01.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:42:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:42:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:02 smithi122 ceph-mon[32255]: pgmap v4906: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:42:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:42:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:02 smithi022 ceph-mon[31360]: pgmap v4906: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:02 smithi022 ceph-mon[40250]: pgmap v4906: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:03.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:03.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:03.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:04 smithi122 ceph-mon[32255]: pgmap v4907: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2181584591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:04 smithi022 ceph-mon[31360]: pgmap v4907: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2181584591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:04 smithi022 ceph-mon[40250]: pgmap v4907: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2181584591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:06 smithi122 ceph-mon[32255]: pgmap v4908: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:06 smithi022 ceph-mon[31360]: pgmap v4908: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:06 smithi022 ceph-mon[40250]: pgmap v4908: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:42:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:08 smithi122 ceph-mon[32255]: pgmap v4909: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:08 smithi022 ceph-mon[31360]: pgmap v4909: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:08 smithi022 ceph-mon[40250]: pgmap v4909: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:08.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:09.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:09.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:09.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/928950552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/928950552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/928950552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:42:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:10 smithi122 ceph-mon[32255]: pgmap v4910: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:10 smithi022 ceph-mon[31360]: pgmap v4910: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:10 smithi022 ceph-mon[40250]: pgmap v4910: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:11.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:42:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:42:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:12 smithi122 ceph-mon[32255]: pgmap v4911: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:12.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:42:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:42:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:12 smithi022 ceph-mon[31360]: pgmap v4911: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:12 smithi022 ceph-mon[40250]: pgmap v4911: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:14.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:14 smithi022 ceph-mon[31360]: pgmap v4912: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:14 smithi022 ceph-mon[40250]: pgmap v4912: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:14 smithi122 ceph-mon[32255]: pgmap v4912: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2132690787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2132690787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2132690787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:16 smithi122 ceph-mon[32255]: pgmap v4913: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:16 smithi022 ceph-mon[31360]: pgmap v4913: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:16 smithi022 ceph-mon[40250]: pgmap v4913: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:42:18.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:18 smithi122 ceph-mon[32255]: pgmap v4914: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:18 smithi022 ceph-mon[31360]: pgmap v4914: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:18 smithi022 ceph-mon[40250]: pgmap v4914: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:19.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:19.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:19.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:19.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:20] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:42:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:20 smithi122 ceph-mon[32255]: pgmap v4915: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4279550656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:20 smithi022 ceph-mon[31360]: pgmap v4915: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4279550656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:20 smithi022 ceph-mon[40250]: pgmap v4915: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4279550656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:21.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:42:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:42:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:22 smithi122 ceph-mon[32255]: pgmap v4916: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:42:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:42:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:22 smithi022 ceph-mon[31360]: pgmap v4916: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:22 smithi022 ceph-mon[40250]: pgmap v4916: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:24.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:24 smithi122 ceph-mon[32255]: pgmap v4917: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:24 smithi022 ceph-mon[31360]: pgmap v4917: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:24 smithi022 ceph-mon[40250]: pgmap v4917: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:24.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:25.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:25.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:25.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:25.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1855337227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1855337227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1855337227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:26 smithi122 ceph-mon[32255]: pgmap v4918: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:26 smithi022 ceph-mon[31360]: pgmap v4918: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:26 smithi022 ceph-mon[40250]: pgmap v4918: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:42:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:28 smithi122 ceph-mon[32255]: pgmap v4919: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:28 smithi022 ceph-mon[31360]: pgmap v4919: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:28 smithi022 ceph-mon[40250]: pgmap v4919: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:30.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:30 smithi122 ceph-mon[32255]: pgmap v4920: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:30] "GET /metrics HTTP/1.1" 200 202333 "" "Prometheus/2.18.1" 2021-12-15T20:42:30.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:30.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:30.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:30.498 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:30 smithi022 ceph-mon[31360]: pgmap v4920: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:30.499 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:30 smithi022 ceph-mon[40250]: pgmap v4920: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1423493275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1423493275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1423493275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:31.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:42:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:42:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:32 smithi122 ceph-mon[32255]: pgmap v4921: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:42:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:42:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:32 smithi022 ceph-mon[31360]: pgmap v4921: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:32 smithi022 ceph-mon[40250]: pgmap v4921: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:42:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:42:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:42:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:42:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:42:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:42:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:42:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:42:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:42:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:42:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:42:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:42:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:34 smithi122 ceph-mon[32255]: pgmap v4922: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:34 smithi022 ceph-mon[31360]: pgmap v4922: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:34 smithi022 ceph-mon[40250]: pgmap v4922: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:42:35.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:35.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:35.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:35.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:36.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:36 smithi122 ceph-mon[32255]: pgmap v4923: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:36.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:42:36.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:42:36.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:42:36.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:42:36.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/311023700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[31360]: pgmap v4923: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:42:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:42:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:42:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:42:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/311023700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[40250]: pgmap v4923: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:42:36.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:42:36.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:42:36.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:42:36.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/311023700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:42:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:38 smithi122 ceph-mon[32255]: pgmap v4924: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:38 smithi022 ceph-mon[31360]: pgmap v4924: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:38 smithi022 ceph-mon[40250]: pgmap v4924: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:40] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:42:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:40 smithi122 ceph-mon[32255]: pgmap v4925: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:40 smithi022 ceph-mon[31360]: pgmap v4925: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:40 smithi022 ceph-mon[40250]: pgmap v4925: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:40.863 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:41.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:41.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:41.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:42.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:42:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:42:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:42 smithi122 ceph-mon[32255]: pgmap v4926: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4028089881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:42:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:42:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:42 smithi022 ceph-mon[31360]: pgmap v4926: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4028089881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:42 smithi022 ceph-mon[40250]: pgmap v4926: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4028089881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:44 smithi122 ceph-mon[32255]: pgmap v4927: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:44 smithi022 ceph-mon[31360]: pgmap v4927: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:44 smithi022 ceph-mon[40250]: pgmap v4927: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:46.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:46 smithi122 ceph-mon[32255]: pgmap v4928: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:46 smithi022 ceph-mon[31360]: pgmap v4928: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:46 smithi022 ceph-mon[40250]: pgmap v4928: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:46.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:46.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:46.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3576425032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3576425032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:42:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3576425032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:48 smithi122 ceph-mon[32255]: pgmap v4929: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:48 smithi022 ceph-mon[31360]: pgmap v4929: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:48 smithi022 ceph-mon[40250]: pgmap v4929: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:50] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:42:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:50 smithi122 ceph-mon[32255]: pgmap v4930: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:50 smithi022 ceph-mon[31360]: pgmap v4930: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:50 smithi022 ceph-mon[40250]: pgmap v4930: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:51.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:51.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:52.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:42:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:42:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:52 smithi022 ceph-mon[31360]: pgmap v4931: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2567914338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:52 smithi022 ceph-mon[40250]: pgmap v4931: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2567914338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:42:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:42:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:52 smithi122 ceph-mon[32255]: pgmap v4931: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2567914338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:53.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:42:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:54 smithi122 ceph-mon[32255]: pgmap v4932: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:54 smithi022 ceph-mon[31360]: pgmap v4932: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:54 smithi022 ceph-mon[40250]: pgmap v4932: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:56 smithi122 ceph-mon[32255]: pgmap v4933: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:56 smithi022 ceph-mon[31360]: pgmap v4933: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:56 smithi022 ceph-mon[40250]: pgmap v4933: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:42:56.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:42:57.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:42:57.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:42:57.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:42:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:42:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:58 smithi122 ceph-mon[32255]: pgmap v4934: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:42:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2681617225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:58.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:58 smithi022 ceph-mon[31360]: pgmap v4934: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:58.539 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:42:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2681617225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:42:58.539 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:58 smithi022 ceph-mon[40250]: pgmap v4934: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:42:58.540 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:42:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2681617225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:00 smithi122 ceph-mon[32255]: pgmap v4935: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:00] "GET /metrics HTTP/1.1" 200 202336 "" "Prometheus/2.18.1" 2021-12-15T20:43:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:00 smithi022 ceph-mon[31360]: pgmap v4935: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:00 smithi022 ceph-mon[40250]: pgmap v4935: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:02.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:43:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:43:02.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:02 smithi022 ceph-mon[31360]: pgmap v4936: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:02 smithi022 ceph-mon[40250]: pgmap v4936: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:02 smithi122 ceph-mon[32255]: pgmap v4936: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:43:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:43:02.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:02.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:02.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2319845200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:03.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2319845200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2319845200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:04 smithi022 ceph-mon[31360]: pgmap v4937: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:04 smithi022 ceph-mon[40250]: pgmap v4937: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:04 smithi122 ceph-mon[32255]: pgmap v4937: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:06 smithi022 ceph-mon[31360]: pgmap v4938: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:06 smithi022 ceph-mon[40250]: pgmap v4938: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:06 smithi122 ceph-mon[32255]: pgmap v4938: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:07.715 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:43:07.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:08.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:08.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:08.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:08 smithi022 ceph-mon[31360]: pgmap v4939: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/975585377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:08 smithi022 ceph-mon[40250]: pgmap v4939: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/975585377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:08 smithi122 ceph-mon[32255]: pgmap v4939: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/975585377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:10 smithi122 ceph-mon[32255]: pgmap v4940: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:10.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:43:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:10 smithi022 ceph-mon[31360]: pgmap v4940: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:10 smithi022 ceph-mon[40250]: pgmap v4940: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:12.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:43:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:43:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:12 smithi022 ceph-mon[31360]: pgmap v4941: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:12 smithi022 ceph-mon[40250]: pgmap v4941: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:12 smithi122 ceph-mon[32255]: pgmap v4941: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:43:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:43:13.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:13.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:13.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:13.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:13.462 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:13.463 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:14 smithi022 ceph-mon[31360]: pgmap v4942: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:14.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2599483437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:14.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:14 smithi022 ceph-mon[40250]: pgmap v4942: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:14.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2599483437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:14 smithi122 ceph-mon[32255]: pgmap v4942: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2599483437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:16 smithi022 ceph-mon[31360]: pgmap v4943: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:16 smithi022 ceph-mon[40250]: pgmap v4943: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:16 smithi122 ceph-mon[32255]: pgmap v4943: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:43:18.465 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:18.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:18 smithi022 ceph-mon[31360]: pgmap v4944: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:18.466 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:18 smithi022 ceph-mon[40250]: pgmap v4944: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:18 smithi122 ceph-mon[32255]: pgmap v4944: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:18.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:18.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:18.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4190832421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4190832421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:20 smithi122 ceph-mon[32255]: pgmap v4945: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:20] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:43:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:20 smithi022 ceph-mon[31360]: pgmap v4945: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4190832421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:20 smithi022 ceph-mon[40250]: pgmap v4945: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:22.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:43:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:43:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:22 smithi022 ceph-mon[31360]: pgmap v4946: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:22 smithi022 ceph-mon[40250]: pgmap v4946: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:22 smithi122 ceph-mon[32255]: pgmap v4946: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:43:22.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:43:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:23.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:24.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:24.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:24.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:24 smithi022 ceph-mon[31360]: pgmap v4947: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/733803682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:24 smithi022 ceph-mon[40250]: pgmap v4947: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/733803682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:24 smithi122 ceph-mon[32255]: pgmap v4947: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/733803682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:26 smithi022 ceph-mon[31360]: pgmap v4948: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:26 smithi022 ceph-mon[40250]: pgmap v4948: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:26 smithi122 ceph-mon[32255]: pgmap v4948: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:43:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:28 smithi022 ceph-mon[31360]: pgmap v4949: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:28 smithi022 ceph-mon[40250]: pgmap v4949: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:28 smithi122 ceph-mon[32255]: pgmap v4949: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:29.210 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:29 smithi022 ceph-mon[31360]: pgmap v4950: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:29 smithi022 ceph-mon[40250]: pgmap v4950: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:29.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:29.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:29.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:29 smithi122 ceph-mon[32255]: pgmap v4950: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:30] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:43:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3274797890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3274797890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3274797890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:31 smithi122 ceph-mon[32255]: pgmap v4951: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:31 smithi022 ceph-mon[31360]: pgmap v4951: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:31 smithi022 ceph-mon[40250]: pgmap v4951: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:32.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:43:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:43:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:43:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:43:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:43:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:43:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:43:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:43:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[31360]: pgmap v4952: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:43:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:43:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:43:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:43:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:33 smithi022 ceph-mon[40250]: pgmap v4952: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:43:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:43:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:43:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:43:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:33 smithi122 ceph-mon[32255]: pgmap v4952: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:34.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:34.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:34.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:34.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/698950883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/698950883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/698950883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:35 smithi122 ceph-mon[32255]: pgmap v4953: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:43:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:43:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:43:36.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[40250]: pgmap v4953: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:36.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:43:36.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:43:36.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:43:36.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[31360]: pgmap v4953: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:36.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:43:36.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:43:36.229 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:43:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:43:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:43:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:37 smithi022 ceph-mon[31360]: pgmap v4954: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:43:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:37 smithi022 ceph-mon[40250]: pgmap v4954: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:43:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:37 smithi122 ceph-mon[32255]: pgmap v4954: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:39.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:40 smithi022 ceph-mon[31360]: pgmap v4955: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:40 smithi022 ceph-mon[40250]: pgmap v4955: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:40.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:40.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:40.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:40] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:43:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:40 smithi122 ceph-mon[32255]: pgmap v4955: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3147858281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3147858281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3147858281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:42.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:43:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:43:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:42 smithi122 ceph-mon[32255]: pgmap v4956: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:42.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:43:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:43:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:42 smithi022 ceph-mon[31360]: pgmap v4956: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:42 smithi022 ceph-mon[40250]: pgmap v4956: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:44 smithi122 ceph-mon[32255]: pgmap v4957: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:44 smithi022 ceph-mon[31360]: pgmap v4957: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:44 smithi022 ceph-mon[40250]: pgmap v4957: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:45.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:45.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:45.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:45.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:46 smithi122 ceph-mon[32255]: pgmap v4958: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/541267905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:46 smithi022 ceph-mon[31360]: pgmap v4958: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/541267905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:46 smithi022 ceph-mon[40250]: pgmap v4958: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/541267905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:43:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:48 smithi122 ceph-mon[32255]: pgmap v4959: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:48 smithi022 ceph-mon[31360]: pgmap v4959: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:48 smithi022 ceph-mon[40250]: pgmap v4959: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:50] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:43:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:50 smithi122 ceph-mon[32255]: pgmap v4960: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:50 smithi022 ceph-mon[31360]: pgmap v4960: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:50 smithi022 ceph-mon[40250]: pgmap v4960: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:50.688 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:51.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:51.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:51.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/810789607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/810789607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/810789607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:52.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:43:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:43:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:52 smithi122 ceph-mon[32255]: pgmap v4961: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:43:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:43:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:52 smithi022 ceph-mon[31360]: pgmap v4961: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:52.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:52 smithi022 ceph-mon[40250]: pgmap v4961: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:43:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:54 smithi122 ceph-mon[32255]: pgmap v4962: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:54 smithi022 ceph-mon[31360]: pgmap v4962: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:54 smithi022 ceph-mon[40250]: pgmap v4962: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:56.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:43:56.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:43:56.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:43:56.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:43:56.432 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:56 smithi022 ceph-mon[31360]: pgmap v4963: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:56.433 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:56 smithi022 ceph-mon[40250]: pgmap v4963: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:56 smithi122 ceph-mon[32255]: pgmap v4963: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:43:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3509616382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3509616382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3509616382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:43:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:43:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:43:58 smithi122 ceph-mon[32255]: pgmap v4964: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:43:58 smithi022 ceph-mon[31360]: pgmap v4964: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:43:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:43:58 smithi022 ceph-mon[40250]: pgmap v4964: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:00] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:44:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:00 smithi122 ceph-mon[32255]: pgmap v4965: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:00 smithi022 ceph-mon[31360]: pgmap v4965: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:00 smithi022 ceph-mon[40250]: pgmap v4965: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:01.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:01.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:01.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:01.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:02.129 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:44:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:44:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:44:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:44:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:02 smithi122 ceph-mon[32255]: pgmap v4966: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1339454535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:02 smithi022 ceph-mon[31360]: pgmap v4966: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1339454535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:02 smithi022 ceph-mon[40250]: pgmap v4966: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1339454535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:04 smithi122 ceph-mon[32255]: pgmap v4967: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:04 smithi022 ceph-mon[31360]: pgmap v4967: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:04 smithi022 ceph-mon[40250]: pgmap v4967: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:06 smithi122 ceph-mon[32255]: pgmap v4968: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:06 smithi022 ceph-mon[31360]: pgmap v4968: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:06 smithi022 ceph-mon[40250]: pgmap v4968: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:06.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:07.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:07.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:07.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:44:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:08 smithi122 ceph-mon[32255]: pgmap v4969: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3850851028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:08 smithi022 ceph-mon[31360]: pgmap v4969: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3850851028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:08 smithi022 ceph-mon[40250]: pgmap v4969: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3850851028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:10 smithi122 ceph-mon[32255]: pgmap v4970: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:44:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:10 smithi022 ceph-mon[31360]: pgmap v4970: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:10 smithi022 ceph-mon[40250]: pgmap v4970: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:12.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:44:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:44:12.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:12 smithi122 ceph-mon[32255]: pgmap v4971: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:44:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:44:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:12 smithi022 ceph-mon[40250]: pgmap v4971: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:12 smithi022 ceph-mon[31360]: pgmap v4971: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:12.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:12.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:12.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2246812517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2246812517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2246812517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:14 smithi122 ceph-mon[32255]: pgmap v4972: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:14 smithi022 ceph-mon[31360]: pgmap v4972: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:14 smithi022 ceph-mon[40250]: pgmap v4972: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:16 smithi122 ceph-mon[32255]: pgmap v4973: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:16 smithi022 ceph-mon[31360]: pgmap v4973: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:16 smithi022 ceph-mon[40250]: pgmap v4973: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:17.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:44:17.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:17.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:17.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:18 smithi022 ceph-mon[31360]: pgmap v4974: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2218970604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:18 smithi022 ceph-mon[40250]: pgmap v4974: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2218970604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:18 smithi122 ceph-mon[32255]: pgmap v4974: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2218970604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:20 smithi122 ceph-mon[32255]: pgmap v4975: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:20] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:44:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:20 smithi022 ceph-mon[31360]: pgmap v4975: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:20 smithi022 ceph-mon[40250]: pgmap v4975: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:22.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:44:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:44:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:22 smithi122 ceph-mon[32255]: pgmap v4976: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:44:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:44:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:22 smithi022 ceph-mon[31360]: pgmap v4976: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:22 smithi022 ceph-mon[40250]: pgmap v4976: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:22.924 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:23.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:23.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:23.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:24 smithi122 ceph-mon[32255]: pgmap v4977: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1673467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:24 smithi022 ceph-mon[31360]: pgmap v4977: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1673467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:24 smithi022 ceph-mon[40250]: pgmap v4977: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1673467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:26 smithi122 ceph-mon[32255]: pgmap v4978: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:26 smithi022 ceph-mon[31360]: pgmap v4978: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:26 smithi022 ceph-mon[40250]: pgmap v4978: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:44:28.297 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:28 smithi122 ceph-mon[32255]: pgmap v4979: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:28 smithi022 ceph-mon[31360]: pgmap v4979: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:28 smithi022 ceph-mon[40250]: pgmap v4979: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:28.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:28.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:28.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3541845269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3541845269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3541845269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:30] "GET /metrics HTTP/1.1" 200 202332 "" "Prometheus/2.18.1" 2021-12-15T20:44:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:30 smithi122 ceph-mon[32255]: pgmap v4980: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:30 smithi022 ceph-mon[31360]: pgmap v4980: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:30 smithi022 ceph-mon[40250]: pgmap v4980: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:32.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:44:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:44:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:32 smithi122 ceph-mon[32255]: pgmap v4981: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:44:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:44:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:32 smithi022 ceph-mon[31360]: pgmap v4981: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:32 smithi022 ceph-mon[40250]: pgmap v4981: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:44:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:44:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:44:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:44:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:44:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:44:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:44:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:44:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:44:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:44:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:44:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:44:33.666 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:34.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:34.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:34.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:34 smithi122 ceph-mon[32255]: pgmap v4982: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1564604591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:34 smithi022 ceph-mon[31360]: pgmap v4982: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1564604591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:34 smithi022 ceph-mon[40250]: pgmap v4982: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1564604591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:36 smithi122 ceph-mon[32255]: pgmap v4983: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:36 smithi022 ceph-mon[31360]: pgmap v4983: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:36 smithi022 ceph-mon[40250]: pgmap v4983: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:44:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:44:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:44:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:44:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:44:37.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:44:37.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:44:37.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:44:37.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:44:37.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:44:37.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:44:37.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:44:37.537 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:44:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:38 smithi122 ceph-mon[32255]: pgmap v4984: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:38 smithi022 ceph-mon[31360]: pgmap v4984: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:38 smithi022 ceph-mon[40250]: pgmap v4984: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:39.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:39.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:39.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:39.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:40] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:44:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:40 smithi122 ceph-mon[32255]: pgmap v4985: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3288682724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:40 smithi022 ceph-mon[31360]: pgmap v4985: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3288682724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:40 smithi022 ceph-mon[40250]: pgmap v4985: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3288682724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:42.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:44:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:44:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:42 smithi022 ceph-mon[31360]: pgmap v4986: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:42 smithi022 ceph-mon[40250]: pgmap v4986: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:42 smithi122 ceph-mon[32255]: pgmap v4986: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:44:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:44:43.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:43.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:44.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:44 smithi022 ceph-mon[31360]: pgmap v4987: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:44 smithi022 ceph-mon[40250]: pgmap v4987: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:44 smithi122 ceph-mon[32255]: pgmap v4987: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:44.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:44.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:44.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/198946661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/198946661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/198946661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:46 smithi022 ceph-mon[31360]: pgmap v4988: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:46 smithi022 ceph-mon[40250]: pgmap v4988: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:46 smithi122 ceph-mon[32255]: pgmap v4988: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:44:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:48 smithi022 ceph-mon[31360]: pgmap v4989: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:48 smithi022 ceph-mon[40250]: pgmap v4989: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:48 smithi122 ceph-mon[32255]: pgmap v4989: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:49.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:50.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:50.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:50.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:50] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:44:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:50 smithi122 ceph-mon[32255]: pgmap v4990: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2241987680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:50 smithi022 ceph-mon[40250]: pgmap v4990: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2241987680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:50 smithi022 ceph-mon[31360]: pgmap v4990: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2241987680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:52.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:44:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:44:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:52 smithi022 ceph-mon[40250]: pgmap v4991: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:52 smithi022 ceph-mon[31360]: pgmap v4991: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:44:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:44:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:52 smithi122 ceph-mon[32255]: pgmap v4991: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:44:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:54 smithi022 ceph-mon[40250]: pgmap v4992: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:54 smithi022 ceph-mon[31360]: pgmap v4992: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:54 smithi122 ceph-mon[32255]: pgmap v4992: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:55.149 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:44:55.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:44:55.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:44:55.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:44:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:56 smithi022 ceph-mon[31360]: pgmap v4993: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2164513609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:56 smithi022 ceph-mon[40250]: pgmap v4993: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2164513609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:56 smithi122 ceph-mon[32255]: pgmap v4993: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:44:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2164513609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:44:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:44:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:44:58 smithi022 ceph-mon[31360]: pgmap v4994: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:44:58 smithi022 ceph-mon[40250]: pgmap v4994: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:44:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:44:58 smithi122 ceph-mon[32255]: pgmap v4994: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:00] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:45:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:00 smithi122 ceph-mon[32255]: pgmap v4995: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:00.522 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:00.523 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:00 smithi022 ceph-mon[31360]: pgmap v4995: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:00.523 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:00 smithi022 ceph-mon[40250]: pgmap v4995: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:00.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:00.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:00.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1909929540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1909929540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1909929540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:02.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:45:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:45:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:02 smithi022 ceph-mon[31360]: pgmap v4996: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:02 smithi022 ceph-mon[40250]: pgmap v4996: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:02 smithi122 ceph-mon[32255]: pgmap v4996: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:45:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:45:03.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:03.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:04 smithi022 ceph-mon[31360]: pgmap v4997: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:04 smithi022 ceph-mon[40250]: pgmap v4997: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:04 smithi122 ceph-mon[32255]: pgmap v4997: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:05.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:06.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:06.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:06.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:06 smithi022 ceph-mon[31360]: pgmap v4998: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:06 smithi022 ceph-mon[40250]: pgmap v4998: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:06 smithi122 ceph-mon[32255]: pgmap v4998: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/94660585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:45:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/94660585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/94660585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:08 smithi022 ceph-mon[31360]: pgmap v4999: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:08 smithi022 ceph-mon[40250]: pgmap v4999: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:08 smithi122 ceph-mon[32255]: pgmap v4999: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:10 smithi122 ceph-mon[32255]: pgmap v5000: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:10] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:45:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:10 smithi022 ceph-mon[31360]: pgmap v5000: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:10 smithi022 ceph-mon[40250]: pgmap v5000: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:11.261 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:11.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:11.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:11.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:12.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:45:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:45:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:12 smithi022 ceph-mon[31360]: pgmap v5001: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/386792152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:12 smithi022 ceph-mon[40250]: pgmap v5001: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/386792152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:12 smithi122 ceph-mon[32255]: pgmap v5001: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/386792152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:45:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:45:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:14 smithi022 ceph-mon[31360]: pgmap v5002: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:14 smithi022 ceph-mon[40250]: pgmap v5002: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:14 smithi122 ceph-mon[32255]: pgmap v5002: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:16 smithi022 ceph-mon[31360]: pgmap v5003: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:16 smithi022 ceph-mon[40250]: pgmap v5003: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:16.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:16 smithi122 ceph-mon[32255]: pgmap v5003: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:17.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:17.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:17.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3621441736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:17 smithi022 ceph-mon[31360]: pgmap v5004: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3621441736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:17 smithi022 ceph-mon[40250]: pgmap v5004: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3621441736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:17 smithi122 ceph-mon[32255]: pgmap v5004: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:45:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:20] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:45:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:20 smithi122 ceph-mon[32255]: pgmap v5005: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:20 smithi022 ceph-mon[31360]: pgmap v5005: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:20 smithi022 ceph-mon[40250]: pgmap v5005: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:22.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:22.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:45:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:45:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:22 smithi022 ceph-mon[31360]: pgmap v5006: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:22 smithi022 ceph-mon[40250]: pgmap v5006: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:22.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:22.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:22.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:22 smithi122 ceph-mon[32255]: pgmap v5006: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:45:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:45:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/18244262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/18244262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/18244262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:24 smithi122 ceph-mon[32255]: pgmap v5007: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:24 smithi022 ceph-mon[31360]: pgmap v5007: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:24 smithi022 ceph-mon[40250]: pgmap v5007: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:26 smithi122 ceph-mon[32255]: pgmap v5008: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:26 smithi022 ceph-mon[31360]: pgmap v5008: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:26 smithi022 ceph-mon[40250]: pgmap v5008: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:27.377 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:27.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:27.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:27.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:27.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:45:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:28 smithi122 ceph-mon[32255]: pgmap v5009: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2640234961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:28 smithi022 ceph-mon[31360]: pgmap v5009: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2640234961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:28 smithi022 ceph-mon[40250]: pgmap v5009: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2640234961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:30] "GET /metrics HTTP/1.1" 200 202338 "" "Prometheus/2.18.1" 2021-12-15T20:45:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:30 smithi122 ceph-mon[32255]: pgmap v5010: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:30 smithi022 ceph-mon[31360]: pgmap v5010: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:30 smithi022 ceph-mon[40250]: pgmap v5010: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:32.136 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:45:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:45:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:32 smithi122 ceph-mon[32255]: pgmap v5011: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:45:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:45:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:32 smithi022 ceph-mon[31360]: pgmap v5011: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:32 smithi022 ceph-mon[40250]: pgmap v5011: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:32.748 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:33.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:33.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:33.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:45:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:45:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:45:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:45:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/560960095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:45:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:45:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:45:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:45:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/560960095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:45:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:45:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:45:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:45:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/560960095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:34 smithi122 ceph-mon[32255]: pgmap v5012: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:34.834 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:34 smithi022 ceph-mon[31360]: pgmap v5012: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:34.835 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:34 smithi022 ceph-mon[40250]: pgmap v5012: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:36 smithi122 ceph-mon[32255]: pgmap v5013: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:36 smithi022 ceph-mon[31360]: pgmap v5013: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:36 smithi022 ceph-mon[40250]: pgmap v5013: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:45:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:45:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:45:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:45:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:45:37.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:45:37.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:45:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:45:37.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:45:37.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:45:38.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:38 smithi122 ceph-mon[32255]: pgmap v5014: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:45:38.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:38.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:38.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:38.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:38 smithi022 ceph-mon[31360]: pgmap v5014: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:38.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:45:38.489 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:38 smithi022 ceph-mon[40250]: pgmap v5014: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:38.490 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:45:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1727050099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1727050099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1727050099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:40 smithi122 ceph-mon[32255]: pgmap v5015: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:40] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:45:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:40 smithi022 ceph-mon[31360]: pgmap v5015: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:40 smithi022 ceph-mon[40250]: pgmap v5015: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:42.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:45:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:45:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:42 smithi122 ceph-mon[32255]: pgmap v5016: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:45:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:45:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:42 smithi022 ceph-mon[31360]: pgmap v5016: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:42 smithi022 ceph-mon[40250]: pgmap v5016: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:43.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:43.490 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:43.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:43.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:43.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:43.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:44 smithi122 ceph-mon[32255]: pgmap v5017: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/781282119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:44 smithi022 ceph-mon[31360]: pgmap v5017: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/781282119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:44 smithi022 ceph-mon[40250]: pgmap v5017: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/781282119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:46 smithi122 ceph-mon[32255]: pgmap v5018: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:46 smithi022 ceph-mon[31360]: pgmap v5018: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:46 smithi022 ceph-mon[40250]: pgmap v5018: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:45:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:48 smithi122 ceph-mon[32255]: pgmap v5019: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:48 smithi022 ceph-mon[31360]: pgmap v5019: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:48 smithi022 ceph-mon[40250]: pgmap v5019: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:48.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:49.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:49.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:50] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:45:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:50 smithi122 ceph-mon[32255]: pgmap v5020: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2608508102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:50 smithi022 ceph-mon[31360]: pgmap v5020: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2608508102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:50 smithi022 ceph-mon[40250]: pgmap v5020: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2608508102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:52.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:45:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:45:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:52 smithi022 ceph-mon[31360]: pgmap v5021: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:52 smithi022 ceph-mon[40250]: pgmap v5021: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:45:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:45:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:52 smithi122 ceph-mon[32255]: pgmap v5021: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:45:54.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:54 smithi022 ceph-mon[31360]: pgmap v5022: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:54 smithi022 ceph-mon[40250]: pgmap v5022: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:54.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:54.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:54.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:45:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:54 smithi122 ceph-mon[32255]: pgmap v5022: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:45:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2314257993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2314257993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2314257993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:45:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:56 smithi022 ceph-mon[31360]: pgmap v5023: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:56 smithi022 ceph-mon[40250]: pgmap v5023: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:56 smithi122 ceph-mon[32255]: pgmap v5023: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:45:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:45:58 smithi022 ceph-mon[31360]: pgmap v5024: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:45:58 smithi022 ceph-mon[40250]: pgmap v5024: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:45:58 smithi122 ceph-mon[32255]: pgmap v5024: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:45:59.599 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:45:59.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:45:59.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:45:59.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:00 smithi022 ceph-mon[31360]: pgmap v5025: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3349097620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:00 smithi022 ceph-mon[40250]: pgmap v5025: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3349097620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:00] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:46:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:00 smithi122 ceph-mon[32255]: pgmap v5025: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3349097620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:02.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:46:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:46:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:02 smithi022 ceph-mon[31360]: pgmap v5026: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:02 smithi022 ceph-mon[40250]: pgmap v5026: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:02 smithi122 ceph-mon[32255]: pgmap v5026: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:46:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:46:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:04 smithi022 ceph-mon[31360]: pgmap v5027: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:04 smithi022 ceph-mon[40250]: pgmap v5027: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:04 smithi122 ceph-mon[32255]: pgmap v5027: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:04.969 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:05.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:05.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:05.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:06 smithi022 ceph-mon[31360]: pgmap v5028: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1256266212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:06 smithi022 ceph-mon[40250]: pgmap v5028: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1256266212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:06 smithi122 ceph-mon[32255]: pgmap v5028: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1256266212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:46:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:08 smithi022 ceph-mon[31360]: pgmap v5029: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:08 smithi022 ceph-mon[40250]: pgmap v5029: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:08 smithi122 ceph-mon[32255]: pgmap v5029: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:10.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:10 smithi122 ceph-mon[32255]: pgmap v5030: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:10] "GET /metrics HTTP/1.1" 200 202334 "" "Prometheus/2.18.1" 2021-12-15T20:46:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:10 smithi022 ceph-mon[31360]: pgmap v5030: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:10 smithi022 ceph-mon[40250]: pgmap v5030: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:10.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:10.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:10.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2506369828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2506369828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2506369828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:12.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:46:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:46:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:12 smithi022 ceph-mon[31360]: pgmap v5031: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:12 smithi022 ceph-mon[40250]: pgmap v5031: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:12 smithi122 ceph-mon[32255]: pgmap v5031: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:46:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:46:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:14 smithi022 ceph-mon[31360]: pgmap v5032: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:14 smithi022 ceph-mon[40250]: pgmap v5032: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:14 smithi122 ceph-mon[32255]: pgmap v5032: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:15.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:16.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:16.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:16.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:16 smithi022 ceph-mon[31360]: pgmap v5033: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/64959883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:16 smithi022 ceph-mon[40250]: pgmap v5033: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/64959883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:16 smithi122 ceph-mon[32255]: pgmap v5033: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/64959883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:46:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:18 smithi022 ceph-mon[31360]: pgmap v5034: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:18 smithi022 ceph-mon[40250]: pgmap v5034: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:18 smithi122 ceph-mon[32255]: pgmap v5034: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:20 smithi122 ceph-mon[32255]: pgmap v5035: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:46:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:20 smithi022 ceph-mon[31360]: pgmap v5035: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:20 smithi022 ceph-mon[40250]: pgmap v5035: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:21.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:21.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:21.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:21.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:22.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:46:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:46:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:22 smithi022 ceph-mon[31360]: pgmap v5036: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2904865479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:22 smithi022 ceph-mon[40250]: pgmap v5036: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2904865479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:22 smithi122 ceph-mon[32255]: pgmap v5036: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2904865479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:46:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:46:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:24 smithi022 ceph-mon[31360]: pgmap v5037: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:24 smithi022 ceph-mon[40250]: pgmap v5037: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:24 smithi122 ceph-mon[32255]: pgmap v5037: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:26 smithi022 ceph-mon[31360]: pgmap v5038: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:26 smithi022 ceph-mon[40250]: pgmap v5038: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:26 smithi122 ceph-mon[32255]: pgmap v5038: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:26.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:26.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:26.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1253819442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:46:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1253819442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1253819442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:28 smithi022 ceph-mon[31360]: pgmap v5039: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:28 smithi022 ceph-mon[40250]: pgmap v5039: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:28 smithi122 ceph-mon[32255]: pgmap v5039: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:30 smithi122 ceph-mon[32255]: pgmap v5040: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:46:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:30 smithi022 ceph-mon[31360]: pgmap v5040: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:30 smithi022 ceph-mon[40250]: pgmap v5040: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:32.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:46:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:46:32.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:32.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:32.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:32 smithi022 ceph-mon[31360]: pgmap v5041: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3711490544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:32 smithi022 ceph-mon[40250]: pgmap v5041: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3711490544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:46:32.313Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:46:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:32 smithi122 ceph-mon[32255]: pgmap v5041: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3711490544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:46:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:46:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:46:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:46:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:46:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:46:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:46:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:46:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:46:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:46:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:46:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:46:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:34 smithi022 ceph-mon[31360]: pgmap v5042: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:34 smithi022 ceph-mon[40250]: pgmap v5042: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:34 smithi122 ceph-mon[32255]: pgmap v5042: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:35 smithi122 ceph-mon[32255]: pgmap v5043: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:35 smithi022 ceph-mon[31360]: pgmap v5043: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:35.802 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:35 smithi022 ceph-mon[40250]: pgmap v5043: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:37.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:46:37.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:37.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:37.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:38 smithi122 ceph-mon[32255]: pgmap v5044: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:46:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:46:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:46:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/836379863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:46:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[40250]: pgmap v5044: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:46:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:46:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:46:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/836379863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:46:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[31360]: pgmap v5044: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:46:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:46:38.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:46:38.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/836379863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:38.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:46:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:40 smithi122 ceph-mon[32255]: pgmap v5045: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:40] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:46:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:40 smithi022 ceph-mon[31360]: pgmap v5045: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:40 smithi022 ceph-mon[40250]: pgmap v5045: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:42.148 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:46:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:46:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:42 smithi122 ceph-mon[32255]: pgmap v5046: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:46:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:46:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:42 smithi022 ceph-mon[31360]: pgmap v5046: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:42 smithi022 ceph-mon[40250]: pgmap v5046: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:42.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:42.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:42.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:42.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1529275764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1529275764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1529275764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:44 smithi122 ceph-mon[32255]: pgmap v5047: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:44 smithi022 ceph-mon[31360]: pgmap v5047: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:44 smithi022 ceph-mon[40250]: pgmap v5047: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:46 smithi122 ceph-mon[32255]: pgmap v5048: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:46 smithi022 ceph-mon[31360]: pgmap v5048: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:46 smithi022 ceph-mon[40250]: pgmap v5048: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:46:47.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:48 smithi022 ceph-mon[31360]: pgmap v5049: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:48 smithi022 ceph-mon[40250]: pgmap v5049: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:48.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:48.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:48.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:48 smithi122 ceph-mon[32255]: pgmap v5049: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/208338319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/208338319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/208338319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:50] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:46:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:50 smithi122 ceph-mon[32255]: pgmap v5050: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:50 smithi022 ceph-mon[31360]: pgmap v5050: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:50 smithi022 ceph-mon[40250]: pgmap v5050: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:52.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:46:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:46:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:52 smithi122 ceph-mon[32255]: pgmap v5051: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:46:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:46:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:52 smithi022 ceph-mon[31360]: pgmap v5051: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:52 smithi022 ceph-mon[40250]: pgmap v5051: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:53.314 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:46:53.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:53.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:53.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:54 smithi122 ceph-mon[32255]: pgmap v5052: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2314448036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:54 smithi022 ceph-mon[31360]: pgmap v5052: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2314448036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:54 smithi022 ceph-mon[40250]: pgmap v5052: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:46:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2314448036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:56 smithi122 ceph-mon[32255]: pgmap v5053: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:56 smithi022 ceph-mon[31360]: pgmap v5053: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:56 smithi022 ceph-mon[40250]: pgmap v5053: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:46:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:58 smithi122 ceph-mon[32255]: pgmap v5054: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:58 smithi022 ceph-mon[40250]: pgmap v5054: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:58 smithi022 ceph-mon[31360]: pgmap v5054: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:46:58.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:46:59.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:46:59.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:46:59.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:46:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:46:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/650901853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:46:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/650901853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:46:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:46:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/650901853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:00] "GET /metrics HTTP/1.1" 200 202331 "" "Prometheus/2.18.1" 2021-12-15T20:47:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:00 smithi122 ceph-mon[32255]: pgmap v5055: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:00 smithi022 ceph-mon[31360]: pgmap v5055: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:00 smithi022 ceph-mon[40250]: pgmap v5055: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:02.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:47:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:47:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:02 smithi122 ceph-mon[32255]: pgmap v5056: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:47:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:47:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:02 smithi022 ceph-mon[31360]: pgmap v5056: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:02 smithi022 ceph-mon[40250]: pgmap v5056: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:04.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:04.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:04.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:04.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:04.420 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:04 smithi022 ceph-mon[40250]: pgmap v5057: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:04.421 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:04 smithi022 ceph-mon[31360]: pgmap v5057: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:04 smithi122 ceph-mon[32255]: pgmap v5057: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1048503442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1048503442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1048503442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:06 smithi022 ceph-mon[31360]: pgmap v5058: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:06 smithi022 ceph-mon[40250]: pgmap v5058: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:06 smithi122 ceph-mon[32255]: pgmap v5058: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:47:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:08 smithi022 ceph-mon[31360]: pgmap v5059: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:08 smithi022 ceph-mon[40250]: pgmap v5059: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:08 smithi122 ceph-mon[32255]: pgmap v5059: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:09.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:09.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:09.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:10.194 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:10 smithi022 ceph-mon[40250]: pgmap v5060: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:10.194 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1936367087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:10] "GET /metrics HTTP/1.1" 200 202327 "" "Prometheus/2.18.1" 2021-12-15T20:47:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:10 smithi122 ceph-mon[32255]: pgmap v5060: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1936367087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:10 smithi022 ceph-mon[31360]: pgmap v5060: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1936367087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:12.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:47:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:47:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:12 smithi022 ceph-mon[31360]: pgmap v5061: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:12 smithi022 ceph-mon[40250]: pgmap v5061: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:12 smithi122 ceph-mon[32255]: pgmap v5061: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:47:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:47:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:14 smithi022 ceph-mon[31360]: pgmap v5062: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:14 smithi022 ceph-mon[40250]: pgmap v5062: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:14 smithi122 ceph-mon[32255]: pgmap v5062: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:14.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:15.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:15.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:15.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2793348095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2793348095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2793348095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:16 smithi022 ceph-mon[31360]: pgmap v5063: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:16 smithi022 ceph-mon[40250]: pgmap v5063: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:16 smithi122 ceph-mon[32255]: pgmap v5063: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:47:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:18 smithi022 ceph-mon[31360]: pgmap v5064: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:18 smithi022 ceph-mon[40250]: pgmap v5064: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:18 smithi122 ceph-mon[32255]: pgmap v5064: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:20.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:20 smithi122 ceph-mon[32255]: pgmap v5065: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:20] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:47:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:20 smithi022 ceph-mon[31360]: pgmap v5065: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:20 smithi022 ceph-mon[40250]: pgmap v5065: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:20.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:20.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:20.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/983339700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/983339700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/983339700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:22.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:47:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:47:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:22 smithi022 ceph-mon[31360]: pgmap v5066: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:22 smithi022 ceph-mon[40250]: pgmap v5066: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:22 smithi122 ceph-mon[32255]: pgmap v5066: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:47:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:47:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:24 smithi022 ceph-mon[31360]: pgmap v5067: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:24 smithi022 ceph-mon[40250]: pgmap v5067: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:24 smithi122 ceph-mon[32255]: pgmap v5067: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:25.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:25.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:25.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:25.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:26 smithi022 ceph-mon[31360]: pgmap v5068: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3798032736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:26 smithi022 ceph-mon[40250]: pgmap v5068: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3798032736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:26 smithi122 ceph-mon[32255]: pgmap v5068: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3798032736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:47:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:28 smithi022 ceph-mon[31360]: pgmap v5069: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:28 smithi022 ceph-mon[40250]: pgmap v5069: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:28 smithi122 ceph-mon[32255]: pgmap v5069: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:30] "GET /metrics HTTP/1.1" 200 202330 "" "Prometheus/2.18.1" 2021-12-15T20:47:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:30 smithi122 ceph-mon[32255]: pgmap v5070: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:30 smithi022 ceph-mon[31360]: pgmap v5070: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:30 smithi022 ceph-mon[40250]: pgmap v5070: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:30.906 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:31.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:31.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:31.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:32.230 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:47:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:47:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:32 smithi022 ceph-mon[31360]: pgmap v5071: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3929502290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:32 smithi022 ceph-mon[40250]: pgmap v5071: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3929502290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:32 smithi122 ceph-mon[32255]: pgmap v5071: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3929502290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:47:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:47:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:47:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:47:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:47:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:47:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:47:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:47:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:47:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:47:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:47:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:34 smithi022 ceph-mon[31360]: pgmap v5072: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:34 smithi022 ceph-mon[40250]: pgmap v5072: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:34 smithi122 ceph-mon[32255]: pgmap v5072: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:36.278 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:36 smithi022 ceph-mon[40250]: pgmap v5073: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:36 smithi022 ceph-mon[31360]: pgmap v5073: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:36.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:36.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:36.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:36 smithi122 ceph-mon[32255]: pgmap v5073: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4177207657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:47:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4177207657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:37.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4177207657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:38 smithi122 ceph-mon[32255]: pgmap v5074: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:47:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:47:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:47:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:38.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[31360]: pgmap v5074: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:38.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:47:38.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:47:38.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:47:38.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:38.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:38.487 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[40250]: pgmap v5074: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:38.487 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:47:38.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:47:38.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:47:38.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:38.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:39.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:39 smithi022 ceph-mon[31360]: pgmap v5075: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:39.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:39.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:39 smithi022 ceph-mon[40250]: pgmap v5075: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:47:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:39 smithi122 ceph-mon[32255]: pgmap v5075: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:40] "GET /metrics HTTP/1.1" 200 202329 "" "Prometheus/2.18.1" 2021-12-15T20:47:41.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:42.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:42.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:42.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:42 smithi022 ceph-mon[31360]: pgmap v5076: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1384324120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:42 smithi022 ceph-mon[40250]: pgmap v5076: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1384324120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:42 smithi122 ceph-mon[32255]: pgmap v5076: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1384324120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:42.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:47:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:47:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:47:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:47:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:44 smithi122 ceph-mon[32255]: pgmap v5077: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:44 smithi022 ceph-mon[31360]: pgmap v5077: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:44 smithi022 ceph-mon[40250]: pgmap v5077: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:46 smithi122 ceph-mon[32255]: pgmap v5078: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:46 smithi022 ceph-mon[31360]: pgmap v5078: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:46 smithi022 ceph-mon[40250]: pgmap v5078: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:47.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:47.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:47.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:47.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:47:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:48 smithi122 ceph-mon[32255]: pgmap v5079: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1633824616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:48 smithi022 ceph-mon[31360]: pgmap v5079: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1633824616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:48 smithi022 ceph-mon[40250]: pgmap v5079: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1633824616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:50] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:47:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:50 smithi122 ceph-mon[32255]: pgmap v5080: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:50 smithi022 ceph-mon[31360]: pgmap v5080: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:50 smithi022 ceph-mon[40250]: pgmap v5080: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:47:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:52 smithi122 ceph-mon[32255]: pgmap v5081: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:52.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:47:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:47:52.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:52 smithi022 ceph-mon[31360]: pgmap v5081: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:52 smithi022 ceph-mon[40250]: pgmap v5081: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:47:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:47:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:47:52.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:52.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:52.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2629370756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2629370756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:47:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2629370756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:54 smithi122 ceph-mon[32255]: pgmap v5082: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:54 smithi022 ceph-mon[31360]: pgmap v5082: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:54 smithi022 ceph-mon[40250]: pgmap v5082: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:56 smithi122 ceph-mon[32255]: pgmap v5083: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:56 smithi022 ceph-mon[31360]: pgmap v5083: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:56 smithi022 ceph-mon[40250]: pgmap v5083: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:57.770 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:47:57.771 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:47:58.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:47:58.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:47:58.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:47:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:58 smithi022 ceph-mon[31360]: pgmap v5084: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:47:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/590314182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:58 smithi022 ceph-mon[40250]: pgmap v5084: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:47:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/590314182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:47:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:58 smithi122 ceph-mon[32255]: pgmap v5084: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:47:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:47:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/590314182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:00 smithi122 ceph-mon[32255]: pgmap v5085: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T20:48:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:00] "GET /metrics HTTP/1.1" 200 202335 "" "Prometheus/2.18.1" 2021-12-15T20:48:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:00 smithi022 ceph-mon[31360]: pgmap v5085: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T20:48:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:00 smithi022 ceph-mon[40250]: pgmap v5085: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T20:48:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:02 smithi122 ceph-mon[32255]: pgmap v5086: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T20:48:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:48:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:48:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:02 smithi022 ceph-mon[31360]: pgmap v5086: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T20:48:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:02 smithi022 ceph-mon[40250]: pgmap v5086: 132 pgs: 132 active+clean; 582 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2021-12-15T20:48:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:48:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:48:03.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:03.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:03.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:03.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:04 smithi022 ceph-mon[31360]: pgmap v5087: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:48:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/170612964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:04 smithi022 ceph-mon[40250]: pgmap v5087: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:48:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/170612964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:04 smithi122 ceph-mon[32255]: pgmap v5087: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:48:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/170612964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:06 smithi022 ceph-mon[31360]: pgmap v5088: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:48:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:06 smithi022 ceph-mon[40250]: pgmap v5088: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:48:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:06 smithi122 ceph-mon[32255]: pgmap v5088: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:48:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:48:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:08 smithi022 ceph-mon[31360]: pgmap v5089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:48:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:08 smithi022 ceph-mon[40250]: pgmap v5089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:48:08.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:08 smithi122 ceph-mon[32255]: pgmap v5089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:48:08.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:08.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:08.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/358978441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/358978441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/358978441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:10 smithi122 ceph-mon[32255]: pgmap v5090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:48:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:10] "GET /metrics HTTP/1.1" 200 202343 "" "Prometheus/2.18.1" 2021-12-15T20:48:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:10 smithi022 ceph-mon[31360]: pgmap v5090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:48:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:10 smithi022 ceph-mon[40250]: pgmap v5090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:48:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:12 smithi122 ceph-mon[32255]: pgmap v5091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T20:48:12.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:48:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:48:12.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:12 smithi022 ceph-mon[31360]: pgmap v5091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T20:48:12.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:12 smithi022 ceph-mon[40250]: pgmap v5091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T20:48:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:48:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:48:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:13.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:14 smithi022 ceph-mon[31360]: pgmap v5092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T20:48:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:14 smithi022 ceph-mon[40250]: pgmap v5092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T20:48:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:14 smithi122 ceph-mon[32255]: pgmap v5092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-15T20:48:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1748926390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1748926390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1748926390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:16 smithi022 ceph-mon[31360]: pgmap v5093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:16 smithi022 ceph-mon[40250]: pgmap v5093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:16 smithi122 ceph-mon[32255]: pgmap v5093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:48:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:18 smithi022 ceph-mon[31360]: pgmap v5094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:18 smithi022 ceph-mon[40250]: pgmap v5094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:18 smithi122 ceph-mon[32255]: pgmap v5094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:19.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:19.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:19.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:19.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:20] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:48:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:20 smithi122 ceph-mon[32255]: pgmap v5095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1157082114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:20 smithi022 ceph-mon[31360]: pgmap v5095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1157082114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:20 smithi022 ceph-mon[40250]: pgmap v5095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1157082114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:22.227 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:48:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:48:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:22 smithi022 ceph-mon[31360]: pgmap v5096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:22 smithi022 ceph-mon[40250]: pgmap v5096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:22 smithi122 ceph-mon[32255]: pgmap v5096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:48:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:48:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:24 smithi022 ceph-mon[31360]: pgmap v5097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:24 smithi022 ceph-mon[40250]: pgmap v5097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:24.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:24 smithi122 ceph-mon[32255]: pgmap v5097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:25.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:25.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:25.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2929857366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2929857366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2929857366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:26 smithi022 ceph-mon[31360]: pgmap v5098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:26 smithi022 ceph-mon[40250]: pgmap v5098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:26 smithi122 ceph-mon[32255]: pgmap v5098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:48:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:28 smithi022 ceph-mon[31360]: pgmap v5099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:28 smithi022 ceph-mon[40250]: pgmap v5099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:28 smithi122 ceph-mon[32255]: pgmap v5099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:30.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:30 smithi022 ceph-mon[31360]: pgmap v5100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:30 smithi022 ceph-mon[40250]: pgmap v5100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:30.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:30.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:30] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:48:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:30 smithi122 ceph-mon[32255]: pgmap v5100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/396362192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/396362192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/396362192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:32.247 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:48:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:48:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:32 smithi022 ceph-mon[31360]: pgmap v5101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:32 smithi022 ceph-mon[40250]: pgmap v5101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:32 smithi122 ceph-mon[32255]: pgmap v5101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:48:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:48:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:48:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:48:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:48:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:48:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:48:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:48:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:48:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:48:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:48:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:48:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:48:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:48:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:34 smithi022 ceph-mon[31360]: pgmap v5102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:34 smithi022 ceph-mon[40250]: pgmap v5102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:34 smithi122 ceph-mon[32255]: pgmap v5102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:35.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:35.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:35.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:35.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:36 smithi022 ceph-mon[31360]: pgmap v5103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/787929274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:36 smithi022 ceph-mon[40250]: pgmap v5103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/787929274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:36 smithi122 ceph-mon[32255]: pgmap v5103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/787929274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:48:38.529 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:38 smithi122 ceph-mon[32255]: pgmap v5104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:38 smithi022 ceph-mon[31360]: pgmap v5104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:38 smithi022 ceph-mon[40250]: pgmap v5104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:48:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:48:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:48:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:48:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:48:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:48:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:48:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:48:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:48:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:48:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:48:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:48:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:40] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T20:48:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:40 smithi122 ceph-mon[32255]: pgmap v5105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:40 smithi022 ceph-mon[31360]: pgmap v5105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:40 smithi022 ceph-mon[40250]: pgmap v5105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:40.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:41.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:41.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:41.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/189428113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/189428113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/189428113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:42.264 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:48:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:48:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:42 smithi022 ceph-mon[31360]: pgmap v5106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:42 smithi022 ceph-mon[40250]: pgmap v5106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:42 smithi122 ceph-mon[32255]: pgmap v5106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:48:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:48:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:44 smithi022 ceph-mon[31360]: pgmap v5107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:44 smithi022 ceph-mon[40250]: pgmap v5107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:44 smithi122 ceph-mon[32255]: pgmap v5107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:46.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:46.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:46.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:46.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:46.508 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:46 smithi022 ceph-mon[31360]: pgmap v5108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:46.508 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:46 smithi022 ceph-mon[40250]: pgmap v5108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:46 smithi122 ceph-mon[32255]: pgmap v5108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3065209035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:48:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3065209035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3065209035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:48 smithi022 ceph-mon[31360]: pgmap v5109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:48 smithi022 ceph-mon[40250]: pgmap v5109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:48 smithi122 ceph-mon[32255]: pgmap v5109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:50] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T20:48:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:50 smithi122 ceph-mon[32255]: pgmap v5110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:50 smithi022 ceph-mon[31360]: pgmap v5110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:50 smithi022 ceph-mon[40250]: pgmap v5110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:51.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:51 smithi122 ceph-mon[32255]: pgmap v5111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:51 smithi022 ceph-mon[31360]: pgmap v5111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:51 smithi022 ceph-mon[40250]: pgmap v5111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:51.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:51.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:51.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:52.285 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:48:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:48:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3186434693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:48:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:48:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3186434693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3186434693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:48:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:53 smithi122 ceph-mon[32255]: pgmap v5112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:53 smithi022 ceph-mon[31360]: pgmap v5112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:53 smithi022 ceph-mon[40250]: pgmap v5112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:48:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:56 smithi122 ceph-mon[32255]: pgmap v5113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:56 smithi022 ceph-mon[31360]: pgmap v5113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:56 smithi022 ceph-mon[40250]: pgmap v5113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:56.880 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:48:57.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:48:57.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:48:57.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:48:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:48:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:58 smithi022 ceph-mon[31360]: pgmap v5114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:48:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1596882345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:58 smithi022 ceph-mon[40250]: pgmap v5114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:48:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1596882345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:48:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:58 smithi122 ceph-mon[32255]: pgmap v5114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:48:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:48:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1596882345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:00 smithi122 ceph-mon[32255]: pgmap v5115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:00] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T20:49:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:00 smithi022 ceph-mon[31360]: pgmap v5115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:00 smithi022 ceph-mon[40250]: pgmap v5115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:02.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:49:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:49:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:02 smithi122 ceph-mon[32255]: pgmap v5116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:02 smithi022 ceph-mon[31360]: pgmap v5116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:02 smithi022 ceph-mon[40250]: pgmap v5116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:02.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:02.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:02.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:49:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:49:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2873310429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2873310429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2873310429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:04 smithi022 ceph-mon[31360]: pgmap v5117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:04 smithi022 ceph-mon[40250]: pgmap v5117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:04 smithi122 ceph-mon[32255]: pgmap v5117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:06 smithi022 ceph-mon[31360]: pgmap v5118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:06 smithi022 ceph-mon[40250]: pgmap v5118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:06 smithi122 ceph-mon[32255]: pgmap v5118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:07.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:49:08.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:08.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:08.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:08 smithi022 ceph-mon[31360]: pgmap v5119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/841843921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:08 smithi022 ceph-mon[40250]: pgmap v5119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/841843921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:08 smithi122 ceph-mon[32255]: pgmap v5119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/841843921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:10] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:49:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:10 smithi122 ceph-mon[32255]: pgmap v5120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:10 smithi022 ceph-mon[31360]: pgmap v5120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:10 smithi022 ceph-mon[40250]: pgmap v5120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:12.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:49:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:49:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:12 smithi122 ceph-mon[32255]: pgmap v5121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:12 smithi022 ceph-mon[31360]: pgmap v5121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:12 smithi022 ceph-mon[40250]: pgmap v5121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:49:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:49:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:13.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:13.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:13.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:14 smithi022 ceph-mon[31360]: pgmap v5122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:14.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/280410697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:14 smithi022 ceph-mon[40250]: pgmap v5122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/280410697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:14.589 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:14 smithi122 ceph-mon[32255]: pgmap v5122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:14.589 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/280410697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:16 smithi122 ceph-mon[32255]: pgmap v5123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:16 smithi022 ceph-mon[31360]: pgmap v5123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:16 smithi022 ceph-mon[40250]: pgmap v5123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:49:18.374 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:18 smithi022 ceph-mon[31360]: pgmap v5124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:18 smithi022 ceph-mon[40250]: pgmap v5124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:18 smithi122 ceph-mon[32255]: pgmap v5124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:18.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:18.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:18.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/921640582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/921640582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/921640582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:20] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:49:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:20 smithi122 ceph-mon[32255]: pgmap v5125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:20 smithi022 ceph-mon[31360]: pgmap v5125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:20 smithi022 ceph-mon[40250]: pgmap v5125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:22 smithi122 ceph-mon[32255]: pgmap v5126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:22.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:49:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:49:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:22 smithi022 ceph-mon[31360]: pgmap v5126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:22 smithi022 ceph-mon[40250]: pgmap v5126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:49:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:49:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:23.745 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:24.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:24.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:24.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:24 smithi022 ceph-mon[31360]: pgmap v5127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1955645232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:24 smithi022 ceph-mon[40250]: pgmap v5127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1955645232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:24 smithi122 ceph-mon[32255]: pgmap v5127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1955645232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:26 smithi022 ceph-mon[31360]: pgmap v5128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:26 smithi022 ceph-mon[40250]: pgmap v5128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:26 smithi122 ceph-mon[32255]: pgmap v5128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:49:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:28 smithi022 ceph-mon[31360]: pgmap v5129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:28 smithi022 ceph-mon[40250]: pgmap v5129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:28 smithi122 ceph-mon[32255]: pgmap v5129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:29.116 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:29.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:29.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:29.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:30] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:49:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:30 smithi122 ceph-mon[32255]: pgmap v5130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1749935439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:30 smithi022 ceph-mon[31360]: pgmap v5130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1749935439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:30 smithi022 ceph-mon[40250]: pgmap v5130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1749935439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:32 smithi122 ceph-mon[32255]: pgmap v5131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:32.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:49:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:49:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:32 smithi022 ceph-mon[31360]: pgmap v5131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:32 smithi022 ceph-mon[40250]: pgmap v5131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:49:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:49:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:49:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:49:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:49:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:49:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:49:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:49:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:49:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:49:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:49:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:49:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:49:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:49:34.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:34 smithi022 ceph-mon[40250]: pgmap v5132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:34 smithi022 ceph-mon[31360]: pgmap v5132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:34 smithi122 ceph-mon[32255]: pgmap v5132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:34.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:34.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:34.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2102204252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2102204252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2102204252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:36 smithi022 ceph-mon[31360]: pgmap v5133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:36 smithi022 ceph-mon[40250]: pgmap v5133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:36 smithi122 ceph-mon[32255]: pgmap v5133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:49:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:38 smithi022 ceph-mon[31360]: pgmap v5134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:38 smithi022 ceph-mon[40250]: pgmap v5134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:38 smithi122 ceph-mon[32255]: pgmap v5134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:49:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:49:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:49:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:49:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:49:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:49:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:49:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:49:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:49:39.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:40.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:40.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:40.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:40] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:49:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:40 smithi122 ceph-mon[32255]: pgmap v5135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:49:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2862706269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:40 smithi022 ceph-mon[40250]: pgmap v5135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:49:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2862706269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:40 smithi022 ceph-mon[31360]: pgmap v5135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:49:40.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2862706269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:42 smithi122 ceph-mon[32255]: pgmap v5136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:42.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:49:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:49:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:42 smithi022 ceph-mon[31360]: pgmap v5136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:42 smithi022 ceph-mon[40250]: pgmap v5136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:49:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:49:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:44 smithi022 ceph-mon[31360]: pgmap v5137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:44 smithi022 ceph-mon[40250]: pgmap v5137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:44 smithi122 ceph-mon[32255]: pgmap v5137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:45.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:45.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:45.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:45.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:46 smithi022 ceph-mon[31360]: pgmap v5138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3048899596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:46 smithi022 ceph-mon[40250]: pgmap v5138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3048899596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:46 smithi122 ceph-mon[32255]: pgmap v5138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3048899596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:49:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:48 smithi022 ceph-mon[31360]: pgmap v5139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:48 smithi022 ceph-mon[40250]: pgmap v5139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:48 smithi122 ceph-mon[32255]: pgmap v5139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:50] "GET /metrics HTTP/1.1" 200 202343 "" "Prometheus/2.18.1" 2021-12-15T20:49:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:50 smithi122 ceph-mon[32255]: pgmap v5140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:50 smithi022 ceph-mon[31360]: pgmap v5140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:50 smithi022 ceph-mon[40250]: pgmap v5140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:50.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:50.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:50.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:50.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3814495826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3814495826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3814495826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:52 smithi122 ceph-mon[32255]: pgmap v5141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:52.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:49:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:49:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:52 smithi022 ceph-mon[31360]: pgmap v5141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:52 smithi022 ceph-mon[40250]: pgmap v5141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:49:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:49:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:49:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:54 smithi022 ceph-mon[31360]: pgmap v5142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:54 smithi022 ceph-mon[40250]: pgmap v5142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:54 smithi122 ceph-mon[32255]: pgmap v5142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:49:55.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:49:56.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:49:56.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:49:56.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:49:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:56 smithi022 ceph-mon[31360]: pgmap v5143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:56 smithi022 ceph-mon[40250]: pgmap v5143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:56 smithi122 ceph-mon[32255]: pgmap v5143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/417839527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/417839527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:49:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/417839527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:49:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:49:58 smithi122 ceph-mon[32255]: pgmap v5144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:49:58 smithi022 ceph-mon[31360]: pgmap v5144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:49:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:49:58 smithi022 ceph-mon[40250]: pgmap v5144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:00] "GET /metrics HTTP/1.1" 200 202343 "" "Prometheus/2.18.1" 2021-12-15T20:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:00 smithi122 ceph-mon[32255]: pgmap v5145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T20:50:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:00 smithi022 ceph-mon[31360]: pgmap v5145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T20:50:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:00 smithi022 ceph-mon[40250]: pgmap v5145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T20:50:01.336 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:01.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:01.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:01.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:02.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:50:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:50:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:02 smithi122 ceph-mon[32255]: pgmap v5146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1914563929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:50:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:50:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:02 smithi022 ceph-mon[31360]: pgmap v5146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1914563929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:02 smithi022 ceph-mon[40250]: pgmap v5146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1914563929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:04 smithi122 ceph-mon[32255]: pgmap v5147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:04 smithi022 ceph-mon[31360]: pgmap v5147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:04 smithi022 ceph-mon[40250]: pgmap v5147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:05 smithi122 ceph-mon[32255]: pgmap v5148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:05 smithi022 ceph-mon[31360]: pgmap v5148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:05 smithi022 ceph-mon[40250]: pgmap v5148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:06.704 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:07.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/312366694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/312366694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/312366694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:50:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:08 smithi122 ceph-mon[32255]: pgmap v5149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:08 smithi022 ceph-mon[31360]: pgmap v5149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:08 smithi022 ceph-mon[40250]: pgmap v5149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:10] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T20:50:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:10 smithi122 ceph-mon[32255]: pgmap v5150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:10 smithi022 ceph-mon[31360]: pgmap v5150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:10 smithi022 ceph-mon[40250]: pgmap v5150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:12.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:12 smithi122 ceph-mon[32255]: pgmap v5151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:12.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:50:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:50:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:50:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:50:12.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:12.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:12.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:12.443 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:12 smithi022 ceph-mon[40250]: pgmap v5151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:12.443 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:12 smithi022 ceph-mon[31360]: pgmap v5151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/485615203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/485615203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/485615203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:14 smithi022 ceph-mon[31360]: pgmap v5152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:14 smithi022 ceph-mon[40250]: pgmap v5152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:14 smithi122 ceph-mon[32255]: pgmap v5152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:16 smithi022 ceph-mon[31360]: pgmap v5153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:16 smithi022 ceph-mon[40250]: pgmap v5153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:16 smithi122 ceph-mon[32255]: pgmap v5153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:17.444 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:50:17.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:17.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:17.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:18 smithi022 ceph-mon[31360]: pgmap v5154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1303713784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:18 smithi022 ceph-mon[40250]: pgmap v5154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1303713784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:18 smithi122 ceph-mon[32255]: pgmap v5154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1303713784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:20 smithi122 ceph-mon[32255]: pgmap v5155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:20] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T20:50:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:20 smithi022 ceph-mon[31360]: pgmap v5155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:20 smithi022 ceph-mon[40250]: pgmap v5155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:22 smithi122 ceph-mon[32255]: pgmap v5156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:50:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:50:22.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:50:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:50:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:22 smithi022 ceph-mon[31360]: pgmap v5156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:22 smithi022 ceph-mon[40250]: pgmap v5156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:22.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:23.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:23.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:23.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1516356084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1516356084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1516356084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:24 smithi022 ceph-mon[31360]: pgmap v5157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:24 smithi022 ceph-mon[40250]: pgmap v5157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:24 smithi122 ceph-mon[32255]: pgmap v5157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:26 smithi022 ceph-mon[31360]: pgmap v5158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:26 smithi022 ceph-mon[40250]: pgmap v5158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:26 smithi122 ceph-mon[32255]: pgmap v5158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:50:28.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:28 smithi022 ceph-mon[31360]: pgmap v5159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:28 smithi022 ceph-mon[40250]: pgmap v5159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:28.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:28.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:28.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:28 smithi122 ceph-mon[32255]: pgmap v5159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3285313068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3285313068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3285313068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:30 smithi122 ceph-mon[32255]: pgmap v5160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:30] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T20:50:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:30 smithi022 ceph-mon[31360]: pgmap v5160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:30 smithi022 ceph-mon[40250]: pgmap v5160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:32.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:50:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:50:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:50:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:50:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:32 smithi122 ceph-mon[32255]: pgmap v5161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:32 smithi022 ceph-mon[31360]: pgmap v5161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:32 smithi022 ceph-mon[40250]: pgmap v5161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:50:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:50:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:50:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:50:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:50:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:50:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:50:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:50:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:50:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:50:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:50:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:50:33.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:33.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:34 smithi022 ceph-mon[31360]: pgmap v5162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/610417975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:34 smithi022 ceph-mon[40250]: pgmap v5162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/610417975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:34 smithi122 ceph-mon[32255]: pgmap v5162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/610417975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:36 smithi022 ceph-mon[31360]: pgmap v5163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:36 smithi022 ceph-mon[40250]: pgmap v5163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:36 smithi122 ceph-mon[32255]: pgmap v5163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:50:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:38 smithi022 ceph-mon[31360]: pgmap v5164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:38 smithi022 ceph-mon[40250]: pgmap v5164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:38 smithi122 ceph-mon[32255]: pgmap v5164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:38.926 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:39.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:39.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:39.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:40] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:50:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:40 smithi122 ceph-mon[32255]: pgmap v5165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3494959006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:50:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:50:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:50:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:50:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[31360]: pgmap v5165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3494959006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:50:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:50:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:50:40.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:50:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[40250]: pgmap v5165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3494959006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:40.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:50:40.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:50:40.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:50:40.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:50:42.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:50:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:50:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:50:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:50:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:42 smithi122 ceph-mon[32255]: pgmap v5166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:42 smithi022 ceph-mon[31360]: pgmap v5166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:42 smithi022 ceph-mon[40250]: pgmap v5166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:44.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:44 smithi022 ceph-mon[31360]: pgmap v5167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:44 smithi022 ceph-mon[40250]: pgmap v5167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:44.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:44.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:44.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:44 smithi122 ceph-mon[32255]: pgmap v5167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3308044605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3308044605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3308044605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:46 smithi022 ceph-mon[31360]: pgmap v5168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:46 smithi022 ceph-mon[40250]: pgmap v5168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:46 smithi122 ceph-mon[32255]: pgmap v5168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:50:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:48 smithi022 ceph-mon[31360]: pgmap v5169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:48 smithi022 ceph-mon[40250]: pgmap v5169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:48 smithi122 ceph-mon[32255]: pgmap v5169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:49.669 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:50.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:50.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:50.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:50] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:50:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:50 smithi122 ceph-mon[32255]: pgmap v5170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2384679418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:50 smithi022 ceph-mon[31360]: pgmap v5170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2384679418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:50 smithi022 ceph-mon[40250]: pgmap v5170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2384679418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:52.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:50:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:50:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:52 smithi122 ceph-mon[32255]: pgmap v5171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:50:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:50:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:52 smithi022 ceph-mon[31360]: pgmap v5171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:52 smithi022 ceph-mon[40250]: pgmap v5171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:50:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:54 smithi022 ceph-mon[31360]: pgmap v5172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:54 smithi022 ceph-mon[40250]: pgmap v5172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:54 smithi122 ceph-mon[32255]: pgmap v5172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:50:55.043 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:50:55.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:50:55.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:50:55.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:50:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:56 smithi122 ceph-mon[32255]: pgmap v5173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2182021752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:56 smithi022 ceph-mon[31360]: pgmap v5173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2182021752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:56 smithi022 ceph-mon[40250]: pgmap v5173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2182021752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:50:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:50:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:50:58 smithi122 ceph-mon[32255]: pgmap v5174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:50:58 smithi022 ceph-mon[31360]: pgmap v5174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:50:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:50:58 smithi022 ceph-mon[40250]: pgmap v5174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:00.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:00] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:51:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:00 smithi122 ceph-mon[32255]: pgmap v5175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:00.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:00 smithi022 ceph-mon[31360]: pgmap v5175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:00 smithi022 ceph-mon[40250]: pgmap v5175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:00.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:00.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:00.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2206935444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2206935444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2206935444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:02 smithi122 ceph-mon[32255]: pgmap v5176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:02.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:51:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:51:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:51:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:51:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:02 smithi022 ceph-mon[31360]: pgmap v5176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:02 smithi022 ceph-mon[40250]: pgmap v5176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:04 smithi122 ceph-mon[32255]: pgmap v5177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:04 smithi022 ceph-mon[31360]: pgmap v5177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:04 smithi022 ceph-mon[40250]: pgmap v5177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:05.782 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:06.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:06.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:06.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:06 smithi022 ceph-mon[31360]: pgmap v5178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3281632128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:06 smithi022 ceph-mon[40250]: pgmap v5178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3281632128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:06 smithi122 ceph-mon[32255]: pgmap v5178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3281632128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:51:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:08 smithi122 ceph-mon[32255]: pgmap v5179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:08 smithi022 ceph-mon[31360]: pgmap v5179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:08 smithi022 ceph-mon[40250]: pgmap v5179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:10 smithi122 ceph-mon[32255]: pgmap v5180: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:10] "GET /metrics HTTP/1.1" 200 202345 "" "Prometheus/2.18.1" 2021-12-15T20:51:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:10 smithi022 ceph-mon[31360]: pgmap v5180: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:10 smithi022 ceph-mon[40250]: pgmap v5180: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:11.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:11 smithi022 ceph-mon[31360]: pgmap v5181: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:11 smithi022 ceph-mon[40250]: pgmap v5181: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:11.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:11.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:11.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:11 smithi122 ceph-mon[32255]: pgmap v5181: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:12.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:51:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:51:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:51:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:51:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1882786803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:12.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1882786803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1882786803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:13 smithi122 ceph-mon[32255]: pgmap v5182: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:13 smithi022 ceph-mon[31360]: pgmap v5182: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:13 smithi022 ceph-mon[40250]: pgmap v5182: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:16 smithi022 ceph-mon[31360]: pgmap v5183: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:16 smithi022 ceph-mon[40250]: pgmap v5183: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:16.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:16 smithi122 ceph-mon[32255]: pgmap v5183: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:16.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:16.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:16.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/229849988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:17.560 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:51:17.561 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/229849988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/229849988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:18 smithi022 ceph-mon[31360]: pgmap v5184: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:18 smithi022 ceph-mon[40250]: pgmap v5184: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:18 smithi122 ceph-mon[32255]: pgmap v5184: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:20 smithi122 ceph-mon[32255]: pgmap v5185: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:20] "GET /metrics HTTP/1.1" 200 202344 "" "Prometheus/2.18.1" 2021-12-15T20:51:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:20 smithi022 ceph-mon[31360]: pgmap v5185: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:20 smithi022 ceph-mon[40250]: pgmap v5185: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:21.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:22.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:22.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:22.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:22.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:51:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:51:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:51:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:51:22.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:22 smithi122 ceph-mon[32255]: pgmap v5186: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:22 smithi022 ceph-mon[31360]: pgmap v5186: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:22 smithi022 ceph-mon[40250]: pgmap v5186: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4241827394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4241827394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4241827394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:24 smithi022 ceph-mon[31360]: pgmap v5187: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:24 smithi022 ceph-mon[40250]: pgmap v5187: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:24 smithi122 ceph-mon[32255]: pgmap v5187: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:26 smithi022 ceph-mon[31360]: pgmap v5188: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:26 smithi022 ceph-mon[40250]: pgmap v5188: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:26 smithi122 ceph-mon[32255]: pgmap v5188: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:27.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:51:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:28 smithi022 ceph-mon[31360]: pgmap v5189: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4148292173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:28 smithi022 ceph-mon[40250]: pgmap v5189: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4148292173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:28 smithi122 ceph-mon[32255]: pgmap v5189: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4148292173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:30 smithi122 ceph-mon[32255]: pgmap v5190: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:30] "GET /metrics HTTP/1.1" 200 202344 "" "Prometheus/2.18.1" 2021-12-15T20:51:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:30 smithi022 ceph-mon[31360]: pgmap v5190: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:30 smithi022 ceph-mon[40250]: pgmap v5190: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:32.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:51:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:51:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:32 smithi122 ceph-mon[32255]: pgmap v5191: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:51:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:51:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:32 smithi022 ceph-mon[31360]: pgmap v5191: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:32 smithi022 ceph-mon[40250]: pgmap v5191: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:32.641 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:33.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:33.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:33.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:51:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:51:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:51:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:51:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3507902467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:51:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:51:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:51:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:51:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3507902467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:51:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:51:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:51:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:51:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3507902467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:34 smithi022 ceph-mon[31360]: pgmap v5192: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:34 smithi022 ceph-mon[40250]: pgmap v5192: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:34 smithi122 ceph-mon[32255]: pgmap v5192: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:36 smithi022 ceph-mon[31360]: pgmap v5193: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:36 smithi022 ceph-mon[40250]: pgmap v5193: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:36 smithi122 ceph-mon[32255]: pgmap v5193: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:51:38.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:38 smithi022 ceph-mon[31360]: pgmap v5194: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:38 smithi022 ceph-mon[40250]: pgmap v5194: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:38.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:38.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:38.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:38 smithi122 ceph-mon[32255]: pgmap v5194: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2400733777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2400733777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2400733777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:40] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:51:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:40 smithi122 ceph-mon[32255]: pgmap v5195: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:51:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:51:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:51:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[40250]: pgmap v5195: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:51:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:51:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:51:40.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[31360]: pgmap v5195: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:40.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:51:40.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:51:40.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:51:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:42 smithi122 ceph-mon[32255]: pgmap v5196: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:42.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:51:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:51:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:51:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:51:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:42 smithi022 ceph-mon[31360]: pgmap v5196: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:42 smithi022 ceph-mon[40250]: pgmap v5196: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:43.379 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:43.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:43.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:43.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:44 smithi022 ceph-mon[31360]: pgmap v5197: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3042689138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:44 smithi022 ceph-mon[40250]: pgmap v5197: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3042689138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:44 smithi122 ceph-mon[32255]: pgmap v5197: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3042689138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:45 smithi122 ceph-mon[32255]: pgmap v5198: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:45.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:45 smithi022 ceph-mon[31360]: pgmap v5198: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:45.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:51:45.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:45 smithi022 ceph-mon[40250]: pgmap v5198: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:51:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:48 smithi022 ceph-mon[31360]: pgmap v5199: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:48 smithi022 ceph-mon[40250]: pgmap v5199: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:48 smithi122 ceph-mon[32255]: pgmap v5199: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:48.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:49.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:49.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:49.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/712104679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/712104679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/712104679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:50 smithi122 ceph-mon[32255]: pgmap v5200: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:50] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:51:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:50 smithi022 ceph-mon[31360]: pgmap v5200: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:50 smithi022 ceph-mon[40250]: pgmap v5200: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:52.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:51:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:51:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:52 smithi122 ceph-mon[32255]: pgmap v5201: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:51:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:51:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:52 smithi022 ceph-mon[31360]: pgmap v5201: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:52 smithi022 ceph-mon[40250]: pgmap v5201: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:51:54.124 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:54.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:54.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:54.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:51:54.497 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:54 smithi022 ceph-mon[31360]: pgmap v5202: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:54.498 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:54 smithi022 ceph-mon[40250]: pgmap v5202: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:54 smithi122 ceph-mon[32255]: pgmap v5202: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:51:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1492248550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1492248550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1492248550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:51:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:56 smithi022 ceph-mon[31360]: pgmap v5203: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:56 smithi022 ceph-mon[40250]: pgmap v5203: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:56 smithi122 ceph-mon[32255]: pgmap v5203: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:51:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:51:58 smithi022 ceph-mon[31360]: pgmap v5204: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:51:58 smithi022 ceph-mon[40250]: pgmap v5204: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:51:58 smithi122 ceph-mon[32255]: pgmap v5204: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:51:59.498 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:51:59.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:51:59.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:51:59.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:00] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:52:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:00 smithi122 ceph-mon[32255]: pgmap v5205: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2766400818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:00 smithi022 ceph-mon[40250]: pgmap v5205: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2766400818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:00 smithi022 ceph-mon[31360]: pgmap v5205: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2766400818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:02 smithi122 ceph-mon[32255]: pgmap v5206: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:02.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:52:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:52:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:52:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:52:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:02 smithi022 ceph-mon[31360]: pgmap v5206: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:02 smithi022 ceph-mon[40250]: pgmap v5206: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:04 smithi022 ceph-mon[31360]: pgmap v5207: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:04 smithi022 ceph-mon[40250]: pgmap v5207: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:04 smithi122 ceph-mon[32255]: pgmap v5207: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:04.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:05.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:05.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:05.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3686640932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3686640932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3686640932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:06 smithi022 ceph-mon[31360]: pgmap v5208: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:06 smithi022 ceph-mon[40250]: pgmap v5208: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:06 smithi122 ceph-mon[32255]: pgmap v5208: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:52:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:08 smithi022 ceph-mon[31360]: pgmap v5209: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:08 smithi022 ceph-mon[40250]: pgmap v5209: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:08 smithi122 ceph-mon[32255]: pgmap v5209: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:10] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:52:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:10 smithi122 ceph-mon[32255]: pgmap v5210: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:10 smithi022 ceph-mon[31360]: pgmap v5210: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:10 smithi022 ceph-mon[40250]: pgmap v5210: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:10.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:10.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:10.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3349454895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3349454895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3349454895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:12 smithi122 ceph-mon[32255]: pgmap v5211: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:12.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:52:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:52:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:52:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:52:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:12 smithi022 ceph-mon[31360]: pgmap v5211: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:12 smithi022 ceph-mon[40250]: pgmap v5211: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:14 smithi122 ceph-mon[32255]: pgmap v5212: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:14 smithi022 ceph-mon[31360]: pgmap v5212: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:14 smithi022 ceph-mon[40250]: pgmap v5212: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:15.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:15.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:15.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:15.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:16 smithi122 ceph-mon[32255]: pgmap v5213: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4263511887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:16 smithi022 ceph-mon[31360]: pgmap v5213: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4263511887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:16 smithi022 ceph-mon[40250]: pgmap v5213: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4263511887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:52:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:18 smithi122 ceph-mon[32255]: pgmap v5214: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:18 smithi022 ceph-mon[31360]: pgmap v5214: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:18.863 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:18 smithi022 ceph-mon[40250]: pgmap v5214: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:20 smithi122 ceph-mon[32255]: pgmap v5215: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:20] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:52:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:20 smithi022 ceph-mon[31360]: pgmap v5215: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:20 smithi022 ceph-mon[40250]: pgmap v5215: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:20.979 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:21.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:21.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:21.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:52:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:52:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:22 smithi122 ceph-mon[32255]: pgmap v5216: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3145804429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:52:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:52:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:22 smithi022 ceph-mon[31360]: pgmap v5216: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3145804429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:22 smithi022 ceph-mon[40250]: pgmap v5216: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3145804429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:24 smithi122 ceph-mon[32255]: pgmap v5217: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:24 smithi022 ceph-mon[31360]: pgmap v5217: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:24 smithi022 ceph-mon[40250]: pgmap v5217: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:26.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:26 smithi122 ceph-mon[32255]: pgmap v5218: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:26.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:26.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:26.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:26.719 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:26 smithi022 ceph-mon[31360]: pgmap v5218: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:26.720 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:26 smithi022 ceph-mon[40250]: pgmap v5218: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1085922564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1085922564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:52:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1085922564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:28 smithi122 ceph-mon[32255]: pgmap v5219: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:28 smithi022 ceph-mon[31360]: pgmap v5219: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:28 smithi022 ceph-mon[40250]: pgmap v5219: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T20:52:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:30 smithi122 ceph-mon[32255]: pgmap v5220: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:30] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:52:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:30 smithi022 ceph-mon[31360]: pgmap v5220: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:30 smithi022 ceph-mon[40250]: pgmap v5220: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:31.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:32.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:32.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:32.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:32 smithi022 ceph-mon[31360]: pgmap v5221: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1277562776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:32 smithi022 ceph-mon[40250]: pgmap v5221: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1277562776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:52:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:52:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:52:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:52:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:32 smithi122 ceph-mon[32255]: pgmap v5221: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1277562776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:52:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:52:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:52:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:52:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:33 smithi122 ceph-mon[32255]: pgmap v5222: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:52:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:52:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:52:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:52:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[40250]: pgmap v5222: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:52:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:52:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:52:33.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:52:33.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:33 smithi022 ceph-mon[31360]: pgmap v5222: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:36 smithi022 ceph-mon[31360]: pgmap v5223: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:36 smithi022 ceph-mon[40250]: pgmap v5223: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:36 smithi122 ceph-mon[32255]: pgmap v5223: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:37.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:37.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:37.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:37.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:52:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:38 smithi022 ceph-mon[31360]: pgmap v5224: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/213898020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:38 smithi022 ceph-mon[40250]: pgmap v5224: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/213898020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:38 smithi122 ceph-mon[32255]: pgmap v5224: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/213898020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:40 smithi122 ceph-mon[32255]: pgmap v5225: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:40] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:52:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:40 smithi022 ceph-mon[31360]: pgmap v5225: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:40 smithi022 ceph-mon[40250]: pgmap v5225: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:42.465 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:42 smithi022 ceph-mon[31360]: pgmap v5226: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:42 smithi022 ceph-mon[40250]: pgmap v5226: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:52:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:52:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:52:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:52:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:42 smithi122 ceph-mon[32255]: pgmap v5226: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:42.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:42.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:42.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2305400270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2305400270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2305400270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:44 smithi022 ceph-mon[31360]: pgmap v5227: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:44.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:44 smithi022 ceph-mon[40250]: pgmap v5227: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:44 smithi122 ceph-mon[32255]: pgmap v5227: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:52:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:52:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:52:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:52:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:52:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:52:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:52:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:52:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:52:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:52:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:52:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:52:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:46 smithi022 ceph-mon[31360]: pgmap v5228: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:46 smithi022 ceph-mon[40250]: pgmap v5228: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:46 smithi122 ceph-mon[32255]: pgmap v5228: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:52:47.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:48.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:48.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:48.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:48 smithi022 ceph-mon[31360]: pgmap v5229: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2859244798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:48 smithi022 ceph-mon[40250]: pgmap v5229: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2859244798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:48 smithi122 ceph-mon[32255]: pgmap v5229: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2859244798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:50 smithi122 ceph-mon[32255]: pgmap v5230: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:50] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T20:52:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:50 smithi022 ceph-mon[31360]: pgmap v5230: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:50 smithi022 ceph-mon[40250]: pgmap v5230: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:52 smithi022 ceph-mon[31360]: pgmap v5231: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:52 smithi022 ceph-mon[40250]: pgmap v5231: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:52 smithi122 ceph-mon[32255]: pgmap v5231: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:52.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:52:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:52:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:52:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:52:53.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:53.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:53.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:53.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:52:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:54 smithi022 ceph-mon[31360]: pgmap v5232: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1759870797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:54 smithi022 ceph-mon[40250]: pgmap v5232: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1759870797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:54 smithi122 ceph-mon[32255]: pgmap v5232: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:52:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1759870797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:56 smithi022 ceph-mon[31360]: pgmap v5233: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:56 smithi022 ceph-mon[40250]: pgmap v5233: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:56 smithi122 ceph-mon[32255]: pgmap v5233: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:52:58.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:52:58.594 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:58 smithi022 ceph-mon[31360]: pgmap v5234: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:58.594 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:58 smithi022 ceph-mon[40250]: pgmap v5234: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:58 smithi122 ceph-mon[32255]: pgmap v5234: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:52:58.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:52:58.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:52:58.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:52:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:52:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1858299318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:52:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1858299318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:52:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:52:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1858299318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:00] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T20:53:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:00 smithi122 ceph-mon[32255]: pgmap v5235: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:00 smithi022 ceph-mon[31360]: pgmap v5235: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:00 smithi022 ceph-mon[40250]: pgmap v5235: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:02 smithi122 ceph-mon[32255]: pgmap v5236: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:02.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:53:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:53:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:53:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:53:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:02 smithi022 ceph-mon[31360]: pgmap v5236: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:02 smithi022 ceph-mon[40250]: pgmap v5236: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:03.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:04.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:04.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:04.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:04 smithi122 ceph-mon[32255]: pgmap v5237: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:04 smithi022 ceph-mon[31360]: pgmap v5237: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:04 smithi022 ceph-mon[40250]: pgmap v5237: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/900359639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/900359639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/900359639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:06 smithi122 ceph-mon[32255]: pgmap v5238: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:06 smithi022 ceph-mon[31360]: pgmap v5238: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:06 smithi022 ceph-mon[40250]: pgmap v5238: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:53:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:08 smithi122 ceph-mon[32255]: pgmap v5239: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:08 smithi022 ceph-mon[31360]: pgmap v5239: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:08 smithi022 ceph-mon[40250]: pgmap v5239: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:09.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:09.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:09.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:09.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:10] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:53:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:10 smithi122 ceph-mon[32255]: pgmap v5240: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1329839811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:10 smithi022 ceph-mon[31360]: pgmap v5240: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1329839811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:10 smithi022 ceph-mon[40250]: pgmap v5240: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1329839811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:12 smithi122 ceph-mon[32255]: pgmap v5241: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:12.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:53:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:53:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:53:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:53:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:12 smithi022 ceph-mon[31360]: pgmap v5241: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:12 smithi022 ceph-mon[40250]: pgmap v5241: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:14 smithi122 ceph-mon[32255]: pgmap v5242: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:14.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:14.708 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:14 smithi022 ceph-mon[31360]: pgmap v5242: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:14.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:14 smithi022 ceph-mon[40250]: pgmap v5242: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:15.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:15.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:15.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3425499792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3425499792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3425499792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:16 smithi122 ceph-mon[32255]: pgmap v5243: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:16 smithi022 ceph-mon[31360]: pgmap v5243: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:16 smithi022 ceph-mon[40250]: pgmap v5243: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:53:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:18 smithi122 ceph-mon[32255]: pgmap v5244: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:18 smithi022 ceph-mon[31360]: pgmap v5244: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:18 smithi022 ceph-mon[40250]: pgmap v5244: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:20.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:20 smithi122 ceph-mon[32255]: pgmap v5245: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:20] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T20:53:20.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:20.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:20.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:20.446 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:20 smithi022 ceph-mon[31360]: pgmap v5245: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:20.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:20 smithi022 ceph-mon[40250]: pgmap v5245: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4172375223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4172375223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4172375223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:53:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:53:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:22 smithi122 ceph-mon[32255]: pgmap v5246: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:53:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:53:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:22 smithi022 ceph-mon[31360]: pgmap v5246: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:22 smithi022 ceph-mon[40250]: pgmap v5246: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:23 smithi122 ceph-mon[32255]: pgmap v5247: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:23 smithi022 ceph-mon[31360]: pgmap v5247: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:23 smithi022 ceph-mon[40250]: pgmap v5247: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:25.447 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:25.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:25.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:25.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:26 smithi022 ceph-mon[31360]: pgmap v5248: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2140645771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:26 smithi022 ceph-mon[40250]: pgmap v5248: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2140645771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:26 smithi122 ceph-mon[32255]: pgmap v5248: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2140645771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:53:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:28 smithi022 ceph-mon[31360]: pgmap v5249: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:28 smithi022 ceph-mon[40250]: pgmap v5249: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:28 smithi122 ceph-mon[32255]: pgmap v5249: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:30] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T20:53:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:30 smithi122 ceph-mon[32255]: pgmap v5250: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:30 smithi022 ceph-mon[31360]: pgmap v5250: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:30 smithi022 ceph-mon[40250]: pgmap v5250: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:30.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:31.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:31.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:31.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2617102375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2617102375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2617102375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:32 smithi022 ceph-mon[31360]: pgmap v5251: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:32 smithi022 ceph-mon[40250]: pgmap v5251: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:32 smithi122 ceph-mon[32255]: pgmap v5251: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:53:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:53:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:53:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:53:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:53:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:53:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:53:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:53:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:53:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:53:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:53:33.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:53:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:53:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:53:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:53:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:53:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:34 smithi022 ceph-mon[31360]: pgmap v5252: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:34 smithi022 ceph-mon[40250]: pgmap v5252: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:34 smithi122 ceph-mon[32255]: pgmap v5252: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:36.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:36 smithi022 ceph-mon[31360]: pgmap v5253: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:36 smithi022 ceph-mon[40250]: pgmap v5253: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:36.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:36.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:36.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:36 smithi122 ceph-mon[32255]: pgmap v5253: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2643129011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2643129011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:53:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2643129011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:38 smithi122 ceph-mon[32255]: pgmap v5254: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:38 smithi022 ceph-mon[31360]: pgmap v5254: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:38 smithi022 ceph-mon[40250]: pgmap v5254: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:40] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T20:53:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:40 smithi122 ceph-mon[32255]: pgmap v5255: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:40 smithi022 ceph-mon[31360]: pgmap v5255: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:40 smithi022 ceph-mon[40250]: pgmap v5255: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:41.554 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:41.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:41.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:41.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:53:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:53:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:42 smithi122 ceph-mon[32255]: pgmap v5256: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1535171129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:53:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:53:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:42 smithi022 ceph-mon[31360]: pgmap v5256: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1535171129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:42 smithi022 ceph-mon[40250]: pgmap v5256: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1535171129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:44 smithi122 ceph-mon[32255]: pgmap v5257: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:44 smithi022 ceph-mon[31360]: pgmap v5257: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:44 smithi022 ceph-mon[40250]: pgmap v5257: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:53:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:53:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:53:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:53:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:53:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:53:45.633 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:53:45.633 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:53:45.633 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:53:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:46 smithi122 ceph-mon[32255]: pgmap v5258: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:53:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:46 smithi022 ceph-mon[31360]: pgmap v5258: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:53:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:46 smithi022 ceph-mon[40250]: pgmap v5258: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:53:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:47.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:47.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:47.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1681820990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1681820990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:53:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1681820990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:48 smithi122 ceph-mon[32255]: pgmap v5259: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:48 smithi022 ceph-mon[31360]: pgmap v5259: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:48 smithi022 ceph-mon[40250]: pgmap v5259: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:50] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:53:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:50 smithi122 ceph-mon[32255]: pgmap v5260: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:50 smithi022 ceph-mon[31360]: pgmap v5260: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:50 smithi022 ceph-mon[40250]: pgmap v5260: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:52.292 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:52.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:52.662 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:52 smithi022 ceph-mon[31360]: pgmap v5261: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:52.662 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:52 smithi022 ceph-mon[40250]: pgmap v5261: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:52 smithi122 ceph-mon[32255]: pgmap v5261: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:52.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:53:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:53:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:53:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:53:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2766821523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2766821523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:53:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2766821523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:54 smithi122 ceph-mon[32255]: pgmap v5262: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:54 smithi022 ceph-mon[31360]: pgmap v5262: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:54 smithi022 ceph-mon[40250]: pgmap v5262: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:53:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:56 smithi122 ceph-mon[32255]: pgmap v5263: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:56 smithi022 ceph-mon[31360]: pgmap v5263: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:56 smithi022 ceph-mon[40250]: pgmap v5263: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:57.663 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:53:57.664 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:53:58.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:53:58.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:53:58.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:53:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:58 smithi122 ceph-mon[32255]: pgmap v5264: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:53:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4217402040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:58 smithi022 ceph-mon[31360]: pgmap v5264: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:53:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4217402040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:53:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:58 smithi022 ceph-mon[40250]: pgmap v5264: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:53:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:53:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4217402040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:00 smithi122 ceph-mon[32255]: pgmap v5265: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:00] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:54:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:00 smithi022 ceph-mon[31360]: pgmap v5265: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:00 smithi022 ceph-mon[40250]: pgmap v5265: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:02 smithi122 ceph-mon[32255]: pgmap v5266: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:02.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:54:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:54:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:54:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:54:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:02 smithi022 ceph-mon[31360]: pgmap v5266: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:02 smithi022 ceph-mon[40250]: pgmap v5266: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:03.034 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:03.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:03.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:03.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:04 smithi122 ceph-mon[32255]: pgmap v5267: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/777693184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:04 smithi022 ceph-mon[31360]: pgmap v5267: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/777693184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:04 smithi022 ceph-mon[40250]: pgmap v5267: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/777693184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:05 smithi122 ceph-mon[32255]: pgmap v5268: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:05 smithi022 ceph-mon[31360]: pgmap v5268: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:05 smithi022 ceph-mon[40250]: pgmap v5268: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:54:08.405 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:08 smithi022 ceph-mon[31360]: pgmap v5269: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:08 smithi022 ceph-mon[40250]: pgmap v5269: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:08 smithi122 ceph-mon[32255]: pgmap v5269: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:08.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:08.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:08.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2752924928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2752924928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2752924928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:10] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:54:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:10 smithi122 ceph-mon[32255]: pgmap v5270: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:10 smithi022 ceph-mon[31360]: pgmap v5270: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:10 smithi022 ceph-mon[40250]: pgmap v5270: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:12 smithi122 ceph-mon[32255]: pgmap v5271: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:54:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:54:12.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:54:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:54:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:12 smithi022 ceph-mon[31360]: pgmap v5271: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:12 smithi022 ceph-mon[40250]: pgmap v5271: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:13.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:14.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:14.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:14.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:14 smithi022 ceph-mon[31360]: pgmap v5272: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3552131751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:14 smithi022 ceph-mon[40250]: pgmap v5272: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3552131751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:14 smithi122 ceph-mon[32255]: pgmap v5272: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3552131751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:16 smithi122 ceph-mon[32255]: pgmap v5273: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:16 smithi022 ceph-mon[31360]: pgmap v5273: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:16 smithi022 ceph-mon[40250]: pgmap v5273: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:54:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:18 smithi122 ceph-mon[32255]: pgmap v5274: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:18 smithi022 ceph-mon[31360]: pgmap v5274: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:18 smithi022 ceph-mon[40250]: pgmap v5274: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:19.151 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:19.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:19.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:19.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:20] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:54:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:20 smithi122 ceph-mon[32255]: pgmap v5275: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1318868430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:20 smithi022 ceph-mon[31360]: pgmap v5275: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1318868430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:20 smithi022 ceph-mon[40250]: pgmap v5275: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1318868430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:54:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:54:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:22 smithi122 ceph-mon[32255]: pgmap v5276: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:54:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:54:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:22 smithi022 ceph-mon[40250]: pgmap v5276: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:22 smithi022 ceph-mon[31360]: pgmap v5276: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:24.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:24 smithi122 ceph-mon[32255]: pgmap v5277: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:24 smithi022 ceph-mon[31360]: pgmap v5277: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:24 smithi022 ceph-mon[40250]: pgmap v5277: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2062796839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2062796839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2062796839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:26 smithi122 ceph-mon[32255]: pgmap v5278: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:26 smithi022 ceph-mon[31360]: pgmap v5278: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:26 smithi022 ceph-mon[40250]: pgmap v5278: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:54:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:28 smithi122 ceph-mon[32255]: pgmap v5279: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:28 smithi022 ceph-mon[31360]: pgmap v5279: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:28 smithi022 ceph-mon[40250]: pgmap v5279: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:29.893 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:30.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:30.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:30.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:30] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:54:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:30 smithi122 ceph-mon[32255]: pgmap v5280: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3762940371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:30 smithi022 ceph-mon[31360]: pgmap v5280: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3762940371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:30 smithi022 ceph-mon[40250]: pgmap v5280: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3762940371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:32 smithi122 ceph-mon[32255]: pgmap v5281: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:54:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:54:32.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:54:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:54:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:32 smithi022 ceph-mon[31360]: pgmap v5281: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:32 smithi022 ceph-mon[40250]: pgmap v5281: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:54:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:54:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:54:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:54:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:54:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:54:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:54:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:54:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:54:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:54:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:54:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:54:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:34 smithi122 ceph-mon[32255]: pgmap v5282: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:34 smithi022 ceph-mon[31360]: pgmap v5282: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:34 smithi022 ceph-mon[40250]: pgmap v5282: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:35.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:35.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:35.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:35.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:36 smithi122 ceph-mon[32255]: pgmap v5283: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3536044855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:36 smithi022 ceph-mon[31360]: pgmap v5283: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3536044855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:36 smithi022 ceph-mon[40250]: pgmap v5283: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3536044855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:54:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:38 smithi122 ceph-mon[32255]: pgmap v5284: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:38 smithi022 ceph-mon[31360]: pgmap v5284: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:38 smithi022 ceph-mon[40250]: pgmap v5284: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:40 smithi122 ceph-mon[32255]: pgmap v5285: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:40] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:54:40.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:40.639 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:40 smithi022 ceph-mon[31360]: pgmap v5285: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:40.639 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:40 smithi022 ceph-mon[40250]: pgmap v5285: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:41.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:41.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:41.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3275956909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3275956909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3275956909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:42 smithi122 ceph-mon[32255]: pgmap v5286: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:54:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:54:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:54:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:54:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:42 smithi022 ceph-mon[31360]: pgmap v5286: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:42 smithi022 ceph-mon[40250]: pgmap v5286: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:44 smithi122 ceph-mon[32255]: pgmap v5287: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:44 smithi022 ceph-mon[31360]: pgmap v5287: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:44 smithi022 ceph-mon[40250]: pgmap v5287: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:46.008 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:46.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:46.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:46.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:46 smithi122 ceph-mon[32255]: pgmap v5288: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:54:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:54:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:54:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:54:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[31360]: pgmap v5288: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:54:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:54:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:54:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:54:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[40250]: pgmap v5288: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:54:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:54:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:54:46.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:54:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3332956393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3332956393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:54:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3332956393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:48 smithi122 ceph-mon[32255]: pgmap v5289: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:48 smithi022 ceph-mon[31360]: pgmap v5289: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:48 smithi022 ceph-mon[40250]: pgmap v5289: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:49 smithi122 ceph-mon[32255]: pgmap v5290: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:49 smithi022 ceph-mon[31360]: pgmap v5290: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:49 smithi022 ceph-mon[40250]: pgmap v5290: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:50] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:54:51.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:51.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:51.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:51.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:52.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:54:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:54:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:52 smithi122 ceph-mon[32255]: pgmap v5291: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/669013913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:54:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:54:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:52 smithi022 ceph-mon[31360]: pgmap v5291: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/669013913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:52 smithi022 ceph-mon[40250]: pgmap v5291: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/669013913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:54:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:54 smithi122 ceph-mon[32255]: pgmap v5292: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:54 smithi022 ceph-mon[31360]: pgmap v5292: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:54 smithi022 ceph-mon[40250]: pgmap v5292: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:54:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:56 smithi122 ceph-mon[32255]: pgmap v5293: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:56.741 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:54:56.742 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:56 smithi022 ceph-mon[31360]: pgmap v5293: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:56.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:56 smithi022 ceph-mon[40250]: pgmap v5293: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:57.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:54:57.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:54:57.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:54:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2092780418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2092780418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2092780418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:54:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:54:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:54:58 smithi122 ceph-mon[32255]: pgmap v5294: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:54:58 smithi022 ceph-mon[31360]: pgmap v5294: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:54:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:54:58 smithi022 ceph-mon[40250]: pgmap v5294: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:00] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:55:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:00 smithi122 ceph-mon[32255]: pgmap v5295: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:00 smithi022 ceph-mon[31360]: pgmap v5295: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:00 smithi022 ceph-mon[40250]: pgmap v5295: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:02.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:02.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:02.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:02.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:02.481 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:02 smithi022 ceph-mon[31360]: pgmap v5296: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:02.481 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:02 smithi022 ceph-mon[40250]: pgmap v5296: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:02.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:55:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:55:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:02 smithi122 ceph-mon[32255]: pgmap v5296: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:55:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:55:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2261647033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2261647033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2261647033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:04 smithi122 ceph-mon[32255]: pgmap v5297: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:04 smithi022 ceph-mon[31360]: pgmap v5297: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:04 smithi022 ceph-mon[40250]: pgmap v5297: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:06 smithi122 ceph-mon[32255]: pgmap v5298: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:06 smithi022 ceph-mon[31360]: pgmap v5298: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:06 smithi022 ceph-mon[40250]: pgmap v5298: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:07.481 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:55:07.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:07.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:07.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:08 smithi122 ceph-mon[32255]: pgmap v5299: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/257473611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:08 smithi022 ceph-mon[31360]: pgmap v5299: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/257473611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:08 smithi022 ceph-mon[40250]: pgmap v5299: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/257473611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:10 smithi122 ceph-mon[32255]: pgmap v5300: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:10] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T20:55:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:10 smithi022 ceph-mon[31360]: pgmap v5300: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:10 smithi022 ceph-mon[40250]: pgmap v5300: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:12 smithi122 ceph-mon[32255]: pgmap v5301: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:12.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:55:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:55:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:55:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:55:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:12 smithi022 ceph-mon[31360]: pgmap v5301: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:12 smithi022 ceph-mon[40250]: pgmap v5301: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:12.851 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:13.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:13.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:13.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1445190653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1445190653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:13.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1445190653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:14 smithi122 ceph-mon[32255]: pgmap v5302: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:14 smithi022 ceph-mon[31360]: pgmap v5302: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:14 smithi022 ceph-mon[40250]: pgmap v5302: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:16 smithi122 ceph-mon[32255]: pgmap v5303: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:16 smithi022 ceph-mon[31360]: pgmap v5303: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:16 smithi022 ceph-mon[40250]: pgmap v5303: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:55:18.220 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:18 smithi022 ceph-mon[31360]: pgmap v5304: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:18 smithi022 ceph-mon[40250]: pgmap v5304: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:18.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:18.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:18.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:18 smithi122 ceph-mon[32255]: pgmap v5304: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3795433237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3795433237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3795433237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:20] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:55:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:20 smithi122 ceph-mon[32255]: pgmap v5305: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:20 smithi022 ceph-mon[31360]: pgmap v5305: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:20 smithi022 ceph-mon[40250]: pgmap v5305: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:22 smithi122 ceph-mon[32255]: pgmap v5306: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:22.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:55:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:55:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:55:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:55:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:22 smithi022 ceph-mon[31360]: pgmap v5306: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:22 smithi022 ceph-mon[40250]: pgmap v5306: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:23.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:23.595 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:23.596 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:23.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:23.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:23.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:24 smithi122 ceph-mon[32255]: pgmap v5307: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1365140479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:24 smithi022 ceph-mon[31360]: pgmap v5307: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1365140479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:24.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:24 smithi022 ceph-mon[40250]: pgmap v5307: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:24.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1365140479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:26 smithi122 ceph-mon[32255]: pgmap v5308: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:26 smithi022 ceph-mon[31360]: pgmap v5308: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:26 smithi022 ceph-mon[40250]: pgmap v5308: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:55:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:28 smithi122 ceph-mon[32255]: pgmap v5309: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:28 smithi022 ceph-mon[31360]: pgmap v5309: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:28 smithi022 ceph-mon[40250]: pgmap v5309: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:28.965 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:29.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:29.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:29.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:29 smithi122 ceph-mon[32255]: pgmap v5310: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3908103911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:29 smithi022 ceph-mon[31360]: pgmap v5310: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3908103911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:29 smithi022 ceph-mon[40250]: pgmap v5310: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3908103911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:30] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:55:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:32 smithi122 ceph-mon[32255]: pgmap v5311: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:32.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:55:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:55:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:55:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:55:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:32 smithi022 ceph-mon[31360]: pgmap v5311: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:32 smithi022 ceph-mon[40250]: pgmap v5311: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:55:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:55:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:55:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:55:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:55:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:55:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:55:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:55:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:55:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:55:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:55:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:55:34.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:34 smithi122 ceph-mon[32255]: pgmap v5312: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:34.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:34.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:34.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:34.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:34 smithi022 ceph-mon[40250]: pgmap v5312: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:34.708 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:34 smithi022 ceph-mon[31360]: pgmap v5312: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3620151442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3620151442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3620151442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:36 smithi122 ceph-mon[32255]: pgmap v5313: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:36 smithi022 ceph-mon[31360]: pgmap v5313: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:36 smithi022 ceph-mon[40250]: pgmap v5313: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:55:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:38 smithi122 ceph-mon[32255]: pgmap v5314: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:38 smithi022 ceph-mon[31360]: pgmap v5314: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:38 smithi022 ceph-mon[40250]: pgmap v5314: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:39.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:40.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:40.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:40.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:40] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:55:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:40 smithi122 ceph-mon[32255]: pgmap v5315: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1370228492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:40 smithi022 ceph-mon[31360]: pgmap v5315: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1370228492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:40 smithi022 ceph-mon[40250]: pgmap v5315: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1370228492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:42 smithi122 ceph-mon[32255]: pgmap v5316: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:55:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:55:42.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:55:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:55:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:42 smithi022 ceph-mon[31360]: pgmap v5316: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:42 smithi022 ceph-mon[40250]: pgmap v5316: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:43 smithi122 ceph-mon[32255]: pgmap v5317: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:43 smithi022 ceph-mon[31360]: pgmap v5317: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:43 smithi022 ceph-mon[40250]: pgmap v5317: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:45.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:45.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:45.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:45.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:46 smithi122 ceph-mon[32255]: pgmap v5318: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396216760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:55:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:55:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:55:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[31360]: pgmap v5318: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396216760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:55:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:55:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:55:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[40250]: pgmap v5318: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396216760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:55:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:55:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:55:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:55:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:55:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:47 smithi022 ceph-mon[31360]: pgmap v5319: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:55:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:47 smithi022 ceph-mon[40250]: pgmap v5319: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:55:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:47 smithi122 ceph-mon[32255]: pgmap v5319: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:50] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:55:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:50 smithi122 ceph-mon[32255]: pgmap v5320: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:50.450 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:50 smithi022 ceph-mon[31360]: pgmap v5320: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:50.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:50 smithi022 ceph-mon[40250]: pgmap v5320: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:50.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:50.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:50.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/553929597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/553929597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/553929597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:52 smithi122 ceph-mon[32255]: pgmap v5321: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:52.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:55:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:55:52.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:55:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:55:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:52 smithi022 ceph-mon[31360]: pgmap v5321: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:52 smithi022 ceph-mon[40250]: pgmap v5321: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:53 smithi122 ceph-mon[32255]: pgmap v5322: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:53 smithi022 ceph-mon[31360]: pgmap v5322: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:55:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:53 smithi022 ceph-mon[40250]: pgmap v5322: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:55:55.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:55:56.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:55:56.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:55:56.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:55:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:56 smithi022 ceph-mon[31360]: pgmap v5323: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2816188984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:56 smithi022 ceph-mon[40250]: pgmap v5323: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2816188984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:56 smithi122 ceph-mon[32255]: pgmap v5323: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2816188984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:55:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:55:58.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:55:58 smithi122 ceph-mon[32255]: pgmap v5324: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:55:58 smithi022 ceph-mon[31360]: pgmap v5324: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:55:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:55:58 smithi022 ceph-mon[40250]: pgmap v5324: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:00 smithi122 ceph-mon[32255]: pgmap v5325: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:00] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:56:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:00 smithi022 ceph-mon[31360]: pgmap v5325: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:00 smithi022 ceph-mon[40250]: pgmap v5325: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:01.191 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:01.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:01.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:01.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:02.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:56:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:56:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:56:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:56:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:02 smithi122 ceph-mon[32255]: pgmap v5326: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2157749503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:02 smithi022 ceph-mon[31360]: pgmap v5326: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2157749503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:02 smithi022 ceph-mon[40250]: pgmap v5326: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2157749503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:03 smithi122 ceph-mon[32255]: pgmap v5327: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:03 smithi022 ceph-mon[31360]: pgmap v5327: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:03 smithi022 ceph-mon[40250]: pgmap v5327: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:06.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:06.565 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:06 smithi022 ceph-mon[40250]: pgmap v5328: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:06.566 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:06 smithi022 ceph-mon[31360]: pgmap v5328: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:06 smithi122 ceph-mon[32255]: pgmap v5328: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:06.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:06.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:06.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/824643570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/824643570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:56:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/824643570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:08 smithi122 ceph-mon[32255]: pgmap v5329: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:08 smithi022 ceph-mon[31360]: pgmap v5329: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:08 smithi022 ceph-mon[40250]: pgmap v5329: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:10 smithi122 ceph-mon[32255]: pgmap v5330: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:10] "GET /metrics HTTP/1.1" 200 202345 "" "Prometheus/2.18.1" 2021-12-15T20:56:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:10 smithi022 ceph-mon[31360]: pgmap v5330: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:10 smithi022 ceph-mon[40250]: pgmap v5330: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:11.936 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:12.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:12.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:12.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:12.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:56:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:56:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:56:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:56:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:12 smithi122 ceph-mon[32255]: pgmap v5331: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806680873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:12 smithi022 ceph-mon[31360]: pgmap v5331: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806680873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:12 smithi022 ceph-mon[40250]: pgmap v5331: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806680873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:13 smithi122 ceph-mon[32255]: pgmap v5332: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:13 smithi022 ceph-mon[31360]: pgmap v5332: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:13 smithi022 ceph-mon[40250]: pgmap v5332: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:16 smithi122 ceph-mon[32255]: pgmap v5333: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:16 smithi022 ceph-mon[31360]: pgmap v5333: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:16 smithi022 ceph-mon[40250]: pgmap v5333: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:17.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:17.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:17.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:17.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:17.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:56:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:18 smithi122 ceph-mon[32255]: pgmap v5334: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1415285266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:18 smithi022 ceph-mon[31360]: pgmap v5334: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1415285266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:18 smithi022 ceph-mon[40250]: pgmap v5334: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1415285266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:20 smithi122 ceph-mon[32255]: pgmap v5335: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:20] "GET /metrics HTTP/1.1" 200 202346 "" "Prometheus/2.18.1" 2021-12-15T20:56:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:20 smithi022 ceph-mon[31360]: pgmap v5335: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:20 smithi022 ceph-mon[40250]: pgmap v5335: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:22.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:22.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:22 smithi022 ceph-mon[31360]: pgmap v5336: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:22.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:22 smithi022 ceph-mon[40250]: pgmap v5336: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:56:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:56:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:22 smithi122 ceph-mon[32255]: pgmap v5336: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:56:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:56:23.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:23.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:23.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3127666816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:23 smithi122 ceph-mon[32255]: pgmap v5337: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3127666816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:23 smithi022 ceph-mon[31360]: pgmap v5337: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3127666816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:23 smithi022 ceph-mon[40250]: pgmap v5337: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:26 smithi122 ceph-mon[32255]: pgmap v5338: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:26 smithi022 ceph-mon[31360]: pgmap v5338: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:26 smithi022 ceph-mon[40250]: pgmap v5338: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:56:28.047 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:28.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:28.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:28.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:28 smithi122 ceph-mon[32255]: pgmap v5339: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:28 smithi022 ceph-mon[31360]: pgmap v5339: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:28 smithi022 ceph-mon[40250]: pgmap v5339: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2959369738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2959369738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2959369738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:30 smithi122 ceph-mon[32255]: pgmap v5340: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:30] "GET /metrics HTTP/1.1" 200 202346 "" "Prometheus/2.18.1" 2021-12-15T20:56:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:30 smithi022 ceph-mon[31360]: pgmap v5340: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:30 smithi022 ceph-mon[40250]: pgmap v5340: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:32 smithi122 ceph-mon[32255]: pgmap v5341: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:32.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:56:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:56:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:56:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:56:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:32 smithi022 ceph-mon[31360]: pgmap v5341: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:32 smithi022 ceph-mon[40250]: pgmap v5341: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:33.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:33.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:56:33.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:56:33.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:56:33.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:56:33.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[40250]: pgmap v5342: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:33.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:33.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:33.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:56:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:33 smithi022 ceph-mon[31360]: pgmap v5342: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:56:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:56:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:56:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:56:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:33 smithi122 ceph-mon[32255]: pgmap v5342: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2041629777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2041629777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2041629777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:35 smithi022 ceph-mon[31360]: pgmap v5343: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:35 smithi022 ceph-mon[40250]: pgmap v5343: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:35 smithi122 ceph-mon[32255]: pgmap v5343: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:56:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:38 smithi122 ceph-mon[32255]: pgmap v5344: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:38 smithi022 ceph-mon[31360]: pgmap v5344: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:38 smithi022 ceph-mon[40250]: pgmap v5344: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:38.788 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2560515709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2560515709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2560515709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:40] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:56:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:40 smithi122 ceph-mon[32255]: pgmap v5345: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:40 smithi022 ceph-mon[31360]: pgmap v5345: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:40 smithi022 ceph-mon[40250]: pgmap v5345: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:42 smithi122 ceph-mon[32255]: pgmap v5346: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:42.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:56:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:56:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:56:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:56:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:42 smithi022 ceph-mon[31360]: pgmap v5346: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:42 smithi022 ceph-mon[40250]: pgmap v5346: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:43 smithi122 ceph-mon[32255]: pgmap v5347: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:43 smithi022 ceph-mon[31360]: pgmap v5347: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:43 smithi022 ceph-mon[40250]: pgmap v5347: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:44.161 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:44.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:44.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:44.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2079248060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2079248060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2079248060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:45 smithi022 ceph-mon[31360]: pgmap v5348: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:45 smithi022 ceph-mon[40250]: pgmap v5348: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:45 smithi122 ceph-mon[32255]: pgmap v5348: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:56:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:56:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:56:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:56:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:56:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:56:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:56:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:56:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:56:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:56:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:56:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:48 smithi122 ceph-mon[32255]: pgmap v5349: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:56:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:48 smithi022 ceph-mon[31360]: pgmap v5349: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:56:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:48 smithi022 ceph-mon[40250]: pgmap v5349: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:49.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:49.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:49.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:49.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:50] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:56:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:50 smithi122 ceph-mon[32255]: pgmap v5350: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2639143726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:50 smithi022 ceph-mon[31360]: pgmap v5350: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2639143726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:50 smithi022 ceph-mon[40250]: pgmap v5350: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2639143726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:52 smithi122 ceph-mon[32255]: pgmap v5351: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:52.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:56:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:56:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:56:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:56:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:52 smithi022 ceph-mon[31360]: pgmap v5351: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:52 smithi022 ceph-mon[40250]: pgmap v5351: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:53 smithi022 ceph-mon[31360]: pgmap v5352: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:53 smithi022 ceph-mon[40250]: pgmap v5352: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:56:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:53 smithi122 ceph-mon[32255]: pgmap v5352: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:56:54.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:56:55.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:56:55.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:56:55.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:56:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396742334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396742334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396742334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:56:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:56 smithi122 ceph-mon[32255]: pgmap v5353: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:56 smithi022 ceph-mon[31360]: pgmap v5353: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:56 smithi022 ceph-mon[40250]: pgmap v5353: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:56:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:56:58 smithi122 ceph-mon[32255]: pgmap v5354: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:56:58 smithi022 ceph-mon[31360]: pgmap v5354: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:56:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:56:58 smithi022 ceph-mon[40250]: pgmap v5354: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:00.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:00] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T20:57:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:00 smithi122 ceph-mon[32255]: pgmap v5355: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:00 smithi022 ceph-mon[31360]: pgmap v5355: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:00 smithi022 ceph-mon[40250]: pgmap v5355: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:00.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:00.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:00.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4037548413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4037548413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4037548413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:02.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:57:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:57:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:02 smithi122 ceph-mon[32255]: pgmap v5356: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:57:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:57:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:02 smithi022 ceph-mon[31360]: pgmap v5356: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:02 smithi022 ceph-mon[40250]: pgmap v5356: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:03 smithi022 ceph-mon[31360]: pgmap v5357: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:03 smithi022 ceph-mon[40250]: pgmap v5357: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:03 smithi122 ceph-mon[32255]: pgmap v5357: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:05.645 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:06.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:06.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:06.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:06 smithi122 ceph-mon[32255]: pgmap v5358: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1095132999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:06 smithi022 ceph-mon[31360]: pgmap v5358: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1095132999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:06 smithi022 ceph-mon[40250]: pgmap v5358: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1095132999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:57:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:08 smithi122 ceph-mon[32255]: pgmap v5359: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:08 smithi022 ceph-mon[40250]: pgmap v5359: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:08 smithi022 ceph-mon[31360]: pgmap v5359: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:10] "GET /metrics HTTP/1.1" 200 202346 "" "Prometheus/2.18.1" 2021-12-15T20:57:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:10 smithi122 ceph-mon[32255]: pgmap v5360: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:10 smithi022 ceph-mon[31360]: pgmap v5360: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:10 smithi022 ceph-mon[40250]: pgmap v5360: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:11.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:11.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:11.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:12.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:57:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:57:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:12 smithi122 ceph-mon[32255]: pgmap v5361: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/762546664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:57:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:57:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:12 smithi022 ceph-mon[31360]: pgmap v5361: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/762546664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:12 smithi022 ceph-mon[40250]: pgmap v5361: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/762546664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:13 smithi022 ceph-mon[31360]: pgmap v5362: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:13 smithi022 ceph-mon[40250]: pgmap v5362: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:13 smithi122 ceph-mon[32255]: pgmap v5362: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:16.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:16 smithi122 ceph-mon[32255]: pgmap v5363: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:16.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:16.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:16.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:16.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:16 smithi022 ceph-mon[31360]: pgmap v5363: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:16.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:16 smithi022 ceph-mon[40250]: pgmap v5363: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3408498962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3408498962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:57:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3408498962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:18 smithi122 ceph-mon[32255]: pgmap v5364: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:18 smithi022 ceph-mon[31360]: pgmap v5364: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:18 smithi022 ceph-mon[40250]: pgmap v5364: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:20] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T20:57:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:20 smithi122 ceph-mon[32255]: pgmap v5365: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:20 smithi022 ceph-mon[31360]: pgmap v5365: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:20 smithi022 ceph-mon[40250]: pgmap v5365: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:21.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:22.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:22.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:22.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:22 smithi022 ceph-mon[31360]: pgmap v5366: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3349935495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:22 smithi022 ceph-mon[40250]: pgmap v5366: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3349935495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:57:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:57:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:22 smithi122 ceph-mon[32255]: pgmap v5366: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3349935495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:22.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:57:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:57:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:23 smithi022 ceph-mon[31360]: pgmap v5367: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:23 smithi022 ceph-mon[40250]: pgmap v5367: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:23 smithi122 ceph-mon[32255]: pgmap v5367: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:26 smithi122 ceph-mon[32255]: pgmap v5368: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:26 smithi022 ceph-mon[31360]: pgmap v5368: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:26 smithi022 ceph-mon[40250]: pgmap v5368: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:27.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:27.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:27.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:27.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:57:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:28 smithi122 ceph-mon[32255]: pgmap v5369: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1530574479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:28 smithi022 ceph-mon[31360]: pgmap v5369: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1530574479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:28 smithi022 ceph-mon[40250]: pgmap v5369: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1530574479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:30 smithi122 ceph-mon[32255]: pgmap v5370: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:30] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T20:57:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:30 smithi022 ceph-mon[31360]: pgmap v5370: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:30 smithi022 ceph-mon[40250]: pgmap v5370: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:32.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:32 smithi122 ceph-mon[32255]: pgmap v5371: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:32.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:57:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:57:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:57:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:57:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:32 smithi022 ceph-mon[40250]: pgmap v5371: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:32 smithi022 ceph-mon[31360]: pgmap v5371: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:32.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:32.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:32.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:57:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/688589679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[31360]: pgmap v5372: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:57:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:57:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:57:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:57:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/688589679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:33 smithi022 ceph-mon[40250]: pgmap v5372: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:57:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:57:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:57:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:57:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/688589679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:33 smithi122 ceph-mon[32255]: pgmap v5372: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:36 smithi122 ceph-mon[32255]: pgmap v5373: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:36 smithi022 ceph-mon[31360]: pgmap v5373: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:36 smithi022 ceph-mon[40250]: pgmap v5373: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:57:37.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:38.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:38.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:38.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:38 smithi022 ceph-mon[31360]: pgmap v5374: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/800421019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:38 smithi022 ceph-mon[40250]: pgmap v5374: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/800421019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:38 smithi122 ceph-mon[32255]: pgmap v5374: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/800421019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:40 smithi122 ceph-mon[32255]: pgmap v5375: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:40] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T20:57:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:40 smithi022 ceph-mon[31360]: pgmap v5375: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:40 smithi022 ceph-mon[40250]: pgmap v5375: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:57:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:57:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:57:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:57:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:42 smithi122 ceph-mon[32255]: pgmap v5376: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:42 smithi022 ceph-mon[31360]: pgmap v5376: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:42 smithi022 ceph-mon[40250]: pgmap v5376: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:43.247 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:43.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:43.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:43.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:43 smithi022 ceph-mon[31360]: pgmap v5377: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2761673745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:43 smithi022 ceph-mon[40250]: pgmap v5377: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2761673745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:43 smithi122 ceph-mon[32255]: pgmap v5377: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2761673745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:46 smithi122 ceph-mon[32255]: pgmap v5378: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:46 smithi022 ceph-mon[31360]: pgmap v5378: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:46 smithi022 ceph-mon[40250]: pgmap v5378: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:47.348 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:57:47.349 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:57:47.349 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:57:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:57:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:57:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:57:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:57:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:57:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:57:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:57:48.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:48.623 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[31360]: pgmap v5379: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:48.623 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.623 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.624 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.624 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[40250]: pgmap v5379: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:48.624 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.625 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.625 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:48 smithi122 ceph-mon[32255]: pgmap v5379: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:57:48.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:48.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:48.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1171642600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1171642600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1171642600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:50] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T20:57:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:50 smithi122 ceph-mon[32255]: pgmap v5380: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:50 smithi022 ceph-mon[31360]: pgmap v5380: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:50.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:50 smithi022 ceph-mon[40250]: pgmap v5380: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:57:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:52 smithi122 ceph-mon[32255]: pgmap v5381: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:52.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:57:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:57:52.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:57:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:57:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:52 smithi022 ceph-mon[31360]: pgmap v5381: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:52 smithi022 ceph-mon[40250]: pgmap v5381: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:57:53.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:53.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:53.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:53 smithi022 ceph-mon[31360]: pgmap v5382: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:53.995 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:53.995 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:53 smithi022 ceph-mon[40250]: pgmap v5382: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:57:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:53 smithi122 ceph-mon[32255]: pgmap v5382: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:54.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:54.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:54.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:57:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1832441864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1832441864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1832441864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:57:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:55 smithi022 ceph-mon[31360]: pgmap v5383: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:55 smithi022 ceph-mon[40250]: pgmap v5383: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:55 smithi122 ceph-mon[32255]: pgmap v5383: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:57:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:57:58 smithi122 ceph-mon[32255]: pgmap v5384: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:57:58 smithi022 ceph-mon[31360]: pgmap v5384: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:57:58 smithi022 ceph-mon[40250]: pgmap v5384: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:57:59.366 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:57:59.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:57:59.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:57:59.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:00] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T20:58:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:00 smithi122 ceph-mon[32255]: pgmap v5385: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:58:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/114582334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:00 smithi022 ceph-mon[31360]: pgmap v5385: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:58:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/114582334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:00 smithi022 ceph-mon[40250]: pgmap v5385: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:58:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/114582334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:02 smithi122 ceph-mon[32255]: pgmap v5386: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:58:02.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:58:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:58:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:58:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:58:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:02 smithi022 ceph-mon[31360]: pgmap v5386: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:58:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:02 smithi022 ceph-mon[40250]: pgmap v5386: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T20:58:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:03 smithi022 ceph-mon[31360]: pgmap v5387: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:58:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:03 smithi022 ceph-mon[40250]: pgmap v5387: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:58:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:03 smithi122 ceph-mon[32255]: pgmap v5387: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T20:58:04.734 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:05.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:05.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:05.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/60901416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/60901416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/60901416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:06 smithi122 ceph-mon[32255]: pgmap v5388: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:06 smithi022 ceph-mon[31360]: pgmap v5388: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:06 smithi022 ceph-mon[40250]: pgmap v5388: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:58:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:08 smithi122 ceph-mon[32255]: pgmap v5389: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:08 smithi022 ceph-mon[31360]: pgmap v5389: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:08 smithi022 ceph-mon[40250]: pgmap v5389: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:10.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:10] "GET /metrics HTTP/1.1" 200 202342 "" "Prometheus/2.18.1" 2021-12-15T20:58:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:10 smithi122 ceph-mon[32255]: pgmap v5390: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:58:10.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:10.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:10.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:10.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:10 smithi022 ceph-mon[31360]: pgmap v5390: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:58:10.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:10 smithi022 ceph-mon[40250]: pgmap v5390: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:58:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1691812013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1691812013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1691812013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:12 smithi122 ceph-mon[32255]: pgmap v5391: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:12.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:58:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:58:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:58:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:58:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:12 smithi022 ceph-mon[31360]: pgmap v5391: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:12 smithi022 ceph-mon[40250]: pgmap v5391: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T20:58:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:13 smithi022 ceph-mon[31360]: pgmap v5392: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:58:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:13 smithi022 ceph-mon[40250]: pgmap v5392: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:58:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:13 smithi122 ceph-mon[32255]: pgmap v5392: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T20:58:15.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:15.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:15.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:15.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:16 smithi122 ceph-mon[32255]: pgmap v5393: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2344176811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:16 smithi022 ceph-mon[31360]: pgmap v5393: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2344176811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:16 smithi022 ceph-mon[40250]: pgmap v5393: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2344176811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:58:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:18 smithi122 ceph-mon[32255]: pgmap v5394: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:18 smithi022 ceph-mon[31360]: pgmap v5394: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:18.834 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:18 smithi022 ceph-mon[40250]: pgmap v5394: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:20 smithi122 ceph-mon[32255]: pgmap v5395: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:20] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T20:58:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:20 smithi022 ceph-mon[40250]: pgmap v5395: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:20 smithi022 ceph-mon[31360]: pgmap v5395: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2902036949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2902036949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2902036949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:22 smithi122 ceph-mon[32255]: pgmap v5396: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:22.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:58:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:58:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:58:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:58:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:22 smithi022 ceph-mon[31360]: pgmap v5396: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:22 smithi022 ceph-mon[40250]: pgmap v5396: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:23 smithi022 ceph-mon[31360]: pgmap v5397: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:23 smithi022 ceph-mon[40250]: pgmap v5397: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:23 smithi122 ceph-mon[32255]: pgmap v5397: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:26.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:26 smithi022 ceph-mon[31360]: pgmap v5398: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:26 smithi022 ceph-mon[40250]: pgmap v5398: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:26.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:26.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:26.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:26 smithi122 ceph-mon[32255]: pgmap v5398: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/695743369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/695743369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:58:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/695743369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:28 smithi122 ceph-mon[32255]: pgmap v5399: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:28 smithi022 ceph-mon[31360]: pgmap v5399: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:28 smithi022 ceph-mon[40250]: pgmap v5399: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:30] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T20:58:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:30 smithi122 ceph-mon[32255]: pgmap v5400: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:30 smithi022 ceph-mon[31360]: pgmap v5400: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:30 smithi022 ceph-mon[40250]: pgmap v5400: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:31.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:31.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:31.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:31.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:58:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:58:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:58:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:58:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:32 smithi122 ceph-mon[32255]: pgmap v5401: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/106069976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:32 smithi022 ceph-mon[31360]: pgmap v5401: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/106069976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:32 smithi022 ceph-mon[40250]: pgmap v5401: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/106069976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:58:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[31360]: pgmap v5402: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:58:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:58:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:58:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:58:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:33 smithi022 ceph-mon[40250]: pgmap v5402: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:58:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:58:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:58:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:58:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:33 smithi122 ceph-mon[32255]: pgmap v5402: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:36 smithi122 ceph-mon[32255]: pgmap v5403: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:36 smithi022 ceph-mon[31360]: pgmap v5403: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:36.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:36 smithi022 ceph-mon[40250]: pgmap v5403: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:36.960 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:37.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:37.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:37.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2846167125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2846167125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:58:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2846167125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:38 smithi122 ceph-mon[32255]: pgmap v5404: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:38 smithi022 ceph-mon[31360]: pgmap v5404: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:38 smithi022 ceph-mon[40250]: pgmap v5404: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:40 smithi122 ceph-mon[32255]: pgmap v5405: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:40] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T20:58:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:40 smithi022 ceph-mon[31360]: pgmap v5405: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:40 smithi022 ceph-mon[40250]: pgmap v5405: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:42.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:58:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:58:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:42 smithi122 ceph-mon[32255]: pgmap v5406: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:58:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:58:42.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:42.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:42.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:42.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:42 smithi022 ceph-mon[31360]: pgmap v5406: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:42.706 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:42 smithi022 ceph-mon[40250]: pgmap v5406: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2006892361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2006892361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2006892361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:44 smithi122 ceph-mon[32255]: pgmap v5407: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:44 smithi022 ceph-mon[31360]: pgmap v5407: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:44 smithi022 ceph-mon[40250]: pgmap v5407: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:46 smithi122 ceph-mon[32255]: pgmap v5408: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:46 smithi022 ceph-mon[31360]: pgmap v5408: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:46 smithi022 ceph-mon[40250]: pgmap v5408: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:47 smithi122 ceph-mon[32255]: pgmap v5409: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:47.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:47.707 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:58:47.708 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:47 smithi022 ceph-mon[31360]: pgmap v5409: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:47.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:47 smithi022 ceph-mon[40250]: pgmap v5409: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:48.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:48.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:48.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:58:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:58:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:58:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1473868091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:58:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:58:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:58:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:58:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1473868091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:58:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:58:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:58:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:58:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1473868091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:58:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:49 smithi022 ceph-mon[31360]: pgmap v5410: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:49 smithi022 ceph-mon[40250]: pgmap v5410: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:49 smithi122 ceph-mon[32255]: pgmap v5410: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:50] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T20:58:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:52 smithi122 ceph-mon[32255]: pgmap v5411: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:52.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:58:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:58:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:58:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:58:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:52 smithi022 ceph-mon[31360]: pgmap v5411: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:52 smithi022 ceph-mon[40250]: pgmap v5411: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:53.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:53.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:53.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:53.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:53 smithi022 ceph-mon[31360]: pgmap v5412: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1974609148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:53 smithi022 ceph-mon[40250]: pgmap v5412: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1974609148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:58:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:53 smithi122 ceph-mon[32255]: pgmap v5412: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:58:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1974609148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:56 smithi122 ceph-mon[32255]: pgmap v5413: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:56 smithi022 ceph-mon[31360]: pgmap v5413: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:56 smithi022 ceph-mon[40250]: pgmap v5413: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:58:58.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:58:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:58 smithi122 ceph-mon[32255]: pgmap v5414: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:58 smithi022 ceph-mon[31360]: pgmap v5414: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:58 smithi022 ceph-mon[40250]: pgmap v5414: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:58:58.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:58:58.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:58:58.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:58:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:58:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2096505363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:58:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2096505363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:58:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:58:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2096505363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:00 smithi122 ceph-mon[32255]: pgmap v5415: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:00] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T20:59:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:00 smithi022 ceph-mon[31360]: pgmap v5415: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:00 smithi022 ceph-mon[40250]: pgmap v5415: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:02 smithi122 ceph-mon[32255]: pgmap v5416: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:59:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:59:02.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:59:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:59:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:02 smithi022 ceph-mon[31360]: pgmap v5416: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:02 smithi022 ceph-mon[40250]: pgmap v5416: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:03 smithi022 ceph-mon[31360]: pgmap v5417: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:03 smithi022 ceph-mon[40250]: pgmap v5417: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:03 smithi122 ceph-mon[32255]: pgmap v5417: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:04.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:04.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:04.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3725912184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3725912184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3725912184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:05 smithi022 ceph-mon[31360]: pgmap v5418: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:05 smithi022 ceph-mon[40250]: pgmap v5418: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:05 smithi122 ceph-mon[32255]: pgmap v5418: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:59:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:08 smithi122 ceph-mon[32255]: pgmap v5419: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:08 smithi022 ceph-mon[31360]: pgmap v5419: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:08 smithi022 ceph-mon[40250]: pgmap v5419: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T20:59:09.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:09.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:09.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:09.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:10 smithi122 ceph-mon[32255]: pgmap v5420: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2359702784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:10] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T20:59:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:10 smithi022 ceph-mon[31360]: pgmap v5420: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2359702784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:10 smithi022 ceph-mon[40250]: pgmap v5420: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T20:59:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2359702784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:12 smithi122 ceph-mon[32255]: pgmap v5421: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:59:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:59:12.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:59:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:59:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:12 smithi022 ceph-mon[31360]: pgmap v5421: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:12 smithi022 ceph-mon[40250]: pgmap v5421: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:13 smithi022 ceph-mon[31360]: pgmap v5422: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:13 smithi022 ceph-mon[40250]: pgmap v5422: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:13 smithi122 ceph-mon[32255]: pgmap v5422: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:14.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:14.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:14.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:14.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2855577079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2855577079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2855577079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:15 smithi022 ceph-mon[31360]: pgmap v5423: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:15 smithi022 ceph-mon[40250]: pgmap v5423: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:15 smithi122 ceph-mon[32255]: pgmap v5423: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:59:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:18 smithi122 ceph-mon[32255]: pgmap v5424: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:18 smithi022 ceph-mon[31360]: pgmap v5424: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:18 smithi022 ceph-mon[40250]: pgmap v5424: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:19.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:20.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:20.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:20.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:20] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T20:59:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:20 smithi122 ceph-mon[32255]: pgmap v5425: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3453914981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:20 smithi022 ceph-mon[31360]: pgmap v5425: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3453914981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:20 smithi022 ceph-mon[40250]: pgmap v5425: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3453914981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:22 smithi122 ceph-mon[32255]: pgmap v5426: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:22.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:59:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:59:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:59:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:59:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:22 smithi022 ceph-mon[31360]: pgmap v5426: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:22 smithi022 ceph-mon[40250]: pgmap v5426: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:23 smithi022 ceph-mon[31360]: pgmap v5427: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:23 smithi022 ceph-mon[40250]: pgmap v5427: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:23 smithi122 ceph-mon[32255]: pgmap v5427: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:25.310 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:25.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:25.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:25.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:26 smithi122 ceph-mon[32255]: pgmap v5428: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3696715497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:26 smithi022 ceph-mon[31360]: pgmap v5428: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3696715497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:26 smithi022 ceph-mon[40250]: pgmap v5428: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3696715497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:59:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:28 smithi122 ceph-mon[32255]: pgmap v5429: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:28 smithi022 ceph-mon[31360]: pgmap v5429: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:28 smithi022 ceph-mon[40250]: pgmap v5429: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:30 smithi122 ceph-mon[32255]: pgmap v5430: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:30] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T20:59:30.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:30.684 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:30 smithi022 ceph-mon[31360]: pgmap v5430: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:30.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:30 smithi022 ceph-mon[40250]: pgmap v5430: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:31.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:31.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:31.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2810037697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2810037697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2810037697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:59:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:59:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:59:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:59:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:32 smithi122 ceph-mon[32255]: pgmap v5431: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:32 smithi022 ceph-mon[31360]: pgmap v5431: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:32 smithi022 ceph-mon[40250]: pgmap v5431: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:59:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:59:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:59:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:59:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[31360]: pgmap v5432: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:59:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:59:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:59:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:59:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:33 smithi022 ceph-mon[40250]: pgmap v5432: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:59:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:59:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T20:59:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T20:59:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:33 smithi122 ceph-mon[32255]: pgmap v5432: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:36.053 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:36 smithi122 ceph-mon[32255]: pgmap v5433: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:36 smithi022 ceph-mon[31360]: pgmap v5433: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:36 smithi022 ceph-mon[40250]: pgmap v5433: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1442949660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1442949660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:59:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1442949660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:38 smithi122 ceph-mon[32255]: pgmap v5434: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:38 smithi022 ceph-mon[31360]: pgmap v5434: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:38 smithi022 ceph-mon[40250]: pgmap v5434: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:39 smithi122 ceph-mon[32255]: pgmap v5435: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:39 smithi022 ceph-mon[31360]: pgmap v5435: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:39 smithi022 ceph-mon[40250]: pgmap v5435: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:40] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T20:59:41.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:41.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:41.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:41.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:42.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:59:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:59:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:42 smithi122 ceph-mon[32255]: pgmap v5436: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1299905868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:59:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:59:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:42 smithi022 ceph-mon[31360]: pgmap v5436: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1299905868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:42 smithi022 ceph-mon[40250]: pgmap v5436: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1299905868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:43 smithi022 ceph-mon[31360]: pgmap v5437: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:43 smithi022 ceph-mon[40250]: pgmap v5437: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:43 smithi122 ceph-mon[32255]: pgmap v5437: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:46 smithi122 ceph-mon[32255]: pgmap v5438: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:46 smithi022 ceph-mon[31360]: pgmap v5438: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:46 smithi022 ceph-mon[40250]: pgmap v5438: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:46.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:47.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:47.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:47.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1179552569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:59:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1179552569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1179552569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:48 smithi122 ceph-mon[32255]: pgmap v5439: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:59:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:59:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:59:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[31360]: pgmap v5439: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:59:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:59:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:59:48.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[40250]: pgmap v5439: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:48.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T20:59:48.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T20:59:48.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T20:59:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:59:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:49 smithi122 ceph-mon[32255]: pgmap v5440: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:59:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:49 smithi022 ceph-mon[31360]: pgmap v5440: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T20:59:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:49 smithi022 ceph-mon[40250]: pgmap v5440: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 20:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:50] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T20:59:52.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:52 smithi022 ceph-mon[31360]: pgmap v5441: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:52 smithi022 ceph-mon[40250]: pgmap v5441: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:52.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:52.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:52.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:52.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 20:59:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T20:59:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:52 smithi122 ceph-mon[32255]: pgmap v5441: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 20:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T20:59:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T20:59:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3272131711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3272131711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3272131711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:54 smithi122 ceph-mon[32255]: pgmap v5442: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:54 smithi022 ceph-mon[31360]: pgmap v5442: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T20:59:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:54 smithi022 ceph-mon[40250]: pgmap v5442: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T20:59:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:56 smithi122 ceph-mon[32255]: pgmap v5443: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:56 smithi022 ceph-mon[31360]: pgmap v5443: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:56 smithi022 ceph-mon[40250]: pgmap v5443: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:57.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T20:59:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:57 smithi122 ceph-mon[32255]: pgmap v5444: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:57 smithi022 ceph-mon[31360]: pgmap v5444: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 20:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:20:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T20:59:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:57 smithi022 ceph-mon[40250]: pgmap v5444: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T20:59:57.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T20:59:57.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T20:59:57.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T20:59:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3017040471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3017040471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T20:59:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3017040471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 20:59:59 smithi022 ceph-mon[31360]: pgmap v5445: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 20:59:59 smithi022 ceph-mon[40250]: pgmap v5445: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 20:59:59 smithi122 ceph-mon[32255]: pgmap v5445: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:00] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:00:00.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:00 smithi122 conmon[74079]: level=info ts=2021-12-15T21:00:00.230Z caller=compact.go:495 component=tsdb msg="write block" mint=1639591200000 maxt=1639598400000 ulid=01FPZYA49J2357CZXXSSNE5PMB duration=51.59483ms 2021-12-15T21:00:00.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:00 smithi122 conmon[74079]: level=info ts=2021-12-15T21:00:00.239Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=2.704841ms 2021-12-15T21:00:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T21:00:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T21:00:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T21:00:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:01 smithi022 ceph-mon[31360]: pgmap v5446: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:01 smithi022 ceph-mon[40250]: pgmap v5446: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:01 smithi122 ceph-mon[32255]: pgmap v5446: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:02.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:00:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:00:02.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:02.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:00:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:00:03.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:03.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:03.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2050974295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:03 smithi022 ceph-mon[40250]: pgmap v5447: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:04.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:04.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2050974295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:04.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:03 smithi022 ceph-mon[31360]: pgmap v5447: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2050974295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:03 smithi122 ceph-mon[32255]: pgmap v5447: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:06 smithi122 ceph-mon[32255]: pgmap v5448: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:06 smithi022 ceph-mon[40250]: pgmap v5448: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:06 smithi022 ceph-mon[31360]: pgmap v5448: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:00:08.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:08.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:08.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:08.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:08.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:08 smithi022 ceph-mon[40250]: pgmap v5449: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:08.655 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:08 smithi022 ceph-mon[31360]: pgmap v5449: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:08 smithi122 ceph-mon[32255]: pgmap v5449: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1177017615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:09 smithi122 ceph-mon[32255]: pgmap v5450: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1177017615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:09 smithi022 ceph-mon[31360]: pgmap v5450: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1177017615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:09 smithi022 ceph-mon[40250]: pgmap v5450: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:00:12.578 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:12 smithi122 ceph-mon[32255]: pgmap v5451: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:12.579 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:00:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:00:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:12 smithi022 ceph-mon[31360]: pgmap v5451: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:12 smithi022 ceph-mon[40250]: pgmap v5451: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:12.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:00:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:00:13.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:14.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:14.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:14.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:14.026 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:14.026 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:13 smithi022 ceph-mon[31360]: pgmap v5452: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:14.027 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:14.027 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:13 smithi022 ceph-mon[40250]: pgmap v5452: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:13 smithi122 ceph-mon[32255]: pgmap v5452: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/815130649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/815130649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/815130649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:15 smithi022 ceph-mon[31360]: pgmap v5453: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:15 smithi022 ceph-mon[40250]: pgmap v5453: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:15 smithi122 ceph-mon[32255]: pgmap v5453: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:00:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:18 smithi122 ceph-mon[32255]: pgmap v5454: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:18 smithi022 ceph-mon[31360]: pgmap v5454: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:18 smithi022 ceph-mon[40250]: pgmap v5454: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:19.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:19.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:19.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:19.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1920786786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1920786786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1920786786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:20 smithi122 ceph-mon[32255]: pgmap v5455: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:20] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:00:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:20 smithi022 ceph-mon[31360]: pgmap v5455: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:20 smithi022 ceph-mon[40250]: pgmap v5455: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:21 smithi122 ceph-mon[32255]: pgmap v5456: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:21 smithi022 ceph-mon[31360]: pgmap v5456: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:21.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:21 smithi022 ceph-mon[40250]: pgmap v5456: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:22.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:00:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:00:22.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:00:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:00:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:23 smithi022 ceph-mon[31360]: pgmap v5457: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:23 smithi022 ceph-mon[40250]: pgmap v5457: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:23 smithi122 ceph-mon[32255]: pgmap v5457: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:24.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:24.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:24.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:24.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2682336037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2682336037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2682336037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:25 smithi122 ceph-mon[32255]: pgmap v5458: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:25 smithi022 ceph-mon[31360]: pgmap v5458: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:25 smithi022 ceph-mon[40250]: pgmap v5458: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:00:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:28 smithi122 ceph-mon[32255]: pgmap v5459: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:28 smithi022 ceph-mon[31360]: pgmap v5459: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:28 smithi022 ceph-mon[40250]: pgmap v5459: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:29 smithi122 ceph-mon[32255]: pgmap v5460: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:29.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:29 smithi022 ceph-mon[31360]: pgmap v5460: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:29.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:29 smithi022 ceph-mon[40250]: pgmap v5460: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:29.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:30.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:30.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:30.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:30] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:00:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/985963337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/985963337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/985963337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:31 smithi122 ceph-mon[32255]: pgmap v5461: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:31 smithi022 ceph-mon[31360]: pgmap v5461: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:31 smithi022 ceph-mon[40250]: pgmap v5461: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:32.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:00:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:00:32.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:00:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:00:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:00:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:00:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:00:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:00:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[31360]: pgmap v5462: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:00:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:00:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:00:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:00:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:33 smithi022 ceph-mon[40250]: pgmap v5462: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:00:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:00:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:00:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:00:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:33 smithi122 ceph-mon[32255]: pgmap v5462: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:35.144 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:35 smithi022 ceph-mon[31360]: pgmap v5463: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:35 smithi022 ceph-mon[40250]: pgmap v5463: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:35 smithi122 ceph-mon[32255]: pgmap v5463: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3961772727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3961772727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3961772727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:37 smithi022 ceph-mon[31360]: pgmap v5464: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:00:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:37 smithi022 ceph-mon[40250]: pgmap v5464: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:37 smithi122 ceph-mon[32255]: pgmap v5464: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:39 smithi022 ceph-mon[31360]: pgmap v5465: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:39 smithi022 ceph-mon[40250]: pgmap v5465: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:39 smithi122 ceph-mon[32255]: pgmap v5465: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:40] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:00:40.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:40.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:40.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:40.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1668211693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1668211693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1668211693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:41 smithi122 ceph-mon[32255]: pgmap v5466: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:41 smithi022 ceph-mon[31360]: pgmap v5466: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:41 smithi022 ceph-mon[40250]: pgmap v5466: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:42.606 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:00:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:00:42.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:00:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:00:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:43 smithi022 ceph-mon[31360]: pgmap v5467: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:43 smithi022 ceph-mon[40250]: pgmap v5467: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:43 smithi122 ceph-mon[32255]: pgmap v5467: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:45 smithi022 ceph-mon[31360]: pgmap v5468: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:45 smithi022 ceph-mon[40250]: pgmap v5468: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:45.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:45 smithi122 ceph-mon[32255]: pgmap v5468: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:46.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:46.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:46.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3345063906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3345063906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3345063906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:47 smithi022 ceph-mon[31360]: pgmap v5469: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:00:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:47 smithi022 ceph-mon[40250]: pgmap v5469: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:47 smithi122 ceph-mon[32255]: pgmap v5469: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:00:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:00:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:00:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:00:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:00:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:00:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:00:49.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:00:49.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:00:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:50] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:00:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:00:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:50 smithi122 ceph-mon[32255]: pgmap v5470: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:00:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:50 smithi022 ceph-mon[31360]: pgmap v5470: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:00:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:50 smithi022 ceph-mon[40250]: pgmap v5470: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:51.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:51 smithi022 ceph-mon[31360]: pgmap v5471: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:51 smithi022 ceph-mon[40250]: pgmap v5471: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:51.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:51 smithi122 ceph-mon[32255]: pgmap v5471: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:52.612 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2089993562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:52.612 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:00:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:00:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2089993562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2089993562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:52.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:00:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:00:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:53 smithi022 ceph-mon[31360]: pgmap v5472: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:53 smithi022 ceph-mon[40250]: pgmap v5472: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:00:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:53 smithi122 ceph-mon[32255]: pgmap v5472: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:55 smithi022 ceph-mon[31360]: pgmap v5473: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:55 smithi022 ceph-mon[40250]: pgmap v5473: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:55 smithi122 ceph-mon[32255]: pgmap v5473: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:56.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:00:56.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:00:56.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:00:56.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:00:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/777209117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/777209117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/777209117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:00:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:00:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:58 smithi122 ceph-mon[32255]: pgmap v5474: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:58 smithi022 ceph-mon[31360]: pgmap v5474: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:58 smithi022 ceph-mon[40250]: pgmap v5474: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:00:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:00:59 smithi022 ceph-mon[31360]: pgmap v5475: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:00:59 smithi022 ceph-mon[40250]: pgmap v5475: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:00:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:00:59 smithi122 ceph-mon[32255]: pgmap v5475: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:00] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:01:01.716 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:01 smithi022 ceph-mon[31360]: pgmap v5476: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:01.717 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:01 smithi022 ceph-mon[40250]: pgmap v5476: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:01 smithi122 ceph-mon[32255]: pgmap v5476: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:01.996 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:02.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:02.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:02.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:02.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/34092056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:02.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:01:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:01:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/34092056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/34092056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:02.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:01:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:01:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:03 smithi022 ceph-mon[31360]: pgmap v5477: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:03 smithi022 ceph-mon[40250]: pgmap v5477: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:03 smithi122 ceph-mon[32255]: pgmap v5477: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:05 smithi022 ceph-mon[31360]: pgmap v5478: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:05 smithi022 ceph-mon[40250]: pgmap v5478: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:05 smithi122 ceph-mon[32255]: pgmap v5478: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:07.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:07.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:07.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:07.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:07.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:07 smithi022 ceph-mon[31360]: pgmap v5479: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:07.739 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:01:07.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:07 smithi022 ceph-mon[40250]: pgmap v5479: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:07 smithi122 ceph-mon[32255]: pgmap v5479: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3875922523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3875922523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3875922523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:09 smithi022 ceph-mon[31360]: pgmap v5480: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:09 smithi022 ceph-mon[40250]: pgmap v5480: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:09 smithi122 ceph-mon[32255]: pgmap v5480: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:10] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:01:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:11 smithi022 ceph-mon[31360]: pgmap v5481: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:11 smithi022 ceph-mon[40250]: pgmap v5481: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:11 smithi122 ceph-mon[32255]: pgmap v5481: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:12.617 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:01:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:01:12.740 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:12.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:01:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:01:13.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:13.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:13.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1932282069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:13 smithi022 ceph-mon[31360]: pgmap v5482: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1932282069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:13 smithi022 ceph-mon[40250]: pgmap v5482: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1932282069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:13 smithi122 ceph-mon[32255]: pgmap v5482: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:15 smithi022 ceph-mon[31360]: pgmap v5483: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:15 smithi022 ceph-mon[40250]: pgmap v5483: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:15 smithi122 ceph-mon[32255]: pgmap v5483: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:17 smithi022 ceph-mon[31360]: pgmap v5484: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:01:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:17 smithi022 ceph-mon[40250]: pgmap v5484: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:17 smithi122 ceph-mon[32255]: pgmap v5484: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:18.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:18.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:18.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:18.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/326474359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/326474359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/326474359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:19 smithi022 ceph-mon[31360]: pgmap v5485: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:19 smithi022 ceph-mon[40250]: pgmap v5485: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:19 smithi122 ceph-mon[32255]: pgmap v5485: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:20] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:01:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:21 smithi022 ceph-mon[31360]: pgmap v5486: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:21 smithi022 ceph-mon[40250]: pgmap v5486: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:21 smithi122 ceph-mon[32255]: pgmap v5486: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:22.627 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:01:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:01:22.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:01:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:01:23.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:23.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:23.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:23.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:23 smithi022 ceph-mon[31360]: pgmap v5487: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:23 smithi022 ceph-mon[40250]: pgmap v5487: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:23 smithi122 ceph-mon[32255]: pgmap v5487: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3647884124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3647884124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3647884124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:25 smithi022 ceph-mon[31360]: pgmap v5488: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:25 smithi022 ceph-mon[40250]: pgmap v5488: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:25 smithi122 ceph-mon[32255]: pgmap v5488: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:27 smithi022 ceph-mon[31360]: pgmap v5489: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:01:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:27 smithi022 ceph-mon[40250]: pgmap v5489: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:27 smithi122 ceph-mon[32255]: pgmap v5489: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:28.848 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:29.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:29.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:29.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1326110252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1326110252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1326110252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:30 smithi122 ceph-mon[32255]: pgmap v5490: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:30] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:01:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:30 smithi022 ceph-mon[31360]: pgmap v5490: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:30 smithi022 ceph-mon[40250]: pgmap v5490: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:31 smithi022 ceph-mon[31360]: pgmap v5491: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:31 smithi022 ceph-mon[40250]: pgmap v5491: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:31 smithi122 ceph-mon[32255]: pgmap v5491: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:32.639 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:01:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:01:32.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:01:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:01:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:01:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[31360]: pgmap v5492: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:01:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:01:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:01:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:01:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:33 smithi022 ceph-mon[40250]: pgmap v5492: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:01:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:01:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:01:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:01:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:33 smithi122 ceph-mon[32255]: pgmap v5492: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:34.217 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:34.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:34.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:34.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2628729927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2628729927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2628729927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:35 smithi022 ceph-mon[31360]: pgmap v5493: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:35 smithi022 ceph-mon[40250]: pgmap v5493: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:35 smithi122 ceph-mon[32255]: pgmap v5493: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:37 smithi022 ceph-mon[31360]: pgmap v5494: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:37 smithi022 ceph-mon[40250]: pgmap v5494: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:01:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:37 smithi122 ceph-mon[32255]: pgmap v5494: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:39.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:39 smithi022 ceph-mon[31360]: pgmap v5495: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:39 smithi022 ceph-mon[40250]: pgmap v5495: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:39 smithi122 ceph-mon[32255]: pgmap v5495: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:39.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:39.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:39.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:40] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:01:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3661550145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3661550145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3661550145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:41 smithi022 ceph-mon[31360]: pgmap v5496: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:41 smithi022 ceph-mon[40250]: pgmap v5496: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:41 smithi122 ceph-mon[32255]: pgmap v5496: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:42.640 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:01:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:01:42.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:01:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:01:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:43 smithi022 ceph-mon[31360]: pgmap v5497: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:43 smithi022 ceph-mon[40250]: pgmap v5497: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:43 smithi122 ceph-mon[32255]: pgmap v5497: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:44.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:45.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:45.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:45.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/489422381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/489422381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/489422381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:46 smithi122 ceph-mon[32255]: pgmap v5498: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:46 smithi022 ceph-mon[31360]: pgmap v5498: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:46 smithi022 ceph-mon[40250]: pgmap v5498: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:01:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:48 smithi122 ceph-mon[32255]: pgmap v5499: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:48 smithi022 ceph-mon[31360]: pgmap v5499: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:48 smithi022 ceph-mon[40250]: pgmap v5499: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:01:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:01:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:01:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:01:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:01:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:01:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:01:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:01:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:01:50.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:50] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:01:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:50 smithi122 ceph-mon[32255]: pgmap v5500: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:50.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:50.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:50.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:50.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:50 smithi022 ceph-mon[40250]: pgmap v5500: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:50.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:50 smithi022 ceph-mon[31360]: pgmap v5500: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2890433538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2890433538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2890433538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:52.650 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:01:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:01:52.651 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:52 smithi122 ceph-mon[32255]: pgmap v5501: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:52 smithi022 ceph-mon[31360]: pgmap v5501: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:52 smithi022 ceph-mon[40250]: pgmap v5501: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:52.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:01:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:01:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:54 smithi122 ceph-mon[32255]: pgmap v5502: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[31360]: pgmap v5502: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:01:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[40250]: pgmap v5502: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:55.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:01:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:55 smithi122 ceph-mon[32255]: pgmap v5503: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:55 smithi022 ceph-mon[31360]: pgmap v5503: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:55 smithi022 ceph-mon[40250]: pgmap v5503: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:56.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:01:56.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:01:56.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:01:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1626082183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1626082183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:57.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1626082183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:01:57.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:01:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:58 smithi022 ceph-mon[31360]: pgmap v5504: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:58 smithi022 ceph-mon[40250]: pgmap v5504: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:01:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:58 smithi122 ceph-mon[32255]: pgmap v5504: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:01:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:01:59 smithi022 ceph-mon[31360]: pgmap v5505: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:01:59 smithi022 ceph-mon[40250]: pgmap v5505: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:01:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:01:59 smithi122 ceph-mon[32255]: pgmap v5505: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:00] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:02:01.082 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:01.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:01.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:01.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:01 smithi022 ceph-mon[31360]: pgmap v5506: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/217242077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:01 smithi022 ceph-mon[40250]: pgmap v5506: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/217242077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:01 smithi122 ceph-mon[32255]: pgmap v5506: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/217242077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:02.656 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:02:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:02:02.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:02:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:02:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:03 smithi022 ceph-mon[31360]: pgmap v5507: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:03 smithi022 ceph-mon[40250]: pgmap v5507: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:03 smithi122 ceph-mon[32255]: pgmap v5507: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:05 smithi022 ceph-mon[31360]: pgmap v5508: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:05 smithi022 ceph-mon[40250]: pgmap v5508: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:05 smithi122 ceph-mon[32255]: pgmap v5508: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:06.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:06.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:06.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:06.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1170539705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1170539705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1170539705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:02:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:07 smithi122 ceph-mon[32255]: pgmap v5509: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:07 smithi022 ceph-mon[31360]: pgmap v5509: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:07 smithi022 ceph-mon[40250]: pgmap v5509: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:09 smithi022 ceph-mon[31360]: pgmap v5510: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:09 smithi022 ceph-mon[40250]: pgmap v5510: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:09 smithi122 ceph-mon[32255]: pgmap v5510: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:10] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:02:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:11 smithi022 ceph-mon[31360]: pgmap v5511: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:11 smithi022 ceph-mon[40250]: pgmap v5511: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:11.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:11 smithi122 ceph-mon[32255]: pgmap v5511: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:12.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:12.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:12.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:12.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2643787974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:12.666 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:02:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:02:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2643787974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2643787974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:12.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:02:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:02:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:13 smithi022 ceph-mon[31360]: pgmap v5512: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:13 smithi022 ceph-mon[40250]: pgmap v5512: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:13 smithi122 ceph-mon[32255]: pgmap v5512: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:15 smithi022 ceph-mon[31360]: pgmap v5513: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:15 smithi022 ceph-mon[40250]: pgmap v5513: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:15 smithi122 ceph-mon[32255]: pgmap v5513: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:17.189 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:17.460 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:02:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:17.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:17.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:17 smithi022 ceph-mon[31360]: pgmap v5514: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:17 smithi022 ceph-mon[40250]: pgmap v5514: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:17 smithi122 ceph-mon[32255]: pgmap v5514: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/282592898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/282592898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/282592898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:19 smithi022 ceph-mon[31360]: pgmap v5515: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:19 smithi022 ceph-mon[40250]: pgmap v5515: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:19 smithi122 ceph-mon[32255]: pgmap v5515: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:20] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:02:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:21 smithi022 ceph-mon[31360]: pgmap v5516: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:21 smithi022 ceph-mon[40250]: pgmap v5516: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:21 smithi122 ceph-mon[32255]: pgmap v5516: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:22.558 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:22.673 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:02:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:02:22.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:22.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:22.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:22.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:02:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:02:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2835851297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:23 smithi022 ceph-mon[31360]: pgmap v5517: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2835851297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:23 smithi022 ceph-mon[40250]: pgmap v5517: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2835851297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:23 smithi122 ceph-mon[32255]: pgmap v5517: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:25 smithi022 ceph-mon[31360]: pgmap v5518: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:25 smithi022 ceph-mon[40250]: pgmap v5518: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:25 smithi122 ceph-mon[32255]: pgmap v5518: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:27 smithi022 ceph-mon[31360]: pgmap v5519: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:02:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:27 smithi022 ceph-mon[40250]: pgmap v5519: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:27.928 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:27 smithi122 ceph-mon[32255]: pgmap v5519: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:28.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:28.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:28.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2894084268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2894084268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2894084268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:29 smithi022 ceph-mon[31360]: pgmap v5520: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:29 smithi022 ceph-mon[40250]: pgmap v5520: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:29 smithi122 ceph-mon[32255]: pgmap v5520: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:30] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:02:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:31 smithi022 ceph-mon[31360]: pgmap v5521: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:31 smithi022 ceph-mon[40250]: pgmap v5521: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:31 smithi122 ceph-mon[32255]: pgmap v5521: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:32.677 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:02:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:02:32.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:02:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:02:33.302 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:33.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:33.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:33.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:02:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:02:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:02:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:02:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[31360]: pgmap v5522: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/559212963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[40250]: pgmap v5522: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/559212963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:02:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:02:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:02:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:02:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:33 smithi122 ceph-mon[32255]: pgmap v5522: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/559212963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:35 smithi022 ceph-mon[31360]: pgmap v5523: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:35 smithi022 ceph-mon[40250]: pgmap v5523: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:35 smithi122 ceph-mon[32255]: pgmap v5523: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:37 smithi022 ceph-mon[31360]: pgmap v5524: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:02:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:37 smithi022 ceph-mon[40250]: pgmap v5524: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:37 smithi122 ceph-mon[32255]: pgmap v5524: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:38.669 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:39.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:39.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:39.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3137655723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3137655723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3137655723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:40 smithi122 ceph-mon[32255]: pgmap v5525: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:40] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:02:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:40 smithi022 ceph-mon[31360]: pgmap v5525: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:40 smithi022 ceph-mon[40250]: pgmap v5525: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:41 smithi022 ceph-mon[31360]: pgmap v5526: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:41 smithi022 ceph-mon[40250]: pgmap v5526: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:41 smithi122 ceph-mon[32255]: pgmap v5526: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:42.681 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:02:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:02:42.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:02:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:02:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:43 smithi022 ceph-mon[31360]: pgmap v5527: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:43 smithi022 ceph-mon[40250]: pgmap v5527: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:43 smithi122 ceph-mon[32255]: pgmap v5527: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:44.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:44.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:44.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/551141494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/551141494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/551141494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:45 smithi022 ceph-mon[31360]: pgmap v5528: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:45 smithi022 ceph-mon[40250]: pgmap v5528: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:45 smithi122 ceph-mon[32255]: pgmap v5528: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:47 smithi022 ceph-mon[31360]: pgmap v5529: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:02:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:47 smithi022 ceph-mon[40250]: pgmap v5529: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:47 smithi122 ceph-mon[32255]: pgmap v5529: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:02:49.412 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:49 smithi022 ceph-mon[31360]: pgmap v5530: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:49 smithi022 ceph-mon[40250]: pgmap v5530: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:49.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:49.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:49.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:49 smithi122 ceph-mon[32255]: pgmap v5530: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:50] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:02:50.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1038281058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:50.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1038281058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1038281058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:51 smithi022 ceph-mon[31360]: pgmap v5531: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:51 smithi022 ceph-mon[40250]: pgmap v5531: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:51 smithi122 ceph-mon[32255]: pgmap v5531: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:02:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:02:53.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:02:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:02:54.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:54.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:53 smithi022 ceph-mon[31360]: pgmap v5532: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:54.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:54.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:53 smithi022 ceph-mon[40250]: pgmap v5532: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:02:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:53 smithi122 ceph-mon[32255]: pgmap v5532: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:54.785 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:02:55.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:02:55.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:02:55.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:02:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1826802846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1826802846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1826802846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:02:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:56 smithi022 ceph-mon[31360]: pgmap v5533: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:56 smithi022 ceph-mon[40250]: pgmap v5533: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:56.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:56 smithi122 ceph-mon[32255]: pgmap v5533: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:02:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:02:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:02:57.746 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:02:57.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:02:57.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:02:57.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:02:57.748 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:02:57.748 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:02:57.748 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:02:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:58 smithi122 ceph-mon[32255]: pgmap v5534: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:02:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:58 smithi022 ceph-mon[31360]: pgmap v5534: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:02:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:58 smithi022 ceph-mon[40250]: pgmap v5534: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:02:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:02:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:02:59 smithi022 ceph-mon[31360]: pgmap v5535: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:02:59 smithi022 ceph-mon[40250]: pgmap v5535: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:02:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:02:59 smithi122 ceph-mon[32255]: pgmap v5535: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:00.153 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:00] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:03:00.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:00.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:00.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2793994362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2793994362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2793994362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:01 smithi122 ceph-mon[32255]: pgmap v5536: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:01 smithi022 ceph-mon[31360]: pgmap v5536: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:01 smithi022 ceph-mon[40250]: pgmap v5536: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:03:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:03:03.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:03:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:03:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:03 smithi022 ceph-mon[31360]: pgmap v5537: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:03 smithi022 ceph-mon[40250]: pgmap v5537: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:03 smithi122 ceph-mon[32255]: pgmap v5537: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:05.522 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:05 smithi022 ceph-mon[31360]: pgmap v5538: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:05 smithi022 ceph-mon[40250]: pgmap v5538: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:05.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:05.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:05.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:05 smithi122 ceph-mon[32255]: pgmap v5538: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3064200960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3064200960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3064200960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:07 smithi022 ceph-mon[31360]: pgmap v5539: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:03:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:07 smithi022 ceph-mon[40250]: pgmap v5539: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:07 smithi122 ceph-mon[32255]: pgmap v5539: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:09 smithi022 ceph-mon[31360]: pgmap v5540: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:09 smithi022 ceph-mon[40250]: pgmap v5540: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:09 smithi122 ceph-mon[32255]: pgmap v5540: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:10] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:03:10.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:11.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:11.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:11.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/903938735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/903938735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/903938735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:12 smithi122 ceph-mon[32255]: pgmap v5541: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:03:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:03:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:12 smithi022 ceph-mon[31360]: pgmap v5541: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:12 smithi022 ceph-mon[40250]: pgmap v5541: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:13.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:03:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:03:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:13 smithi022 ceph-mon[31360]: pgmap v5542: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:13 smithi022 ceph-mon[40250]: pgmap v5542: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:13 smithi122 ceph-mon[32255]: pgmap v5542: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:15 smithi022 ceph-mon[31360]: pgmap v5543: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:15 smithi022 ceph-mon[40250]: pgmap v5543: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:15 smithi122 ceph-mon[32255]: pgmap v5543: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:16.261 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:16.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:16.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:16.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4100680052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4100680052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4100680052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:17.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:03:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:17 smithi122 ceph-mon[32255]: pgmap v5544: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:17 smithi022 ceph-mon[31360]: pgmap v5544: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:17 smithi022 ceph-mon[40250]: pgmap v5544: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:19 smithi022 ceph-mon[31360]: pgmap v5545: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:19 smithi022 ceph-mon[40250]: pgmap v5545: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:19 smithi122 ceph-mon[32255]: pgmap v5545: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:20] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:03:21.631 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:21 smithi022 ceph-mon[40250]: pgmap v5546: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:21 smithi022 ceph-mon[31360]: pgmap v5546: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:21 smithi122 ceph-mon[32255]: pgmap v5546: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:22.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:22.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:22.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/376022048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:03:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:03:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/376022048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/376022048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:23.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:03:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:03:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:23 smithi022 ceph-mon[31360]: pgmap v5547: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:23 smithi022 ceph-mon[40250]: pgmap v5547: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:23 smithi122 ceph-mon[32255]: pgmap v5547: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:25 smithi022 ceph-mon[31360]: pgmap v5548: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:25 smithi022 ceph-mon[40250]: pgmap v5548: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:25 smithi122 ceph-mon[32255]: pgmap v5548: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:27.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:27.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:27.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:27.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/599717769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/599717769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:03:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/599717769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:28 smithi122 ceph-mon[32255]: pgmap v5549: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:28 smithi022 ceph-mon[31360]: pgmap v5549: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:28 smithi022 ceph-mon[40250]: pgmap v5549: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:30 smithi122 ceph-mon[32255]: pgmap v5550: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:30] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:03:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:30 smithi022 ceph-mon[31360]: pgmap v5550: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:30 smithi022 ceph-mon[40250]: pgmap v5550: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:32.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:32 smithi122 ceph-mon[32255]: pgmap v5551: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:03:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:03:32.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:32.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:32.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:32.745 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:32 smithi022 ceph-mon[31360]: pgmap v5551: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:32.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:32 smithi022 ceph-mon[40250]: pgmap v5551: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:33.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:03:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:03:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2876814272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:03:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:03:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:03:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:03:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2876814272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:03:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:03:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:03:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:03:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2876814272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:03:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:03:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:03:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:03:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:34 smithi122 ceph-mon[32255]: pgmap v5552: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:34 smithi022 ceph-mon[31360]: pgmap v5552: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:34 smithi022 ceph-mon[40250]: pgmap v5552: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:35 smithi022 ceph-mon[31360]: pgmap v5553: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:35 smithi022 ceph-mon[40250]: pgmap v5553: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:35 smithi122 ceph-mon[32255]: pgmap v5553: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:37.746 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:37.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:03:37.747 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:37 smithi022 ceph-mon[31360]: pgmap v5554: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:37.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:37 smithi022 ceph-mon[40250]: pgmap v5554: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:37 smithi122 ceph-mon[32255]: pgmap v5554: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:38.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:38.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:38.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/419262129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/419262129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/419262129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:39 smithi022 ceph-mon[31360]: pgmap v5555: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:39 smithi022 ceph-mon[40250]: pgmap v5555: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:39 smithi122 ceph-mon[32255]: pgmap v5555: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:40] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:03:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:41 smithi022 ceph-mon[31360]: pgmap v5556: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:41 smithi022 ceph-mon[40250]: pgmap v5556: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:41 smithi122 ceph-mon[32255]: pgmap v5556: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:03:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:03:43.115 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:43.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:03:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:03:43.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:43.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:43.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:43 smithi022 ceph-mon[40250]: pgmap v5557: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/338933565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:43 smithi022 ceph-mon[31360]: pgmap v5557: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/338933565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:43 smithi122 ceph-mon[32255]: pgmap v5557: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/338933565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:45 smithi022 ceph-mon[31360]: pgmap v5558: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:45 smithi022 ceph-mon[40250]: pgmap v5558: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:45 smithi122 ceph-mon[32255]: pgmap v5558: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:47 smithi022 ceph-mon[31360]: pgmap v5559: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:03:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:47 smithi022 ceph-mon[40250]: pgmap v5559: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:47 smithi122 ceph-mon[32255]: pgmap v5559: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1883496585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1883496585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1883496585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:49 smithi122 ceph-mon[32255]: pgmap v5560: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:49 smithi022 ceph-mon[40250]: pgmap v5560: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:49 smithi022 ceph-mon[31360]: pgmap v5560: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:50] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:03:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:51 smithi022 ceph-mon[31360]: pgmap v5561: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:51 smithi022 ceph-mon[40250]: pgmap v5561: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:51 smithi122 ceph-mon[32255]: pgmap v5561: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:03:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:03:53.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:03:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:03:53.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:53 smithi122 ceph-mon[32255]: pgmap v5562: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:54.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:54.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:54.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:54.227 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:54.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:53 smithi022 ceph-mon[31360]: pgmap v5562: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:54.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:03:54.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:53 smithi022 ceph-mon[40250]: pgmap v5562: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2203409788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2203409788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2203409788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:03:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:55 smithi122 ceph-mon[32255]: pgmap v5563: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:55 smithi022 ceph-mon[31360]: pgmap v5563: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:55 smithi022 ceph-mon[40250]: pgmap v5563: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:57 smithi022 ceph-mon[31360]: pgmap v5564: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:03:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:57 smithi022 ceph-mon[40250]: pgmap v5564: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:57.789 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:57 smithi122 ceph-mon[32255]: pgmap v5564: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:03:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:03:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:03:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:03:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:03:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:03:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:03:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:03:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:03:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:03:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:03:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:03:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:03:59.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:03:59.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:03:59 smithi022 ceph-mon[40250]: pgmap v5565: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:59.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:03:59.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:03:59.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:03:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:03:59 smithi022 ceph-mon[31360]: pgmap v5565: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:03:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:03:59 smithi122 ceph-mon[32255]: pgmap v5565: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:00] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:04:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1204249267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1204249267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1204249267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:01 smithi022 ceph-mon[31360]: pgmap v5566: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:01 smithi022 ceph-mon[40250]: pgmap v5566: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:01 smithi122 ceph-mon[32255]: pgmap v5566: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:04:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:04:03.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:04:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:04:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:03 smithi122 ceph-mon[32255]: pgmap v5567: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:03 smithi022 ceph-mon[31360]: pgmap v5567: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:03 smithi022 ceph-mon[40250]: pgmap v5567: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:04.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:04.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:04.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/113709041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/113709041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/113709041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:06 smithi022 ceph-mon[31360]: pgmap v5568: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:06 smithi022 ceph-mon[40250]: pgmap v5568: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:06 smithi122 ceph-mon[32255]: pgmap v5568: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:04:07.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:07 smithi022 ceph-mon[31360]: pgmap v5569: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:07 smithi022 ceph-mon[40250]: pgmap v5569: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:07 smithi122 ceph-mon[32255]: pgmap v5569: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:09 smithi022 ceph-mon[31360]: pgmap v5570: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:09 smithi022 ceph-mon[40250]: pgmap v5570: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:09 smithi122 ceph-mon[32255]: pgmap v5570: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:09.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:10.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:10.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:10.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:10] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:04:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3830104941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3830104941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3830104941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:11 smithi022 ceph-mon[31360]: pgmap v5571: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:11 smithi022 ceph-mon[40250]: pgmap v5571: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:11 smithi122 ceph-mon[32255]: pgmap v5571: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:04:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:04:13.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:04:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:04:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:13 smithi122 ceph-mon[32255]: pgmap v5572: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:13 smithi022 ceph-mon[31360]: pgmap v5572: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:13 smithi022 ceph-mon[40250]: pgmap v5572: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:15.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:15.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:15.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:15.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:15.714 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:15 smithi022 ceph-mon[31360]: pgmap v5573: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:15.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:15 smithi022 ceph-mon[40250]: pgmap v5573: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:15 smithi122 ceph-mon[32255]: pgmap v5573: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2252622597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2252622597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2252622597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:04:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:17 smithi022 ceph-mon[31360]: pgmap v5574: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:17 smithi022 ceph-mon[40250]: pgmap v5574: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:17 smithi122 ceph-mon[32255]: pgmap v5574: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:19 smithi022 ceph-mon[31360]: pgmap v5575: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:19 smithi022 ceph-mon[40250]: pgmap v5575: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:19 smithi122 ceph-mon[32255]: pgmap v5575: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:20] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:04:20.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:21.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:21.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:21.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/440827470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/440827470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/440827470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:22 smithi122 ceph-mon[32255]: pgmap v5576: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:04:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:04:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:22 smithi022 ceph-mon[31360]: pgmap v5576: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:22.609 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:22 smithi022 ceph-mon[40250]: pgmap v5576: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:23.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:04:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:04:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:23 smithi122 ceph-mon[32255]: pgmap v5577: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:23 smithi022 ceph-mon[31360]: pgmap v5577: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:23 smithi022 ceph-mon[40250]: pgmap v5577: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:25 smithi022 ceph-mon[31360]: pgmap v5578: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:25 smithi022 ceph-mon[40250]: pgmap v5578: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:25 smithi122 ceph-mon[32255]: pgmap v5578: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:26.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:26.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:26.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:26.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3113676163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3113676163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3113676163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:27 smithi022 ceph-mon[31360]: pgmap v5579: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:04:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:27 smithi022 ceph-mon[40250]: pgmap v5579: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:27 smithi122 ceph-mon[32255]: pgmap v5579: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:29 smithi022 ceph-mon[31360]: pgmap v5580: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:29 smithi022 ceph-mon[40250]: pgmap v5580: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:29 smithi122 ceph-mon[32255]: pgmap v5580: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:30] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:04:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:31 smithi022 ceph-mon[31360]: pgmap v5581: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:31 smithi022 ceph-mon[40250]: pgmap v5581: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:31.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:31.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:31.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:31 smithi122 ceph-mon[32255]: pgmap v5581: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3942472348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:04:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:04:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3942472348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3942472348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:33.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:04:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:04:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:04:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:04:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:04:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:04:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:33 smithi122 ceph-mon[32255]: pgmap v5582: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:04:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:04:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:04:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:04:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[31360]: pgmap v5582: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:04:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:04:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:04:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:04:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:33 smithi022 ceph-mon[40250]: pgmap v5582: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:35 smithi022 ceph-mon[31360]: pgmap v5583: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:35 smithi022 ceph-mon[40250]: pgmap v5583: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:35 smithi122 ceph-mon[32255]: pgmap v5583: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:36.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:37.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:37.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:37.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2392285227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:04:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2392285227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2392285227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:38 smithi022 ceph-mon[31360]: pgmap v5584: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:38 smithi022 ceph-mon[40250]: pgmap v5584: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:38 smithi122 ceph-mon[32255]: pgmap v5584: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:39 smithi022 ceph-mon[31360]: pgmap v5585: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:39 smithi022 ceph-mon[40250]: pgmap v5585: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:39 smithi122 ceph-mon[32255]: pgmap v5585: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:40] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:04:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:41 smithi022 ceph-mon[31360]: pgmap v5586: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:41.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:41 smithi022 ceph-mon[40250]: pgmap v5586: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:41 smithi122 ceph-mon[32255]: pgmap v5586: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:42.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:42.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:42.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:42.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:42.609 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:04:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:04:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1215064285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:42.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:04:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:04:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1215064285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1215064285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:43 smithi122 ceph-mon[32255]: pgmap v5587: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:43 smithi022 ceph-mon[31360]: pgmap v5587: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:43 smithi022 ceph-mon[40250]: pgmap v5587: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:45 smithi022 ceph-mon[31360]: pgmap v5588: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:45 smithi022 ceph-mon[40250]: pgmap v5588: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:45 smithi122 ceph-mon[32255]: pgmap v5588: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:47.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:47 smithi022 ceph-mon[31360]: pgmap v5589: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:04:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:47 smithi022 ceph-mon[40250]: pgmap v5589: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:47 smithi122 ceph-mon[32255]: pgmap v5589: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:47.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:47.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:47.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1364977178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1364977178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1364977178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:49 smithi122 ceph-mon[32255]: pgmap v5590: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:49 smithi022 ceph-mon[31360]: pgmap v5590: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:49 smithi022 ceph-mon[40250]: pgmap v5590: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:50] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:04:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:51 smithi022 ceph-mon[31360]: pgmap v5591: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:51 smithi022 ceph-mon[40250]: pgmap v5591: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:51 smithi122 ceph-mon[32255]: pgmap v5591: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:04:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:04:52.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:53.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:04:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:04:53.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:53.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:53.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/747714093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:53 smithi122 ceph-mon[32255]: pgmap v5592: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/747714093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:53 smithi022 ceph-mon[31360]: pgmap v5592: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:04:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/747714093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:53 smithi022 ceph-mon[40250]: pgmap v5592: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:04:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:55 smithi022 ceph-mon[31360]: pgmap v5593: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:55 smithi022 ceph-mon[40250]: pgmap v5593: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:55 smithi122 ceph-mon[32255]: pgmap v5593: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:04:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:57 smithi022 ceph-mon[31360]: pgmap v5594: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:57 smithi022 ceph-mon[40250]: pgmap v5594: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:57 smithi122 ceph-mon[32255]: pgmap v5594: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:04:58.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:04:58.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:04:58.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:04:58.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:04:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:04:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:04:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:04:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/140056568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:04:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:04:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:04:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:04:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/140056568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:04:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:04:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:04:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:04:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/140056568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:04:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:05:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:04:59 smithi122 ceph-mon[32255]: pgmap v5595: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:04:59 smithi022 ceph-mon[31360]: pgmap v5595: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:04:59 smithi022 ceph-mon[40250]: pgmap v5595: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:00] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:05:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:01 smithi022 ceph-mon[31360]: pgmap v5596: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:01 smithi022 ceph-mon[40250]: pgmap v5596: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:01 smithi122 ceph-mon[32255]: pgmap v5596: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:05:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:05:03.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:05:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:05:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:03 smithi022 ceph-mon[31360]: pgmap v5597: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:03 smithi022 ceph-mon[40250]: pgmap v5597: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:04.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:04.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:04.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:03 smithi122 ceph-mon[32255]: pgmap v5597: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1657939957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1657939957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1657939957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:05 smithi122 ceph-mon[32255]: pgmap v5598: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:05 smithi022 ceph-mon[31360]: pgmap v5598: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:05 smithi022 ceph-mon[40250]: pgmap v5598: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:07 smithi022 ceph-mon[31360]: pgmap v5599: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:05:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:07 smithi022 ceph-mon[40250]: pgmap v5599: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:07 smithi122 ceph-mon[32255]: pgmap v5599: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:09.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:09.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:09.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:09.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:09 smithi022 ceph-mon[31360]: pgmap v5600: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1814337801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:09 smithi022 ceph-mon[40250]: pgmap v5600: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1814337801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:09 smithi122 ceph-mon[32255]: pgmap v5600: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1814337801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:10] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:05:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:11 smithi022 ceph-mon[31360]: pgmap v5601: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:11 smithi022 ceph-mon[40250]: pgmap v5601: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:11 smithi122 ceph-mon[32255]: pgmap v5601: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:05:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:05:13.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:05:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:05:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:13 smithi122 ceph-mon[32255]: pgmap v5602: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:13 smithi022 ceph-mon[31360]: pgmap v5602: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:13 smithi022 ceph-mon[40250]: pgmap v5602: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:14.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:14.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:14.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:14.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1305984101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1305984101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1305984101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:15 smithi122 ceph-mon[32255]: pgmap v5603: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:15 smithi022 ceph-mon[31360]: pgmap v5603: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:15 smithi022 ceph-mon[40250]: pgmap v5603: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:17 smithi022 ceph-mon[31360]: pgmap v5604: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:05:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:17 smithi022 ceph-mon[40250]: pgmap v5604: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:17 smithi122 ceph-mon[32255]: pgmap v5604: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:19 smithi022 ceph-mon[31360]: pgmap v5605: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:19 smithi022 ceph-mon[40250]: pgmap v5605: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:19.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:19 smithi122 ceph-mon[32255]: pgmap v5605: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:20.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:20.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:20.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:20] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:05:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2502813408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2502813408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2502813408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:21 smithi022 ceph-mon[31360]: pgmap v5606: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:21 smithi022 ceph-mon[40250]: pgmap v5606: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:21 smithi122 ceph-mon[32255]: pgmap v5606: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:05:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:05:23.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:05:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:05:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:23 smithi122 ceph-mon[32255]: pgmap v5607: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:23 smithi022 ceph-mon[31360]: pgmap v5607: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:23 smithi022 ceph-mon[40250]: pgmap v5607: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:25.189 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:25.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:25.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:25.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:25.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:25 smithi022 ceph-mon[31360]: pgmap v5608: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:25.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:25 smithi022 ceph-mon[40250]: pgmap v5608: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:25 smithi122 ceph-mon[32255]: pgmap v5608: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4051451734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4051451734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4051451734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:27 smithi022 ceph-mon[31360]: pgmap v5609: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:05:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:27 smithi022 ceph-mon[40250]: pgmap v5609: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:27 smithi122 ceph-mon[32255]: pgmap v5609: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:29 smithi022 ceph-mon[31360]: pgmap v5610: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:29 smithi022 ceph-mon[40250]: pgmap v5610: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:29 smithi122 ceph-mon[32255]: pgmap v5610: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:30] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:05:30.562 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:30.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:30.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:30.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1219154081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1219154081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1219154081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:31 smithi022 ceph-mon[31360]: pgmap v5611: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:31 smithi022 ceph-mon[40250]: pgmap v5611: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:31 smithi122 ceph-mon[32255]: pgmap v5611: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:05:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:05:33.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:05:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:05:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:05:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:05:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:05:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:05:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:33 smithi122 ceph-mon[32255]: pgmap v5612: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:05:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:05:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:05:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:05:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[31360]: pgmap v5612: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:05:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:05:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:05:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:05:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:33 smithi022 ceph-mon[40250]: pgmap v5612: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:35 smithi022 ceph-mon[31360]: pgmap v5613: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:35 smithi022 ceph-mon[40250]: pgmap v5613: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:35.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:35 smithi122 ceph-mon[32255]: pgmap v5613: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:36.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:36.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:36.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4261105226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4261105226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4261105226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:37 smithi022 ceph-mon[31360]: pgmap v5614: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:05:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:37 smithi022 ceph-mon[40250]: pgmap v5614: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:37 smithi122 ceph-mon[32255]: pgmap v5614: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:39 smithi022 ceph-mon[31360]: pgmap v5615: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:39 smithi022 ceph-mon[40250]: pgmap v5615: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:39 smithi122 ceph-mon[32255]: pgmap v5615: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:40] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:05:41.301 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:41.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:41.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:41.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:41.670 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:41 smithi022 ceph-mon[31360]: pgmap v5616: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:41.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:41 smithi022 ceph-mon[40250]: pgmap v5616: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:41 smithi122 ceph-mon[32255]: pgmap v5616: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/851194913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:05:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:05:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/851194913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/851194913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:43.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:05:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:05:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:43 smithi122 ceph-mon[32255]: pgmap v5617: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:43 smithi022 ceph-mon[31360]: pgmap v5617: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:43 smithi022 ceph-mon[40250]: pgmap v5617: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:45 smithi022 ceph-mon[31360]: pgmap v5618: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:45 smithi022 ceph-mon[40250]: pgmap v5618: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:45 smithi122 ceph-mon[32255]: pgmap v5618: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:46.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:47.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:47.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:47.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1488958614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1488958614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1488958614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:05:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:48 smithi122 ceph-mon[32255]: pgmap v5619: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:48 smithi022 ceph-mon[31360]: pgmap v5619: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:48 smithi022 ceph-mon[40250]: pgmap v5619: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:49 smithi022 ceph-mon[31360]: pgmap v5620: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:49 smithi022 ceph-mon[40250]: pgmap v5620: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:49 smithi122 ceph-mon[32255]: pgmap v5620: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:50] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T21:05:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:51 smithi022 ceph-mon[31360]: pgmap v5621: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:51 smithi022 ceph-mon[40250]: pgmap v5621: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:51 smithi122 ceph-mon[32255]: pgmap v5621: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:52.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:52.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:52.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:52.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1070226648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:05:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:05:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1070226648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1070226648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:53.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:05:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:05:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:53 smithi122 ceph-mon[32255]: pgmap v5622: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:53 smithi022 ceph-mon[31360]: pgmap v5622: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:05:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:53 smithi022 ceph-mon[40250]: pgmap v5622: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:55 smithi022 ceph-mon[31360]: pgmap v5623: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:55 smithi022 ceph-mon[40250]: pgmap v5623: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:55 smithi122 ceph-mon[32255]: pgmap v5623: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:57.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:05:57.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:05:57.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:05:57.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:05:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:05:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:57 smithi022 ceph-mon[31360]: pgmap v5624: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:57 smithi022 ceph-mon[40250]: pgmap v5624: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:57 smithi122 ceph-mon[32255]: pgmap v5624: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:05:58.750 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3671011886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:58.770 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3671011886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:58.771 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3671011886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:05:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:05:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:05:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:05:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:05:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[31360]: pgmap v5625: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:05:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:05:59.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:05:59.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:05:59.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:05:59 smithi022 ceph-mon[40250]: pgmap v5625: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:05:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:05:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:05:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:05:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:05:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:05:59 smithi122 ceph-mon[32255]: pgmap v5625: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:00] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T21:06:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:01 smithi022 ceph-mon[31360]: pgmap v5626: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:01 smithi022 ceph-mon[40250]: pgmap v5626: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:01 smithi122 ceph-mon[32255]: pgmap v5626: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:06:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:06:02.780 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:03.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:03.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:03.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:03.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:06:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:06:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3818589561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:03 smithi122 ceph-mon[32255]: pgmap v5627: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3818589561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:03 smithi022 ceph-mon[31360]: pgmap v5627: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3818589561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:03 smithi022 ceph-mon[40250]: pgmap v5627: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:05 smithi022 ceph-mon[31360]: pgmap v5628: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:05 smithi022 ceph-mon[40250]: pgmap v5628: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:05 smithi122 ceph-mon[32255]: pgmap v5628: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:07 smithi022 ceph-mon[31360]: pgmap v5629: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:06:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:07 smithi022 ceph-mon[40250]: pgmap v5629: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:07 smithi122 ceph-mon[32255]: pgmap v5629: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:08.151 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:08.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1358690283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1358690283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1358690283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:09 smithi122 ceph-mon[32255]: pgmap v5630: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:10.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:09 smithi022 ceph-mon[31360]: pgmap v5630: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:10.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:09 smithi022 ceph-mon[40250]: pgmap v5630: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:06:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:11 smithi022 ceph-mon[31360]: pgmap v5631: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:11 smithi022 ceph-mon[40250]: pgmap v5631: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:11 smithi122 ceph-mon[32255]: pgmap v5631: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:06:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:06:13.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:06:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:06:13.523 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:13.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:13.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:13.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:13 smithi122 ceph-mon[32255]: pgmap v5632: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:13 smithi022 ceph-mon[31360]: pgmap v5632: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:13 smithi022 ceph-mon[40250]: pgmap v5632: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3208042493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3208042493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3208042493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:15 smithi122 ceph-mon[32255]: pgmap v5633: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:15 smithi022 ceph-mon[31360]: pgmap v5633: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:15 smithi022 ceph-mon[40250]: pgmap v5633: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:17 smithi022 ceph-mon[31360]: pgmap v5634: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:06:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:17 smithi022 ceph-mon[40250]: pgmap v5634: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:17 smithi122 ceph-mon[32255]: pgmap v5634: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:18.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:19.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:19.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:19.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3371973477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3371973477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3371973477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:20 smithi122 ceph-mon[32255]: pgmap v5635: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:20] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:06:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:20 smithi022 ceph-mon[31360]: pgmap v5635: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:20 smithi022 ceph-mon[40250]: pgmap v5635: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:21.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:21 smithi022 ceph-mon[31360]: pgmap v5636: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:21 smithi022 ceph-mon[40250]: pgmap v5636: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:21 smithi122 ceph-mon[32255]: pgmap v5636: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:06:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:06:23.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:06:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:06:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:23 smithi122 ceph-mon[32255]: pgmap v5637: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:24.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:24.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:24.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:23 smithi022 ceph-mon[31360]: pgmap v5637: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:24.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:24.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:23 smithi022 ceph-mon[40250]: pgmap v5637: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:24.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:24.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:24.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1198926971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1198926971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1198926971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:25 smithi122 ceph-mon[32255]: pgmap v5638: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:25 smithi022 ceph-mon[31360]: pgmap v5638: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:25 smithi022 ceph-mon[40250]: pgmap v5638: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:27 smithi022 ceph-mon[31360]: pgmap v5639: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:06:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:27 smithi022 ceph-mon[40250]: pgmap v5639: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:27 smithi122 ceph-mon[32255]: pgmap v5639: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:29.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:29 smithi022 ceph-mon[31360]: pgmap v5640: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:29 smithi022 ceph-mon[40250]: pgmap v5640: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:29 smithi122 ceph-mon[32255]: pgmap v5640: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:30.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:30.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:30] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:06:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/247320941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/247320941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/247320941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:31 smithi022 ceph-mon[31360]: pgmap v5641: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:31 smithi022 ceph-mon[40250]: pgmap v5641: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:31 smithi122 ceph-mon[32255]: pgmap v5641: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:06:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:06:33.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:06:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:06:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:06:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:33 smithi122 ceph-mon[32255]: pgmap v5642: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:06:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:06:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:06:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:06:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[40250]: pgmap v5642: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:06:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:06:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:06:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:06:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:33 smithi022 ceph-mon[31360]: pgmap v5642: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:35.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:35.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:35.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:35.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1489421670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:35.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1489421670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:35.903 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1489421670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:37 smithi122 ceph-mon[32255]: pgmap v5643: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:37 smithi022 ceph-mon[31360]: pgmap v5643: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:06:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:37 smithi022 ceph-mon[40250]: pgmap v5643: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:39 smithi122 ceph-mon[32255]: pgmap v5644: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:39 smithi022 ceph-mon[31360]: pgmap v5644: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:39 smithi022 ceph-mon[40250]: pgmap v5644: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:40.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:40] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:06:40.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:40.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:40.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:41 smithi122 ceph-mon[32255]: pgmap v5645: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/515378129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:41 smithi022 ceph-mon[31360]: pgmap v5645: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/515378129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:41 smithi022 ceph-mon[40250]: pgmap v5645: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/515378129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:06:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:06:43.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:06:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:06:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:43 smithi122 ceph-mon[32255]: pgmap v5646: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:43 smithi022 ceph-mon[31360]: pgmap v5646: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:43 smithi022 ceph-mon[40250]: pgmap v5646: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:45 smithi122 ceph-mon[32255]: pgmap v5647: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:45.741 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:45.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:45 smithi022 ceph-mon[40250]: pgmap v5647: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:45.743 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:45 smithi022 ceph-mon[31360]: pgmap v5647: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:46.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:46.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:46.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/742209224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/742209224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/742209224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:47 smithi022 ceph-mon[31360]: pgmap v5648: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:06:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:47 smithi022 ceph-mon[40250]: pgmap v5648: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:47 smithi122 ceph-mon[32255]: pgmap v5648: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:49 smithi022 ceph-mon[31360]: pgmap v5649: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:49 smithi022 ceph-mon[40250]: pgmap v5649: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:49 smithi122 ceph-mon[32255]: pgmap v5649: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:50] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:06:51.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:51.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:51.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:51.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:51 smithi022 ceph-mon[31360]: pgmap v5650: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:51 smithi022 ceph-mon[40250]: pgmap v5650: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:51 smithi122 ceph-mon[32255]: pgmap v5650: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3167167058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:52.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:06:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:06:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3167167058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3167167058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:53.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:06:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:06:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:53 smithi022 ceph-mon[31360]: pgmap v5651: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:53 smithi022 ceph-mon[40250]: pgmap v5651: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:53 smithi122 ceph-mon[32255]: pgmap v5651: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:06:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:55 smithi022 ceph-mon[31360]: pgmap v5652: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:55 smithi022 ceph-mon[40250]: pgmap v5652: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:55 smithi122 ceph-mon[32255]: pgmap v5652: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:06:56.476 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:06:56.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:06:56.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:06:56.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:06:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:06:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:57 smithi022 ceph-mon[31360]: pgmap v5653: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3069091353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:57 smithi022 ceph-mon[40250]: pgmap v5653: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3069091353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:57 smithi122 ceph-mon[32255]: pgmap v5653: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3069091353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:06:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[31360]: pgmap v5654: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:06:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:06:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:06:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[40250]: pgmap v5654: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:06:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:06:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:06:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:06:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:59 smithi122 ceph-mon[32255]: pgmap v5654: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:06:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:06:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:06:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:06:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:07:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:00] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:07:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:00 smithi122 ceph-mon[32255]: pgmap v5655: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:07:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:00 smithi022 ceph-mon[31360]: pgmap v5655: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:07:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:00 smithi022 ceph-mon[40250]: pgmap v5655: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:07:01.846 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:02.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:02.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:02.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:02.611 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:07:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:07:02.905 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:02 smithi122 ceph-mon[32255]: pgmap v5656: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:02.905 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/398103740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:02 smithi022 ceph-mon[31360]: pgmap v5656: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/398103740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:02 smithi022 ceph-mon[40250]: pgmap v5656: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:03.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/398103740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:03.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:07:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:07:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:04 smithi122 ceph-mon[32255]: pgmap v5657: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:04 smithi022 ceph-mon[31360]: pgmap v5657: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:04 smithi022 ceph-mon[40250]: pgmap v5657: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:06 smithi122 ceph-mon[32255]: pgmap v5658: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:07.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:07.217 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:06 smithi022 ceph-mon[31360]: pgmap v5658: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:07.217 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:06 smithi022 ceph-mon[40250]: pgmap v5658: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:07:07.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:07.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:07.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4020262951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4020262951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4020262951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:08 smithi122 ceph-mon[32255]: pgmap v5659: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:08 smithi022 ceph-mon[31360]: pgmap v5659: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:08 smithi022 ceph-mon[40250]: pgmap v5659: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:10] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:07:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:10 smithi122 ceph-mon[32255]: pgmap v5660: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:10 smithi022 ceph-mon[31360]: pgmap v5660: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:10 smithi022 ceph-mon[40250]: pgmap v5660: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:12.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:07:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:07:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:12.960 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:12 smithi022 ceph-mon[31360]: pgmap v5661: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:12.960 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:12 smithi022 ceph-mon[40250]: pgmap v5661: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:12 smithi122 ceph-mon[32255]: pgmap v5661: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:13.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:07:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:07:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3610945386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3610945386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3610945386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:14 smithi122 ceph-mon[32255]: pgmap v5662: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:14 smithi022 ceph-mon[31360]: pgmap v5662: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:14 smithi022 ceph-mon[40250]: pgmap v5662: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:16 smithi122 ceph-mon[32255]: pgmap v5663: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:16 smithi022 ceph-mon[31360]: pgmap v5663: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:16 smithi022 ceph-mon[40250]: pgmap v5663: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:07:17.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:18.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:18.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:18.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:18 smithi122 ceph-mon[32255]: pgmap v5664: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2097195317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:18 smithi022 ceph-mon[31360]: pgmap v5664: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2097195317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:18 smithi022 ceph-mon[40250]: pgmap v5664: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2097195317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:20] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:07:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:20 smithi122 ceph-mon[32255]: pgmap v5665: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:20 smithi022 ceph-mon[31360]: pgmap v5665: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:20 smithi022 ceph-mon[40250]: pgmap v5665: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:07:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:07:23.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:07:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:07:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:22 smithi122 ceph-mon[32255]: pgmap v5666: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:22 smithi022 ceph-mon[31360]: pgmap v5666: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:22 smithi022 ceph-mon[40250]: pgmap v5666: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:23.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:23.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:23.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1111459931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1111459931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1111459931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:24 smithi122 ceph-mon[32255]: pgmap v5667: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:24 smithi022 ceph-mon[31360]: pgmap v5667: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:24 smithi022 ceph-mon[40250]: pgmap v5667: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:26 smithi122 ceph-mon[32255]: pgmap v5668: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:26 smithi022 ceph-mon[31360]: pgmap v5668: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:26 smithi022 ceph-mon[40250]: pgmap v5668: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:07:28.702 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:28 smithi022 ceph-mon[31360]: pgmap v5669: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:28 smithi022 ceph-mon[40250]: pgmap v5669: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:29.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:29.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:29.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:28 smithi122 ceph-mon[32255]: pgmap v5669: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/524581881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/524581881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/524581881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:30] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:07:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:30 smithi122 ceph-mon[32255]: pgmap v5670: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:30 smithi022 ceph-mon[31360]: pgmap v5670: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:30 smithi022 ceph-mon[40250]: pgmap v5670: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:07:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:07:33.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:07:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:07:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:32 smithi122 ceph-mon[32255]: pgmap v5671: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:07:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:07:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:07:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:07:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[31360]: pgmap v5671: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:07:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:07:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:07:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:07:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[40250]: pgmap v5671: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:07:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:07:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:07:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:07:34.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:34.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:34.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:34.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:34 smithi122 ceph-mon[32255]: pgmap v5672: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1166496195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:34 smithi022 ceph-mon[31360]: pgmap v5672: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1166496195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:34 smithi022 ceph-mon[40250]: pgmap v5672: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1166496195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:36 smithi122 ceph-mon[32255]: pgmap v5673: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:36 smithi022 ceph-mon[31360]: pgmap v5673: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:36 smithi022 ceph-mon[40250]: pgmap v5673: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:07:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:38 smithi122 ceph-mon[32255]: pgmap v5674: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:38 smithi022 ceph-mon[31360]: pgmap v5674: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:38 smithi022 ceph-mon[40250]: pgmap v5674: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:39.447 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:39.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:39.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:39.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1040808442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1040808442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1040808442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:40] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:07:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:40 smithi122 ceph-mon[32255]: pgmap v5675: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:40 smithi022 ceph-mon[40250]: pgmap v5675: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:40 smithi022 ceph-mon[31360]: pgmap v5675: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:07:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:07:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:42 smithi122 ceph-mon[32255]: pgmap v5676: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:43.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:07:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:07:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:42 smithi022 ceph-mon[31360]: pgmap v5676: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:42 smithi022 ceph-mon[40250]: pgmap v5676: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:44.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:44 smithi122 ceph-mon[32255]: pgmap v5677: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:45.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:45.193 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:44 smithi022 ceph-mon[31360]: pgmap v5677: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:45.193 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:44 smithi022 ceph-mon[40250]: pgmap v5677: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/143279530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/143279530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/143279530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:46 smithi122 ceph-mon[32255]: pgmap v5678: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:46 smithi022 ceph-mon[31360]: pgmap v5678: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:46 smithi022 ceph-mon[40250]: pgmap v5678: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:07:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:48 smithi122 ceph-mon[32255]: pgmap v5679: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:48 smithi022 ceph-mon[31360]: pgmap v5679: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:48 smithi022 ceph-mon[40250]: pgmap v5679: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:50.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:50] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:07:50.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:50.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:50.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:50 smithi122 ceph-mon[32255]: pgmap v5680: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1085284231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:51.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:50 smithi022 ceph-mon[31360]: pgmap v5680: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:51.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1085284231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:51.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:50 smithi022 ceph-mon[40250]: pgmap v5680: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:07:51.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1085284231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:07:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:07:53.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:07:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:07:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:52 smithi122 ceph-mon[32255]: pgmap v5681: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:52 smithi022 ceph-mon[31360]: pgmap v5681: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:52 smithi022 ceph-mon[40250]: pgmap v5681: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:07:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:07:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:54 smithi122 ceph-mon[32255]: pgmap v5682: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:54 smithi022 ceph-mon[31360]: pgmap v5682: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:54 smithi022 ceph-mon[40250]: pgmap v5682: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:55.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:07:55.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:07:55.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:07:55.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:07:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/82699196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/82699196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/82699196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:07:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:56 smithi022 ceph-mon[31360]: pgmap v5683: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:56 smithi022 ceph-mon[40250]: pgmap v5683: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:56 smithi122 ceph-mon[32255]: pgmap v5683: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:07:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:58 smithi022 ceph-mon[31360]: pgmap v5684: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:58 smithi022 ceph-mon[40250]: pgmap v5684: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:58 smithi122 ceph-mon[32255]: pgmap v5684: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:07:59.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:07:59.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:07:59.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:07:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:08:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:08:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:08:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:07:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:08:00.407 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:00] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:08:00.408 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:08:00.408 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:08:00.408 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:07:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:08:00.933 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[31360]: pgmap v5685: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:08:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[40250]: pgmap v5685: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:08:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:01.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:01.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:01 smithi122 ceph-mon[32255]: pgmap v5685: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:08:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:08:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4237123213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4237123213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4237123213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:08:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:08:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:03 smithi022 ceph-mon[31360]: pgmap v5686: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:08:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:03 smithi022 ceph-mon[40250]: pgmap v5686: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:08:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:03 smithi122 ceph-mon[32255]: pgmap v5686: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:08:03.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:08:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:08:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:05 smithi022 ceph-mon[31360]: pgmap v5687: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:08:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:05 smithi022 ceph-mon[40250]: pgmap v5687: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:08:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:05 smithi122 ceph-mon[32255]: pgmap v5687: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:08:06.301 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:06.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:06.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:06.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:07 smithi022 ceph-mon[31360]: pgmap v5688: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/330897723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:07 smithi022 ceph-mon[40250]: pgmap v5688: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/330897723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:07 smithi122 ceph-mon[32255]: pgmap v5688: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/330897723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:08:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:09 smithi022 ceph-mon[31360]: pgmap v5689: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:09 smithi022 ceph-mon[40250]: pgmap v5689: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:09 smithi122 ceph-mon[32255]: pgmap v5689: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:10] "GET /metrics HTTP/1.1" 200 202345 "" "Prometheus/2.18.1" 2021-12-15T21:08:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:11 smithi022 ceph-mon[31360]: pgmap v5690: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:08:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:11 smithi022 ceph-mon[40250]: pgmap v5690: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:08:11.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:11 smithi122 ceph-mon[32255]: pgmap v5690: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:08:11.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:12.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:12.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:12.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:08:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:08:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:13 smithi022 ceph-mon[31360]: pgmap v5691: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4099553419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:13 smithi022 ceph-mon[40250]: pgmap v5691: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4099553419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:13.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:08:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:08:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:13 smithi122 ceph-mon[32255]: pgmap v5691: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:08:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4099553419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:15 smithi122 ceph-mon[32255]: pgmap v5692: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:08:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:15 smithi022 ceph-mon[31360]: pgmap v5692: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:08:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:15 smithi022 ceph-mon[40250]: pgmap v5692: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:08:17.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:17 smithi022 ceph-mon[31360]: pgmap v5693: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:17 smithi022 ceph-mon[40250]: pgmap v5693: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:17.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:17.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:17.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:17 smithi122 ceph-mon[32255]: pgmap v5693: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:08:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2086903103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2086903103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2086903103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:19 smithi122 ceph-mon[32255]: pgmap v5694: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:19 smithi022 ceph-mon[31360]: pgmap v5694: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:19 smithi022 ceph-mon[40250]: pgmap v5694: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:20] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:08:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:21 smithi122 ceph-mon[32255]: pgmap v5695: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:21 smithi022 ceph-mon[31360]: pgmap v5695: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:21 smithi022 ceph-mon[40250]: pgmap v5695: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:22.412 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:08:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:08:22.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:22.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:22.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:23.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:08:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:08:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:23 smithi122 ceph-mon[32255]: pgmap v5696: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3216316975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:23 smithi022 ceph-mon[31360]: pgmap v5696: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3216316975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:23 smithi022 ceph-mon[40250]: pgmap v5696: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3216316975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:25 smithi122 ceph-mon[32255]: pgmap v5697: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:25 smithi022 ceph-mon[31360]: pgmap v5697: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:25 smithi022 ceph-mon[40250]: pgmap v5697: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:27 smithi022 ceph-mon[31360]: pgmap v5698: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:27 smithi022 ceph-mon[40250]: pgmap v5698: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:27 smithi122 ceph-mon[32255]: pgmap v5698: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:08:27.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:28.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:28.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:28.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:29 smithi122 ceph-mon[32255]: pgmap v5699: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/826047007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:29 smithi022 ceph-mon[31360]: pgmap v5699: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/826047007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:29 smithi022 ceph-mon[40250]: pgmap v5699: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/826047007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:30] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:08:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:31 smithi122 ceph-mon[32255]: pgmap v5700: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:31 smithi022 ceph-mon[31360]: pgmap v5700: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:31 smithi022 ceph-mon[40250]: pgmap v5700: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:08:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:08:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:33.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:08:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:08:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:33 smithi122 ceph-mon[32255]: pgmap v5701: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:08:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:08:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:08:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:08:33.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:33.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:33.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:33.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[31360]: pgmap v5701: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:08:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:08:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:08:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:08:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[40250]: pgmap v5701: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:08:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:08:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:08:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:08:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1030720968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1030720968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1030720968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:35 smithi122 ceph-mon[32255]: pgmap v5702: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:35 smithi022 ceph-mon[31360]: pgmap v5702: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:35 smithi022 ceph-mon[40250]: pgmap v5702: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:37 smithi022 ceph-mon[31360]: pgmap v5703: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:37 smithi022 ceph-mon[40250]: pgmap v5703: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:37 smithi122 ceph-mon[32255]: pgmap v5703: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:08:38.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:38.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:38.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:38.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:39 smithi022 ceph-mon[31360]: pgmap v5704: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2885834729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:39 smithi022 ceph-mon[40250]: pgmap v5704: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2885834729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:39 smithi122 ceph-mon[32255]: pgmap v5704: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2885834729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:40] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:08:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:41 smithi122 ceph-mon[32255]: pgmap v5705: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:41 smithi022 ceph-mon[31360]: pgmap v5705: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:41 smithi022 ceph-mon[40250]: pgmap v5705: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:08:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:08:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:43 smithi122 ceph-mon[32255]: pgmap v5706: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:43.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:08:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:08:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:43 smithi022 ceph-mon[31360]: pgmap v5706: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:43 smithi022 ceph-mon[40250]: pgmap v5706: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:43.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:44.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:44.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:44.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:44.270 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:44.271 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:45 smithi122 ceph-mon[32255]: pgmap v5707: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3838922219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:45 smithi022 ceph-mon[31360]: pgmap v5707: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3838922219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:45 smithi022 ceph-mon[40250]: pgmap v5707: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3838922219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:47 smithi022 ceph-mon[31360]: pgmap v5708: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:47 smithi022 ceph-mon[40250]: pgmap v5708: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:47 smithi122 ceph-mon[32255]: pgmap v5708: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:08:49.271 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:49 smithi122 ceph-mon[32255]: pgmap v5709: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:49 smithi022 ceph-mon[31360]: pgmap v5709: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:49 smithi022 ceph-mon[40250]: pgmap v5709: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:49.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:49.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:49.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1585020136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:50] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:08:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1585020136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1585020136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:51 smithi122 ceph-mon[32255]: pgmap v5710: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:51 smithi022 ceph-mon[31360]: pgmap v5710: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:51 smithi022 ceph-mon[40250]: pgmap v5710: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:08:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:08:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:53 smithi122 ceph-mon[32255]: pgmap v5711: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:53.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:08:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:08:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:53 smithi022 ceph-mon[31360]: pgmap v5711: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:53.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:53 smithi022 ceph-mon[40250]: pgmap v5711: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:08:54.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:08:55.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:08:55.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:08:55.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:08:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:55 smithi022 ceph-mon[31360]: pgmap v5712: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3613507736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:55 smithi022 ceph-mon[40250]: pgmap v5712: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3613507736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:55 smithi122 ceph-mon[32255]: pgmap v5712: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:08:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3613507736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:08:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:57 smithi022 ceph-mon[31360]: pgmap v5713: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:57 smithi022 ceph-mon[40250]: pgmap v5713: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:57 smithi122 ceph-mon[32255]: pgmap v5713: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:08:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:08:59 smithi122 ceph-mon[32255]: pgmap v5714: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:08:59 smithi022 ceph-mon[31360]: pgmap v5714: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:08:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:08:59 smithi022 ceph-mon[40250]: pgmap v5714: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:00.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:00.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:00.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:00.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:00] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:09:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:01 smithi122 ceph-mon[32255]: pgmap v5715: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4108981508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:01.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:09:01.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:09:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:09:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:09:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[31360]: pgmap v5715: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4108981508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:09:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:09:01.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:09:01.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:09:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[40250]: pgmap v5715: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4108981508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:01.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:09:01.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:09:01.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:09:01.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:09:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:09:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:09:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:03 smithi122 ceph-mon[32255]: pgmap v5716: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:03.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:09:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:09:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:03 smithi022 ceph-mon[31360]: pgmap v5716: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:03 smithi022 ceph-mon[40250]: pgmap v5716: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:05.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:05 smithi122 ceph-mon[32255]: pgmap v5717: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:05 smithi022 ceph-mon[31360]: pgmap v5717: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:05 smithi022 ceph-mon[40250]: pgmap v5717: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:05.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:05.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:05.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2602433338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2602433338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2602433338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:07 smithi022 ceph-mon[31360]: pgmap v5718: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:07 smithi022 ceph-mon[40250]: pgmap v5718: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:07 smithi122 ceph-mon[32255]: pgmap v5718: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:09:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:09 smithi122 ceph-mon[32255]: pgmap v5719: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:09 smithi022 ceph-mon[31360]: pgmap v5719: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:09 smithi022 ceph-mon[40250]: pgmap v5719: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:10] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:09:10.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:11 smithi122 ceph-mon[32255]: pgmap v5720: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/311293481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:11 smithi022 ceph-mon[31360]: pgmap v5720: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/311293481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:11 smithi022 ceph-mon[40250]: pgmap v5720: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/311293481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:09:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:09:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:13 smithi122 ceph-mon[32255]: pgmap v5721: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:13.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:09:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:09:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:13 smithi022 ceph-mon[31360]: pgmap v5721: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:13 smithi022 ceph-mon[40250]: pgmap v5721: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:15 smithi122 ceph-mon[32255]: pgmap v5722: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:15 smithi022 ceph-mon[31360]: pgmap v5722: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:15 smithi022 ceph-mon[40250]: pgmap v5722: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:16.123 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:16.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:16.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:16.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:17 smithi122 ceph-mon[32255]: pgmap v5723: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/62108439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:09:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:17 smithi022 ceph-mon[31360]: pgmap v5723: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/62108439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:17 smithi022 ceph-mon[40250]: pgmap v5723: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/62108439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:19 smithi122 ceph-mon[32255]: pgmap v5724: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:19 smithi022 ceph-mon[31360]: pgmap v5724: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:19 smithi022 ceph-mon[40250]: pgmap v5724: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:20] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:09:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:21 smithi122 ceph-mon[32255]: pgmap v5725: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:21.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:21.497 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:21 smithi022 ceph-mon[31360]: pgmap v5725: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:21.498 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:21 smithi022 ceph-mon[40250]: pgmap v5725: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:21.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:21.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:21.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1876447404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:09:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:09:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1876447404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1876447404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:23 smithi122 ceph-mon[32255]: pgmap v5726: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:23.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:09:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:09:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:23 smithi022 ceph-mon[31360]: pgmap v5726: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:23 smithi022 ceph-mon[40250]: pgmap v5726: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:25 smithi122 ceph-mon[32255]: pgmap v5727: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:25 smithi022 ceph-mon[31360]: pgmap v5727: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:25 smithi022 ceph-mon[40250]: pgmap v5727: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:26.865 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:27.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:27 smithi122 ceph-mon[32255]: pgmap v5728: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:27 smithi022 ceph-mon[31360]: pgmap v5728: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:09:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:27 smithi022 ceph-mon[40250]: pgmap v5728: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3705570703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3705570703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3705570703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:29 smithi122 ceph-mon[32255]: pgmap v5729: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:29 smithi022 ceph-mon[31360]: pgmap v5729: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:29 smithi022 ceph-mon[40250]: pgmap v5729: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:30] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:09:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:31 smithi022 ceph-mon[31360]: pgmap v5730: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:31 smithi022 ceph-mon[40250]: pgmap v5730: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:31 smithi122 ceph-mon[32255]: pgmap v5730: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:32.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:32.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:32.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:32.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:09:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:09:33.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:09:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:09:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:33 smithi122 ceph-mon[32255]: pgmap v5731: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/840237389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:09:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:09:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:09:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:09:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[31360]: pgmap v5731: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/840237389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:09:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:09:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:09:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:09:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[40250]: pgmap v5731: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/840237389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:09:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:09:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:09:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:09:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:35 smithi022 ceph-mon[31360]: pgmap v5732: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:35 smithi022 ceph-mon[40250]: pgmap v5732: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:35 smithi122 ceph-mon[32255]: pgmap v5732: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:37 smithi022 ceph-mon[31360]: pgmap v5733: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:09:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:37 smithi022 ceph-mon[40250]: pgmap v5733: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:37.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:37 smithi122 ceph-mon[32255]: pgmap v5733: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:37.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:37.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:37.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:38 smithi122 ceph-mon[32255]: pgmap v5734: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:39.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3118179009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:38 smithi022 ceph-mon[31360]: pgmap v5734: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3118179009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:38 smithi022 ceph-mon[40250]: pgmap v5734: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3118179009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:40] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:09:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:40 smithi122 ceph-mon[32255]: pgmap v5735: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:40 smithi022 ceph-mon[31360]: pgmap v5735: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:40 smithi022 ceph-mon[40250]: pgmap v5735: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:09:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:09:42.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:43.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:09:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:09:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:42 smithi122 ceph-mon[32255]: pgmap v5736: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:42 smithi022 ceph-mon[31360]: pgmap v5736: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:42 smithi022 ceph-mon[40250]: pgmap v5736: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3028709613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3028709613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3028709613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:44 smithi122 ceph-mon[32255]: pgmap v5737: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:44 smithi022 ceph-mon[31360]: pgmap v5737: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:44 smithi022 ceph-mon[40250]: pgmap v5737: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:46 smithi122 ceph-mon[32255]: pgmap v5738: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:46 smithi022 ceph-mon[31360]: pgmap v5738: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:46 smithi022 ceph-mon[40250]: pgmap v5738: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:09:48.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:48.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:48.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:48.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:48 smithi022 ceph-mon[31360]: pgmap v5739: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1126509500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:48 smithi022 ceph-mon[40250]: pgmap v5739: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1126509500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:48 smithi122 ceph-mon[32255]: pgmap v5739: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1126509500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:50] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:09:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:50 smithi122 ceph-mon[32255]: pgmap v5740: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:50 smithi022 ceph-mon[31360]: pgmap v5740: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:50 smithi022 ceph-mon[40250]: pgmap v5740: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:09:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:09:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:52 smithi122 ceph-mon[32255]: pgmap v5741: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:53.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:09:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:09:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:52 smithi022 ceph-mon[31360]: pgmap v5741: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:52 smithi022 ceph-mon[40250]: pgmap v5741: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:53.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:54.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:54.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:54.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:09:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:09:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:54 smithi122 ceph-mon[32255]: pgmap v5742: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:55.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2635727948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:54 smithi022 ceph-mon[31360]: pgmap v5742: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2635727948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:54 smithi022 ceph-mon[40250]: pgmap v5742: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2635727948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:09:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:56 smithi122 ceph-mon[32255]: pgmap v5743: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:56 smithi022 ceph-mon[31360]: pgmap v5743: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:56 smithi022 ceph-mon[40250]: pgmap v5743: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:09:59.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:09:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:58 smithi122 ceph-mon[32255]: pgmap v5744: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:58 smithi022 ceph-mon[40250]: pgmap v5744: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:58 smithi022 ceph-mon[31360]: pgmap v5744: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:09:59.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:09:59.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:09:59.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:09:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/274155391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:09:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/274155391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:09:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/274155391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:00] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:10:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:00 smithi122 ceph-mon[32255]: pgmap v5745: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T21:10:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:00 smithi022 ceph-mon[31360]: pgmap v5745: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T21:10:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:00 smithi022 ceph-mon[40250]: pgmap v5745: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T21:10:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:10:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:10:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:10:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:10:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:10:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:10:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:10:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:10:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:10:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:10:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:10:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:10:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:10:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:10:03.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:10:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:10:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:03 smithi122 ceph-mon[32255]: pgmap v5746: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:03 smithi022 ceph-mon[31360]: pgmap v5746: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:03 smithi022 ceph-mon[40250]: pgmap v5746: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:04.472 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:04.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:04.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:04.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:05 smithi122 ceph-mon[32255]: pgmap v5747: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2263327648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:05 smithi022 ceph-mon[31360]: pgmap v5747: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2263327648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:05 smithi022 ceph-mon[40250]: pgmap v5747: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2263327648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:07 smithi122 ceph-mon[32255]: pgmap v5748: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:07 smithi022 ceph-mon[31360]: pgmap v5748: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:10:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:07 smithi022 ceph-mon[40250]: pgmap v5748: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:09 smithi122 ceph-mon[32255]: pgmap v5749: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:09 smithi022 ceph-mon[31360]: pgmap v5749: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:09 smithi022 ceph-mon[40250]: pgmap v5749: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:09.842 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:10.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:10.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3544313032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:10:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3544313032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3544313032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:11 smithi122 ceph-mon[32255]: pgmap v5750: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:11 smithi022 ceph-mon[31360]: pgmap v5750: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:11 smithi022 ceph-mon[40250]: pgmap v5750: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:10:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:10:13.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:10:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:10:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:13 smithi122 ceph-mon[32255]: pgmap v5751: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:13 smithi022 ceph-mon[31360]: pgmap v5751: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:13 smithi022 ceph-mon[40250]: pgmap v5751: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:15.213 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:15 smithi022 ceph-mon[31360]: pgmap v5752: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:15 smithi022 ceph-mon[40250]: pgmap v5752: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:15.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:15.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:15.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:15 smithi122 ceph-mon[32255]: pgmap v5752: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/786093818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/786093818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/786093818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:17 smithi122 ceph-mon[32255]: pgmap v5753: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:17 smithi022 ceph-mon[31360]: pgmap v5753: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:10:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:17 smithi022 ceph-mon[40250]: pgmap v5753: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:19 smithi122 ceph-mon[32255]: pgmap v5754: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:19 smithi022 ceph-mon[31360]: pgmap v5754: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:19 smithi022 ceph-mon[40250]: pgmap v5754: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:20] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:10:20.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:20.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:20.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:20.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:21 smithi122 ceph-mon[32255]: pgmap v5755: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2998207421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:21 smithi022 ceph-mon[31360]: pgmap v5755: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2998207421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:21 smithi022 ceph-mon[40250]: pgmap v5755: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2998207421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:10:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:10:23.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:10:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:10:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:23 smithi122 ceph-mon[32255]: pgmap v5756: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:23 smithi022 ceph-mon[31360]: pgmap v5756: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:23 smithi022 ceph-mon[40250]: pgmap v5756: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:25 smithi122 ceph-mon[32255]: pgmap v5757: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:25 smithi022 ceph-mon[31360]: pgmap v5757: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:25 smithi022 ceph-mon[40250]: pgmap v5757: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:25.955 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:26.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:26.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:26.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1416651866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1416651866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1416651866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:27 smithi122 ceph-mon[32255]: pgmap v5758: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:27 smithi022 ceph-mon[31360]: pgmap v5758: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:10:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:27 smithi022 ceph-mon[40250]: pgmap v5758: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:29 smithi022 ceph-mon[31360]: pgmap v5759: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:29 smithi022 ceph-mon[40250]: pgmap v5759: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:29 smithi122 ceph-mon[32255]: pgmap v5759: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:30] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:10:31.326 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:31.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:31.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:31.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:31.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:31 smithi022 ceph-mon[31360]: pgmap v5760: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:31.695 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:31 smithi022 ceph-mon[40250]: pgmap v5760: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:31 smithi122 ceph-mon[32255]: pgmap v5760: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1754318267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:10:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:10:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1754318267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1754318267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:33.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:10:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:10:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[31360]: pgmap v5761: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:10:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:10:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:10:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:10:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[40250]: pgmap v5761: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:10:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:10:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:10:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:10:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:33 smithi122 ceph-mon[32255]: pgmap v5761: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:10:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:10:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:10:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:10:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:35 smithi022 ceph-mon[31360]: pgmap v5762: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:35 smithi022 ceph-mon[40250]: pgmap v5762: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:35 smithi122 ceph-mon[32255]: pgmap v5762: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:37.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:37.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:37.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:10:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:37 smithi022 ceph-mon[31360]: pgmap v5763: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4026210972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:37 smithi022 ceph-mon[40250]: pgmap v5763: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4026210972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:37 smithi122 ceph-mon[32255]: pgmap v5763: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4026210972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:39 smithi022 ceph-mon[31360]: pgmap v5764: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:39 smithi022 ceph-mon[40250]: pgmap v5764: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:39 smithi122 ceph-mon[32255]: pgmap v5764: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:40] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:10:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:41 smithi022 ceph-mon[31360]: pgmap v5765: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:41 smithi022 ceph-mon[40250]: pgmap v5765: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:41 smithi122 ceph-mon[32255]: pgmap v5765: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:42.065 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:42.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:42.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:42.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2163697226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:10:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:10:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2163697226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2163697226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:43.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:10:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:10:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:43 smithi022 ceph-mon[31360]: pgmap v5766: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:43 smithi022 ceph-mon[40250]: pgmap v5766: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:43 smithi122 ceph-mon[32255]: pgmap v5766: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:45 smithi022 ceph-mon[31360]: pgmap v5767: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:45 smithi022 ceph-mon[40250]: pgmap v5767: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:45 smithi122 ceph-mon[32255]: pgmap v5767: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:47.435 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:47 smithi022 ceph-mon[31360]: pgmap v5768: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:10:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:47 smithi022 ceph-mon[40250]: pgmap v5768: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:47.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:47.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:47.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:47 smithi122 ceph-mon[32255]: pgmap v5768: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1879755579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1879755579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1879755579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:49 smithi022 ceph-mon[31360]: pgmap v5769: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:49 smithi022 ceph-mon[40250]: pgmap v5769: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:49 smithi122 ceph-mon[32255]: pgmap v5769: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:50] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:10:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:51 smithi022 ceph-mon[31360]: pgmap v5770: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:51 smithi022 ceph-mon[40250]: pgmap v5770: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:51 smithi122 ceph-mon[32255]: pgmap v5770: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:10:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:10:52.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:53.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:53.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:53.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:53.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:10:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:10:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:53 smithi022 ceph-mon[31360]: pgmap v5771: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/927375078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:53 smithi022 ceph-mon[40250]: pgmap v5771: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/927375078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:53 smithi122 ceph-mon[32255]: pgmap v5771: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/927375078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:10:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:55 smithi022 ceph-mon[31360]: pgmap v5772: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:55 smithi022 ceph-mon[40250]: pgmap v5772: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:55 smithi122 ceph-mon[32255]: pgmap v5772: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:10:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:57 smithi022 ceph-mon[31360]: pgmap v5773: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:10:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:57 smithi022 ceph-mon[40250]: pgmap v5773: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:57 smithi122 ceph-mon[32255]: pgmap v5773: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:58.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:10:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:10:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:10:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:10:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:59 smithi022 ceph-mon[31360]: pgmap v5774: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:59.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:10:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1433871025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:59.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:59 smithi022 ceph-mon[40250]: pgmap v5774: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:59.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:10:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1433871025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:10:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:59 smithi122 ceph-mon[32255]: pgmap v5774: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:10:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:10:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1433871025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:00] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:11:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:01 smithi122 ceph-mon[32255]: pgmap v5775: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:01.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:11:01.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:11:01.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:11:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[40250]: pgmap v5775: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:11:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:11:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:11:01.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[31360]: pgmap v5775: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:01.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:11:01.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:11:01.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:11:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:11:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:11:03.114 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:02 smithi122 ceph-mon[32255]: pgmap v5776: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:03.115 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:11:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:02 smithi022 ceph-mon[31360]: pgmap v5776: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:11:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:02 smithi022 ceph-mon[40250]: pgmap v5776: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:11:03.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:11:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:11:03.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:03.874 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:03.874 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:03.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:03.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:03.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:04 smithi122 ceph-mon[32255]: pgmap v5777: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1676046027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:04 smithi022 ceph-mon[31360]: pgmap v5777: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1676046027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:04 smithi022 ceph-mon[40250]: pgmap v5777: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1676046027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:06 smithi122 ceph-mon[32255]: pgmap v5778: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:06 smithi022 ceph-mon[31360]: pgmap v5778: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:06 smithi022 ceph-mon[40250]: pgmap v5778: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:11:08.914 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:08 smithi122 ceph-mon[32255]: pgmap v5779: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:08 smithi022 ceph-mon[31360]: pgmap v5779: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:08 smithi022 ceph-mon[40250]: pgmap v5779: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:09.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:09.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:09.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/564090096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/564090096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/564090096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:10] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:11:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:10 smithi122 ceph-mon[32255]: pgmap v5780: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:10 smithi022 ceph-mon[31360]: pgmap v5780: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:10 smithi022 ceph-mon[40250]: pgmap v5780: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:11:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:11:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:12 smithi122 ceph-mon[32255]: pgmap v5781: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:13.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:11:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:11:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:12 smithi022 ceph-mon[31360]: pgmap v5781: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:12 smithi022 ceph-mon[40250]: pgmap v5781: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:14.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:14.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:14.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:14 smithi022 ceph-mon[31360]: pgmap v5782: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/632723642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:14 smithi022 ceph-mon[40250]: pgmap v5782: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/632723642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:14 smithi122 ceph-mon[32255]: pgmap v5782: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/632723642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:16 smithi122 ceph-mon[32255]: pgmap v5783: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:16 smithi022 ceph-mon[31360]: pgmap v5783: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:16 smithi022 ceph-mon[40250]: pgmap v5783: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:11:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:18 smithi122 ceph-mon[32255]: pgmap v5784: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:18 smithi022 ceph-mon[31360]: pgmap v5784: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:18 smithi022 ceph-mon[40250]: pgmap v5784: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:19.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:20.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:20.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:20.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:20] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:11:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:20 smithi122 ceph-mon[32255]: pgmap v5785: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2627436372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:20 smithi022 ceph-mon[31360]: pgmap v5785: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2627436372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:20 smithi022 ceph-mon[40250]: pgmap v5785: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2627436372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:11:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:11:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:22 smithi122 ceph-mon[32255]: pgmap v5786: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:23.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:11:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:11:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:22 smithi022 ceph-mon[31360]: pgmap v5786: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:22 smithi022 ceph-mon[40250]: pgmap v5786: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:25.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:24 smithi122 ceph-mon[32255]: pgmap v5787: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:24 smithi022 ceph-mon[31360]: pgmap v5787: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:24 smithi022 ceph-mon[40250]: pgmap v5787: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:25.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:25.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:25.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4269230935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4269230935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4269230935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:26 smithi122 ceph-mon[32255]: pgmap v5788: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:26 smithi022 ceph-mon[31360]: pgmap v5788: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:26 smithi022 ceph-mon[40250]: pgmap v5788: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:11:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:28 smithi122 ceph-mon[32255]: pgmap v5789: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:28 smithi022 ceph-mon[31360]: pgmap v5789: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:28 smithi022 ceph-mon[40250]: pgmap v5789: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:30.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:30] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:11:30.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:30.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:30.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:30 smithi022 ceph-mon[31360]: pgmap v5790: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/285192526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:30 smithi022 ceph-mon[40250]: pgmap v5790: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/285192526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:30 smithi122 ceph-mon[32255]: pgmap v5790: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/285192526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:11:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:11:33.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:11:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:11:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:32 smithi122 ceph-mon[32255]: pgmap v5791: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:11:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:11:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:11:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:11:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[31360]: pgmap v5791: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:11:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:11:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:11:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:11:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[40250]: pgmap v5791: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:11:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:11:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:11:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:11:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:34 smithi122 ceph-mon[32255]: pgmap v5792: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:34 smithi022 ceph-mon[31360]: pgmap v5792: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:34 smithi022 ceph-mon[40250]: pgmap v5792: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:35.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:36.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:36.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:36.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:36 smithi122 ceph-mon[32255]: pgmap v5793: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1719351565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:36 smithi022 ceph-mon[31360]: pgmap v5793: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1719351565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:36 smithi022 ceph-mon[40250]: pgmap v5793: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1719351565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:11:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:38 smithi022 ceph-mon[31360]: pgmap v5794: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:38 smithi022 ceph-mon[40250]: pgmap v5794: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:38 smithi122 ceph-mon[32255]: pgmap v5794: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:40] "GET /metrics HTTP/1.1" 200 202343 "" "Prometheus/2.18.1" 2021-12-15T21:11:41.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:40 smithi022 ceph-mon[31360]: pgmap v5795: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:40 smithi022 ceph-mon[40250]: pgmap v5795: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:40 smithi122 ceph-mon[32255]: pgmap v5795: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1958933311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1958933311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1958933311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:11:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:11:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:42 smithi022 ceph-mon[31360]: pgmap v5796: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:42 smithi022 ceph-mon[40250]: pgmap v5796: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:42 smithi122 ceph-mon[32255]: pgmap v5796: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:43.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:11:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:11:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:44 smithi022 ceph-mon[31360]: pgmap v5797: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:44 smithi022 ceph-mon[40250]: pgmap v5797: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:44 smithi122 ceph-mon[32255]: pgmap v5797: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:46.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:46.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:46.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:46.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:46 smithi022 ceph-mon[31360]: pgmap v5798: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3046297625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:46 smithi022 ceph-mon[40250]: pgmap v5798: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3046297625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:46 smithi122 ceph-mon[32255]: pgmap v5798: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3046297625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:11:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:48 smithi022 ceph-mon[31360]: pgmap v5799: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:48 smithi022 ceph-mon[40250]: pgmap v5799: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:48 smithi122 ceph-mon[32255]: pgmap v5799: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:50] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:11:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:50 smithi022 ceph-mon[31360]: pgmap v5800: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:50 smithi022 ceph-mon[40250]: pgmap v5800: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:50 smithi122 ceph-mon[32255]: pgmap v5800: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:51.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:52.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:52.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:52.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:11:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:11:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:52 smithi022 ceph-mon[31360]: pgmap v5801: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/885010021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:52 smithi022 ceph-mon[40250]: pgmap v5801: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/885010021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:53.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:11:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:11:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:52 smithi122 ceph-mon[32255]: pgmap v5801: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/885010021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:11:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:54 smithi022 ceph-mon[31360]: pgmap v5802: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:54 smithi022 ceph-mon[40250]: pgmap v5802: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:54 smithi122 ceph-mon[32255]: pgmap v5802: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:11:57.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:11:57.249 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:56 smithi022 ceph-mon[31360]: pgmap v5803: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:57.250 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:56 smithi022 ceph-mon[40250]: pgmap v5803: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:56 smithi122 ceph-mon[32255]: pgmap v5803: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:11:57.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:11:57.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:11:57.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:11:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/873735054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/873735054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/873735054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:11:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:11:58 smithi022 ceph-mon[31360]: pgmap v5804: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:11:58 smithi022 ceph-mon[40250]: pgmap v5804: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:11:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:11:58 smithi122 ceph-mon[32255]: pgmap v5804: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:00] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:12:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:00 smithi022 ceph-mon[31360]: pgmap v5805: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:00 smithi022 ceph-mon[40250]: pgmap v5805: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:00 smithi122 ceph-mon[32255]: pgmap v5805: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:12:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:12:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:12:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:12:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:12:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:12:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:12:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:12:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:12:02.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:12:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:12:02.991 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:02 smithi022 ceph-mon[31360]: pgmap v5806: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:02.992 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1493163268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:02.992 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:02 smithi022 ceph-mon[40250]: pgmap v5806: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:02.992 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1493163268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:03.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:03.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:03.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:03.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:12:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:12:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:02 smithi122 ceph-mon[32255]: pgmap v5806: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1493163268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:04 smithi022 ceph-mon[31360]: pgmap v5807: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:04 smithi022 ceph-mon[40250]: pgmap v5807: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:04 smithi122 ceph-mon[32255]: pgmap v5807: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:12:06.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:06.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:07 smithi122 ceph-mon[32255]: pgmap v5808: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:07 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3267M 2021-12-15T21:12:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:12:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:07 smithi022 ceph-mon[31360]: pgmap v5808: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:07 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3267M 2021-12-15T21:12:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:07 smithi022 ceph-mon[40250]: pgmap v5808: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:07 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3267M 2021-12-15T21:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:12:08.002 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:08.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:08.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:08.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:08 smithi122 ceph-mon[32255]: pgmap v5809: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3472469323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:08 smithi022 ceph-mon[31360]: pgmap v5809: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3472469323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:08 smithi022 ceph-mon[40250]: pgmap v5809: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3472469323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:10] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:12:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:10 smithi122 ceph-mon[32255]: pgmap v5810: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:10 smithi022 ceph-mon[31360]: pgmap v5810: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:10 smithi022 ceph-mon[40250]: pgmap v5810: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:12:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:12:13.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:12 smithi122 ceph-mon[32255]: pgmap v5811: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:12 smithi022 ceph-mon[31360]: pgmap v5811: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:12 smithi022 ceph-mon[40250]: pgmap v5811: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:13.374 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:13.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:12:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:12:13.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:13.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:13.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3536730748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3536730748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3536730748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:14 smithi122 ceph-mon[32255]: pgmap v5812: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:14 smithi022 ceph-mon[31360]: pgmap v5812: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:14 smithi022 ceph-mon[40250]: pgmap v5812: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:16 smithi122 ceph-mon[32255]: pgmap v5813: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:16 smithi022 ceph-mon[31360]: pgmap v5813: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:16 smithi022 ceph-mon[40250]: pgmap v5813: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:12:18.745 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:18 smithi022 ceph-mon[31360]: pgmap v5814: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:18 smithi022 ceph-mon[40250]: pgmap v5814: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:19.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:19.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:19.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:18 smithi122 ceph-mon[32255]: pgmap v5814: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2019294533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2019294533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2019294533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:20] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:12:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:20 smithi122 ceph-mon[32255]: pgmap v5815: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:20 smithi022 ceph-mon[31360]: pgmap v5815: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:20 smithi022 ceph-mon[40250]: pgmap v5815: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:12:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:12:23.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:22 smithi122 ceph-mon[32255]: pgmap v5816: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:22 smithi022 ceph-mon[31360]: pgmap v5816: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:22 smithi022 ceph-mon[40250]: pgmap v5816: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:23.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:12:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:12:24.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:24.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:24.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:24.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:24 smithi122 ceph-mon[32255]: pgmap v5817: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3550094365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:24 smithi022 ceph-mon[31360]: pgmap v5817: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3550094365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:24 smithi022 ceph-mon[40250]: pgmap v5817: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3550094365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:26 smithi122 ceph-mon[32255]: pgmap v5818: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:26 smithi022 ceph-mon[31360]: pgmap v5818: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:26 smithi022 ceph-mon[40250]: pgmap v5818: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:12:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:28 smithi122 ceph-mon[32255]: pgmap v5819: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:28 smithi022 ceph-mon[31360]: pgmap v5819: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:28 smithi022 ceph-mon[40250]: pgmap v5819: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:29.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:29.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:29.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:29.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3680326644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3680326644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3680326644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:30] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:12:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:30 smithi122 ceph-mon[32255]: pgmap v5820: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:30 smithi022 ceph-mon[31360]: pgmap v5820: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:30 smithi022 ceph-mon[40250]: pgmap v5820: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:12:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:12:33.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:32 smithi122 ceph-mon[32255]: pgmap v5821: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:33.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:12:33.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:12:33.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:12:33.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:12:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[31360]: pgmap v5821: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:12:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:12:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:12:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:12:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[40250]: pgmap v5821: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:12:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:12:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:12:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:12:33.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:12:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:12:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:34.858 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:34 smithi122 ceph-mon[32255]: pgmap v5822: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:35.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:35.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:35.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:35.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:34 smithi022 ceph-mon[31360]: pgmap v5822: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:35.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:34 smithi022 ceph-mon[40250]: pgmap v5822: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3233034144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3233034144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3233034144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:36 smithi022 ceph-mon[31360]: pgmap v5823: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:36 smithi022 ceph-mon[40250]: pgmap v5823: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:36 smithi122 ceph-mon[32255]: pgmap v5823: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:12:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:38 smithi022 ceph-mon[31360]: pgmap v5824: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:38 smithi022 ceph-mon[40250]: pgmap v5824: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:38 smithi122 ceph-mon[32255]: pgmap v5824: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:40.229 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:40] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:12:40.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:40.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:40.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:40 smithi022 ceph-mon[31360]: pgmap v5825: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/930420457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:40 smithi022 ceph-mon[40250]: pgmap v5825: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:41.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/930420457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:40 smithi122 ceph-mon[32255]: pgmap v5825: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:41.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/930420457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:12:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:12:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:42 smithi022 ceph-mon[40250]: pgmap v5826: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:42 smithi022 ceph-mon[31360]: pgmap v5826: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:42 smithi122 ceph-mon[32255]: pgmap v5826: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:43.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:12:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:12:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:44 smithi022 ceph-mon[31360]: pgmap v5827: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:44 smithi022 ceph-mon[40250]: pgmap v5827: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:44 smithi122 ceph-mon[32255]: pgmap v5827: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:45.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:45.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:45.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:45.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1174787180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1174787180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1174787180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:47 smithi022 ceph-mon[31360]: pgmap v5828: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:47.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:47 smithi022 ceph-mon[40250]: pgmap v5828: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:47 smithi122 ceph-mon[32255]: pgmap v5828: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:12:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:49 smithi022 ceph-mon[31360]: pgmap v5829: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:49 smithi022 ceph-mon[40250]: pgmap v5829: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:49 smithi122 ceph-mon[32255]: pgmap v5829: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:50] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:12:50.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:51 smithi022 ceph-mon[31360]: pgmap v5830: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:51 smithi022 ceph-mon[40250]: pgmap v5830: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:51.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:51.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:51.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:51 smithi122 ceph-mon[32255]: pgmap v5830: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1420606566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1420606566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1420606566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:12:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:12:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:53 smithi022 ceph-mon[31360]: pgmap v5831: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:53 smithi022 ceph-mon[40250]: pgmap v5831: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:53 smithi122 ceph-mon[32255]: pgmap v5831: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:53.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:12:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:12:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:12:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:55 smithi022 ceph-mon[31360]: pgmap v5832: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:55 smithi022 ceph-mon[40250]: pgmap v5832: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:55 smithi122 ceph-mon[32255]: pgmap v5832: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:12:56.342 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:12:56.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:12:56.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:12:56.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:12:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:57 smithi022 ceph-mon[31360]: pgmap v5833: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3954279847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:57 smithi022 ceph-mon[40250]: pgmap v5833: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3954279847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:57 smithi122 ceph-mon[32255]: pgmap v5833: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:57.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3954279847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:12:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:12:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:12:59 smithi122 ceph-mon[32255]: pgmap v5834: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:12:59 smithi022 ceph-mon[31360]: pgmap v5834: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:12:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:12:59 smithi022 ceph-mon[40250]: pgmap v5834: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:00] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:13:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:01 smithi122 ceph-mon[32255]: pgmap v5835: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:01 smithi022 ceph-mon[31360]: pgmap v5835: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:01 smithi022 ceph-mon[40250]: pgmap v5835: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:01.710 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:02.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:02.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:02.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:13:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:13:03.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:13:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:13:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:03 smithi122 ceph-mon[32255]: pgmap v5836: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1880712541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:03 smithi022 ceph-mon[31360]: pgmap v5836: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1880712541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:03 smithi022 ceph-mon[40250]: pgmap v5836: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1880712541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:05 smithi122 ceph-mon[32255]: pgmap v5837: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:05 smithi022 ceph-mon[31360]: pgmap v5837: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:05 smithi022 ceph-mon[40250]: pgmap v5837: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:07.081 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:07 smithi122 ceph-mon[32255]: pgmap v5838: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:13:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:13:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:13:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[31360]: pgmap v5838: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:13:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:13:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:13:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[40250]: pgmap v5838: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:13:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:13:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:13:07.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:13:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:13:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/506388219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:13:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/506388219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:13:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/506388219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:09 smithi122 ceph-mon[32255]: pgmap v5839: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:09 smithi022 ceph-mon[40250]: pgmap v5839: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:09 smithi022 ceph-mon[31360]: pgmap v5839: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:10] "GET /metrics HTTP/1.1" 200 202342 "" "Prometheus/2.18.1" 2021-12-15T21:13:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:11 smithi122 ceph-mon[32255]: pgmap v5840: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:11 smithi022 ceph-mon[31360]: pgmap v5840: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:11 smithi022 ceph-mon[40250]: pgmap v5840: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:12.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:13:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:13:12.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:12.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:12.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:13.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:13:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:13:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:13 smithi122 ceph-mon[32255]: pgmap v5841: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3782612777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:13 smithi022 ceph-mon[40250]: pgmap v5841: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3782612777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:13 smithi022 ceph-mon[31360]: pgmap v5841: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3782612777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:15 smithi122 ceph-mon[32255]: pgmap v5842: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:15 smithi022 ceph-mon[40250]: pgmap v5842: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:15 smithi022 ceph-mon[31360]: pgmap v5842: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:17 smithi122 ceph-mon[32255]: pgmap v5843: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:17 smithi022 ceph-mon[31360]: pgmap v5843: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:13:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:17 smithi022 ceph-mon[40250]: pgmap v5843: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:17.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:18.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:18 smithi122 ceph-mon[32255]: pgmap v5844: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/156551732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:18 smithi022 ceph-mon[31360]: pgmap v5844: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/156551732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:18 smithi022 ceph-mon[40250]: pgmap v5844: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/156551732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:20] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:13:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:20 smithi122 ceph-mon[32255]: pgmap v5845: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:20 smithi022 ceph-mon[31360]: pgmap v5845: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:20 smithi022 ceph-mon[40250]: pgmap v5845: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:13:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:13:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:22 smithi122 ceph-mon[32255]: pgmap v5846: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:23.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:23.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:22 smithi022 ceph-mon[31360]: pgmap v5846: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:23.188 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:22 smithi022 ceph-mon[40250]: pgmap v5846: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:23.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:23.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:23.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:23.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:13:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:13:23.908 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:23.908 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1207402877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1207402877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1207402877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:24 smithi122 ceph-mon[32255]: pgmap v5847: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:24 smithi022 ceph-mon[31360]: pgmap v5847: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:24 smithi022 ceph-mon[40250]: pgmap v5847: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:26 smithi122 ceph-mon[32255]: pgmap v5848: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:26 smithi022 ceph-mon[31360]: pgmap v5848: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:26 smithi022 ceph-mon[40250]: pgmap v5848: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:13:28.558 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:28.914 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:28 smithi022 ceph-mon[31360]: pgmap v5849: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:28.915 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:28 smithi022 ceph-mon[40250]: pgmap v5849: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:28.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:28.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:28.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:28 smithi122 ceph-mon[32255]: pgmap v5849: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3059958581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3059958581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3059958581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:30] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:13:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:30 smithi122 ceph-mon[32255]: pgmap v5850: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:31.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:30 smithi022 ceph-mon[31360]: pgmap v5850: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:30 smithi022 ceph-mon[40250]: pgmap v5850: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:13:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:13:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:32 smithi122 ceph-mon[32255]: pgmap v5851: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:13:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:13:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:13:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:13:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[31360]: pgmap v5851: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:13:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:13:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:13:33.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:13:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[40250]: pgmap v5851: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:13:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:13:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:13:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:13:33.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:13:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:13:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:34.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:34.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:34.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:34 smithi122 ceph-mon[32255]: pgmap v5852: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2606667209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:34 smithi022 ceph-mon[31360]: pgmap v5852: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2606667209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:34 smithi022 ceph-mon[40250]: pgmap v5852: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2606667209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:36 smithi122 ceph-mon[32255]: pgmap v5853: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:36 smithi022 ceph-mon[31360]: pgmap v5853: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:36 smithi022 ceph-mon[40250]: pgmap v5853: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:13:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:38 smithi122 ceph-mon[32255]: pgmap v5854: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:38 smithi022 ceph-mon[31360]: pgmap v5854: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:38 smithi022 ceph-mon[40250]: pgmap v5854: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:39.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:39.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:39.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:39.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4099916369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4099916369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:40] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:13:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4099916369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:40 smithi022 ceph-mon[31360]: pgmap v5855: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:40 smithi022 ceph-mon[40250]: pgmap v5855: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:40 smithi122 ceph-mon[32255]: pgmap v5855: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:13:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:13:43.248 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:42 smithi122 ceph-mon[32255]: pgmap v5856: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:42 smithi022 ceph-mon[31360]: pgmap v5856: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:42 smithi022 ceph-mon[40250]: pgmap v5856: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:13:43.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:13:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:13:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:44.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:45.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:45.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:45.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:44 smithi022 ceph-mon[31360]: pgmap v5857: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:44 smithi022 ceph-mon[40250]: pgmap v5857: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:44 smithi122 ceph-mon[32255]: pgmap v5857: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2763048408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2763048408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2763048408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:46 smithi022 ceph-mon[31360]: pgmap v5858: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:46 smithi022 ceph-mon[40250]: pgmap v5858: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:46 smithi122 ceph-mon[32255]: pgmap v5858: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:13:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:48 smithi022 ceph-mon[31360]: pgmap v5859: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:48 smithi022 ceph-mon[40250]: pgmap v5859: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:48 smithi122 ceph-mon[32255]: pgmap v5859: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:50.042 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:50.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:50.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:50] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T21:13:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:50 smithi022 ceph-mon[31360]: pgmap v5860: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1239848842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:50 smithi022 ceph-mon[40250]: pgmap v5860: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1239848842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:50 smithi122 ceph-mon[32255]: pgmap v5860: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1239848842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:13:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:13:53.252 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:52 smithi122 ceph-mon[32255]: pgmap v5861: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:52 smithi022 ceph-mon[31360]: pgmap v5861: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:52 smithi022 ceph-mon[40250]: pgmap v5861: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:53.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:13:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:13:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:13:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:54 smithi022 ceph-mon[31360]: pgmap v5862: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:54 smithi022 ceph-mon[40250]: pgmap v5862: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:55.419 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:13:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:54 smithi122 ceph-mon[32255]: pgmap v5862: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:13:55.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:13:55.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:13:55.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:13:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1100060508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1100060508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1100060508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:13:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:56 smithi022 ceph-mon[31360]: pgmap v5863: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:56 smithi022 ceph-mon[40250]: pgmap v5863: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:56 smithi122 ceph-mon[32255]: pgmap v5863: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:13:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:13:58 smithi022 ceph-mon[31360]: pgmap v5864: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:13:58 smithi022 ceph-mon[40250]: pgmap v5864: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:13:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:13:58 smithi122 ceph-mon[32255]: pgmap v5864: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:00] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T21:14:00.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:01.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:01.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:01.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:01.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:00 smithi022 ceph-mon[31360]: pgmap v5865: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:01.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:00 smithi022 ceph-mon[40250]: pgmap v5865: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:00 smithi122 ceph-mon[32255]: pgmap v5865: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1137663050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1137663050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1137663050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:14:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:14:03.263 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:02 smithi122 ceph-mon[32255]: pgmap v5866: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:02 smithi022 ceph-mon[31360]: pgmap v5866: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:02 smithi022 ceph-mon[40250]: pgmap v5866: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:03.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:14:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:14:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:04 smithi022 ceph-mon[31360]: pgmap v5867: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:05.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:04 smithi022 ceph-mon[40250]: pgmap v5867: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:04 smithi122 ceph-mon[32255]: pgmap v5867: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:06.158 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:06.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:06.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:06.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:06 smithi022 ceph-mon[31360]: pgmap v5868: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2265378218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:06 smithi022 ceph-mon[40250]: pgmap v5868: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2265378218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:07.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:06 smithi122 ceph-mon[32255]: pgmap v5868: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:07.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2265378218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:07.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:14:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:14:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:14:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:14:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:14:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:14:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:14:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:14:08.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:14:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:14:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:14:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:14:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:14:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:08 smithi022 ceph-mon[31360]: pgmap v5869: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:08 smithi022 ceph-mon[40250]: pgmap v5869: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:08 smithi122 ceph-mon[32255]: pgmap v5869: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:14:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:11 smithi022 ceph-mon[31360]: pgmap v5870: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:11 smithi022 ceph-mon[40250]: pgmap v5870: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:11 smithi122 ceph-mon[32255]: pgmap v5870: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:11.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:11.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:11.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:11.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/63375946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/63375946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/63375946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:14:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:14:13.274 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:13 smithi122 ceph-mon[32255]: pgmap v5871: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:13 smithi022 ceph-mon[31360]: pgmap v5871: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:13 smithi022 ceph-mon[40250]: pgmap v5871: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:13.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:14:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:14:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:15 smithi022 ceph-mon[31360]: pgmap v5872: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:15 smithi022 ceph-mon[40250]: pgmap v5872: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:15 smithi122 ceph-mon[32255]: pgmap v5872: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:16.897 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:17.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:17.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:17.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:17.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:17 smithi022 ceph-mon[31360]: pgmap v5873: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:17.268 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:17 smithi022 ceph-mon[40250]: pgmap v5873: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:17 smithi122 ceph-mon[32255]: pgmap v5873: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:14:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1309213532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1309213532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1309213532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:19 smithi022 ceph-mon[31360]: pgmap v5874: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:19 smithi022 ceph-mon[40250]: pgmap v5874: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:19 smithi122 ceph-mon[32255]: pgmap v5874: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:20] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:14:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:21 smithi022 ceph-mon[31360]: pgmap v5875: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:21 smithi022 ceph-mon[40250]: pgmap v5875: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:21 smithi122 ceph-mon[32255]: pgmap v5875: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:22.268 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:22.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:22.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:22.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:14:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:14:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:23 smithi022 ceph-mon[31360]: pgmap v5876: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/139506582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:23 smithi022 ceph-mon[40250]: pgmap v5876: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/139506582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:23.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:14:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:14:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:23 smithi122 ceph-mon[32255]: pgmap v5876: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/139506582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:25 smithi122 ceph-mon[32255]: pgmap v5877: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:25 smithi022 ceph-mon[31360]: pgmap v5877: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:25 smithi022 ceph-mon[40250]: pgmap v5877: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:27 smithi022 ceph-mon[31360]: pgmap v5878: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:27 smithi022 ceph-mon[40250]: pgmap v5878: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:27 smithi122 ceph-mon[32255]: pgmap v5878: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:27.635 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:14:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:28 smithi122 ceph-mon[32255]: pgmap v5879: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4255437284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:28 smithi022 ceph-mon[31360]: pgmap v5879: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4255437284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:28 smithi022 ceph-mon[40250]: pgmap v5879: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4255437284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:30] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:14:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:30 smithi122 ceph-mon[32255]: pgmap v5880: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:30 smithi022 ceph-mon[31360]: pgmap v5880: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:30 smithi022 ceph-mon[40250]: pgmap v5880: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:14:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:14:33.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:32 smithi122 ceph-mon[32255]: pgmap v5881: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:14:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:14:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:14:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:14:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[31360]: pgmap v5881: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:14:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:14:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:14:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:14:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[40250]: pgmap v5881: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:14:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:14:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:14:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:14:33.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:33.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:33.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:33.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:14:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:14:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2450832382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2450832382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2450832382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:34 smithi122 ceph-mon[32255]: pgmap v5882: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:34 smithi022 ceph-mon[31360]: pgmap v5882: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:34 smithi022 ceph-mon[40250]: pgmap v5882: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:36 smithi122 ceph-mon[32255]: pgmap v5883: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:36 smithi022 ceph-mon[31360]: pgmap v5883: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:36 smithi022 ceph-mon[40250]: pgmap v5883: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:14:38.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:38.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:38.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:38.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:38 smithi022 ceph-mon[31360]: pgmap v5884: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2115713907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:38 smithi022 ceph-mon[40250]: pgmap v5884: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2115713907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:38 smithi122 ceph-mon[32255]: pgmap v5884: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2115713907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:40] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:14:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:40 smithi022 ceph-mon[31360]: pgmap v5885: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:40 smithi022 ceph-mon[40250]: pgmap v5885: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:40 smithi122 ceph-mon[32255]: pgmap v5885: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:14:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:14:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:42 smithi022 ceph-mon[31360]: pgmap v5886: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:42 smithi022 ceph-mon[40250]: pgmap v5886: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:43.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:42 smithi122 ceph-mon[32255]: pgmap v5886: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:43.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:14:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:14:43.756 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:44.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:44.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:44.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:44 smithi022 ceph-mon[31360]: pgmap v5887: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3667118817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:44 smithi022 ceph-mon[40250]: pgmap v5887: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3667118817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:44 smithi122 ceph-mon[32255]: pgmap v5887: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3667118817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:46 smithi022 ceph-mon[31360]: pgmap v5888: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:46 smithi022 ceph-mon[40250]: pgmap v5888: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:46 smithi122 ceph-mon[32255]: pgmap v5888: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:14:49.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:48 smithi022 ceph-mon[31360]: pgmap v5889: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:48 smithi022 ceph-mon[40250]: pgmap v5889: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:48 smithi122 ceph-mon[32255]: pgmap v5889: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:49.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:49.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:49.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/18994235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/18994235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/18994235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:50] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:14:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:50 smithi022 ceph-mon[31360]: pgmap v5890: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:50 smithi022 ceph-mon[40250]: pgmap v5890: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:50 smithi122 ceph-mon[32255]: pgmap v5890: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:14:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:14:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:52 smithi022 ceph-mon[31360]: pgmap v5891: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:52 smithi022 ceph-mon[40250]: pgmap v5891: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:53.296 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:52 smithi122 ceph-mon[32255]: pgmap v5891: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:53.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:14:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:14:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:14:54.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:14:54.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:14:54.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:14:54.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:14:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:54 smithi022 ceph-mon[40250]: pgmap v5892: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1234681744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:54 smithi022 ceph-mon[31360]: pgmap v5892: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:55.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1234681744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:54 smithi122 ceph-mon[32255]: pgmap v5892: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:14:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1234681744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:14:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:56 smithi022 ceph-mon[31360]: pgmap v5893: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:56 smithi022 ceph-mon[40250]: pgmap v5893: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:56 smithi122 ceph-mon[32255]: pgmap v5893: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:14:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:14:58 smithi022 ceph-mon[31360]: pgmap v5894: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:14:58 smithi022 ceph-mon[40250]: pgmap v5894: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:14:58 smithi122 ceph-mon[32255]: pgmap v5894: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:14:59.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:00.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:00.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:00.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:00] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:15:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:00 smithi022 ceph-mon[31360]: pgmap v5895: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4104280793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:00 smithi022 ceph-mon[40250]: pgmap v5895: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4104280793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:00 smithi122 ceph-mon[32255]: pgmap v5895: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4104280793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:15:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:15:03.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:02 smithi022 ceph-mon[31360]: pgmap v5896: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:03.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:02 smithi022 ceph-mon[40250]: pgmap v5896: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:03.297 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:02 smithi122 ceph-mon[32255]: pgmap v5896: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:03.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:15:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:15:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:05.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:04 smithi022 ceph-mon[31360]: pgmap v5897: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:04 smithi022 ceph-mon[40250]: pgmap v5897: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:04 smithi122 ceph-mon[32255]: pgmap v5897: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:05.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:05.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:05.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:05.993 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1289198257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1289198257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1289198257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:06 smithi022 ceph-mon[31360]: pgmap v5898: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:06 smithi022 ceph-mon[40250]: pgmap v5898: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:06 smithi122 ceph-mon[32255]: pgmap v5898: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:15:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:15:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:15:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:15:08.200 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:15:08.201 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:15:08.201 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:15:08.201 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:15:08.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:15:08.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:15:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:09 smithi022 ceph-mon[31360]: pgmap v5899: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:15:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:09 smithi022 ceph-mon[40250]: pgmap v5899: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:15:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:09 smithi122 ceph-mon[32255]: pgmap v5899: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:15:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:10] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:15:10.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:10.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:10.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:10.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:11 smithi022 ceph-mon[31360]: pgmap v5900: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2676465572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:11 smithi022 ceph-mon[40250]: pgmap v5900: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2676465572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:11 smithi122 ceph-mon[32255]: pgmap v5900: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2676465572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:15:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:15:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:13 smithi022 ceph-mon[31360]: pgmap v5901: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:13 smithi022 ceph-mon[40250]: pgmap v5901: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:13 smithi122 ceph-mon[32255]: pgmap v5901: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:13.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:15:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:15:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:15 smithi022 ceph-mon[31360]: pgmap v5902: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:15 smithi022 ceph-mon[40250]: pgmap v5902: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:15 smithi122 ceph-mon[32255]: pgmap v5902: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:15.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:16.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:15:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:17 smithi022 ceph-mon[31360]: pgmap v5903: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2260385003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:17 smithi022 ceph-mon[40250]: pgmap v5903: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2260385003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:17 smithi122 ceph-mon[32255]: pgmap v5903: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2260385003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:19 smithi022 ceph-mon[31360]: pgmap v5904: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:19 smithi022 ceph-mon[40250]: pgmap v5904: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:19 smithi122 ceph-mon[32255]: pgmap v5904: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:20] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:15:21.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:21 smithi022 ceph-mon[31360]: pgmap v5905: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:21 smithi022 ceph-mon[40250]: pgmap v5905: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:21 smithi122 ceph-mon[32255]: pgmap v5905: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:21.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:21.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:21.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2568839844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2568839844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2568839844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:15:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:15:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:23 smithi022 ceph-mon[31360]: pgmap v5906: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:23 smithi022 ceph-mon[40250]: pgmap v5906: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:23 smithi122 ceph-mon[32255]: pgmap v5906: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:23.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:15:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:15:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:25 smithi022 ceph-mon[31360]: pgmap v5907: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:25 smithi022 ceph-mon[40250]: pgmap v5907: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:25 smithi122 ceph-mon[32255]: pgmap v5907: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:26.722 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:27.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:27.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:27.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:27 smithi022 ceph-mon[31360]: pgmap v5908: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/288127281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:27 smithi022 ceph-mon[40250]: pgmap v5908: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/288127281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:27 smithi122 ceph-mon[32255]: pgmap v5908: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/288127281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:15:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:29 smithi022 ceph-mon[31360]: pgmap v5909: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:29 smithi022 ceph-mon[40250]: pgmap v5909: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:29 smithi122 ceph-mon[32255]: pgmap v5909: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:30] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:15:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:31 smithi022 ceph-mon[31360]: pgmap v5910: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:31 smithi022 ceph-mon[40250]: pgmap v5910: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:31 smithi122 ceph-mon[32255]: pgmap v5910: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:32.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:32.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:32.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:32.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:15:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:15:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[31360]: pgmap v5911: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1728369122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:15:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:15:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:15:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:15:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[40250]: pgmap v5911: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1728369122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:15:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:15:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:15:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:15:33.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:15:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:15:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:33 smithi122 ceph-mon[32255]: pgmap v5911: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1728369122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:15:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:15:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:15:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:15:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:35 smithi022 ceph-mon[31360]: pgmap v5912: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:35 smithi022 ceph-mon[40250]: pgmap v5912: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:35 smithi122 ceph-mon[32255]: pgmap v5912: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:37.465 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:37 smithi022 ceph-mon[31360]: pgmap v5913: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:15:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:37 smithi022 ceph-mon[40250]: pgmap v5913: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:37 smithi122 ceph-mon[32255]: pgmap v5913: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:37.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:37.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:37.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:38.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2958772532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2958772532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2958772532' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:39 smithi022 ceph-mon[31360]: pgmap v5914: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:39 smithi022 ceph-mon[40250]: pgmap v5914: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:39 smithi122 ceph-mon[32255]: pgmap v5914: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:40] "GET /metrics HTTP/1.1" 200 202344 "" "Prometheus/2.18.1" 2021-12-15T21:15:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:41 smithi022 ceph-mon[31360]: pgmap v5915: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:41 smithi022 ceph-mon[40250]: pgmap v5915: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:41 smithi122 ceph-mon[32255]: pgmap v5915: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:15:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:15:42.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:43.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:43.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:43.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:43 smithi022 ceph-mon[31360]: pgmap v5916: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3175425883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:43 smithi022 ceph-mon[40250]: pgmap v5916: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3175425883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:43.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:15:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:15:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:43 smithi122 ceph-mon[32255]: pgmap v5916: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3175425883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:45 smithi022 ceph-mon[31360]: pgmap v5917: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:45 smithi022 ceph-mon[40250]: pgmap v5917: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:45 smithi122 ceph-mon[32255]: pgmap v5917: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:47 smithi022 ceph-mon[31360]: pgmap v5918: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:15:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:47 smithi022 ceph-mon[40250]: pgmap v5918: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:47 smithi122 ceph-mon[32255]: pgmap v5918: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:48.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:48.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:48.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:48.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:49 smithi122 ceph-mon[32255]: pgmap v5919: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4196002856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:49 smithi022 ceph-mon[31360]: pgmap v5919: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4196002856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:49 smithi022 ceph-mon[40250]: pgmap v5919: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4196002856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:50] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:15:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:51 smithi122 ceph-mon[32255]: pgmap v5920: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:51 smithi022 ceph-mon[31360]: pgmap v5920: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:51 smithi022 ceph-mon[40250]: pgmap v5920: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:15:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:15:53.585 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:53.585 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:53 smithi022 ceph-mon[31360]: pgmap v5921: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:53.586 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:53 smithi022 ceph-mon[40250]: pgmap v5921: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:53 smithi122 ceph-mon[32255]: pgmap v5921: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:53.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:15:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:15:53.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:53.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:53.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2152299809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2152299809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:15:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2152299809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:15:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:55 smithi122 ceph-mon[32255]: pgmap v5922: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:55 smithi022 ceph-mon[31360]: pgmap v5922: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:55 smithi022 ceph-mon[40250]: pgmap v5922: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:15:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:57 smithi122 ceph-mon[32255]: pgmap v5923: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:57 smithi022 ceph-mon[31360]: pgmap v5923: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:15:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:57 smithi022 ceph-mon[40250]: pgmap v5923: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:58.954 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:15:59.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:15:59.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:15:59.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:15:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:15:59 smithi122 ceph-mon[32255]: pgmap v5924: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:15:59 smithi022 ceph-mon[31360]: pgmap v5924: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:15:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:15:59 smithi022 ceph-mon[40250]: pgmap v5924: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:00] "GET /metrics HTTP/1.1" 200 202348 "" "Prometheus/2.18.1" 2021-12-15T21:16:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2099179374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2099179374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2099179374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:01 smithi122 ceph-mon[32255]: pgmap v5925: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:01 smithi022 ceph-mon[31360]: pgmap v5925: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:01 smithi022 ceph-mon[40250]: pgmap v5925: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:16:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:16:03.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:16:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:16:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:03 smithi122 ceph-mon[32255]: pgmap v5926: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:03 smithi022 ceph-mon[31360]: pgmap v5926: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:03 smithi022 ceph-mon[40250]: pgmap v5926: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:04.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:04.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:04.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:04.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:04.691 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:04.692 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:05 smithi122 ceph-mon[32255]: pgmap v5927: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2504295418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:05 smithi022 ceph-mon[31360]: pgmap v5927: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2504295418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:05 smithi022 ceph-mon[40250]: pgmap v5927: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2504295418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:07 smithi122 ceph-mon[32255]: pgmap v5928: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:07 smithi022 ceph-mon[31360]: pgmap v5928: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:16:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:07 smithi022 ceph-mon[40250]: pgmap v5928: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[31360]: pgmap v5929: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:16:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:16:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:16:09.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:16:09.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[40250]: pgmap v5929: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:09.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:16:09.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:16:09.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:16:09.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:16:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:08 smithi122 ceph-mon[32255]: pgmap v5929: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:16:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:16:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:16:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:16:09.693 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:10.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:10.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:10.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:10] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:16:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:10 smithi022 ceph-mon[31360]: pgmap v5930: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3075098949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:10 smithi022 ceph-mon[40250]: pgmap v5930: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3075098949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:10 smithi122 ceph-mon[32255]: pgmap v5930: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3075098949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:16:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:16:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:12 smithi022 ceph-mon[31360]: pgmap v5931: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:12 smithi022 ceph-mon[40250]: pgmap v5931: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:13.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:12 smithi122 ceph-mon[32255]: pgmap v5931: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:13.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:16:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:16:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:15.066 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:14 smithi022 ceph-mon[31360]: pgmap v5932: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:14 smithi022 ceph-mon[40250]: pgmap v5932: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:15.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:15.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:15.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:14 smithi122 ceph-mon[32255]: pgmap v5932: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1483274533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1483274533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1483274533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:16 smithi022 ceph-mon[31360]: pgmap v5933: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:16 smithi022 ceph-mon[40250]: pgmap v5933: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:16 smithi122 ceph-mon[32255]: pgmap v5933: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:16:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:18 smithi022 ceph-mon[31360]: pgmap v5934: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:18 smithi022 ceph-mon[40250]: pgmap v5934: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:18 smithi122 ceph-mon[32255]: pgmap v5934: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:20] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:16:20.437 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:20.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:20.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:20.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:20 smithi022 ceph-mon[31360]: pgmap v5935: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3739590345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:20 smithi022 ceph-mon[40250]: pgmap v5935: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3739590345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:20 smithi122 ceph-mon[32255]: pgmap v5935: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3739590345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:16:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:16:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:22 smithi022 ceph-mon[31360]: pgmap v5936: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:22 smithi022 ceph-mon[40250]: pgmap v5936: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:23.355 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:22 smithi122 ceph-mon[32255]: pgmap v5936: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:23.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:16:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:16:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:24 smithi022 ceph-mon[31360]: pgmap v5937: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:24 smithi022 ceph-mon[40250]: pgmap v5937: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:24 smithi122 ceph-mon[32255]: pgmap v5937: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:25.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:26 smithi022 ceph-mon[31360]: pgmap v5938: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3140026274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:26 smithi022 ceph-mon[40250]: pgmap v5938: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3140026274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:27.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:26 smithi122 ceph-mon[32255]: pgmap v5938: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:27.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3140026274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:16:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:28 smithi022 ceph-mon[40250]: pgmap v5939: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:28 smithi022 ceph-mon[31360]: pgmap v5939: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:28 smithi122 ceph-mon[32255]: pgmap v5939: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:30] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:16:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:30 smithi022 ceph-mon[31360]: pgmap v5940: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:30 smithi022 ceph-mon[40250]: pgmap v5940: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:30 smithi122 ceph-mon[32255]: pgmap v5940: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:31.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:31.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:31.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:31.983 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2592540082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2592540082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2592540082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:16:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:16:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[31360]: pgmap v5941: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:16:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:16:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:16:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:16:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[40250]: pgmap v5941: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:16:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:16:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:16:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:16:33.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:32 smithi122 ceph-mon[32255]: pgmap v5941: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:33.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:16:33.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:16:33.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:16:33.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:16:33.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:16:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:16:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:34 smithi022 ceph-mon[31360]: pgmap v5942: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:34 smithi022 ceph-mon[40250]: pgmap v5942: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:34 smithi122 ceph-mon[32255]: pgmap v5942: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:36.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:36.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:36.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:36.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:36 smithi022 ceph-mon[31360]: pgmap v5943: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2464424146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:36 smithi022 ceph-mon[40250]: pgmap v5943: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2464424146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:36 smithi122 ceph-mon[32255]: pgmap v5943: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2464424146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:16:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:38 smithi022 ceph-mon[31360]: pgmap v5944: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:38 smithi022 ceph-mon[40250]: pgmap v5944: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:38 smithi122 ceph-mon[32255]: pgmap v5944: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:40] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:16:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:40 smithi022 ceph-mon[31360]: pgmap v5945: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:40 smithi022 ceph-mon[40250]: pgmap v5945: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:40 smithi122 ceph-mon[32255]: pgmap v5945: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:41.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:42.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:42.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:42.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:16:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:16:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:43 smithi022 ceph-mon[31360]: pgmap v5946: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1575957199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:43 smithi022 ceph-mon[40250]: pgmap v5946: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1575957199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:43.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:43 smithi122 ceph-mon[32255]: pgmap v5946: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:43.369 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1575957199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:43.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:16:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:16:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:45 smithi022 ceph-mon[31360]: pgmap v5947: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:45 smithi022 ceph-mon[40250]: pgmap v5947: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:45 smithi122 ceph-mon[32255]: pgmap v5947: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:47 smithi022 ceph-mon[31360]: pgmap v5948: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T21:16:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:47 smithi022 ceph-mon[40250]: pgmap v5948: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T21:16:47.292 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:47 smithi122 ceph-mon[32255]: pgmap v5948: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T21:16:47.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:47.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:47.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:47.665 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:16:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3688340866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3688340866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3688340866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:49 smithi022 ceph-mon[31360]: pgmap v5949: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:49 smithi022 ceph-mon[40250]: pgmap v5949: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:49 smithi122 ceph-mon[32255]: pgmap v5949: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:50] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:16:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:51 smithi022 ceph-mon[31360]: pgmap v5950: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:51 smithi022 ceph-mon[40250]: pgmap v5950: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:51 smithi122 ceph-mon[32255]: pgmap v5950: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:52.666 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:16:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:16:53.026 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:53 smithi022 ceph-mon[40250]: pgmap v5951: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:53.026 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/943195030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:53.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:53 smithi022 ceph-mon[31360]: pgmap v5951: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:53.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/943195030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:53.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:53.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:53.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:53.379 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:53 smithi122 ceph-mon[32255]: pgmap v5951: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:53.379 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/943195030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:53.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:16:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:16:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:16:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:55 smithi122 ceph-mon[32255]: pgmap v5952: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:55 smithi022 ceph-mon[31360]: pgmap v5952: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:55 smithi022 ceph-mon[40250]: pgmap v5952: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:16:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:57 smithi022 ceph-mon[31360]: pgmap v5953: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:57 smithi022 ceph-mon[40250]: pgmap v5953: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:57 smithi122 ceph-mon[32255]: pgmap v5953: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:16:58.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:16:58.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:16:58.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:16:58.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:16:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:59 smithi122 ceph-mon[32255]: pgmap v5954: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:16:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3716233255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:59 smithi022 ceph-mon[31360]: pgmap v5954: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:16:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3716233255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:16:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:59 smithi022 ceph-mon[40250]: pgmap v5954: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:16:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:16:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3716233255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:00] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:17:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:01 smithi122 ceph-mon[32255]: pgmap v5955: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:01 smithi022 ceph-mon[31360]: pgmap v5955: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:01 smithi022 ceph-mon[40250]: pgmap v5955: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:17:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:17:03.390 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:03 smithi122 ceph-mon[32255]: pgmap v5956: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:03.406 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:03.406 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:03 smithi022 ceph-mon[31360]: pgmap v5956: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:03.406 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:03 smithi022 ceph-mon[40250]: pgmap v5956: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:03.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:17:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:17:03.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:03.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:03.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3813188824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3813188824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3813188824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:05 smithi122 ceph-mon[32255]: pgmap v5957: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:05 smithi022 ceph-mon[31360]: pgmap v5957: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:05 smithi022 ceph-mon[40250]: pgmap v5957: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:07 smithi022 ceph-mon[31360]: pgmap v5958: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:07 smithi022 ceph-mon[40250]: pgmap v5958: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:07 smithi122 ceph-mon[32255]: pgmap v5958: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:17:08.769 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:09.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:09.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:09.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:09 smithi122 ceph-mon[32255]: pgmap v5959: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:17:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:17:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:17:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[31360]: pgmap v5959: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:17:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:17:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:17:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[40250]: pgmap v5959: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:17:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:17:09.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:17:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:10] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:17:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/28654506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:17:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/28654506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:17:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/28654506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:17:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:11 smithi122 ceph-mon[32255]: pgmap v5960: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:11 smithi022 ceph-mon[31360]: pgmap v5960: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:11 smithi022 ceph-mon[40250]: pgmap v5960: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:17:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:17:13.399 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:13 smithi122 ceph-mon[32255]: pgmap v5961: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:13 smithi022 ceph-mon[31360]: pgmap v5961: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:13 smithi022 ceph-mon[40250]: pgmap v5961: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:13.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:17:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:17:14.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:14.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:14.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:14.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:14.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:14.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:15 smithi122 ceph-mon[32255]: pgmap v5962: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3490013030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:15 smithi022 ceph-mon[31360]: pgmap v5962: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3490013030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:15 smithi022 ceph-mon[40250]: pgmap v5962: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3490013030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:17 smithi122 ceph-mon[32255]: pgmap v5963: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:17 smithi022 ceph-mon[31360]: pgmap v5963: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:17:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:17 smithi022 ceph-mon[40250]: pgmap v5963: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:19 smithi122 ceph-mon[32255]: pgmap v5964: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:19.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:19 smithi022 ceph-mon[31360]: pgmap v5964: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:19.519 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:19 smithi022 ceph-mon[40250]: pgmap v5964: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:19.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:19.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:19.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:19.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:20.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1119846114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:20] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:17:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1119846114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1119846114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:21 smithi122 ceph-mon[32255]: pgmap v5965: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:21 smithi022 ceph-mon[31360]: pgmap v5965: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:21 smithi022 ceph-mon[40250]: pgmap v5965: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:17:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:17:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:23 smithi122 ceph-mon[32255]: pgmap v5966: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:23.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:17:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:17:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:23 smithi022 ceph-mon[31360]: pgmap v5966: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:23 smithi022 ceph-mon[40250]: pgmap v5966: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:25.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:25.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:25.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:25 smithi122 ceph-mon[32255]: pgmap v5967: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:25 smithi022 ceph-mon[31360]: pgmap v5967: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:25 smithi022 ceph-mon[40250]: pgmap v5967: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3945964222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3945964222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3945964222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:27 smithi122 ceph-mon[32255]: pgmap v5968: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:27 smithi022 ceph-mon[31360]: pgmap v5968: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:17:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:27 smithi022 ceph-mon[40250]: pgmap v5968: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:29 smithi122 ceph-mon[32255]: pgmap v5969: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:29 smithi022 ceph-mon[31360]: pgmap v5969: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:29 smithi022 ceph-mon[40250]: pgmap v5969: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:30.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:30] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:17:30.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:30.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:30.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:31 smithi122 ceph-mon[32255]: pgmap v5970: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/386804444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:31 smithi022 ceph-mon[31360]: pgmap v5970: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/386804444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:31 smithi022 ceph-mon[40250]: pgmap v5970: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/386804444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:17:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:17:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[40250]: pgmap v5971: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:17:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:17:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:17:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:17:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[31360]: pgmap v5971: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:17:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:17:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:17:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:17:33.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:17:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:17:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:33 smithi122 ceph-mon[32255]: pgmap v5971: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:17:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:17:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:17:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:17:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:35 smithi022 ceph-mon[31360]: pgmap v5972: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:35 smithi022 ceph-mon[40250]: pgmap v5972: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:35.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:35 smithi122 ceph-mon[32255]: pgmap v5972: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:36.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:36.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:36.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/140284903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/140284903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/140284903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:37 smithi022 ceph-mon[31360]: pgmap v5973: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:17:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:37 smithi022 ceph-mon[40250]: pgmap v5973: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:37 smithi122 ceph-mon[32255]: pgmap v5973: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:39 smithi022 ceph-mon[31360]: pgmap v5974: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:39 smithi022 ceph-mon[40250]: pgmap v5974: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:39 smithi122 ceph-mon[32255]: pgmap v5974: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:40] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:17:41.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:41 smithi022 ceph-mon[31360]: pgmap v5975: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:41 smithi022 ceph-mon[40250]: pgmap v5975: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:41.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:41.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:41.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:41 smithi122 ceph-mon[32255]: pgmap v5975: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3386772794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3386772794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3386772794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:17:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:17:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:43 smithi022 ceph-mon[31360]: pgmap v5976: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:43 smithi022 ceph-mon[40250]: pgmap v5976: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:43.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:17:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:17:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:43 smithi122 ceph-mon[32255]: pgmap v5976: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:45 smithi022 ceph-mon[31360]: pgmap v5977: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:45 smithi022 ceph-mon[40250]: pgmap v5977: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:45 smithi122 ceph-mon[32255]: pgmap v5977: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:46.374 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:46.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:46.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:46.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:17:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:47 smithi022 ceph-mon[31360]: pgmap v5978: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3687130436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:47.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:47 smithi022 ceph-mon[40250]: pgmap v5978: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:47.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3687130436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:47 smithi122 ceph-mon[32255]: pgmap v5978: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3687130436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:49 smithi022 ceph-mon[31360]: pgmap v5979: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:49 smithi022 ceph-mon[40250]: pgmap v5979: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:49 smithi122 ceph-mon[32255]: pgmap v5979: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:50] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:17:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:51 smithi022 ceph-mon[31360]: pgmap v5980: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:51 smithi022 ceph-mon[40250]: pgmap v5980: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:51 smithi122 ceph-mon[32255]: pgmap v5980: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:17:51.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:52.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:52.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:52.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/733391673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/733391673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/733391673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:17:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:17:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:53 smithi022 ceph-mon[31360]: pgmap v5981: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:53 smithi022 ceph-mon[40250]: pgmap v5981: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:53.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:17:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:17:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:53 smithi122 ceph-mon[32255]: pgmap v5981: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:17:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:17:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:55 smithi022 ceph-mon[31360]: pgmap v5982: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T21:17:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:55 smithi022 ceph-mon[40250]: pgmap v5982: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T21:17:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:55 smithi122 ceph-mon[32255]: pgmap v5982: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T21:17:57.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:17:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:57 smithi022 ceph-mon[31360]: pgmap v5983: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:17:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:57 smithi022 ceph-mon[40250]: pgmap v5983: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:17:57.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:17:57.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:17:57.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:17:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:57 smithi122 ceph-mon[32255]: pgmap v5983: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:17:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:17:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3356743541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3356743541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3356743541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:17:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:17:59 smithi022 ceph-mon[31360]: pgmap v5984: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:17:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:17:59 smithi022 ceph-mon[40250]: pgmap v5984: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:17:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:17:59 smithi122 ceph-mon[32255]: pgmap v5984: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:18:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:00] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:18:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:01 smithi022 ceph-mon[31360]: pgmap v5985: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T21:18:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:01 smithi022 ceph-mon[40250]: pgmap v5985: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T21:18:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:01 smithi122 ceph-mon[32255]: pgmap v5985: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T21:18:02.508 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:18:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:18:02.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:02.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:02.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:03 smithi022 ceph-mon[31360]: pgmap v5986: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:18:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3945970418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:03 smithi022 ceph-mon[40250]: pgmap v5986: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:18:03.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3945970418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:03.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:18:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:18:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:03 smithi122 ceph-mon[32255]: pgmap v5986: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T21:18:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3945970418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:05 smithi022 ceph-mon[31360]: pgmap v5987: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T21:18:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:05 smithi022 ceph-mon[40250]: pgmap v5987: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T21:18:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:05 smithi122 ceph-mon[32255]: pgmap v5987: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T21:18:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:07 smithi022 ceph-mon[31360]: pgmap v5988: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:18:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:07 smithi022 ceph-mon[40250]: pgmap v5988: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:07 smithi122 ceph-mon[32255]: pgmap v5988: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:07.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:08.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:09.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:08 smithi022 ceph-mon[31360]: pgmap v5989: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:09.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/69370658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:09.220 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:08 smithi022 ceph-mon[40250]: pgmap v5989: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:09.221 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/69370658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:08 smithi122 ceph-mon[32255]: pgmap v5989: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/69370658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:09.965 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:18:09.966 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:18:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:18:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:18:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:18:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:18:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:10] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:10 smithi022 ceph-mon[31360]: pgmap v5990: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:18:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:10 smithi022 ceph-mon[40250]: pgmap v5990: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:18:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:10 smithi122 ceph-mon[32255]: pgmap v5990: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:18:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:18:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:18:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:18:13.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:13.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:12 smithi022 ceph-mon[40250]: pgmap v5991: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:13.252 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:12 smithi022 ceph-mon[31360]: pgmap v5991: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:12 smithi122 ceph-mon[32255]: pgmap v5991: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:18:13.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:13.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:13.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:13.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:18:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:18:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3379798357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3379798357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3379798357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:15 smithi022 ceph-mon[31360]: pgmap v5992: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:18:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:15 smithi022 ceph-mon[40250]: pgmap v5992: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:18:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:15 smithi122 ceph-mon[32255]: pgmap v5992: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:18:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:17 smithi022 ceph-mon[31360]: pgmap v5993: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:17 smithi022 ceph-mon[40250]: pgmap v5993: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:17 smithi122 ceph-mon[32255]: pgmap v5993: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:18:18.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:19 smithi022 ceph-mon[31360]: pgmap v5994: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/801092522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:19 smithi022 ceph-mon[40250]: pgmap v5994: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/801092522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:19 smithi122 ceph-mon[32255]: pgmap v5994: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/801092522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:20] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:18:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:21 smithi022 ceph-mon[31360]: pgmap v5995: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:21 smithi022 ceph-mon[40250]: pgmap v5995: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:21 smithi122 ceph-mon[32255]: pgmap v5995: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:18:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:18:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:23 smithi022 ceph-mon[31360]: pgmap v5996: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:23 smithi022 ceph-mon[40250]: pgmap v5996: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:23 smithi122 ceph-mon[32255]: pgmap v5996: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:23.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:18:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:18:23.993 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:24.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:24.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:24.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:25 smithi022 ceph-mon[31360]: pgmap v5997: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1228995386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:25 smithi022 ceph-mon[40250]: pgmap v5997: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1228995386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:25 smithi122 ceph-mon[32255]: pgmap v5997: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1228995386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:27 smithi022 ceph-mon[31360]: pgmap v5998: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:27 smithi022 ceph-mon[40250]: pgmap v5998: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:27 smithi122 ceph-mon[32255]: pgmap v5998: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:18:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:29 smithi022 ceph-mon[31360]: pgmap v5999: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:29 smithi022 ceph-mon[40250]: pgmap v5999: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:29.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:29 smithi122 ceph-mon[32255]: pgmap v5999: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:29.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:29.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:29.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/536781206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/536781206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:30] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:18:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/536781206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:31 smithi122 ceph-mon[32255]: pgmap v6000: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:31 smithi022 ceph-mon[31360]: pgmap v6000: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:31 smithi022 ceph-mon[40250]: pgmap v6000: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:18:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:18:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:33 smithi122 ceph-mon[32255]: pgmap v6001: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:18:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:18:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:18:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:18:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[40250]: pgmap v6001: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:18:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:18:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:18:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:18:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[31360]: pgmap v6001: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:18:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:18:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:18:33.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:18:33.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:18:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:18:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:34.733 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:35.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:35.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:35.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:35 smithi122 ceph-mon[32255]: pgmap v6002: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:35 smithi022 ceph-mon[31360]: pgmap v6002: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:35 smithi022 ceph-mon[40250]: pgmap v6002: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2662421098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2662421098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2662421098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:37 smithi022 ceph-mon[31360]: pgmap v6003: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:37 smithi022 ceph-mon[40250]: pgmap v6003: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:37 smithi122 ceph-mon[32255]: pgmap v6003: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:18:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:39 smithi122 ceph-mon[32255]: pgmap v6004: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:39 smithi022 ceph-mon[31360]: pgmap v6004: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:39 smithi022 ceph-mon[40250]: pgmap v6004: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:40.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:40] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:18:40.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:40.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:40.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:41 smithi122 ceph-mon[32255]: pgmap v6005: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3495904735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:41 smithi022 ceph-mon[31360]: pgmap v6005: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3495904735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:41 smithi022 ceph-mon[40250]: pgmap v6005: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:41.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3495904735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:18:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:18:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:43 smithi122 ceph-mon[32255]: pgmap v6006: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:43 smithi022 ceph-mon[31360]: pgmap v6006: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:43 smithi022 ceph-mon[40250]: pgmap v6006: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:43.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:18:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:18:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:45 smithi122 ceph-mon[32255]: pgmap v6007: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:45.469 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:45.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:45 smithi022 ceph-mon[31360]: pgmap v6007: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:45.470 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:45 smithi022 ceph-mon[40250]: pgmap v6007: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:45.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:45.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:45.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2414120305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2414120305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2414120305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:47 smithi022 ceph-mon[31360]: pgmap v6008: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:47 smithi022 ceph-mon[40250]: pgmap v6008: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:47 smithi122 ceph-mon[32255]: pgmap v6008: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:18:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:49 smithi122 ceph-mon[32255]: pgmap v6009: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:49 smithi022 ceph-mon[31360]: pgmap v6009: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:49 smithi022 ceph-mon[40250]: pgmap v6009: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:50] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:18:50.843 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:51.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:51.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:51.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:51.214 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:51 smithi022 ceph-mon[31360]: pgmap v6010: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:51.215 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:51 smithi022 ceph-mon[40250]: pgmap v6010: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:51 smithi122 ceph-mon[32255]: pgmap v6010: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3509465248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:18:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:18:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3509465248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:52.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3509465248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:53 smithi122 ceph-mon[32255]: pgmap v6011: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:53 smithi022 ceph-mon[31360]: pgmap v6011: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:53 smithi022 ceph-mon[40250]: pgmap v6011: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:53.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:18:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:18:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:18:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:55 smithi122 ceph-mon[32255]: pgmap v6012: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:55 smithi022 ceph-mon[40250]: pgmap v6012: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:55 smithi022 ceph-mon[31360]: pgmap v6012: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:18:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:18:56.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:18:56.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:18:56.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:18:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:57 smithi022 ceph-mon[31360]: pgmap v6013: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1874869921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:57 smithi022 ceph-mon[40250]: pgmap v6013: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1874869921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:57 smithi122 ceph-mon[32255]: pgmap v6013: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1874869921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:18:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:18:59.094 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:18:59 smithi022 ceph-mon[31360]: pgmap v6014: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:59.095 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:18:59 smithi022 ceph-mon[40250]: pgmap v6014: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:18:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:18:59 smithi122 ceph-mon[32255]: pgmap v6014: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:00] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:19:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:01 smithi122 ceph-mon[32255]: pgmap v6015: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:01 smithi022 ceph-mon[31360]: pgmap v6015: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:01 smithi022 ceph-mon[40250]: pgmap v6015: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:01.587 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:01.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:01.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:01.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4250663687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4250663687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4250663687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:19:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:19:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:03 smithi122 ceph-mon[32255]: pgmap v6016: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:03 smithi022 ceph-mon[31360]: pgmap v6016: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:03 smithi022 ceph-mon[40250]: pgmap v6016: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:03.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:19:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:19:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:05 smithi122 ceph-mon[32255]: pgmap v6017: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:05 smithi022 ceph-mon[31360]: pgmap v6017: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:05 smithi022 ceph-mon[40250]: pgmap v6017: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:06.960 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:07 smithi022 ceph-mon[31360]: pgmap v6018: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:07 smithi022 ceph-mon[40250]: pgmap v6018: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:07.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:07.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:07.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:07 smithi122 ceph-mon[32255]: pgmap v6018: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:19:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3139545402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3139545402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:08.633 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3139545402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:09 smithi122 ceph-mon[32255]: pgmap v6019: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:09 smithi022 ceph-mon[31360]: pgmap v6019: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:09 smithi022 ceph-mon[40250]: pgmap v6019: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:19:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:19:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:19:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:10] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:19:10.456 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:19:10.456 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:19:10.456 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:19:10.457 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:19:10.457 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:19:10.457 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:19:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:11 smithi022 ceph-mon[40250]: pgmap v6020: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:19:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:11 smithi022 ceph-mon[31360]: pgmap v6020: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:19:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:11 smithi122 ceph-mon[32255]: pgmap v6020: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:19:12.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:19:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:19:12.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:12.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:12.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:13 smithi022 ceph-mon[31360]: pgmap v6021: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2940681714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:13 smithi022 ceph-mon[40250]: pgmap v6021: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2940681714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:13 smithi122 ceph-mon[32255]: pgmap v6021: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2940681714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:13.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:19:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:19:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:15 smithi022 ceph-mon[31360]: pgmap v6022: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:15 smithi022 ceph-mon[40250]: pgmap v6022: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:15 smithi122 ceph-mon[32255]: pgmap v6022: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:17.701 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:17 smithi022 ceph-mon[31360]: pgmap v6023: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:17.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:19:17.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:17 smithi022 ceph-mon[40250]: pgmap v6023: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:17 smithi122 ceph-mon[32255]: pgmap v6023: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:18.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:18.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:18.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:18 smithi022 ceph-mon[31360]: pgmap v6024: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1668618158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:18 smithi022 ceph-mon[40250]: pgmap v6024: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1668618158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:18 smithi122 ceph-mon[32255]: pgmap v6024: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1668618158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:20] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:19:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:20 smithi022 ceph-mon[31360]: pgmap v6025: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:20 smithi022 ceph-mon[40250]: pgmap v6025: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:20 smithi122 ceph-mon[32255]: pgmap v6025: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:19:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:19:23.069 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:22 smithi022 ceph-mon[31360]: pgmap v6026: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:22 smithi022 ceph-mon[40250]: pgmap v6026: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:22 smithi122 ceph-mon[32255]: pgmap v6026: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:23.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:23.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:23.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:23.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:19:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:19:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2866766682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2866766682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2866766682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:24 smithi022 ceph-mon[31360]: pgmap v6027: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:24 smithi022 ceph-mon[40250]: pgmap v6027: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:24 smithi122 ceph-mon[32255]: pgmap v6027: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:26 smithi022 ceph-mon[31360]: pgmap v6028: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:26 smithi022 ceph-mon[40250]: pgmap v6028: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:26 smithi122 ceph-mon[32255]: pgmap v6028: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:19:28.439 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:28.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:28.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:28.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:28 smithi022 ceph-mon[31360]: pgmap v6029: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2215642120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:29.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:28 smithi022 ceph-mon[40250]: pgmap v6029: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:29.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2215642120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:28 smithi122 ceph-mon[32255]: pgmap v6029: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2215642120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:30] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:19:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:30 smithi022 ceph-mon[31360]: pgmap v6030: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:30 smithi022 ceph-mon[40250]: pgmap v6030: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:30 smithi122 ceph-mon[32255]: pgmap v6030: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:19:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:19:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[31360]: pgmap v6031: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:19:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:19:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:19:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:19:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[40250]: pgmap v6031: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:19:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:19:33.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:19:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:19:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:33 smithi122 ceph-mon[32255]: pgmap v6031: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:19:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:19:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:19:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:19:33.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:33.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:19:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:19:34.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:34.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:34.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:34.180 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:34.180 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:35 smithi022 ceph-mon[31360]: pgmap v6032: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/769295194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:35 smithi022 ceph-mon[40250]: pgmap v6032: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/769295194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:35 smithi122 ceph-mon[32255]: pgmap v6032: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/769295194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:37 smithi022 ceph-mon[31360]: pgmap v6033: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:37 smithi022 ceph-mon[40250]: pgmap v6033: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:37 smithi122 ceph-mon[32255]: pgmap v6033: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:19:39.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:39 smithi022 ceph-mon[31360]: pgmap v6034: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:39.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:39 smithi022 ceph-mon[40250]: pgmap v6034: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:39 smithi122 ceph-mon[32255]: pgmap v6034: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:39.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:39.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:39.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3280369231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3280369231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:40] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:19:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3280369231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:41 smithi022 ceph-mon[31360]: pgmap v6035: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:41 smithi022 ceph-mon[40250]: pgmap v6035: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:41 smithi122 ceph-mon[32255]: pgmap v6035: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:19:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:19:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:43 smithi022 ceph-mon[31360]: pgmap v6036: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:43 smithi022 ceph-mon[40250]: pgmap v6036: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:43 smithi122 ceph-mon[32255]: pgmap v6036: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:43.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:19:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:19:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:44.551 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:44.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:45 smithi022 ceph-mon[31360]: pgmap v6037: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2445506768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:45 smithi022 ceph-mon[40250]: pgmap v6037: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2445506768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:45 smithi122 ceph-mon[32255]: pgmap v6037: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2445506768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:47 smithi022 ceph-mon[31360]: pgmap v6038: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:47 smithi022 ceph-mon[40250]: pgmap v6038: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:47 smithi122 ceph-mon[32255]: pgmap v6038: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:19:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:49 smithi122 ceph-mon[32255]: pgmap v6039: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:49 smithi022 ceph-mon[31360]: pgmap v6039: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:49 smithi022 ceph-mon[40250]: pgmap v6039: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:49.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:50.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:50.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:50.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:50] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:19:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:51 smithi122 ceph-mon[32255]: pgmap v6040: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1759850443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:51 smithi022 ceph-mon[31360]: pgmap v6040: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1759850443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:51 smithi022 ceph-mon[40250]: pgmap v6040: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1759850443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:19:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:19:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:53 smithi122 ceph-mon[32255]: pgmap v6041: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:53 smithi022 ceph-mon[31360]: pgmap v6041: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:53 smithi022 ceph-mon[40250]: pgmap v6041: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:53.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:19:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:19:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:19:55.297 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:19:55.298 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:55 smithi022 ceph-mon[40250]: pgmap v6042: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:55.299 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:55 smithi022 ceph-mon[31360]: pgmap v6042: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:55 smithi122 ceph-mon[32255]: pgmap v6042: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:19:55.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:19:55.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:19:55.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:19:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1167029492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1167029492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1167029492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:19:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:57 smithi022 ceph-mon[31360]: pgmap v6043: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:57 smithi022 ceph-mon[40250]: pgmap v6043: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:57 smithi122 ceph-mon[32255]: pgmap v6043: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:19:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:19:59 smithi122 ceph-mon[32255]: pgmap v6044: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:19:59 smithi022 ceph-mon[31360]: pgmap v6044: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:19:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:19:59 smithi022 ceph-mon[40250]: pgmap v6044: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:00] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:20:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T21:20:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T21:20:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T21:20:00.665 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:01.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:01.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:01.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:01 smithi122 ceph-mon[32255]: pgmap v6045: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/294052197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:01 smithi022 ceph-mon[31360]: pgmap v6045: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:01.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/294052197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:01 smithi022 ceph-mon[40250]: pgmap v6045: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/294052197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:20:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:20:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:03 smithi122 ceph-mon[32255]: pgmap v6046: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:03 smithi022 ceph-mon[31360]: pgmap v6046: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:03 smithi022 ceph-mon[40250]: pgmap v6046: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:03.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:20:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:20:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:05 smithi122 ceph-mon[32255]: pgmap v6047: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:05 smithi022 ceph-mon[31360]: pgmap v6047: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:05 smithi022 ceph-mon[40250]: pgmap v6047: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:06.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:06.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:06.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:06.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:07 smithi022 ceph-mon[31360]: pgmap v6048: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3830208328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:07 smithi022 ceph-mon[40250]: pgmap v6048: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3830208328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:07 smithi122 ceph-mon[32255]: pgmap v6048: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3830208328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:20:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:09 smithi122 ceph-mon[32255]: pgmap v6049: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:09 smithi022 ceph-mon[31360]: pgmap v6049: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:09 smithi022 ceph-mon[40250]: pgmap v6049: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:10] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:20:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:11 smithi122 ceph-mon[32255]: pgmap v6050: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:20:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:20:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:20:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:20:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[31360]: pgmap v6050: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:20:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:20:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:20:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:20:11.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[40250]: pgmap v6050: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:11.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:20:11.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:20:11.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:20:11.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:20:11.405 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:11.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:11.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:11.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:20:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:20:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/450223906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/450223906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/450223906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:13 smithi122 ceph-mon[32255]: pgmap v6051: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:13 smithi022 ceph-mon[31360]: pgmap v6051: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:13 smithi022 ceph-mon[40250]: pgmap v6051: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:13.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:20:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:20:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:15 smithi122 ceph-mon[32255]: pgmap v6052: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:15 smithi022 ceph-mon[31360]: pgmap v6052: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:15 smithi022 ceph-mon[40250]: pgmap v6052: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:16.768 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:17.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:17.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:17.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:17 smithi022 ceph-mon[31360]: pgmap v6053: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:17 smithi022 ceph-mon[40250]: pgmap v6053: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:17 smithi122 ceph-mon[32255]: pgmap v6053: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:20:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3824295195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3824295195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3824295195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:19 smithi122 ceph-mon[32255]: pgmap v6054: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:19 smithi022 ceph-mon[31360]: pgmap v6054: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:19 smithi022 ceph-mon[40250]: pgmap v6054: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:20] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:20:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:21 smithi122 ceph-mon[32255]: pgmap v6055: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:21 smithi022 ceph-mon[31360]: pgmap v6055: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:21 smithi022 ceph-mon[40250]: pgmap v6055: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:22.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:22.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:22.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:22.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:20:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:20:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:23 smithi122 ceph-mon[32255]: pgmap v6056: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4234952075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:23 smithi022 ceph-mon[31360]: pgmap v6056: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4234952075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:23 smithi022 ceph-mon[40250]: pgmap v6056: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4234952075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:23.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:20:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:20:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:25 smithi122 ceph-mon[32255]: pgmap v6057: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:25 smithi022 ceph-mon[31360]: pgmap v6057: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:25 smithi022 ceph-mon[40250]: pgmap v6057: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:27 smithi022 ceph-mon[31360]: pgmap v6058: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:27 smithi022 ceph-mon[40250]: pgmap v6058: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:27 smithi122 ceph-mon[32255]: pgmap v6058: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:27.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:20:27.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:27.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:27.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2330736170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2330736170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2330736170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:29 smithi122 ceph-mon[32255]: pgmap v6059: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:29 smithi022 ceph-mon[31360]: pgmap v6059: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:29 smithi022 ceph-mon[40250]: pgmap v6059: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:30] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:20:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:31 smithi122 ceph-mon[32255]: pgmap v6060: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:31 smithi022 ceph-mon[31360]: pgmap v6060: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:31 smithi022 ceph-mon[40250]: pgmap v6060: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:20:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:20:32.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:33.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:33.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:33.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:33 smithi122 ceph-mon[32255]: pgmap v6061: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:20:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:20:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:20:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:20:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[31360]: pgmap v6061: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:20:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:20:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[40250]: pgmap v6061: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:20:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:20:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:20:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:20:33.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:20:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:20:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3315830109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3315830109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3315830109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:35 smithi122 ceph-mon[32255]: pgmap v6062: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:35 smithi022 ceph-mon[31360]: pgmap v6062: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:35 smithi022 ceph-mon[40250]: pgmap v6062: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:37 smithi022 ceph-mon[31360]: pgmap v6063: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:37 smithi022 ceph-mon[40250]: pgmap v6063: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:37 smithi122 ceph-mon[32255]: pgmap v6063: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:20:38.247 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:38.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:38.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:38.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:39.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:39 smithi122 ceph-mon[32255]: pgmap v6064: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:39.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744304290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:39 smithi022 ceph-mon[31360]: pgmap v6064: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744304290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:39 smithi022 ceph-mon[40250]: pgmap v6064: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744304290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:40] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:20:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:41 smithi122 ceph-mon[32255]: pgmap v6065: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:41 smithi022 ceph-mon[31360]: pgmap v6065: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:41 smithi022 ceph-mon[40250]: pgmap v6065: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:20:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:20:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:43 smithi122 ceph-mon[32255]: pgmap v6066: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:43 smithi022 ceph-mon[31360]: pgmap v6066: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:43 smithi022 ceph-mon[40250]: pgmap v6066: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:43.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:43.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:20:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:20:43.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:43.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:43.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3018130673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3018130673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3018130673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:45 smithi122 ceph-mon[32255]: pgmap v6067: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:45 smithi022 ceph-mon[31360]: pgmap v6067: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:45 smithi022 ceph-mon[40250]: pgmap v6067: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:47 smithi122 ceph-mon[32255]: pgmap v6068: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:47 smithi022 ceph-mon[31360]: pgmap v6068: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:20:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:47 smithi022 ceph-mon[40250]: pgmap v6068: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:48.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:49 smithi022 ceph-mon[31360]: pgmap v6069: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:49 smithi022 ceph-mon[40250]: pgmap v6069: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:49.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:49.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:49.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:49 smithi122 ceph-mon[32255]: pgmap v6069: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1683738215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:50] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:20:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1683738215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1683738215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:51 smithi122 ceph-mon[32255]: pgmap v6070: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:51 smithi022 ceph-mon[31360]: pgmap v6070: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:51 smithi022 ceph-mon[40250]: pgmap v6070: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:20:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:20:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:53 smithi122 ceph-mon[32255]: pgmap v6071: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:53 smithi022 ceph-mon[31360]: pgmap v6071: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:53 smithi022 ceph-mon[40250]: pgmap v6071: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:53.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:20:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:20:54.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:20:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:20:54.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:20:54.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:20:54.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:20:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:55 smithi122 ceph-mon[32255]: pgmap v6072: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:55.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/411587054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:55 smithi022 ceph-mon[31360]: pgmap v6072: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/411587054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:55 smithi022 ceph-mon[40250]: pgmap v6072: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:20:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/411587054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:20:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:57 smithi122 ceph-mon[32255]: pgmap v6073: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:57 smithi022 ceph-mon[31360]: pgmap v6073: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:20:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:57 smithi022 ceph-mon[40250]: pgmap v6073: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:20:59 smithi122 ceph-mon[32255]: pgmap v6074: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:20:59 smithi022 ceph-mon[31360]: pgmap v6074: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:20:59 smithi022 ceph-mon[40250]: pgmap v6074: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:20:59.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:00.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:00.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:00.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:00] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:21:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3685922058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3685922058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3685922058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:01 smithi122 ceph-mon[32255]: pgmap v6075: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:01 smithi022 ceph-mon[31360]: pgmap v6075: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:01 smithi022 ceph-mon[40250]: pgmap v6075: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:21:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:21:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:03 smithi022 ceph-mon[31360]: pgmap v6076: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:03 smithi022 ceph-mon[40250]: pgmap v6076: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:03.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:03 smithi122 ceph-mon[32255]: pgmap v6076: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:03.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:21:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:21:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:05.095 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:05.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:05.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:05.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:05.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:05 smithi022 ceph-mon[31360]: pgmap v6077: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:05.465 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:05 smithi022 ceph-mon[40250]: pgmap v6077: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:05 smithi122 ceph-mon[32255]: pgmap v6077: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2819174366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:06.600 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2819174366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2819174366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:07 smithi022 ceph-mon[31360]: pgmap v6078: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:21:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:07 smithi022 ceph-mon[40250]: pgmap v6078: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:07 smithi122 ceph-mon[32255]: pgmap v6078: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:09 smithi022 ceph-mon[31360]: pgmap v6079: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:09 smithi022 ceph-mon[40250]: pgmap v6079: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:09 smithi122 ceph-mon[32255]: pgmap v6079: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:10] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:21:10.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:10.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:10.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:10.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:11.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:11 smithi122 ceph-mon[32255]: pgmap v6080: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:11.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3320538629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:11.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:21:11.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:21:11.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:21:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[31360]: pgmap v6080: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3320538629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:21:11.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:21:11.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:21:11.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[40250]: pgmap v6080: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:11.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3320538629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:11.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:21:11.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:21:11.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:21:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:21:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:21:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:21:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:21:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:21:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:13 smithi122 ceph-mon[32255]: pgmap v6081: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:13.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:21:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:21:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:13 smithi022 ceph-mon[31360]: pgmap v6081: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:13 smithi022 ceph-mon[40250]: pgmap v6081: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:15 smithi122 ceph-mon[32255]: pgmap v6082: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:15 smithi022 ceph-mon[31360]: pgmap v6082: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:15 smithi022 ceph-mon[40250]: pgmap v6082: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:16.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3008467883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3008467883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3008467883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:17 smithi122 ceph-mon[32255]: pgmap v6083: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:17 smithi022 ceph-mon[31360]: pgmap v6083: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:21:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:17 smithi022 ceph-mon[40250]: pgmap v6083: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:19 smithi122 ceph-mon[32255]: pgmap v6084: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:19 smithi022 ceph-mon[31360]: pgmap v6084: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:19 smithi022 ceph-mon[40250]: pgmap v6084: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:20] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:21:21.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:21 smithi022 ceph-mon[31360]: pgmap v6085: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:21 smithi022 ceph-mon[40250]: pgmap v6085: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:21.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:21.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:21.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:21 smithi122 ceph-mon[32255]: pgmap v6085: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:21:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:21:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3785139056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3785139056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3785139056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:23 smithi122 ceph-mon[32255]: pgmap v6086: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:23.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:21:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:21:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:23 smithi022 ceph-mon[31360]: pgmap v6086: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:23 smithi022 ceph-mon[40250]: pgmap v6086: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:25 smithi122 ceph-mon[32255]: pgmap v6087: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:25 smithi022 ceph-mon[31360]: pgmap v6087: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:25 smithi022 ceph-mon[40250]: pgmap v6087: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:26.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:26.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:26.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:27 smithi122 ceph-mon[32255]: pgmap v6088: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/753847964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:21:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:27 smithi022 ceph-mon[31360]: pgmap v6088: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:27.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/753847964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:27 smithi022 ceph-mon[40250]: pgmap v6088: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/753847964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:29 smithi122 ceph-mon[32255]: pgmap v6089: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:29 smithi022 ceph-mon[31360]: pgmap v6089: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:29 smithi022 ceph-mon[40250]: pgmap v6089: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:30] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:21:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:31 smithi122 ceph-mon[32255]: pgmap v6090: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:31 smithi022 ceph-mon[31360]: pgmap v6090: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:31 smithi022 ceph-mon[40250]: pgmap v6090: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:31.949 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:32.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:32.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:32.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/795487102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:21:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:21:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/795487102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/795487102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[31360]: pgmap v6091: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:21:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:21:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:21:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:21:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[40250]: pgmap v6091: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:21:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:21:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:21:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:21:33.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:21:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:21:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:33 smithi122 ceph-mon[32255]: pgmap v6091: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:21:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:21:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:21:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:21:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:35 smithi022 ceph-mon[31360]: pgmap v6092: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:35 smithi022 ceph-mon[40250]: pgmap v6092: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:35 smithi122 ceph-mon[32255]: pgmap v6092: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:37.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:37.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:37.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:37.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:37.689 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:21:37.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:37 smithi022 ceph-mon[31360]: pgmap v6093: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:37.689 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:37 smithi022 ceph-mon[40250]: pgmap v6093: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:37 smithi122 ceph-mon[32255]: pgmap v6093: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/136975775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/136975775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/136975775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:39 smithi022 ceph-mon[31360]: pgmap v6094: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:39 smithi022 ceph-mon[40250]: pgmap v6094: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:39 smithi122 ceph-mon[32255]: pgmap v6094: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:40] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:21:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:41 smithi022 ceph-mon[31360]: pgmap v6095: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:41 smithi022 ceph-mon[40250]: pgmap v6095: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:41.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:41 smithi122 ceph-mon[32255]: pgmap v6095: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:21:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:21:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:43.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:43.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:43.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:43 smithi022 ceph-mon[31360]: pgmap v6096: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3279563047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:43 smithi022 ceph-mon[40250]: pgmap v6096: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3279563047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:43.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:21:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:21:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:43 smithi122 ceph-mon[32255]: pgmap v6096: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3279563047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:45 smithi022 ceph-mon[31360]: pgmap v6097: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:45 smithi022 ceph-mon[40250]: pgmap v6097: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:45 smithi122 ceph-mon[32255]: pgmap v6097: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:47 smithi022 ceph-mon[31360]: pgmap v6098: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:21:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:47 smithi022 ceph-mon[40250]: pgmap v6098: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:47 smithi122 ceph-mon[32255]: pgmap v6098: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:48.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:48.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:48.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:48.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:49 smithi022 ceph-mon[31360]: pgmap v6099: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1235931210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:49 smithi022 ceph-mon[40250]: pgmap v6099: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1235931210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:49 smithi122 ceph-mon[32255]: pgmap v6099: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1235931210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:50] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:21:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:51 smithi022 ceph-mon[31360]: pgmap v6100: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:51 smithi022 ceph-mon[40250]: pgmap v6100: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:51 smithi122 ceph-mon[32255]: pgmap v6100: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:21:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:21:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:53 smithi022 ceph-mon[31360]: pgmap v6101: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:53 smithi022 ceph-mon[40250]: pgmap v6101: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:53.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:53 smithi122 ceph-mon[32255]: pgmap v6101: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:53.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:53.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:53.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:53.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:21:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:21:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2703691429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2703691429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2703691429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:21:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:55 smithi022 ceph-mon[31360]: pgmap v6102: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:55 smithi022 ceph-mon[40250]: pgmap v6102: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:21:55.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:55 smithi122 ceph-mon[32255]: pgmap v6102: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:21:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:57 smithi022 ceph-mon[31360]: pgmap v6103: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:57 smithi022 ceph-mon[40250]: pgmap v6103: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:57 smithi122 ceph-mon[32255]: pgmap v6103: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:21:58.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:21:59.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:21:59.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:21:59.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:21:59.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:21:59 smithi022 ceph-mon[31360]: pgmap v6104: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:59.173 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:21:59 smithi022 ceph-mon[40250]: pgmap v6104: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:21:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:21:59 smithi122 ceph-mon[32255]: pgmap v6104: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1231075441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:00] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:22:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1231075441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1231075441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:01 smithi122 ceph-mon[32255]: pgmap v6105: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:01 smithi022 ceph-mon[31360]: pgmap v6105: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:01 smithi022 ceph-mon[40250]: pgmap v6105: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:22:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:22:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:03 smithi122 ceph-mon[32255]: pgmap v6106: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:03 smithi022 ceph-mon[31360]: pgmap v6106: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:03 smithi022 ceph-mon[40250]: pgmap v6106: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:03.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:22:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:22:04.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:04.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:04.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:04.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:05 smithi122 ceph-mon[32255]: pgmap v6107: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2786914674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:05 smithi022 ceph-mon[31360]: pgmap v6107: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2786914674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:05 smithi022 ceph-mon[40250]: pgmap v6107: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2786914674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:07 smithi022 ceph-mon[31360]: pgmap v6108: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:07 smithi022 ceph-mon[40250]: pgmap v6108: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:07 smithi122 ceph-mon[32255]: pgmap v6108: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:22:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:09 smithi122 ceph-mon[32255]: pgmap v6109: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:09.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:09 smithi022 ceph-mon[31360]: pgmap v6109: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:09 smithi022 ceph-mon[40250]: pgmap v6109: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:09.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:09.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/27325601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/27325601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/27325601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:10] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:22:11.429 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:11 smithi022 ceph-mon[31360]: pgmap v6110: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:11.429 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:11 smithi022 ceph-mon[40250]: pgmap v6110: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:11.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:11 smithi122 ceph-mon[32255]: pgmap v6110: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:22:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:22:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:22:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:22:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:22:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:22:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:22:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:22:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:22:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:22:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:22:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:13 smithi122 ceph-mon[32255]: pgmap v6111: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:13 smithi022 ceph-mon[31360]: pgmap v6111: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:13 smithi022 ceph-mon[40250]: pgmap v6111: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:13.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:22:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:22:14.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:15 smithi022 ceph-mon[31360]: pgmap v6112: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:15 smithi022 ceph-mon[40250]: pgmap v6112: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:15.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:15.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:15.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:15 smithi122 ceph-mon[32255]: pgmap v6112: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2366689090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3258M 2021-12-15T21:22:16.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:16.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2366689090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3258M 2021-12-15T21:22:16.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:16.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:16.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2366689090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:16.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:22:16.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3258M 2021-12-15T21:22:16.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:16.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:17 smithi022 ceph-mon[31360]: pgmap v6113: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:22:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:17 smithi022 ceph-mon[40250]: pgmap v6113: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:17 smithi122 ceph-mon[32255]: pgmap v6113: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:22:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:19 smithi022 ceph-mon[31360]: pgmap v6114: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:19 smithi022 ceph-mon[40250]: pgmap v6114: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:19 smithi122 ceph-mon[32255]: pgmap v6114: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:20.290 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:20] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:22:20.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:20.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:20.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:21 smithi022 ceph-mon[31360]: pgmap v6115: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/307074737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:21 smithi022 ceph-mon[40250]: pgmap v6115: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/307074737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:21 smithi122 ceph-mon[32255]: pgmap v6115: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/307074737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:22:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:22:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:23 smithi022 ceph-mon[31360]: pgmap v6116: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:23 smithi022 ceph-mon[40250]: pgmap v6116: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:23.645 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:23 smithi122 ceph-mon[32255]: pgmap v6116: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:23.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:22:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:22:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:25 smithi022 ceph-mon[31360]: pgmap v6117: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:25.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:25.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:25 smithi022 ceph-mon[40250]: pgmap v6117: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:25.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:25 smithi122 ceph-mon[32255]: pgmap v6117: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:26.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:26.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:26.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1834196633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1834196633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1834196633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:27 smithi022 ceph-mon[31360]: pgmap v6118: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:22:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:27 smithi022 ceph-mon[40250]: pgmap v6118: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:27 smithi122 ceph-mon[32255]: pgmap v6118: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:29 smithi022 ceph-mon[31360]: pgmap v6119: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:29 smithi022 ceph-mon[40250]: pgmap v6119: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:29 smithi122 ceph-mon[32255]: pgmap v6119: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:30] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:22:31.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:31.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:31.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:31.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:31.409 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:31 smithi022 ceph-mon[31360]: pgmap v6120: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:31.410 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:31 smithi022 ceph-mon[40250]: pgmap v6120: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:31 smithi122 ceph-mon[32255]: pgmap v6120: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2682296980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2682296980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2682296980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:22:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:22:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[31360]: pgmap v6121: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:22:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:22:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:22:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:22:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[40250]: pgmap v6121: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:22:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:22:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:22:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:22:33.656 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:33 smithi122 ceph-mon[32255]: pgmap v6121: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:33.656 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:22:33.656 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:22:33.656 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:22:33.657 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:22:33.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:22:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:22:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:35 smithi022 ceph-mon[31360]: pgmap v6122: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:35 smithi022 ceph-mon[40250]: pgmap v6122: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:35 smithi122 ceph-mon[32255]: pgmap v6122: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:36.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:36.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:36.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:37 smithi122 ceph-mon[32255]: pgmap v6123: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2311231020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:22:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:37 smithi022 ceph-mon[31360]: pgmap v6123: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2311231020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:37 smithi022 ceph-mon[40250]: pgmap v6123: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2311231020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:39 smithi122 ceph-mon[32255]: pgmap v6124: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:39 smithi022 ceph-mon[31360]: pgmap v6124: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:39 smithi022 ceph-mon[40250]: pgmap v6124: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:40] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:22:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:41 smithi122 ceph-mon[32255]: pgmap v6125: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:41 smithi022 ceph-mon[31360]: pgmap v6125: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:41 smithi022 ceph-mon[40250]: pgmap v6125: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:41.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:42.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:42.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:42.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4114816298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4114816298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4114816298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:22:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:22:43.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:43 smithi122 ceph-mon[32255]: pgmap v6126: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:43 smithi022 ceph-mon[31360]: pgmap v6126: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:43 smithi022 ceph-mon[40250]: pgmap v6126: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:43.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:22:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:22:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:45 smithi122 ceph-mon[32255]: pgmap v6127: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:45 smithi022 ceph-mon[31360]: pgmap v6127: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:45 smithi022 ceph-mon[40250]: pgmap v6127: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:47.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:47.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:47.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:47.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:47.517 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:22:47.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:47 smithi022 ceph-mon[31360]: pgmap v6128: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:47.519 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:47 smithi022 ceph-mon[40250]: pgmap v6128: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:47 smithi122 ceph-mon[32255]: pgmap v6128: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3713739281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3713739281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3713739281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:49 smithi122 ceph-mon[32255]: pgmap v6129: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:49 smithi022 ceph-mon[31360]: pgmap v6129: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:49 smithi022 ceph-mon[40250]: pgmap v6129: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:50] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:22:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:51 smithi122 ceph-mon[32255]: pgmap v6130: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:51 smithi022 ceph-mon[31360]: pgmap v6130: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:51 smithi022 ceph-mon[40250]: pgmap v6130: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:52.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:22:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:22:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:53.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:53 smithi122 ceph-mon[32255]: pgmap v6131: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:53.678 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2090873198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:53 smithi022 ceph-mon[31360]: pgmap v6131: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2090873198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:53 smithi022 ceph-mon[40250]: pgmap v6131: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2090873198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:53.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:22:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:22:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:55 smithi122 ceph-mon[32255]: pgmap v6132: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:55 smithi022 ceph-mon[31360]: pgmap v6132: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:22:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:55 smithi022 ceph-mon[40250]: pgmap v6132: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:22:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:57 smithi122 ceph-mon[32255]: pgmap v6133: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:57 smithi022 ceph-mon[31360]: pgmap v6133: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:22:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:57 smithi022 ceph-mon[40250]: pgmap v6133: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:57.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:22:58.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:22:58.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:22:58.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:22:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:59 smithi022 ceph-mon[40250]: pgmap v6134: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:22:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/598369551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:59 smithi022 ceph-mon[31360]: pgmap v6134: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:22:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/598369551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:22:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:59 smithi122 ceph-mon[32255]: pgmap v6134: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:22:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:22:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/598369551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:00] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:23:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:01 smithi022 ceph-mon[31360]: pgmap v6135: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:01 smithi022 ceph-mon[40250]: pgmap v6135: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:01 smithi122 ceph-mon[32255]: pgmap v6135: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:23:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:23:03.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:03 smithi122 ceph-mon[32255]: pgmap v6136: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:03 smithi022 ceph-mon[31360]: pgmap v6136: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:03 smithi022 ceph-mon[40250]: pgmap v6136: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:03.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:03.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:03.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:04.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:23:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:23:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3492951529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3492951529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3492951529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:05 smithi122 ceph-mon[32255]: pgmap v6137: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:05 smithi022 ceph-mon[31360]: pgmap v6137: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:05 smithi022 ceph-mon[40250]: pgmap v6137: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:07 smithi022 ceph-mon[31360]: pgmap v6138: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:07 smithi022 ceph-mon[40250]: pgmap v6138: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:07 smithi122 ceph-mon[32255]: pgmap v6138: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:23:08.631 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:08.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:08.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:08.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:09 smithi022 ceph-mon[31360]: pgmap v6139: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1635340386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:09.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:09 smithi022 ceph-mon[40250]: pgmap v6139: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:09.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1635340386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:09 smithi122 ceph-mon[32255]: pgmap v6139: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1635340386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:10] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:23:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:11 smithi122 ceph-mon[32255]: pgmap v6140: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:11 smithi022 ceph-mon[31360]: pgmap v6140: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:11 smithi022 ceph-mon[40250]: pgmap v6140: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:23:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:23:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:13 smithi122 ceph-mon[32255]: pgmap v6141: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:13 smithi022 ceph-mon[31360]: pgmap v6141: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:13 smithi022 ceph-mon[40250]: pgmap v6141: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:14.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:23:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:23:14.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:14.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:14.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:15 smithi122 ceph-mon[32255]: pgmap v6142: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/734458793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:15 smithi022 ceph-mon[31360]: pgmap v6142: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/734458793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:15 smithi022 ceph-mon[40250]: pgmap v6142: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/734458793' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[40250]: pgmap v6143: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:23:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:23:17.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:23:17.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:23:17.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[31360]: pgmap v6143: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:17.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:23:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:23:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:23:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:23:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:17 smithi122 ceph-mon[32255]: pgmap v6143: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:23:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:23:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:23:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:23:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:23:19.374 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:19.374 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:19 smithi022 ceph-mon[31360]: pgmap v6144: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:19.375 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:19 smithi022 ceph-mon[40250]: pgmap v6144: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:19 smithi122 ceph-mon[32255]: pgmap v6144: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:19.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:19.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:19.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2469443591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:20] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2469443591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2469443591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:21 smithi122 ceph-mon[32255]: pgmap v6145: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:21 smithi022 ceph-mon[31360]: pgmap v6145: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:21 smithi022 ceph-mon[40250]: pgmap v6145: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:23:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:23:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:23 smithi122 ceph-mon[32255]: pgmap v6146: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:23 smithi022 ceph-mon[31360]: pgmap v6146: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:23 smithi022 ceph-mon[40250]: pgmap v6146: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:24.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:23:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:23:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:25.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:25.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:25 smithi122 ceph-mon[32255]: pgmap v6147: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:25 smithi022 ceph-mon[31360]: pgmap v6147: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:25 smithi022 ceph-mon[40250]: pgmap v6147: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2619838202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2619838202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2619838202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:27 smithi022 ceph-mon[31360]: pgmap v6148: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:27 smithi022 ceph-mon[40250]: pgmap v6148: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:27 smithi122 ceph-mon[32255]: pgmap v6148: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:23:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:29 smithi122 ceph-mon[32255]: pgmap v6149: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:29 smithi022 ceph-mon[31360]: pgmap v6149: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:29 smithi022 ceph-mon[40250]: pgmap v6149: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:30.113 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:30] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:23:30.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:30.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:30.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:31 smithi122 ceph-mon[32255]: pgmap v6150: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2433255375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:31 smithi022 ceph-mon[31360]: pgmap v6150: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2433255375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:31 smithi022 ceph-mon[40250]: pgmap v6150: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2433255375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:23:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:23:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:33 smithi122 ceph-mon[32255]: pgmap v6151: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:23:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:23:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:23:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:23:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[31360]: pgmap v6151: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:23:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:23:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:23:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:23:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[40250]: pgmap v6151: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:23:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:23:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:23:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:23:34.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:23:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:23:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:35 smithi122 ceph-mon[32255]: pgmap v6152: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:35.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:35.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:35 smithi022 ceph-mon[40250]: pgmap v6152: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:35.483 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:35.484 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:35 smithi022 ceph-mon[31360]: pgmap v6152: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:35.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:35.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:35.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4283177734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4283177734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4283177734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:37 smithi022 ceph-mon[31360]: pgmap v6153: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:37 smithi022 ceph-mon[40250]: pgmap v6153: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:37 smithi122 ceph-mon[32255]: pgmap v6153: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:23:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:39 smithi122 ceph-mon[32255]: pgmap v6154: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:39 smithi022 ceph-mon[31360]: pgmap v6154: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:39 smithi022 ceph-mon[40250]: pgmap v6154: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:40] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:23:40.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:41.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:41.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:41.215 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:41 smithi022 ceph-mon[31360]: pgmap v6155: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:41.215 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:41 smithi022 ceph-mon[40250]: pgmap v6155: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:41 smithi122 ceph-mon[32255]: pgmap v6155: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1954854658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:23:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:23:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1954854658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1954854658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:43 smithi122 ceph-mon[32255]: pgmap v6156: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:43 smithi022 ceph-mon[31360]: pgmap v6156: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:43 smithi022 ceph-mon[40250]: pgmap v6156: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:23:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:23:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:45 smithi122 ceph-mon[32255]: pgmap v6157: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:45 smithi022 ceph-mon[31360]: pgmap v6157: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:45 smithi022 ceph-mon[40250]: pgmap v6157: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:46.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:46.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:46.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:46.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:47 smithi022 ceph-mon[31360]: pgmap v6158: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1253813159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:47 smithi022 ceph-mon[40250]: pgmap v6158: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1253813159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:47 smithi122 ceph-mon[32255]: pgmap v6158: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1253813159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:23:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:49 smithi122 ceph-mon[32255]: pgmap v6159: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:49 smithi022 ceph-mon[31360]: pgmap v6159: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:49 smithi022 ceph-mon[40250]: pgmap v6159: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:50] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:23:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:51 smithi122 ceph-mon[32255]: pgmap v6160: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:51 smithi022 ceph-mon[31360]: pgmap v6160: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:51 smithi022 ceph-mon[40250]: pgmap v6160: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:51.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:51.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:51.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:51.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3251862118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3251862118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3251862118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:23:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:23:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:53 smithi122 ceph-mon[32255]: pgmap v6161: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:53 smithi022 ceph-mon[31360]: pgmap v6161: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:53 smithi022 ceph-mon[40250]: pgmap v6161: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:54.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:23:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:23:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:55.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:55 smithi122 ceph-mon[32255]: pgmap v6162: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:55 smithi022 ceph-mon[31360]: pgmap v6162: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:23:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:55 smithi022 ceph-mon[40250]: pgmap v6162: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:23:56.953 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:23:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:57 smithi022 ceph-mon[31360]: pgmap v6163: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:57 smithi022 ceph-mon[40250]: pgmap v6163: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:57.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:23:57.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:23:57.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:23:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:57 smithi122 ceph-mon[32255]: pgmap v6163: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:23:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1393629398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1393629398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1393629398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:23:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:23:59 smithi122 ceph-mon[32255]: pgmap v6164: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:23:59 smithi022 ceph-mon[31360]: pgmap v6164: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:23:59.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:23:59 smithi022 ceph-mon[40250]: pgmap v6164: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:00] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:24:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:01 smithi122 ceph-mon[32255]: pgmap v6165: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:01 smithi022 ceph-mon[31360]: pgmap v6165: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:01 smithi022 ceph-mon[40250]: pgmap v6165: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:02.324 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:24:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:24:02.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:02.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:02.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:03 smithi122 ceph-mon[32255]: pgmap v6166: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/810603752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:03 smithi022 ceph-mon[31360]: pgmap v6166: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/810603752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:03.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:03 smithi022 ceph-mon[40250]: pgmap v6166: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:03.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/810603752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:04.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:24:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:24:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:05 smithi122 ceph-mon[32255]: pgmap v6167: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:05 smithi022 ceph-mon[31360]: pgmap v6167: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:05 smithi022 ceph-mon[40250]: pgmap v6167: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:07 smithi122 ceph-mon[32255]: pgmap v6168: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:07 smithi022 ceph-mon[31360]: pgmap v6168: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:24:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:07 smithi022 ceph-mon[40250]: pgmap v6168: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:07.692 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:08.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:08.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:08.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:09 smithi122 ceph-mon[32255]: pgmap v6169: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1558824444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:09 smithi022 ceph-mon[31360]: pgmap v6169: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:09.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1558824444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:09 smithi022 ceph-mon[40250]: pgmap v6169: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:09.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1558824444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:10] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:24:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:11 smithi122 ceph-mon[32255]: pgmap v6170: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:11.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:11 smithi022 ceph-mon[31360]: pgmap v6170: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:11.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:11 smithi022 ceph-mon[40250]: pgmap v6170: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:24:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:24:13.066 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:13.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:13.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:13.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:13.436 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:13 smithi022 ceph-mon[31360]: pgmap v6171: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:13.437 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:13 smithi022 ceph-mon[40250]: pgmap v6171: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:13 smithi122 ceph-mon[32255]: pgmap v6171: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:14.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:24:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:24:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/42859044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/42859044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/42859044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:15 smithi122 ceph-mon[32255]: pgmap v6172: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:15 smithi022 ceph-mon[31360]: pgmap v6172: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:15 smithi022 ceph-mon[40250]: pgmap v6172: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[31360]: pgmap v6173: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:24:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:24:17.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:24:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[40250]: pgmap v6173: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:24:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:24:17.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:24:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:17 smithi122 ceph-mon[32255]: pgmap v6173: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:24:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:24:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:24:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:24:18.438 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:18.438 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:24:18.439 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:24:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:24:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:19.188 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:19 smithi022 ceph-mon[40250]: pgmap v6174: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:19.189 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4091563483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:19 smithi022 ceph-mon[31360]: pgmap v6174: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4091563483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:19 smithi122 ceph-mon[32255]: pgmap v6174: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:24:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4091563483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:20] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:24:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:21 smithi022 ceph-mon[31360]: pgmap v6175: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:21 smithi022 ceph-mon[40250]: pgmap v6175: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:21 smithi122 ceph-mon[32255]: pgmap v6175: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:24:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:24:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:23 smithi022 ceph-mon[31360]: pgmap v6176: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:23 smithi022 ceph-mon[40250]: pgmap v6176: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:23 smithi122 ceph-mon[32255]: pgmap v6176: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:23.806 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:24.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:24.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:24.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:24.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:24:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:24:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3657358354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3657358354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3657358354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:25 smithi022 ceph-mon[31360]: pgmap v6177: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:25 smithi022 ceph-mon[40250]: pgmap v6177: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:25 smithi122 ceph-mon[32255]: pgmap v6177: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:27 smithi022 ceph-mon[31360]: pgmap v6178: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:24:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:27 smithi022 ceph-mon[40250]: pgmap v6178: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:27 smithi122 ceph-mon[32255]: pgmap v6178: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:29.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:29 smithi022 ceph-mon[31360]: pgmap v6179: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:29 smithi022 ceph-mon[40250]: pgmap v6179: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:29.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:29.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:29.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:29 smithi122 ceph-mon[32255]: pgmap v6179: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/385465133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:30] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:24:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/385465133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/385465133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:31 smithi022 ceph-mon[31360]: pgmap v6180: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:31 smithi022 ceph-mon[40250]: pgmap v6180: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:31 smithi122 ceph-mon[32255]: pgmap v6180: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:24:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:24:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[31360]: pgmap v6181: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:24:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:24:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:24:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:24:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[40250]: pgmap v6181: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:24:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:24:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:24:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:24:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:33 smithi122 ceph-mon[32255]: pgmap v6181: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:24:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:24:34.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:24:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:24:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:34.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:34.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:34.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:35 smithi022 ceph-mon[31360]: pgmap v6182: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3730222153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:35 smithi022 ceph-mon[40250]: pgmap v6182: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3730222153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:35 smithi122 ceph-mon[32255]: pgmap v6182: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3730222153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:37 smithi022 ceph-mon[31360]: pgmap v6183: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:24:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:37 smithi022 ceph-mon[40250]: pgmap v6183: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:37 smithi122 ceph-mon[32255]: pgmap v6183: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:39 smithi022 ceph-mon[31360]: pgmap v6184: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:39 smithi022 ceph-mon[40250]: pgmap v6184: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:39 smithi122 ceph-mon[32255]: pgmap v6184: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:39.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:40.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:40.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:40.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:40] "GET /metrics HTTP/1.1" 200 202371 "" "Prometheus/2.18.1" 2021-12-15T21:24:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:41 smithi022 ceph-mon[31360]: pgmap v6185: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:42.685 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1078953079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:42.686 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:41 smithi022 ceph-mon[40250]: pgmap v6185: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:42.686 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1078953079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:41 smithi122 ceph-mon[32255]: pgmap v6185: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1078953079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:24:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:24:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:43 smithi022 ceph-mon[31360]: pgmap v6186: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:43 smithi022 ceph-mon[40250]: pgmap v6186: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:43 smithi122 ceph-mon[32255]: pgmap v6186: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:24:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:24:45.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:45 smithi022 ceph-mon[31360]: pgmap v6187: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:45 smithi022 ceph-mon[40250]: pgmap v6187: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:45.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:45.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:45.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:45 smithi122 ceph-mon[32255]: pgmap v6187: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3934658916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3934658916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3934658916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:47 smithi022 ceph-mon[31360]: pgmap v6188: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:24:47.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:47 smithi022 ceph-mon[40250]: pgmap v6188: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:47 smithi122 ceph-mon[32255]: pgmap v6188: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:49 smithi022 ceph-mon[40250]: pgmap v6189: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:49.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:49 smithi022 ceph-mon[31360]: pgmap v6189: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:49 smithi122 ceph-mon[32255]: pgmap v6189: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:50] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:24:50.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:51.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:51.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:51.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:51 smithi022 ceph-mon[31360]: pgmap v6190: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4250712325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:51 smithi022 ceph-mon[40250]: pgmap v6190: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4250712325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:51 smithi122 ceph-mon[32255]: pgmap v6190: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4250712325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:24:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:24:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:53 smithi022 ceph-mon[31360]: pgmap v6191: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:53 smithi022 ceph-mon[40250]: pgmap v6191: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:53 smithi122 ceph-mon[32255]: pgmap v6191: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:54.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:24:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:24:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:55 smithi022 ceph-mon[31360]: pgmap v6192: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:55 smithi022 ceph-mon[40250]: pgmap v6192: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:24:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:55 smithi122 ceph-mon[32255]: pgmap v6192: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:24:56.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:24:56.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:24:56.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:24:56.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:24:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:24:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:57 smithi022 ceph-mon[31360]: pgmap v6193: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:57.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1328564328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:57 smithi022 ceph-mon[40250]: pgmap v6193: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1328564328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:57 smithi122 ceph-mon[32255]: pgmap v6193: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1328564328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:24:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:24:59 smithi022 ceph-mon[31360]: pgmap v6194: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:59.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:24:59 smithi022 ceph-mon[40250]: pgmap v6194: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:24:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:24:59 smithi122 ceph-mon[32255]: pgmap v6194: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:00] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:25:01.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:01 smithi022 ceph-mon[31360]: pgmap v6195: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:01 smithi022 ceph-mon[40250]: pgmap v6195: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:01 smithi122 ceph-mon[32255]: pgmap v6195: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/908766090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/908766090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/908766090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:25:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:25:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:03 smithi022 ceph-mon[31360]: pgmap v6196: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:03 smithi022 ceph-mon[40250]: pgmap v6196: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:03 smithi122 ceph-mon[32255]: pgmap v6196: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:04.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:25:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:25:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:05 smithi022 ceph-mon[31360]: pgmap v6197: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:05 smithi022 ceph-mon[40250]: pgmap v6197: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:05 smithi122 ceph-mon[32255]: pgmap v6197: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:06.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:07.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:07.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:07.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:07 smithi022 ceph-mon[31360]: pgmap v6198: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3090315410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:07 smithi022 ceph-mon[40250]: pgmap v6198: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3090315410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:07 smithi122 ceph-mon[32255]: pgmap v6198: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3090315410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:25:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:09 smithi022 ceph-mon[31360]: pgmap v6199: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:09 smithi022 ceph-mon[40250]: pgmap v6199: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:09 smithi122 ceph-mon[32255]: pgmap v6199: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:10] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:25:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:11 smithi022 ceph-mon[31360]: pgmap v6200: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:11 smithi022 ceph-mon[40250]: pgmap v6200: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:11 smithi122 ceph-mon[32255]: pgmap v6200: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:12.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:12.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:12.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:12.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:25:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:25:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:13 smithi022 ceph-mon[31360]: pgmap v6201: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2461621083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:13 smithi022 ceph-mon[40250]: pgmap v6201: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:13.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2461621083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:13 smithi122 ceph-mon[32255]: pgmap v6201: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2461621083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:14.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:25:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:25:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:15 smithi022 ceph-mon[40250]: pgmap v6202: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:15.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:15.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:15 smithi022 ceph-mon[31360]: pgmap v6202: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:15 smithi122 ceph-mon[32255]: pgmap v6202: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:17.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:17.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[31360]: pgmap v6203: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:17.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:25:17.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:25:17.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:25:17.507 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:25:17.507 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[40250]: pgmap v6203: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:17.508 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:25:17.508 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:25:17.508 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:25:17.643 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:17 smithi122 ceph-mon[32255]: pgmap v6203: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:17.644 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:25:17.644 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:25:17.644 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:25:17.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:17.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:17.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:25:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:18 smithi122 ceph-mon[32255]: pgmap v6204: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2629320382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:25:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:18 smithi022 ceph-mon[31360]: pgmap v6204: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2629320382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:25:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:18 smithi022 ceph-mon[40250]: pgmap v6204: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2629320382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:20] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:25:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:20 smithi122 ceph-mon[32255]: pgmap v6205: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:20 smithi022 ceph-mon[31360]: pgmap v6205: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:20 smithi022 ceph-mon[40250]: pgmap v6205: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:25:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:25:22.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:22 smithi122 ceph-mon[32255]: pgmap v6206: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:23.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:23.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:23.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:22 smithi022 ceph-mon[31360]: pgmap v6206: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:23.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:22 smithi022 ceph-mon[40250]: pgmap v6206: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:24.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:25:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:25:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/952172986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/952172986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/952172986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:25 smithi122 ceph-mon[32255]: pgmap v6207: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:25 smithi022 ceph-mon[31360]: pgmap v6207: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:25 smithi022 ceph-mon[40250]: pgmap v6207: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:27.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:27 smithi022 ceph-mon[31360]: pgmap v6208: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:27 smithi022 ceph-mon[40250]: pgmap v6208: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:27 smithi122 ceph-mon[32255]: pgmap v6208: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:25:28.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:28.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:28.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:28.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:29 smithi122 ceph-mon[32255]: pgmap v6209: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/621496615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:29 smithi022 ceph-mon[31360]: pgmap v6209: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/621496615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:29 smithi022 ceph-mon[40250]: pgmap v6209: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/621496615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:31.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:30] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:25:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:31 smithi122 ceph-mon[32255]: pgmap v6210: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:31 smithi022 ceph-mon[31360]: pgmap v6210: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:31 smithi022 ceph-mon[40250]: pgmap v6210: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:25:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:25:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:33 smithi122 ceph-mon[32255]: pgmap v6211: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:25:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:25:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:25:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:25:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[31360]: pgmap v6211: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:25:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:25:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:25:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:25:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[40250]: pgmap v6211: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:25:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:25:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:25:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:25:33.604 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:33.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:33.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:33.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:34.084 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:25:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:25:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3926532883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3926532883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3926532883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:35 smithi122 ceph-mon[32255]: pgmap v6212: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:35 smithi022 ceph-mon[31360]: pgmap v6212: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:35 smithi022 ceph-mon[40250]: pgmap v6212: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:37 smithi022 ceph-mon[31360]: pgmap v6213: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:37 smithi022 ceph-mon[40250]: pgmap v6213: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:37 smithi122 ceph-mon[32255]: pgmap v6213: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:25:38.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:39 smithi022 ceph-mon[31360]: pgmap v6214: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:39 smithi022 ceph-mon[40250]: pgmap v6214: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:39.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:39.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:39.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:39 smithi122 ceph-mon[32255]: pgmap v6214: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2864026834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:40] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:25:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2864026834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2864026834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:41 smithi122 ceph-mon[32255]: pgmap v6215: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:41 smithi022 ceph-mon[31360]: pgmap v6215: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:41 smithi022 ceph-mon[40250]: pgmap v6215: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:25:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:25:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:43 smithi122 ceph-mon[32255]: pgmap v6216: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:43 smithi022 ceph-mon[31360]: pgmap v6216: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:43 smithi022 ceph-mon[40250]: pgmap v6216: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:25:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:25:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:44.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:44.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:44.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:45 smithi122 ceph-mon[32255]: pgmap v6217: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1208627440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:45 smithi022 ceph-mon[31360]: pgmap v6217: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1208627440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:45 smithi022 ceph-mon[40250]: pgmap v6217: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1208627440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:47 smithi022 ceph-mon[31360]: pgmap v6218: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:47 smithi022 ceph-mon[40250]: pgmap v6218: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:47 smithi122 ceph-mon[32255]: pgmap v6218: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:25:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:49 smithi122 ceph-mon[32255]: pgmap v6219: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:49 smithi022 ceph-mon[31360]: pgmap v6219: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:49 smithi022 ceph-mon[40250]: pgmap v6219: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:49.719 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:50.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:50.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:50.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:50] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:25:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2973959186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2973959186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2973959186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:51 smithi122 ceph-mon[32255]: pgmap v6220: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:51 smithi022 ceph-mon[31360]: pgmap v6220: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:51 smithi022 ceph-mon[40250]: pgmap v6220: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:25:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:25:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:53 smithi122 ceph-mon[32255]: pgmap v6221: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:53 smithi022 ceph-mon[31360]: pgmap v6221: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:53 smithi022 ceph-mon[40250]: pgmap v6221: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:54.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:25:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:25:55.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:25:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:55 smithi122 ceph-mon[32255]: pgmap v6222: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:55.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:55.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:25:55.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:25:55.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:25:55.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:55 smithi022 ceph-mon[40250]: pgmap v6222: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:55.460 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:55.460 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:55 smithi022 ceph-mon[31360]: pgmap v6222: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:25:55.460 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:25:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2283726143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2283726143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2283726143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:25:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:57 smithi022 ceph-mon[31360]: pgmap v6223: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:57 smithi022 ceph-mon[40250]: pgmap v6223: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:57 smithi122 ceph-mon[32255]: pgmap v6223: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:25:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:25:59 smithi122 ceph-mon[32255]: pgmap v6224: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:25:59 smithi022 ceph-mon[31360]: pgmap v6224: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:25:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:25:59 smithi022 ceph-mon[40250]: pgmap v6224: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:00] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:26:00.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:00.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:00.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:00.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:01 smithi122 ceph-mon[32255]: pgmap v6225: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3648400814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:01 smithi022 ceph-mon[31360]: pgmap v6225: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:01.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3648400814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:01 smithi022 ceph-mon[40250]: pgmap v6225: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3648400814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:26:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:26:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:03 smithi122 ceph-mon[32255]: pgmap v6226: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:03 smithi022 ceph-mon[31360]: pgmap v6226: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:03 smithi022 ceph-mon[40250]: pgmap v6226: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:04.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:26:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:26:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:05 smithi122 ceph-mon[32255]: pgmap v6227: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:05 smithi022 ceph-mon[31360]: pgmap v6227: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:05 smithi022 ceph-mon[40250]: pgmap v6227: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:05.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:06.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:06.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:06.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:07 smithi022 ceph-mon[31360]: pgmap v6228: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2138365154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:07 smithi022 ceph-mon[40250]: pgmap v6228: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2138365154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:07 smithi122 ceph-mon[32255]: pgmap v6228: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2138365154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:26:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:09 smithi122 ceph-mon[32255]: pgmap v6229: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:09 smithi022 ceph-mon[31360]: pgmap v6229: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:09 smithi022 ceph-mon[40250]: pgmap v6229: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:10] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:26:11.208 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:11 smithi122 ceph-mon[32255]: pgmap v6230: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:11 smithi022 ceph-mon[31360]: pgmap v6230: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:11 smithi022 ceph-mon[40250]: pgmap v6230: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:11.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:11.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:11.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2061975105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:26:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:26:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2061975105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2061975105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:13 smithi122 ceph-mon[32255]: pgmap v6231: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:13 smithi022 ceph-mon[31360]: pgmap v6231: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:13 smithi022 ceph-mon[40250]: pgmap v6231: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:14.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:26:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:26:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:15 smithi122 ceph-mon[32255]: pgmap v6232: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:15 smithi022 ceph-mon[31360]: pgmap v6232: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:15 smithi022 ceph-mon[40250]: pgmap v6232: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:16.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:16.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:16.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:16.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:17 smithi022 ceph-mon[31360]: pgmap v6233: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:17.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3568523210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:17 smithi022 ceph-mon[40250]: pgmap v6233: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3568523210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:17.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:17 smithi122 ceph-mon[32255]: pgmap v6233: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:17.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3568523210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:17.696 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:26:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:26:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:26:18.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:26:18.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:26:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:26:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:26:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:26:18.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:26:18.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:26:18.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:26:18.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:26:18.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:26:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:19 smithi122 ceph-mon[32255]: pgmap v6234: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:19 smithi022 ceph-mon[31360]: pgmap v6234: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:19 smithi022 ceph-mon[40250]: pgmap v6234: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:20] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:26:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:21 smithi122 ceph-mon[32255]: pgmap v6235: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:21 smithi022 ceph-mon[31360]: pgmap v6235: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:21 smithi022 ceph-mon[40250]: pgmap v6235: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:21.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:22.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:22.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:22.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:26:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:26:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:23 smithi122 ceph-mon[32255]: pgmap v6236: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3330751788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:23 smithi022 ceph-mon[31360]: pgmap v6236: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3330751788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:23 smithi022 ceph-mon[40250]: pgmap v6236: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3330751788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:24.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:26:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:26:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:25 smithi122 ceph-mon[32255]: pgmap v6237: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:25 smithi022 ceph-mon[31360]: pgmap v6237: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:25 smithi022 ceph-mon[40250]: pgmap v6237: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:27.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:27 smithi122 ceph-mon[32255]: pgmap v6238: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:27 smithi022 ceph-mon[31360]: pgmap v6238: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:26:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:27 smithi022 ceph-mon[40250]: pgmap v6238: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:27.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:27.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:27.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/426746502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/426746502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/426746502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:29 smithi122 ceph-mon[32255]: pgmap v6239: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:29 smithi022 ceph-mon[31360]: pgmap v6239: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:29 smithi022 ceph-mon[40250]: pgmap v6239: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:30] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:26:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:31 smithi122 ceph-mon[32255]: pgmap v6240: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:31 smithi022 ceph-mon[31360]: pgmap v6240: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:31 smithi022 ceph-mon[40250]: pgmap v6240: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:26:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:26:32.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:33.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:33.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:33.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:33 smithi122 ceph-mon[32255]: pgmap v6241: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:26:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:26:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:26:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:26:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2926218715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[31360]: pgmap v6241: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:26:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:26:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:26:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:26:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2926218715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[40250]: pgmap v6241: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:26:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:26:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:26:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:26:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2926218715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:34.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:26:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:26:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:35 smithi122 ceph-mon[32255]: pgmap v6242: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:35 smithi022 ceph-mon[31360]: pgmap v6242: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:35 smithi022 ceph-mon[40250]: pgmap v6242: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:37 smithi122 ceph-mon[32255]: pgmap v6243: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:37 smithi022 ceph-mon[31360]: pgmap v6243: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:26:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:37 smithi022 ceph-mon[40250]: pgmap v6243: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:38.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:38.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:38.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:38.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:39 smithi022 ceph-mon[31360]: pgmap v6244: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:39.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/461553837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:39.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:39 smithi022 ceph-mon[40250]: pgmap v6244: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:39.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/461553837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:39.739 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:39 smithi122 ceph-mon[32255]: pgmap v6244: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:39.739 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/461553837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:40] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:26:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:41 smithi022 ceph-mon[31360]: pgmap v6245: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:41 smithi022 ceph-mon[40250]: pgmap v6245: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:41 smithi122 ceph-mon[32255]: pgmap v6245: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:26:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:26:43.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:43 smithi022 ceph-mon[31360]: pgmap v6246: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:43 smithi022 ceph-mon[40250]: pgmap v6246: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:43 smithi122 ceph-mon[32255]: pgmap v6246: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:43.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:43.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:43.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:26:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:26:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3707992413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3707992413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3707992413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:45 smithi022 ceph-mon[31360]: pgmap v6247: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:45 smithi022 ceph-mon[40250]: pgmap v6247: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:45 smithi122 ceph-mon[32255]: pgmap v6247: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:47 smithi022 ceph-mon[31360]: pgmap v6248: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:26:47.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:47 smithi022 ceph-mon[40250]: pgmap v6248: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:47 smithi122 ceph-mon[32255]: pgmap v6248: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:49.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:49.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:49.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:49 smithi022 ceph-mon[31360]: pgmap v6249: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1091427328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:49 smithi022 ceph-mon[40250]: pgmap v6249: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1091427328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:49 smithi122 ceph-mon[32255]: pgmap v6249: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1091427328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:50] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:26:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:51 smithi022 ceph-mon[31360]: pgmap v6250: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:51 smithi022 ceph-mon[40250]: pgmap v6250: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:51 smithi122 ceph-mon[32255]: pgmap v6250: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:26:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:26:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:53 smithi022 ceph-mon[31360]: pgmap v6251: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:53 smithi022 ceph-mon[40250]: pgmap v6251: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:53 smithi122 ceph-mon[32255]: pgmap v6251: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:54.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:26:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:26:54.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:54.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:54.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:26:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:55 smithi022 ceph-mon[31360]: pgmap v6252: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/241337677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:55 smithi022 ceph-mon[40250]: pgmap v6252: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/241337677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:55 smithi122 ceph-mon[32255]: pgmap v6252: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:26:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:26:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/241337677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:26:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:57 smithi022 ceph-mon[31360]: pgmap v6253: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:26:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:57 smithi022 ceph-mon[40250]: pgmap v6253: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:57 smithi122 ceph-mon[32255]: pgmap v6253: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:26:59 smithi022 ceph-mon[31360]: pgmap v6254: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:26:59 smithi022 ceph-mon[40250]: pgmap v6254: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:59.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:26:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:26:59 smithi122 ceph-mon[32255]: pgmap v6254: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:26:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:26:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:26:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/711195611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:00] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:27:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/711195611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/711195611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:01.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:01 smithi022 ceph-mon[31360]: pgmap v6255: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:01 smithi022 ceph-mon[40250]: pgmap v6255: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:01 smithi122 ceph-mon[32255]: pgmap v6255: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:27:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:27:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:03 smithi022 ceph-mon[31360]: pgmap v6256: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:03 smithi022 ceph-mon[40250]: pgmap v6256: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:03 smithi122 ceph-mon[32255]: pgmap v6256: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:04.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:27:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:27:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:05.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:05.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:05.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:05 smithi022 ceph-mon[31360]: pgmap v6257: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:05.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:05 smithi022 ceph-mon[40250]: pgmap v6257: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:05 smithi122 ceph-mon[32255]: pgmap v6257: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:06.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2530566551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:06.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2530566551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2530566551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:07 smithi022 ceph-mon[31360]: pgmap v6258: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:27:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:07 smithi022 ceph-mon[40250]: pgmap v6258: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:07 smithi122 ceph-mon[32255]: pgmap v6258: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:09 smithi022 ceph-mon[31360]: pgmap v6259: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:09 smithi022 ceph-mon[40250]: pgmap v6259: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:09 smithi122 ceph-mon[32255]: pgmap v6259: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:10] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:27:10.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:10.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:10.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:11 smithi022 ceph-mon[31360]: pgmap v6260: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1418085363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:11.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:11 smithi022 ceph-mon[40250]: pgmap v6260: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:11.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1418085363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:11 smithi122 ceph-mon[32255]: pgmap v6260: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1418085363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:27:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:27:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:13 smithi022 ceph-mon[31360]: pgmap v6261: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:13 smithi022 ceph-mon[40250]: pgmap v6261: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:13 smithi122 ceph-mon[32255]: pgmap v6261: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:14.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:27:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:27:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:15 smithi022 ceph-mon[31360]: pgmap v6262: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:15 smithi022 ceph-mon[40250]: pgmap v6262: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:15 smithi122 ceph-mon[32255]: pgmap v6262: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:16.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:16.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:16.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3767283042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3767283042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3767283042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:17 smithi022 ceph-mon[31360]: pgmap v6263: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:27:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:17 smithi022 ceph-mon[40250]: pgmap v6263: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:17 smithi122 ceph-mon[32255]: pgmap v6263: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:19 smithi122 ceph-mon[32255]: pgmap v6264: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:27:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:27:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:27:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:27:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[31360]: pgmap v6264: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:27:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:27:19.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:27:19.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:27:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[40250]: pgmap v6264: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:19.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:27:19.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:27:19.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:27:19.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:27:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:20] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:27:21.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:21 smithi022 ceph-mon[31360]: pgmap v6265: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:21 smithi022 ceph-mon[40250]: pgmap v6265: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:21.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:21.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:21.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:21 smithi122 ceph-mon[32255]: pgmap v6265: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2103808900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:27:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:27:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2103808900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2103808900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:23 smithi122 ceph-mon[32255]: pgmap v6266: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:23 smithi022 ceph-mon[31360]: pgmap v6266: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:23 smithi022 ceph-mon[40250]: pgmap v6266: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:24.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:27:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:27:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:25 smithi122 ceph-mon[32255]: pgmap v6267: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:25 smithi022 ceph-mon[31360]: pgmap v6267: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:25 smithi022 ceph-mon[40250]: pgmap v6267: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:26.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:26.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:26.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:26.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:27 smithi022 ceph-mon[31360]: pgmap v6268: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3235238010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:27 smithi022 ceph-mon[40250]: pgmap v6268: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3235238010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:27 smithi122 ceph-mon[32255]: pgmap v6268: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:27.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3235238010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:27:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:29 smithi122 ceph-mon[32255]: pgmap v6269: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:29 smithi022 ceph-mon[31360]: pgmap v6269: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:29 smithi022 ceph-mon[40250]: pgmap v6269: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:30] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:27:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:31 smithi122 ceph-mon[32255]: pgmap v6270: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:31 smithi022 ceph-mon[31360]: pgmap v6270: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:31 smithi022 ceph-mon[40250]: pgmap v6270: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:31.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:32.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:32.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:32.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:27:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:27:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:33 smithi122 ceph-mon[32255]: pgmap v6271: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3523089441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:27:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:27:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:27:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:27:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[31360]: pgmap v6271: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3523089441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:27:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:27:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:27:33.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:27:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[40250]: pgmap v6271: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3523089441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:27:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:27:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:27:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:27:34.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:27:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:27:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:35 smithi122 ceph-mon[32255]: pgmap v6272: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:35 smithi022 ceph-mon[31360]: pgmap v6272: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:35 smithi022 ceph-mon[40250]: pgmap v6272: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:37.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:37 smithi022 ceph-mon[40250]: pgmap v6273: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:37 smithi022 ceph-mon[31360]: pgmap v6273: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:37 smithi122 ceph-mon[32255]: pgmap v6273: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:27:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/300918820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/300918820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/300918820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:39 smithi122 ceph-mon[32255]: pgmap v6274: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:39 smithi022 ceph-mon[31360]: pgmap v6274: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:39 smithi022 ceph-mon[40250]: pgmap v6274: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:40] "GET /metrics HTTP/1.1" 200 202362 "" "Prometheus/2.18.1" 2021-12-15T21:27:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:41 smithi122 ceph-mon[32255]: pgmap v6275: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:41 smithi022 ceph-mon[31360]: pgmap v6275: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:41 smithi022 ceph-mon[40250]: pgmap v6275: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:42.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:27:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:27:42.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:42.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:42.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:43 smithi022 ceph-mon[31360]: pgmap v6276: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/223950710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:43 smithi022 ceph-mon[40250]: pgmap v6276: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:43.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/223950710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:43 smithi122 ceph-mon[32255]: pgmap v6276: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:43.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/223950710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:27:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:27:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:45 smithi122 ceph-mon[32255]: pgmap v6277: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:45 smithi022 ceph-mon[31360]: pgmap v6277: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:45 smithi022 ceph-mon[40250]: pgmap v6277: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:47 smithi022 ceph-mon[31360]: pgmap v6278: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:47 smithi022 ceph-mon[40250]: pgmap v6278: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:47 smithi122 ceph-mon[32255]: pgmap v6278: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:27:47.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:48.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:48.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:48.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:49 smithi122 ceph-mon[32255]: pgmap v6279: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3070447717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:49 smithi022 ceph-mon[31360]: pgmap v6279: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3070447717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:49 smithi022 ceph-mon[40250]: pgmap v6279: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3070447717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:50] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:27:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:51 smithi122 ceph-mon[32255]: pgmap v6280: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:51 smithi022 ceph-mon[31360]: pgmap v6280: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:51 smithi022 ceph-mon[40250]: pgmap v6280: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:27:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:27:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:27:53.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:53 smithi122 ceph-mon[32255]: pgmap v6281: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:53 smithi022 ceph-mon[31360]: pgmap v6281: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:53 smithi022 ceph-mon[40250]: pgmap v6281: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:27:53.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:53.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:53.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1746416633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:54.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:27:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:27:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1746416633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1746416633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:55 smithi122 ceph-mon[32255]: pgmap v6282: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:55.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:55 smithi022 ceph-mon[31360]: pgmap v6282: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:55 smithi022 ceph-mon[40250]: pgmap v6282: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:55.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:27:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:57 smithi122 ceph-mon[32255]: pgmap v6283: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:57 smithi022 ceph-mon[31360]: pgmap v6283: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:27:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:57 smithi022 ceph-mon[40250]: pgmap v6283: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:58.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:27:58.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:27:58.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:27:58.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:27:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:59 smithi022 ceph-mon[40250]: pgmap v6284: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:27:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/910376703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:59 smithi022 ceph-mon[31360]: pgmap v6284: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:27:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/910376703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:27:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:59 smithi122 ceph-mon[32255]: pgmap v6284: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:27:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:27:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/910376703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:00] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:28:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:01 smithi122 ceph-mon[32255]: pgmap v6285: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:28:01.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:01 smithi022 ceph-mon[31360]: pgmap v6285: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:28:01.565 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:01 smithi022 ceph-mon[40250]: pgmap v6285: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:28:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:28:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:28:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:03 smithi122 ceph-mon[32255]: pgmap v6286: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:28:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:03 smithi022 ceph-mon[31360]: pgmap v6286: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:28:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:03 smithi022 ceph-mon[40250]: pgmap v6286: 132 pgs: 132 active+clean; 582 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:28:03.986 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:04.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:28:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:28:04.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:04.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:04.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:05 smithi122 ceph-mon[32255]: pgmap v6287: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:28:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/444258139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:05 smithi022 ceph-mon[31360]: pgmap v6287: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:28:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/444258139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:05 smithi022 ceph-mon[40250]: pgmap v6287: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:28:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/444258139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:07 smithi122 ceph-mon[32255]: pgmap v6288: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:28:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:07 smithi022 ceph-mon[31360]: pgmap v6288: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:07 smithi022 ceph-mon[40250]: pgmap v6288: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:09.358 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:09 smithi122 ceph-mon[32255]: pgmap v6289: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:09 smithi022 ceph-mon[31360]: pgmap v6289: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:09 smithi022 ceph-mon[40250]: pgmap v6289: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:09.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:09.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:09.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3494516278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:10] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:28:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3494516278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3494516278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:11 smithi122 ceph-mon[32255]: pgmap v6290: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:28:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:11 smithi022 ceph-mon[31360]: pgmap v6290: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:28:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:11 smithi022 ceph-mon[40250]: pgmap v6290: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:28:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:28:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:28:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:13 smithi122 ceph-mon[32255]: pgmap v6291: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:13 smithi022 ceph-mon[31360]: pgmap v6291: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:13 smithi022 ceph-mon[40250]: pgmap v6291: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:28:14.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:28:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:28:14.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:15.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:15.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:15.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:15 smithi122 ceph-mon[32255]: pgmap v6292: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:28:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2660485958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:15 smithi022 ceph-mon[31360]: pgmap v6292: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:28:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2660485958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:15 smithi022 ceph-mon[40250]: pgmap v6292: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:28:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2660485958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:17 smithi122 ceph-mon[32255]: pgmap v6293: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:17 smithi022 ceph-mon[31360]: pgmap v6293: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:28:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:17 smithi022 ceph-mon[40250]: pgmap v6293: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[31360]: pgmap v6294: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:19.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:28:19.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:28:19.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:28:19.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:19.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:19.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[40250]: pgmap v6294: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:19.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:28:19.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:28:19.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:28:19.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:19.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:19.377 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:19 smithi122 ceph-mon[32255]: pgmap v6294: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:19.377 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:28:19.377 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:28:19.377 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:28:19.378 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:19.378 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:20.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:20] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:28:20.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:20.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:20.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:28:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:21 smithi122 ceph-mon[32255]: pgmap v6295: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4207175619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:21 smithi022 ceph-mon[31360]: pgmap v6295: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4207175619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:21 smithi022 ceph-mon[40250]: pgmap v6295: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4207175619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:22.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:28:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:28:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:23 smithi122 ceph-mon[32255]: pgmap v6296: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:23 smithi022 ceph-mon[31360]: pgmap v6296: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:23 smithi022 ceph-mon[40250]: pgmap v6296: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:24.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:28:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:28:25.475 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:25 smithi122 ceph-mon[32255]: pgmap v6297: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:25 smithi022 ceph-mon[40250]: pgmap v6297: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:25 smithi022 ceph-mon[31360]: pgmap v6297: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/764122665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/764122665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/764122665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:27 smithi122 ceph-mon[32255]: pgmap v6298: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:27 smithi022 ceph-mon[31360]: pgmap v6298: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:28:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:27 smithi022 ceph-mon[40250]: pgmap v6298: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:29 smithi022 ceph-mon[31360]: pgmap v6299: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:29 smithi022 ceph-mon[40250]: pgmap v6299: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:29 smithi122 ceph-mon[32255]: pgmap v6299: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:30] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:28:30.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:31 smithi022 ceph-mon[31360]: pgmap v6300: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2256965272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:31 smithi022 ceph-mon[40250]: pgmap v6300: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2256965272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:31 smithi122 ceph-mon[32255]: pgmap v6300: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2256965272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:28:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:28:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[31360]: pgmap v6301: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:28:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:28:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:28:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:28:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[40250]: pgmap v6301: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:28:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:28:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:28:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:28:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:33 smithi122 ceph-mon[32255]: pgmap v6301: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:28:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:28:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:28:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:28:34.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:28:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:28:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:35 smithi022 ceph-mon[31360]: pgmap v6302: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:35 smithi022 ceph-mon[40250]: pgmap v6302: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:35 smithi122 ceph-mon[32255]: pgmap v6302: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:36.213 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:36.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:36.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:36.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:28:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:37 smithi022 ceph-mon[31360]: pgmap v6303: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:37.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4262302009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:37 smithi022 ceph-mon[40250]: pgmap v6303: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4262302009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:37 smithi122 ceph-mon[32255]: pgmap v6303: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4262302009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:39 smithi022 ceph-mon[31360]: pgmap v6304: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:39 smithi022 ceph-mon[40250]: pgmap v6304: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:39 smithi122 ceph-mon[32255]: pgmap v6304: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:40] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:28:41.585 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:41 smithi022 ceph-mon[31360]: pgmap v6305: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:41 smithi022 ceph-mon[40250]: pgmap v6305: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:41 smithi122 ceph-mon[32255]: pgmap v6305: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:41.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:41.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:41.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:28:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:28:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3579650634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3579650634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3579650634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:43 smithi022 ceph-mon[31360]: pgmap v6306: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:43 smithi022 ceph-mon[40250]: pgmap v6306: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:43 smithi122 ceph-mon[32255]: pgmap v6306: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:44.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:28:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:28:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:45 smithi022 ceph-mon[31360]: pgmap v6307: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:45 smithi022 ceph-mon[40250]: pgmap v6307: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:45 smithi122 ceph-mon[32255]: pgmap v6307: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:46.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:47.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:47.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:47.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:28:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:47 smithi022 ceph-mon[31360]: pgmap v6308: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1988482790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:47 smithi022 ceph-mon[40250]: pgmap v6308: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1988482790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:47 smithi122 ceph-mon[32255]: pgmap v6308: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1988482790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:49 smithi022 ceph-mon[31360]: pgmap v6309: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:49 smithi022 ceph-mon[40250]: pgmap v6309: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:49 smithi122 ceph-mon[32255]: pgmap v6309: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:50] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:28:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:51 smithi022 ceph-mon[31360]: pgmap v6310: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:51 smithi022 ceph-mon[40250]: pgmap v6310: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:51 smithi122 ceph-mon[32255]: pgmap v6310: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:52.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:28:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:28:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:52.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:52.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:53 smithi022 ceph-mon[31360]: pgmap v6311: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3246707609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:53 smithi022 ceph-mon[40250]: pgmap v6311: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3246707609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:53 smithi122 ceph-mon[32255]: pgmap v6311: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3246707609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:54.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:28:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:28:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:55 smithi022 ceph-mon[31360]: pgmap v6312: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:55 smithi022 ceph-mon[40250]: pgmap v6312: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:55 smithi122 ceph-mon[32255]: pgmap v6312: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:28:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:28:57.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:28:57.699 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:28:57.700 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:57 smithi022 ceph-mon[40250]: pgmap v6313: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:57.700 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:57 smithi022 ceph-mon[31360]: pgmap v6313: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:57 smithi122 ceph-mon[32255]: pgmap v6313: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:58.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:28:58.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:28:58.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:28:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3829081022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3829081022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3829081022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:28:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:28:59 smithi022 ceph-mon[31360]: pgmap v6314: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:28:59 smithi022 ceph-mon[40250]: pgmap v6314: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:28:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:28:59 smithi122 ceph-mon[32255]: pgmap v6314: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:00] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:29:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:01 smithi022 ceph-mon[31360]: pgmap v6315: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:01 smithi022 ceph-mon[40250]: pgmap v6315: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:01 smithi122 ceph-mon[32255]: pgmap v6315: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:29:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:29:03.069 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:03 smithi022 ceph-mon[31360]: pgmap v6316: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/660495295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:03 smithi022 ceph-mon[40250]: pgmap v6316: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/660495295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:03 smithi122 ceph-mon[32255]: pgmap v6316: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/660495295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:04.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:29:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:29:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:05 smithi022 ceph-mon[31360]: pgmap v6317: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:05 smithi022 ceph-mon[40250]: pgmap v6317: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:05 smithi122 ceph-mon[32255]: pgmap v6317: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:07 smithi022 ceph-mon[31360]: pgmap v6318: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:29:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:07 smithi022 ceph-mon[40250]: pgmap v6318: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:07 smithi122 ceph-mon[32255]: pgmap v6318: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:08.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:08.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:08.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:08.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:09 smithi022 ceph-mon[31360]: pgmap v6319: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3865393612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:09 smithi022 ceph-mon[40250]: pgmap v6319: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3865393612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:09 smithi122 ceph-mon[32255]: pgmap v6319: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3865393612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:10] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:29:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:11 smithi022 ceph-mon[31360]: pgmap v6320: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:11 smithi022 ceph-mon[40250]: pgmap v6320: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:11 smithi122 ceph-mon[32255]: pgmap v6320: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:29:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:29:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:13 smithi022 ceph-mon[31360]: pgmap v6321: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:13 smithi022 ceph-mon[40250]: pgmap v6321: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:13.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:13 smithi122 ceph-mon[32255]: pgmap v6321: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:14.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:14.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:14.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:14.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:29:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:29:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/387402369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/387402369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/387402369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:15 smithi022 ceph-mon[31360]: pgmap v6322: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:15 smithi022 ceph-mon[40250]: pgmap v6322: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:15 smithi122 ceph-mon[32255]: pgmap v6322: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:15.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:17 smithi022 ceph-mon[31360]: pgmap v6323: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:29:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:17 smithi022 ceph-mon[40250]: pgmap v6323: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:17 smithi122 ceph-mon[32255]: pgmap v6323: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:19.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:19.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:19.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:19.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[31360]: pgmap v6324: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:29:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:29:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:29:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[40250]: pgmap v6324: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:29:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:29:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:29:19.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:19 smithi122 ceph-mon[32255]: pgmap v6324: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:19.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:29:19.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:29:19.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:29:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:20] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:29:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2735598660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:29:20.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2735598660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:20.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:29:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2735598660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:29:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:21 smithi022 ceph-mon[40250]: pgmap v6325: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:21 smithi022 ceph-mon[31360]: pgmap v6325: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:21 smithi122 ceph-mon[32255]: pgmap v6325: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:29:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:29:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:23 smithi022 ceph-mon[31360]: pgmap v6326: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:23 smithi022 ceph-mon[40250]: pgmap v6326: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:23 smithi122 ceph-mon[32255]: pgmap v6326: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:24.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:29:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:29:24.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:24.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:24.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:24.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:25 smithi022 ceph-mon[40250]: pgmap v6327: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/467642435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:25.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:25 smithi022 ceph-mon[31360]: pgmap v6327: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:25.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:25.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/467642435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:25 smithi122 ceph-mon[32255]: pgmap v6327: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/467642435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:29:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:27 smithi022 ceph-mon[40250]: pgmap v6328: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:27.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:27 smithi022 ceph-mon[31360]: pgmap v6328: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:27 smithi122 ceph-mon[32255]: pgmap v6328: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:29.925 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:29.926 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:29 smithi022 ceph-mon[31360]: pgmap v6329: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:29.926 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:29 smithi022 ceph-mon[40250]: pgmap v6329: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:29 smithi122 ceph-mon[32255]: pgmap v6329: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:30.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:30.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:30.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:30] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:29:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2084637717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2084637717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2084637717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:31 smithi122 ceph-mon[32255]: pgmap v6330: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:31 smithi022 ceph-mon[31360]: pgmap v6330: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:31 smithi022 ceph-mon[40250]: pgmap v6330: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:29:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:29:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:33 smithi122 ceph-mon[32255]: pgmap v6331: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:29:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:29:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:29:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:29:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[31360]: pgmap v6331: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:29:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:29:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:29:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:29:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[40250]: pgmap v6331: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:29:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:29:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:29:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:29:34.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:29:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:29:35.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:35.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:35.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:35.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:35 smithi122 ceph-mon[32255]: pgmap v6332: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:35 smithi022 ceph-mon[40250]: pgmap v6332: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:35 smithi022 ceph-mon[31360]: pgmap v6332: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2460571798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2460571798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2460571798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:37 smithi022 ceph-mon[31360]: pgmap v6333: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:29:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:37 smithi022 ceph-mon[40250]: pgmap v6333: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:37 smithi122 ceph-mon[32255]: pgmap v6333: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:39 smithi122 ceph-mon[32255]: pgmap v6334: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:39 smithi022 ceph-mon[31360]: pgmap v6334: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:39 smithi022 ceph-mon[40250]: pgmap v6334: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:40] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:29:40.665 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:41.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:41.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:41.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:41 smithi122 ceph-mon[32255]: pgmap v6335: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2976717206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:41 smithi022 ceph-mon[31360]: pgmap v6335: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2976717206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:41 smithi022 ceph-mon[40250]: pgmap v6335: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2976717206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:29:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:29:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:43 smithi122 ceph-mon[32255]: pgmap v6336: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:43 smithi022 ceph-mon[31360]: pgmap v6336: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:43 smithi022 ceph-mon[40250]: pgmap v6336: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:44.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:29:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:29:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:45 smithi122 ceph-mon[32255]: pgmap v6337: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:45 smithi022 ceph-mon[31360]: pgmap v6337: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:45 smithi022 ceph-mon[40250]: pgmap v6337: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:46.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:46.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:46.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:46.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/327374267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/327374267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/327374267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:47 smithi022 ceph-mon[31360]: pgmap v6338: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:29:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:47 smithi022 ceph-mon[40250]: pgmap v6338: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:47 smithi122 ceph-mon[32255]: pgmap v6338: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:49 smithi122 ceph-mon[32255]: pgmap v6339: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:49 smithi022 ceph-mon[31360]: pgmap v6339: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:49 smithi022 ceph-mon[40250]: pgmap v6339: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:50] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:29:51.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:51 smithi022 ceph-mon[31360]: pgmap v6340: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:51 smithi022 ceph-mon[40250]: pgmap v6340: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:51.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:51.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:51.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:51 smithi122 ceph-mon[32255]: pgmap v6340: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:52.583 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:29:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:29:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/752401289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/752401289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/752401289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:53 smithi122 ceph-mon[32255]: pgmap v6341: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:53 smithi022 ceph-mon[31360]: pgmap v6341: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:53 smithi022 ceph-mon[40250]: pgmap v6341: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:54.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:29:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:29:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:55 smithi122 ceph-mon[32255]: pgmap v6342: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:55 smithi022 ceph-mon[31360]: pgmap v6342: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:55 smithi022 ceph-mon[40250]: pgmap v6342: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:29:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:29:56.782 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:29:57.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:29:57.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:29:57.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:29:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:29:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:57 smithi122 ceph-mon[32255]: pgmap v6343: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2372976492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:57 smithi022 ceph-mon[40250]: pgmap v6343: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2372976492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:57 smithi022 ceph-mon[31360]: pgmap v6343: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:29:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2372976492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:29:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:29:59 smithi122 ceph-mon[32255]: pgmap v6344: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:29:59 smithi022 ceph-mon[31360]: pgmap v6344: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:29:59 smithi022 ceph-mon[40250]: pgmap v6344: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:00] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:30:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T21:30:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T21:30:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T21:30:01.872 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:01 smithi122 ceph-mon[32255]: pgmap v6345: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:01 smithi022 ceph-mon[31360]: pgmap v6345: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:01 smithi022 ceph-mon[40250]: pgmap v6345: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:02.154 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:02.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:02.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:02.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:02.603 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:30:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:30:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/194743994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/194743994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/194743994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:03 smithi122 ceph-mon[32255]: pgmap v6346: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:03 smithi022 ceph-mon[31360]: pgmap v6346: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:03 smithi022 ceph-mon[40250]: pgmap v6346: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:04.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:30:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:30:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:05 smithi122 ceph-mon[32255]: pgmap v6347: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:05 smithi022 ceph-mon[31360]: pgmap v6347: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:05 smithi022 ceph-mon[40250]: pgmap v6347: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:07.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:07 smithi022 ceph-mon[31360]: pgmap v6348: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:30:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:07 smithi022 ceph-mon[40250]: pgmap v6348: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:07.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:07.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:07.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:07 smithi122 ceph-mon[32255]: pgmap v6348: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3814797149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3814797149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3814797149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:09 smithi122 ceph-mon[32255]: pgmap v6349: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:09 smithi022 ceph-mon[31360]: pgmap v6349: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:09 smithi022 ceph-mon[40250]: pgmap v6349: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:10] "GET /metrics HTTP/1.1" 200 202341 "" "Prometheus/2.18.1" 2021-12-15T21:30:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:11 smithi122 ceph-mon[32255]: pgmap v6350: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:11 smithi022 ceph-mon[31360]: pgmap v6350: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:11 smithi022 ceph-mon[40250]: pgmap v6350: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:30:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:30:12.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:13.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:13.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:13.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:13 smithi122 ceph-mon[32255]: pgmap v6351: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/77419827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:13 smithi022 ceph-mon[31360]: pgmap v6351: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/77419827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:13 smithi022 ceph-mon[40250]: pgmap v6351: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/77419827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:14.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:30:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:30:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:15 smithi122 ceph-mon[32255]: pgmap v6352: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:15 smithi022 ceph-mon[31360]: pgmap v6352: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:15 smithi022 ceph-mon[40250]: pgmap v6352: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:17 smithi022 ceph-mon[31360]: pgmap v6353: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:30:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:17 smithi022 ceph-mon[40250]: pgmap v6353: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:17 smithi122 ceph-mon[32255]: pgmap v6353: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:18.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:18.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:18.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:18.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:19.911 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:19 smithi122 ceph-mon[32255]: pgmap v6354: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:19.911 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/833121931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:19.912 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:19 smithi022 ceph-mon[31360]: pgmap v6354: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:19.912 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/833121931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:19.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:19 smithi022 ceph-mon[40250]: pgmap v6354: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:19.913 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/833121931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:20] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:30:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:30:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:30:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:30:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:30:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:30:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:30:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:30:21.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:30:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:30:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:30:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:30:21.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:30:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:21 smithi122 ceph-mon[32255]: pgmap v6355: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:21 smithi022 ceph-mon[31360]: pgmap v6355: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:21 smithi022 ceph-mon[40250]: pgmap v6355: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:30:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:30:23.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:23 smithi122 ceph-mon[32255]: pgmap v6356: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:24.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:24.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:24.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:24.013 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:23 smithi022 ceph-mon[40250]: pgmap v6356: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:24.013 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:23 smithi022 ceph-mon[31360]: pgmap v6356: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:24.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:30:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:30:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3767466507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3767466507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3767466507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:25 smithi122 ceph-mon[32255]: pgmap v6357: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:25 smithi022 ceph-mon[40250]: pgmap v6357: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:25 smithi022 ceph-mon[31360]: pgmap v6357: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:27.648 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:30:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:27 smithi122 ceph-mon[32255]: pgmap v6358: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:27 smithi022 ceph-mon[31360]: pgmap v6358: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:27 smithi022 ceph-mon[40250]: pgmap v6358: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:29.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:29.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:29.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:29.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:29 smithi122 ceph-mon[32255]: pgmap v6359: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1244005111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:29 smithi022 ceph-mon[31360]: pgmap v6359: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1244005111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:29 smithi022 ceph-mon[40250]: pgmap v6359: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1244005111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:30] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:30:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:31 smithi122 ceph-mon[32255]: pgmap v6360: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:31 smithi022 ceph-mon[31360]: pgmap v6360: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:31 smithi022 ceph-mon[40250]: pgmap v6360: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:30:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:30:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:33 smithi122 ceph-mon[32255]: pgmap v6361: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:30:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:30:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:30:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:30:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[31360]: pgmap v6361: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:30:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:30:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:30:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:30:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[40250]: pgmap v6361: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:30:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:30:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:30:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:30:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:34.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:30:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:30:34.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:34.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:34.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:35 smithi122 ceph-mon[32255]: pgmap v6362: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/187253550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:35 smithi022 ceph-mon[31360]: pgmap v6362: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/187253550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:35 smithi022 ceph-mon[40250]: pgmap v6362: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/187253550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:37.665 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:30:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:37 smithi122 ceph-mon[32255]: pgmap v6363: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:37 smithi022 ceph-mon[31360]: pgmap v6363: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:37 smithi022 ceph-mon[40250]: pgmap v6363: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:39.753 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:39 smithi122 ceph-mon[32255]: pgmap v6364: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:39 smithi022 ceph-mon[31360]: pgmap v6364: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:39 smithi022 ceph-mon[40250]: pgmap v6364: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:40.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:40.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:40.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:40] "GET /metrics HTTP/1.1" 200 202344 "" "Prometheus/2.18.1" 2021-12-15T21:30:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2977065301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2977065301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2977065301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:41 smithi122 ceph-mon[32255]: pgmap v6365: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:41 smithi022 ceph-mon[31360]: pgmap v6365: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:41 smithi022 ceph-mon[40250]: pgmap v6365: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:30:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:30:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:43 smithi122 ceph-mon[32255]: pgmap v6366: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:43 smithi022 ceph-mon[31360]: pgmap v6366: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:43 smithi022 ceph-mon[40250]: pgmap v6366: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:44.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:30:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:30:45.123 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:45.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:45.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:45.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:45 smithi022 ceph-mon[31360]: pgmap v6367: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:45.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/940610447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:45 smithi022 ceph-mon[40250]: pgmap v6367: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:45.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/940610447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:45 smithi122 ceph-mon[32255]: pgmap v6367: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/940610447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:47.681 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:30:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:47 smithi122 ceph-mon[32255]: pgmap v6368: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:47 smithi022 ceph-mon[31360]: pgmap v6368: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:47 smithi022 ceph-mon[40250]: pgmap v6368: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:49 smithi122 ceph-mon[32255]: pgmap v6369: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:49 smithi022 ceph-mon[31360]: pgmap v6369: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:49 smithi022 ceph-mon[40250]: pgmap v6369: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:50] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:30:50.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:50.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:50.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:50.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:51 smithi122 ceph-mon[32255]: pgmap v6370: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1436855324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:51 smithi022 ceph-mon[31360]: pgmap v6370: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1436855324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:51 smithi022 ceph-mon[40250]: pgmap v6370: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1436855324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:30:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:30:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:53 smithi022 ceph-mon[31360]: pgmap v6371: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:53 smithi022 ceph-mon[40250]: pgmap v6371: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:54.064 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:53 smithi122 ceph-mon[32255]: pgmap v6371: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:54.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:30:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:30:55.863 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:30:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:55 smithi022 ceph-mon[31360]: pgmap v6372: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:55 smithi022 ceph-mon[40250]: pgmap v6372: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:55 smithi122 ceph-mon[32255]: pgmap v6372: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:30:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:30:56.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:30:56.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:30:56.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:30:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2672677246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2672677246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2672677246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:30:57.695 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:30:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:57 smithi022 ceph-mon[31360]: pgmap v6373: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:57 smithi022 ceph-mon[40250]: pgmap v6373: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:30:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:57 smithi122 ceph-mon[32255]: pgmap v6373: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:30:59 smithi022 ceph-mon[31360]: pgmap v6374: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:30:59 smithi022 ceph-mon[40250]: pgmap v6374: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:30:59 smithi122 ceph-mon[32255]: pgmap v6374: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:00] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:31:01.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:01.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:01.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:01.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:01 smithi022 ceph-mon[31360]: pgmap v6375: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3938828553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:01 smithi022 ceph-mon[40250]: pgmap v6375: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3938828553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:01 smithi122 ceph-mon[32255]: pgmap v6375: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3938828553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:31:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:31:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:03 smithi022 ceph-mon[31360]: pgmap v6376: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:03 smithi022 ceph-mon[40250]: pgmap v6376: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:04.070 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:03 smithi122 ceph-mon[32255]: pgmap v6376: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:04.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:31:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:31:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:05 smithi022 ceph-mon[31360]: pgmap v6377: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:05 smithi022 ceph-mon[40250]: pgmap v6377: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:05 smithi122 ceph-mon[32255]: pgmap v6377: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:06.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:06.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:06.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:06.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:07.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:31:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:07 smithi022 ceph-mon[31360]: pgmap v6378: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/131095562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:07 smithi022 ceph-mon[40250]: pgmap v6378: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/131095562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:07 smithi122 ceph-mon[32255]: pgmap v6378: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/131095562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:09 smithi022 ceph-mon[31360]: pgmap v6379: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:09 smithi022 ceph-mon[40250]: pgmap v6379: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:09 smithi122 ceph-mon[32255]: pgmap v6379: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:10] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:31:11.971 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:11 smithi022 ceph-mon[40250]: pgmap v6380: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:11.971 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:11 smithi022 ceph-mon[31360]: pgmap v6380: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:11.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:11 smithi122 ceph-mon[32255]: pgmap v6380: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:12.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:12.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:12.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:31:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:31:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1987566596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1987566596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1987566596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:13 smithi022 ceph-mon[31360]: pgmap v6381: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:13 smithi022 ceph-mon[40250]: pgmap v6381: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:14.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:13 smithi122 ceph-mon[32255]: pgmap v6381: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:14.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:31:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:31:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:14 smithi022 ceph-mon[31360]: pgmap v6382: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:14 smithi022 ceph-mon[40250]: pgmap v6382: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:14 smithi122 ceph-mon[32255]: pgmap v6382: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:15.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:16 smithi022 ceph-mon[31360]: pgmap v6383: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:16 smithi022 ceph-mon[40250]: pgmap v6383: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:17.343 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:16 smithi122 ceph-mon[32255]: pgmap v6383: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:17.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:17.710 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:17.710 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:17.711 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:31:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4261727077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4261727077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4261727077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:19 smithi122 ceph-mon[32255]: pgmap v6384: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:19 smithi022 ceph-mon[31360]: pgmap v6384: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:19 smithi022 ceph-mon[40250]: pgmap v6384: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:20] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:31:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:21 smithi122 ceph-mon[32255]: pgmap v6385: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:31:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:31:21.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:31:21.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:31:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[31360]: pgmap v6385: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:31:21.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:31:21.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:31:21.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:31:21.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[40250]: pgmap v6385: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:21.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:31:21.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:31:21.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:31:21.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:31:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:31:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:31:22.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:23.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:23 smithi122 ceph-mon[32255]: pgmap v6386: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2438245226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:23 smithi022 ceph-mon[31360]: pgmap v6386: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2438245226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:23 smithi022 ceph-mon[40250]: pgmap v6386: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2438245226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:24.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:31:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:31:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:25 smithi122 ceph-mon[32255]: pgmap v6387: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:25 smithi022 ceph-mon[40250]: pgmap v6387: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:25 smithi022 ceph-mon[31360]: pgmap v6387: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:25.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:27 smithi122 ceph-mon[32255]: pgmap v6388: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:27 smithi022 ceph-mon[31360]: pgmap v6388: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:31:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:27 smithi022 ceph-mon[40250]: pgmap v6388: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:28.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:28.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:28.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:28.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:29 smithi122 ceph-mon[32255]: pgmap v6389: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2803390288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:29 smithi022 ceph-mon[31360]: pgmap v6389: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2803390288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:29 smithi022 ceph-mon[40250]: pgmap v6389: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2803390288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:30] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:31:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:31 smithi122 ceph-mon[32255]: pgmap v6390: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:31 smithi022 ceph-mon[31360]: pgmap v6390: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:31 smithi022 ceph-mon[40250]: pgmap v6390: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:31:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:31:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:33 smithi122 ceph-mon[32255]: pgmap v6391: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:31:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:31:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:31:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:31:33.453 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:33.453 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[31360]: pgmap v6391: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:33.454 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:31:33.454 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:31:33.454 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:31:33.455 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:31:33.455 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[40250]: pgmap v6391: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:33.455 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:31:33.455 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:31:33.456 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:31:33.456 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:31:33.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:33.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:33.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:34.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:31:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:31:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3156128086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3156128086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3156128086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:35 smithi122 ceph-mon[32255]: pgmap v6392: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:35 smithi022 ceph-mon[31360]: pgmap v6392: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:35 smithi022 ceph-mon[40250]: pgmap v6392: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:37 smithi122 ceph-mon[32255]: pgmap v6393: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:37 smithi022 ceph-mon[31360]: pgmap v6393: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:31:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:37 smithi022 ceph-mon[40250]: pgmap v6393: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:39.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:39.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:39.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:39 smithi122 ceph-mon[32255]: pgmap v6394: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:39 smithi022 ceph-mon[31360]: pgmap v6394: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:39 smithi022 ceph-mon[40250]: pgmap v6394: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/889364268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:40] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:31:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/889364268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/889364268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:41 smithi122 ceph-mon[32255]: pgmap v6395: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:41 smithi022 ceph-mon[31360]: pgmap v6395: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:41 smithi022 ceph-mon[40250]: pgmap v6395: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:31:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:31:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:43 smithi122 ceph-mon[32255]: pgmap v6396: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:43 smithi022 ceph-mon[31360]: pgmap v6396: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:43 smithi022 ceph-mon[40250]: pgmap v6396: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:44.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:44.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:31:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:31:44.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:44.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:44.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:45 smithi122 ceph-mon[32255]: pgmap v6397: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1281861548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:45 smithi022 ceph-mon[31360]: pgmap v6397: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1281861548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:45 smithi022 ceph-mon[40250]: pgmap v6397: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1281861548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:47 smithi122 ceph-mon[32255]: pgmap v6398: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:47 smithi022 ceph-mon[31360]: pgmap v6398: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:31:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:47 smithi022 ceph-mon[40250]: pgmap v6398: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:49 smithi122 ceph-mon[32255]: pgmap v6399: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:49 smithi022 ceph-mon[31360]: pgmap v6399: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:49 smithi022 ceph-mon[40250]: pgmap v6399: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:49.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:49.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:49.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301483552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:50] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:31:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301483552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301483552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:51 smithi022 ceph-mon[31360]: pgmap v6400: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:51 smithi022 ceph-mon[40250]: pgmap v6400: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:51 smithi122 ceph-mon[32255]: pgmap v6400: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:31:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:31:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:53 smithi022 ceph-mon[31360]: pgmap v6401: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:53 smithi022 ceph-mon[40250]: pgmap v6401: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:53 smithi122 ceph-mon[32255]: pgmap v6401: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:54.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:31:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:31:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:31:55.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:31:55.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:31:55.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:31:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:55 smithi022 ceph-mon[31360]: pgmap v6402: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:55 smithi022 ceph-mon[40250]: pgmap v6402: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:55 smithi122 ceph-mon[32255]: pgmap v6402: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:31:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:31:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/836664512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/836664512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/836664512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:31:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:57 smithi022 ceph-mon[31360]: pgmap v6403: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:31:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:57 smithi022 ceph-mon[40250]: pgmap v6403: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:57 smithi122 ceph-mon[32255]: pgmap v6403: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:31:59 smithi022 ceph-mon[31360]: pgmap v6404: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:31:59 smithi022 ceph-mon[40250]: pgmap v6404: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:31:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:31:59 smithi122 ceph-mon[32255]: pgmap v6404: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:00.305 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:00] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:32:00.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:00.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:00.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:01 smithi022 ceph-mon[40250]: pgmap v6405: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1450202736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:01.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:01 smithi022 ceph-mon[31360]: pgmap v6405: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:01.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1450202736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:01 smithi122 ceph-mon[32255]: pgmap v6405: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1450202736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:32:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:32:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:03 smithi022 ceph-mon[31360]: pgmap v6406: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:03 smithi022 ceph-mon[40250]: pgmap v6406: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:03 smithi122 ceph-mon[32255]: pgmap v6406: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:04.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:32:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:32:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:05 smithi022 ceph-mon[31360]: pgmap v6407: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:05 smithi022 ceph-mon[40250]: pgmap v6407: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:05.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:05 smithi122 ceph-mon[32255]: pgmap v6407: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:06.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:06.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:06.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3517694411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3517694411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3517694411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:07 smithi022 ceph-mon[31360]: pgmap v6408: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:32:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:07 smithi022 ceph-mon[40250]: pgmap v6408: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:07 smithi122 ceph-mon[32255]: pgmap v6408: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:09 smithi022 ceph-mon[31360]: pgmap v6409: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:09 smithi022 ceph-mon[40250]: pgmap v6409: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:09 smithi122 ceph-mon[32255]: pgmap v6409: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:32:11.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:11.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:11.417 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:11 smithi022 ceph-mon[31360]: pgmap v6410: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:11.418 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:11 smithi022 ceph-mon[40250]: pgmap v6410: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:11 smithi122 ceph-mon[32255]: pgmap v6410: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1701041062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1701041062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1701041062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:32:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:32:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:13 smithi022 ceph-mon[31360]: pgmap v6411: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:13 smithi022 ceph-mon[40250]: pgmap v6411: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:13 smithi122 ceph-mon[32255]: pgmap v6411: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:14.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:32:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:32:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:15 smithi022 ceph-mon[31360]: pgmap v6412: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:15 smithi022 ceph-mon[40250]: pgmap v6412: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:15 smithi122 ceph-mon[32255]: pgmap v6412: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:16.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:16.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:16.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:32:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:17 smithi022 ceph-mon[31360]: pgmap v6413: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:17.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/791750689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:17 smithi022 ceph-mon[40250]: pgmap v6413: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:17.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/791750689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:17 smithi122 ceph-mon[32255]: pgmap v6413: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/791750689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:19 smithi022 ceph-mon[31360]: pgmap v6414: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:19 smithi022 ceph-mon[40250]: pgmap v6414: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:19 smithi122 ceph-mon[32255]: pgmap v6414: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:20] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:32:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[31360]: pgmap v6415: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:32:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:32:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:32:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[40250]: pgmap v6415: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:32:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:32:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:32:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:21 smithi122 ceph-mon[32255]: pgmap v6415: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:32:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:32:21.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:32:21.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:22.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:22.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:22.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1536273957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1536273957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1536273957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:32:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:32:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:23 smithi022 ceph-mon[31360]: pgmap v6416: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:23 smithi022 ceph-mon[40250]: pgmap v6416: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:23 smithi122 ceph-mon[32255]: pgmap v6416: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:24.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:32:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:32:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[31360]: pgmap v6417: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[40250]: pgmap v6417: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:25 smithi122 ceph-mon[32255]: pgmap v6417: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:25.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:27.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:27.269 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:27 smithi122 ceph-mon[32255]: pgmap v6418: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:27.270 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:27.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:27 smithi022 ceph-mon[31360]: pgmap v6418: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:27.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:27 smithi022 ceph-mon[40250]: pgmap v6418: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:27.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:27.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:27.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:27.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:32:28.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3375217574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:28.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:28.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3375217574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:28.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3375217574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:32:28.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:29 smithi122 ceph-mon[32255]: pgmap v6419: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:29 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3247M 2021-12-15T21:32:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[31360]: pgmap v6419: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3247M 2021-12-15T21:32:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[40250]: pgmap v6419: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3247M 2021-12-15T21:32:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:32:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:30] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:32:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:31 smithi122 ceph-mon[32255]: pgmap v6420: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:31 smithi022 ceph-mon[31360]: pgmap v6420: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:31 smithi022 ceph-mon[40250]: pgmap v6420: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:32.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:32:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:32:32.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:32.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:32.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:33 smithi122 ceph-mon[32255]: pgmap v6421: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2471780697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:32:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:32:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:32:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:32:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[31360]: pgmap v6421: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2471780697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:32:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:32:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:32:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:32:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[40250]: pgmap v6421: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2471780697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:32:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:32:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:32:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:32:34.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:32:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:32:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:35 smithi122 ceph-mon[32255]: pgmap v6422: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:35 smithi022 ceph-mon[31360]: pgmap v6422: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:35 smithi022 ceph-mon[40250]: pgmap v6422: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:37.676 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:32:37.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:37 smithi122 ceph-mon[32255]: pgmap v6423: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:37 smithi022 ceph-mon[31360]: pgmap v6423: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:37 smithi022 ceph-mon[40250]: pgmap v6423: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:38.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:38.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:38.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:39 smithi122 ceph-mon[32255]: pgmap v6424: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2019271216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:39 smithi022 ceph-mon[31360]: pgmap v6424: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2019271216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:39 smithi022 ceph-mon[40250]: pgmap v6424: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2019271216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:40] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:32:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:41 smithi122 ceph-mon[32255]: pgmap v6425: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:41 smithi022 ceph-mon[31360]: pgmap v6425: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:41 smithi022 ceph-mon[40250]: pgmap v6425: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:32:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:32:43.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:43 smithi122 ceph-mon[32255]: pgmap v6426: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:43 smithi022 ceph-mon[31360]: pgmap v6426: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:43 smithi022 ceph-mon[40250]: pgmap v6426: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3038938530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:44.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:32:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:32:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3038938530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3038938530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:45 smithi122 ceph-mon[32255]: pgmap v6427: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:45 smithi022 ceph-mon[31360]: pgmap v6427: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:45 smithi022 ceph-mon[40250]: pgmap v6427: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:47 smithi122 ceph-mon[32255]: pgmap v6428: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:47 smithi022 ceph-mon[31360]: pgmap v6428: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:32:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:47 smithi022 ceph-mon[40250]: pgmap v6428: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:48.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:49.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:49.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:49.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:49 smithi022 ceph-mon[31360]: pgmap v6429: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1363464282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:49 smithi022 ceph-mon[40250]: pgmap v6429: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1363464282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:49 smithi122 ceph-mon[32255]: pgmap v6429: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1363464282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:50] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:32:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:51 smithi122 ceph-mon[32255]: pgmap v6430: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:51 smithi022 ceph-mon[31360]: pgmap v6430: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:51 smithi022 ceph-mon[40250]: pgmap v6430: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:32:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:32:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:53 smithi122 ceph-mon[32255]: pgmap v6431: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:53 smithi022 ceph-mon[31360]: pgmap v6431: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:53 smithi022 ceph-mon[40250]: pgmap v6431: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:54.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:54.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:54.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:32:54.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:32:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:32:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:55 smithi022 ceph-mon[31360]: pgmap v6432: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3760030390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:55 smithi022 ceph-mon[40250]: pgmap v6432: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3760030390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:55 smithi122 ceph-mon[32255]: pgmap v6432: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:32:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:32:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3760030390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:32:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:57 smithi022 ceph-mon[31360]: pgmap v6433: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:32:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:57 smithi022 ceph-mon[40250]: pgmap v6433: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:57 smithi122 ceph-mon[32255]: pgmap v6433: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:59.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:32:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:32:59 smithi022 ceph-mon[31360]: pgmap v6434: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:32:59 smithi022 ceph-mon[40250]: pgmap v6434: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:32:59 smithi122 ceph-mon[32255]: pgmap v6434: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:32:59.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:32:59.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:32:59.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3061963814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:00] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:33:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3061963814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3061963814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:01 smithi022 ceph-mon[31360]: pgmap v6435: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:01 smithi022 ceph-mon[40250]: pgmap v6435: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:01 smithi122 ceph-mon[32255]: pgmap v6435: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:33:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:33:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:03 smithi022 ceph-mon[31360]: pgmap v6436: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:03 smithi022 ceph-mon[40250]: pgmap v6436: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:03 smithi122 ceph-mon[32255]: pgmap v6436: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:04.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:33:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:33:04.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:05.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:05.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:05.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:05 smithi022 ceph-mon[31360]: pgmap v6437: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:05.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1522727601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:05 smithi022 ceph-mon[40250]: pgmap v6437: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1522727601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:05 smithi122 ceph-mon[32255]: pgmap v6437: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1522727601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:07 smithi022 ceph-mon[31360]: pgmap v6438: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:33:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:07 smithi022 ceph-mon[40250]: pgmap v6438: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:07 smithi122 ceph-mon[32255]: pgmap v6438: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:09 smithi022 ceph-mon[31360]: pgmap v6439: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:09 smithi022 ceph-mon[40250]: pgmap v6439: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:09 smithi122 ceph-mon[32255]: pgmap v6439: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:10.145 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:10] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:33:10.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:10.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:10.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:11 smithi022 ceph-mon[31360]: pgmap v6440: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2604567771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:11 smithi022 ceph-mon[40250]: pgmap v6440: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2604567771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:11 smithi122 ceph-mon[32255]: pgmap v6440: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2604567771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:33:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:33:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:13 smithi022 ceph-mon[31360]: pgmap v6441: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:13 smithi022 ceph-mon[40250]: pgmap v6441: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:13 smithi122 ceph-mon[32255]: pgmap v6441: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:14.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:33:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:33:15.516 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:15.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:15 smithi022 ceph-mon[31360]: pgmap v6442: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:15.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:15.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:15 smithi022 ceph-mon[40250]: pgmap v6442: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:15.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:15 smithi122 ceph-mon[32255]: pgmap v6442: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:15.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:15.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:15.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3779413976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3779413976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3779413976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:17 smithi022 ceph-mon[31360]: pgmap v6443: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:33:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:17 smithi022 ceph-mon[40250]: pgmap v6443: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:17 smithi122 ceph-mon[32255]: pgmap v6443: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:19 smithi022 ceph-mon[31360]: pgmap v6444: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:19 smithi022 ceph-mon[40250]: pgmap v6444: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:19 smithi122 ceph-mon[32255]: pgmap v6444: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:20] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:33:20.882 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:21.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:21.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:21.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:21 smithi022 ceph-mon[31360]: pgmap v6445: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:21 smithi022 ceph-mon[40250]: pgmap v6445: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:21 smithi122 ceph-mon[32255]: pgmap v6445: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1860902774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1860902774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1860902774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:33:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:33:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:23 smithi022 ceph-mon[31360]: pgmap v6446: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:23 smithi022 ceph-mon[40250]: pgmap v6446: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:23 smithi122 ceph-mon[32255]: pgmap v6446: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:24.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:33:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:33:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:25 smithi022 ceph-mon[31360]: pgmap v6447: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:25 smithi022 ceph-mon[40250]: pgmap v6447: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:25 smithi122 ceph-mon[32255]: pgmap v6447: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:26.253 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:26.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:27 smithi022 ceph-mon[31360]: pgmap v6448: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1915131945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:33:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:27 smithi022 ceph-mon[40250]: pgmap v6448: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1915131945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:27 smithi122 ceph-mon[32255]: pgmap v6448: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1915131945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:29 smithi122 ceph-mon[32255]: pgmap v6449: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:33:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:33:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:33:29.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:33:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[31360]: pgmap v6449: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:33:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:33:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:33:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:33:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[40250]: pgmap v6449: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:33:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:33:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:33:29.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:33:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:30] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:33:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:31 smithi022 ceph-mon[31360]: pgmap v6450: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:31 smithi022 ceph-mon[40250]: pgmap v6450: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:31.628 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:31 smithi122 ceph-mon[32255]: pgmap v6450: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:32.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:32.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:32.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/217922470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/217922470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/217922470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:33:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:33:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[31360]: pgmap v6451: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:33:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:33:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:33:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:33:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[40250]: pgmap v6451: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:33:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:33:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:33:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:33:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:33 smithi122 ceph-mon[32255]: pgmap v6451: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:33:33.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:33:33.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:33:33.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:33:34.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:33:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:33:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:35 smithi022 ceph-mon[31360]: pgmap v6452: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:35 smithi022 ceph-mon[40250]: pgmap v6452: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:35 smithi122 ceph-mon[32255]: pgmap v6452: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:37.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:37 smithi022 ceph-mon[31360]: pgmap v6453: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:37 smithi022 ceph-mon[40250]: pgmap v6453: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:37.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:37.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:37.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:37 smithi122 ceph-mon[32255]: pgmap v6453: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:33:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/317247171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/317247171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/317247171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:39 smithi022 ceph-mon[31360]: pgmap v6454: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:39 smithi022 ceph-mon[40250]: pgmap v6454: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:39 smithi122 ceph-mon[32255]: pgmap v6454: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:40] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:33:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:41 smithi022 ceph-mon[31360]: pgmap v6455: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:41 smithi022 ceph-mon[40250]: pgmap v6455: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:41 smithi122 ceph-mon[32255]: pgmap v6455: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:33:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:33:42.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:42.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:42.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:43 smithi022 ceph-mon[31360]: pgmap v6456: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1173574146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:43 smithi022 ceph-mon[40250]: pgmap v6456: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1173574146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:43 smithi122 ceph-mon[32255]: pgmap v6456: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1173574146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:44.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:33:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:33:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:45 smithi022 ceph-mon[31360]: pgmap v6457: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:45 smithi022 ceph-mon[40250]: pgmap v6457: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:45 smithi122 ceph-mon[32255]: pgmap v6457: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:47 smithi022 ceph-mon[31360]: pgmap v6458: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:33:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:47 smithi022 ceph-mon[40250]: pgmap v6458: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:47 smithi122 ceph-mon[32255]: pgmap v6458: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:47.751 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:48.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:48.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:48.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3285911312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3285911312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3285911312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:49 smithi122 ceph-mon[32255]: pgmap v6459: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:49 smithi022 ceph-mon[31360]: pgmap v6459: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:49 smithi022 ceph-mon[40250]: pgmap v6459: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:50] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:33:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:51 smithi122 ceph-mon[32255]: pgmap v6460: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:51 smithi022 ceph-mon[31360]: pgmap v6460: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:51 smithi022 ceph-mon[40250]: pgmap v6460: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:33:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:33:53.123 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:53.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:53.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:53.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:53.499 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:53 smithi022 ceph-mon[31360]: pgmap v6461: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:53.499 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:53 smithi022 ceph-mon[40250]: pgmap v6461: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:53 smithi122 ceph-mon[32255]: pgmap v6461: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:54.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:33:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:33:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2853043014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2853043014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2853043014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:55 smithi122 ceph-mon[32255]: pgmap v6462: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:55 smithi022 ceph-mon[31360]: pgmap v6462: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:55 smithi022 ceph-mon[40250]: pgmap v6462: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:33:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:33:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:57 smithi122 ceph-mon[32255]: pgmap v6463: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:57 smithi022 ceph-mon[31360]: pgmap v6463: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:33:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:57 smithi022 ceph-mon[40250]: pgmap v6463: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:58.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:33:58.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:33:58.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:33:58.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:33:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:59 smithi122 ceph-mon[32255]: pgmap v6464: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:33:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3687138665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:59 smithi022 ceph-mon[31360]: pgmap v6464: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:33:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3687138665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:33:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:59 smithi022 ceph-mon[40250]: pgmap v6464: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:33:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:33:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3687138665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:00] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:34:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:01 smithi122 ceph-mon[32255]: pgmap v6465: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:01 smithi022 ceph-mon[31360]: pgmap v6465: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:01 smithi022 ceph-mon[40250]: pgmap v6465: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:34:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:34:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:03 smithi122 ceph-mon[32255]: pgmap v6466: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:03 smithi022 ceph-mon[31360]: pgmap v6466: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:03 smithi022 ceph-mon[40250]: pgmap v6466: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:03.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:04.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:04.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:04.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4094324445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4094324445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4094324445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:04.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:34:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:34:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:05 smithi122 ceph-mon[32255]: pgmap v6467: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:05 smithi022 ceph-mon[31360]: pgmap v6467: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:05 smithi022 ceph-mon[40250]: pgmap v6467: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:07 smithi122 ceph-mon[32255]: pgmap v6468: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:07 smithi022 ceph-mon[31360]: pgmap v6468: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:34:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:07 smithi022 ceph-mon[40250]: pgmap v6468: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:09.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:09 smithi022 ceph-mon[31360]: pgmap v6469: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:09 smithi022 ceph-mon[40250]: pgmap v6469: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:09.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:09.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:09.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:09 smithi122 ceph-mon[32255]: pgmap v6469: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2368963996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:10] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:34:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2368963996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2368963996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:11 smithi122 ceph-mon[32255]: pgmap v6470: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:11 smithi022 ceph-mon[31360]: pgmap v6470: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:11 smithi022 ceph-mon[40250]: pgmap v6470: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:34:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:34:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:13 smithi122 ceph-mon[32255]: pgmap v6471: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:13 smithi022 ceph-mon[31360]: pgmap v6471: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:13 smithi022 ceph-mon[40250]: pgmap v6471: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:14.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:34:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:34:14.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:14.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:14.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:15 smithi122 ceph-mon[32255]: pgmap v6472: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3625400071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:15 smithi022 ceph-mon[31360]: pgmap v6472: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3625400071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:15 smithi022 ceph-mon[40250]: pgmap v6472: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:15.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:15.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3625400071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:17 smithi122 ceph-mon[32255]: pgmap v6473: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:17 smithi022 ceph-mon[31360]: pgmap v6473: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:17 smithi022 ceph-mon[40250]: pgmap v6473: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:34:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:19 smithi122 ceph-mon[32255]: pgmap v6474: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:19 smithi022 ceph-mon[31360]: pgmap v6474: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:19 smithi022 ceph-mon[40250]: pgmap v6474: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:19.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:20.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:20.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:20.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:20] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:34:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:21 smithi122 ceph-mon[32255]: pgmap v6475: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2986474096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:21 smithi022 ceph-mon[31360]: pgmap v6475: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2986474096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:21 smithi022 ceph-mon[40250]: pgmap v6475: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2986474096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:34:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:34:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:23 smithi122 ceph-mon[32255]: pgmap v6476: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:23 smithi022 ceph-mon[31360]: pgmap v6476: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:23 smithi022 ceph-mon[40250]: pgmap v6476: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:24.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:34:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:34:25.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:25 smithi122 ceph-mon[32255]: pgmap v6477: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:25.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:25.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:25.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:25.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:25 smithi022 ceph-mon[40250]: pgmap v6477: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:25.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:25.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:25 smithi022 ceph-mon[31360]: pgmap v6477: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:25.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/714298790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/714298790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/714298790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:27 smithi122 ceph-mon[32255]: pgmap v6478: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:27 smithi022 ceph-mon[31360]: pgmap v6478: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:34:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:27 smithi022 ceph-mon[40250]: pgmap v6478: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[40250]: pgmap v6479: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:34:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:34:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:34:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[31360]: pgmap v6479: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:34:29.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:34:29.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:34:29.593 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:29 smithi122 ceph-mon[32255]: pgmap v6479: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:34:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:34:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:34:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:34:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:30] "GET /metrics HTTP/1.1" 200 202354 "" "Prometheus/2.18.1" 2021-12-15T21:34:30.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:34:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:34:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:34:31.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:31.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:31.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:31 smithi122 ceph-mon[32255]: pgmap v6480: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4106394087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:31 smithi022 ceph-mon[31360]: pgmap v6480: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4106394087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:31 smithi022 ceph-mon[40250]: pgmap v6480: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4106394087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:34:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:34:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:33 smithi122 ceph-mon[32255]: pgmap v6481: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:34:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:34:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:34:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:34:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[31360]: pgmap v6481: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:34:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:34:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:34:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:34:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[40250]: pgmap v6481: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:34:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:34:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:34:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:34:34.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:34:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:34:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:35 smithi122 ceph-mon[32255]: pgmap v6482: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:35 smithi022 ceph-mon[31360]: pgmap v6482: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:35 smithi022 ceph-mon[40250]: pgmap v6482: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:36.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:36.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:36.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:36.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/944838010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/944838010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/944838010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:37 smithi022 ceph-mon[31360]: pgmap v6483: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:34:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:37 smithi022 ceph-mon[40250]: pgmap v6483: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:37 smithi122 ceph-mon[32255]: pgmap v6483: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:39 smithi122 ceph-mon[32255]: pgmap v6484: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:39 smithi022 ceph-mon[31360]: pgmap v6484: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:39 smithi022 ceph-mon[40250]: pgmap v6484: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:40] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:34:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:41 smithi022 ceph-mon[31360]: pgmap v6485: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:41 smithi022 ceph-mon[40250]: pgmap v6485: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:41.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:41.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:41.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:41 smithi122 ceph-mon[32255]: pgmap v6485: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:42.648 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:34:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:34:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1106232163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1106232163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1106232163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:43 smithi122 ceph-mon[32255]: pgmap v6486: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:43 smithi022 ceph-mon[31360]: pgmap v6486: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:43 smithi022 ceph-mon[40250]: pgmap v6486: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:44.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:34:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:34:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:45 smithi122 ceph-mon[32255]: pgmap v6487: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:45 smithi022 ceph-mon[31360]: pgmap v6487: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:45 smithi022 ceph-mon[40250]: pgmap v6487: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:46.856 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:47.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:47.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:47.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:34:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:47 smithi122 ceph-mon[32255]: pgmap v6488: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/905863786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:47 smithi022 ceph-mon[31360]: pgmap v6488: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/905863786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:47 smithi022 ceph-mon[40250]: pgmap v6488: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/905863786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:49 smithi122 ceph-mon[32255]: pgmap v6489: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:49 smithi022 ceph-mon[31360]: pgmap v6489: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:49 smithi022 ceph-mon[40250]: pgmap v6489: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:50] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:34:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:51 smithi122 ceph-mon[32255]: pgmap v6490: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:51 smithi022 ceph-mon[31360]: pgmap v6490: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:51 smithi022 ceph-mon[40250]: pgmap v6490: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:52.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:52.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:52.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:52.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:52.663 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:34:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:34:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2239915786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2239915786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2239915786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:53 smithi122 ceph-mon[32255]: pgmap v6491: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:53 smithi022 ceph-mon[31360]: pgmap v6491: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:53 smithi022 ceph-mon[40250]: pgmap v6491: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:54.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:34:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:34:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:55 smithi122 ceph-mon[32255]: pgmap v6492: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:55 smithi022 ceph-mon[31360]: pgmap v6492: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:55 smithi022 ceph-mon[40250]: pgmap v6492: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:34:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:34:57.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:34:57.668 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:34:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:57 smithi122 ceph-mon[32255]: pgmap v6493: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:57.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:34:57.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:34:57.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:34:57.963 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:57 smithi022 ceph-mon[31360]: pgmap v6493: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:57.964 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:57 smithi022 ceph-mon[40250]: pgmap v6493: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:34:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2263942290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2263942290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2263942290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:34:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:34:59 smithi122 ceph-mon[32255]: pgmap v6494: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:34:59 smithi022 ceph-mon[31360]: pgmap v6494: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:34:59 smithi022 ceph-mon[40250]: pgmap v6494: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:00] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:35:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:01 smithi122 ceph-mon[32255]: pgmap v6495: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:01 smithi022 ceph-mon[31360]: pgmap v6495: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:01 smithi022 ceph-mon[40250]: pgmap v6495: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:35:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:35:02.964 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:03.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:03.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:03.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:03 smithi122 ceph-mon[32255]: pgmap v6496: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2718252356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:03 smithi022 ceph-mon[31360]: pgmap v6496: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2718252356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:03 smithi022 ceph-mon[40250]: pgmap v6496: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2718252356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:04.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:35:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:35:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:05 smithi122 ceph-mon[32255]: pgmap v6497: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:05 smithi022 ceph-mon[31360]: pgmap v6497: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:05 smithi022 ceph-mon[40250]: pgmap v6497: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:07.685 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:35:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:07 smithi022 ceph-mon[31360]: pgmap v6498: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:07 smithi022 ceph-mon[40250]: pgmap v6498: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:07 smithi122 ceph-mon[32255]: pgmap v6498: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:08.341 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:08.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:08.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:08.710 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:09 smithi022 ceph-mon[31360]: pgmap v6499: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/881301430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:09 smithi022 ceph-mon[40250]: pgmap v6499: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/881301430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:09 smithi122 ceph-mon[32255]: pgmap v6499: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/881301430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:35:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:11 smithi022 ceph-mon[31360]: pgmap v6500: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:11 smithi022 ceph-mon[40250]: pgmap v6500: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:11 smithi122 ceph-mon[32255]: pgmap v6500: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:35:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:35:13.710 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:13 smithi022 ceph-mon[31360]: pgmap v6501: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:13 smithi022 ceph-mon[40250]: pgmap v6501: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:14.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:14.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:14.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:13 smithi122 ceph-mon[32255]: pgmap v6501: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:14.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:35:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:35:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1770342139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1770342139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1770342139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:15 smithi022 ceph-mon[31360]: pgmap v6502: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:15 smithi022 ceph-mon[40250]: pgmap v6502: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:15 smithi122 ceph-mon[32255]: pgmap v6502: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:17.702 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:35:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:17 smithi022 ceph-mon[31360]: pgmap v6503: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:17 smithi022 ceph-mon[40250]: pgmap v6503: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:17 smithi122 ceph-mon[32255]: pgmap v6503: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:19.083 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:19.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:19.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:19.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:19 smithi022 ceph-mon[31360]: pgmap v6504: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/486852908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:19 smithi022 ceph-mon[40250]: pgmap v6504: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/486852908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:19 smithi122 ceph-mon[32255]: pgmap v6504: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/486852908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:20] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:35:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:21 smithi022 ceph-mon[31360]: pgmap v6505: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:21 smithi022 ceph-mon[40250]: pgmap v6505: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:21 smithi122 ceph-mon[32255]: pgmap v6505: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:35:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:35:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:23 smithi022 ceph-mon[31360]: pgmap v6506: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:23 smithi022 ceph-mon[40250]: pgmap v6506: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:23 smithi122 ceph-mon[32255]: pgmap v6506: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:24.453 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:24.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:35:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:35:24.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:24.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:24.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:25 smithi022 ceph-mon[31360]: pgmap v6507: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3470519415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:25 smithi022 ceph-mon[40250]: pgmap v6507: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3470519415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:25 smithi122 ceph-mon[32255]: pgmap v6507: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3470519415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:27.714 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:35:27.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:27 smithi022 ceph-mon[40250]: pgmap v6508: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:27 smithi022 ceph-mon[31360]: pgmap v6508: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:27 smithi122 ceph-mon[32255]: pgmap v6508: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:29.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:29 smithi122 ceph-mon[32255]: pgmap v6509: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:35:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:35:29.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:35:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[40250]: pgmap v6509: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:35:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:35:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:35:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[31360]: pgmap v6509: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:35:30.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:35:30.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:35:30.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:30.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:30.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:30] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:35:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:31 smithi122 ceph-mon[32255]: pgmap v6510: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:35:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3428611850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:31 smithi022 ceph-mon[31360]: pgmap v6510: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:35:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3428611850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:31 smithi022 ceph-mon[40250]: pgmap v6510: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:35:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3428611850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:35:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:33 smithi122 ceph-mon[32255]: pgmap v6511: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:35:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:35:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[31360]: pgmap v6511: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:35:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:35:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:35:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:35:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[40250]: pgmap v6511: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:35:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:35:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:35:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:35:34.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:35:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:35:35.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:35 smithi022 ceph-mon[31360]: pgmap v6512: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:35 smithi022 ceph-mon[40250]: pgmap v6512: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:35.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:35.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:35.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:35 smithi122 ceph-mon[32255]: pgmap v6512: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/194221890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/194221890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/194221890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:37 smithi022 ceph-mon[31360]: pgmap v6513: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:35:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:37 smithi022 ceph-mon[40250]: pgmap v6513: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:37 smithi122 ceph-mon[32255]: pgmap v6513: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:39 smithi022 ceph-mon[31360]: pgmap v6514: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:39 smithi022 ceph-mon[40250]: pgmap v6514: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:39 smithi122 ceph-mon[32255]: pgmap v6514: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:40] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:35:40.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:40.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:40.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:40.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:41 smithi022 ceph-mon[40250]: pgmap v6515: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1707434199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:41 smithi022 ceph-mon[31360]: pgmap v6515: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1707434199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:41 smithi122 ceph-mon[32255]: pgmap v6515: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1707434199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:35:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:35:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:43 smithi022 ceph-mon[31360]: pgmap v6516: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:43 smithi022 ceph-mon[40250]: pgmap v6516: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:43 smithi122 ceph-mon[32255]: pgmap v6516: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:44.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:35:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:35:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:45 smithi022 ceph-mon[31360]: pgmap v6517: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:45 smithi022 ceph-mon[40250]: pgmap v6517: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:45.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:45 smithi122 ceph-mon[32255]: pgmap v6517: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:45.932 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:46.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:46.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:46.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:35:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:47 smithi022 ceph-mon[31360]: pgmap v6518: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:47.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2951660924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:47.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:47 smithi022 ceph-mon[40250]: pgmap v6518: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:47.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2951660924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:47 smithi122 ceph-mon[32255]: pgmap v6518: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2951660924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:49 smithi022 ceph-mon[31360]: pgmap v6519: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:49 smithi022 ceph-mon[40250]: pgmap v6519: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:49 smithi122 ceph-mon[32255]: pgmap v6519: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:50] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:35:51.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:51 smithi022 ceph-mon[31360]: pgmap v6520: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:51 smithi022 ceph-mon[40250]: pgmap v6520: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:51 smithi122 ceph-mon[32255]: pgmap v6520: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/325590217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/325590217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/325590217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:35:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:35:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:53 smithi022 ceph-mon[40250]: pgmap v6521: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:53 smithi022 ceph-mon[31360]: pgmap v6521: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:53 smithi122 ceph-mon[32255]: pgmap v6521: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:54.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:35:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:35:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:55 smithi022 ceph-mon[40250]: pgmap v6522: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:55 smithi022 ceph-mon[31360]: pgmap v6522: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:55 smithi122 ceph-mon[32255]: pgmap v6522: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:35:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:35:56.673 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:35:57.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:35:57.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:35:57.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:35:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:57 smithi022 ceph-mon[31360]: pgmap v6523: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3986431179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:57 smithi022 ceph-mon[40250]: pgmap v6523: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3986431179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:57 smithi122 ceph-mon[32255]: pgmap v6523: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3986431179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:35:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:35:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:35:59 smithi022 ceph-mon[31360]: pgmap v6524: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:35:59 smithi022 ceph-mon[40250]: pgmap v6524: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:35:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:35:59 smithi122 ceph-mon[32255]: pgmap v6524: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:00] "GET /metrics HTTP/1.1" 200 202351 "" "Prometheus/2.18.1" 2021-12-15T21:36:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:01 smithi022 ceph-mon[31360]: pgmap v6525: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:01 smithi022 ceph-mon[40250]: pgmap v6525: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:01 smithi122 ceph-mon[32255]: pgmap v6525: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:02.043 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:02.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:02.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:02.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:36:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:36:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:03 smithi022 ceph-mon[40250]: pgmap v6526: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1527300554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:03 smithi022 ceph-mon[31360]: pgmap v6526: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1527300554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:03 smithi122 ceph-mon[32255]: pgmap v6526: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1527300554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:04.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:36:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:36:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:05 smithi022 ceph-mon[31360]: pgmap v6527: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:05 smithi022 ceph-mon[40250]: pgmap v6527: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:05 smithi122 ceph-mon[32255]: pgmap v6527: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:07.415 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:07 smithi022 ceph-mon[31360]: pgmap v6528: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:36:07.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:07 smithi022 ceph-mon[40250]: pgmap v6528: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:07 smithi122 ceph-mon[32255]: pgmap v6528: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:07.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:07.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:07.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3477728931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3477728931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3477728931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:09 smithi022 ceph-mon[31360]: pgmap v6529: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:09 smithi022 ceph-mon[40250]: pgmap v6529: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:09 smithi122 ceph-mon[32255]: pgmap v6529: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:10] "GET /metrics HTTP/1.1" 200 202349 "" "Prometheus/2.18.1" 2021-12-15T21:36:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:11 smithi022 ceph-mon[31360]: pgmap v6530: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:11 smithi022 ceph-mon[40250]: pgmap v6530: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:11 smithi122 ceph-mon[32255]: pgmap v6530: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:36:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:36:12.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:13.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:13.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:13.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:13 smithi022 ceph-mon[31360]: pgmap v6531: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4207579906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:13 smithi022 ceph-mon[40250]: pgmap v6531: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4207579906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:13 smithi122 ceph-mon[32255]: pgmap v6531: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4207579906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:14.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:36:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:36:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:15 smithi022 ceph-mon[31360]: pgmap v6532: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:15 smithi022 ceph-mon[40250]: pgmap v6532: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:15 smithi122 ceph-mon[32255]: pgmap v6532: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:17 smithi022 ceph-mon[31360]: pgmap v6533: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:36:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:17 smithi022 ceph-mon[40250]: pgmap v6533: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:17 smithi122 ceph-mon[32255]: pgmap v6533: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:18.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:18.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:18.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:18.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:19 smithi022 ceph-mon[31360]: pgmap v6534: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1321278545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:19 smithi022 ceph-mon[40250]: pgmap v6534: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1321278545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:19 smithi122 ceph-mon[32255]: pgmap v6534: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1321278545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:20] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:36:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:21 smithi022 ceph-mon[31360]: pgmap v6535: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:21 smithi022 ceph-mon[40250]: pgmap v6535: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:21 smithi122 ceph-mon[32255]: pgmap v6535: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:36:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:36:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:23 smithi022 ceph-mon[31360]: pgmap v6536: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:23 smithi022 ceph-mon[40250]: pgmap v6536: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:23.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:23 smithi122 ceph-mon[32255]: pgmap v6536: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:23.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:23.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:23.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2891879663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:24.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:36:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:36:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2891879663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2891879663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:25 smithi122 ceph-mon[32255]: pgmap v6537: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:25 smithi022 ceph-mon[31360]: pgmap v6537: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:25 smithi022 ceph-mon[40250]: pgmap v6537: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:27 smithi122 ceph-mon[32255]: pgmap v6538: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:27 smithi022 ceph-mon[31360]: pgmap v6538: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:36:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:27 smithi022 ceph-mon[40250]: pgmap v6538: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:28.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:29.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:29.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:29 smithi022 ceph-mon[31360]: pgmap v6539: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/165180340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:29 smithi022 ceph-mon[40250]: pgmap v6539: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/165180340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:29 smithi122 ceph-mon[32255]: pgmap v6539: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/165180340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:30] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:36:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:36:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:36:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:36:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:36:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:36:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:36:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:36:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:36:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:36:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:31 smithi122 ceph-mon[32255]: pgmap v6540: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:36:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:31 smithi022 ceph-mon[31360]: pgmap v6540: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:36:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:31 smithi022 ceph-mon[40250]: pgmap v6540: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:36:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:36:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:36:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:33 smithi122 ceph-mon[32255]: pgmap v6541: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:36:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:36:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:36:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:36:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[31360]: pgmap v6541: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:36:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:36:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:36:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:36:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[40250]: pgmap v6541: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:36:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:36:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:36:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:36:34.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:34.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:34.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:34.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:34.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:36:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:36:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:35 smithi122 ceph-mon[32255]: pgmap v6542: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:35.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:35.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3197439521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:35 smithi022 ceph-mon[31360]: pgmap v6542: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3197439521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:35 smithi022 ceph-mon[40250]: pgmap v6542: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3197439521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:37 smithi022 ceph-mon[31360]: pgmap v6543: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:36:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:37 smithi022 ceph-mon[40250]: pgmap v6543: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:37 smithi122 ceph-mon[32255]: pgmap v6543: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:39.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:39 smithi122 ceph-mon[32255]: pgmap v6544: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:40.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:40.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:40.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:40.011 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:39 smithi022 ceph-mon[31360]: pgmap v6544: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:40.011 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:39 smithi022 ceph-mon[40250]: pgmap v6544: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:40] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:36:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2926786516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2926786516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2926786516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:41 smithi122 ceph-mon[32255]: pgmap v6545: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:41 smithi022 ceph-mon[31360]: pgmap v6545: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:41 smithi022 ceph-mon[40250]: pgmap v6545: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:36:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:36:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:43 smithi122 ceph-mon[32255]: pgmap v6546: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:43 smithi022 ceph-mon[31360]: pgmap v6546: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:43 smithi022 ceph-mon[40250]: pgmap v6546: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:44.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:36:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:36:45.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:45.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:45.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:45.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:45 smithi022 ceph-mon[31360]: pgmap v6547: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/299020491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:45 smithi022 ceph-mon[40250]: pgmap v6547: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/299020491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:45 smithi122 ceph-mon[32255]: pgmap v6547: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/299020491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:47 smithi022 ceph-mon[31360]: pgmap v6548: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:36:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:47 smithi022 ceph-mon[40250]: pgmap v6548: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:47 smithi122 ceph-mon[32255]: pgmap v6548: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:49 smithi122 ceph-mon[32255]: pgmap v6549: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:49 smithi022 ceph-mon[31360]: pgmap v6549: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:49 smithi022 ceph-mon[40250]: pgmap v6549: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:50.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:50] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:36:50.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:50.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:50.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:51 smithi122 ceph-mon[32255]: pgmap v6550: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4267958311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:51 smithi022 ceph-mon[31360]: pgmap v6550: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4267958311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:51 smithi022 ceph-mon[40250]: pgmap v6550: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4267958311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:36:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:36:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:53 smithi122 ceph-mon[32255]: pgmap v6551: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:53 smithi022 ceph-mon[31360]: pgmap v6551: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:53 smithi022 ceph-mon[40250]: pgmap v6551: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:54.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:36:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:36:55.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:36:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:55 smithi122 ceph-mon[32255]: pgmap v6552: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:55 smithi022 ceph-mon[31360]: pgmap v6552: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:55 smithi022 ceph-mon[40250]: pgmap v6552: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:36:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:36:56.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:36:56.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:36:56.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:36:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3769267122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3769267122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3769267122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:36:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:57 smithi022 ceph-mon[31360]: pgmap v6553: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:36:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:57 smithi022 ceph-mon[40250]: pgmap v6553: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:57 smithi122 ceph-mon[32255]: pgmap v6553: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:36:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:36:59 smithi122 ceph-mon[32255]: pgmap v6554: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:36:59 smithi022 ceph-mon[31360]: pgmap v6554: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:36:59 smithi022 ceph-mon[40250]: pgmap v6554: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:00] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:37:01.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:01.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:01.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:01.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:01 smithi022 ceph-mon[31360]: pgmap v6555: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/538062803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:01 smithi022 ceph-mon[40250]: pgmap v6555: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/538062803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:01 smithi122 ceph-mon[32255]: pgmap v6555: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/538062803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:37:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:37:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:03 smithi122 ceph-mon[32255]: pgmap v6556: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:03 smithi022 ceph-mon[31360]: pgmap v6556: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:03 smithi022 ceph-mon[40250]: pgmap v6556: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:04.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:37:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:37:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:05 smithi122 ceph-mon[32255]: pgmap v6557: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:05 smithi022 ceph-mon[31360]: pgmap v6557: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:05 smithi022 ceph-mon[40250]: pgmap v6557: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:06.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:06.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:06.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:06.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:37:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:07 smithi022 ceph-mon[31360]: pgmap v6558: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4571578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:07 smithi022 ceph-mon[40250]: pgmap v6558: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4571578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:07 smithi122 ceph-mon[32255]: pgmap v6558: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4571578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:09 smithi122 ceph-mon[32255]: pgmap v6559: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:09 smithi022 ceph-mon[31360]: pgmap v6559: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:09 smithi022 ceph-mon[40250]: pgmap v6559: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:37:11.865 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:11 smithi122 ceph-mon[32255]: pgmap v6560: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:11 smithi022 ceph-mon[31360]: pgmap v6560: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:11 smithi022 ceph-mon[40250]: pgmap v6560: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:12.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:12.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:12.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:12.625 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:37:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:37:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2913760503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2913760503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2913760503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:13 smithi122 ceph-mon[32255]: pgmap v6561: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:13 smithi022 ceph-mon[31360]: pgmap v6561: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:13 smithi022 ceph-mon[40250]: pgmap v6561: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:14.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:37:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:37:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:15 smithi122 ceph-mon[32255]: pgmap v6562: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:15 smithi022 ceph-mon[31360]: pgmap v6562: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:15 smithi022 ceph-mon[40250]: pgmap v6562: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:17.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:37:17.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:17.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:17.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:17 smithi122 ceph-mon[32255]: pgmap v6563: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3294233308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:17 smithi022 ceph-mon[31360]: pgmap v6563: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3294233308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:17 smithi022 ceph-mon[40250]: pgmap v6563: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3294233308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:19 smithi122 ceph-mon[32255]: pgmap v6564: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:19 smithi022 ceph-mon[31360]: pgmap v6564: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:19 smithi022 ceph-mon[40250]: pgmap v6564: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:20] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:37:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:21 smithi122 ceph-mon[32255]: pgmap v6565: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:21 smithi022 ceph-mon[31360]: pgmap v6565: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:21 smithi022 ceph-mon[40250]: pgmap v6565: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:22.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:37:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:37:22.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:22.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:22.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:23 smithi122 ceph-mon[32255]: pgmap v6566: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2423777448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:23 smithi022 ceph-mon[31360]: pgmap v6566: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2423777448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:23 smithi022 ceph-mon[40250]: pgmap v6566: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2423777448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:24.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:37:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:37:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:25 smithi122 ceph-mon[32255]: pgmap v6567: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:25 smithi022 ceph-mon[31360]: pgmap v6567: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:25 smithi022 ceph-mon[40250]: pgmap v6567: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:27 smithi022 ceph-mon[31360]: pgmap v6568: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:37:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:27 smithi022 ceph-mon[40250]: pgmap v6568: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:27 smithi122 ceph-mon[32255]: pgmap v6568: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:27.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:28.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:28.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:28.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:29 smithi022 ceph-mon[31360]: pgmap v6569: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1453873503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:29 smithi022 ceph-mon[40250]: pgmap v6569: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1453873503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:29 smithi122 ceph-mon[32255]: pgmap v6569: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1453873503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:30] "GET /metrics HTTP/1.1" 200 202350 "" "Prometheus/2.18.1" 2021-12-15T21:37:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[31360]: pgmap v6570: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:37:31.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:37:31.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:37:31.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:37:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[40250]: pgmap v6570: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:37:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:37:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:37:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:37:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:31 smithi122 ceph-mon[32255]: pgmap v6570: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:37:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:37:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:37:31.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:37:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:37:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:37:33.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[31360]: pgmap v6571: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[40250]: pgmap v6571: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:37:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:37:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:33 smithi122 ceph-mon[32255]: pgmap v6571: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:37:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:37:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:37:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:37:33.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:33.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:33.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2372615698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2372615698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2372615698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:34.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:37:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:37:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:35 smithi022 ceph-mon[31360]: pgmap v6572: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:35 smithi022 ceph-mon[40250]: pgmap v6572: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:35 smithi122 ceph-mon[32255]: pgmap v6572: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:37 smithi022 ceph-mon[31360]: pgmap v6573: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:37:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:37 smithi022 ceph-mon[40250]: pgmap v6573: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:37 smithi122 ceph-mon[32255]: pgmap v6573: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:38.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:39.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:39.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:39.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:39 smithi022 ceph-mon[31360]: pgmap v6574: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4273793546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:39 smithi022 ceph-mon[40250]: pgmap v6574: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4273793546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:39 smithi122 ceph-mon[32255]: pgmap v6574: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:39.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4273793546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:40] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:37:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:41 smithi022 ceph-mon[31360]: pgmap v6575: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:41.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:41 smithi022 ceph-mon[40250]: pgmap v6575: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:41 smithi122 ceph-mon[32255]: pgmap v6575: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:37:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:37:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:43 smithi022 ceph-mon[31360]: pgmap v6576: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:43 smithi022 ceph-mon[40250]: pgmap v6576: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:43 smithi122 ceph-mon[32255]: pgmap v6576: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:44.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:44.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:44.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:44.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:37:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:37:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:45 smithi022 ceph-mon[31360]: pgmap v6577: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2146132255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:45 smithi022 ceph-mon[40250]: pgmap v6577: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:45.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2146132255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:45 smithi122 ceph-mon[32255]: pgmap v6577: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2146132255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:47 smithi022 ceph-mon[31360]: pgmap v6578: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:37:47.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:47 smithi022 ceph-mon[40250]: pgmap v6578: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:47 smithi122 ceph-mon[32255]: pgmap v6578: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:49.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:49 smithi022 ceph-mon[31360]: pgmap v6579: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:49 smithi022 ceph-mon[40250]: pgmap v6579: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:49 smithi122 ceph-mon[32255]: pgmap v6579: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:49.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:50.257 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2440662196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:50] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T21:37:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2440662196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2440662196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:51 smithi022 ceph-mon[31360]: pgmap v6580: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:51 smithi022 ceph-mon[40250]: pgmap v6580: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:51 smithi122 ceph-mon[32255]: pgmap v6580: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:37:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:37:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:37:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:53 smithi022 ceph-mon[31360]: pgmap v6581: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:53 smithi022 ceph-mon[40250]: pgmap v6581: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:53 smithi122 ceph-mon[32255]: pgmap v6581: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:37:54.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:37:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:37:54.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:37:55.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:37:55.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:37:55.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:37:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:55 smithi022 ceph-mon[31360]: pgmap v6582: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1711274870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:55 smithi022 ceph-mon[40250]: pgmap v6582: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:55.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1711274870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:55 smithi122 ceph-mon[32255]: pgmap v6582: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:37:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1711274870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:37:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:57 smithi022 ceph-mon[31360]: pgmap v6583: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:37:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:57 smithi022 ceph-mon[40250]: pgmap v6583: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:57 smithi122 ceph-mon[32255]: pgmap v6583: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:37:59 smithi022 ceph-mon[31360]: pgmap v6584: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:37:59 smithi022 ceph-mon[40250]: pgmap v6584: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:37:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:37:59 smithi122 ceph-mon[32255]: pgmap v6584: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:38:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:00] "GET /metrics HTTP/1.1" 200 202347 "" "Prometheus/2.18.1" 2021-12-15T21:38:00.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:00.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:00.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:01 smithi022 ceph-mon[31360]: pgmap v6585: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:38:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/393028883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:01 smithi022 ceph-mon[40250]: pgmap v6585: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:38:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/393028883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:01 smithi122 ceph-mon[32255]: pgmap v6585: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:38:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/393028883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:38:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:38:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:03 smithi022 ceph-mon[31360]: pgmap v6586: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:38:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:03 smithi022 ceph-mon[40250]: pgmap v6586: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:38:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:03 smithi122 ceph-mon[32255]: pgmap v6586: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:38:04.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:38:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:38:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:05 smithi022 ceph-mon[31360]: pgmap v6587: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:38:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:05 smithi022 ceph-mon[40250]: pgmap v6587: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:38:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:05.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:05 smithi122 ceph-mon[32255]: pgmap v6587: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:38:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:05.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:05.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:05.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1267529608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:06.913 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1267529608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:06.913 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1267529608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:07 smithi122 ceph-mon[32255]: pgmap v6588: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:07 smithi022 ceph-mon[31360]: pgmap v6588: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:38:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:07 smithi022 ceph-mon[40250]: pgmap v6588: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:09 smithi122 ceph-mon[32255]: pgmap v6589: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:09 smithi022 ceph-mon[31360]: pgmap v6589: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:09 smithi022 ceph-mon[40250]: pgmap v6589: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:10] "GET /metrics HTTP/1.1" 200 202352 "" "Prometheus/2.18.1" 2021-12-15T21:38:10.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:11.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:11.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:11.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:11 smithi122 ceph-mon[32255]: pgmap v6590: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:38:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1341923923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:11 smithi022 ceph-mon[31360]: pgmap v6590: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:38:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1341923923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:11 smithi022 ceph-mon[40250]: pgmap v6590: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:38:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1341923923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:38:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:38:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:13 smithi122 ceph-mon[32255]: pgmap v6591: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:13 smithi022 ceph-mon[31360]: pgmap v6591: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:13 smithi022 ceph-mon[40250]: pgmap v6591: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:38:14.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:38:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:38:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:15 smithi122 ceph-mon[32255]: pgmap v6592: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:38:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:15 smithi022 ceph-mon[31360]: pgmap v6592: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:38:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:15 smithi022 ceph-mon[40250]: pgmap v6592: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:38:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:16.302 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:16.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:16.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:16.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:17 smithi122 ceph-mon[32255]: pgmap v6593: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3209113656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:38:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:17 smithi022 ceph-mon[31360]: pgmap v6593: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3209113656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:17 smithi022 ceph-mon[40250]: pgmap v6593: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3209113656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:19 smithi122 ceph-mon[32255]: pgmap v6594: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:19 smithi022 ceph-mon[31360]: pgmap v6594: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:19 smithi022 ceph-mon[40250]: pgmap v6594: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:20] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:38:21.669 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:21.670 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:21 smithi022 ceph-mon[31360]: pgmap v6595: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:21.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:21 smithi022 ceph-mon[40250]: pgmap v6595: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:21 smithi122 ceph-mon[32255]: pgmap v6595: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:22.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:22.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:22.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2237933847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:38:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:38:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2237933847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2237933847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:23 smithi122 ceph-mon[32255]: pgmap v6596: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:23 smithi022 ceph-mon[31360]: pgmap v6596: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:23 smithi022 ceph-mon[40250]: pgmap v6596: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:24.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:38:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:38:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:25 smithi122 ceph-mon[32255]: pgmap v6597: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:25 smithi022 ceph-mon[31360]: pgmap v6597: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:25 smithi022 ceph-mon[40250]: pgmap v6597: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:27.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:27.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:27.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:27 smithi122 ceph-mon[32255]: pgmap v6598: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:38:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:27 smithi022 ceph-mon[31360]: pgmap v6598: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:27 smithi022 ceph-mon[40250]: pgmap v6598: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3652185059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3652185059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3652185059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:29 smithi122 ceph-mon[32255]: pgmap v6599: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:29 smithi022 ceph-mon[31360]: pgmap v6599: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:29 smithi022 ceph-mon[40250]: pgmap v6599: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:30] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:38:31.437 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[40250]: pgmap v6600: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:31.438 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:38:31.438 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:38:31.438 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:38:31.438 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[31360]: pgmap v6600: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:31.439 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:38:31.439 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:38:31.439 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:38:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:31 smithi122 ceph-mon[32255]: pgmap v6600: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:38:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:38:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:38:32.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:38:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:38:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:32.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:32.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:32.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:38:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:33 smithi122 ceph-mon[32255]: pgmap v6601: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1089064838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:38:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:38:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:38:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:38:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[40250]: pgmap v6601: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1089064838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:38:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:38:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:38:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:38:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[31360]: pgmap v6601: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1089064838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:38:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:38:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:38:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:38:34.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:38:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:38:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:35 smithi122 ceph-mon[32255]: pgmap v6602: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:35 smithi022 ceph-mon[31360]: pgmap v6602: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:35 smithi022 ceph-mon[40250]: pgmap v6602: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:37 smithi122 ceph-mon[32255]: pgmap v6603: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:37.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:38:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:37 smithi022 ceph-mon[40250]: pgmap v6603: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:37 smithi022 ceph-mon[31360]: pgmap v6603: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:38.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:38.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:38.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2110510129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2110510129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2110510129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:39 smithi122 ceph-mon[32255]: pgmap v6604: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:39 smithi022 ceph-mon[31360]: pgmap v6604: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:39 smithi022 ceph-mon[40250]: pgmap v6604: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:40] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:38:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:41 smithi122 ceph-mon[32255]: pgmap v6605: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:41 smithi022 ceph-mon[31360]: pgmap v6605: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:41 smithi022 ceph-mon[40250]: pgmap v6605: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:38:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:38:43.150 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:43.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:43.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:43.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:43 smithi122 ceph-mon[32255]: pgmap v6606: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:43 smithi022 ceph-mon[31360]: pgmap v6606: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:43 smithi022 ceph-mon[40250]: pgmap v6606: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:44.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:38:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:38:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/721757694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/721757694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/721757694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:45 smithi022 ceph-mon[31360]: pgmap v6607: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:45 smithi022 ceph-mon[40250]: pgmap v6607: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:45 smithi122 ceph-mon[32255]: pgmap v6607: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:47 smithi022 ceph-mon[31360]: pgmap v6608: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:38:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:47 smithi022 ceph-mon[40250]: pgmap v6608: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:47 smithi122 ceph-mon[32255]: pgmap v6608: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:48.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:48.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:48.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:48.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:49 smithi022 ceph-mon[31360]: pgmap v6609: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1288030921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:49 smithi022 ceph-mon[40250]: pgmap v6609: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1288030921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:49 smithi122 ceph-mon[32255]: pgmap v6609: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1288030921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:50] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:38:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:51 smithi022 ceph-mon[31360]: pgmap v6610: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:51 smithi022 ceph-mon[40250]: pgmap v6610: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:51 smithi122 ceph-mon[32255]: pgmap v6610: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:38:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:38:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:53 smithi022 ceph-mon[31360]: pgmap v6611: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:53 smithi022 ceph-mon[40250]: pgmap v6611: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:53.893 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:53 smithi122 ceph-mon[32255]: pgmap v6611: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:54.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:54.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:54.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1825482936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1825482936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:54.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:38:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:38:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1825482936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:38:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:55 smithi022 ceph-mon[31360]: pgmap v6612: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:55 smithi022 ceph-mon[40250]: pgmap v6612: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:55 smithi122 ceph-mon[32255]: pgmap v6612: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:38:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:38:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:57 smithi022 ceph-mon[31360]: pgmap v6613: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:38:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:57 smithi022 ceph-mon[40250]: pgmap v6613: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:57.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:57 smithi122 ceph-mon[32255]: pgmap v6613: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:59.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:38:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:38:59 smithi022 ceph-mon[31360]: pgmap v6614: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:38:59 smithi022 ceph-mon[40250]: pgmap v6614: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:38:59.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:38:59.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:38:59.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:38:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:38:59 smithi122 ceph-mon[32255]: pgmap v6614: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:00] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:39:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2643929505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2643929505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2643929505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:01 smithi022 ceph-mon[31360]: pgmap v6615: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:01 smithi022 ceph-mon[40250]: pgmap v6615: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:01 smithi122 ceph-mon[32255]: pgmap v6615: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:39:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:39:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:03 smithi022 ceph-mon[31360]: pgmap v6616: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:03 smithi022 ceph-mon[40250]: pgmap v6616: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:03 smithi122 ceph-mon[32255]: pgmap v6616: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:04.635 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:04.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:39:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:39:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:05 smithi022 ceph-mon[31360]: pgmap v6617: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2610253308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:05 smithi022 ceph-mon[40250]: pgmap v6617: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2610253308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:05 smithi122 ceph-mon[32255]: pgmap v6617: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2610253308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:07 smithi022 ceph-mon[31360]: pgmap v6618: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:39:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:07 smithi022 ceph-mon[40250]: pgmap v6618: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:07 smithi122 ceph-mon[32255]: pgmap v6618: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:09 smithi022 ceph-mon[31360]: pgmap v6619: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:09 smithi022 ceph-mon[40250]: pgmap v6619: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:09 smithi122 ceph-mon[32255]: pgmap v6619: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:10.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:10.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:10.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:10.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:10] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:39:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/241436197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/241436197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/241436197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:11 smithi022 ceph-mon[31360]: pgmap v6620: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:11 smithi022 ceph-mon[40250]: pgmap v6620: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:11 smithi122 ceph-mon[32255]: pgmap v6620: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:39:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:39:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:13 smithi022 ceph-mon[31360]: pgmap v6621: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:13 smithi022 ceph-mon[40250]: pgmap v6621: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:13 smithi122 ceph-mon[32255]: pgmap v6621: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:14.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:39:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:39:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:15.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:15.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:15.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:15.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:15 smithi022 ceph-mon[40250]: pgmap v6622: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:15.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:15.747 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:15 smithi022 ceph-mon[31360]: pgmap v6622: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:15.748 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:15 smithi122 ceph-mon[32255]: pgmap v6622: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/923178039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/923178039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/923178039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:17 smithi022 ceph-mon[31360]: pgmap v6623: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:39:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:17 smithi022 ceph-mon[40250]: pgmap v6623: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:17 smithi122 ceph-mon[32255]: pgmap v6623: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:19 smithi022 ceph-mon[31360]: pgmap v6624: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:19 smithi022 ceph-mon[40250]: pgmap v6624: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:19 smithi122 ceph-mon[32255]: pgmap v6624: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:20] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:39:20.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:21.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:21.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:21.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:21 smithi022 ceph-mon[31360]: pgmap v6625: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1490082155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:21 smithi022 ceph-mon[40250]: pgmap v6625: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1490082155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:21 smithi122 ceph-mon[32255]: pgmap v6625: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1490082155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:39:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:39:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:23 smithi022 ceph-mon[31360]: pgmap v6626: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:23 smithi022 ceph-mon[40250]: pgmap v6626: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:23 smithi122 ceph-mon[32255]: pgmap v6626: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:24.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:39:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:39:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:25 smithi022 ceph-mon[31360]: pgmap v6627: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:25 smithi022 ceph-mon[40250]: pgmap v6627: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:25 smithi122 ceph-mon[32255]: pgmap v6627: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:26.115 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:26.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:26.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:26.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/86951620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/86951620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/86951620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:27 smithi022 ceph-mon[31360]: pgmap v6628: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:39:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:27 smithi022 ceph-mon[40250]: pgmap v6628: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:27 smithi122 ceph-mon[32255]: pgmap v6628: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:29 smithi022 ceph-mon[31360]: pgmap v6629: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:29 smithi022 ceph-mon[40250]: pgmap v6629: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:29 smithi122 ceph-mon[32255]: pgmap v6629: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:30] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:39:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:31 smithi022 ceph-mon[31360]: pgmap v6630: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:31 smithi022 ceph-mon[40250]: pgmap v6630: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:31.904 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:31 smithi122 ceph-mon[32255]: pgmap v6630: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:39:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:39:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2920159923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:39:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:39:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:39:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:39:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2920159923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:39:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:39:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:39:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:39:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2920159923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:39:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:39:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:39:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:39:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:33 smithi122 ceph-mon[32255]: pgmap v6631: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:39:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:39:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:39:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:39:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[31360]: pgmap v6631: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:39:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:39:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:39:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:39:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[40250]: pgmap v6631: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:39:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:39:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:39:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:39:34.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:39:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:39:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:35 smithi122 ceph-mon[32255]: pgmap v6632: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:35 smithi022 ceph-mon[31360]: pgmap v6632: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:35 smithi022 ceph-mon[40250]: pgmap v6632: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:36.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:39:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:37 smithi122 ceph-mon[32255]: pgmap v6633: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3362130930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:37 smithi022 ceph-mon[31360]: pgmap v6633: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3362130930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:37 smithi022 ceph-mon[40250]: pgmap v6633: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3362130930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:39 smithi122 ceph-mon[32255]: pgmap v6634: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:39 smithi022 ceph-mon[31360]: pgmap v6634: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:39 smithi022 ceph-mon[40250]: pgmap v6634: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:40] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:39:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:41 smithi122 ceph-mon[32255]: pgmap v6635: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:41 smithi022 ceph-mon[31360]: pgmap v6635: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:41 smithi022 ceph-mon[40250]: pgmap v6635: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:42.228 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:42.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:42.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:42.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:39:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:39:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:43 smithi122 ceph-mon[32255]: pgmap v6636: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1148675896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:43 smithi022 ceph-mon[31360]: pgmap v6636: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:44.572 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1148675896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:44.572 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:43 smithi022 ceph-mon[40250]: pgmap v6636: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:44.573 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1148675896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:44.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:39:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:39:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:45 smithi122 ceph-mon[32255]: pgmap v6637: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:45 smithi022 ceph-mon[31360]: pgmap v6637: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:45 smithi022 ceph-mon[40250]: pgmap v6637: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:47.601 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:47 smithi022 ceph-mon[31360]: pgmap v6638: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:39:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:47 smithi022 ceph-mon[40250]: pgmap v6638: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:47 smithi122 ceph-mon[32255]: pgmap v6638: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:47.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:47.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:47.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1476112909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1476112909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1476112909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:49 smithi122 ceph-mon[32255]: pgmap v6639: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:49 smithi022 ceph-mon[31360]: pgmap v6639: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:49 smithi022 ceph-mon[40250]: pgmap v6639: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:50] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:39:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:51 smithi122 ceph-mon[32255]: pgmap v6640: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:51 smithi022 ceph-mon[31360]: pgmap v6640: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:51 smithi022 ceph-mon[40250]: pgmap v6640: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:39:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:39:52.970 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:53.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:53.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:53.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:53 smithi022 ceph-mon[31360]: pgmap v6641: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2563618694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:53 smithi022 ceph-mon[40250]: pgmap v6641: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2563618694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:53 smithi122 ceph-mon[32255]: pgmap v6641: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2563618694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:39:54.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:39:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:39:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:55 smithi122 ceph-mon[32255]: pgmap v6642: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:56.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:55 smithi022 ceph-mon[31360]: pgmap v6642: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:56.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:55 smithi022 ceph-mon[40250]: pgmap v6642: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:39:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:39:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:57 smithi022 ceph-mon[31360]: pgmap v6643: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:39:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:57 smithi022 ceph-mon[40250]: pgmap v6643: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:57 smithi122 ceph-mon[32255]: pgmap v6643: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:58.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:39:58.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:39:58.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:39:58.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:39:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:59 smithi122 ceph-mon[32255]: pgmap v6644: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:39:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:39:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3908255089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:59 smithi022 ceph-mon[31360]: pgmap v6644: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:39:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3908255089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:59 smithi022 ceph-mon[40250]: pgmap v6644: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:39:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3908255089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:00] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:40:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T21:40:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T21:40:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T21:40:01.872 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:01 smithi122 ceph-mon[32255]: pgmap v6645: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:01 smithi022 ceph-mon[31360]: pgmap v6645: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:01 smithi022 ceph-mon[40250]: pgmap v6645: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:40:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:40:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:03 smithi122 ceph-mon[32255]: pgmap v6646: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:03 smithi022 ceph-mon[31360]: pgmap v6646: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:03 smithi022 ceph-mon[40250]: pgmap v6646: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:04.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:04.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:04.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4066481795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:04.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:40:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:40:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4066481795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4066481795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:05 smithi122 ceph-mon[32255]: pgmap v6647: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:05 smithi022 ceph-mon[31360]: pgmap v6647: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:05 smithi022 ceph-mon[40250]: pgmap v6647: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:07 smithi022 ceph-mon[31360]: pgmap v6648: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:40:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:07 smithi022 ceph-mon[40250]: pgmap v6648: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:07 smithi122 ceph-mon[32255]: pgmap v6648: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:09.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:09 smithi022 ceph-mon[31360]: pgmap v6649: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:09.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2794046657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:09 smithi022 ceph-mon[40250]: pgmap v6649: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2794046657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:09 smithi122 ceph-mon[32255]: pgmap v6649: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2794046657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:10] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:40:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:11 smithi122 ceph-mon[32255]: pgmap v6650: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:11 smithi022 ceph-mon[31360]: pgmap v6650: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:11 smithi022 ceph-mon[40250]: pgmap v6650: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:40:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:40:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:13 smithi122 ceph-mon[32255]: pgmap v6651: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:13 smithi022 ceph-mon[31360]: pgmap v6651: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:13 smithi022 ceph-mon[40250]: pgmap v6651: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:14.459 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:14.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:14.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:14.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:14.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:40:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:40:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:15 smithi122 ceph-mon[32255]: pgmap v6652: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3705251760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:15 smithi022 ceph-mon[40250]: pgmap v6652: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3705251760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:15 smithi022 ceph-mon[31360]: pgmap v6652: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:16.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3705251760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:17 smithi022 ceph-mon[31360]: pgmap v6653: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:40:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:17 smithi022 ceph-mon[40250]: pgmap v6653: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:17 smithi122 ceph-mon[32255]: pgmap v6653: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:19.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:19 smithi122 ceph-mon[32255]: pgmap v6654: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:19 smithi022 ceph-mon[31360]: pgmap v6654: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:19 smithi022 ceph-mon[40250]: pgmap v6654: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:20.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:20.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:20.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:20] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:40:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1429360482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1429360482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1429360482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:21 smithi122 ceph-mon[32255]: pgmap v6655: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:21 smithi022 ceph-mon[31360]: pgmap v6655: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:21 smithi022 ceph-mon[40250]: pgmap v6655: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:40:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:40:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:23 smithi122 ceph-mon[32255]: pgmap v6656: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:23 smithi022 ceph-mon[31360]: pgmap v6656: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:23 smithi022 ceph-mon[40250]: pgmap v6656: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:24.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:40:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:40:25.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:25.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:25 smithi122 ceph-mon[32255]: pgmap v6657: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/10629017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:25 smithi022 ceph-mon[40250]: pgmap v6657: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/10629017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:25 smithi022 ceph-mon[31360]: pgmap v6657: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/10629017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:27 smithi022 ceph-mon[31360]: pgmap v6658: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:40:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:27 smithi022 ceph-mon[40250]: pgmap v6658: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:27 smithi122 ceph-mon[32255]: pgmap v6658: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:29 smithi122 ceph-mon[32255]: pgmap v6659: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:29 smithi022 ceph-mon[31360]: pgmap v6659: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:29 smithi022 ceph-mon[40250]: pgmap v6659: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:30] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:40:30.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:30.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:30.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:30.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:31 smithi122 ceph-mon[32255]: pgmap v6660: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/488836115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:31 smithi022 ceph-mon[31360]: pgmap v6660: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/488836115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:31 smithi022 ceph-mon[40250]: pgmap v6660: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/488836115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:32.630 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:40:32.630 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:40:32.630 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:40:32.631 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:40:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:40:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:40:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:40:32.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:40:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:40:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:40:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:40:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:33 smithi122 ceph-mon[32255]: pgmap v6661: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:40:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:40:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:40:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:40:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:40:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[31360]: pgmap v6661: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:40:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:40:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:40:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:40:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:40:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[40250]: pgmap v6661: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:40:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:40:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:40:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:40:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:40:34.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:40:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:40:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:35 smithi122 ceph-mon[32255]: pgmap v6662: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:35 smithi022 ceph-mon[31360]: pgmap v6662: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:35 smithi022 ceph-mon[40250]: pgmap v6662: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:36.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:36.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:36.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1844549165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1844549165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1844549165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:40:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:37 smithi122 ceph-mon[32255]: pgmap v6663: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:37 smithi022 ceph-mon[31360]: pgmap v6663: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:37 smithi022 ceph-mon[40250]: pgmap v6663: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:39 smithi122 ceph-mon[32255]: pgmap v6664: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:39 smithi022 ceph-mon[31360]: pgmap v6664: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:39 smithi022 ceph-mon[40250]: pgmap v6664: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:40] "GET /metrics HTTP/1.1" 200 202355 "" "Prometheus/2.18.1" 2021-12-15T21:40:41.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:41.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:41.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:41.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:41 smithi022 ceph-mon[31360]: pgmap v6665: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2876918552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:41 smithi022 ceph-mon[40250]: pgmap v6665: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2876918552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:41 smithi122 ceph-mon[32255]: pgmap v6665: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2876918552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:40:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:40:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:43 smithi122 ceph-mon[32255]: pgmap v6666: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:43 smithi022 ceph-mon[31360]: pgmap v6666: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:43 smithi022 ceph-mon[40250]: pgmap v6666: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:44.867 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:40:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:40:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:44 smithi122 ceph-mon[32255]: pgmap v6667: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:44 smithi022 ceph-mon[31360]: pgmap v6667: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:44 smithi022 ceph-mon[40250]: pgmap v6667: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:46.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:47.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:47.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:47.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:47 smithi022 ceph-mon[31360]: pgmap v6668: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:47.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2121487747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:47 smithi022 ceph-mon[40250]: pgmap v6668: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2121487747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:47 smithi122 ceph-mon[32255]: pgmap v6668: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2121487747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:40:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:49 smithi022 ceph-mon[31360]: pgmap v6669: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:49 smithi022 ceph-mon[40250]: pgmap v6669: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:49 smithi122 ceph-mon[32255]: pgmap v6669: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:50] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:40:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:51 smithi022 ceph-mon[31360]: pgmap v6670: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:51 smithi022 ceph-mon[40250]: pgmap v6670: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:51 smithi122 ceph-mon[32255]: pgmap v6670: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:52.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:52.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:52.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:52.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:40:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:40:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:53 smithi022 ceph-mon[31360]: pgmap v6671: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1968363385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:53 smithi022 ceph-mon[40250]: pgmap v6671: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1968363385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:53 smithi122 ceph-mon[32255]: pgmap v6671: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1968363385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:54.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:40:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:40:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:55 smithi022 ceph-mon[31360]: pgmap v6672: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:55 smithi022 ceph-mon[40250]: pgmap v6672: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:55 smithi122 ceph-mon[32255]: pgmap v6672: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:40:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:40:57.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:40:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:57 smithi022 ceph-mon[31360]: pgmap v6673: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:40:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:57 smithi022 ceph-mon[40250]: pgmap v6673: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:57 smithi122 ceph-mon[32255]: pgmap v6673: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:57.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:40:57.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:40:57.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:40:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/976762803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/976762803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/976762803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:40:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:40:59 smithi122 ceph-mon[32255]: pgmap v6674: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:40:59 smithi022 ceph-mon[31360]: pgmap v6674: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:40:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:40:59 smithi022 ceph-mon[40250]: pgmap v6674: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:00] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:41:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:01 smithi122 ceph-mon[32255]: pgmap v6675: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:01 smithi022 ceph-mon[31360]: pgmap v6675: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:01 smithi022 ceph-mon[40250]: pgmap v6675: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:41:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:41:02.803 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:03.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:03 smithi022 ceph-mon[31360]: pgmap v6676: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1753284340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:03 smithi022 ceph-mon[40250]: pgmap v6676: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1753284340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:03 smithi122 ceph-mon[32255]: pgmap v6676: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1753284340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:04.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:41:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:41:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:05 smithi122 ceph-mon[32255]: pgmap v6677: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:05 smithi022 ceph-mon[31360]: pgmap v6677: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:05 smithi022 ceph-mon[40250]: pgmap v6677: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:07 smithi122 ceph-mon[32255]: pgmap v6678: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:07 smithi022 ceph-mon[31360]: pgmap v6678: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:41:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:07 smithi022 ceph-mon[40250]: pgmap v6678: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:08.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:08.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:08.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:08.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:09 smithi122 ceph-mon[32255]: pgmap v6679: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/526363744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:09 smithi022 ceph-mon[31360]: pgmap v6679: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/526363744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:09 smithi022 ceph-mon[40250]: pgmap v6679: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/526363744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:10] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:41:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:11 smithi122 ceph-mon[32255]: pgmap v6680: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:11 smithi022 ceph-mon[31360]: pgmap v6680: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:11 smithi022 ceph-mon[40250]: pgmap v6680: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:41:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:41:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:13 smithi122 ceph-mon[32255]: pgmap v6681: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:13 smithi022 ceph-mon[31360]: pgmap v6681: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:13 smithi022 ceph-mon[40250]: pgmap v6681: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:13.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:13.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:13.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:14.558 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3543155111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3543155111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3543155111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:14.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:41:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:41:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:15 smithi122 ceph-mon[32255]: pgmap v6682: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:15 smithi022 ceph-mon[31360]: pgmap v6682: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:15 smithi022 ceph-mon[40250]: pgmap v6682: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:17 smithi122 ceph-mon[32255]: pgmap v6683: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:17 smithi022 ceph-mon[31360]: pgmap v6683: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:41:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:17 smithi022 ceph-mon[40250]: pgmap v6683: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:18.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:19.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:19.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:19.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:19 smithi122 ceph-mon[32255]: pgmap v6684: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:19.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3763993318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:19 smithi022 ceph-mon[31360]: pgmap v6684: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3763993318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:19 smithi022 ceph-mon[40250]: pgmap v6684: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3763993318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:20] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:41:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:21 smithi122 ceph-mon[32255]: pgmap v6685: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:21 smithi022 ceph-mon[31360]: pgmap v6685: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:21 smithi022 ceph-mon[40250]: pgmap v6685: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:41:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:41:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:23 smithi122 ceph-mon[32255]: pgmap v6686: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:23 smithi022 ceph-mon[31360]: pgmap v6686: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:23 smithi022 ceph-mon[40250]: pgmap v6686: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:24.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:24.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:24.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:24.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:24.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:41:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:41:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:25 smithi122 ceph-mon[32255]: pgmap v6687: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/140191503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:25 smithi022 ceph-mon[31360]: pgmap v6687: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/140191503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:25 smithi022 ceph-mon[40250]: pgmap v6687: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/140191503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:27 smithi122 ceph-mon[32255]: pgmap v6688: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:27 smithi022 ceph-mon[31360]: pgmap v6688: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:41:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:27 smithi022 ceph-mon[40250]: pgmap v6688: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:29.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:29.655 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:29 smithi022 ceph-mon[31360]: pgmap v6689: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:29.655 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:29 smithi022 ceph-mon[40250]: pgmap v6689: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:29 smithi122 ceph-mon[32255]: pgmap v6689: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:30.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:30.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:30.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:30] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:41:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/439305482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/439305482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/439305482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:31 smithi122 ceph-mon[32255]: pgmap v6690: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:31 smithi022 ceph-mon[31360]: pgmap v6690: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:31 smithi022 ceph-mon[40250]: pgmap v6690: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:41:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:41:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: pgmap v6691: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:41:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:41:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:41:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:41:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: pgmap v6691: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:41:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:41:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:41:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:41:33.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:41:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: pgmap v6691: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:41:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:41:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:41:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:41:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:41:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:41:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:41:33.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:41:34.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:41:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:41:35.026 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:35.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:35.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:35.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:35 smithi122 ceph-mon[32255]: pgmap v6692: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:35 smithi022 ceph-mon[31360]: pgmap v6692: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:35 smithi022 ceph-mon[40250]: pgmap v6692: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2181398144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2181398144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2181398144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:37 smithi122 ceph-mon[32255]: pgmap v6693: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:37 smithi022 ceph-mon[31360]: pgmap v6693: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:41:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:37 smithi022 ceph-mon[40250]: pgmap v6693: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:39 smithi122 ceph-mon[32255]: pgmap v6694: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:39 smithi022 ceph-mon[31360]: pgmap v6694: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:39 smithi022 ceph-mon[40250]: pgmap v6694: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:40.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:40] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:41:40.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:40.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:40.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:41 smithi122 ceph-mon[32255]: pgmap v6695: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3839511958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:41 smithi022 ceph-mon[31360]: pgmap v6695: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3839511958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:41 smithi022 ceph-mon[40250]: pgmap v6695: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3839511958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:41:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:41:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:43 smithi122 ceph-mon[32255]: pgmap v6696: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:43 smithi022 ceph-mon[31360]: pgmap v6696: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:43 smithi022 ceph-mon[40250]: pgmap v6696: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:44.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:41:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:41:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:45 smithi122 ceph-mon[32255]: pgmap v6697: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:45.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:45.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:45 smithi022 ceph-mon[31360]: pgmap v6697: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:45.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:45.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:45 smithi022 ceph-mon[40250]: pgmap v6697: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:45.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:46.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:46.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:46.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1779972711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1779972711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1779972711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:47 smithi122 ceph-mon[32255]: pgmap v6698: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:41:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:47 smithi022 ceph-mon[31360]: pgmap v6698: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:47 smithi022 ceph-mon[40250]: pgmap v6698: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:49 smithi122 ceph-mon[32255]: pgmap v6699: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:49 smithi022 ceph-mon[31360]: pgmap v6699: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:49 smithi022 ceph-mon[40250]: pgmap v6699: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:50] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:41:51.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:51.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:51.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:51.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:51.504 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:51 smithi022 ceph-mon[40250]: pgmap v6700: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:51.504 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:51 smithi022 ceph-mon[31360]: pgmap v6700: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:51 smithi122 ceph-mon[32255]: pgmap v6700: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1028433160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:52.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:41:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:41:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1028433160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1028433160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:53 smithi122 ceph-mon[32255]: pgmap v6701: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:53 smithi022 ceph-mon[31360]: pgmap v6701: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:53 smithi022 ceph-mon[40250]: pgmap v6701: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:54.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:41:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:41:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:55 smithi122 ceph-mon[32255]: pgmap v6702: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:55 smithi022 ceph-mon[31360]: pgmap v6702: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:55 smithi022 ceph-mon[40250]: pgmap v6702: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:41:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:41:56.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:41:56.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:41:56.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:41:56.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:41:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:57 smithi122 ceph-mon[32255]: pgmap v6703: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2031830024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:57 smithi022 ceph-mon[31360]: pgmap v6703: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2031830024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:41:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:57 smithi022 ceph-mon[40250]: pgmap v6703: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2031830024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:41:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:41:59 smithi122 ceph-mon[32255]: pgmap v6704: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:41:59 smithi022 ceph-mon[31360]: pgmap v6704: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:41:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:41:59 smithi022 ceph-mon[40250]: pgmap v6704: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:00] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:42:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:01 smithi122 ceph-mon[32255]: pgmap v6705: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:01 smithi022 ceph-mon[31360]: pgmap v6705: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:01 smithi022 ceph-mon[40250]: pgmap v6705: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:01.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:02.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:02.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:02.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2454183800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2454183800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2454183800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:42:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:42:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:03 smithi122 ceph-mon[32255]: pgmap v6706: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:03 smithi022 ceph-mon[31360]: pgmap v6706: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:03 smithi022 ceph-mon[40250]: pgmap v6706: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:04.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:42:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:42:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:05 smithi122 ceph-mon[32255]: pgmap v6707: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:05 smithi022 ceph-mon[31360]: pgmap v6707: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:05 smithi022 ceph-mon[40250]: pgmap v6707: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:07.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:07 smithi022 ceph-mon[31360]: pgmap v6708: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:42:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:07 smithi022 ceph-mon[40250]: pgmap v6708: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:07.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:07.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:07.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:07 smithi122 ceph-mon[32255]: pgmap v6708: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3202488904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3202488904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3202488904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:09 smithi122 ceph-mon[32255]: pgmap v6709: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:09 smithi022 ceph-mon[31360]: pgmap v6709: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:09 smithi022 ceph-mon[40250]: pgmap v6709: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:10] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:42:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:11 smithi122 ceph-mon[32255]: pgmap v6710: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:11 smithi022 ceph-mon[31360]: pgmap v6710: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:11 smithi022 ceph-mon[40250]: pgmap v6710: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:12.616 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:42:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:42:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:13 smithi122 ceph-mon[32255]: pgmap v6711: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/449956201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:13 smithi022 ceph-mon[31360]: pgmap v6711: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/449956201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:13 smithi022 ceph-mon[40250]: pgmap v6711: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/449956201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:14.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:42:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:42:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:15 smithi122 ceph-mon[32255]: pgmap v6712: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:15 smithi022 ceph-mon[31360]: pgmap v6712: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:15 smithi022 ceph-mon[40250]: pgmap v6712: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:15.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:17 smithi122 ceph-mon[32255]: pgmap v6713: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:17 smithi022 ceph-mon[31360]: pgmap v6713: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:42:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:17 smithi022 ceph-mon[40250]: pgmap v6713: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:18.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:18.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:18.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:19 smithi122 ceph-mon[32255]: pgmap v6714: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2091956949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:19 smithi022 ceph-mon[31360]: pgmap v6714: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2091956949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:19 smithi022 ceph-mon[40250]: pgmap v6714: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2091956949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:20] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:42:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:21 smithi122 ceph-mon[32255]: pgmap v6715: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:21 smithi022 ceph-mon[31360]: pgmap v6715: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:21 smithi022 ceph-mon[40250]: pgmap v6715: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:42:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:42:23.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:23 smithi122 ceph-mon[32255]: pgmap v6716: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:23.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:23.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:23.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:23.728 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:23 smithi022 ceph-mon[31360]: pgmap v6716: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:23.728 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:23 smithi022 ceph-mon[40250]: pgmap v6716: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:24.626 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1501784236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1501784236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1501784236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:24.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:42:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:42:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:25 smithi122 ceph-mon[32255]: pgmap v6717: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:25 smithi022 ceph-mon[31360]: pgmap v6717: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:25 smithi022 ceph-mon[40250]: pgmap v6717: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:27 smithi122 ceph-mon[32255]: pgmap v6718: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:27 smithi022 ceph-mon[31360]: pgmap v6718: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:42:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:27 smithi022 ceph-mon[40250]: pgmap v6718: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:28.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:29.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:29.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:29.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:29 smithi022 ceph-mon[31360]: pgmap v6719: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3772137289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:29 smithi022 ceph-mon[40250]: pgmap v6719: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3772137289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:29 smithi122 ceph-mon[32255]: pgmap v6719: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3772137289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:30] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:42:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:31 smithi122 ceph-mon[32255]: pgmap v6720: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:31 smithi022 ceph-mon[31360]: pgmap v6720: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:31 smithi022 ceph-mon[40250]: pgmap v6720: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:42:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:42:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: pgmap v6721: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:42:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:42:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:42:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:42:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:42:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:42:33.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:42:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: pgmap v6721: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:42:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:42:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: pgmap v6721: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:42:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:42:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:42:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:42:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:42:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:42:33.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:42:34.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:34.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:34.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:34.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:34.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:42:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:42:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:35 smithi122 ceph-mon[32255]: pgmap v6722: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2238190381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:35 smithi022 ceph-mon[31360]: pgmap v6722: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2238190381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:35 smithi022 ceph-mon[40250]: pgmap v6722: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2238190381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:42:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: pgmap v6723: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: pgmap v6723: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: pgmap v6723: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:42:38.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:38.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:39.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:38 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3238M 2021-12-15T21:42:39.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:38 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3238M 2021-12-15T21:42:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:38 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3238M 2021-12-15T21:42:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:42:39.475 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:39.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:39.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:39.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:39 smithi122 ceph-mon[32255]: pgmap v6724: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:39 smithi022 ceph-mon[31360]: pgmap v6724: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:39 smithi022 ceph-mon[40250]: pgmap v6724: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:40] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:42:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/961723247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/961723247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/961723247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:41 smithi122 ceph-mon[32255]: pgmap v6725: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:41 smithi022 ceph-mon[31360]: pgmap v6725: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:41 smithi022 ceph-mon[40250]: pgmap v6725: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:42:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:42:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:42 smithi122 ceph-mon[32255]: pgmap v6726: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:42 smithi022 ceph-mon[31360]: pgmap v6726: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:42 smithi022 ceph-mon[40250]: pgmap v6726: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:44.850 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:44.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:42:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:42:45.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:45.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:45.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:45 smithi022 ceph-mon[31360]: pgmap v6727: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3812923381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:45 smithi022 ceph-mon[40250]: pgmap v6727: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3812923381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:45 smithi122 ceph-mon[32255]: pgmap v6727: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3812923381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:47 smithi122 ceph-mon[32255]: pgmap v6728: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:47 smithi022 ceph-mon[31360]: pgmap v6728: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:42:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:47 smithi022 ceph-mon[40250]: pgmap v6728: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:49 smithi122 ceph-mon[32255]: pgmap v6729: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:49 smithi022 ceph-mon[31360]: pgmap v6729: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:49 smithi022 ceph-mon[40250]: pgmap v6729: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:50.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:50] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:42:50.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:50.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:50.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:51 smithi122 ceph-mon[32255]: pgmap v6730: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4022988077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:51 smithi022 ceph-mon[31360]: pgmap v6730: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4022988077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:51 smithi022 ceph-mon[40250]: pgmap v6730: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4022988077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:42:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:42:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:53 smithi122 ceph-mon[32255]: pgmap v6731: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:53 smithi022 ceph-mon[31360]: pgmap v6731: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:53 smithi022 ceph-mon[40250]: pgmap v6731: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:54.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:42:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:42:55.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:42:55.593 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:55 smithi022 ceph-mon[31360]: pgmap v6732: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:55.593 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:55.594 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:55 smithi022 ceph-mon[40250]: pgmap v6732: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:55.594 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:55 smithi122 ceph-mon[32255]: pgmap v6732: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:42:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:42:55.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:42:55.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:42:55.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:42:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2311408814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2311408814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2311408814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:42:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:57 smithi122 ceph-mon[32255]: pgmap v6733: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:57 smithi022 ceph-mon[31360]: pgmap v6733: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:42:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:57 smithi022 ceph-mon[40250]: pgmap v6733: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:42:59 smithi122 ceph-mon[32255]: pgmap v6734: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:42:59 smithi022 ceph-mon[31360]: pgmap v6734: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:42:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:42:59 smithi022 ceph-mon[40250]: pgmap v6734: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:00] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:43:00.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:01.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:01.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:01.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:01 smithi122 ceph-mon[32255]: pgmap v6735: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:01 smithi022 ceph-mon[31360]: pgmap v6735: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:01 smithi022 ceph-mon[40250]: pgmap v6735: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1956649985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:43:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:43:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1956649985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1956649985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:03 smithi122 ceph-mon[32255]: pgmap v6736: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:03 smithi022 ceph-mon[31360]: pgmap v6736: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:03 smithi022 ceph-mon[40250]: pgmap v6736: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:04.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:43:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:43:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:05 smithi122 ceph-mon[32255]: pgmap v6737: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:05 smithi022 ceph-mon[31360]: pgmap v6737: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:05 smithi022 ceph-mon[40250]: pgmap v6737: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:06.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:06.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:06.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:06.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:07 smithi122 ceph-mon[32255]: pgmap v6738: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1131529116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:43:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:07 smithi022 ceph-mon[31360]: pgmap v6738: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1131529116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:07 smithi022 ceph-mon[40250]: pgmap v6738: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1131529116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:09 smithi122 ceph-mon[32255]: pgmap v6739: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:09 smithi022 ceph-mon[31360]: pgmap v6739: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:09 smithi022 ceph-mon[40250]: pgmap v6739: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:10] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:43:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:11 smithi122 ceph-mon[32255]: pgmap v6740: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:11.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:11 smithi022 ceph-mon[40250]: pgmap v6740: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:11.702 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:11 smithi022 ceph-mon[31360]: pgmap v6740: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:11.703 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:12.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:12.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:12.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:12.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1397639313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:12.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1397639313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:43:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:43:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1397639313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:13 smithi122 ceph-mon[32255]: pgmap v6741: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:13 smithi022 ceph-mon[31360]: pgmap v6741: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:13 smithi022 ceph-mon[40250]: pgmap v6741: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:14.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:43:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:43:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:15 smithi122 ceph-mon[32255]: pgmap v6742: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:15 smithi022 ceph-mon[31360]: pgmap v6742: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:15 smithi022 ceph-mon[40250]: pgmap v6742: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:17.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:17.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:17.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:17.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:17 smithi122 ceph-mon[32255]: pgmap v6743: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:17 smithi022 ceph-mon[31360]: pgmap v6743: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:43:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:17 smithi022 ceph-mon[40250]: pgmap v6743: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/137290576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/137290576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/137290576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:19 smithi122 ceph-mon[32255]: pgmap v6744: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:19 smithi022 ceph-mon[31360]: pgmap v6744: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:19 smithi022 ceph-mon[40250]: pgmap v6744: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:20] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:43:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:21 smithi122 ceph-mon[32255]: pgmap v6745: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:21 smithi022 ceph-mon[31360]: pgmap v6745: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:21 smithi022 ceph-mon[40250]: pgmap v6745: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:22.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:43:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:43:22.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:22.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:22.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:23 smithi122 ceph-mon[32255]: pgmap v6746: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2611115663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:23 smithi022 ceph-mon[31360]: pgmap v6746: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2611115663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:23 smithi022 ceph-mon[40250]: pgmap v6746: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2611115663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:24.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:43:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:43:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:25 smithi122 ceph-mon[32255]: pgmap v6747: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:25 smithi022 ceph-mon[31360]: pgmap v6747: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:25 smithi022 ceph-mon[40250]: pgmap v6747: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:27 smithi122 ceph-mon[32255]: pgmap v6748: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:27 smithi022 ceph-mon[31360]: pgmap v6748: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:43:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:27 smithi022 ceph-mon[40250]: pgmap v6748: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:27.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:28.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:28.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4263651009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4263651009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4263651009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:29 smithi122 ceph-mon[32255]: pgmap v6749: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:29 smithi022 ceph-mon[31360]: pgmap v6749: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:29 smithi022 ceph-mon[40250]: pgmap v6749: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:30] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:43:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:31 smithi122 ceph-mon[32255]: pgmap v6750: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:31 smithi022 ceph-mon[31360]: pgmap v6750: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:31 smithi022 ceph-mon[40250]: pgmap v6750: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:43:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:43:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[31360]: pgmap v6751: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:43:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:43:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:43:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:43:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[40250]: pgmap v6751: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:43:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:43:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:43:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:43:33.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:33.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:33.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:33 smithi122 ceph-mon[32255]: pgmap v6751: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:43:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:43:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:43:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:43:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/624858252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/624858252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/624858252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:34.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:43:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:43:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:35 smithi122 ceph-mon[32255]: pgmap v6752: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:35 smithi022 ceph-mon[31360]: pgmap v6752: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:35 smithi022 ceph-mon[40250]: pgmap v6752: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:37 smithi122 ceph-mon[32255]: pgmap v6753: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:37 smithi022 ceph-mon[31360]: pgmap v6753: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:43:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:37 smithi022 ceph-mon[40250]: pgmap v6753: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:43:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:43:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:43:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:43:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:43:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:43:38.562 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:38.660 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:43:38.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:43:38.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:43:38.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:38.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:38.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:39 smithi022 ceph-mon[31360]: pgmap v6754: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:43:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1611923660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:39 smithi022 ceph-mon[40250]: pgmap v6754: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:43:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1611923660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:39 smithi122 ceph-mon[32255]: pgmap v6754: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:40.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:43:40.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1611923660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:40] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:43:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:41 smithi022 ceph-mon[31360]: pgmap v6755: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:41 smithi022 ceph-mon[40250]: pgmap v6755: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:41 smithi122 ceph-mon[32255]: pgmap v6755: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:43:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:43:43.927 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:43 smithi022 ceph-mon[40250]: pgmap v6756: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:43 smithi022 ceph-mon[31360]: pgmap v6756: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:43 smithi122 ceph-mon[32255]: pgmap v6756: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:44.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:44.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:44.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2014822912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2014822912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2014822912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:45.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:43:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:43:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:45 smithi022 ceph-mon[31360]: pgmap v6757: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:45 smithi022 ceph-mon[40250]: pgmap v6757: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:45 smithi122 ceph-mon[32255]: pgmap v6757: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:47.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:43:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:47 smithi022 ceph-mon[31360]: pgmap v6758: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:47 smithi022 ceph-mon[40250]: pgmap v6758: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:47 smithi122 ceph-mon[32255]: pgmap v6758: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:49.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:49.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:49.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:49.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:49 smithi022 ceph-mon[31360]: pgmap v6759: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3614334609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:49 smithi022 ceph-mon[40250]: pgmap v6759: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3614334609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:49 smithi122 ceph-mon[32255]: pgmap v6759: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3614334609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:50] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:43:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:51 smithi022 ceph-mon[31360]: pgmap v6760: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:51 smithi022 ceph-mon[40250]: pgmap v6760: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:51 smithi122 ceph-mon[32255]: pgmap v6760: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:43:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:43:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:53 smithi022 ceph-mon[31360]: pgmap v6761: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:53 smithi022 ceph-mon[40250]: pgmap v6761: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:53 smithi122 ceph-mon[32255]: pgmap v6761: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:54.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:43:55.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:43:55.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:43:55.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:43:55.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:43:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:43:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:55 smithi022 ceph-mon[31360]: pgmap v6762: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2306176475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:55 smithi022 ceph-mon[40250]: pgmap v6762: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2306176475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:55 smithi122 ceph-mon[32255]: pgmap v6762: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:43:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:43:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2306176475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:43:57.732 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:43:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:57 smithi022 ceph-mon[31360]: pgmap v6763: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:57 smithi022 ceph-mon[40250]: pgmap v6763: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:43:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:57 smithi122 ceph-mon[32255]: pgmap v6763: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:43:59 smithi022 ceph-mon[31360]: pgmap v6764: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:43:59 smithi022 ceph-mon[40250]: pgmap v6764: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:00.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:43:59 smithi122 ceph-mon[32255]: pgmap v6764: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:00.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:00.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:00.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:00] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:44:00.740 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1045709731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1045709731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1045709731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:01 smithi022 ceph-mon[31360]: pgmap v6765: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:02.058 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:01 smithi022 ceph-mon[40250]: pgmap v6765: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:01 smithi122 ceph-mon[32255]: pgmap v6765: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:44:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:44:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:03 smithi022 ceph-mon[31360]: pgmap v6766: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:03 smithi022 ceph-mon[40250]: pgmap v6766: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:03 smithi122 ceph-mon[32255]: pgmap v6766: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:05.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:44:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:44:05.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:05.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:05.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:05.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:05 smithi022 ceph-mon[31360]: pgmap v6767: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:05 smithi022 ceph-mon[40250]: pgmap v6767: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:05 smithi122 ceph-mon[32255]: pgmap v6767: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1069184321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1069184321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1069184321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:07.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:44:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:07 smithi022 ceph-mon[31360]: pgmap v6768: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:07 smithi022 ceph-mon[40250]: pgmap v6768: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:07 smithi122 ceph-mon[32255]: pgmap v6768: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:09 smithi022 ceph-mon[31360]: pgmap v6769: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:09 smithi022 ceph-mon[40250]: pgmap v6769: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:09 smithi122 ceph-mon[32255]: pgmap v6769: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:10] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:44:10.785 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:11.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:11.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:11.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:11 smithi022 ceph-mon[31360]: pgmap v6770: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3764788055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:11 smithi022 ceph-mon[40250]: pgmap v6770: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3764788055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:11 smithi122 ceph-mon[32255]: pgmap v6770: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3764788055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:44:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:44:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:13 smithi022 ceph-mon[31360]: pgmap v6771: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:13 smithi022 ceph-mon[40250]: pgmap v6771: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:13 smithi122 ceph-mon[32255]: pgmap v6771: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:15.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:44:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:44:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:15 smithi022 ceph-mon[31360]: pgmap v6772: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:16.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:15 smithi022 ceph-mon[40250]: pgmap v6772: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:16.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:15 smithi122 ceph-mon[32255]: pgmap v6772: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:16.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:16.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1270461887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1270461887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1270461887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:17.769 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:44:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:17 smithi022 ceph-mon[40250]: pgmap v6773: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:17 smithi022 ceph-mon[31360]: pgmap v6773: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:17 smithi122 ceph-mon[32255]: pgmap v6773: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:19 smithi022 ceph-mon[31360]: pgmap v6774: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:19 smithi022 ceph-mon[40250]: pgmap v6774: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:19 smithi122 ceph-mon[32255]: pgmap v6774: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:20] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:44:21.522 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:21.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:21.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:21.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:21 smithi022 ceph-mon[31360]: pgmap v6775: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:21 smithi022 ceph-mon[40250]: pgmap v6775: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:21 smithi122 ceph-mon[32255]: pgmap v6775: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:44:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:44:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2564563980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2564563980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2564563980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:23 smithi122 ceph-mon[32255]: pgmap v6776: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:23 smithi022 ceph-mon[31360]: pgmap v6776: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:23 smithi022 ceph-mon[40250]: pgmap v6776: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:25.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:44:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:44:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:25 smithi122 ceph-mon[32255]: pgmap v6777: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:25 smithi022 ceph-mon[31360]: pgmap v6777: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:25 smithi022 ceph-mon[40250]: pgmap v6777: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:26.889 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:27.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:27.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:27.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:44:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:27 smithi122 ceph-mon[32255]: pgmap v6778: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2111925579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:27 smithi022 ceph-mon[31360]: pgmap v6778: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2111925579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:27 smithi022 ceph-mon[40250]: pgmap v6778: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2111925579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:29 smithi122 ceph-mon[32255]: pgmap v6779: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:29 smithi022 ceph-mon[31360]: pgmap v6779: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:29 smithi022 ceph-mon[40250]: pgmap v6779: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:30] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:44:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:31 smithi122 ceph-mon[32255]: pgmap v6780: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:32.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:32.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:31 smithi022 ceph-mon[31360]: pgmap v6780: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:32.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:31 smithi022 ceph-mon[40250]: pgmap v6780: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:32.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:32.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:32.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:44:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:44:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2531425222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:33.055 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2531425222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2531425222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:33 smithi122 ceph-mon[32255]: pgmap v6781: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:44:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:44:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:44:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:44:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[31360]: pgmap v6781: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:44:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:44:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:44:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:44:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[40250]: pgmap v6781: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:44:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:44:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:44:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:44:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:44:35.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:44:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:44:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:35 smithi122 ceph-mon[32255]: pgmap v6782: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:35 smithi022 ceph-mon[31360]: pgmap v6782: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:35 smithi022 ceph-mon[40250]: pgmap v6782: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:37.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:44:38.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:38.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:38.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:37 smithi122 ceph-mon[32255]: pgmap v6783: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:37 smithi022 ceph-mon[31360]: pgmap v6783: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:37 smithi022 ceph-mon[40250]: pgmap v6783: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/796593148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:44:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:44:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:44:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/796593148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:44:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:44:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:44:39.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/796593148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:39.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:44:39.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:44:39.151 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:44:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:39 smithi122 ceph-mon[32255]: pgmap v6784: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:44:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:39 smithi022 ceph-mon[31360]: pgmap v6784: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:44:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:39 smithi022 ceph-mon[40250]: pgmap v6784: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:40.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:44:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:40] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:44:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:41 smithi122 ceph-mon[32255]: pgmap v6785: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:41 smithi022 ceph-mon[31360]: pgmap v6785: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:41 smithi022 ceph-mon[40250]: pgmap v6785: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:44:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:44:43.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:43.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:43 smithi122 ceph-mon[32255]: pgmap v6786: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/182914559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:43 smithi022 ceph-mon[31360]: pgmap v6786: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/182914559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:43 smithi022 ceph-mon[40250]: pgmap v6786: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/182914559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:45.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:44:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:44:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:45 smithi122 ceph-mon[32255]: pgmap v6787: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:46.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:45 smithi022 ceph-mon[31360]: pgmap v6787: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:45 smithi022 ceph-mon[40250]: pgmap v6787: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:44:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:47 smithi122 ceph-mon[32255]: pgmap v6788: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:47 smithi022 ceph-mon[40250]: pgmap v6788: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:47 smithi022 ceph-mon[31360]: pgmap v6788: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:48.376 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:48.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:48.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:48.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:49 smithi122 ceph-mon[32255]: pgmap v6789: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3159917195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:49 smithi022 ceph-mon[31360]: pgmap v6789: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3159917195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:49 smithi022 ceph-mon[40250]: pgmap v6789: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3159917195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:50] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:44:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:51 smithi122 ceph-mon[32255]: pgmap v6790: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:51 smithi022 ceph-mon[31360]: pgmap v6790: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:51 smithi022 ceph-mon[40250]: pgmap v6790: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:44:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:44:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:53 smithi122 ceph-mon[32255]: pgmap v6791: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:53.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:53.742 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:53 smithi022 ceph-mon[31360]: pgmap v6791: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:53.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:53 smithi022 ceph-mon[40250]: pgmap v6791: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:54.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:54.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:54.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1558138615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1558138615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1558138615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:44:55.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:44:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:44:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:55 smithi122 ceph-mon[32255]: pgmap v6792: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:55 smithi022 ceph-mon[31360]: pgmap v6792: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:55 smithi022 ceph-mon[40250]: pgmap v6792: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:44:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:57 smithi122 ceph-mon[32255]: pgmap v6793: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:57 smithi022 ceph-mon[31360]: pgmap v6793: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:44:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:57 smithi022 ceph-mon[40250]: pgmap v6793: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:44:59.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:44:59.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:44:59.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:44:59.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:44:59.481 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:44:59 smithi022 ceph-mon[31360]: pgmap v6794: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:59.481 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:44:59 smithi022 ceph-mon[40250]: pgmap v6794: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:44:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:44:59 smithi122 ceph-mon[32255]: pgmap v6794: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1397518364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:00] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T21:45:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1397518364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1397518364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:01 smithi122 ceph-mon[32255]: pgmap v6795: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:01 smithi022 ceph-mon[31360]: pgmap v6795: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:01 smithi022 ceph-mon[40250]: pgmap v6795: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:45:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:45:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:03 smithi122 ceph-mon[32255]: pgmap v6796: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:03 smithi022 ceph-mon[31360]: pgmap v6796: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:03 smithi022 ceph-mon[40250]: pgmap v6796: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:04.482 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:04.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:05.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:45:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:45:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:05 smithi122 ceph-mon[32255]: pgmap v6797: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3888162720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:05 smithi022 ceph-mon[31360]: pgmap v6797: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3888162720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:05 smithi022 ceph-mon[40250]: pgmap v6797: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3888162720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:07 smithi122 ceph-mon[32255]: pgmap v6798: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:07 smithi022 ceph-mon[31360]: pgmap v6798: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:45:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:07 smithi022 ceph-mon[40250]: pgmap v6798: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:09 smithi122 ceph-mon[32255]: pgmap v6799: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:09 smithi022 ceph-mon[31360]: pgmap v6799: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:09 smithi022 ceph-mon[40250]: pgmap v6799: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:09.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:10.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:10.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:10.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/905036806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:10] "GET /metrics HTTP/1.1" 200 202358 "" "Prometheus/2.18.1" 2021-12-15T21:45:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/905036806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/905036806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:11 smithi122 ceph-mon[32255]: pgmap v6800: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:11 smithi022 ceph-mon[31360]: pgmap v6800: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:11 smithi022 ceph-mon[40250]: pgmap v6800: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:45:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:45:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:13 smithi122 ceph-mon[32255]: pgmap v6801: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:13 smithi022 ceph-mon[31360]: pgmap v6801: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:13 smithi022 ceph-mon[40250]: pgmap v6801: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:15.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:45:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:45:15.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:15 smithi022 ceph-mon[31360]: pgmap v6802: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:15 smithi022 ceph-mon[40250]: pgmap v6802: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:15.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:15.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:15.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:15 smithi122 ceph-mon[32255]: pgmap v6802: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/102304345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/102304345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/102304345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:17 smithi122 ceph-mon[32255]: pgmap v6803: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:17 smithi022 ceph-mon[31360]: pgmap v6803: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:45:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:17 smithi022 ceph-mon[40250]: pgmap v6803: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:19 smithi122 ceph-mon[32255]: pgmap v6804: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:19 smithi022 ceph-mon[31360]: pgmap v6804: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:19 smithi022 ceph-mon[40250]: pgmap v6804: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:20] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:45:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:20.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:20.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:20.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:21 smithi122 ceph-mon[32255]: pgmap v6805: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1150093588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:21 smithi022 ceph-mon[31360]: pgmap v6805: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1150093588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:21 smithi022 ceph-mon[40250]: pgmap v6805: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1150093588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:45:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:45:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:23 smithi122 ceph-mon[32255]: pgmap v6806: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:23 smithi022 ceph-mon[31360]: pgmap v6806: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:23 smithi022 ceph-mon[40250]: pgmap v6806: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:25.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:45:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:45:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:25 smithi122 ceph-mon[32255]: pgmap v6807: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:25 smithi022 ceph-mon[31360]: pgmap v6807: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:25 smithi022 ceph-mon[40250]: pgmap v6807: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:25.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:26.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:26.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:26.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1631882726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1631882726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1631882726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:27 smithi122 ceph-mon[32255]: pgmap v6808: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:27 smithi022 ceph-mon[31360]: pgmap v6808: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:45:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:27 smithi022 ceph-mon[40250]: pgmap v6808: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:29 smithi122 ceph-mon[32255]: pgmap v6809: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:29 smithi022 ceph-mon[31360]: pgmap v6809: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:29 smithi022 ceph-mon[40250]: pgmap v6809: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:30] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:45:31.338 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:31 smithi122 ceph-mon[32255]: pgmap v6810: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:31.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:31.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:31.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:31.709 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:31 smithi022 ceph-mon[31360]: pgmap v6810: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:31.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:31 smithi022 ceph-mon[40250]: pgmap v6810: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2324389360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:45:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:45:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2324389360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2324389360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:33 smithi122 ceph-mon[32255]: pgmap v6811: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:45:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:45:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:45:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:45:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[31360]: pgmap v6811: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[40250]: pgmap v6811: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:45:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:45:35.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:45:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:45:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:35 smithi122 ceph-mon[32255]: pgmap v6812: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:35 smithi022 ceph-mon[31360]: pgmap v6812: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:35 smithi022 ceph-mon[40250]: pgmap v6812: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:36.711 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:37.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:37.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:37.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:37 smithi122 ceph-mon[32255]: pgmap v6813: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1044909070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:45:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:37 smithi022 ceph-mon[31360]: pgmap v6813: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1044909070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:37 smithi022 ceph-mon[40250]: pgmap v6813: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1044909070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[31360]: pgmap v6814: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:45:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:45:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:45:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[40250]: pgmap v6814: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:45:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:45:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:45:39.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:39 smithi122 ceph-mon[32255]: pgmap v6814: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:39.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:45:39.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:45:39.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:45:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:40] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:45:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:45:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:45:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:45:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:41 smithi122 ceph-mon[32255]: pgmap v6815: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:41 smithi022 ceph-mon[31360]: pgmap v6815: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:41 smithi022 ceph-mon[40250]: pgmap v6815: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:42.078 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:42.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:42.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:42.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:42.650 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:45:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:45:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1742208683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1742208683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:42.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1742208683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:43 smithi122 ceph-mon[32255]: pgmap v6816: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:43 smithi022 ceph-mon[31360]: pgmap v6816: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:43 smithi022 ceph-mon[40250]: pgmap v6816: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:45.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:45:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:45:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:45 smithi122 ceph-mon[32255]: pgmap v6817: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:45 smithi022 ceph-mon[31360]: pgmap v6817: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:45 smithi022 ceph-mon[40250]: pgmap v6817: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:47.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:47.655 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:45:47.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:47.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:47.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:47 smithi122 ceph-mon[32255]: pgmap v6818: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:47 smithi022 ceph-mon[31360]: pgmap v6818: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:47 smithi022 ceph-mon[40250]: pgmap v6818: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/750168422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/750168422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/750168422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:49 smithi122 ceph-mon[32255]: pgmap v6819: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:49 smithi022 ceph-mon[31360]: pgmap v6819: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:49 smithi022 ceph-mon[40250]: pgmap v6819: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:50] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:45:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:51 smithi122 ceph-mon[32255]: pgmap v6820: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:51 smithi022 ceph-mon[31360]: pgmap v6820: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:51 smithi022 ceph-mon[40250]: pgmap v6820: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:45:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:45:52.818 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:53 smithi122 ceph-mon[32255]: pgmap v6821: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4261941832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:53 smithi022 ceph-mon[31360]: pgmap v6821: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4261941832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:53 smithi022 ceph-mon[40250]: pgmap v6821: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4261941832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:55.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:45:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:45:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:55 smithi122 ceph-mon[32255]: pgmap v6822: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:55 smithi022 ceph-mon[31360]: pgmap v6822: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:55 smithi022 ceph-mon[40250]: pgmap v6822: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:45:57.672 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:45:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:57 smithi122 ceph-mon[32255]: pgmap v6823: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:57 smithi022 ceph-mon[31360]: pgmap v6823: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:57 smithi022 ceph-mon[40250]: pgmap v6823: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:45:58.189 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:45:58.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:45:58.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:45:58.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:45:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:59 smithi122 ceph-mon[32255]: pgmap v6824: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:45:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4287019289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:59 smithi022 ceph-mon[31360]: pgmap v6824: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:45:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4287019289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:45:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:59 smithi022 ceph-mon[40250]: pgmap v6824: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:45:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:45:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4287019289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:00] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:46:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:01 smithi122 ceph-mon[32255]: pgmap v6825: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:01 smithi022 ceph-mon[31360]: pgmap v6825: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:01 smithi022 ceph-mon[40250]: pgmap v6825: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:46:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:46:03.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:03 smithi122 ceph-mon[32255]: pgmap v6826: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:03 smithi022 ceph-mon[31360]: pgmap v6826: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:03 smithi022 ceph-mon[40250]: pgmap v6826: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:03.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:03.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1474771973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1474771973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1474771973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:05.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:46:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:46:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:05 smithi122 ceph-mon[32255]: pgmap v6827: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:05.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:05 smithi022 ceph-mon[31360]: pgmap v6827: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:06.099 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:06.100 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:05 smithi022 ceph-mon[40250]: pgmap v6827: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:06.100 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:07 smithi122 ceph-mon[32255]: pgmap v6828: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:07 smithi022 ceph-mon[31360]: pgmap v6828: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:46:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:07 smithi022 ceph-mon[40250]: pgmap v6828: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:08.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:09.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:09.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:09.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:09 smithi122 ceph-mon[32255]: pgmap v6829: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2362154320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:09 smithi022 ceph-mon[31360]: pgmap v6829: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2362154320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:09 smithi022 ceph-mon[40250]: pgmap v6829: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2362154320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:10] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:46:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:11 smithi122 ceph-mon[32255]: pgmap v6830: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:11 smithi022 ceph-mon[31360]: pgmap v6830: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:11 smithi022 ceph-mon[40250]: pgmap v6830: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:46:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:46:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:13 smithi122 ceph-mon[32255]: pgmap v6831: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:13 smithi022 ceph-mon[31360]: pgmap v6831: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:13 smithi022 ceph-mon[40250]: pgmap v6831: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:14.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:14.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:14.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:14.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:15.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:46:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:46:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:15 smithi122 ceph-mon[32255]: pgmap v6832: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3854803374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:15 smithi022 ceph-mon[31360]: pgmap v6832: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3854803374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:15.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:15 smithi022 ceph-mon[40250]: pgmap v6832: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:15.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3854803374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:17 smithi122 ceph-mon[32255]: pgmap v6833: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:17 smithi022 ceph-mon[31360]: pgmap v6833: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:46:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:17 smithi022 ceph-mon[40250]: pgmap v6833: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:19.674 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:19.674 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:19 smithi022 ceph-mon[31360]: pgmap v6834: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:19.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:19 smithi022 ceph-mon[40250]: pgmap v6834: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:19 smithi122 ceph-mon[32255]: pgmap v6834: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:20.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:20.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:20.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:20] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:46:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/654463161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/654463161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/654463161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:21 smithi122 ceph-mon[32255]: pgmap v6835: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:21 smithi022 ceph-mon[31360]: pgmap v6835: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:21 smithi022 ceph-mon[40250]: pgmap v6835: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:46:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:46:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:23 smithi122 ceph-mon[32255]: pgmap v6836: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:23 smithi022 ceph-mon[31360]: pgmap v6836: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:23 smithi022 ceph-mon[40250]: pgmap v6836: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:25.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:25.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:46:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:46:25.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:25.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:25.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:25 smithi122 ceph-mon[32255]: pgmap v6837: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:25 smithi022 ceph-mon[31360]: pgmap v6837: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:25 smithi022 ceph-mon[40250]: pgmap v6837: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1986329341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1986329341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1986329341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:27.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:27 smithi122 ceph-mon[32255]: pgmap v6838: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:46:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:27 smithi022 ceph-mon[31360]: pgmap v6838: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:27 smithi022 ceph-mon[40250]: pgmap v6838: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:29 smithi122 ceph-mon[32255]: pgmap v6839: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:29 smithi022 ceph-mon[31360]: pgmap v6839: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:29 smithi022 ceph-mon[40250]: pgmap v6839: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:30.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:30] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:46:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:31 smithi122 ceph-mon[32255]: pgmap v6840: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3286910966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:31 smithi022 ceph-mon[31360]: pgmap v6840: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3286910966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:31 smithi022 ceph-mon[40250]: pgmap v6840: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3286910966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:46:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:46:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:33 smithi122 ceph-mon[32255]: pgmap v6841: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:46:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:46:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:46:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:46:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[31360]: pgmap v6841: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:46:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:46:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:46:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:46:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[40250]: pgmap v6841: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:46:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:46:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:46:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:46:35.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:46:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:46:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:35 smithi122 ceph-mon[32255]: pgmap v6842: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:35 smithi022 ceph-mon[31360]: pgmap v6842: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:35 smithi022 ceph-mon[40250]: pgmap v6842: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:35.786 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:36.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:36.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:36.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4274722012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4274722012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4274722012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:37 smithi122 ceph-mon[32255]: pgmap v6843: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:46:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:37 smithi022 ceph-mon[31360]: pgmap v6843: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:37 smithi022 ceph-mon[40250]: pgmap v6843: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:39.677 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:39 smithi022 ceph-mon[31360]: pgmap v6844: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:39.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:39 smithi022 ceph-mon[40250]: pgmap v6844: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:39 smithi122 ceph-mon[32255]: pgmap v6844: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:40.404 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:40] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:46:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:46:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:46:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:46:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:46:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:46:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:46:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:46:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:46:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:46:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:46:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:46:40.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:46:41.158 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:41.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:41.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:41.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:41 smithi022 ceph-mon[31360]: pgmap v6845: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:41 smithi022 ceph-mon[40250]: pgmap v6845: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:41 smithi122 ceph-mon[32255]: pgmap v6845: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1139181190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:46:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:46:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1139181190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1139181190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:43 smithi122 ceph-mon[32255]: pgmap v6846: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:43 smithi022 ceph-mon[31360]: pgmap v6846: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:43 smithi022 ceph-mon[40250]: pgmap v6846: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:45.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:46:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:46:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:45 smithi122 ceph-mon[32255]: pgmap v6847: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:45 smithi022 ceph-mon[31360]: pgmap v6847: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:45 smithi022 ceph-mon[40250]: pgmap v6847: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:46.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:46.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:46.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:47 smithi122 ceph-mon[32255]: pgmap v6848: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/753977397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:46:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:47 smithi022 ceph-mon[31360]: pgmap v6848: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/753977397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:47 smithi022 ceph-mon[40250]: pgmap v6848: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/753977397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:49 smithi122 ceph-mon[32255]: pgmap v6849: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:49 smithi022 ceph-mon[31360]: pgmap v6849: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:49 smithi022 ceph-mon[40250]: pgmap v6849: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:50] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:46:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:51 smithi122 ceph-mon[32255]: pgmap v6850: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:51 smithi022 ceph-mon[31360]: pgmap v6850: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:51 smithi022 ceph-mon[40250]: pgmap v6850: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:51.901 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:52.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:52.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:52.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3018568449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3018568449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3018568449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:46:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:46:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:53 smithi122 ceph-mon[32255]: pgmap v6851: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:53 smithi022 ceph-mon[31360]: pgmap v6851: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:53 smithi022 ceph-mon[40250]: pgmap v6851: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:55.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:46:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:46:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:55 smithi122 ceph-mon[32255]: pgmap v6852: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:55 smithi022 ceph-mon[31360]: pgmap v6852: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:55 smithi022 ceph-mon[40250]: pgmap v6852: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:46:57.271 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:46:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:57 smithi022 ceph-mon[31360]: pgmap v6853: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:46:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:57 smithi022 ceph-mon[40250]: pgmap v6853: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:57.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:46:57.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:46:57.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:46:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:57 smithi122 ceph-mon[32255]: pgmap v6853: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:46:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1565486808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:58.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1565486808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1565486808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:46:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:46:59 smithi022 ceph-mon[31360]: pgmap v6854: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:46:59 smithi022 ceph-mon[40250]: pgmap v6854: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:46:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:46:59 smithi122 ceph-mon[32255]: pgmap v6854: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:00] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:47:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:01 smithi022 ceph-mon[31360]: pgmap v6855: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:01 smithi022 ceph-mon[40250]: pgmap v6855: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:01 smithi122 ceph-mon[32255]: pgmap v6855: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:02.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:47:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:47:03.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:03.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:03.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:03 smithi022 ceph-mon[31360]: pgmap v6856: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/951304307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:03 smithi022 ceph-mon[40250]: pgmap v6856: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/951304307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:03 smithi122 ceph-mon[32255]: pgmap v6856: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/951304307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:05.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:47:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:47:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:05 smithi022 ceph-mon[31360]: pgmap v6857: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:05 smithi022 ceph-mon[40250]: pgmap v6857: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:05 smithi122 ceph-mon[32255]: pgmap v6857: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:07 smithi022 ceph-mon[31360]: pgmap v6858: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:47:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:07 smithi022 ceph-mon[40250]: pgmap v6858: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:07 smithi122 ceph-mon[32255]: pgmap v6858: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:08.016 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:08.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:08.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:08.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:09 smithi122 ceph-mon[32255]: pgmap v6859: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:09.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4284768567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:09 smithi022 ceph-mon[31360]: pgmap v6859: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4284768567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:09 smithi022 ceph-mon[40250]: pgmap v6859: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4284768567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:10] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:47:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:11 smithi122 ceph-mon[32255]: pgmap v6860: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:11 smithi022 ceph-mon[31360]: pgmap v6860: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:11 smithi022 ceph-mon[40250]: pgmap v6860: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:47:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:47:13.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:13 smithi122 ceph-mon[32255]: pgmap v6861: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:13.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:13.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:13.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:13.758 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:13 smithi022 ceph-mon[31360]: pgmap v6861: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:13.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:13 smithi022 ceph-mon[40250]: pgmap v6861: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2016382521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2016382521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2016382521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:15.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:47:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:47:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:15 smithi122 ceph-mon[32255]: pgmap v6862: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:15 smithi022 ceph-mon[31360]: pgmap v6862: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:15 smithi022 ceph-mon[40250]: pgmap v6862: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:17 smithi122 ceph-mon[32255]: pgmap v6863: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:17 smithi022 ceph-mon[31360]: pgmap v6863: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:47:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:17 smithi022 ceph-mon[40250]: pgmap v6863: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:19.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:19.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:19.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:19 smithi022 ceph-mon[31360]: pgmap v6864: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2904789151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:19 smithi022 ceph-mon[40250]: pgmap v6864: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2904789151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:19 smithi122 ceph-mon[32255]: pgmap v6864: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2904789151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:20] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:47:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:21 smithi122 ceph-mon[32255]: pgmap v6865: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:21 smithi022 ceph-mon[31360]: pgmap v6865: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:21 smithi022 ceph-mon[40250]: pgmap v6865: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:47:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:47:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:23 smithi122 ceph-mon[32255]: pgmap v6866: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:23 smithi022 ceph-mon[31360]: pgmap v6866: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:23 smithi022 ceph-mon[40250]: pgmap v6866: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:24.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:24.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:24.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:24.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:25.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:47:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:47:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:25 smithi122 ceph-mon[32255]: pgmap v6867: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1480494491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:25 smithi022 ceph-mon[31360]: pgmap v6867: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1480494491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:25 smithi022 ceph-mon[40250]: pgmap v6867: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1480494491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:27 smithi122 ceph-mon[32255]: pgmap v6868: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:27 smithi022 ceph-mon[31360]: pgmap v6868: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:47:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:27 smithi022 ceph-mon[40250]: pgmap v6868: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:29.498 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:29 smithi122 ceph-mon[32255]: pgmap v6869: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:29 smithi022 ceph-mon[31360]: pgmap v6869: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:29 smithi022 ceph-mon[40250]: pgmap v6869: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:29.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:29.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:29.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2173046720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:30] "GET /metrics HTTP/1.1" 200 202359 "" "Prometheus/2.18.1" 2021-12-15T21:47:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2173046720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2173046720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:31 smithi122 ceph-mon[32255]: pgmap v6870: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:31 smithi022 ceph-mon[31360]: pgmap v6870: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:31 smithi022 ceph-mon[40250]: pgmap v6870: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:47:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:33 smithi122 ceph-mon[32255]: pgmap v6871: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:47:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:47:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[31360]: pgmap v6871: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:47:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:47:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[40250]: pgmap v6871: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:47:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:47:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:47:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:47:34.866 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:35.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:47:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:47:35.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:35.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:35.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:35 smithi022 ceph-mon[31360]: pgmap v6872: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2339497741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:35 smithi022 ceph-mon[40250]: pgmap v6872: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2339497741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:35 smithi122 ceph-mon[32255]: pgmap v6872: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2339497741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:37 smithi122 ceph-mon[32255]: pgmap v6873: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:37 smithi022 ceph-mon[31360]: pgmap v6873: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:47:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:37 smithi022 ceph-mon[40250]: pgmap v6873: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:39 smithi122 ceph-mon[32255]: pgmap v6874: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:39 smithi022 ceph-mon[31360]: pgmap v6874: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:39 smithi022 ceph-mon[40250]: pgmap v6874: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:40.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:40.405 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:40] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:47:40.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:40.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:40.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:40.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:47:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:47:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:47:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:47:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:47:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:47:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:47:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:47:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:47:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:41 smithi122 ceph-mon[32255]: pgmap v6875: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:47:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2566579536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:41 smithi022 ceph-mon[31360]: pgmap v6875: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:47:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2566579536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:41 smithi022 ceph-mon[40250]: pgmap v6875: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:47:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2566579536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:47:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:47:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:43 smithi122 ceph-mon[32255]: pgmap v6876: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:43 smithi022 ceph-mon[31360]: pgmap v6876: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:43 smithi022 ceph-mon[40250]: pgmap v6876: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:45.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:47:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:47:45.610 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:45 smithi122 ceph-mon[32255]: pgmap v6877: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:45.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:45.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:45.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:45.982 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:45 smithi022 ceph-mon[31360]: pgmap v6877: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:45.983 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:45 smithi022 ceph-mon[40250]: pgmap v6877: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/911110134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/911110134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/911110134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:47 smithi022 ceph-mon[31360]: pgmap v6878: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:47:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:47 smithi022 ceph-mon[40250]: pgmap v6878: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:47 smithi122 ceph-mon[32255]: pgmap v6878: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:49 smithi122 ceph-mon[32255]: pgmap v6879: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:49 smithi022 ceph-mon[31360]: pgmap v6879: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:49 smithi022 ceph-mon[40250]: pgmap v6879: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:47:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:50] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:47:50.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:51.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:51.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:51.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:51 smithi022 ceph-mon[31360]: pgmap v6880: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/931093719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:51 smithi022 ceph-mon[40250]: pgmap v6880: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/931093719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:51 smithi122 ceph-mon[32255]: pgmap v6880: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/931093719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:47:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:47:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:53 smithi122 ceph-mon[32255]: pgmap v6881: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:53 smithi022 ceph-mon[31360]: pgmap v6881: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:53 smithi022 ceph-mon[40250]: pgmap v6881: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:47:55.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:47:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:47:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:55 smithi122 ceph-mon[32255]: pgmap v6882: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:47:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:55 smithi022 ceph-mon[31360]: pgmap v6882: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:47:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:55 smithi022 ceph-mon[40250]: pgmap v6882: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:47:56.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:47:56.604 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:56.605 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:56.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:47:56.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:47:56.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:47:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:47:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:47:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:57 smithi022 ceph-mon[31360]: pgmap v6883: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:47:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1105504738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:57 smithi022 ceph-mon[40250]: pgmap v6883: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:47:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1105504738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:57 smithi122 ceph-mon[32255]: pgmap v6883: 132 pgs: 132 active+clean; 582 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:47:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1105504738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:47:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:47:59 smithi122 ceph-mon[32255]: pgmap v6884: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:47:59 smithi022 ceph-mon[31360]: pgmap v6884: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:47:59 smithi022 ceph-mon[40250]: pgmap v6884: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:00] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:48:01.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:01 smithi122 ceph-mon[32255]: pgmap v6885: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:01 smithi022 ceph-mon[31360]: pgmap v6885: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:01 smithi022 ceph-mon[40250]: pgmap v6885: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:02.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:02.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:02.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:02.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:48:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:48:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4132050221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4132050221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4132050221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:03 smithi122 ceph-mon[32255]: pgmap v6886: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:03 smithi022 ceph-mon[31360]: pgmap v6886: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:03 smithi022 ceph-mon[40250]: pgmap v6886: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:48:05.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:48:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:48:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:05 smithi122 ceph-mon[32255]: pgmap v6887: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:48:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:05 smithi022 ceph-mon[31360]: pgmap v6887: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:48:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:05 smithi022 ceph-mon[40250]: pgmap v6887: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:48:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:07.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:07.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:07.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:07.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:48:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:07 smithi022 ceph-mon[31360]: pgmap v6888: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3433021972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:07 smithi022 ceph-mon[40250]: pgmap v6888: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3433021972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:07 smithi122 ceph-mon[32255]: pgmap v6888: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3433021972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:09 smithi122 ceph-mon[32255]: pgmap v6889: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:48:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:09 smithi022 ceph-mon[31360]: pgmap v6889: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:48:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:09 smithi022 ceph-mon[40250]: pgmap v6889: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:48:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:10] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:48:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:11 smithi122 ceph-mon[32255]: pgmap v6890: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:11 smithi022 ceph-mon[31360]: pgmap v6890: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:11 smithi022 ceph-mon[40250]: pgmap v6890: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:12.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:48:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:48:12.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:12.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:12.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:13 smithi122 ceph-mon[32255]: pgmap v6891: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/103597094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:13 smithi022 ceph-mon[31360]: pgmap v6891: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/103597094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:13 smithi022 ceph-mon[40250]: pgmap v6891: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:48:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/103597094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:15.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:48:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:48:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:15 smithi122 ceph-mon[32255]: pgmap v6892: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:48:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:15 smithi022 ceph-mon[31360]: pgmap v6892: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:48:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:15 smithi022 ceph-mon[40250]: pgmap v6892: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:48:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:17 smithi022 ceph-mon[31360]: pgmap v6893: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:48:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:17 smithi022 ceph-mon[40250]: pgmap v6893: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:17.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:17 smithi122 ceph-mon[32255]: pgmap v6893: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2461633521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2461633521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2461633521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:19 smithi122 ceph-mon[32255]: pgmap v6894: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:19 smithi022 ceph-mon[31360]: pgmap v6894: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:19 smithi022 ceph-mon[40250]: pgmap v6894: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:20] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:48:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:21 smithi122 ceph-mon[32255]: pgmap v6895: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:21 smithi022 ceph-mon[31360]: pgmap v6895: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:21 smithi022 ceph-mon[40250]: pgmap v6895: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:48:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:48:23.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:23.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:23.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:23.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:23 smithi122 ceph-mon[32255]: pgmap v6896: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1483872544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:23 smithi022 ceph-mon[31360]: pgmap v6896: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1483872544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:23 smithi022 ceph-mon[40250]: pgmap v6896: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1483872544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:25.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:48:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:48:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:25 smithi122 ceph-mon[32255]: pgmap v6897: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:25 smithi022 ceph-mon[31360]: pgmap v6897: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:25 smithi022 ceph-mon[40250]: pgmap v6897: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:27.664 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:48:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:27 smithi122 ceph-mon[32255]: pgmap v6898: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:27 smithi022 ceph-mon[31360]: pgmap v6898: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:27 smithi022 ceph-mon[40250]: pgmap v6898: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:28.572 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:28.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:29 smithi122 ceph-mon[32255]: pgmap v6899: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/143408031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:29 smithi022 ceph-mon[31360]: pgmap v6899: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/143408031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:29 smithi022 ceph-mon[40250]: pgmap v6899: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/143408031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:30] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:48:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:31 smithi122 ceph-mon[32255]: pgmap v6900: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:31 smithi022 ceph-mon[31360]: pgmap v6900: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:31 smithi022 ceph-mon[40250]: pgmap v6900: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:48:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:48:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:33 smithi122 ceph-mon[32255]: pgmap v6901: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:48:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:48:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:48:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:48:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:33.944 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[31360]: pgmap v6901: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:33.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:48:33.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:48:33.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:48:33.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:48:33.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[40250]: pgmap v6901: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:33.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:48:33.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:48:33.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:48:33.948 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:48:34.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:34.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:34.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3587383589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3587383589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3587383589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:35.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:48:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:48:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:35 smithi122 ceph-mon[32255]: pgmap v6902: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:35 smithi022 ceph-mon[31360]: pgmap v6902: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:35 smithi022 ceph-mon[40250]: pgmap v6902: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:37.689 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:48:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:37 smithi022 ceph-mon[31360]: pgmap v6903: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:37 smithi022 ceph-mon[40250]: pgmap v6903: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:37 smithi122 ceph-mon[32255]: pgmap v6903: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:39.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:39.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:39.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:39.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:39 smithi022 ceph-mon[31360]: pgmap v6904: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/600132972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:39 smithi022 ceph-mon[40250]: pgmap v6904: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/600132972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:39 smithi122 ceph-mon[32255]: pgmap v6904: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/600132972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:40] "GET /metrics HTTP/1.1" 200 202371 "" "Prometheus/2.18.1" 2021-12-15T21:48:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:48:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:48:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:48:40.992 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:48:40.992 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:48:40.992 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:48:40.993 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:48:40.993 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:48:40.993 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:48:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[31360]: pgmap v6905: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[40250]: pgmap v6905: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:41 smithi122 ceph-mon[32255]: pgmap v6905: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:48:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:48:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:48:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:43 smithi022 ceph-mon[31360]: pgmap v6906: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:43 smithi022 ceph-mon[40250]: pgmap v6906: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:43 smithi122 ceph-mon[32255]: pgmap v6906: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:44.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:45.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:45.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:45.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:45.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:48:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:48:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:45 smithi022 ceph-mon[40250]: pgmap v6907: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4177417649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:45 smithi022 ceph-mon[31360]: pgmap v6907: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:46.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4177417649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:45 smithi122 ceph-mon[32255]: pgmap v6907: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4177417649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:47.710 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:48:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:47 smithi022 ceph-mon[31360]: pgmap v6908: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:47 smithi022 ceph-mon[40250]: pgmap v6908: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:47 smithi122 ceph-mon[32255]: pgmap v6908: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:49 smithi022 ceph-mon[31360]: pgmap v6909: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:50.101 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:49 smithi022 ceph-mon[40250]: pgmap v6909: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:50.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:49 smithi122 ceph-mon[32255]: pgmap v6909: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:50.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:50.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:50] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T21:48:50.717 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1077343269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1077343269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:51.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1077343269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:51 smithi022 ceph-mon[31360]: pgmap v6910: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:51 smithi022 ceph-mon[40250]: pgmap v6910: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:51 smithi122 ceph-mon[32255]: pgmap v6910: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:48:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:48:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:53 smithi022 ceph-mon[31360]: pgmap v6911: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:53 smithi022 ceph-mon[40250]: pgmap v6911: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:53 smithi122 ceph-mon[32255]: pgmap v6911: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:55.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:48:55.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:48:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:48:55.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:48:55.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:48:55.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:48:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:55 smithi022 ceph-mon[31360]: pgmap v6912: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:55 smithi022 ceph-mon[40250]: pgmap v6912: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:55 smithi122 ceph-mon[32255]: pgmap v6912: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:48:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3789108517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3789108517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:48:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3789108517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:48:57.730 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:48:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:57 smithi022 ceph-mon[31360]: pgmap v6913: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:58.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:57 smithi022 ceph-mon[40250]: pgmap v6913: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:48:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:57 smithi122 ceph-mon[32255]: pgmap v6913: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:48:59 smithi022 ceph-mon[31360]: pgmap v6914: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:48:59 smithi022 ceph-mon[40250]: pgmap v6914: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:48:59 smithi122 ceph-mon[32255]: pgmap v6914: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:00] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T21:49:00.793 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:01.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:01.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:01.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:01 smithi022 ceph-mon[31360]: pgmap v6915: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/561955325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:01 smithi022 ceph-mon[40250]: pgmap v6915: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/561955325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:01 smithi122 ceph-mon[32255]: pgmap v6915: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:02.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/561955325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:49:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:49:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:03 smithi022 ceph-mon[31360]: pgmap v6916: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:03 smithi022 ceph-mon[40250]: pgmap v6916: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:03 smithi122 ceph-mon[32255]: pgmap v6916: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:05.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:49:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:49:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:05 smithi022 ceph-mon[31360]: pgmap v6917: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:05 smithi022 ceph-mon[40250]: pgmap v6917: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:06.163 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:05 smithi122 ceph-mon[32255]: pgmap v6917: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:06.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:06.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:06.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3816328303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3816328303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3816328303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:07.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:49:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:07 smithi022 ceph-mon[31360]: pgmap v6918: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:07 smithi022 ceph-mon[40250]: pgmap v6918: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:07 smithi122 ceph-mon[32255]: pgmap v6918: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:09 smithi022 ceph-mon[31360]: pgmap v6919: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:09 smithi022 ceph-mon[40250]: pgmap v6919: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:09 smithi122 ceph-mon[32255]: pgmap v6919: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:10] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:49:11.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:11.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:11.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:11.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:11.898 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:11 smithi022 ceph-mon[40250]: pgmap v6920: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:11.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:11 smithi022 ceph-mon[31360]: pgmap v6920: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:11 smithi122 ceph-mon[32255]: pgmap v6920: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:49:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:49:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/808818924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/808818924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/808818924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:13 smithi022 ceph-mon[31360]: pgmap v6921: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:13 smithi022 ceph-mon[40250]: pgmap v6921: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:13 smithi122 ceph-mon[32255]: pgmap v6921: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:15.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:49:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:49:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:15 smithi022 ceph-mon[31360]: pgmap v6922: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:15 smithi022 ceph-mon[40250]: pgmap v6922: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:15 smithi122 ceph-mon[32255]: pgmap v6922: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:16.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:17.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:17.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:17.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:49:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:17 smithi022 ceph-mon[31360]: pgmap v6923: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3546639940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:17 smithi022 ceph-mon[40250]: pgmap v6923: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3546639940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:17 smithi122 ceph-mon[32255]: pgmap v6923: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3546639940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:19 smithi022 ceph-mon[31360]: pgmap v6924: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:19 smithi022 ceph-mon[40250]: pgmap v6924: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:19 smithi122 ceph-mon[32255]: pgmap v6924: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:20] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:49:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:21 smithi022 ceph-mon[31360]: pgmap v6925: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:21 smithi022 ceph-mon[40250]: pgmap v6925: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:21 smithi122 ceph-mon[32255]: pgmap v6925: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:22.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:22.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:22.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:22.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:49:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:49:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2528938251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2528938251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2528938251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:23 smithi022 ceph-mon[31360]: pgmap v6926: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:23 smithi022 ceph-mon[40250]: pgmap v6926: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:23 smithi122 ceph-mon[32255]: pgmap v6926: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:25.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:49:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:49:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:25 smithi122 ceph-mon[32255]: pgmap v6927: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:25 smithi022 ceph-mon[31360]: pgmap v6927: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:25 smithi022 ceph-mon[40250]: pgmap v6927: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:27 smithi122 ceph-mon[32255]: pgmap v6928: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:27 smithi022 ceph-mon[31360]: pgmap v6928: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:27 smithi022 ceph-mon[40250]: pgmap v6928: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4237448122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4237448122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4237448122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:29 smithi122 ceph-mon[32255]: pgmap v6929: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:29 smithi022 ceph-mon[31360]: pgmap v6929: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:29 smithi022 ceph-mon[40250]: pgmap v6929: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:30] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:49:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:31 smithi122 ceph-mon[32255]: pgmap v6930: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:31 smithi022 ceph-mon[31360]: pgmap v6930: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:31 smithi022 ceph-mon[40250]: pgmap v6930: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:49:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:49:33.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:33.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:33.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:33.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:33 smithi122 ceph-mon[32255]: pgmap v6931: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:49:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:49:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:49:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:49:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1803898510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[31360]: pgmap v6931: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:49:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:49:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:49:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:49:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1803898510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[40250]: pgmap v6931: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:49:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:49:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:49:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:49:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1803898510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:35.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:49:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:49:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:35 smithi122 ceph-mon[32255]: pgmap v6932: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:35 smithi022 ceph-mon[31360]: pgmap v6932: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:35 smithi022 ceph-mon[40250]: pgmap v6932: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:49:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:37 smithi122 ceph-mon[32255]: pgmap v6933: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:37 smithi022 ceph-mon[31360]: pgmap v6933: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:37 smithi022 ceph-mon[40250]: pgmap v6933: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:38.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:38.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:38.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:38.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:39 smithi122 ceph-mon[32255]: pgmap v6934: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/279714354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:39 smithi022 ceph-mon[31360]: pgmap v6934: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/279714354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:39 smithi022 ceph-mon[40250]: pgmap v6934: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/279714354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:40] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:49:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:41 smithi122 ceph-mon[32255]: pgmap v6935: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:41 smithi022 ceph-mon[31360]: pgmap v6935: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:41 smithi022 ceph-mon[40250]: pgmap v6935: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:49:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:49:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:49:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:49:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:49:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:49:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:49:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:49:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:49:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:49:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:49:42.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:49:42.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:49:42.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:49:43.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:43 smithi022 ceph-mon[31360]: pgmap v6936: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:43 smithi022 ceph-mon[40250]: pgmap v6936: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:44.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:44.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:44.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:43 smithi122 ceph-mon[32255]: pgmap v6936: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:45.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:49:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:49:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4147972420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4147972420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4147972420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:45 smithi122 ceph-mon[32255]: pgmap v6937: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:45 smithi022 ceph-mon[31360]: pgmap v6937: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:45 smithi022 ceph-mon[40250]: pgmap v6937: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:49:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:47 smithi122 ceph-mon[32255]: pgmap v6938: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:47 smithi022 ceph-mon[31360]: pgmap v6938: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:47 smithi022 ceph-mon[40250]: pgmap v6938: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:49.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:49.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:49.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:49 smithi122 ceph-mon[32255]: pgmap v6939: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:49 smithi022 ceph-mon[31360]: pgmap v6939: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:49 smithi022 ceph-mon[40250]: pgmap v6939: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3444040465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:50] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:49:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3444040465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3444040465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:51 smithi122 ceph-mon[32255]: pgmap v6940: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:51 smithi022 ceph-mon[31360]: pgmap v6940: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:51 smithi022 ceph-mon[40250]: pgmap v6940: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:49:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:49:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:53 smithi122 ceph-mon[32255]: pgmap v6941: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:53 smithi022 ceph-mon[31360]: pgmap v6941: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:53 smithi022 ceph-mon[40250]: pgmap v6941: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:54.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:49:54.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:49:54.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:49:54.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:49:55.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:49:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:49:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:55 smithi122 ceph-mon[32255]: pgmap v6942: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2078737021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:55 smithi022 ceph-mon[31360]: pgmap v6942: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2078737021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:55 smithi022 ceph-mon[40250]: pgmap v6942: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2078737021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:49:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:49:57.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:57 smithi122 ceph-mon[32255]: pgmap v6943: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:57 smithi022 ceph-mon[31360]: pgmap v6943: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:49:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:57 smithi022 ceph-mon[40250]: pgmap v6943: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:49:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:49:59 smithi122 ceph-mon[32255]: pgmap v6944: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:49:59 smithi022 ceph-mon[31360]: pgmap v6944: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:49:59 smithi022 ceph-mon[40250]: pgmap v6944: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:49:59.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:00.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:00.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:00.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:00] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:50:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T21:50:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1449094173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T21:50:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1449094173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T21:50:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1449094173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:01.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:01 smithi122 ceph-mon[32255]: pgmap v6945: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:01 smithi022 ceph-mon[31360]: pgmap v6945: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:01 smithi022 ceph-mon[40250]: pgmap v6945: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:50:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:50:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:03 smithi122 ceph-mon[32255]: pgmap v6946: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:03 smithi022 ceph-mon[31360]: pgmap v6946: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:03 smithi022 ceph-mon[40250]: pgmap v6946: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:05.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:05.420 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:50:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:50:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:05 smithi022 ceph-mon[31360]: pgmap v6947: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:05 smithi022 ceph-mon[40250]: pgmap v6947: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:05.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:05.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:05.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:05 smithi122 ceph-mon[32255]: pgmap v6947: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2401182558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2401182558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2401182558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:07 smithi122 ceph-mon[32255]: pgmap v6948: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:07 smithi022 ceph-mon[31360]: pgmap v6948: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:50:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:07 smithi022 ceph-mon[40250]: pgmap v6948: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:09 smithi122 ceph-mon[32255]: pgmap v6949: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:09 smithi022 ceph-mon[31360]: pgmap v6949: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:09 smithi022 ceph-mon[40250]: pgmap v6949: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:10] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:50:10.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:10.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:10.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:10.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:11 smithi122 ceph-mon[32255]: pgmap v6950: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3139248839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:11 smithi022 ceph-mon[31360]: pgmap v6950: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3139248839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:11 smithi022 ceph-mon[40250]: pgmap v6950: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3139248839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:50:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:50:13.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:13 smithi122 ceph-mon[32255]: pgmap v6951: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:13 smithi022 ceph-mon[31360]: pgmap v6951: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:13 smithi022 ceph-mon[40250]: pgmap v6951: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:15.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:50:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:50:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:15 smithi022 ceph-mon[31360]: pgmap v6952: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:15 smithi022 ceph-mon[40250]: pgmap v6952: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:15 smithi122 ceph-mon[32255]: pgmap v6952: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:16.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:16.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:16.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2288384588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2288384588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2288384588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:17 smithi022 ceph-mon[31360]: pgmap v6953: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:50:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:17 smithi022 ceph-mon[40250]: pgmap v6953: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:17 smithi122 ceph-mon[32255]: pgmap v6953: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:19 smithi022 ceph-mon[31360]: pgmap v6954: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:19 smithi022 ceph-mon[40250]: pgmap v6954: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:19 smithi122 ceph-mon[32255]: pgmap v6954: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:20] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:50:21.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:21.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:21.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:21.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:21.714 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:21 smithi022 ceph-mon[31360]: pgmap v6955: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:21.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:21 smithi022 ceph-mon[40250]: pgmap v6955: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:21 smithi122 ceph-mon[32255]: pgmap v6955: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1024082824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:50:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:50:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1024082824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1024082824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:23 smithi022 ceph-mon[31360]: pgmap v6956: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:23 smithi022 ceph-mon[40250]: pgmap v6956: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:23 smithi122 ceph-mon[32255]: pgmap v6956: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:25.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:50:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:50:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:25 smithi022 ceph-mon[31360]: pgmap v6957: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:25 smithi022 ceph-mon[40250]: pgmap v6957: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:25 smithi122 ceph-mon[32255]: pgmap v6957: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:26.716 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:26.717 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:27.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:27.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:27.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:27 smithi022 ceph-mon[31360]: pgmap v6958: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3695182294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:50:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:27 smithi022 ceph-mon[40250]: pgmap v6958: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3695182294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:27 smithi122 ceph-mon[32255]: pgmap v6958: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3695182294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:29 smithi022 ceph-mon[40250]: pgmap v6959: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:29 smithi022 ceph-mon[31360]: pgmap v6959: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:29 smithi122 ceph-mon[32255]: pgmap v6959: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:30] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T21:50:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:31 smithi022 ceph-mon[31360]: pgmap v6960: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:31 smithi022 ceph-mon[40250]: pgmap v6960: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:31 smithi122 ceph-mon[32255]: pgmap v6960: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:32.084 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:32.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:32.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:32.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2899221977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:50:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:50:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2899221977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2899221977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[31360]: pgmap v6961: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:50:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:50:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:50:33.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:50:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[40250]: pgmap v6961: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:50:33.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:50:33.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:50:33.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:50:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:33 smithi122 ceph-mon[32255]: pgmap v6961: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:50:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:50:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:50:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:50:35.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:50:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:50:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:35 smithi022 ceph-mon[31360]: pgmap v6962: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:35 smithi022 ceph-mon[40250]: pgmap v6962: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:35 smithi122 ceph-mon[32255]: pgmap v6962: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:37.453 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:50:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:37 smithi022 ceph-mon[31360]: pgmap v6963: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:37 smithi022 ceph-mon[40250]: pgmap v6963: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:37.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:37.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:37.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:37 smithi122 ceph-mon[32255]: pgmap v6963: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/453927632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/453927632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/453927632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:39 smithi022 ceph-mon[31360]: pgmap v6964: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:39 smithi022 ceph-mon[40250]: pgmap v6964: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:39 smithi122 ceph-mon[32255]: pgmap v6964: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:40] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:50:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:41 smithi022 ceph-mon[31360]: pgmap v6965: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:41 smithi022 ceph-mon[40250]: pgmap v6965: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:41.933 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:41 smithi122 ceph-mon[32255]: pgmap v6965: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:50:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:50:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:50:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:50:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:50:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:50:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:50:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:50:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:50:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:50:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:50:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:50:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:50:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:50:42.825 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:43.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:43.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:43.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:43 smithi022 ceph-mon[31360]: pgmap v6966: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/460397737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:43 smithi022 ceph-mon[40250]: pgmap v6966: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/460397737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:43 smithi122 ceph-mon[32255]: pgmap v6966: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/460397737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:45.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:50:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:50:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:45 smithi022 ceph-mon[31360]: pgmap v6967: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:45 smithi022 ceph-mon[40250]: pgmap v6967: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:45 smithi122 ceph-mon[32255]: pgmap v6967: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:47 smithi022 ceph-mon[31360]: pgmap v6968: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:47.863 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:50:47.864 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:47 smithi022 ceph-mon[40250]: pgmap v6968: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:47 smithi122 ceph-mon[32255]: pgmap v6968: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:48.194 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:49 smithi022 ceph-mon[31360]: pgmap v6969: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/135283291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:49 smithi022 ceph-mon[40250]: pgmap v6969: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/135283291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:49 smithi122 ceph-mon[32255]: pgmap v6969: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/135283291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:50] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:50:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:51 smithi022 ceph-mon[40250]: pgmap v6970: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:51 smithi022 ceph-mon[31360]: pgmap v6970: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:51 smithi122 ceph-mon[32255]: pgmap v6970: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:50:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:50:53.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:53 smithi022 ceph-mon[31360]: pgmap v6971: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:53 smithi022 ceph-mon[40250]: pgmap v6971: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:53.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:53.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:53.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:53 smithi122 ceph-mon[32255]: pgmap v6971: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3345064413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3345064413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3345064413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:55.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:50:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:50:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:55 smithi022 ceph-mon[31360]: pgmap v6972: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:55 smithi022 ceph-mon[40250]: pgmap v6972: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:55 smithi122 ceph-mon[32255]: pgmap v6972: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:50:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:57 smithi022 ceph-mon[31360]: pgmap v6973: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:50:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:57 smithi022 ceph-mon[40250]: pgmap v6973: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:57 smithi122 ceph-mon[32255]: pgmap v6973: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:50:58.933 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:50:59.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:50:59.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:50:59.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:50:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:59 smithi022 ceph-mon[40250]: pgmap v6974: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:50:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1294257688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:59.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:59 smithi022 ceph-mon[31360]: pgmap v6974: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:59.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:50:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1294257688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:50:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:59 smithi122 ceph-mon[32255]: pgmap v6974: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:50:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:50:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1294257688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:00] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:51:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:01 smithi122 ceph-mon[32255]: pgmap v6975: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:01 smithi022 ceph-mon[31360]: pgmap v6975: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:01 smithi022 ceph-mon[40250]: pgmap v6975: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:51:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:51:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:03 smithi122 ceph-mon[32255]: pgmap v6976: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:03 smithi022 ceph-mon[40250]: pgmap v6976: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:03 smithi022 ceph-mon[31360]: pgmap v6976: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:04.302 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:04.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:04.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:04.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:05.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:51:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:51:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:05 smithi122 ceph-mon[32255]: pgmap v6977: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4172425743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:05 smithi022 ceph-mon[31360]: pgmap v6977: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4172425743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:05 smithi022 ceph-mon[40250]: pgmap v6977: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4172425743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:07 smithi022 ceph-mon[31360]: pgmap v6978: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:51:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:07 smithi022 ceph-mon[40250]: pgmap v6978: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:07 smithi122 ceph-mon[32255]: pgmap v6978: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:09.673 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:09 smithi122 ceph-mon[32255]: pgmap v6979: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:09 smithi022 ceph-mon[31360]: pgmap v6979: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:09 smithi022 ceph-mon[40250]: pgmap v6979: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:10.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:10.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:10.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:10] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:51:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/229815671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/229815671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/229815671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:11 smithi122 ceph-mon[32255]: pgmap v6980: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:11 smithi022 ceph-mon[31360]: pgmap v6980: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:11 smithi022 ceph-mon[40250]: pgmap v6980: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:51:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:51:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:13 smithi122 ceph-mon[32255]: pgmap v6981: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:14.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:13 smithi022 ceph-mon[31360]: pgmap v6981: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:14.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:13 smithi022 ceph-mon[40250]: pgmap v6981: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:15.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:15.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:15.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:15.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:15.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:51:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:51:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:15 smithi022 ceph-mon[31360]: pgmap v6982: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3567838926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:15.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:15 smithi022 ceph-mon[40250]: pgmap v6982: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3567838926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:15 smithi122 ceph-mon[32255]: pgmap v6982: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3567838926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:17 smithi022 ceph-mon[31360]: pgmap v6983: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:51:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:17 smithi022 ceph-mon[40250]: pgmap v6983: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:17.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:17 smithi122 ceph-mon[32255]: pgmap v6983: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:19 smithi122 ceph-mon[32255]: pgmap v6984: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:19 smithi022 ceph-mon[31360]: pgmap v6984: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:19 smithi022 ceph-mon[40250]: pgmap v6984: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:20.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:20] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:51:20.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:20.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:20.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:21 smithi122 ceph-mon[32255]: pgmap v6985: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2079291114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:21 smithi022 ceph-mon[31360]: pgmap v6985: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2079291114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:21 smithi022 ceph-mon[40250]: pgmap v6985: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2079291114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:51:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:51:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:23 smithi122 ceph-mon[32255]: pgmap v6986: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:23 smithi022 ceph-mon[31360]: pgmap v6986: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:23 smithi022 ceph-mon[40250]: pgmap v6986: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:25.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:51:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:51:25.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:25 smithi122 ceph-mon[32255]: pgmap v6987: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:25 smithi022 ceph-mon[31360]: pgmap v6987: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:25 smithi022 ceph-mon[40250]: pgmap v6987: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:26.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3795316943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3795316943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3795316943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:27 smithi022 ceph-mon[31360]: pgmap v6988: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:51:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:27 smithi022 ceph-mon[40250]: pgmap v6988: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:27 smithi122 ceph-mon[32255]: pgmap v6988: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:29 smithi122 ceph-mon[32255]: pgmap v6989: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:29 smithi022 ceph-mon[31360]: pgmap v6989: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:29 smithi022 ceph-mon[40250]: pgmap v6989: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:30] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:51:31.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:31.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:31.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:31.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:31 smithi022 ceph-mon[31360]: pgmap v6990: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2909017838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:31 smithi022 ceph-mon[40250]: pgmap v6990: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2909017838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:31 smithi122 ceph-mon[32255]: pgmap v6990: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2909017838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:51:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:51:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:33 smithi122 ceph-mon[32255]: pgmap v6991: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:51:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:51:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:51:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:51:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[31360]: pgmap v6991: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:51:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:51:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:51:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:51:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[40250]: pgmap v6991: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:51:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:51:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:51:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:51:35.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:51:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:51:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:35 smithi122 ceph-mon[32255]: pgmap v6992: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:35 smithi022 ceph-mon[31360]: pgmap v6992: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:35 smithi022 ceph-mon[40250]: pgmap v6992: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:36.526 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:36.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:36.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:36.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:37 smithi022 ceph-mon[31360]: pgmap v6993: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2702629428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:51:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:37 smithi022 ceph-mon[40250]: pgmap v6993: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2702629428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:37 smithi122 ceph-mon[32255]: pgmap v6993: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2702629428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:39 smithi122 ceph-mon[32255]: pgmap v6994: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:39 smithi022 ceph-mon[31360]: pgmap v6994: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:39 smithi022 ceph-mon[40250]: pgmap v6994: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:40] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:51:41.893 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:41.894 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:41 smithi022 ceph-mon[31360]: pgmap v6995: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:41.894 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:41 smithi022 ceph-mon[40250]: pgmap v6995: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:41 smithi122 ceph-mon[32255]: pgmap v6995: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:42.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:42.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:42.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:42.611 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:51:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:51:42.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3660237402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:42.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:51:42.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:51:42.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:51:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3660237402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:51:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:51:43.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:51:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3660237402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:51:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:51:43.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:51:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:43 smithi122 ceph-mon[32255]: pgmap v6996: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:51:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:43 smithi022 ceph-mon[31360]: pgmap v6996: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:51:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:43 smithi022 ceph-mon[40250]: pgmap v6996: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:51:45.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:51:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:51:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:45 smithi122 ceph-mon[32255]: pgmap v6997: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:45 smithi022 ceph-mon[31360]: pgmap v6997: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:45 smithi022 ceph-mon[40250]: pgmap v6997: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:47.266 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:47.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:47.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:51:47.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:47.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:47.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:47 smithi022 ceph-mon[31360]: pgmap v6998: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1468258480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:47 smithi022 ceph-mon[40250]: pgmap v6998: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1468258480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:47 smithi122 ceph-mon[32255]: pgmap v6998: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1468258480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:49 smithi122 ceph-mon[32255]: pgmap v6999: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:49 smithi022 ceph-mon[31360]: pgmap v6999: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:49 smithi022 ceph-mon[40250]: pgmap v6999: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:50] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:51:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:51 smithi122 ceph-mon[32255]: pgmap v7000: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:51 smithi022 ceph-mon[31360]: pgmap v7000: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:51 smithi022 ceph-mon[40250]: pgmap v7000: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:52.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:51:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:51:53.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:53.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:53.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:53 smithi122 ceph-mon[32255]: pgmap v7001: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2900810251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:53 smithi022 ceph-mon[31360]: pgmap v7001: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2900810251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:53 smithi022 ceph-mon[40250]: pgmap v7001: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2900810251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:55.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:51:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:51:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:55 smithi122 ceph-mon[32255]: pgmap v7002: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:55 smithi022 ceph-mon[31360]: pgmap v7002: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:55 smithi022 ceph-mon[40250]: pgmap v7002: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:51:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:51:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:51:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:51:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:57 smithi122 ceph-mon[32255]: pgmap v7003: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:57 smithi022 ceph-mon[31360]: pgmap v7003: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:57 smithi022 ceph-mon[40250]: pgmap v7003: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:51:58.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:51:58.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:51:58.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:51:59.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3986182193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3986182193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:51:59.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3986182193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:51:59 smithi122 ceph-mon[32255]: pgmap v7004: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:51:59 smithi022 ceph-mon[31360]: pgmap v7004: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:51:59 smithi022 ceph-mon[40250]: pgmap v7004: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:00] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:52:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:01 smithi122 ceph-mon[32255]: pgmap v7005: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:01 smithi022 ceph-mon[31360]: pgmap v7005: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:01 smithi022 ceph-mon[40250]: pgmap v7005: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:52:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:52:03.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:03.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:03.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:03.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:03 smithi022 ceph-mon[31360]: pgmap v7006: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2152951260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:03 smithi022 ceph-mon[40250]: pgmap v7006: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2152951260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:03 smithi122 ceph-mon[32255]: pgmap v7006: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2152951260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:05.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:52:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:52:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:05 smithi122 ceph-mon[32255]: pgmap v7007: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:05 smithi022 ceph-mon[31360]: pgmap v7007: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:05 smithi022 ceph-mon[40250]: pgmap v7007: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:52:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:07 smithi122 ceph-mon[32255]: pgmap v7008: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:07 smithi022 ceph-mon[31360]: pgmap v7008: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:07 smithi022 ceph-mon[40250]: pgmap v7008: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:08.744 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:09.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:09.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:09.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:09 smithi122 ceph-mon[32255]: pgmap v7009: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3300563104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:09 smithi022 ceph-mon[31360]: pgmap v7009: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3300563104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:09 smithi022 ceph-mon[40250]: pgmap v7009: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3300563104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:10] "GET /metrics HTTP/1.1" 200 202357 "" "Prometheus/2.18.1" 2021-12-15T21:52:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:11 smithi122 ceph-mon[32255]: pgmap v7010: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:11 smithi022 ceph-mon[31360]: pgmap v7010: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:11 smithi022 ceph-mon[40250]: pgmap v7010: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:52:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:52:14.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:13 smithi122 ceph-mon[32255]: pgmap v7011: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:13 smithi022 ceph-mon[31360]: pgmap v7011: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:13 smithi022 ceph-mon[40250]: pgmap v7011: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:14.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:14.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:14.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:15.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:52:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:52:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1108422083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1108422083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1108422083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:15 smithi122 ceph-mon[32255]: pgmap v7012: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:15 smithi022 ceph-mon[31360]: pgmap v7012: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:15 smithi022 ceph-mon[40250]: pgmap v7012: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:52:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:17 smithi122 ceph-mon[32255]: pgmap v7013: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:17 smithi022 ceph-mon[31360]: pgmap v7013: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:17 smithi022 ceph-mon[40250]: pgmap v7013: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:19.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:19.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:19.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:19.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:19 smithi122 ceph-mon[32255]: pgmap v7014: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/131835697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:19 smithi022 ceph-mon[31360]: pgmap v7014: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/131835697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:19 smithi022 ceph-mon[40250]: pgmap v7014: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/131835697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:20] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:52:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:21 smithi122 ceph-mon[32255]: pgmap v7015: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:21 smithi022 ceph-mon[31360]: pgmap v7015: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:21 smithi022 ceph-mon[40250]: pgmap v7015: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:52:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:52:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:23 smithi122 ceph-mon[32255]: pgmap v7016: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:23 smithi022 ceph-mon[40250]: pgmap v7016: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:23 smithi022 ceph-mon[31360]: pgmap v7016: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:24.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:25.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:25.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:25.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:25.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:52:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:52:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:25 smithi122 ceph-mon[32255]: pgmap v7017: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/430774778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:25 smithi022 ceph-mon[31360]: pgmap v7017: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/430774778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:25 smithi022 ceph-mon[40250]: pgmap v7017: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/430774778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:52:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:27 smithi122 ceph-mon[32255]: pgmap v7018: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:28.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:27 smithi022 ceph-mon[31360]: pgmap v7018: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:27 smithi022 ceph-mon[40250]: pgmap v7018: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:29 smithi122 ceph-mon[32255]: pgmap v7019: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:30.278 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:29 smithi022 ceph-mon[31360]: pgmap v7019: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:29 smithi022 ceph-mon[40250]: pgmap v7019: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:30] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:52:30.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:30.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:30.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1766656396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:31.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1766656396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1766656396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:31 smithi122 ceph-mon[32255]: pgmap v7020: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:31 smithi022 ceph-mon[31360]: pgmap v7020: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:31 smithi022 ceph-mon[40250]: pgmap v7020: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:52:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:52:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:33 smithi122 ceph-mon[32255]: pgmap v7021: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:52:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:52:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:52:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:52:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[31360]: pgmap v7021: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:52:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:52:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:52:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[40250]: pgmap v7021: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:52:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:52:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:52:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:52:35.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:52:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:52:35.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:35.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:35.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:35.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:35 smithi122 ceph-mon[32255]: pgmap v7022: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:35 smithi022 ceph-mon[31360]: pgmap v7022: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:35 smithi022 ceph-mon[40250]: pgmap v7022: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1158649442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1158649442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1158649442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:52:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:37 smithi122 ceph-mon[32255]: pgmap v7023: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:37 smithi022 ceph-mon[31360]: pgmap v7023: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:37 smithi022 ceph-mon[40250]: pgmap v7023: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:40.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:40] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:52:40.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:39 smithi122 ceph-mon[32255]: pgmap v7024: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:39 smithi022 ceph-mon[31360]: pgmap v7024: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:39 smithi022 ceph-mon[40250]: pgmap v7024: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:41.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:41.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:41.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:41 smithi122 ceph-mon[32255]: pgmap v7025: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1008918279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:41 smithi022 ceph-mon[31360]: pgmap v7025: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1008918279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:41 smithi022 ceph-mon[40250]: pgmap v7025: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1008918279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:52:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:52:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:52:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:52:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:52:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:52:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:52:43.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:52:43.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:52:43.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:52:43.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:52:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:43 smithi022 ceph-mon[31360]: pgmap v7026: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:43 smithi022 ceph-mon[40250]: pgmap v7026: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:43 smithi122 ceph-mon[32255]: pgmap v7026: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:44 smithi022 ceph-mon[31360]: pgmap v7027: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:44 smithi022 ceph-mon[40250]: pgmap v7027: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:44 smithi122 ceph-mon[32255]: pgmap v7027: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:45.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:52:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:52:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:46.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:46.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:46.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:46.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: pgmap v7028: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2498964490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:47.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3228M 2021-12-15T21:52:47.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:52:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: pgmap v7028: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2498964490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3228M 2021-12-15T21:52:47.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: pgmap v7028: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2498964490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T21:52:47.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3228M 2021-12-15T21:52:47.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:47.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:52:50.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:49 smithi122 ceph-mon[32255]: pgmap v7029: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:49 smithi022 ceph-mon[31360]: pgmap v7029: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:49 smithi022 ceph-mon[40250]: pgmap v7029: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:50] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:52:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:51 smithi022 ceph-mon[40250]: pgmap v7030: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:51 smithi022 ceph-mon[31360]: pgmap v7030: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:52.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:52.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:51 smithi122 ceph-mon[32255]: pgmap v7030: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:52:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:52:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1660261617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1660261617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1660261617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:53 smithi122 ceph-mon[32255]: pgmap v7031: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:53 smithi022 ceph-mon[31360]: pgmap v7031: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:53 smithi022 ceph-mon[40250]: pgmap v7031: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:55.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:52:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:52:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:55 smithi122 ceph-mon[32255]: pgmap v7032: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:55 smithi022 ceph-mon[31360]: pgmap v7032: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:55 smithi022 ceph-mon[40250]: pgmap v7032: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:52:57.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:52:57.100 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:57.101 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:52:57.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:52:57.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:52:57.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:52:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:52:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:57 smithi122 ceph-mon[32255]: pgmap v7033: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2181624996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:57 smithi022 ceph-mon[31360]: pgmap v7033: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2181624996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:52:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:57 smithi022 ceph-mon[40250]: pgmap v7033: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:52:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2181624996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:52:59 smithi122 ceph-mon[32255]: pgmap v7034: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:52:59 smithi022 ceph-mon[31360]: pgmap v7034: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:52:59 smithi022 ceph-mon[40250]: pgmap v7034: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:00] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:53:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:01 smithi122 ceph-mon[32255]: pgmap v7035: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:01 smithi022 ceph-mon[31360]: pgmap v7035: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:01 smithi022 ceph-mon[40250]: pgmap v7035: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:02.468 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:53:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:53:02.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:02.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:02.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:03 smithi122 ceph-mon[32255]: pgmap v7036: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3732756661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:03 smithi022 ceph-mon[31360]: pgmap v7036: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3732756661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:03 smithi022 ceph-mon[40250]: pgmap v7036: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3732756661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:05.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:53:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:53:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:05 smithi122 ceph-mon[32255]: pgmap v7037: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:05 smithi022 ceph-mon[31360]: pgmap v7037: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:05 smithi022 ceph-mon[40250]: pgmap v7037: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:53:07.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:07 smithi122 ceph-mon[32255]: pgmap v7038: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:08.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:08.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:08.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:08.211 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:07 smithi022 ceph-mon[40250]: pgmap v7038: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:08.211 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:07 smithi022 ceph-mon[31360]: pgmap v7038: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3417348203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3417348203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3417348203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:09 smithi122 ceph-mon[32255]: pgmap v7039: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:09 smithi022 ceph-mon[31360]: pgmap v7039: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:09 smithi022 ceph-mon[40250]: pgmap v7039: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:10] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T21:53:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:11 smithi122 ceph-mon[32255]: pgmap v7040: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:11 smithi022 ceph-mon[31360]: pgmap v7040: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:11 smithi022 ceph-mon[40250]: pgmap v7040: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:53:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:53:13.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:13 smithi022 ceph-mon[31360]: pgmap v7041: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3677721472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:13 smithi022 ceph-mon[40250]: pgmap v7041: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3677721472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:13 smithi122 ceph-mon[32255]: pgmap v7041: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3677721472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:15.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:53:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:53:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:15 smithi122 ceph-mon[32255]: pgmap v7042: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:15 smithi022 ceph-mon[31360]: pgmap v7042: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:16.391 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:15 smithi022 ceph-mon[40250]: pgmap v7042: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:53:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:17 smithi122 ceph-mon[32255]: pgmap v7043: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:17 smithi022 ceph-mon[31360]: pgmap v7043: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:17 smithi022 ceph-mon[40250]: pgmap v7043: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:18.585 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:18.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:18.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:18.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:19 smithi122 ceph-mon[32255]: pgmap v7044: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1363957676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:19 smithi022 ceph-mon[40250]: pgmap v7044: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1363957676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:19 smithi022 ceph-mon[31360]: pgmap v7044: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1363957676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:20] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T21:53:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:21 smithi122 ceph-mon[32255]: pgmap v7045: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:21 smithi022 ceph-mon[31360]: pgmap v7045: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:21 smithi022 ceph-mon[40250]: pgmap v7045: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:53:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:53:23.957 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:23 smithi122 ceph-mon[32255]: pgmap v7046: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:23 smithi022 ceph-mon[31360]: pgmap v7046: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:23 smithi022 ceph-mon[40250]: pgmap v7046: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:24.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:24.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:24.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:25.160 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1413628303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1413628303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1413628303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:25.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:53:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:53:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:25 smithi122 ceph-mon[32255]: pgmap v7047: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:25 smithi022 ceph-mon[31360]: pgmap v7047: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:25 smithi022 ceph-mon[40250]: pgmap v7047: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:27.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:27.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:53:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:27 smithi122 ceph-mon[32255]: pgmap v7048: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:27 smithi022 ceph-mon[40250]: pgmap v7048: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:27 smithi022 ceph-mon[31360]: pgmap v7048: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:29.332 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:29.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:29.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:29.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:30.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:29 smithi022 ceph-mon[31360]: pgmap v7049: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:30.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2980405974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:29 smithi022 ceph-mon[40250]: pgmap v7049: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:30.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2980405974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:29 smithi122 ceph-mon[32255]: pgmap v7049: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2980405974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:30] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T21:53:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:31 smithi122 ceph-mon[32255]: pgmap v7050: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:31 smithi022 ceph-mon[31360]: pgmap v7050: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:31 smithi022 ceph-mon[40250]: pgmap v7050: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:53:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:53:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:33 smithi122 ceph-mon[32255]: pgmap v7051: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:53:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:53:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:53:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:53:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[31360]: pgmap v7051: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:53:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:53:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:53:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:53:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[40250]: pgmap v7051: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:53:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:53:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:53:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:53:34.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:35.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:35.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:35.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:35.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:53:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:53:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:35 smithi122 ceph-mon[32255]: pgmap v7052: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2403992554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:35 smithi022 ceph-mon[40250]: pgmap v7052: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2403992554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:36.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:35 smithi022 ceph-mon[31360]: pgmap v7052: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:36.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2403992554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:53:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:37 smithi122 ceph-mon[32255]: pgmap v7053: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:37 smithi022 ceph-mon[31360]: pgmap v7053: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:37 smithi022 ceph-mon[40250]: pgmap v7053: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:40.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:39 smithi122 ceph-mon[32255]: pgmap v7054: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:39 smithi022 ceph-mon[31360]: pgmap v7054: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:39 smithi022 ceph-mon[40250]: pgmap v7054: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:40] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T21:53:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2834478625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2834478625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2834478625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:41 smithi122 ceph-mon[32255]: pgmap v7055: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:41 smithi022 ceph-mon[31360]: pgmap v7055: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:41 smithi022 ceph-mon[40250]: pgmap v7055: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:53:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:53:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:43 smithi122 ceph-mon[32255]: pgmap v7056: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:43 smithi022 ceph-mon[31360]: pgmap v7056: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:43 smithi022 ceph-mon[40250]: pgmap v7056: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:45.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:53:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:53:45.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:45.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:45.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:45.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:45 smithi122 ceph-mon[32255]: pgmap v7057: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/579822696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:45 smithi022 ceph-mon[31360]: pgmap v7057: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/579822696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:45 smithi022 ceph-mon[40250]: pgmap v7057: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/579822696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:47.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:53:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:47 smithi122 ceph-mon[32255]: pgmap v7058: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:53:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:53:48.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:53:48.250 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[31360]: pgmap v7058: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:48.250 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:53:48.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:53:48.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:53:48.251 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[40250]: pgmap v7058: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:48.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:53:48.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:53:48.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:53:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:53:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:49 smithi022 ceph-mon[31360]: pgmap v7059: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:53:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:49 smithi022 ceph-mon[40250]: pgmap v7059: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:53:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:49 smithi122 ceph-mon[32255]: pgmap v7059: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:50] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:53:50.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:51.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:51.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:51.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:51 smithi022 ceph-mon[31360]: pgmap v7060: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3048929723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:51 smithi022 ceph-mon[40250]: pgmap v7060: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3048929723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:51 smithi122 ceph-mon[32255]: pgmap v7060: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3048929723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:53:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:53:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:53 smithi122 ceph-mon[32255]: pgmap v7061: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:53 smithi022 ceph-mon[31360]: pgmap v7061: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:53 smithi022 ceph-mon[40250]: pgmap v7061: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:55.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:53:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:53:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:55 smithi022 ceph-mon[31360]: pgmap v7062: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:55 smithi022 ceph-mon[40250]: pgmap v7062: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:55 smithi122 ceph-mon[32255]: pgmap v7062: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:56.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:53:56.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:53:56.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:53:56.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:53:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:53:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:53:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:57 smithi022 ceph-mon[31360]: pgmap v7063: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1481719603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:57 smithi022 ceph-mon[40250]: pgmap v7063: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1481719603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:57 smithi122 ceph-mon[32255]: pgmap v7063: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:53:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1481719603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:53:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:53:59 smithi022 ceph-mon[31360]: pgmap v7064: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:53:59 smithi022 ceph-mon[40250]: pgmap v7064: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:53:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:53:59 smithi122 ceph-mon[32255]: pgmap v7064: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:00] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T21:54:01.554 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:01 smithi022 ceph-mon[31360]: pgmap v7065: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:01 smithi022 ceph-mon[40250]: pgmap v7065: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:01.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:01.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:01.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:01 smithi122 ceph-mon[32255]: pgmap v7065: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3029769498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:54:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:54:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3029769498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3029769498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:03 smithi022 ceph-mon[31360]: pgmap v7066: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:03 smithi022 ceph-mon[40250]: pgmap v7066: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:03 smithi122 ceph-mon[32255]: pgmap v7066: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:05.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:54:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:54:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:05 smithi022 ceph-mon[31360]: pgmap v7067: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:05 smithi022 ceph-mon[40250]: pgmap v7067: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:05 smithi122 ceph-mon[32255]: pgmap v7067: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:06.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:07.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:07.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:07.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:54:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:07 smithi022 ceph-mon[31360]: pgmap v7068: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1082474635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:07 smithi022 ceph-mon[40250]: pgmap v7068: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1082474635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:07 smithi122 ceph-mon[32255]: pgmap v7068: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1082474635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:09 smithi022 ceph-mon[31360]: pgmap v7069: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:09 smithi022 ceph-mon[40250]: pgmap v7069: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:09 smithi122 ceph-mon[32255]: pgmap v7069: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:10] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T21:54:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:11 smithi022 ceph-mon[31360]: pgmap v7070: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:11 smithi022 ceph-mon[40250]: pgmap v7070: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:11 smithi122 ceph-mon[32255]: pgmap v7070: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:12.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:12.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:12.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:12.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:54:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:54:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:13 smithi022 ceph-mon[31360]: pgmap v7071: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2642462480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:13 smithi022 ceph-mon[40250]: pgmap v7071: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2642462480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:13 smithi122 ceph-mon[32255]: pgmap v7071: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2642462480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:15.483 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:54:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:54:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:15 smithi022 ceph-mon[31360]: pgmap v7072: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:15 smithi022 ceph-mon[40250]: pgmap v7072: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:15 smithi122 ceph-mon[32255]: pgmap v7072: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:17.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:17.669 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:54:17.669 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:17 smithi022 ceph-mon[31360]: pgmap v7073: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:17.670 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:17 smithi022 ceph-mon[40250]: pgmap v7073: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:17 smithi122 ceph-mon[32255]: pgmap v7073: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2062450857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2062450857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2062450857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:19 smithi022 ceph-mon[31360]: pgmap v7074: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:19 smithi022 ceph-mon[40250]: pgmap v7074: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:19 smithi122 ceph-mon[32255]: pgmap v7074: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:20] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T21:54:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:21 smithi022 ceph-mon[31360]: pgmap v7075: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:21 smithi022 ceph-mon[40250]: pgmap v7075: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:21 smithi122 ceph-mon[32255]: pgmap v7075: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:54:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:54:23.036 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:23.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:23.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:23.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:23 smithi022 ceph-mon[31360]: pgmap v7076: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3456539995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:23.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:23 smithi022 ceph-mon[40250]: pgmap v7076: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:23.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3456539995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:23 smithi122 ceph-mon[32255]: pgmap v7076: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3456539995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:25.500 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:54:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:54:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:25 smithi022 ceph-mon[31360]: pgmap v7077: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:25 smithi022 ceph-mon[40250]: pgmap v7077: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:25 smithi122 ceph-mon[32255]: pgmap v7077: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:27 smithi022 ceph-mon[31360]: pgmap v7078: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:54:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:27 smithi022 ceph-mon[40250]: pgmap v7078: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:27 smithi122 ceph-mon[32255]: pgmap v7078: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T21:54:28.406 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:28.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:28.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:28.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:29 smithi022 ceph-mon[31360]: pgmap v7079: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:29.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/331932942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:29 smithi022 ceph-mon[40250]: pgmap v7079: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/331932942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:29 smithi122 ceph-mon[32255]: pgmap v7079: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T21:54:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/331932942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:30] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T21:54:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:31 smithi022 ceph-mon[31360]: pgmap v7080: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:31 smithi022 ceph-mon[40250]: pgmap v7080: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:31 smithi122 ceph-mon[32255]: pgmap v7080: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:54:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:54:33.775 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:33.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[31360]: pgmap v7081: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:33.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:54:33.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:54:33.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:54:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:54:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[40250]: pgmap v7081: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:54:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:54:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:54:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:33 smithi122 ceph-mon[32255]: pgmap v7081: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:54:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:54:34.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:34.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:34.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:34.520 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3841302817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:34.521 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3841302817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3841302817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:35.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:54:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:54:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:35 smithi022 ceph-mon[31360]: pgmap v7082: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:35 smithi022 ceph-mon[40250]: pgmap v7082: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:35 smithi122 ceph-mon[32255]: pgmap v7082: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:37 smithi022 ceph-mon[31360]: pgmap v7083: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:54:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:37 smithi022 ceph-mon[40250]: pgmap v7083: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:37 smithi122 ceph-mon[32255]: pgmap v7083: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:39.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:39 smithi022 ceph-mon[31360]: pgmap v7084: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1272196240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:39 smithi022 ceph-mon[40250]: pgmap v7084: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1272196240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:39 smithi122 ceph-mon[32255]: pgmap v7084: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1272196240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:40] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:54:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:41 smithi122 ceph-mon[32255]: pgmap v7085: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:41 smithi022 ceph-mon[31360]: pgmap v7085: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:41 smithi022 ceph-mon[40250]: pgmap v7085: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:54:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:54:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:43 smithi122 ceph-mon[32255]: pgmap v7086: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:43 smithi022 ceph-mon[31360]: pgmap v7086: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:43 smithi022 ceph-mon[40250]: pgmap v7086: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:44.516 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:45.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:54:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:54:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:45 smithi122 ceph-mon[32255]: pgmap v7087: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/778164259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:45 smithi022 ceph-mon[31360]: pgmap v7087: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/778164259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:45 smithi022 ceph-mon[40250]: pgmap v7087: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/778164259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:47 smithi022 ceph-mon[31360]: pgmap v7088: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:54:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:47 smithi022 ceph-mon[40250]: pgmap v7088: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:47 smithi122 ceph-mon[32255]: pgmap v7088: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:48.549 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:54:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:54:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:54:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:54:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:54:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:54:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:54:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:54:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:54:49.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:49 smithi022 ceph-mon[31360]: pgmap v7089: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:54:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:49 smithi022 ceph-mon[40250]: pgmap v7089: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:54:50.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:49 smithi122 ceph-mon[32255]: pgmap v7089: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:50.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:54:50.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:50.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:50.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:50] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T21:54:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1342773693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:51.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1342773693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:51.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1342773693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:51 smithi022 ceph-mon[31360]: pgmap v7090: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:51 smithi022 ceph-mon[40250]: pgmap v7090: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:51 smithi122 ceph-mon[32255]: pgmap v7090: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:54:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:54:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:53 smithi022 ceph-mon[31360]: pgmap v7091: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:53 smithi022 ceph-mon[40250]: pgmap v7091: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:53 smithi122 ceph-mon[32255]: pgmap v7091: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:55.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:54:55.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:54:55.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:54:55.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:54:55.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:54:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:54:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:55 smithi022 ceph-mon[31360]: pgmap v7092: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3535833868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:55 smithi022 ceph-mon[40250]: pgmap v7092: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3535833868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:55 smithi122 ceph-mon[32255]: pgmap v7092: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:54:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3535833868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:54:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:54:57.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:54:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:57 smithi022 ceph-mon[31360]: pgmap v7093: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:57 smithi022 ceph-mon[40250]: pgmap v7093: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:54:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:57 smithi122 ceph-mon[32255]: pgmap v7093: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:54:59 smithi022 ceph-mon[31360]: pgmap v7094: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:54:59 smithi022 ceph-mon[40250]: pgmap v7094: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:54:59 smithi122 ceph-mon[32255]: pgmap v7094: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:00] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T21:55:00.633 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:01.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:01.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:01.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:01 smithi022 ceph-mon[31360]: pgmap v7095: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/612489255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:01 smithi022 ceph-mon[40250]: pgmap v7095: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/612489255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:01 smithi122 ceph-mon[32255]: pgmap v7095: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/612489255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:55:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:55:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:03 smithi022 ceph-mon[31360]: pgmap v7096: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:03 smithi022 ceph-mon[40250]: pgmap v7096: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:03 smithi122 ceph-mon[32255]: pgmap v7096: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:05.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:55:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:55:06.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:06.008 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:05 smithi022 ceph-mon[40250]: pgmap v7097: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:06.008 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:05 smithi022 ceph-mon[31360]: pgmap v7097: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:05 smithi122 ceph-mon[32255]: pgmap v7097: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:06.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:06.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:06.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/532542236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/532542236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/532542236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:07.760 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:55:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:07 smithi022 ceph-mon[31360]: pgmap v7098: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:07 smithi022 ceph-mon[40250]: pgmap v7098: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:07 smithi122 ceph-mon[32255]: pgmap v7098: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:09 smithi022 ceph-mon[31360]: pgmap v7099: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:09 smithi022 ceph-mon[40250]: pgmap v7099: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:09 smithi122 ceph-mon[32255]: pgmap v7099: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:10] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:55:11.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:11.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:11.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:11.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:11 smithi022 ceph-mon[31360]: pgmap v7100: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3862985382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:11 smithi022 ceph-mon[40250]: pgmap v7100: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3862985382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:11 smithi122 ceph-mon[32255]: pgmap v7100: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3862985382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:55:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:55:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:13 smithi022 ceph-mon[31360]: pgmap v7101: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:13 smithi022 ceph-mon[40250]: pgmap v7101: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:13 smithi122 ceph-mon[32255]: pgmap v7101: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:15.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:55:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:55:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:15 smithi022 ceph-mon[31360]: pgmap v7102: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:15 smithi022 ceph-mon[40250]: pgmap v7102: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:15 smithi122 ceph-mon[32255]: pgmap v7102: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:16.746 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:17.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:17.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:17.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:17.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:55:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:17 smithi022 ceph-mon[31360]: pgmap v7103: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4246843288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:17 smithi022 ceph-mon[40250]: pgmap v7103: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4246843288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:17 smithi122 ceph-mon[32255]: pgmap v7103: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4246843288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:19 smithi022 ceph-mon[31360]: pgmap v7104: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:19 smithi022 ceph-mon[40250]: pgmap v7104: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:19 smithi122 ceph-mon[32255]: pgmap v7104: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:20] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T21:55:22.121 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:22.122 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:21 smithi022 ceph-mon[40250]: pgmap v7105: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:22.122 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:21 smithi022 ceph-mon[31360]: pgmap v7105: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:21 smithi122 ceph-mon[32255]: pgmap v7105: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:22.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:22.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:55:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:55:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3302346560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3302346560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3302346560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:23 smithi122 ceph-mon[32255]: pgmap v7106: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:23 smithi022 ceph-mon[31360]: pgmap v7106: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:23 smithi022 ceph-mon[40250]: pgmap v7106: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:25.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:55:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:55:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:25 smithi122 ceph-mon[32255]: pgmap v7107: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:25 smithi022 ceph-mon[31360]: pgmap v7107: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:25 smithi022 ceph-mon[40250]: pgmap v7107: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:27.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:55:27.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:27.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:27.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:27 smithi122 ceph-mon[32255]: pgmap v7108: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:27 smithi022 ceph-mon[31360]: pgmap v7108: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:27 smithi022 ceph-mon[40250]: pgmap v7108: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2073424609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2073424609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2073424609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:29 smithi122 ceph-mon[32255]: pgmap v7109: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:29 smithi022 ceph-mon[31360]: pgmap v7109: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:29 smithi022 ceph-mon[40250]: pgmap v7109: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:30] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T21:55:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:31 smithi122 ceph-mon[32255]: pgmap v7110: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:31 smithi022 ceph-mon[31360]: pgmap v7110: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:31 smithi022 ceph-mon[40250]: pgmap v7110: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:55:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:55:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:33.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:33.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:33.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:33 smithi122 ceph-mon[32255]: pgmap v7111: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:55:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:55:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:55:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:55:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/146570191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[31360]: pgmap v7111: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:55:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:55:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:55:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:55:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/146570191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[40250]: pgmap v7111: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:55:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:55:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:55:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:55:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/146570191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:35.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:55:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:55:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:35 smithi122 ceph-mon[32255]: pgmap v7112: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:35 smithi022 ceph-mon[31360]: pgmap v7112: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:35 smithi022 ceph-mon[40250]: pgmap v7112: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:55:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:37 smithi122 ceph-mon[32255]: pgmap v7113: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:38.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:38.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:37 smithi022 ceph-mon[31360]: pgmap v7113: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:38.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:37 smithi022 ceph-mon[40250]: pgmap v7113: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:38.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:38.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:38.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:39 smithi022 ceph-mon[31360]: pgmap v7114: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2543919062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:39 smithi022 ceph-mon[40250]: pgmap v7114: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2543919062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:39 smithi122 ceph-mon[32255]: pgmap v7114: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2543919062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:40] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T21:55:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:41 smithi022 ceph-mon[31360]: pgmap v7115: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:41 smithi022 ceph-mon[40250]: pgmap v7115: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:41 smithi122 ceph-mon[32255]: pgmap v7115: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:55:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:55:43.613 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:43 smithi022 ceph-mon[31360]: pgmap v7116: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:43 smithi022 ceph-mon[40250]: pgmap v7116: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:43 smithi122 ceph-mon[32255]: pgmap v7116: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T21:55:43.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/292939041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:44.829 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/292939041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/292939041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:45 smithi122 ceph-mon[32255]: pgmap v7117: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:45.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:55:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:55:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:45 smithi022 ceph-mon[31360]: pgmap v7117: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:45 smithi022 ceph-mon[40250]: pgmap v7117: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:55:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:47 smithi022 ceph-mon[40250]: pgmap v7118: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:47 smithi022 ceph-mon[31360]: pgmap v7118: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:47 smithi122 ceph-mon[32255]: pgmap v7118: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:48.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:49.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:49.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:49.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[31360]: pgmap v7119: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:55:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:55:49.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:55:49.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:55:49.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/980354195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[40250]: pgmap v7119: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:55:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:55:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:55:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:55:49.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/980354195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:49 smithi122 ceph-mon[32255]: pgmap v7119: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:55:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:55:49.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:55:49.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:55:49.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/980354195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:50] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:55:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:51 smithi022 ceph-mon[31360]: pgmap v7120: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:51 smithi022 ceph-mon[40250]: pgmap v7120: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:51 smithi122 ceph-mon[32255]: pgmap v7120: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:55:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:55:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:53 smithi022 ceph-mon[31360]: pgmap v7121: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:53 smithi022 ceph-mon[40250]: pgmap v7121: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:53 smithi122 ceph-mon[32255]: pgmap v7121: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:54.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:54.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:55:54.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:55:54.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:55:55.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:55:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:55:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:55 smithi122 ceph-mon[32255]: pgmap v7122: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2855946309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:55 smithi022 ceph-mon[31360]: pgmap v7122: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2855946309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:55 smithi022 ceph-mon[40250]: pgmap v7122: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2855946309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:55:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:55:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:57 smithi022 ceph-mon[31360]: pgmap v7123: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:55:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:57 smithi022 ceph-mon[40250]: pgmap v7123: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:57 smithi122 ceph-mon[32255]: pgmap v7123: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:55:59.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:55:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:55:59 smithi022 ceph-mon[31360]: pgmap v7124: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:55:59 smithi022 ceph-mon[40250]: pgmap v7124: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:55:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:55:59 smithi122 ceph-mon[32255]: pgmap v7124: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:00.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:00.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:00.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:00] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T21:56:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3932585624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3932585624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3932585624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:01 smithi022 ceph-mon[31360]: pgmap v7125: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:01 smithi022 ceph-mon[40250]: pgmap v7125: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:01 smithi122 ceph-mon[32255]: pgmap v7125: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:56:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:56:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:03 smithi022 ceph-mon[31360]: pgmap v7126: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:03 smithi022 ceph-mon[40250]: pgmap v7126: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:03 smithi122 ceph-mon[32255]: pgmap v7126: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:05.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:05.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:05.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:05.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:56:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:56:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:05 smithi122 ceph-mon[32255]: pgmap v7127: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:05.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/400881284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:05 smithi022 ceph-mon[31360]: pgmap v7127: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/400881284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:05 smithi022 ceph-mon[40250]: pgmap v7127: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/400881284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:56:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:07 smithi022 ceph-mon[31360]: pgmap v7128: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:07 smithi022 ceph-mon[40250]: pgmap v7128: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:07 smithi122 ceph-mon[32255]: pgmap v7128: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:09 smithi022 ceph-mon[31360]: pgmap v7129: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:09 smithi022 ceph-mon[40250]: pgmap v7129: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:09 smithi122 ceph-mon[32255]: pgmap v7129: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:10] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:56:10.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:10.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:10.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:10.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:11 smithi022 ceph-mon[31360]: pgmap v7130: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3790240863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:11 smithi022 ceph-mon[40250]: pgmap v7130: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3790240863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:11 smithi122 ceph-mon[32255]: pgmap v7130: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3790240863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:56:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:56:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:13 smithi122 ceph-mon[32255]: pgmap v7131: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:13 smithi022 ceph-mon[31360]: pgmap v7131: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:13 smithi022 ceph-mon[40250]: pgmap v7131: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:15 smithi122 ceph-mon[32255]: pgmap v7132: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:15.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:56:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:56:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:15.835 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:15 smithi022 ceph-mon[40250]: pgmap v7132: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:15.836 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:15 smithi022 ceph-mon[31360]: pgmap v7132: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:16.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:16.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:16.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2072037825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2072037825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2072037825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:17 smithi022 ceph-mon[31360]: pgmap v7133: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:56:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:17 smithi022 ceph-mon[40250]: pgmap v7133: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:17 smithi122 ceph-mon[32255]: pgmap v7133: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:19 smithi122 ceph-mon[32255]: pgmap v7134: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:19 smithi022 ceph-mon[31360]: pgmap v7134: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:19 smithi022 ceph-mon[40250]: pgmap v7134: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:20] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:56:21.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:21.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:21.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:21.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:21 smithi122 ceph-mon[32255]: pgmap v7135: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:21 smithi022 ceph-mon[31360]: pgmap v7135: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:21 smithi022 ceph-mon[40250]: pgmap v7135: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1613726474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:56:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:56:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1613726474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1613726474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:23 smithi122 ceph-mon[32255]: pgmap v7136: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:23 smithi022 ceph-mon[31360]: pgmap v7136: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:23 smithi022 ceph-mon[40250]: pgmap v7136: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:25.555 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:56:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:56:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:25 smithi122 ceph-mon[32255]: pgmap v7137: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:25 smithi022 ceph-mon[31360]: pgmap v7137: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:25 smithi022 ceph-mon[40250]: pgmap v7137: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:26.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:26.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:26.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:26.943 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:26.944 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:27 smithi022 ceph-mon[31360]: pgmap v7138: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/590772033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:56:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:27 smithi022 ceph-mon[40250]: pgmap v7138: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/590772033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:27 smithi122 ceph-mon[32255]: pgmap v7138: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/590772033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:29 smithi122 ceph-mon[32255]: pgmap v7139: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:29 smithi022 ceph-mon[31360]: pgmap v7139: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:29 smithi022 ceph-mon[40250]: pgmap v7139: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:30] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:56:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:31 smithi122 ceph-mon[32255]: pgmap v7140: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:31.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:31.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:31 smithi022 ceph-mon[31360]: pgmap v7140: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:31.945 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:31 smithi022 ceph-mon[40250]: pgmap v7140: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:32.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:32.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:32.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:32.571 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1155380996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:32.572 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:56:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:56:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1155380996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1155380996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:33 smithi122 ceph-mon[32255]: pgmap v7141: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:56:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:56:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:56:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[31360]: pgmap v7141: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:56:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:56:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:56:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:56:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[40250]: pgmap v7141: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:56:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:56:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:56:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:56:35.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:56:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:56:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:35 smithi122 ceph-mon[32255]: pgmap v7142: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:35 smithi022 ceph-mon[31360]: pgmap v7142: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:35 smithi022 ceph-mon[40250]: pgmap v7142: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:37.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:37.580 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:56:37.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:37.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:37.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:37 smithi122 ceph-mon[32255]: pgmap v7143: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:37 smithi022 ceph-mon[31360]: pgmap v7143: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:37 smithi022 ceph-mon[40250]: pgmap v7143: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2933062219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2933062219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2933062219' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:39 smithi122 ceph-mon[32255]: pgmap v7144: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:39 smithi022 ceph-mon[31360]: pgmap v7144: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:39 smithi022 ceph-mon[40250]: pgmap v7144: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:40] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T21:56:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:41 smithi122 ceph-mon[32255]: pgmap v7145: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:41 smithi022 ceph-mon[31360]: pgmap v7145: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:41 smithi022 ceph-mon[40250]: pgmap v7145: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:56:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:56:42.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:43 smithi122 ceph-mon[32255]: pgmap v7146: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1396152694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:43 smithi022 ceph-mon[40250]: pgmap v7146: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1396152694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:44.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:43 smithi022 ceph-mon[31360]: pgmap v7146: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:44.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1396152694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:45.594 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:56:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:56:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:45 smithi122 ceph-mon[32255]: pgmap v7147: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:45 smithi022 ceph-mon[31360]: pgmap v7147: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:45 smithi022 ceph-mon[40250]: pgmap v7147: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:47 smithi022 ceph-mon[31360]: pgmap v7148: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:56:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:47 smithi022 ceph-mon[40250]: pgmap v7148: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:47 smithi122 ceph-mon[32255]: pgmap v7148: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:48.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:48.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:48.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:48.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/30488860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:48.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/30488860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/30488860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:49 smithi122 ceph-mon[32255]: pgmap v7149: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:56:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:56:49.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:56:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[31360]: pgmap v7149: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:56:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:56:50.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:56:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[40250]: pgmap v7149: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:56:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:56:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:56:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:50] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:56:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:56:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:56:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:56:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:51 smithi122 ceph-mon[32255]: pgmap v7150: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:51 smithi022 ceph-mon[31360]: pgmap v7150: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:51 smithi022 ceph-mon[40250]: pgmap v7150: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:56:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:56:53.434 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:53 smithi022 ceph-mon[31360]: pgmap v7151: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:53 smithi022 ceph-mon[40250]: pgmap v7151: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:53.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:53.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:53.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:53 smithi122 ceph-mon[32255]: pgmap v7151: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3681033478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:55.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3681033478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3681033478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:56:55.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:56:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:56:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:55 smithi122 ceph-mon[32255]: pgmap v7152: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:55 smithi022 ceph-mon[31360]: pgmap v7152: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:55 smithi022 ceph-mon[40250]: pgmap v7152: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:56:57.678 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:56:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:57 smithi122 ceph-mon[32255]: pgmap v7153: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:57 smithi022 ceph-mon[31360]: pgmap v7153: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:57 smithi022 ceph-mon[40250]: pgmap v7153: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:56:58.806 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:56:59.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:56:59.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:56:59.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:56:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:59 smithi122 ceph-mon[32255]: pgmap v7154: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:56:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:56:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1263976386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:59 smithi022 ceph-mon[31360]: pgmap v7154: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:56:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1263976386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:59 smithi022 ceph-mon[40250]: pgmap v7154: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:56:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1263976386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:00] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T21:57:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:01 smithi122 ceph-mon[32255]: pgmap v7155: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:01 smithi022 ceph-mon[31360]: pgmap v7155: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:01 smithi022 ceph-mon[40250]: pgmap v7155: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:57:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:57:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:03 smithi022 ceph-mon[31360]: pgmap v7156: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:03 smithi022 ceph-mon[40250]: pgmap v7156: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:04.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:03 smithi122 ceph-mon[32255]: pgmap v7156: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:04.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:04.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:04.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4201075849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4201075849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4201075849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:05.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:57:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:57:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:05 smithi022 ceph-mon[31360]: pgmap v7157: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:05 smithi022 ceph-mon[40250]: pgmap v7157: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:05 smithi122 ceph-mon[32255]: pgmap v7157: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:07.698 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:57:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:07 smithi022 ceph-mon[31360]: pgmap v7158: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:07 smithi022 ceph-mon[40250]: pgmap v7158: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:07 smithi122 ceph-mon[32255]: pgmap v7158: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:09.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:09.916 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:09 smithi022 ceph-mon[31360]: pgmap v7159: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:09.916 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:09 smithi022 ceph-mon[40250]: pgmap v7159: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:09 smithi122 ceph-mon[32255]: pgmap v7159: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:10] "GET /metrics HTTP/1.1" 200 202374 "" "Prometheus/2.18.1" 2021-12-15T21:57:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3289088514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3289088514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3289088514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:11 smithi022 ceph-mon[31360]: pgmap v7160: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:11 smithi022 ceph-mon[40250]: pgmap v7160: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:11 smithi122 ceph-mon[32255]: pgmap v7160: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:57:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:57:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:13 smithi022 ceph-mon[31360]: pgmap v7161: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:13 smithi022 ceph-mon[40250]: pgmap v7161: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:13 smithi122 ceph-mon[32255]: pgmap v7161: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:14.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:15.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:57:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:57:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:15 smithi022 ceph-mon[31360]: pgmap v7162: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1352376330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:15 smithi022 ceph-mon[40250]: pgmap v7162: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1352376330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:15 smithi122 ceph-mon[32255]: pgmap v7162: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1352376330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:17.717 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:57:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:17 smithi022 ceph-mon[31360]: pgmap v7163: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:17 smithi022 ceph-mon[40250]: pgmap v7163: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:17 smithi122 ceph-mon[32255]: pgmap v7163: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:19 smithi022 ceph-mon[40250]: pgmap v7164: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:20.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:19 smithi022 ceph-mon[31360]: pgmap v7164: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:19 smithi122 ceph-mon[32255]: pgmap v7164: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:20] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:57:20.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:20.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:20.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2166203069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2166203069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2166203069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:21 smithi022 ceph-mon[31360]: pgmap v7165: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:22.094 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:21 smithi022 ceph-mon[40250]: pgmap v7165: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:21 smithi122 ceph-mon[32255]: pgmap v7165: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:57:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:57:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:23 smithi022 ceph-mon[31360]: pgmap v7166: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:23 smithi022 ceph-mon[40250]: pgmap v7166: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:23 smithi122 ceph-mon[32255]: pgmap v7166: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:25.658 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:25.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:57:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:57:26.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:26.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:26.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:25 smithi022 ceph-mon[31360]: pgmap v7167: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:25 smithi022 ceph-mon[40250]: pgmap v7167: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:25 smithi122 ceph-mon[32255]: pgmap v7167: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1989576290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1989576290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1989576290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:27.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:57:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:27 smithi022 ceph-mon[31360]: pgmap v7168: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:27 smithi022 ceph-mon[40250]: pgmap v7168: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:27 smithi122 ceph-mon[32255]: pgmap v7168: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:29 smithi022 ceph-mon[31360]: pgmap v7169: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:29 smithi022 ceph-mon[40250]: pgmap v7169: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:29 smithi122 ceph-mon[32255]: pgmap v7169: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:30] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:57:31.029 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:31.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:31.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:31.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:31 smithi022 ceph-mon[31360]: pgmap v7170: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2729049877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:31 smithi022 ceph-mon[40250]: pgmap v7170: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2729049877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:31 smithi122 ceph-mon[32255]: pgmap v7170: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2729049877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:57:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[31360]: pgmap v7171: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:57:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:57:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:57:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:57:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[40250]: pgmap v7171: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:57:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:57:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:57:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:57:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:33 smithi122 ceph-mon[32255]: pgmap v7171: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:57:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:57:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:57:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:57:35.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:57:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:57:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:35 smithi022 ceph-mon[31360]: pgmap v7172: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:35 smithi022 ceph-mon[40250]: pgmap v7172: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:35 smithi122 ceph-mon[32255]: pgmap v7172: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:36.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:36.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:36.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3121133226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:36.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:36.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:36.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3121133226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3121133226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:37.760 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:57:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:37 smithi022 ceph-mon[31360]: pgmap v7173: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:37 smithi022 ceph-mon[40250]: pgmap v7173: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:37 smithi122 ceph-mon[32255]: pgmap v7173: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:39 smithi022 ceph-mon[31360]: pgmap v7174: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:39 smithi022 ceph-mon[40250]: pgmap v7174: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:39 smithi122 ceph-mon[32255]: pgmap v7174: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:40] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T21:57:41.768 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:41 smithi022 ceph-mon[31360]: pgmap v7175: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:41 smithi022 ceph-mon[40250]: pgmap v7175: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:42.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:42.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:42.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:41 smithi122 ceph-mon[32255]: pgmap v7175: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:57:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:57:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3905905994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3905905994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3905905994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:43 smithi022 ceph-mon[31360]: pgmap v7176: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:43 smithi022 ceph-mon[40250]: pgmap v7176: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:43 smithi122 ceph-mon[32255]: pgmap v7176: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:45.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:57:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:57:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:45 smithi022 ceph-mon[31360]: pgmap v7177: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:45 smithi022 ceph-mon[40250]: pgmap v7177: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:45 smithi122 ceph-mon[32255]: pgmap v7177: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:47.139 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:47.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:47.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:47.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:57:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:47 smithi122 ceph-mon[32255]: pgmap v7178: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1836683053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:47 smithi022 ceph-mon[31360]: pgmap v7178: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1836683053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:47 smithi022 ceph-mon[40250]: pgmap v7178: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1836683053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[31360]: pgmap v7179: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:57:50.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:57:50.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:57:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[40250]: pgmap v7179: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:57:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:57:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:57:50.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:49 smithi122 ceph-mon[32255]: pgmap v7179: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:57:50.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:57:50.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:57:50.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:57:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:50] "GET /metrics HTTP/1.1" 200 202374 "" "Prometheus/2.18.1" 2021-12-15T21:57:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:57:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:51 smithi122 ceph-mon[32255]: pgmap v7180: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:57:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:51 smithi022 ceph-mon[31360]: pgmap v7180: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:57:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:51 smithi022 ceph-mon[40250]: pgmap v7180: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:52.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:57:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:57:52.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:53 smithi022 ceph-mon[31360]: pgmap v7181: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4292172675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:53 smithi022 ceph-mon[40250]: pgmap v7181: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4292172675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:53 smithi122 ceph-mon[32255]: pgmap v7181: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:57:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4292172675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:55.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:55 smithi122 ceph-mon[32255]: pgmap v7182: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:55.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:57:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:57:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:55 smithi022 ceph-mon[31360]: pgmap v7182: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:55.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:55 smithi022 ceph-mon[40250]: pgmap v7182: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:57:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:57 smithi022 ceph-mon[31360]: pgmap v7183: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:57:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:57 smithi022 ceph-mon[40250]: pgmap v7183: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:57.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:57:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:57 smithi122 ceph-mon[32255]: pgmap v7183: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:58.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:57:58.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:57:58.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:57:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1267194246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1267194246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1267194246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:57:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:57:59 smithi022 ceph-mon[31360]: pgmap v7184: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:57:59 smithi022 ceph-mon[40250]: pgmap v7184: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:57:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:57:59 smithi122 ceph-mon[32255]: pgmap v7184: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:58:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:00] "GET /metrics HTTP/1.1" 200 202374 "" "Prometheus/2.18.1" 2021-12-15T21:58:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:01 smithi022 ceph-mon[31360]: pgmap v7185: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:58:01.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:01 smithi022 ceph-mon[40250]: pgmap v7185: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:58:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:01 smithi122 ceph-mon[32255]: pgmap v7185: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:58:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:58:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:58:03.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:03.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:03 smithi022 ceph-mon[40250]: pgmap v7186: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:58:03.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:03 smithi022 ceph-mon[31360]: pgmap v7186: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:58:03.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:03.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:03.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:03 smithi122 ceph-mon[32255]: pgmap v7186: 132 pgs: 132 active+clean; 582 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T21:58:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2806728340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2806728340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2806728340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:05.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:58:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:58:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:05 smithi122 ceph-mon[32255]: pgmap v7187: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:58:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:05 smithi022 ceph-mon[31360]: pgmap v7187: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:58:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:05 smithi022 ceph-mon[40250]: pgmap v7187: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T21:58:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:07 smithi022 ceph-mon[31360]: pgmap v7188: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:58:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:07 smithi022 ceph-mon[40250]: pgmap v7188: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:07 smithi122 ceph-mon[32255]: pgmap v7188: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:08.633 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:09.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:09.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:09.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:09 smithi122 ceph-mon[32255]: pgmap v7189: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:58:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3630078825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:09 smithi022 ceph-mon[31360]: pgmap v7189: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:58:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3630078825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:09 smithi022 ceph-mon[40250]: pgmap v7189: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:58:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3630078825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:10] "GET /metrics HTTP/1.1" 200 202353 "" "Prometheus/2.18.1" 2021-12-15T21:58:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:11 smithi122 ceph-mon[32255]: pgmap v7190: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:11 smithi022 ceph-mon[31360]: pgmap v7190: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:11 smithi022 ceph-mon[40250]: pgmap v7190: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:58:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:58:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:13 smithi122 ceph-mon[32255]: pgmap v7191: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:14.004 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:13 smithi022 ceph-mon[31360]: pgmap v7191: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:14.005 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:13 smithi022 ceph-mon[40250]: pgmap v7191: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T21:58:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:14.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:14.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:14.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3591117697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3591117697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3591117697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:15.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:58:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:58:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:15 smithi122 ceph-mon[32255]: pgmap v7192: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:58:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:15 smithi022 ceph-mon[31360]: pgmap v7192: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:58:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:15 smithi022 ceph-mon[40250]: pgmap v7192: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T21:58:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:58:17.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:17 smithi022 ceph-mon[31360]: pgmap v7193: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:17 smithi022 ceph-mon[40250]: pgmap v7193: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:17 smithi122 ceph-mon[32255]: pgmap v7193: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:19.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:19.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:19.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:19.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:19.748 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:19 smithi022 ceph-mon[31360]: pgmap v7194: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:19.748 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:19 smithi022 ceph-mon[40250]: pgmap v7194: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:19 smithi122 ceph-mon[32255]: pgmap v7194: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:20] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:58:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1067559239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1067559239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1067559239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:21 smithi122 ceph-mon[32255]: pgmap v7195: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:21 smithi022 ceph-mon[31360]: pgmap v7195: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:21 smithi022 ceph-mon[40250]: pgmap v7195: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:58:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:58:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:23 smithi122 ceph-mon[32255]: pgmap v7196: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:23 smithi022 ceph-mon[31360]: pgmap v7196: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:23 smithi022 ceph-mon[40250]: pgmap v7196: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:24.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:25.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:25.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:25.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:25.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:58:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:58:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:25 smithi122 ceph-mon[32255]: pgmap v7197: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:25.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1092951372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:25 smithi022 ceph-mon[31360]: pgmap v7197: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1092951372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:25 smithi022 ceph-mon[40250]: pgmap v7197: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1092951372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:27 smithi022 ceph-mon[31360]: pgmap v7198: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:58:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:27 smithi022 ceph-mon[40250]: pgmap v7198: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:27 smithi122 ceph-mon[32255]: pgmap v7198: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:29 smithi122 ceph-mon[32255]: pgmap v7199: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:29 smithi022 ceph-mon[31360]: pgmap v7199: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:29 smithi022 ceph-mon[40250]: pgmap v7199: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:30.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:30] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:58:30.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:30.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:30.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1462851922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1462851922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1462851922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:31 smithi122 ceph-mon[32255]: pgmap v7200: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:31 smithi022 ceph-mon[31360]: pgmap v7200: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:31 smithi022 ceph-mon[40250]: pgmap v7200: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:58:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:58:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:33 smithi122 ceph-mon[32255]: pgmap v7201: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:58:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:58:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:58:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[31360]: pgmap v7201: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:58:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:58:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[40250]: pgmap v7201: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:58:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:58:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:58:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:58:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:35 smithi122 ceph-mon[32255]: pgmap v7202: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:35.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:58:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:58:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:35 smithi022 ceph-mon[31360]: pgmap v7202: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:35 smithi022 ceph-mon[40250]: pgmap v7202: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:35.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:35.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:35.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3220722017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3220722017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3220722017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:37 smithi022 ceph-mon[31360]: pgmap v7203: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:58:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:37 smithi022 ceph-mon[40250]: pgmap v7203: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:37 smithi122 ceph-mon[32255]: pgmap v7203: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:39 smithi122 ceph-mon[32255]: pgmap v7204: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:39 smithi022 ceph-mon[31360]: pgmap v7204: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:39 smithi022 ceph-mon[40250]: pgmap v7204: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:40] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T21:58:40.856 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:41.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:41.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:41.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:41 smithi122 ceph-mon[32255]: pgmap v7205: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2862695059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:41 smithi022 ceph-mon[31360]: pgmap v7205: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2862695059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:41 smithi022 ceph-mon[40250]: pgmap v7205: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2862695059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:58:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:58:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:43 smithi122 ceph-mon[32255]: pgmap v7206: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:43 smithi022 ceph-mon[31360]: pgmap v7206: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:43 smithi022 ceph-mon[40250]: pgmap v7206: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:45.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:58:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:58:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:45 smithi122 ceph-mon[32255]: pgmap v7207: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:45 smithi022 ceph-mon[31360]: pgmap v7207: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:45 smithi022 ceph-mon[40250]: pgmap v7207: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:46.226 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:46.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:46.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2209440781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2209440781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2209440781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:47 smithi022 ceph-mon[31360]: pgmap v7208: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:58:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:47 smithi022 ceph-mon[40250]: pgmap v7208: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:47 smithi122 ceph-mon[32255]: pgmap v7208: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:49 smithi122 ceph-mon[32255]: pgmap v7209: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:49 smithi022 ceph-mon[31360]: pgmap v7209: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:49 smithi022 ceph-mon[40250]: pgmap v7209: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:50] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:58:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:58:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:58:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:58:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:50.987 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:58:50.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:58:50.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:58:50.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:50.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:50.989 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:58:50.989 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:58:50.989 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:58:50.989 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:50.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:51.597 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:51.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:51.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:51.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:51 smithi022 ceph-mon[31360]: pgmap v7210: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2303350313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:51 smithi022 ceph-mon[40250]: pgmap v7210: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2303350313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:51 smithi122 ceph-mon[32255]: pgmap v7210: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:58:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2303350313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:58:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:58:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:53 smithi022 ceph-mon[31360]: pgmap v7211: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:53 smithi022 ceph-mon[40250]: pgmap v7211: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:53 smithi122 ceph-mon[32255]: pgmap v7211: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:55.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:58:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:58:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:55 smithi022 ceph-mon[31360]: pgmap v7212: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:55 smithi022 ceph-mon[40250]: pgmap v7212: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:55 smithi122 ceph-mon[32255]: pgmap v7212: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:56.964 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:58:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:57.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:58:57.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:58:57.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:58:57.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:58:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:58:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:58 smithi022 ceph-mon[31360]: pgmap v7213: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1434005176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:58 smithi022 ceph-mon[40250]: pgmap v7213: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1434005176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:58 smithi122 ceph-mon[32255]: pgmap v7213: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:58:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1434005176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:58:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:58:59 smithi022 ceph-mon[31360]: pgmap v7214: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:58:59 smithi022 ceph-mon[40250]: pgmap v7214: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:58:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:58:59 smithi122 ceph-mon[32255]: pgmap v7214: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:00] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:59:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:01 smithi022 ceph-mon[31360]: pgmap v7215: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:01 smithi022 ceph-mon[40250]: pgmap v7215: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:01 smithi122 ceph-mon[32255]: pgmap v7215: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:02.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:59:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:59:02.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:02.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:02.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:03 smithi022 ceph-mon[31360]: pgmap v7216: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3056138204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:03 smithi022 ceph-mon[40250]: pgmap v7216: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3056138204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:03 smithi122 ceph-mon[32255]: pgmap v7216: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3056138204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:05 smithi122 ceph-mon[32255]: pgmap v7217: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:05.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:59:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:59:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:05 smithi022 ceph-mon[31360]: pgmap v7217: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:05 smithi022 ceph-mon[40250]: pgmap v7217: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:07.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:07.709 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:59:07.710 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:07 smithi022 ceph-mon[31360]: pgmap v7218: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:07.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:07 smithi022 ceph-mon[40250]: pgmap v7218: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:07 smithi122 ceph-mon[32255]: pgmap v7218: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:08.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:08.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:08.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/727421166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:09.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/727421166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/727421166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:09 smithi122 ceph-mon[32255]: pgmap v7219: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:09 smithi022 ceph-mon[31360]: pgmap v7219: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:09 smithi022 ceph-mon[40250]: pgmap v7219: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:10] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:59:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:11 smithi122 ceph-mon[32255]: pgmap v7220: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:11 smithi022 ceph-mon[31360]: pgmap v7220: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:11 smithi022 ceph-mon[40250]: pgmap v7220: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:59:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:59:13.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:13.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:13.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:13.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:13 smithi022 ceph-mon[31360]: pgmap v7221: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/261831962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:13 smithi022 ceph-mon[40250]: pgmap v7221: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/261831962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:13 smithi122 ceph-mon[32255]: pgmap v7221: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/261831962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:15.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:59:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:59:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:15 smithi122 ceph-mon[32255]: pgmap v7222: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:15 smithi022 ceph-mon[31360]: pgmap v7222: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:15 smithi022 ceph-mon[40250]: pgmap v7222: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:17 smithi022 ceph-mon[31360]: pgmap v7223: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:59:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:17 smithi022 ceph-mon[40250]: pgmap v7223: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:17 smithi122 ceph-mon[32255]: pgmap v7223: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:18.453 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:18.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:18.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:18.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:19 smithi122 ceph-mon[32255]: pgmap v7224: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1415392823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:19 smithi022 ceph-mon[31360]: pgmap v7224: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1415392823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:19 smithi022 ceph-mon[40250]: pgmap v7224: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1415392823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:20] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:59:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:21 smithi122 ceph-mon[32255]: pgmap v7225: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:21 smithi022 ceph-mon[31360]: pgmap v7225: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:21 smithi022 ceph-mon[40250]: pgmap v7225: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:59:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:59:23.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:23.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:23 smithi022 ceph-mon[31360]: pgmap v7226: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:23.823 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:23 smithi022 ceph-mon[40250]: pgmap v7226: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:23 smithi122 ceph-mon[32255]: pgmap v7226: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:24.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:24.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:24.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2819609074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2819609074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2819609074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:25.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:59:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:59:25.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:25 smithi122 ceph-mon[32255]: pgmap v7227: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:25 smithi022 ceph-mon[31360]: pgmap v7227: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:25 smithi022 ceph-mon[40250]: pgmap v7227: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:27 smithi022 ceph-mon[31360]: pgmap v7228: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:59:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:27 smithi022 ceph-mon[40250]: pgmap v7228: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:27 smithi122 ceph-mon[32255]: pgmap v7228: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:29.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:29.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:29.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:29.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:29 smithi122 ceph-mon[32255]: pgmap v7229: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2611135697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:29 smithi022 ceph-mon[31360]: pgmap v7229: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2611135697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:29 smithi022 ceph-mon[40250]: pgmap v7229: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2611135697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:30] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T21:59:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:31 smithi122 ceph-mon[32255]: pgmap v7230: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:31 smithi022 ceph-mon[31360]: pgmap v7230: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:31 smithi022 ceph-mon[40250]: pgmap v7230: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:59:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:59:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:33 smithi122 ceph-mon[32255]: pgmap v7231: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:59:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:59:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:59:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:59:34.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[31360]: pgmap v7231: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:34.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:59:34.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:59:34.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:59:34.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:59:34.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[40250]: pgmap v7231: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:34.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:59:34.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:59:34.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T21:59:34.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T21:59:34.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:34.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:34.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:34.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:35.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:59:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:59:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:35 smithi122 ceph-mon[32255]: pgmap v7232: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:35.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1435008762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:35 smithi022 ceph-mon[31360]: pgmap v7232: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1435008762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:35 smithi022 ceph-mon[40250]: pgmap v7232: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1435008762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:37 smithi022 ceph-mon[31360]: pgmap v7233: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:59:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:37 smithi022 ceph-mon[40250]: pgmap v7233: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:37 smithi122 ceph-mon[32255]: pgmap v7233: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:39 smithi122 ceph-mon[32255]: pgmap v7234: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:39.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:39.946 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:39 smithi022 ceph-mon[31360]: pgmap v7234: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:39.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:39 smithi022 ceph-mon[40250]: pgmap v7234: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:40.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:40.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:40.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:40] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T21:59:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1349681084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1349681084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1349681084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:41 smithi122 ceph-mon[32255]: pgmap v7235: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:41 smithi022 ceph-mon[31360]: pgmap v7235: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:41 smithi022 ceph-mon[40250]: pgmap v7235: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:59:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:59:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:43 smithi122 ceph-mon[32255]: pgmap v7236: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:43 smithi022 ceph-mon[40250]: pgmap v7236: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:43 smithi022 ceph-mon[31360]: pgmap v7236: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:45.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:45.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:45.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:45.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:45 smithi122 ceph-mon[32255]: pgmap v7237: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:59:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:59:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:45 smithi022 ceph-mon[31360]: pgmap v7237: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:45 smithi022 ceph-mon[40250]: pgmap v7237: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4186222611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4186222611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4186222611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:47 smithi022 ceph-mon[31360]: pgmap v7238: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:59:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:47 smithi022 ceph-mon[40250]: pgmap v7238: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:47 smithi122 ceph-mon[32255]: pgmap v7238: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:49 smithi122 ceph-mon[32255]: pgmap v7239: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:49 smithi022 ceph-mon[31360]: pgmap v7239: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:49 smithi022 ceph-mon[40250]: pgmap v7239: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 21:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:50] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T21:59:50.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:51.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:51.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:51.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:51 smithi122 ceph-mon[32255]: pgmap v7240: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:59:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:59:51.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:59:51.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3257740809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:51.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:59:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[31360]: pgmap v7240: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:59:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:59:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:59:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3257740809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:59:51.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[40250]: pgmap v7240: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:51.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T21:59:51.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T21:59:51.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T21:59:51.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3257740809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:51.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T21:59:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 21:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T21:59:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T21:59:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:53 smithi122 ceph-mon[32255]: pgmap v7241: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:53 smithi022 ceph-mon[31360]: pgmap v7241: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:53 smithi022 ceph-mon[40250]: pgmap v7241: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 21:59:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T21:59:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:55 smithi122 ceph-mon[32255]: pgmap v7242: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:55 smithi022 ceph-mon[31360]: pgmap v7242: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:55 smithi022 ceph-mon[40250]: pgmap v7242: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T21:59:56.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T21:59:56.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T21:59:56.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T21:59:56.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T21:59:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4144293681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4144293681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T21:59:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4144293681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T21:59:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:57 smithi022 ceph-mon[31360]: pgmap v7243: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 21:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:21:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T21:59:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:57 smithi022 ceph-mon[40250]: pgmap v7243: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:57 smithi122 ceph-mon[32255]: pgmap v7243: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T21:59:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 21:59:59 smithi122 ceph-mon[32255]: pgmap v7244: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 21:59:59 smithi022 ceph-mon[31360]: pgmap v7244: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 21:59:59 smithi022 ceph-mon[40250]: pgmap v7244: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:00] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T22:00:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T22:00:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T22:00:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T22:00:01.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:01 smithi022 ceph-mon[31360]: pgmap v7245: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:01 smithi022 ceph-mon[40250]: pgmap v7245: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:01 smithi122 ceph-mon[32255]: pgmap v7245: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:02.638 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:00:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:00:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/635161214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/635161214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/635161214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:03 smithi122 ceph-mon[32255]: pgmap v7246: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:03 smithi022 ceph-mon[31360]: pgmap v7246: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:03 smithi022 ceph-mon[40250]: pgmap v7246: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:05.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:00:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:00:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:05 smithi122 ceph-mon[32255]: pgmap v7247: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:05 smithi022 ceph-mon[31360]: pgmap v7247: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:05 smithi022 ceph-mon[40250]: pgmap v7247: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:06.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:07.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:07.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:07.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:00:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:07 smithi122 ceph-mon[32255]: pgmap v7248: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2729048293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:07 smithi022 ceph-mon[31360]: pgmap v7248: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2729048293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:07 smithi022 ceph-mon[40250]: pgmap v7248: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:08.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2729048293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:09 smithi122 ceph-mon[32255]: pgmap v7249: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:09 smithi022 ceph-mon[31360]: pgmap v7249: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:09 smithi022 ceph-mon[40250]: pgmap v7249: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:10] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T22:00:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:11 smithi122 ceph-mon[32255]: pgmap v7250: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:11 smithi022 ceph-mon[31360]: pgmap v7250: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:11 smithi022 ceph-mon[40250]: pgmap v7250: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:12.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:12.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:12.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:12.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:12.660 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:00:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:00:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2734005753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2734005753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2734005753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:13 smithi122 ceph-mon[32255]: pgmap v7251: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:13 smithi022 ceph-mon[31360]: pgmap v7251: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:13 smithi022 ceph-mon[40250]: pgmap v7251: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:15 smithi122 ceph-mon[32255]: pgmap v7252: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:15.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:00:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:00:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:15 smithi022 ceph-mon[31360]: pgmap v7252: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:15 smithi022 ceph-mon[40250]: pgmap v7252: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:17.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:17.670 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:00:17.671 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:17 smithi022 ceph-mon[31360]: pgmap v7253: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:17.671 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:17 smithi022 ceph-mon[40250]: pgmap v7253: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:17.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:17.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:17.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:17 smithi122 ceph-mon[32255]: pgmap v7253: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3647207329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3647207329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3647207329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:19 smithi122 ceph-mon[32255]: pgmap v7254: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:19 smithi022 ceph-mon[31360]: pgmap v7254: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:19 smithi022 ceph-mon[40250]: pgmap v7254: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:20] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T22:00:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:21 smithi122 ceph-mon[32255]: pgmap v7255: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:21 smithi022 ceph-mon[31360]: pgmap v7255: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:21 smithi022 ceph-mon[40250]: pgmap v7255: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:00:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:00:22.921 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:23.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:23.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:23.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:23 smithi122 ceph-mon[32255]: pgmap v7256: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2101385773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:23 smithi022 ceph-mon[31360]: pgmap v7256: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2101385773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:23 smithi022 ceph-mon[40250]: pgmap v7256: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2101385773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:00:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:00:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:25 smithi122 ceph-mon[32255]: pgmap v7257: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:25 smithi022 ceph-mon[31360]: pgmap v7257: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:25 smithi022 ceph-mon[40250]: pgmap v7257: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:27.691 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:00:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:27 smithi022 ceph-mon[31360]: pgmap v7258: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:27 smithi022 ceph-mon[40250]: pgmap v7258: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:27 smithi122 ceph-mon[32255]: pgmap v7258: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:28.290 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:28.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:28.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:28.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:29 smithi122 ceph-mon[32255]: pgmap v7259: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4069319169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:29 smithi022 ceph-mon[31360]: pgmap v7259: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4069319169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:29 smithi022 ceph-mon[40250]: pgmap v7259: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4069319169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:30] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T22:00:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:31 smithi122 ceph-mon[32255]: pgmap v7260: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:31 smithi022 ceph-mon[31360]: pgmap v7260: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:31 smithi022 ceph-mon[40250]: pgmap v7260: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:00:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:00:33.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:33 smithi122 ceph-mon[32255]: pgmap v7261: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:00:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:00:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:00:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:00:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[40250]: pgmap v7261: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:00:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:00:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:00:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:00:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[31360]: pgmap v7261: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:00:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:00:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:00:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:00:34.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:34.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:34.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4122379544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4122379544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4122379544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:35 smithi122 ceph-mon[32255]: pgmap v7262: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:00:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:00:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:35 smithi022 ceph-mon[31360]: pgmap v7262: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:35 smithi022 ceph-mon[40250]: pgmap v7262: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:37 smithi022 ceph-mon[31360]: pgmap v7263: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:00:37.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:37 smithi022 ceph-mon[40250]: pgmap v7263: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:37 smithi122 ceph-mon[32255]: pgmap v7263: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:39.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:39.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:39.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:39.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:39 smithi022 ceph-mon[31360]: pgmap v7264: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:39.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3176214911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:39 smithi022 ceph-mon[40250]: pgmap v7264: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3176214911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:39 smithi122 ceph-mon[32255]: pgmap v7264: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3176214911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:40] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T22:00:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:41 smithi122 ceph-mon[32255]: pgmap v7265: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:41 smithi022 ceph-mon[31360]: pgmap v7265: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:41 smithi022 ceph-mon[40250]: pgmap v7265: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:00:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:00:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:43 smithi122 ceph-mon[32255]: pgmap v7266: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:43 smithi022 ceph-mon[31360]: pgmap v7266: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:43 smithi022 ceph-mon[40250]: pgmap v7266: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:44.405 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:44.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:44.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:44.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:00:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:00:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:45 smithi122 ceph-mon[32255]: pgmap v7267: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2889760910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:45 smithi022 ceph-mon[31360]: pgmap v7267: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2889760910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:45 smithi022 ceph-mon[40250]: pgmap v7267: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2889760910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:47 smithi022 ceph-mon[31360]: pgmap v7268: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:00:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:47 smithi022 ceph-mon[40250]: pgmap v7268: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:47 smithi122 ceph-mon[32255]: pgmap v7268: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:49.776 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:49 smithi122 ceph-mon[32255]: pgmap v7269: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:49 smithi022 ceph-mon[31360]: pgmap v7269: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:49 smithi022 ceph-mon[40250]: pgmap v7269: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:50.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:50.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:50.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:50] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T22:00:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1899871008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1899871008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1899871008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[31360]: pgmap v7270: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:00:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:00:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:00:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[40250]: pgmap v7270: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:00:51.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:00:51.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:00:51.896 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:51 smithi122 ceph-mon[32255]: pgmap v7270: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:51.896 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:00:51.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:00:51.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:00:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:00:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:00:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:00:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:00:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:00:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:53 smithi122 ceph-mon[32255]: pgmap v7271: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:53 smithi022 ceph-mon[31360]: pgmap v7271: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:53 smithi022 ceph-mon[40250]: pgmap v7271: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:55.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:00:55.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:00:55.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:00:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:00:55.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:00:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:00:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:55 smithi122 ceph-mon[32255]: pgmap v7272: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1307855965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:55 smithi022 ceph-mon[31360]: pgmap v7272: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1307855965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:55 smithi022 ceph-mon[40250]: pgmap v7272: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:00:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1307855965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:00:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:00:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:00:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:57 smithi122 ceph-mon[32255]: pgmap v7273: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:57 smithi022 ceph-mon[31360]: pgmap v7273: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:00:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:57 smithi022 ceph-mon[40250]: pgmap v7273: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:00.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:00] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T22:01:00.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:00:59 smithi122 ceph-mon[32255]: pgmap v7274: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:00:59 smithi022 ceph-mon[31360]: pgmap v7274: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:00:59 smithi022 ceph-mon[40250]: pgmap v7274: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:00.520 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:00.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:00.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:00.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/178151900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/178151900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/178151900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:01 smithi022 ceph-mon[31360]: pgmap v7275: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:01 smithi022 ceph-mon[40250]: pgmap v7275: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:01 smithi122 ceph-mon[32255]: pgmap v7275: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:01:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:01:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:03 smithi022 ceph-mon[31360]: pgmap v7276: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:03 smithi022 ceph-mon[40250]: pgmap v7276: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:03 smithi122 ceph-mon[32255]: pgmap v7276: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:05.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:05.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:01:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:01:06.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:06.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:06.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:06.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:05 smithi022 ceph-mon[31360]: pgmap v7277: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:06.261 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:05 smithi022 ceph-mon[40250]: pgmap v7277: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:05 smithi122 ceph-mon[32255]: pgmap v7277: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2514509231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2514509231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2514509231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:01:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:07 smithi022 ceph-mon[31360]: pgmap v7278: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:07 smithi022 ceph-mon[40250]: pgmap v7278: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:07 smithi122 ceph-mon[32255]: pgmap v7278: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:09 smithi022 ceph-mon[31360]: pgmap v7279: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:09 smithi022 ceph-mon[40250]: pgmap v7279: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:10] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:01:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:09 smithi122 ceph-mon[32255]: pgmap v7279: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:11.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:11.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:11.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:11.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:11 smithi022 ceph-mon[31360]: pgmap v7280: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1072415701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:11 smithi022 ceph-mon[40250]: pgmap v7280: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1072415701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:11 smithi122 ceph-mon[32255]: pgmap v7280: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:12.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1072415701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:01:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:01:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:13 smithi022 ceph-mon[31360]: pgmap v7281: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:13 smithi022 ceph-mon[40250]: pgmap v7281: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:13 smithi122 ceph-mon[32255]: pgmap v7281: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:15.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:01:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:01:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:15 smithi022 ceph-mon[31360]: pgmap v7282: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:15 smithi022 ceph-mon[40250]: pgmap v7282: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:15 smithi122 ceph-mon[32255]: pgmap v7282: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:16.630 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:16.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:17.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:17.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:01:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:17 smithi022 ceph-mon[31360]: pgmap v7283: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/246944856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:17 smithi022 ceph-mon[40250]: pgmap v7283: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/246944856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:17 smithi122 ceph-mon[32255]: pgmap v7283: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/246944856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:19 smithi022 ceph-mon[31360]: pgmap v7284: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:19 smithi022 ceph-mon[40250]: pgmap v7284: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:19 smithi122 ceph-mon[32255]: pgmap v7284: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:20] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T22:01:22.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:21 smithi022 ceph-mon[40250]: pgmap v7285: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:21 smithi022 ceph-mon[31360]: pgmap v7285: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:21 smithi122 ceph-mon[32255]: pgmap v7285: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:22.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:22.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:22.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:01:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:01:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221168042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221168042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221168042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:23 smithi022 ceph-mon[31360]: pgmap v7286: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:23 smithi022 ceph-mon[40250]: pgmap v7286: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:23 smithi122 ceph-mon[32255]: pgmap v7286: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:01:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:01:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:25 smithi022 ceph-mon[31360]: pgmap v7287: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:25 smithi022 ceph-mon[40250]: pgmap v7287: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:25 smithi122 ceph-mon[32255]: pgmap v7287: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:27.376 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:27.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:27.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:27.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:27.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:27.746 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:01:28.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:27 smithi022 ceph-mon[31360]: pgmap v7288: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:28.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3761647959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:28.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:27 smithi022 ceph-mon[40250]: pgmap v7288: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:28.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3761647959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:27 smithi122 ceph-mon[32255]: pgmap v7288: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3761647959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:30.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:29 smithi022 ceph-mon[31360]: pgmap v7289: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:29 smithi022 ceph-mon[40250]: pgmap v7289: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:30] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T22:01:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:29 smithi122 ceph-mon[32255]: pgmap v7289: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:31 smithi022 ceph-mon[31360]: pgmap v7290: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:31 smithi022 ceph-mon[40250]: pgmap v7290: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:31 smithi122 ceph-mon[32255]: pgmap v7290: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:01:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:01:32.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:33.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:33.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:33.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[31360]: pgmap v7291: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:01:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:01:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:01:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:01:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/633386622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[40250]: pgmap v7291: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:01:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:01:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:01:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:01:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/633386622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:33 smithi122 ceph-mon[32255]: pgmap v7291: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:01:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:01:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:01:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:01:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/633386622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:01:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:01:36.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:35 smithi022 ceph-mon[31360]: pgmap v7292: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:35 smithi022 ceph-mon[40250]: pgmap v7292: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:36.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:35 smithi122 ceph-mon[32255]: pgmap v7292: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:01:38.123 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:38 smithi022 ceph-mon[31360]: pgmap v7293: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:38 smithi022 ceph-mon[40250]: pgmap v7293: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:38 smithi122 ceph-mon[32255]: pgmap v7293: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:38.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:38.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:38.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:39.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/82488866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:39.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/82488866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/82488866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:40 smithi022 ceph-mon[40250]: pgmap v7294: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:40 smithi022 ceph-mon[31360]: pgmap v7294: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:40] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:01:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:40 smithi122 ceph-mon[32255]: pgmap v7294: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:42 smithi022 ceph-mon[40250]: pgmap v7295: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:42 smithi022 ceph-mon[31360]: pgmap v7295: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:42 smithi122 ceph-mon[32255]: pgmap v7295: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:01:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:01:43.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:43.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:43.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:43.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:44 smithi022 ceph-mon[31360]: pgmap v7296: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1026618266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:44 smithi022 ceph-mon[40250]: pgmap v7296: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1026618266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:44 smithi122 ceph-mon[32255]: pgmap v7296: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1026618266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:01:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:01:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:46 smithi022 ceph-mon[31360]: pgmap v7297: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:46 smithi022 ceph-mon[40250]: pgmap v7297: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:46 smithi122 ceph-mon[32255]: pgmap v7297: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:47.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:01:48.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:48 smithi022 ceph-mon[31360]: pgmap v7298: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:48 smithi022 ceph-mon[40250]: pgmap v7298: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:48 smithi122 ceph-mon[32255]: pgmap v7298: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:48.868 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:49.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:49.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:49.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:50 smithi022 ceph-mon[31360]: pgmap v7299: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3250493445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:50 smithi022 ceph-mon[40250]: pgmap v7299: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3250493445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:50] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T22:01:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:50 smithi122 ceph-mon[32255]: pgmap v7299: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3250493445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[31360]: pgmap v7300: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:52.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:01:52.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:01:52.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:01:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[40250]: pgmap v7300: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:52.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:01:52.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:01:52.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:01:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:52 smithi122 ceph-mon[32255]: pgmap v7300: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:52.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:01:52.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:01:52.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:01:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:01:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:01:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:01:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:53 smithi122 ceph-mon[32255]: pgmap v7301: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:01:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:53 smithi022 ceph-mon[40250]: pgmap v7301: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:01:53.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:53 smithi022 ceph-mon[31360]: pgmap v7301: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:54.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:54.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:54.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:54.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:01:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:01:55.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:55 smithi122 ceph-mon[32255]: pgmap v7302: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:55.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3748197542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:56.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:55 smithi022 ceph-mon[31360]: pgmap v7302: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3748197542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:55 smithi022 ceph-mon[40250]: pgmap v7302: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:56.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3748197542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:01:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:01:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:57 smithi022 ceph-mon[31360]: pgmap v7303: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:01:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:57 smithi022 ceph-mon[40250]: pgmap v7303: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:57 smithi122 ceph-mon[32255]: pgmap v7303: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:01:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:01:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:01:59 smithi122 ceph-mon[32255]: pgmap v7304: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:59.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:01:59.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:01:59.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:01:59.980 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:01:59 smithi022 ceph-mon[31360]: pgmap v7304: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:01:59.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:01:59 smithi022 ceph-mon[40250]: pgmap v7304: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:00] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T22:02:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3463541128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3463541128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3463541128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:01 smithi122 ceph-mon[32255]: pgmap v7305: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:01 smithi022 ceph-mon[31360]: pgmap v7305: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:01 smithi022 ceph-mon[40250]: pgmap v7305: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:02:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:02:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:03 smithi122 ceph-mon[32255]: pgmap v7306: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:03 smithi022 ceph-mon[31360]: pgmap v7306: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:03 smithi022 ceph-mon[40250]: pgmap v7306: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:04.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:05.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:05.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:05 smithi022 ceph-mon[31360]: pgmap v7307: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2796891228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:05 smithi022 ceph-mon[40250]: pgmap v7307: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2796891228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:05.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:02:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:02:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:05 smithi122 ceph-mon[32255]: pgmap v7307: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2796891228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:02:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:07 smithi022 ceph-mon[40250]: pgmap v7308: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:07 smithi022 ceph-mon[31360]: pgmap v7308: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:07 smithi122 ceph-mon[32255]: pgmap v7308: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:09 smithi122 ceph-mon[32255]: pgmap v7309: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:09 smithi022 ceph-mon[31360]: pgmap v7309: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:09 smithi022 ceph-mon[40250]: pgmap v7309: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:10.349 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:10] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T22:02:10.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:10.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:10.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:11 smithi122 ceph-mon[32255]: pgmap v7310: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2841765891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:11 smithi022 ceph-mon[31360]: pgmap v7310: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2841765891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:11 smithi022 ceph-mon[40250]: pgmap v7310: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2841765891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:02:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:02:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:13 smithi122 ceph-mon[32255]: pgmap v7311: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:13 smithi022 ceph-mon[31360]: pgmap v7311: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:13 smithi022 ceph-mon[40250]: pgmap v7311: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:15.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:15 smithi122 ceph-mon[32255]: pgmap v7312: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:15.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:02:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:02:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:15 smithi022 ceph-mon[31360]: pgmap v7312: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:15 smithi022 ceph-mon[40250]: pgmap v7312: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:16.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/679273655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:17.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/679273655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/679273655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:17 smithi022 ceph-mon[31360]: pgmap v7313: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:02:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:17 smithi022 ceph-mon[40250]: pgmap v7313: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:17 smithi122 ceph-mon[32255]: pgmap v7313: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:19 smithi122 ceph-mon[32255]: pgmap v7314: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:19 smithi022 ceph-mon[31360]: pgmap v7314: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:19 smithi022 ceph-mon[40250]: pgmap v7314: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:20] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T22:02:21.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:21.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:21.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:21.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:21 smithi022 ceph-mon[31360]: pgmap v7315: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2578246307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:21 smithi022 ceph-mon[40250]: pgmap v7315: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2578246307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:21 smithi122 ceph-mon[32255]: pgmap v7315: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2578246307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:02:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:02:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:23 smithi122 ceph-mon[32255]: pgmap v7316: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:23 smithi022 ceph-mon[31360]: pgmap v7316: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:23 smithi022 ceph-mon[40250]: pgmap v7316: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:25 smithi122 ceph-mon[32255]: pgmap v7317: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:02:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:02:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:25 smithi022 ceph-mon[31360]: pgmap v7317: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:25 smithi022 ceph-mon[40250]: pgmap v7317: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:26.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:26.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:26.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:26.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:02:27.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:27 smithi022 ceph-mon[31360]: pgmap v7318: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:27.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1465811364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:27 smithi022 ceph-mon[40250]: pgmap v7318: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:27.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1465811364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:27 smithi122 ceph-mon[32255]: pgmap v7318: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1465811364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:29 smithi122 ceph-mon[32255]: pgmap v7319: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:29 smithi022 ceph-mon[31360]: pgmap v7319: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:29 smithi022 ceph-mon[40250]: pgmap v7319: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:30] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T22:02:31.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:31 smithi122 ceph-mon[32255]: pgmap v7320: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:31 smithi022 ceph-mon[31360]: pgmap v7320: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:31 smithi022 ceph-mon[40250]: pgmap v7320: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:32.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:32.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:32.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:32.626 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:02:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:02:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2988209930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2988209930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2988209930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:33 smithi122 ceph-mon[32255]: pgmap v7321: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:02:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:02:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:02:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:02:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[31360]: pgmap v7321: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:02:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:02:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:02:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[40250]: pgmap v7321: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:02:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:02:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:35 smithi122 ceph-mon[32255]: pgmap v7322: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:02:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:02:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:35 smithi022 ceph-mon[31360]: pgmap v7322: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:35 smithi022 ceph-mon[40250]: pgmap v7322: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:37.205 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:02:37.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:37.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:37.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:37 smithi122 ceph-mon[32255]: pgmap v7323: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1475462561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:37 smithi022 ceph-mon[31360]: pgmap v7323: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1475462561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:37 smithi022 ceph-mon[40250]: pgmap v7323: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:38.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1475462561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:39 smithi122 ceph-mon[32255]: pgmap v7324: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:39 smithi022 ceph-mon[31360]: pgmap v7324: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:39 smithi022 ceph-mon[40250]: pgmap v7324: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:40] "GET /metrics HTTP/1.1" 200 202366 "" "Prometheus/2.18.1" 2021-12-15T22:02:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:41 smithi122 ceph-mon[32255]: pgmap v7325: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:41 smithi022 ceph-mon[31360]: pgmap v7325: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:41 smithi022 ceph-mon[40250]: pgmap v7325: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:42.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:02:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:02:42.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:42.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:42.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:43 smithi122 ceph-mon[32255]: pgmap v7326: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:43.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1441496826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:43 smithi022 ceph-mon[31360]: pgmap v7326: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1441496826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:43 smithi022 ceph-mon[40250]: pgmap v7326: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1441496826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:45 smithi122 ceph-mon[32255]: pgmap v7327: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:02:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:02:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:45 smithi022 ceph-mon[31360]: pgmap v7327: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:45 smithi022 ceph-mon[40250]: pgmap v7327: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:47.652 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:02:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:47 smithi122 ceph-mon[32255]: pgmap v7328: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:47.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:47.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:47 smithi022 ceph-mon[31360]: pgmap v7328: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:47.948 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:47 smithi022 ceph-mon[40250]: pgmap v7328: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:48.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:48.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:48.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:48.657 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3794553030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3794553030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3794553030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:49 smithi122 ceph-mon[32255]: pgmap v7329: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:49 smithi022 ceph-mon[31360]: pgmap v7329: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:49 smithi022 ceph-mon[40250]: pgmap v7329: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:50] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:02:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:51 smithi122 ceph-mon[32255]: pgmap v7330: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:51 smithi022 ceph-mon[31360]: pgmap v7330: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:51 smithi022 ceph-mon[40250]: pgmap v7330: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:52.664 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:02:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:02:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:02:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:02:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:02:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:02:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:02:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:02:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:02:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:02:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:02:53.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:53.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:53.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:53.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:53 smithi122 ceph-mon[32255]: pgmap v7331: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:53 smithi022 ceph-mon[40250]: pgmap v7331: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:53 smithi022 ceph-mon[31360]: pgmap v7331: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/85967050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:55.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/85967050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/85967050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:02:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:02:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:55 smithi122 ceph-mon[32255]: pgmap v7332: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:55 smithi022 ceph-mon[31360]: pgmap v7332: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:55 smithi022 ceph-mon[40250]: pgmap v7332: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:02:57.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:57.652 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:02:58.377 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:58 smithi122 ceph-mon[32255]: pgmap v7333: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:58.377 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:58.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:58 smithi022 ceph-mon[31360]: pgmap v7333: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:58.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:58.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:58 smithi022 ceph-mon[40250]: pgmap v7333: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:02:58.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:58.692 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:02:59.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:02:59.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:02:59.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:02:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: pgmap v7334: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2818722137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:59.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:59.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:59.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: pgmap v7334: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:59.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2818722137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:59.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:59.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:02:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:59.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: pgmap v7334: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:02:59.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2818722137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:02:59.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:02:59.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.927 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.927 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:02:59.927 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:02:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:03:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:00] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:03:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:00 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3218M 2021-12-15T22:03:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:03:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:03:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:00 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3218M 2021-12-15T22:03:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:03:01.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:03:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:00 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3218M 2021-12-15T22:03:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:03:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:03:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:01 smithi122 ceph-mon[32255]: pgmap v7335: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:01 smithi022 ceph-mon[31360]: pgmap v7335: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:01 smithi022 ceph-mon[40250]: pgmap v7335: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:03:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:03:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:03 smithi122 ceph-mon[32255]: pgmap v7336: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:03 smithi022 ceph-mon[31360]: pgmap v7336: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:03 smithi022 ceph-mon[40250]: pgmap v7336: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:04.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:04.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:04.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:04.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/997381806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/997381806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/997381806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:05 smithi122 ceph-mon[32255]: pgmap v7337: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:05.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:03:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:03:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:05 smithi022 ceph-mon[31360]: pgmap v7337: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:05 smithi022 ceph-mon[40250]: pgmap v7337: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:07 smithi022 ceph-mon[31360]: pgmap v7338: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:03:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:07 smithi022 ceph-mon[40250]: pgmap v7338: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:07 smithi122 ceph-mon[32255]: pgmap v7338: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:09 smithi022 ceph-mon[31360]: pgmap v7339: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:09 smithi022 ceph-mon[40250]: pgmap v7339: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:09.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:09.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:09.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:09 smithi122 ceph-mon[32255]: pgmap v7339: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:10] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T22:03:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/904830159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/904830159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/904830159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:11 smithi122 ceph-mon[32255]: pgmap v7340: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:11 smithi022 ceph-mon[31360]: pgmap v7340: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:11 smithi022 ceph-mon[40250]: pgmap v7340: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:03:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:03:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:13 smithi122 ceph-mon[32255]: pgmap v7341: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:13 smithi022 ceph-mon[31360]: pgmap v7341: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:13 smithi022 ceph-mon[40250]: pgmap v7341: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:14.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:15.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:15.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:15.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:15.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:03:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:03:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:15 smithi122 ceph-mon[32255]: pgmap v7342: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/834644874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:15 smithi022 ceph-mon[31360]: pgmap v7342: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/834644874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:15 smithi022 ceph-mon[40250]: pgmap v7342: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/834644874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:16.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:17 smithi022 ceph-mon[31360]: pgmap v7343: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:03:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:17 smithi022 ceph-mon[40250]: pgmap v7343: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:17 smithi122 ceph-mon[32255]: pgmap v7343: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:19 smithi122 ceph-mon[32255]: pgmap v7344: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:19 smithi022 ceph-mon[31360]: pgmap v7344: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:19 smithi022 ceph-mon[40250]: pgmap v7344: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:20.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:20] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T22:03:20.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:20.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:20.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/42243999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/42243999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/42243999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:21 smithi122 ceph-mon[32255]: pgmap v7345: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:21 smithi022 ceph-mon[31360]: pgmap v7345: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:21 smithi022 ceph-mon[40250]: pgmap v7345: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:03:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:03:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:23 smithi122 ceph-mon[32255]: pgmap v7346: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:23 smithi022 ceph-mon[31360]: pgmap v7346: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:23 smithi022 ceph-mon[40250]: pgmap v7346: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:25.541 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:25.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:25.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:25.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:25.909 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:25 smithi022 ceph-mon[31360]: pgmap v7347: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:25.910 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:25 smithi022 ceph-mon[40250]: pgmap v7347: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:25 smithi122 ceph-mon[32255]: pgmap v7347: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:03:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:03:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:26.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1124227271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1124227271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1124227271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:27 smithi022 ceph-mon[31360]: pgmap v7348: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:03:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:27 smithi022 ceph-mon[40250]: pgmap v7348: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:27 smithi122 ceph-mon[32255]: pgmap v7348: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:29 smithi122 ceph-mon[32255]: pgmap v7349: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:29 smithi022 ceph-mon[31360]: pgmap v7349: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:29 smithi022 ceph-mon[40250]: pgmap v7349: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:30] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T22:03:30.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:31.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:31.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:31.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:31 smithi122 ceph-mon[32255]: pgmap v7350: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2931986357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:31 smithi022 ceph-mon[31360]: pgmap v7350: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2931986357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:31 smithi022 ceph-mon[40250]: pgmap v7350: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2931986357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:03:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:03:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:33 smithi122 ceph-mon[32255]: pgmap v7351: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:03:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:03:33.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:03:33.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:03:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[31360]: pgmap v7351: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:03:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:03:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:03:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:03:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[40250]: pgmap v7351: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:03:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:03:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:03:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:03:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:03:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:03:35.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:35 smithi122 ceph-mon[32255]: pgmap v7352: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:35 smithi022 ceph-mon[31360]: pgmap v7352: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:35 smithi022 ceph-mon[40250]: pgmap v7352: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:36.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:36.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:36.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:36.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/400608122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/400608122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/400608122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:37.647 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:03:37.647 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:37 smithi022 ceph-mon[40250]: pgmap v7353: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:37 smithi122 ceph-mon[32255]: pgmap v7353: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:37 smithi022 ceph-mon[31360]: pgmap v7353: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:39 smithi122 ceph-mon[32255]: pgmap v7354: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:39 smithi022 ceph-mon[31360]: pgmap v7354: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:39 smithi022 ceph-mon[40250]: pgmap v7354: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:40] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:03:41.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:41 smithi122 ceph-mon[32255]: pgmap v7355: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:42.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:42.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:42.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:42.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:41 smithi022 ceph-mon[31360]: pgmap v7355: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:42.018 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:41 smithi022 ceph-mon[40250]: pgmap v7355: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:42.659 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:03:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:03:42.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2891195379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2891195379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2891195379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:43 smithi122 ceph-mon[32255]: pgmap v7356: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:43 smithi022 ceph-mon[31360]: pgmap v7356: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:43 smithi022 ceph-mon[40250]: pgmap v7356: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:45 smithi122 ceph-mon[32255]: pgmap v7357: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:03:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:03:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:45 smithi022 ceph-mon[31360]: pgmap v7357: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:45 smithi022 ceph-mon[40250]: pgmap v7357: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:47.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:47.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:47.019 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:47.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:47.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:47.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:47.667 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:03:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:47 smithi122 ceph-mon[32255]: pgmap v7358: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3386872348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:47 smithi022 ceph-mon[31360]: pgmap v7358: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3386872348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:47 smithi022 ceph-mon[40250]: pgmap v7358: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3386872348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:49 smithi122 ceph-mon[32255]: pgmap v7359: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:49 smithi022 ceph-mon[31360]: pgmap v7359: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:49 smithi022 ceph-mon[40250]: pgmap v7359: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:50] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T22:03:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:51 smithi122 ceph-mon[32255]: pgmap v7360: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:51 smithi022 ceph-mon[31360]: pgmap v7360: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:51 smithi022 ceph-mon[40250]: pgmap v7360: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:52.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:03:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:03:52.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:52.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:52.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:53 smithi122 ceph-mon[32255]: pgmap v7361: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3546850543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:53 smithi022 ceph-mon[31360]: pgmap v7361: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3546850543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:53 smithi022 ceph-mon[40250]: pgmap v7361: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3546850543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:55.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:03:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:03:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:55 smithi122 ceph-mon[32255]: pgmap v7362: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:55 smithi022 ceph-mon[31360]: pgmap v7362: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:55 smithi022 ceph-mon[40250]: pgmap v7362: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:03:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:03:57.684 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:03:57.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:03:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:57 smithi122 ceph-mon[32255]: pgmap v7363: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:57 smithi022 ceph-mon[31360]: pgmap v7363: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:57 smithi022 ceph-mon[40250]: pgmap v7363: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:03:58.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:03:58.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:03:58.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:03:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3199614633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3199614633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:03:59.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3199614633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:03:59 smithi022 ceph-mon[31360]: pgmap v7364: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:03:59 smithi022 ceph-mon[40250]: pgmap v7364: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:03:59 smithi122 ceph-mon[32255]: pgmap v7364: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:00] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T22:04:00.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:04:00.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:04:00.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:04:00.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:04:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:04:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:04:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:04:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:04:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:04:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:04:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:04:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:04:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:01 smithi022 ceph-mon[31360]: pgmap v7365: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:01 smithi022 ceph-mon[40250]: pgmap v7365: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:01 smithi122 ceph-mon[32255]: pgmap v7365: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:04:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:04:03.132 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:03 smithi022 ceph-mon[31360]: pgmap v7366: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/676882118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:03 smithi022 ceph-mon[40250]: pgmap v7366: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/676882118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:03 smithi122 ceph-mon[32255]: pgmap v7366: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/676882118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:05 smithi122 ceph-mon[32255]: pgmap v7367: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:05.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:04:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:04:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:05 smithi022 ceph-mon[31360]: pgmap v7367: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:05 smithi022 ceph-mon[40250]: pgmap v7367: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:07.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:04:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:07 smithi022 ceph-mon[31360]: pgmap v7368: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:07 smithi022 ceph-mon[40250]: pgmap v7368: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:07 smithi122 ceph-mon[32255]: pgmap v7368: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:08.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:08.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:08.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:08.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:09 smithi022 ceph-mon[31360]: pgmap v7369: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2004023824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:09 smithi022 ceph-mon[40250]: pgmap v7369: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2004023824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:09 smithi122 ceph-mon[32255]: pgmap v7369: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:10.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2004023824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:10] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T22:04:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:11 smithi022 ceph-mon[31360]: pgmap v7370: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:11 smithi022 ceph-mon[40250]: pgmap v7370: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:11 smithi122 ceph-mon[32255]: pgmap v7370: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:04:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:04:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:13 smithi022 ceph-mon[31360]: pgmap v7371: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:13 smithi022 ceph-mon[40250]: pgmap v7371: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:13 smithi122 ceph-mon[32255]: pgmap v7371: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4271470405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4271470405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4271470405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:15 smithi022 ceph-mon[31360]: pgmap v7372: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:15 smithi022 ceph-mon[40250]: pgmap v7372: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:15 smithi122 ceph-mon[32255]: pgmap v7372: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:16.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:04:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:04:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:17.726 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:04:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:17 smithi022 ceph-mon[31360]: pgmap v7373: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:17 smithi022 ceph-mon[40250]: pgmap v7373: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:17 smithi122 ceph-mon[32255]: pgmap v7373: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:19.253 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:19.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:19.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:19.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:19 smithi022 ceph-mon[31360]: pgmap v7374: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3673576418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:19 smithi022 ceph-mon[40250]: pgmap v7374: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3673576418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:19 smithi122 ceph-mon[32255]: pgmap v7374: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3673576418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:20] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:04:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:21 smithi022 ceph-mon[31360]: pgmap v7375: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:21 smithi022 ceph-mon[40250]: pgmap v7375: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:21 smithi122 ceph-mon[32255]: pgmap v7375: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:04:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:04:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:23 smithi022 ceph-mon[31360]: pgmap v7376: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:23 smithi022 ceph-mon[40250]: pgmap v7376: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:23 smithi122 ceph-mon[32255]: pgmap v7376: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:24.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:24.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:24.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:24.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:25 smithi022 ceph-mon[31360]: pgmap v7377: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3147950901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:25 smithi022 ceph-mon[40250]: pgmap v7377: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3147950901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:26.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:04:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:04:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:25 smithi122 ceph-mon[32255]: pgmap v7377: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3147950901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:27.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:04:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:27 smithi022 ceph-mon[31360]: pgmap v7378: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:27 smithi022 ceph-mon[40250]: pgmap v7378: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:27 smithi122 ceph-mon[32255]: pgmap v7378: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:29.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:29 smithi022 ceph-mon[31360]: pgmap v7379: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:29 smithi022 ceph-mon[40250]: pgmap v7379: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:29 smithi122 ceph-mon[32255]: pgmap v7379: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:30.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:30.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:30.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:30] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:04:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3009538030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3009538030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3009538030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:31 smithi022 ceph-mon[31360]: pgmap v7380: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:31 smithi022 ceph-mon[40250]: pgmap v7380: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:31 smithi122 ceph-mon[32255]: pgmap v7380: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:04:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:04:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[31360]: pgmap v7381: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:04:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:04:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:04:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:04:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[40250]: pgmap v7381: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:04:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:04:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:04:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:04:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:33 smithi122 ceph-mon[32255]: pgmap v7381: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:04:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:04:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:04:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:04:35.363 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:35 smithi022 ceph-mon[31360]: pgmap v7382: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4098148674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:35 smithi022 ceph-mon[40250]: pgmap v7382: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4098148674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:36.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:04:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:04:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:35 smithi122 ceph-mon[32255]: pgmap v7382: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4098148674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:37.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:04:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:37 smithi022 ceph-mon[31360]: pgmap v7383: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:37 smithi022 ceph-mon[40250]: pgmap v7383: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:37 smithi122 ceph-mon[32255]: pgmap v7383: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:39 smithi022 ceph-mon[31360]: pgmap v7384: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:39 smithi022 ceph-mon[40250]: pgmap v7384: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:39 smithi122 ceph-mon[32255]: pgmap v7384: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:40] "GET /metrics HTTP/1.1" 200 202356 "" "Prometheus/2.18.1" 2021-12-15T22:04:40.730 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:41.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:41.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:41.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:41 smithi022 ceph-mon[31360]: pgmap v7385: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3051950435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:41 smithi022 ceph-mon[40250]: pgmap v7385: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3051950435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:41 smithi122 ceph-mon[32255]: pgmap v7385: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3051950435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:04:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:04:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:43 smithi022 ceph-mon[31360]: pgmap v7386: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:43 smithi022 ceph-mon[40250]: pgmap v7386: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:43 smithi122 ceph-mon[32255]: pgmap v7386: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:45 smithi022 ceph-mon[31360]: pgmap v7387: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:45 smithi022 ceph-mon[40250]: pgmap v7387: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:46.099 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:45 smithi122 ceph-mon[32255]: pgmap v7387: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:46.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:04:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:04:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:46.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:46.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:47.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3058365762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3058365762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3058365762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:04:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:47 smithi122 ceph-mon[32255]: pgmap v7388: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:47 smithi022 ceph-mon[31360]: pgmap v7388: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:47 smithi022 ceph-mon[40250]: pgmap v7388: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:49 smithi122 ceph-mon[32255]: pgmap v7389: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:49 smithi022 ceph-mon[31360]: pgmap v7389: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:49 smithi022 ceph-mon[40250]: pgmap v7389: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:50] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:04:51.468 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:51.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:51 smithi122 ceph-mon[32255]: pgmap v7390: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:51 smithi022 ceph-mon[31360]: pgmap v7390: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:51 smithi022 ceph-mon[40250]: pgmap v7390: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:04:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:04:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2632993837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2632993837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2632993837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:53 smithi122 ceph-mon[32255]: pgmap v7391: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:53 smithi022 ceph-mon[31360]: pgmap v7391: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:53 smithi022 ceph-mon[40250]: pgmap v7391: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:55 smithi122 ceph-mon[32255]: pgmap v7392: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:56.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:04:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:04:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:55 smithi022 ceph-mon[31360]: pgmap v7392: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:55 smithi022 ceph-mon[40250]: pgmap v7392: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:04:56.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:04:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:57.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:04:57.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:04:57.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:04:57.209 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:57.209 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:04:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:04:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:57 smithi122 ceph-mon[32255]: pgmap v7393: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/926238420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:57 smithi022 ceph-mon[31360]: pgmap v7393: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/926238420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:04:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:57 smithi022 ceph-mon[40250]: pgmap v7393: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:04:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/926238420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:04:59 smithi122 ceph-mon[32255]: pgmap v7394: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:04:59 smithi022 ceph-mon[31360]: pgmap v7394: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:04:59 smithi022 ceph-mon[40250]: pgmap v7394: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:00] "GET /metrics HTTP/1.1" 200 202370 "" "Prometheus/2.18.1" 2021-12-15T22:05:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:05:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:05:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:05:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:05:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:05:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:05:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:05:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:05:01.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:05:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:01 smithi122 ceph-mon[32255]: pgmap v7395: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:05:02.211 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:02.211 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:01 smithi022 ceph-mon[31360]: pgmap v7395: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:02.212 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:05:02.212 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:01 smithi022 ceph-mon[40250]: pgmap v7395: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:02.212 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:05:02.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:02.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:02.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:05:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:05:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2950882576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2950882576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2950882576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:03 smithi122 ceph-mon[32255]: pgmap v7396: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:03 smithi022 ceph-mon[31360]: pgmap v7396: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:03 smithi022 ceph-mon[40250]: pgmap v7396: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:06.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:05:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:05:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:05 smithi122 ceph-mon[32255]: pgmap v7397: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:05 smithi022 ceph-mon[31360]: pgmap v7397: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:05 smithi022 ceph-mon[40250]: pgmap v7397: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:07.582 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:05:07.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:07.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:07.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:07 smithi122 ceph-mon[32255]: pgmap v7398: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:07 smithi022 ceph-mon[31360]: pgmap v7398: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:07 smithi022 ceph-mon[40250]: pgmap v7398: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3648610762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3648610762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3648610762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:09 smithi122 ceph-mon[32255]: pgmap v7399: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:09 smithi022 ceph-mon[31360]: pgmap v7399: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:09 smithi022 ceph-mon[40250]: pgmap v7399: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:10] "GET /metrics HTTP/1.1" 200 202368 "" "Prometheus/2.18.1" 2021-12-15T22:05:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:11 smithi122 ceph-mon[32255]: pgmap v7400: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:11 smithi022 ceph-mon[31360]: pgmap v7400: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:11 smithi022 ceph-mon[40250]: pgmap v7400: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:05:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:05:12.951 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:13.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:13.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:13 smithi122 ceph-mon[32255]: pgmap v7401: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3782873054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:13 smithi022 ceph-mon[31360]: pgmap v7401: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:14.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3782873054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:14.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:13 smithi022 ceph-mon[40250]: pgmap v7401: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:14.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3782873054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:16.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:05:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:05:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:15 smithi122 ceph-mon[32255]: pgmap v7402: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:15 smithi022 ceph-mon[31360]: pgmap v7402: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:15 smithi022 ceph-mon[40250]: pgmap v7402: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:05:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:17 smithi122 ceph-mon[32255]: pgmap v7403: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:17 smithi022 ceph-mon[31360]: pgmap v7403: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:17 smithi022 ceph-mon[40250]: pgmap v7403: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:18.321 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:18.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:18.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:18.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:19 smithi122 ceph-mon[32255]: pgmap v7404: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/27560920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:19 smithi022 ceph-mon[31360]: pgmap v7404: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:20.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/27560920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:19 smithi022 ceph-mon[40250]: pgmap v7404: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/27560920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:20] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T22:05:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:21 smithi122 ceph-mon[32255]: pgmap v7405: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:21 smithi022 ceph-mon[31360]: pgmap v7405: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:21 smithi022 ceph-mon[40250]: pgmap v7405: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:05:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:05:23.695 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:23 smithi122 ceph-mon[32255]: pgmap v7406: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:23 smithi022 ceph-mon[31360]: pgmap v7406: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:23 smithi022 ceph-mon[40250]: pgmap v7406: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:24.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:24.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:24.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3304862450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3304862450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3304862450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:25 smithi122 ceph-mon[32255]: pgmap v7407: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:05:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:05:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:25 smithi022 ceph-mon[31360]: pgmap v7407: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:25 smithi022 ceph-mon[40250]: pgmap v7407: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:27 smithi022 ceph-mon[31360]: pgmap v7408: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:05:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:27 smithi022 ceph-mon[40250]: pgmap v7408: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:27 smithi122 ceph-mon[32255]: pgmap v7408: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:29.065 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:29.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:29.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:29.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:29 smithi022 ceph-mon[31360]: pgmap v7409: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3901621477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:29 smithi022 ceph-mon[40250]: pgmap v7409: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3901621477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:29 smithi122 ceph-mon[32255]: pgmap v7409: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3901621477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:30] "GET /metrics HTTP/1.1" 200 202365 "" "Prometheus/2.18.1" 2021-12-15T22:05:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:31 smithi122 ceph-mon[32255]: pgmap v7410: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:31 smithi022 ceph-mon[31360]: pgmap v7410: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:31 smithi022 ceph-mon[40250]: pgmap v7410: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:05:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:05:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:33 smithi122 ceph-mon[32255]: pgmap v7411: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:05:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:05:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:05:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:05:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[31360]: pgmap v7411: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:05:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:05:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:05:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:05:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[40250]: pgmap v7411: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:05:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:05:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:05:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:05:34.439 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:34.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:34.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:34.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:05:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:05:35.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:35 smithi122 ceph-mon[32255]: pgmap v7412: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:35.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/920143516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:35 smithi022 ceph-mon[31360]: pgmap v7412: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/920143516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:35 smithi022 ceph-mon[40250]: pgmap v7412: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/920143516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:37 smithi022 ceph-mon[31360]: pgmap v7413: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:05:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:37 smithi022 ceph-mon[40250]: pgmap v7413: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:37 smithi122 ceph-mon[32255]: pgmap v7413: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:39.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:39 smithi122 ceph-mon[32255]: pgmap v7414: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:39 smithi022 ceph-mon[31360]: pgmap v7414: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:39 smithi022 ceph-mon[40250]: pgmap v7414: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:40.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:40.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:40.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:40] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T22:05:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/661124470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/661124470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/661124470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:41 smithi122 ceph-mon[32255]: pgmap v7415: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:41 smithi022 ceph-mon[31360]: pgmap v7415: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:41 smithi022 ceph-mon[40250]: pgmap v7415: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:05:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:05:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:43 smithi122 ceph-mon[32255]: pgmap v7416: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:43 smithi022 ceph-mon[31360]: pgmap v7416: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:44.138 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:43 smithi022 ceph-mon[40250]: pgmap v7416: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:45.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:45.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:45.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:45.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:05:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:05:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:45 smithi122 ceph-mon[32255]: pgmap v7417: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/938449043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:45 smithi022 ceph-mon[31360]: pgmap v7417: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/938449043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:45 smithi022 ceph-mon[40250]: pgmap v7417: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/938449043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:47.658 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:05:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:47 smithi122 ceph-mon[32255]: pgmap v7418: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:47 smithi022 ceph-mon[31360]: pgmap v7418: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:47 smithi022 ceph-mon[40250]: pgmap v7418: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:05:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:49 smithi122 ceph-mon[32255]: pgmap v7419: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:49 smithi022 ceph-mon[31360]: pgmap v7419: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:49 smithi022 ceph-mon[40250]: pgmap v7419: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:50] "GET /metrics HTTP/1.1" 200 202371 "" "Prometheus/2.18.1" 2021-12-15T22:05:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:50.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:50.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:50.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:51 smithi122 ceph-mon[32255]: pgmap v7420: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3012829684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:51 smithi022 ceph-mon[31360]: pgmap v7420: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3012829684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:51 smithi022 ceph-mon[40250]: pgmap v7420: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3012829684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:05:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:05:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:53 smithi122 ceph-mon[32255]: pgmap v7421: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:53 smithi022 ceph-mon[31360]: pgmap v7421: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:53 smithi022 ceph-mon[40250]: pgmap v7421: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:55.925 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:05:55.926 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:55 smithi022 ceph-mon[31360]: pgmap v7422: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:55.927 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:55 smithi022 ceph-mon[40250]: pgmap v7422: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:55 smithi122 ceph-mon[32255]: pgmap v7422: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:05:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:05:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:05:56.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:05:56.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:05:56.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:05:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1236812686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1236812686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:05:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1236812686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:05:57.676 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:05:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:57 smithi122 ceph-mon[32255]: pgmap v7423: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:57 smithi022 ceph-mon[31360]: pgmap v7423: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:57 smithi022 ceph-mon[40250]: pgmap v7423: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:05:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:05:59 smithi122 ceph-mon[32255]: pgmap v7424: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:05:59 smithi022 ceph-mon[31360]: pgmap v7424: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:05:59 smithi022 ceph-mon[40250]: pgmap v7424: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:00] "GET /metrics HTTP/1.1" 200 202371 "" "Prometheus/2.18.1" 2021-12-15T22:06:01.299 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:01.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:01.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:01.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:01 smithi122 ceph-mon[32255]: pgmap v7425: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:06:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:06:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:06:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:06:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2358571817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[40250]: pgmap v7425: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:06:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:06:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:06:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:06:02.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2358571817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:02.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[31360]: pgmap v7425: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:02.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:06:02.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:06:02.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:06:02.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:06:02.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2358571817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:06:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:06:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:03 smithi122 ceph-mon[32255]: pgmap v7426: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:03 smithi022 ceph-mon[31360]: pgmap v7426: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:03 smithi022 ceph-mon[40250]: pgmap v7426: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:05 smithi022 ceph-mon[31360]: pgmap v7427: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:05 smithi022 ceph-mon[40250]: pgmap v7427: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:05 smithi122 ceph-mon[32255]: pgmap v7427: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:06.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:06:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:06:06.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:07.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:07.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:07.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:07.693 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:06:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:07 smithi022 ceph-mon[31360]: pgmap v7428: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3321268126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:08.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:07 smithi022 ceph-mon[40250]: pgmap v7428: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:08.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3321268126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:07 smithi122 ceph-mon[32255]: pgmap v7428: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3321268126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:09 smithi022 ceph-mon[31360]: pgmap v7429: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:09 smithi022 ceph-mon[40250]: pgmap v7429: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:09 smithi122 ceph-mon[32255]: pgmap v7429: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:10] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T22:06:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:11 smithi022 ceph-mon[31360]: pgmap v7430: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:11 smithi022 ceph-mon[40250]: pgmap v7430: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:12.043 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:11 smithi122 ceph-mon[32255]: pgmap v7430: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:12.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:12.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:12.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:06:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:06:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1552916725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1552916725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1552916725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:13 smithi022 ceph-mon[31360]: pgmap v7431: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:13 smithi022 ceph-mon[40250]: pgmap v7431: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:13 smithi122 ceph-mon[32255]: pgmap v7431: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:15 smithi022 ceph-mon[31360]: pgmap v7432: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:15 smithi022 ceph-mon[40250]: pgmap v7432: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:16.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:06:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:06:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:15 smithi122 ceph-mon[32255]: pgmap v7432: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:17.415 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:17.717 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:06:17.784 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:17.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:17.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:17 smithi022 ceph-mon[31360]: pgmap v7433: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:17 smithi022 ceph-mon[40250]: pgmap v7433: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:17 smithi122 ceph-mon[32255]: pgmap v7433: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3639754309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3639754309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3639754309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:19 smithi022 ceph-mon[31360]: pgmap v7434: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:19 smithi022 ceph-mon[40250]: pgmap v7434: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:19 smithi122 ceph-mon[32255]: pgmap v7434: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:20] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T22:06:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:21 smithi022 ceph-mon[31360]: pgmap v7435: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:21 smithi022 ceph-mon[40250]: pgmap v7435: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:21 smithi122 ceph-mon[32255]: pgmap v7435: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:06:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:06:22.786 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:23 smithi022 ceph-mon[31360]: pgmap v7436: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1477513530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:23 smithi022 ceph-mon[40250]: pgmap v7436: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1477513530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:23 smithi122 ceph-mon[32255]: pgmap v7436: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1477513530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:25 smithi022 ceph-mon[31360]: pgmap v7437: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:25 smithi022 ceph-mon[40250]: pgmap v7437: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:25 smithi122 ceph-mon[32255]: pgmap v7437: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:26.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:06:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:06:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:27.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:06:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:27 smithi022 ceph-mon[31360]: pgmap v7438: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:27 smithi022 ceph-mon[40250]: pgmap v7438: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:28.156 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:27 smithi122 ceph-mon[32255]: pgmap v7438: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1452875817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1452875817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1452875817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:29 smithi022 ceph-mon[31360]: pgmap v7439: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:29 smithi022 ceph-mon[40250]: pgmap v7439: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:29 smithi122 ceph-mon[32255]: pgmap v7439: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:30] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T22:06:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:31 smithi022 ceph-mon[31360]: pgmap v7440: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:31 smithi022 ceph-mon[40250]: pgmap v7440: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:31 smithi122 ceph-mon[32255]: pgmap v7440: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:06:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:06:33.527 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[31360]: pgmap v7441: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:06:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:06:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:06:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:06:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[40250]: pgmap v7441: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:06:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:06:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:06:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:06:33.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:33.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:33.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:33 smithi122 ceph-mon[32255]: pgmap v7441: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:06:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:06:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:06:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2277672850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2277672850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2277672850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:35 smithi022 ceph-mon[31360]: pgmap v7442: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:35 smithi022 ceph-mon[40250]: pgmap v7442: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:35 smithi122 ceph-mon[32255]: pgmap v7442: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:36.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:06:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:06:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:37.758 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:06:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:37 smithi022 ceph-mon[31360]: pgmap v7443: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:37 smithi022 ceph-mon[40250]: pgmap v7443: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:37 smithi122 ceph-mon[32255]: pgmap v7443: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:38.897 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:39.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:39.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:39.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:39 smithi022 ceph-mon[31360]: pgmap v7444: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2901399372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:39 smithi022 ceph-mon[40250]: pgmap v7444: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2901399372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:39 smithi122 ceph-mon[32255]: pgmap v7444: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2901399372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:40] "GET /metrics HTTP/1.1" 200 202361 "" "Prometheus/2.18.1" 2021-12-15T22:06:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:41 smithi022 ceph-mon[31360]: pgmap v7445: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:41 smithi022 ceph-mon[40250]: pgmap v7445: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:41 smithi122 ceph-mon[32255]: pgmap v7445: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:06:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:06:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:43 smithi022 ceph-mon[31360]: pgmap v7446: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:43 smithi022 ceph-mon[40250]: pgmap v7446: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:43 smithi122 ceph-mon[32255]: pgmap v7446: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:44.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2645078246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2645078246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2645078246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:45 smithi022 ceph-mon[31360]: pgmap v7447: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:45 smithi022 ceph-mon[40250]: pgmap v7447: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:45 smithi122 ceph-mon[32255]: pgmap v7447: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:46.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:06:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:06:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:06:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:47 smithi122 ceph-mon[32255]: pgmap v7448: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:47 smithi022 ceph-mon[31360]: pgmap v7448: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:47 smithi022 ceph-mon[40250]: pgmap v7448: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:49.636 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:50.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:50.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:50.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:50.008 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:49 smithi022 ceph-mon[40250]: pgmap v7449: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:50.008 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:49 smithi022 ceph-mon[31360]: pgmap v7449: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:49 smithi122 ceph-mon[32255]: pgmap v7449: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:50] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T22:06:51.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/878411229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/878411229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/878411229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:51 smithi122 ceph-mon[32255]: pgmap v7450: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:51 smithi022 ceph-mon[40250]: pgmap v7450: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:52.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:51 smithi022 ceph-mon[31360]: pgmap v7450: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:06:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:06:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:53 smithi122 ceph-mon[32255]: pgmap v7451: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:53 smithi022 ceph-mon[31360]: pgmap v7451: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:53 smithi022 ceph-mon[40250]: pgmap v7451: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:55.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:06:55.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:06:55.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:06:55.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:06:56.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:06:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:06:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:55 smithi122 ceph-mon[32255]: pgmap v7452: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:56.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4110619263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:55 smithi022 ceph-mon[31360]: pgmap v7452: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4110619263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:55 smithi022 ceph-mon[40250]: pgmap v7452: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:06:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4110619263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:06:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:06:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:06:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:57 smithi122 ceph-mon[32255]: pgmap v7453: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:57 smithi022 ceph-mon[31360]: pgmap v7453: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:06:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:57 smithi022 ceph-mon[40250]: pgmap v7453: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:06:59 smithi122 ceph-mon[32255]: pgmap v7454: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:06:59 smithi022 ceph-mon[31360]: pgmap v7454: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:06:59 smithi022 ceph-mon[40250]: pgmap v7454: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:00.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:00] "GET /metrics HTTP/1.1" 200 202363 "" "Prometheus/2.18.1" 2021-12-15T22:07:00.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:00.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:00.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1561283975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1561283975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1561283975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:01 smithi122 ceph-mon[32255]: pgmap v7455: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:07:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:07:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:07:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[31360]: pgmap v7455: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:07:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:07:02.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:07:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[40250]: pgmap v7455: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:07:02.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:07:02.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:07:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:07:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:07:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:07:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:07:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:07:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:04 smithi122 ceph-mon[32255]: pgmap v7456: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:04 smithi022 ceph-mon[31360]: pgmap v7456: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:04 smithi022 ceph-mon[40250]: pgmap v7456: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:05.751 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:06.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:06.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:06.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:06 smithi122 ceph-mon[32255]: pgmap v7457: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:06.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:07:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:07:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:06 smithi022 ceph-mon[31360]: pgmap v7457: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:06 smithi022 ceph-mon[40250]: pgmap v7457: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:07.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2461136192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2461136192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2461136192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:07:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:08 smithi122 ceph-mon[32255]: pgmap v7458: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:08 smithi022 ceph-mon[31360]: pgmap v7458: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:08 smithi022 ceph-mon[40250]: pgmap v7458: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:10 smithi122 ceph-mon[32255]: pgmap v7459: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:10] "GET /metrics HTTP/1.1" 200 202367 "" "Prometheus/2.18.1" 2021-12-15T22:07:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:10 smithi022 ceph-mon[31360]: pgmap v7459: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:10 smithi022 ceph-mon[40250]: pgmap v7459: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:11.113 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:11.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:11.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:11.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:12 smithi122 ceph-mon[32255]: pgmap v7460: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2458389555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:12.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:07:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:07:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:12 smithi022 ceph-mon[31360]: pgmap v7460: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2458389555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:12 smithi022 ceph-mon[40250]: pgmap v7460: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2458389555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:14 smithi122 ceph-mon[32255]: pgmap v7461: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:14 smithi022 ceph-mon[31360]: pgmap v7461: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:14 smithi022 ceph-mon[40250]: pgmap v7461: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:16.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:07:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:07:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:16 smithi122 ceph-mon[32255]: pgmap v7462: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:16.483 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:16.483 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:16 smithi022 ceph-mon[31360]: pgmap v7462: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:16.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:16 smithi022 ceph-mon[40250]: pgmap v7462: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:16.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:16.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:16.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2109121969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2109121969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:17.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:17.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2109121969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:07:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:18 smithi122 ceph-mon[32255]: pgmap v7463: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:18 smithi022 ceph-mon[31360]: pgmap v7463: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:18 smithi022 ceph-mon[40250]: pgmap v7463: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:20] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T22:07:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:20 smithi122 ceph-mon[32255]: pgmap v7464: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:20 smithi022 ceph-mon[31360]: pgmap v7464: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:20 smithi022 ceph-mon[40250]: pgmap v7464: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:21 smithi122 ceph-mon[32255]: pgmap v7465: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:21 smithi022 ceph-mon[31360]: pgmap v7465: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:21 smithi022 ceph-mon[40250]: pgmap v7465: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:21.853 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:22.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:22.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:22.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3040034952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3040034952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:07:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:07:22.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3040034952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:23 smithi022 ceph-mon[31360]: pgmap v7466: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:23 smithi022 ceph-mon[40250]: pgmap v7466: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:23 smithi122 ceph-mon[32255]: pgmap v7466: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:25 smithi122 ceph-mon[32255]: pgmap v7467: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:07:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:07:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:25 smithi022 ceph-mon[31360]: pgmap v7467: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:25 smithi022 ceph-mon[40250]: pgmap v7467: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:27.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:07:27.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:27.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:27 smithi122 ceph-mon[32255]: pgmap v7468: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4058498946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:27 smithi022 ceph-mon[31360]: pgmap v7468: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4058498946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:27 smithi022 ceph-mon[40250]: pgmap v7468: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4058498946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:29 smithi122 ceph-mon[32255]: pgmap v7469: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:29 smithi022 ceph-mon[31360]: pgmap v7469: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:29 smithi022 ceph-mon[40250]: pgmap v7469: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:30] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T22:07:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:31 smithi122 ceph-mon[32255]: pgmap v7470: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:31 smithi022 ceph-mon[31360]: pgmap v7470: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:31 smithi022 ceph-mon[40250]: pgmap v7470: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:32.591 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:07:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:07:32.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:32.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:32.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:33 smithi122 ceph-mon[32255]: pgmap v7471: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1165370854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:07:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:07:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:07:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:07:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[31360]: pgmap v7471: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1165370854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:07:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:07:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:07:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:07:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[40250]: pgmap v7471: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1165370854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:07:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:07:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:07:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:07:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:07:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:07:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:35 smithi122 ceph-mon[32255]: pgmap v7472: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:35 smithi022 ceph-mon[31360]: pgmap v7472: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:35 smithi022 ceph-mon[40250]: pgmap v7472: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:37.656 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:07:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:37 smithi122 ceph-mon[32255]: pgmap v7473: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:37.961 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:37 smithi022 ceph-mon[40250]: pgmap v7473: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:37.961 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:37 smithi022 ceph-mon[31360]: pgmap v7473: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:38.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:38.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:38.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/131340272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/131340272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/131340272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:39 smithi122 ceph-mon[32255]: pgmap v7474: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:39 smithi022 ceph-mon[31360]: pgmap v7474: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:39 smithi022 ceph-mon[40250]: pgmap v7474: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:40] "GET /metrics HTTP/1.1" 200 202360 "" "Prometheus/2.18.1" 2021-12-15T22:07:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:41 smithi122 ceph-mon[32255]: pgmap v7475: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:41 smithi022 ceph-mon[31360]: pgmap v7475: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:41 smithi022 ceph-mon[40250]: pgmap v7475: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:07:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:07:43.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:43.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:43.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:43.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:43 smithi122 ceph-mon[32255]: pgmap v7476: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:43 smithi022 ceph-mon[31360]: pgmap v7476: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:43 smithi022 ceph-mon[40250]: pgmap v7476: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1752658295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1752658295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:45.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1752658295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:45 smithi122 ceph-mon[32255]: pgmap v7477: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:45.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:07:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:07:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:45 smithi022 ceph-mon[31360]: pgmap v7477: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:45 smithi022 ceph-mon[40250]: pgmap v7477: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:47.686 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:07:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:47 smithi122 ceph-mon[32255]: pgmap v7478: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:47 smithi022 ceph-mon[31360]: pgmap v7478: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:47 smithi022 ceph-mon[40250]: pgmap v7478: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:48.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:49 smithi022 ceph-mon[31360]: pgmap v7479: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2984012870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:49 smithi022 ceph-mon[40250]: pgmap v7479: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2984012870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:49 smithi122 ceph-mon[32255]: pgmap v7479: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:07:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2984012870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:50] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T22:07:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:51 smithi022 ceph-mon[31360]: pgmap v7480: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:51 smithi022 ceph-mon[40250]: pgmap v7480: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:51 smithi122 ceph-mon[32255]: pgmap v7480: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:07:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:07:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:53 smithi022 ceph-mon[31360]: pgmap v7481: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:53 smithi022 ceph-mon[40250]: pgmap v7481: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:54.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:53 smithi122 ceph-mon[32255]: pgmap v7481: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:07:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:07:55.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/882481605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:55.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/882481605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:55.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/882481605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:07:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:55 smithi022 ceph-mon[31360]: pgmap v7482: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:07:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:55 smithi022 ceph-mon[40250]: pgmap v7482: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:07:56.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:07:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:07:56.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:55 smithi122 ceph-mon[32255]: pgmap v7482: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:07:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:07:57.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:07:58.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:57 smithi022 ceph-mon[31360]: pgmap v7483: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:07:58.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:57 smithi022 ceph-mon[40250]: pgmap v7483: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:07:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:57 smithi122 ceph-mon[32255]: pgmap v7483: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:07:59.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:07:59.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:07:59.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:07:59.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:07:59 smithi022 ceph-mon[31360]: pgmap v7484: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:07:59 smithi022 ceph-mon[40250]: pgmap v7484: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:07:59 smithi122 ceph-mon[32255]: pgmap v7484: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:00] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T22:08:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1622378630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1622378630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1622378630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:01 smithi022 ceph-mon[31360]: pgmap v7485: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:01 smithi022 ceph-mon[40250]: pgmap v7485: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:02.091 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:01 smithi122 ceph-mon[32255]: pgmap v7485: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:08:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:08:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:08:02.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:08:02.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:08:02.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:08:02.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:08:02.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:08:02.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:08:02.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:08:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:08:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:08:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:08:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:08:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:03 smithi022 ceph-mon[31360]: pgmap v7486: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:03 smithi022 ceph-mon[40250]: pgmap v7486: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:03 smithi122 ceph-mon[32255]: pgmap v7486: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:08:04.820 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:05.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:05.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:05.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:05 smithi022 ceph-mon[31360]: pgmap v7487: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:08:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2145796833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:05 smithi022 ceph-mon[40250]: pgmap v7487: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:08:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2145796833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:05 smithi122 ceph-mon[32255]: pgmap v7487: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:08:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2145796833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:06.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:08:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:08:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:07.726 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:08:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:07 smithi022 ceph-mon[31360]: pgmap v7488: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:07 smithi022 ceph-mon[40250]: pgmap v7488: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:07 smithi122 ceph-mon[32255]: pgmap v7488: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:09 smithi022 ceph-mon[31360]: pgmap v7489: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:08:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:09 smithi022 ceph-mon[40250]: pgmap v7489: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:08:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:09 smithi122 ceph-mon[32255]: pgmap v7489: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:08:10.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:10] "GET /metrics HTTP/1.1" 200 202364 "" "Prometheus/2.18.1" 2021-12-15T22:08:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3083773150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3083773150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3083773150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:11 smithi022 ceph-mon[31360]: pgmap v7490: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:11 smithi022 ceph-mon[40250]: pgmap v7490: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:11 smithi122 ceph-mon[32255]: pgmap v7490: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:08:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:08:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:13 smithi022 ceph-mon[31360]: pgmap v7491: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:13 smithi022 ceph-mon[40250]: pgmap v7491: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:13 smithi122 ceph-mon[32255]: pgmap v7491: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:08:15.554 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:15.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:15.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:15.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:15.926 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:15 smithi022 ceph-mon[31360]: pgmap v7492: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:08:15.927 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:15 smithi022 ceph-mon[40250]: pgmap v7492: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:08:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:15 smithi122 ceph-mon[32255]: pgmap v7492: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:08:16.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:08:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:08:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2481925539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2481925539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2481925539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:17.744 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:08:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:17 smithi022 ceph-mon[31360]: pgmap v7493: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:17 smithi022 ceph-mon[40250]: pgmap v7493: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:17 smithi122 ceph-mon[32255]: pgmap v7493: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:19 smithi022 ceph-mon[31360]: pgmap v7494: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:19 smithi022 ceph-mon[40250]: pgmap v7494: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:19 smithi122 ceph-mon[32255]: pgmap v7494: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:20] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:08:20.927 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:21.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:21.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:21.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:21.751 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:21 smithi022 ceph-mon[31360]: pgmap v7495: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2879256012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:21 smithi022 ceph-mon[40250]: pgmap v7495: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2879256012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:21 smithi122 ceph-mon[32255]: pgmap v7495: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2879256012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:08:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:08:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:23 smithi022 ceph-mon[31360]: pgmap v7496: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:23 smithi022 ceph-mon[40250]: pgmap v7496: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:23 smithi122 ceph-mon[32255]: pgmap v7496: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:25 smithi022 ceph-mon[31360]: pgmap v7497: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:25 smithi022 ceph-mon[40250]: pgmap v7497: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:26.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:08:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:08:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:25 smithi122 ceph-mon[32255]: pgmap v7497: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:26.296 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:26.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:26.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:26.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4287309507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4287309507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4287309507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:27.762 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:08:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:27 smithi022 ceph-mon[40250]: pgmap v7498: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:27 smithi022 ceph-mon[31360]: pgmap v7498: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:27 smithi122 ceph-mon[32255]: pgmap v7498: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:29 smithi022 ceph-mon[31360]: pgmap v7499: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:29 smithi022 ceph-mon[40250]: pgmap v7499: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:29 smithi122 ceph-mon[32255]: pgmap v7499: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:30] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:08:31.666 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:31 smithi022 ceph-mon[40250]: pgmap v7500: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:31 smithi022 ceph-mon[31360]: pgmap v7500: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:32.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:32.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:32.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:31 smithi122 ceph-mon[32255]: pgmap v7500: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:08:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:08:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3538249798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3538249798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3538249798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[31360]: pgmap v7501: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:08:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:08:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:08:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:08:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[40250]: pgmap v7501: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:08:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:08:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:08:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:08:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:33 smithi122 ceph-mon[32255]: pgmap v7501: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:08:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:08:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:08:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:08:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:35 smithi022 ceph-mon[31360]: pgmap v7502: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:35 smithi022 ceph-mon[40250]: pgmap v7502: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:36.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:08:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:08:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:35 smithi122 ceph-mon[32255]: pgmap v7502: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:37.036 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:37.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:37.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:37.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:37.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:37.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:08:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:37 smithi122 ceph-mon[32255]: pgmap v7503: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2199427101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:37 smithi022 ceph-mon[31360]: pgmap v7503: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2199427101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:37 smithi022 ceph-mon[40250]: pgmap v7503: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2199427101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:39 smithi122 ceph-mon[32255]: pgmap v7504: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:39 smithi022 ceph-mon[31360]: pgmap v7504: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:39 smithi022 ceph-mon[40250]: pgmap v7504: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:40] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T22:08:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:41 smithi122 ceph-mon[32255]: pgmap v7505: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:41 smithi022 ceph-mon[31360]: pgmap v7505: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:41 smithi022 ceph-mon[40250]: pgmap v7505: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:42.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:08:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:08:42.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:42.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:42.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2923173631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2923173631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2923173631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:43 smithi122 ceph-mon[32255]: pgmap v7506: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:43 smithi022 ceph-mon[31360]: pgmap v7506: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:43 smithi022 ceph-mon[40250]: pgmap v7506: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:45 smithi122 ceph-mon[32255]: pgmap v7507: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:46.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:08:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:08:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:45 smithi022 ceph-mon[31360]: pgmap v7507: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:45 smithi022 ceph-mon[40250]: pgmap v7507: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:47.775 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:47.776 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:08:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:47 smithi022 ceph-mon[40250]: pgmap v7508: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:47 smithi022 ceph-mon[31360]: pgmap v7508: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:48.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:48.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:48.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:47 smithi122 ceph-mon[32255]: pgmap v7508: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2540423679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2540423679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:49.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2540423679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:49 smithi122 ceph-mon[32255]: pgmap v7509: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:50.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:49 smithi022 ceph-mon[31360]: pgmap v7509: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:49 smithi022 ceph-mon[40250]: pgmap v7509: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:50] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:08:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:51 smithi122 ceph-mon[32255]: pgmap v7510: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:51 smithi022 ceph-mon[31360]: pgmap v7510: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:51 smithi022 ceph-mon[40250]: pgmap v7510: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:08:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:08:53.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:53.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:53.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:53.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:08:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:53 smithi122 ceph-mon[32255]: pgmap v7511: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2314672333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:53 smithi022 ceph-mon[31360]: pgmap v7511: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2314672333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:53 smithi022 ceph-mon[40250]: pgmap v7511: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2314672333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:08:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:55 smithi122 ceph-mon[32255]: pgmap v7512: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:56.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:08:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:08:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:55 smithi022 ceph-mon[31360]: pgmap v7512: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:55 smithi022 ceph-mon[40250]: pgmap v7512: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:08:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:57.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:08:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:08:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:57 smithi122 ceph-mon[32255]: pgmap v7513: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:57 smithi022 ceph-mon[31360]: pgmap v7513: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:57 smithi022 ceph-mon[40250]: pgmap v7513: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:08:58.526 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:08:58.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:08:58.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:08:58.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:59 smithi122 ceph-mon[32255]: pgmap v7514: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:08:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2354009439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:00.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:59 smithi022 ceph-mon[31360]: pgmap v7514: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:00.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:08:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2354009439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:00.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:59 smithi022 ceph-mon[40250]: pgmap v7514: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:00.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:08:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2354009439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:00] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:09:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:01 smithi122 ceph-mon[32255]: pgmap v7515: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:01 smithi022 ceph-mon[31360]: pgmap v7515: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:01 smithi022 ceph-mon[40250]: pgmap v7515: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:02.584 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:09:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:09:02.908 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:09:02.908 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:09:02.908 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:09:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:09:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:09:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:09:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:09:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:09:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:09:03.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:03 smithi122 ceph-mon[32255]: pgmap v7516: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:04.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:04.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:04.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[31360]: pgmap v7516: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:04.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[40250]: pgmap v7516: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:04.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:04.265 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:09:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4071559758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4071559758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4071559758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:05 smithi122 ceph-mon[32255]: pgmap v7517: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:06.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:09:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:09:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:05 smithi022 ceph-mon[31360]: pgmap v7517: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:05 smithi022 ceph-mon[40250]: pgmap v7517: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:09:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:07 smithi122 ceph-mon[32255]: pgmap v7518: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:07 smithi022 ceph-mon[31360]: pgmap v7518: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:07 smithi022 ceph-mon[40250]: pgmap v7518: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:09.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:09.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:09.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:09.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:10.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:10] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:09:10.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:09 smithi122 ceph-mon[32255]: pgmap v7519: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:10.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/445056729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:09 smithi022 ceph-mon[31360]: pgmap v7519: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/445056729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:09 smithi022 ceph-mon[40250]: pgmap v7519: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/445056729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:11 smithi122 ceph-mon[32255]: pgmap v7520: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:11 smithi022 ceph-mon[31360]: pgmap v7520: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:11 smithi022 ceph-mon[40250]: pgmap v7520: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:09:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:09:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:13 smithi122 ceph-mon[32255]: pgmap v7521: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:13 smithi022 ceph-mon[31360]: pgmap v7521: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:13 smithi022 ceph-mon[40250]: pgmap v7521: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:14.636 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:15.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:15.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:15.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:15 smithi122 ceph-mon[32255]: pgmap v7522: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1265481453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:16.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:09:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:09:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:15 smithi022 ceph-mon[31360]: pgmap v7522: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:16.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1265481453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:15 smithi022 ceph-mon[40250]: pgmap v7522: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:16.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1265481453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:09:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:17 smithi022 ceph-mon[31360]: pgmap v7523: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:17 smithi022 ceph-mon[40250]: pgmap v7523: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:17 smithi122 ceph-mon[32255]: pgmap v7523: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:20.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:19 smithi022 ceph-mon[31360]: pgmap v7524: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:19 smithi022 ceph-mon[40250]: pgmap v7524: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:20.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:20.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:20.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:20] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:09:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:19 smithi122 ceph-mon[32255]: pgmap v7524: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2912908346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:21.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2912908346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2912908346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:21 smithi022 ceph-mon[31360]: pgmap v7525: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:21 smithi022 ceph-mon[40250]: pgmap v7525: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:21 smithi122 ceph-mon[32255]: pgmap v7525: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:09:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:09:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:23 smithi022 ceph-mon[31360]: pgmap v7526: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:24.312 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:23 smithi022 ceph-mon[40250]: pgmap v7526: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:23 smithi122 ceph-mon[32255]: pgmap v7526: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:25.378 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:25.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:25 smithi022 ceph-mon[31360]: pgmap v7527: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2500971754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:25 smithi022 ceph-mon[40250]: pgmap v7527: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2500971754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:26.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:09:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:09:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:25 smithi122 ceph-mon[32255]: pgmap v7527: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2500971754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:27.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:09:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:27 smithi022 ceph-mon[31360]: pgmap v7528: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:27 smithi022 ceph-mon[40250]: pgmap v7528: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:27 smithi122 ceph-mon[32255]: pgmap v7528: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:29 smithi022 ceph-mon[31360]: pgmap v7529: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:29 smithi022 ceph-mon[40250]: pgmap v7529: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:29 smithi122 ceph-mon[32255]: pgmap v7529: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:30] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:09:30.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:31.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:31.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:31 smithi022 ceph-mon[31360]: pgmap v7530: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/547361647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:31 smithi022 ceph-mon[40250]: pgmap v7530: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/547361647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:31 smithi122 ceph-mon[32255]: pgmap v7530: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/547361647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:09:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:09:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[31360]: pgmap v7531: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:09:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:09:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:09:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:09:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[40250]: pgmap v7531: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:09:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:09:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:09:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:09:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:33 smithi122 ceph-mon[32255]: pgmap v7531: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:09:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:09:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:09:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:09:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:35 smithi022 ceph-mon[31360]: pgmap v7532: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:35 smithi022 ceph-mon[40250]: pgmap v7532: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:36.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:09:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:09:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:35 smithi122 ceph-mon[32255]: pgmap v7532: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:36.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:36.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:36.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2816179718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:37.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:37.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2816179718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2816179718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:09:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:37 smithi022 ceph-mon[31360]: pgmap v7533: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:37 smithi022 ceph-mon[40250]: pgmap v7533: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:37 smithi122 ceph-mon[32255]: pgmap v7533: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:39 smithi022 ceph-mon[31360]: pgmap v7534: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:39 smithi022 ceph-mon[40250]: pgmap v7534: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:39 smithi122 ceph-mon[32255]: pgmap v7534: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:40] "GET /metrics HTTP/1.1" 200 202369 "" "Prometheus/2.18.1" 2021-12-15T22:09:41.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:41.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:41.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:41.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:41 smithi022 ceph-mon[31360]: pgmap v7535: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1565271885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:41 smithi022 ceph-mon[40250]: pgmap v7535: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1565271885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:41 smithi122 ceph-mon[32255]: pgmap v7535: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1565271885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:09:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:09:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:43 smithi022 ceph-mon[31360]: pgmap v7536: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:43 smithi022 ceph-mon[40250]: pgmap v7536: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:43 smithi122 ceph-mon[32255]: pgmap v7536: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:45 smithi022 ceph-mon[31360]: pgmap v7537: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:45 smithi022 ceph-mon[40250]: pgmap v7537: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:45 smithi122 ceph-mon[32255]: pgmap v7537: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:46.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:09:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:09:46.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:47.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:47.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:47.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:47.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:47.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:09:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:47 smithi022 ceph-mon[31360]: pgmap v7538: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1829640622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:47 smithi022 ceph-mon[40250]: pgmap v7538: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1829640622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:47 smithi122 ceph-mon[32255]: pgmap v7538: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1829640622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:49 smithi022 ceph-mon[31360]: pgmap v7539: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:49 smithi022 ceph-mon[40250]: pgmap v7539: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:49 smithi122 ceph-mon[32255]: pgmap v7539: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:50] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:09:52.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:52 smithi022 ceph-mon[31360]: pgmap v7540: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:52 smithi022 ceph-mon[40250]: pgmap v7540: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:52 smithi122 ceph-mon[32255]: pgmap v7540: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:52.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:52.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:52.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:09:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:09:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3756311643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3756311643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3756311643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:54 smithi022 ceph-mon[31360]: pgmap v7541: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:54 smithi022 ceph-mon[40250]: pgmap v7541: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:54 smithi122 ceph-mon[32255]: pgmap v7541: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:56 smithi022 ceph-mon[31360]: pgmap v7542: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:56 smithi022 ceph-mon[40250]: pgmap v7542: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:56 smithi122 ceph-mon[32255]: pgmap v7542: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:09:56.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:09:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:09:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:57.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:09:57.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:09:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:09:57.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:09:57.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:09:57.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:09:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:58 smithi022 ceph-mon[31360]: pgmap v7543: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:09:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2846447478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:58 smithi022 ceph-mon[40250]: pgmap v7543: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:09:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2846447478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:09:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:58 smithi122 ceph-mon[32255]: pgmap v7543: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:09:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:09:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2846447478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:00 smithi022 ceph-mon[31360]: pgmap v7544: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:00.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T22:10:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:00 smithi022 ceph-mon[40250]: pgmap v7544: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T22:10:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:00] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:10:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:00 smithi122 ceph-mon[32255]: pgmap v7544: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T22:10:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:02 smithi122 ceph-mon[32255]: pgmap v7545: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:02 smithi022 ceph-mon[31360]: pgmap v7545: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:02 smithi022 ceph-mon[40250]: pgmap v7545: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:10:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:10:02.973 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:03.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:03.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:03.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:04 smithi122 ceph-mon[32255]: pgmap v7546: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2419827105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:10:04.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:10:04.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:10:04.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:10:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[31360]: pgmap v7546: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2419827105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:10:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:10:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:10:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:10:04.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[40250]: pgmap v7546: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:04.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2419827105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:04.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:10:04.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:10:04.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:10:04.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:10:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:06 smithi022 ceph-mon[31360]: pgmap v7547: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:06 smithi022 ceph-mon[40250]: pgmap v7547: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:06.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:10:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:10:06.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:06 smithi122 ceph-mon[32255]: pgmap v7547: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:10:08.349 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:08 smithi022 ceph-mon[31360]: pgmap v7548: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:08.349 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:08 smithi022 ceph-mon[40250]: pgmap v7548: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:08.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:08 smithi122 ceph-mon[32255]: pgmap v7548: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:08.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:08.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:08.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:09 smithi122 ceph-mon[32255]: pgmap v7549: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/471929387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:09 smithi022 ceph-mon[31360]: pgmap v7549: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/471929387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:09 smithi022 ceph-mon[40250]: pgmap v7549: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/471929387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:10] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T22:10:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:11 smithi122 ceph-mon[32255]: pgmap v7550: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:11 smithi022 ceph-mon[31360]: pgmap v7550: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:11 smithi022 ceph-mon[40250]: pgmap v7550: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:10:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:10:13.718 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:13 smithi122 ceph-mon[32255]: pgmap v7551: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:13 smithi022 ceph-mon[31360]: pgmap v7551: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:13 smithi022 ceph-mon[40250]: pgmap v7551: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:14.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1928457035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1928457035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1928457035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:15 smithi122 ceph-mon[32255]: pgmap v7552: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:15 smithi022 ceph-mon[31360]: pgmap v7552: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:15 smithi022 ceph-mon[40250]: pgmap v7552: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:16.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:10:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:10:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:17.689 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:10:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:17 smithi022 ceph-mon[31360]: pgmap v7553: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:17 smithi022 ceph-mon[40250]: pgmap v7553: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:17 smithi122 ceph-mon[32255]: pgmap v7553: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:19.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:19.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:19.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:19.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:19 smithi022 ceph-mon[31360]: pgmap v7554: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:19.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2374225848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:19 smithi022 ceph-mon[40250]: pgmap v7554: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2374225848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:19 smithi122 ceph-mon[32255]: pgmap v7554: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2374225848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:20] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T22:10:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:21 smithi022 ceph-mon[31360]: pgmap v7555: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:21 smithi022 ceph-mon[40250]: pgmap v7555: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:21 smithi122 ceph-mon[32255]: pgmap v7555: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:10:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:10:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:23 smithi022 ceph-mon[31360]: pgmap v7556: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:23 smithi022 ceph-mon[40250]: pgmap v7556: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:23 smithi122 ceph-mon[32255]: pgmap v7556: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:24.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:24.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:26.003 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:25 smithi122 ceph-mon[32255]: pgmap v7557: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:26.003 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3728287958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:25 smithi022 ceph-mon[31360]: pgmap v7557: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3728287958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:25 smithi022 ceph-mon[40250]: pgmap v7557: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3728287958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:26.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:10:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:10:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:27.704 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:10:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:27 smithi022 ceph-mon[31360]: pgmap v7558: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:27 smithi022 ceph-mon[40250]: pgmap v7558: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:27 smithi122 ceph-mon[32255]: pgmap v7558: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:29.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:29 smithi022 ceph-mon[31360]: pgmap v7559: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:29 smithi022 ceph-mon[40250]: pgmap v7559: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:29 smithi122 ceph-mon[32255]: pgmap v7559: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:30.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:30.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:30.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:30] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T22:10:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2725038061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:31.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2725038061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2725038061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:31 smithi022 ceph-mon[31360]: pgmap v7560: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:31 smithi022 ceph-mon[40250]: pgmap v7560: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:31 smithi122 ceph-mon[32255]: pgmap v7560: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:10:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:10:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[31360]: pgmap v7561: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:10:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:10:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:10:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:10:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[40250]: pgmap v7561: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:34.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:10:34.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:10:34.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:10:34.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:10:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:33 smithi122 ceph-mon[32255]: pgmap v7561: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:10:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:10:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:10:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:10:35.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:35.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:35.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:35.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:36.007 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:35 smithi122 ceph-mon[32255]: pgmap v7562: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:36.007 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1339867877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:35 smithi022 ceph-mon[31360]: pgmap v7562: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:36.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1339867877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:35 smithi022 ceph-mon[40250]: pgmap v7562: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:36.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1339867877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:36.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:10:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:10:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:37.725 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:10:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:37 smithi022 ceph-mon[31360]: pgmap v7563: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:37 smithi022 ceph-mon[40250]: pgmap v7563: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:37 smithi122 ceph-mon[32255]: pgmap v7563: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:39 smithi022 ceph-mon[31360]: pgmap v7564: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:39 smithi022 ceph-mon[40250]: pgmap v7564: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:39 smithi122 ceph-mon[32255]: pgmap v7564: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:40] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:10:40.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:40.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:40.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:40.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:41 smithi022 ceph-mon[31360]: pgmap v7565: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/739654957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:41 smithi022 ceph-mon[40250]: pgmap v7565: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/739654957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:41 smithi122 ceph-mon[32255]: pgmap v7565: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/739654957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:10:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:10:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:43 smithi022 ceph-mon[31360]: pgmap v7566: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:43 smithi022 ceph-mon[40250]: pgmap v7566: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:43 smithi122 ceph-mon[32255]: pgmap v7566: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:45.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:46.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:45 smithi122 ceph-mon[32255]: pgmap v7567: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:45 smithi022 ceph-mon[31360]: pgmap v7567: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:45 smithi022 ceph-mon[40250]: pgmap v7567: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:46.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:46.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:46.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:46.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:10:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:10:46.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:46.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/186045807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:47.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/186045807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/186045807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:47.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:10:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:47 smithi022 ceph-mon[31360]: pgmap v7568: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:47 smithi022 ceph-mon[40250]: pgmap v7568: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:47 smithi122 ceph-mon[32255]: pgmap v7568: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:49 smithi022 ceph-mon[31360]: pgmap v7569: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:49 smithi022 ceph-mon[40250]: pgmap v7569: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:49 smithi122 ceph-mon[32255]: pgmap v7569: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:50] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:10:51.310 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:51.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:51.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:51.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:51 smithi022 ceph-mon[31360]: pgmap v7570: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1619205150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:51 smithi022 ceph-mon[40250]: pgmap v7570: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1619205150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:51 smithi122 ceph-mon[32255]: pgmap v7570: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1619205150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:10:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:10:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:53 smithi022 ceph-mon[31360]: pgmap v7571: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:53 smithi022 ceph-mon[40250]: pgmap v7571: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:53 smithi122 ceph-mon[32255]: pgmap v7571: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:56.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:55 smithi122 ceph-mon[32255]: pgmap v7572: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:55 smithi022 ceph-mon[31360]: pgmap v7572: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:55 smithi022 ceph-mon[40250]: pgmap v7572: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:10:56.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:10:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:10:56.677 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:10:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:57.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:10:57.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:10:57.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:10:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:10:57.759 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:10:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:57 smithi022 ceph-mon[31360]: pgmap v7573: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1053680941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:58.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:57 smithi022 ceph-mon[40250]: pgmap v7573: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:58.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1053680941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:10:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:57 smithi122 ceph-mon[32255]: pgmap v7573: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:10:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1053680941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:10:59 smithi022 ceph-mon[31360]: pgmap v7574: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:10:59 smithi022 ceph-mon[40250]: pgmap v7574: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:10:59 smithi122 ceph-mon[32255]: pgmap v7574: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:00] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:11:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:01 smithi022 ceph-mon[31360]: pgmap v7575: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:01 smithi022 ceph-mon[40250]: pgmap v7575: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:02.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:01 smithi122 ceph-mon[32255]: pgmap v7575: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:02.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:02.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:02.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:11:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:11:03.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1004757334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1004757334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1004757334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:03 smithi022 ceph-mon[31360]: pgmap v7576: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:03 smithi022 ceph-mon[40250]: pgmap v7576: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:03 smithi122 ceph-mon[32255]: pgmap v7576: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:11:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:11:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:11:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:11:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:11:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:11:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:11:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:11:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:11:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:11:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:11:05.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:11:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:05 smithi022 ceph-mon[31360]: pgmap v7577: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:05 smithi022 ceph-mon[40250]: pgmap v7577: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:06.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:05 smithi122 ceph-mon[32255]: pgmap v7577: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:06.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:11:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:11:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:07.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:11:07.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:07.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:07.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:07 smithi022 ceph-mon[31360]: pgmap v7578: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:07 smithi022 ceph-mon[40250]: pgmap v7578: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:07 smithi122 ceph-mon[32255]: pgmap v7578: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/615240779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/615240779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/615240779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:09 smithi122 ceph-mon[32255]: pgmap v7579: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:09 smithi022 ceph-mon[31360]: pgmap v7579: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:09 smithi022 ceph-mon[40250]: pgmap v7579: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:10] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:11:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:11 smithi122 ceph-mon[32255]: pgmap v7580: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:11 smithi022 ceph-mon[31360]: pgmap v7580: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:11 smithi022 ceph-mon[40250]: pgmap v7580: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:11:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:11:12.793 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:13.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:13.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:13.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:13 smithi122 ceph-mon[32255]: pgmap v7581: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1475383332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:13 smithi022 ceph-mon[31360]: pgmap v7581: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1475383332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:13 smithi022 ceph-mon[40250]: pgmap v7581: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1475383332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:16.046 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:15 smithi122 ceph-mon[32255]: pgmap v7582: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:15 smithi022 ceph-mon[31360]: pgmap v7582: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:15 smithi022 ceph-mon[40250]: pgmap v7582: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:16.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:11:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:11:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:17.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:11:18.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:18.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:17 smithi022 ceph-mon[31360]: pgmap v7583: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:18.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:17 smithi022 ceph-mon[40250]: pgmap v7583: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:17 smithi122 ceph-mon[32255]: pgmap v7583: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:18.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:18.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:18.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/534175318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/534175318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/534175318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:19 smithi122 ceph-mon[32255]: pgmap v7584: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:19 smithi022 ceph-mon[31360]: pgmap v7584: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:19 smithi022 ceph-mon[40250]: pgmap v7584: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:20] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:11:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:21 smithi122 ceph-mon[32255]: pgmap v7585: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:21 smithi022 ceph-mon[31360]: pgmap v7585: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:21 smithi022 ceph-mon[40250]: pgmap v7585: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:11:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:11:23.532 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:23.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:23.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:23.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:23 smithi122 ceph-mon[32255]: pgmap v7586: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:23 smithi022 ceph-mon[31360]: pgmap v7586: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:23 smithi022 ceph-mon[40250]: pgmap v7586: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1650059815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1650059815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1650059815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:25 smithi122 ceph-mon[32255]: pgmap v7587: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:26.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:11:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:11:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:25 smithi022 ceph-mon[31360]: pgmap v7587: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:25 smithi022 ceph-mon[40250]: pgmap v7587: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:11:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:27 smithi122 ceph-mon[32255]: pgmap v7588: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:27 smithi022 ceph-mon[31360]: pgmap v7588: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:27 smithi022 ceph-mon[40250]: pgmap v7588: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:28.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:29.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:29 smithi122 ceph-mon[32255]: pgmap v7589: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/264716283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:29 smithi022 ceph-mon[31360]: pgmap v7589: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/264716283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:29 smithi022 ceph-mon[40250]: pgmap v7589: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/264716283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:30] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:11:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:31 smithi122 ceph-mon[32255]: pgmap v7590: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:32.318 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:31 smithi022 ceph-mon[31360]: pgmap v7590: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:32.319 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:31 smithi022 ceph-mon[40250]: pgmap v7590: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:11:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:11:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:33 smithi122 ceph-mon[32255]: pgmap v7591: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:11:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:11:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:11:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:11:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:34.272 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[40250]: pgmap v7591: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:34.272 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:11:34.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:11:34.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:11:34.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:11:34.274 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[31360]: pgmap v7591: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:34.274 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:11:34.274 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:11:34.275 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:11:34.275 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:11:34.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:34.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:34.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1646379280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1646379280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1646379280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:35 smithi122 ceph-mon[32255]: pgmap v7592: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:36.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:11:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:11:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:35 smithi022 ceph-mon[31360]: pgmap v7592: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:35 smithi022 ceph-mon[40250]: pgmap v7592: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:11:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:37 smithi122 ceph-mon[32255]: pgmap v7593: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:37 smithi022 ceph-mon[31360]: pgmap v7593: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:37 smithi022 ceph-mon[40250]: pgmap v7593: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:39.643 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:40.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:40.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:40.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:40.013 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:39 smithi022 ceph-mon[31360]: pgmap v7594: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:40.014 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:39 smithi022 ceph-mon[40250]: pgmap v7594: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:39 smithi122 ceph-mon[32255]: pgmap v7594: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:40] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:11:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/847131330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/847131330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/847131330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:41 smithi122 ceph-mon[32255]: pgmap v7595: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:41 smithi022 ceph-mon[31360]: pgmap v7595: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:41 smithi022 ceph-mon[40250]: pgmap v7595: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:11:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:11:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:43 smithi122 ceph-mon[32255]: pgmap v7596: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:43 smithi022 ceph-mon[31360]: pgmap v7596: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:43 smithi022 ceph-mon[40250]: pgmap v7596: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:45.014 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:45.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:45.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:45.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:45 smithi122 ceph-mon[32255]: pgmap v7597: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3665213376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:46.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:11:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:11:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:45 smithi022 ceph-mon[31360]: pgmap v7597: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3665213376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:45 smithi022 ceph-mon[40250]: pgmap v7597: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3665213376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:11:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:47 smithi122 ceph-mon[32255]: pgmap v7598: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:47 smithi022 ceph-mon[31360]: pgmap v7598: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:47 smithi022 ceph-mon[40250]: pgmap v7598: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:49 smithi122 ceph-mon[32255]: pgmap v7599: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:49 smithi022 ceph-mon[31360]: pgmap v7599: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:49 smithi022 ceph-mon[40250]: pgmap v7599: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:50.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:50] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:11:50.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:50.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:50.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1394163137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:51.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1394163137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:51.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1394163137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:51 smithi122 ceph-mon[32255]: pgmap v7600: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:51 smithi022 ceph-mon[31360]: pgmap v7600: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:51 smithi022 ceph-mon[40250]: pgmap v7600: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:11:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:11:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:53 smithi122 ceph-mon[32255]: pgmap v7601: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:53 smithi022 ceph-mon[31360]: pgmap v7601: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:53 smithi022 ceph-mon[40250]: pgmap v7601: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:55.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:11:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:55 smithi022 ceph-mon[31360]: pgmap v7602: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:55 smithi022 ceph-mon[40250]: pgmap v7602: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:56.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:11:56.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:11:56.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:11:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:55 smithi122 ceph-mon[32255]: pgmap v7602: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:11:56.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:11:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:11:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3998367030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3998367030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:57.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:11:57.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3998367030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:11:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:11:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:57 smithi122 ceph-mon[32255]: pgmap v7603: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:57 smithi022 ceph-mon[31360]: pgmap v7603: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:11:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:57 smithi022 ceph-mon[40250]: pgmap v7603: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:11:59 smithi122 ceph-mon[32255]: pgmap v7604: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:11:59 smithi022 ceph-mon[31360]: pgmap v7604: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:11:59 smithi022 ceph-mon[40250]: pgmap v7604: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:00] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:12:01.121 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:01.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:01.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:01.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:01 smithi122 ceph-mon[32255]: pgmap v7605: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/338518826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:01 smithi022 ceph-mon[31360]: pgmap v7605: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/338518826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:01 smithi022 ceph-mon[40250]: pgmap v7605: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/338518826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:12:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:12:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:03 smithi122 ceph-mon[32255]: pgmap v7606: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:03 smithi022 ceph-mon[40250]: pgmap v7606: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:03 smithi022 ceph-mon[31360]: pgmap v7606: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:12:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:12:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:12:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:12:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:12:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:12:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:12:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:12:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:12:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:12:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:12:05.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:12:06.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:12:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:12:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:05 smithi122 ceph-mon[32255]: pgmap v7607: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:05 smithi022 ceph-mon[31360]: pgmap v7607: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:05 smithi022 ceph-mon[40250]: pgmap v7607: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:06.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:06.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:06.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3570090155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3570090155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3570090155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:12:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:07 smithi122 ceph-mon[32255]: pgmap v7608: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:07 smithi022 ceph-mon[31360]: pgmap v7608: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:07 smithi022 ceph-mon[40250]: pgmap v7608: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:09 smithi122 ceph-mon[32255]: pgmap v7609: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:09 smithi022 ceph-mon[31360]: pgmap v7609: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:09 smithi022 ceph-mon[40250]: pgmap v7609: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:12:11.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:11 smithi122 ceph-mon[32255]: pgmap v7610: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:12.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:12.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:12.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:12.236 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:11 smithi022 ceph-mon[31360]: pgmap v7610: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:12.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:11 smithi022 ceph-mon[40250]: pgmap v7610: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:12:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:12:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1900498046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1900498046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1900498046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:13 smithi122 ceph-mon[32255]: pgmap v7611: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:13 smithi022 ceph-mon[31360]: pgmap v7611: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:13 smithi022 ceph-mon[40250]: pgmap v7611: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:15 smithi122 ceph-mon[32255]: pgmap v7612: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:16.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:12:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:12:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:15 smithi022 ceph-mon[31360]: pgmap v7612: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:15 smithi022 ceph-mon[40250]: pgmap v7612: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:17.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:17.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:17.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:12:17.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:17.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:17.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:17 smithi122 ceph-mon[32255]: pgmap v7613: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4217664351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:17 smithi022 ceph-mon[40250]: pgmap v7613: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4217664351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:17 smithi022 ceph-mon[31360]: pgmap v7613: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4217664351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:20.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:19 smithi122 ceph-mon[32255]: pgmap v7614: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:20.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:12:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:19 smithi022 ceph-mon[31360]: pgmap v7614: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:19 smithi022 ceph-mon[40250]: pgmap v7614: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:21 smithi022 ceph-mon[31360]: pgmap v7615: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:21 smithi022 ceph-mon[40250]: pgmap v7615: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:21 smithi122 ceph-mon[32255]: pgmap v7615: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:22.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:12:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:12:22.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:22.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:22.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:23 smithi022 ceph-mon[40250]: pgmap v7616: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2659689483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:23 smithi022 ceph-mon[31360]: pgmap v7616: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:24.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2659689483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:23 smithi122 ceph-mon[32255]: pgmap v7616: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2659689483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:25 smithi022 ceph-mon[31360]: pgmap v7617: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:25 smithi022 ceph-mon[40250]: pgmap v7617: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:25 smithi122 ceph-mon[32255]: pgmap v7617: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:26.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:12:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:12:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:27.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:12:27.979 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:27 smithi022 ceph-mon[31360]: pgmap v7618: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:27 smithi022 ceph-mon[40250]: pgmap v7618: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:28.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:28.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:28.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:27 smithi122 ceph-mon[32255]: pgmap v7618: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:29.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3490318501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:29.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3490318501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3490318501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:29 smithi022 ceph-mon[31360]: pgmap v7619: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:29 smithi022 ceph-mon[40250]: pgmap v7619: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:29 smithi122 ceph-mon[32255]: pgmap v7619: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:12:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:31 smithi022 ceph-mon[31360]: pgmap v7620: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:31 smithi022 ceph-mon[40250]: pgmap v7620: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:31 smithi122 ceph-mon[32255]: pgmap v7620: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:12:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:12:33.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:33.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:33.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:33.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[31360]: pgmap v7621: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:12:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:12:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:12:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:12:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508889381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[40250]: pgmap v7621: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:12:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:12:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:12:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:12:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508889381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:33 smithi122 ceph-mon[32255]: pgmap v7621: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:12:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:12:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:12:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:12:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508889381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:35 smithi022 ceph-mon[31360]: pgmap v7622: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:35 smithi022 ceph-mon[40250]: pgmap v7622: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:36.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:12:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:12:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:35 smithi122 ceph-mon[32255]: pgmap v7622: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:12:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:37 smithi022 ceph-mon[31360]: pgmap v7623: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:37 smithi022 ceph-mon[40250]: pgmap v7623: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:37 smithi122 ceph-mon[32255]: pgmap v7623: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:38.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:39.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:39.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:39.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:39 smithi022 ceph-mon[31360]: pgmap v7624: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2971119251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:39 smithi022 ceph-mon[40250]: pgmap v7624: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2971119251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:39 smithi122 ceph-mon[32255]: pgmap v7624: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2971119251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:40] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:12:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:41 smithi022 ceph-mon[31360]: pgmap v7625: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:41 smithi022 ceph-mon[40250]: pgmap v7625: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:41 smithi122 ceph-mon[32255]: pgmap v7625: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:12:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:12:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:43 smithi022 ceph-mon[31360]: pgmap v7626: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:43 smithi022 ceph-mon[40250]: pgmap v7626: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:43 smithi122 ceph-mon[32255]: pgmap v7626: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:44.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:44.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:44.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2551634837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2551634837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2551634837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:45 smithi022 ceph-mon[31360]: pgmap v7627: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:45 smithi022 ceph-mon[40250]: pgmap v7627: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:45 smithi122 ceph-mon[32255]: pgmap v7627: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:46.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:12:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:12:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:12:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:47 smithi022 ceph-mon[31360]: pgmap v7628: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:47 smithi022 ceph-mon[40250]: pgmap v7628: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:47 smithi122 ceph-mon[32255]: pgmap v7628: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:49.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:49.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:49.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:49.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:49 smithi022 ceph-mon[31360]: pgmap v7629: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/353795902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:49 smithi022 ceph-mon[40250]: pgmap v7629: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/353795902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:12:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:49 smithi122 ceph-mon[32255]: pgmap v7629: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/353795902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:51 smithi022 ceph-mon[31360]: pgmap v7630: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:51 smithi022 ceph-mon[40250]: pgmap v7630: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:51 smithi122 ceph-mon[32255]: pgmap v7630: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:12:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:12:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:53 smithi022 ceph-mon[31360]: pgmap v7631: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:53 smithi022 ceph-mon[40250]: pgmap v7631: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:53 smithi122 ceph-mon[32255]: pgmap v7631: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:54.834 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:12:55.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:12:55.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:12:55.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:12:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:55 smithi022 ceph-mon[31360]: pgmap v7632: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3280693780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:55 smithi022 ceph-mon[40250]: pgmap v7632: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3280693780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:56.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:12:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:12:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:56 smithi122 ceph-mon[32255]: pgmap v7632: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:12:56.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3280693780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:12:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:57.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:57.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:12:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:12:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:12:58 smithi022 ceph-mon[31360]: pgmap v7633: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:12:58 smithi022 ceph-mon[40250]: pgmap v7633: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:12:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:12:58 smithi122 ceph-mon[32255]: pgmap v7633: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:00.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:00 smithi022 ceph-mon[31360]: pgmap v7634: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:00 smithi022 ceph-mon[40250]: pgmap v7634: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:13:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:00 smithi122 ceph-mon[32255]: pgmap v7634: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4078415935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4078415935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4078415935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:02 smithi022 ceph-mon[31360]: pgmap v7635: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:02 smithi022 ceph-mon[40250]: pgmap v7635: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:02 smithi122 ceph-mon[32255]: pgmap v7635: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:13:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:13:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:04 smithi022 ceph-mon[31360]: pgmap v7636: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:04 smithi022 ceph-mon[40250]: pgmap v7636: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:04 smithi122 ceph-mon[32255]: pgmap v7636: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:13:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:13:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:13:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:13:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:13:05.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:13:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:13:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:13:05.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:13:05.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:05.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:05.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:05.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:06 smithi022 ceph-mon[31360]: pgmap v7637: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2216485686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:06 smithi022 ceph-mon[40250]: pgmap v7637: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2216485686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:06 smithi122 ceph-mon[32255]: pgmap v7637: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2216485686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:06.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:13:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:13:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:13:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:08 smithi022 ceph-mon[31360]: pgmap v7638: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:08 smithi022 ceph-mon[40250]: pgmap v7638: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:08 smithi122 ceph-mon[32255]: pgmap v7638: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: pgmap v7639: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:09.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3208M 2021-12-15T22:13:09.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: pgmap v7639: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:09.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3208M 2021-12-15T22:13:09.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:09.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:09.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: pgmap v7639: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:09.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:13:09.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3208M 2021-12-15T22:13:09.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:10] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:13:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:10.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:13:10.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:11.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:11.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:11 smithi122 ceph-mon[32255]: pgmap v7640: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2408966686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:11 smithi022 ceph-mon[31360]: pgmap v7640: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2408966686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:11 smithi022 ceph-mon[40250]: pgmap v7640: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:11.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2408966686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:13:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:13:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:13 smithi022 ceph-mon[31360]: pgmap v7641: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:13 smithi022 ceph-mon[40250]: pgmap v7641: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:13 smithi122 ceph-mon[32255]: pgmap v7641: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:15 smithi022 ceph-mon[31360]: pgmap v7642: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:15 smithi022 ceph-mon[40250]: pgmap v7642: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:16.134 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:15 smithi122 ceph-mon[32255]: pgmap v7642: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:16.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:13:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:13:16.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:16.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:16.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2135050760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2135050760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2135050760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:17.731 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:13:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:17 smithi022 ceph-mon[31360]: pgmap v7643: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:17 smithi022 ceph-mon[40250]: pgmap v7643: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:17 smithi122 ceph-mon[32255]: pgmap v7643: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:19 smithi022 ceph-mon[31360]: pgmap v7644: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:19 smithi022 ceph-mon[40250]: pgmap v7644: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:19 smithi122 ceph-mon[32255]: pgmap v7644: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:20] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:13:21.731 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:21 smithi022 ceph-mon[31360]: pgmap v7645: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:21 smithi022 ceph-mon[40250]: pgmap v7645: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:22.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:22.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:22.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:21 smithi122 ceph-mon[32255]: pgmap v7645: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:13:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:13:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3869131463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:23.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3869131463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3869131463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:23 smithi022 ceph-mon[31360]: pgmap v7646: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:23 smithi022 ceph-mon[40250]: pgmap v7646: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:23 smithi122 ceph-mon[32255]: pgmap v7646: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:25 smithi022 ceph-mon[31360]: pgmap v7647: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:25 smithi022 ceph-mon[40250]: pgmap v7647: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:26.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:25 smithi122 ceph-mon[32255]: pgmap v7647: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:26.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:13:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:13:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:27.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:27.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:27.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:27.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:27.751 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:13:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:27 smithi022 ceph-mon[31360]: pgmap v7648: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:28.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3667520143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:27 smithi022 ceph-mon[40250]: pgmap v7648: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:28.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3667520143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:27 smithi122 ceph-mon[32255]: pgmap v7648: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3667520143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:29 smithi022 ceph-mon[31360]: pgmap v7649: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:29 smithi022 ceph-mon[40250]: pgmap v7649: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:29 smithi122 ceph-mon[32255]: pgmap v7649: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:30] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:13:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:31 smithi022 ceph-mon[31360]: pgmap v7650: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:31 smithi022 ceph-mon[40250]: pgmap v7650: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:31 smithi122 ceph-mon[32255]: pgmap v7650: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:32.472 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:13:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:13:32.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:32.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:32.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[40250]: pgmap v7651: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1566281907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:13:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:13:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:13:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:13:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[31360]: pgmap v7651: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1566281907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:13:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:13:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:13:34.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:13:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:33 smithi122 ceph-mon[32255]: pgmap v7651: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1566281907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:13:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:13:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:13:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:13:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:35 smithi022 ceph-mon[40250]: pgmap v7652: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:36.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:35 smithi022 ceph-mon[31360]: pgmap v7652: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:36.151 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:35 smithi122 ceph-mon[32255]: pgmap v7652: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:36.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:13:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:13:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:37.768 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:13:37.844 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:37 smithi022 ceph-mon[40250]: pgmap v7653: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:37 smithi022 ceph-mon[31360]: pgmap v7653: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:37 smithi122 ceph-mon[32255]: pgmap v7653: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:38.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:38.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:38.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2882907483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2882907483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:39.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2882907483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:39 smithi022 ceph-mon[31360]: pgmap v7654: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:39 smithi022 ceph-mon[40250]: pgmap v7654: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:39 smithi122 ceph-mon[32255]: pgmap v7654: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:13:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:41 smithi022 ceph-mon[31360]: pgmap v7655: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:41 smithi022 ceph-mon[40250]: pgmap v7655: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:41 smithi122 ceph-mon[32255]: pgmap v7655: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:13:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:13:43.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:43.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:43.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:43.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:43 smithi022 ceph-mon[31360]: pgmap v7656: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/122843424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:43 smithi022 ceph-mon[40250]: pgmap v7656: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/122843424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:43 smithi122 ceph-mon[32255]: pgmap v7656: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/122843424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:46.158 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:45 smithi122 ceph-mon[32255]: pgmap v7657: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:45 smithi022 ceph-mon[31360]: pgmap v7657: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:45 smithi022 ceph-mon[40250]: pgmap v7657: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:46.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:13:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:13:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:13:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:47 smithi122 ceph-mon[32255]: pgmap v7658: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:47 smithi022 ceph-mon[31360]: pgmap v7658: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:47 smithi022 ceph-mon[40250]: pgmap v7658: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:48.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:48.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:48.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:48.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:49 smithi122 ceph-mon[32255]: pgmap v7659: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1652356115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:49 smithi022 ceph-mon[31360]: pgmap v7659: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1652356115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:49 smithi022 ceph-mon[40250]: pgmap v7659: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1652356115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:50] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:13:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:51 smithi122 ceph-mon[32255]: pgmap v7660: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:51 smithi022 ceph-mon[31360]: pgmap v7660: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:51 smithi022 ceph-mon[40250]: pgmap v7660: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:13:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:13:53.964 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:53 smithi122 ceph-mon[32255]: pgmap v7661: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:53 smithi022 ceph-mon[31360]: pgmap v7661: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:53 smithi022 ceph-mon[40250]: pgmap v7661: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:54.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:54.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:54.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:13:55.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1022710386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1022710386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1022710386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:13:56.160 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:55 smithi122 ceph-mon[32255]: pgmap v7662: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:55 smithi022 ceph-mon[31360]: pgmap v7662: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:55 smithi022 ceph-mon[40250]: pgmap v7662: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:13:56.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:13:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:13:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:13:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:57 smithi122 ceph-mon[32255]: pgmap v7663: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:57 smithi022 ceph-mon[31360]: pgmap v7663: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:13:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:57 smithi022 ceph-mon[40250]: pgmap v7663: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:13:59.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:13:59.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:13:59.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:13:59.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:59 smithi022 ceph-mon[40250]: pgmap v7664: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:13:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/420641233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:00.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:59 smithi022 ceph-mon[31360]: pgmap v7664: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:00.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:13:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/420641233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:59 smithi122 ceph-mon[32255]: pgmap v7664: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:13:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/420641233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:00] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:14:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:01 smithi122 ceph-mon[32255]: pgmap v7665: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:01 smithi022 ceph-mon[31360]: pgmap v7665: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:01 smithi022 ceph-mon[40250]: pgmap v7665: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:14:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:14:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:03 smithi122 ceph-mon[32255]: pgmap v7666: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:03 smithi022 ceph-mon[31360]: pgmap v7666: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:03 smithi022 ceph-mon[40250]: pgmap v7666: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:04.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:06.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:05 smithi122 ceph-mon[32255]: pgmap v7667: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:06.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2127458489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:05 smithi022 ceph-mon[31360]: pgmap v7667: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2127458489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:05 smithi022 ceph-mon[40250]: pgmap v7667: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2127458489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:06.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:14:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:14:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:14:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:07 smithi122 ceph-mon[32255]: pgmap v7668: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:07 smithi022 ceph-mon[31360]: pgmap v7668: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:07 smithi022 ceph-mon[40250]: pgmap v7668: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:10.078 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:10.079 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:09 smithi022 ceph-mon[40250]: pgmap v7669: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:10.079 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:09 smithi022 ceph-mon[31360]: pgmap v7669: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:09 smithi122 ceph-mon[32255]: pgmap v7669: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:10] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:14:10.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:10.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:10.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:14:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:14:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:14:11.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:14:11.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4053625770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:14:11.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:14:11.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:14:11.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:14:11.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4053625770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:11.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:14:11.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:14:11.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:14:11.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:14:11.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4053625770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:11 smithi122 ceph-mon[32255]: pgmap v7670: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:12.710 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:11 smithi022 ceph-mon[31360]: pgmap v7670: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:12.710 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:14:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:14:12.711 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:11 smithi022 ceph-mon[40250]: pgmap v7670: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:13 smithi122 ceph-mon[32255]: pgmap v7671: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:13 smithi022 ceph-mon[31360]: pgmap v7671: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:13 smithi022 ceph-mon[40250]: pgmap v7671: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:15.444 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:15.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:15.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:15.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:16.182 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:15 smithi122 ceph-mon[32255]: pgmap v7672: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:16.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/501727809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:15 smithi022 ceph-mon[31360]: pgmap v7672: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/501727809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:15 smithi022 ceph-mon[40250]: pgmap v7672: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/501727809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:16.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:14:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:14:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:14:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:17 smithi122 ceph-mon[32255]: pgmap v7673: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:17 smithi022 ceph-mon[31360]: pgmap v7673: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:17 smithi022 ceph-mon[40250]: pgmap v7673: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:19 smithi122 ceph-mon[32255]: pgmap v7674: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:19 smithi022 ceph-mon[31360]: pgmap v7674: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:19 smithi022 ceph-mon[40250]: pgmap v7674: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:14:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:21.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:21.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:21.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:21 smithi122 ceph-mon[32255]: pgmap v7675: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/691638549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:21 smithi022 ceph-mon[31360]: pgmap v7675: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:22.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/691638549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:21 smithi022 ceph-mon[40250]: pgmap v7675: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/691638549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:14:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:14:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:23 smithi122 ceph-mon[32255]: pgmap v7676: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:23 smithi022 ceph-mon[31360]: pgmap v7676: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:23 smithi022 ceph-mon[40250]: pgmap v7676: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:26.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:25 smithi122 ceph-mon[32255]: pgmap v7677: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:26.191 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:26.191 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:25 smithi022 ceph-mon[40250]: pgmap v7677: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:26.191 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:25 smithi022 ceph-mon[31360]: pgmap v7677: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:26.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:14:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:14:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/646557763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/646557763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/646557763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:14:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:27 smithi122 ceph-mon[32255]: pgmap v7678: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:27 smithi022 ceph-mon[31360]: pgmap v7678: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:27 smithi022 ceph-mon[40250]: pgmap v7678: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:29 smithi122 ceph-mon[32255]: pgmap v7679: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:29 smithi022 ceph-mon[31360]: pgmap v7679: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:29 smithi022 ceph-mon[40250]: pgmap v7679: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:14:31.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:31.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:31.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:31.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:31 smithi122 ceph-mon[32255]: pgmap v7680: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:31 smithi022 ceph-mon[31360]: pgmap v7680: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:31 smithi022 ceph-mon[40250]: pgmap v7680: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:14:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:14:33.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2873853243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2873853243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2873853243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:33 smithi122 ceph-mon[32255]: pgmap v7681: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:14:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:14:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:14:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:14:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[31360]: pgmap v7681: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:14:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:14:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:14:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:14:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[40250]: pgmap v7681: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:14:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:14:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:14:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:14:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:35 smithi122 ceph-mon[32255]: pgmap v7682: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:35 smithi022 ceph-mon[31360]: pgmap v7682: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:35 smithi022 ceph-mon[40250]: pgmap v7682: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:36.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:14:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:14:36.930 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:37.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:37.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:37.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:14:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:37 smithi122 ceph-mon[32255]: pgmap v7683: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1177331881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:37 smithi022 ceph-mon[31360]: pgmap v7683: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1177331881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:37 smithi022 ceph-mon[40250]: pgmap v7683: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:38.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1177331881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:39 smithi122 ceph-mon[32255]: pgmap v7684: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:39 smithi022 ceph-mon[31360]: pgmap v7684: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:39 smithi022 ceph-mon[40250]: pgmap v7684: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:40] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T22:14:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:41 smithi122 ceph-mon[32255]: pgmap v7685: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:41 smithi022 ceph-mon[31360]: pgmap v7685: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:41 smithi022 ceph-mon[40250]: pgmap v7685: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:42.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:42.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:42.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:42.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:42.916 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:14:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:14:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3675296743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3675296743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3675296743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:43 smithi122 ceph-mon[32255]: pgmap v7686: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:43 smithi022 ceph-mon[31360]: pgmap v7686: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:43 smithi022 ceph-mon[40250]: pgmap v7686: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:45 smithi122 ceph-mon[32255]: pgmap v7687: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:45 smithi022 ceph-mon[31360]: pgmap v7687: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:45 smithi022 ceph-mon[40250]: pgmap v7687: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:46.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:14:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:14:47.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:47.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:14:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:47 smithi022 ceph-mon[31360]: pgmap v7688: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:47 smithi022 ceph-mon[40250]: pgmap v7688: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:48.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:48.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:48.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:47 smithi122 ceph-mon[32255]: pgmap v7688: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:49.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3448367841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3448367841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3448367841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:49 smithi122 ceph-mon[32255]: pgmap v7689: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:49 smithi022 ceph-mon[31360]: pgmap v7689: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:49 smithi022 ceph-mon[40250]: pgmap v7689: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:14:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:51 smithi122 ceph-mon[32255]: pgmap v7690: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:51 smithi022 ceph-mon[31360]: pgmap v7690: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:51 smithi022 ceph-mon[40250]: pgmap v7690: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:14:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:14:53.042 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:53.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:53.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:53.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:14:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:53 smithi122 ceph-mon[32255]: pgmap v7691: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/850675179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:53 smithi022 ceph-mon[31360]: pgmap v7691: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:54.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/850675179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:53 smithi022 ceph-mon[40250]: pgmap v7691: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/850675179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:14:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:55 smithi122 ceph-mon[32255]: pgmap v7692: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:55 smithi022 ceph-mon[31360]: pgmap v7692: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:55 smithi022 ceph-mon[40250]: pgmap v7692: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:14:56.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:14:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:14:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:14:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:57 smithi122 ceph-mon[32255]: pgmap v7693: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:57 smithi022 ceph-mon[31360]: pgmap v7693: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:14:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:57 smithi022 ceph-mon[40250]: pgmap v7693: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:14:58.417 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:14:58.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:14:58.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:14:58.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:59 smithi022 ceph-mon[31360]: pgmap v7694: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:00.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:14:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/572002707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:59 smithi022 ceph-mon[40250]: pgmap v7694: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:14:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/572002707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:59 smithi122 ceph-mon[32255]: pgmap v7694: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:14:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/572002707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:15:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:01 smithi022 ceph-mon[31360]: pgmap v7695: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:01 smithi022 ceph-mon[40250]: pgmap v7695: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:01 smithi122 ceph-mon[32255]: pgmap v7695: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:15:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:15:03.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:03 smithi022 ceph-mon[31360]: pgmap v7696: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:03 smithi022 ceph-mon[40250]: pgmap v7696: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:04.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:04.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:04.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:03 smithi122 ceph-mon[32255]: pgmap v7696: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3015878047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3015878047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3015878047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:05 smithi022 ceph-mon[31360]: pgmap v7697: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:05 smithi022 ceph-mon[40250]: pgmap v7697: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:05 smithi122 ceph-mon[32255]: pgmap v7697: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:06.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:15:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:15:07.757 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:15:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:07 smithi022 ceph-mon[31360]: pgmap v7698: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:07 smithi022 ceph-mon[40250]: pgmap v7698: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:07 smithi122 ceph-mon[32255]: pgmap v7698: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:09.161 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:09.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:09.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:09.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:09 smithi022 ceph-mon[31360]: pgmap v7699: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4229197211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:09 smithi022 ceph-mon[40250]: pgmap v7699: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4229197211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:09 smithi122 ceph-mon[32255]: pgmap v7699: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:10.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4229197211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:15:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:15:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:15:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:15:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:15:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:15:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:15:11.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:15:11.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:15:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:15:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:15:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:15:11.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:11 smithi022 ceph-mon[31360]: pgmap v7700: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:11 smithi022 ceph-mon[40250]: pgmap v7700: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:11 smithi122 ceph-mon[32255]: pgmap v7700: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:15:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:15:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:13 smithi022 ceph-mon[31360]: pgmap v7701: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:13 smithi022 ceph-mon[40250]: pgmap v7701: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:13 smithi122 ceph-mon[32255]: pgmap v7701: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:14.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:14.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:14.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:14.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:15 smithi022 ceph-mon[31360]: pgmap v7702: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1845430915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:15 smithi022 ceph-mon[40250]: pgmap v7702: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1845430915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:15 smithi122 ceph-mon[32255]: pgmap v7702: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1845430915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:16.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:15:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:15:17.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:15:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:17 smithi022 ceph-mon[31360]: pgmap v7703: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:17 smithi022 ceph-mon[40250]: pgmap v7703: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:17 smithi122 ceph-mon[32255]: pgmap v7703: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:19.900 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:19 smithi022 ceph-mon[31360]: pgmap v7704: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:19 smithi022 ceph-mon[40250]: pgmap v7704: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:19 smithi122 ceph-mon[32255]: pgmap v7704: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:20.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:20.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:20.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:15:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1597760566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1597760566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1597760566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:21 smithi122 ceph-mon[32255]: pgmap v7705: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:21 smithi022 ceph-mon[31360]: pgmap v7705: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:21 smithi022 ceph-mon[40250]: pgmap v7705: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:15:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:15:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:23 smithi122 ceph-mon[32255]: pgmap v7706: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:23 smithi022 ceph-mon[31360]: pgmap v7706: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:23 smithi022 ceph-mon[40250]: pgmap v7706: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:25.271 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:25.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:25.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:25.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:25 smithi022 ceph-mon[31360]: pgmap v7707: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3880851295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:25 smithi022 ceph-mon[40250]: pgmap v7707: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3880851295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:25 smithi122 ceph-mon[32255]: pgmap v7707: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3880851295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:26.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:15:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:15:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:15:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:27 smithi122 ceph-mon[32255]: pgmap v7708: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:27 smithi022 ceph-mon[31360]: pgmap v7708: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:27 smithi022 ceph-mon[40250]: pgmap v7708: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:29 smithi122 ceph-mon[32255]: pgmap v7709: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:29 smithi022 ceph-mon[31360]: pgmap v7709: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:29 smithi022 ceph-mon[40250]: pgmap v7709: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:15:30.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:31.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:31.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:31.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:31 smithi122 ceph-mon[32255]: pgmap v7710: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4140919011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:31 smithi022 ceph-mon[31360]: pgmap v7710: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4140919011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:31 smithi022 ceph-mon[40250]: pgmap v7710: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4140919011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:15:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:15:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:33 smithi122 ceph-mon[32255]: pgmap v7711: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:15:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:15:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:15:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:15:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[40250]: pgmap v7711: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:15:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:15:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:15:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:15:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[31360]: pgmap v7711: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:15:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:15:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:15:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:15:36.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:35 smithi122 ceph-mon[32255]: pgmap v7712: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:35 smithi022 ceph-mon[31360]: pgmap v7712: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:35 smithi022 ceph-mon[40250]: pgmap v7712: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:36.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:36.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:15:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:15:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3954531094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3954531094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3954531094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:15:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:37 smithi122 ceph-mon[32255]: pgmap v7713: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:37 smithi022 ceph-mon[31360]: pgmap v7713: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:37 smithi022 ceph-mon[40250]: pgmap v7713: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:39 smithi122 ceph-mon[32255]: pgmap v7714: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:39 smithi022 ceph-mon[31360]: pgmap v7714: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:39 smithi022 ceph-mon[40250]: pgmap v7714: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:40] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:15:41.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:41.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:41.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:41.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:41 smithi022 ceph-mon[31360]: pgmap v7715: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4178923433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:41 smithi022 ceph-mon[40250]: pgmap v7715: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4178923433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:41 smithi122 ceph-mon[32255]: pgmap v7715: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4178923433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:15:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:15:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:43 smithi122 ceph-mon[32255]: pgmap v7716: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:43 smithi022 ceph-mon[31360]: pgmap v7716: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:43 smithi022 ceph-mon[40250]: pgmap v7716: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:45 smithi122 ceph-mon[32255]: pgmap v7717: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:45 smithi022 ceph-mon[31360]: pgmap v7717: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:45 smithi022 ceph-mon[40250]: pgmap v7717: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:46.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:15:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:15:46.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:47.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:47.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:47.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:15:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:47 smithi122 ceph-mon[32255]: pgmap v7718: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1912043661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:48.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:47 smithi022 ceph-mon[31360]: pgmap v7718: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:48.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1912043661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:48.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:47 smithi022 ceph-mon[40250]: pgmap v7718: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:48.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1912043661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:49 smithi122 ceph-mon[32255]: pgmap v7719: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:49 smithi022 ceph-mon[31360]: pgmap v7719: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:49 smithi022 ceph-mon[40250]: pgmap v7719: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:50] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:15:52.125 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:51 smithi022 ceph-mon[31360]: pgmap v7720: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:52.126 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:51 smithi022 ceph-mon[40250]: pgmap v7720: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:52.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:51 smithi122 ceph-mon[32255]: pgmap v7720: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:52.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:52.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:52.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:15:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:15:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3176214213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3176214213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3176214213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:53 smithi122 ceph-mon[32255]: pgmap v7721: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:53 smithi022 ceph-mon[31360]: pgmap v7721: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:53 smithi022 ceph-mon[40250]: pgmap v7721: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:15:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:55 smithi122 ceph-mon[32255]: pgmap v7722: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:55 smithi022 ceph-mon[31360]: pgmap v7722: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:55 smithi022 ceph-mon[40250]: pgmap v7722: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:15:56.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:15:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:15:57.493 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:15:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:15:57.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:15:57.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:15:57.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:15:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:57 smithi122 ceph-mon[32255]: pgmap v7723: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:15:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:58.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:57 smithi022 ceph-mon[31360]: pgmap v7723: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:15:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:15:58.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:57 smithi022 ceph-mon[40250]: pgmap v7723: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:15:59.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1558915178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1558915178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:15:59.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1558915178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:15:59 smithi122 ceph-mon[32255]: pgmap v7724: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:15:59 smithi022 ceph-mon[31360]: pgmap v7724: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:15:59 smithi022 ceph-mon[40250]: pgmap v7724: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:00] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:16:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:01 smithi122 ceph-mon[32255]: pgmap v7725: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:16:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:01 smithi022 ceph-mon[31360]: pgmap v7725: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:16:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:01 smithi022 ceph-mon[40250]: pgmap v7725: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:16:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:16:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:16:02.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:03.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:03.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:03.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:03 smithi122 ceph-mon[32255]: pgmap v7726: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:16:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/494978746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:03 smithi022 ceph-mon[31360]: pgmap v7726: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:16:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/494978746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:03 smithi022 ceph-mon[40250]: pgmap v7726: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:16:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/494978746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:05 smithi122 ceph-mon[32255]: pgmap v7727: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:05 smithi022 ceph-mon[31360]: pgmap v7727: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:05 smithi022 ceph-mon[40250]: pgmap v7727: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:06.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:16:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:16:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:16:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:07 smithi122 ceph-mon[32255]: pgmap v7728: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:08.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:08.236 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:08.236 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:07 smithi022 ceph-mon[31360]: pgmap v7728: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:08.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:08.237 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:07 smithi022 ceph-mon[40250]: pgmap v7728: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:08.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:08.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:08.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/446209655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/446209655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:09.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/446209655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:09 smithi122 ceph-mon[32255]: pgmap v7729: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:09 smithi022 ceph-mon[31360]: pgmap v7729: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:09 smithi022 ceph-mon[40250]: pgmap v7729: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:16:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:16:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:16:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:16:11.215 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:16:11.215 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:16:11.215 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:16:11.216 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:16:11.216 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:16:11.216 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:16:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:11 smithi122 ceph-mon[32255]: pgmap v7730: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:16:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:11 smithi022 ceph-mon[31360]: pgmap v7730: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:16:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:11 smithi022 ceph-mon[40250]: pgmap v7730: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:16:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:16:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:16:13.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:13.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:13.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:13.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:13 smithi122 ceph-mon[32255]: pgmap v7731: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:13 smithi022 ceph-mon[31360]: pgmap v7731: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:13 smithi022 ceph-mon[40250]: pgmap v7731: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2472644552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2472644552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2472644552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:15 smithi122 ceph-mon[32255]: pgmap v7732: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:15 smithi022 ceph-mon[31360]: pgmap v7732: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:15 smithi022 ceph-mon[40250]: pgmap v7732: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:16.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:16:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:16:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:16:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:17 smithi122 ceph-mon[32255]: pgmap v7733: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:17 smithi022 ceph-mon[31360]: pgmap v7733: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:17 smithi022 ceph-mon[40250]: pgmap v7733: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:18.974 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:19.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:19.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:19.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:19 smithi122 ceph-mon[32255]: pgmap v7734: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1177815927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:19 smithi022 ceph-mon[31360]: pgmap v7734: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1177815927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:19 smithi022 ceph-mon[40250]: pgmap v7734: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1177815927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:16:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:21 smithi122 ceph-mon[32255]: pgmap v7735: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:21 smithi022 ceph-mon[31360]: pgmap v7735: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:21 smithi022 ceph-mon[40250]: pgmap v7735: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:16:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:16:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:23 smithi122 ceph-mon[32255]: pgmap v7736: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:23 smithi022 ceph-mon[31360]: pgmap v7736: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:23 smithi022 ceph-mon[40250]: pgmap v7736: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:24.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3547565208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3547565208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3547565208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:25 smithi122 ceph-mon[32255]: pgmap v7737: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:25 smithi022 ceph-mon[31360]: pgmap v7737: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:25 smithi022 ceph-mon[40250]: pgmap v7737: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:26.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:16:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:16:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:16:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:27 smithi122 ceph-mon[32255]: pgmap v7738: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:27 smithi022 ceph-mon[31360]: pgmap v7738: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:27 smithi022 ceph-mon[40250]: pgmap v7738: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:29.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:29 smithi022 ceph-mon[31360]: pgmap v7739: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:29 smithi022 ceph-mon[40250]: pgmap v7739: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:30.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:29 smithi122 ceph-mon[32255]: pgmap v7739: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:16:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/708118733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:31.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/708118733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/708118733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:31 smithi122 ceph-mon[32255]: pgmap v7740: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:31 smithi022 ceph-mon[31360]: pgmap v7740: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:31 smithi022 ceph-mon[40250]: pgmap v7740: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:16:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:16:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:33 smithi122 ceph-mon[32255]: pgmap v7741: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:16:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:16:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:16:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:16:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[31360]: pgmap v7741: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:16:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:16:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:16:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:16:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[40250]: pgmap v7741: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:16:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:16:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:16:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:16:35.089 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:35.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:35.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:35.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:35 smithi122 ceph-mon[32255]: pgmap v7742: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1358290737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:35 smithi022 ceph-mon[31360]: pgmap v7742: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1358290737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:35 smithi022 ceph-mon[40250]: pgmap v7742: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1358290737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:36.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:16:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:16:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:16:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:37 smithi122 ceph-mon[32255]: pgmap v7743: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:37 smithi022 ceph-mon[31360]: pgmap v7743: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:37 smithi022 ceph-mon[40250]: pgmap v7743: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:40.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:39 smithi122 ceph-mon[32255]: pgmap v7744: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:39 smithi022 ceph-mon[31360]: pgmap v7744: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:39 smithi022 ceph-mon[40250]: pgmap v7744: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:40] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:16:40.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:40.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:40.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:40.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/597784484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/597784484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:41.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/597784484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:41 smithi122 ceph-mon[32255]: pgmap v7745: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:41 smithi022 ceph-mon[31360]: pgmap v7745: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:41 smithi022 ceph-mon[40250]: pgmap v7745: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:16:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:16:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:43 smithi122 ceph-mon[32255]: pgmap v7746: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:43 smithi022 ceph-mon[31360]: pgmap v7746: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:43 smithi022 ceph-mon[40250]: pgmap v7746: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:45.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:45 smithi122 ceph-mon[32255]: pgmap v7747: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:46.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:46.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:46.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:46.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:45 smithi022 ceph-mon[31360]: pgmap v7747: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:46.196 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:45 smithi022 ceph-mon[40250]: pgmap v7747: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:46.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:16:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:16:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3355937653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3355937653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3355937653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:16:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:47 smithi122 ceph-mon[32255]: pgmap v7748: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:48.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:47 smithi022 ceph-mon[31360]: pgmap v7748: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:47 smithi022 ceph-mon[40250]: pgmap v7748: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:50.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:50] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:16:50.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:49 smithi122 ceph-mon[32255]: pgmap v7749: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:49 smithi022 ceph-mon[31360]: pgmap v7749: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:49 smithi022 ceph-mon[40250]: pgmap v7749: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:51.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:51.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:51.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:51.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:51 smithi022 ceph-mon[31360]: pgmap v7750: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:52.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4011035696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:51 smithi022 ceph-mon[40250]: pgmap v7750: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4011035696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:51 smithi122 ceph-mon[32255]: pgmap v7750: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4011035696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:16:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:16:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:53 smithi022 ceph-mon[31360]: pgmap v7751: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:53 smithi022 ceph-mon[40250]: pgmap v7751: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:53 smithi122 ceph-mon[32255]: pgmap v7751: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:55 smithi022 ceph-mon[31360]: pgmap v7752: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:55 smithi022 ceph-mon[40250]: pgmap v7752: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:56.294 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:55 smithi122 ceph-mon[32255]: pgmap v7752: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:16:56.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:16:56.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:16:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:16:56.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:16:56.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:16:56.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:16:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/499656021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/499656021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:57.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/499656021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:16:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:16:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:57 smithi022 ceph-mon[31360]: pgmap v7753: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:57 smithi022 ceph-mon[40250]: pgmap v7753: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:16:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:16:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:57 smithi122 ceph-mon[32255]: pgmap v7753: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:16:59 smithi022 ceph-mon[31360]: pgmap v7754: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:16:59 smithi022 ceph-mon[40250]: pgmap v7754: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:16:59 smithi122 ceph-mon[32255]: pgmap v7754: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:00] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:17:01.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:01 smithi022 ceph-mon[31360]: pgmap v7755: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:01 smithi022 ceph-mon[40250]: pgmap v7755: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:02.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:02.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:02.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:01 smithi122 ceph-mon[32255]: pgmap v7755: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:17:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:17:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1145964717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1145964717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1145964717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:03 smithi022 ceph-mon[31360]: pgmap v7756: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:03 smithi022 ceph-mon[40250]: pgmap v7756: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:03 smithi122 ceph-mon[32255]: pgmap v7756: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:05 smithi022 ceph-mon[31360]: pgmap v7757: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:05 smithi022 ceph-mon[40250]: pgmap v7757: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:06.297 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:05 smithi122 ceph-mon[32255]: pgmap v7757: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:06.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:17:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:17:07.305 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:07.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:07.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:07.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:07.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:17:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:07 smithi022 ceph-mon[31360]: pgmap v7758: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:08.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/670545042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:07 smithi022 ceph-mon[40250]: pgmap v7758: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/670545042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:07 smithi122 ceph-mon[32255]: pgmap v7758: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/670545042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:09 smithi022 ceph-mon[31360]: pgmap v7759: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:09 smithi022 ceph-mon[40250]: pgmap v7759: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:17:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:09 smithi122 ceph-mon[32255]: pgmap v7759: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[31360]: pgmap v7760: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:12.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:17:12.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:17:12.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:17:12.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:17:12.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[40250]: pgmap v7760: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:12.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:17:12.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:17:12.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:17:12.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:17:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:11 smithi122 ceph-mon[32255]: pgmap v7760: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:17:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:17:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:17:12.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:17:12.676 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:17:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:17:13.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:13.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:13.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:13 smithi022 ceph-mon[31360]: pgmap v7761: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2231843574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:13 smithi022 ceph-mon[40250]: pgmap v7761: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2231843574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:13 smithi122 ceph-mon[32255]: pgmap v7761: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2231843574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:15 smithi022 ceph-mon[31360]: pgmap v7762: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:15 smithi022 ceph-mon[40250]: pgmap v7762: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:16.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:15 smithi122 ceph-mon[32255]: pgmap v7762: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:16.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:17:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:17:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:17:18.044 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:17 smithi022 ceph-mon[31360]: pgmap v7763: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:17 smithi022 ceph-mon[40250]: pgmap v7763: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:18.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:18.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:18.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:17 smithi122 ceph-mon[32255]: pgmap v7763: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/633695839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/633695839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/633695839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:19 smithi022 ceph-mon[31360]: pgmap v7764: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:19 smithi022 ceph-mon[40250]: pgmap v7764: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:20] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:17:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:19 smithi122 ceph-mon[32255]: pgmap v7764: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:22.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:21 smithi022 ceph-mon[31360]: pgmap v7765: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:21 smithi022 ceph-mon[40250]: pgmap v7765: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:21 smithi122 ceph-mon[32255]: pgmap v7765: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:17:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:17:23.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:23.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:23.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:23.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:23 smithi022 ceph-mon[31360]: pgmap v7766: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/729027590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:23 smithi022 ceph-mon[40250]: pgmap v7766: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/729027590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:23 smithi122 ceph-mon[32255]: pgmap v7766: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/729027590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:25 smithi022 ceph-mon[31360]: pgmap v7767: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:25 smithi022 ceph-mon[40250]: pgmap v7767: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:26.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:25 smithi122 ceph-mon[32255]: pgmap v7767: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:26.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:17:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:17:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:17:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:27 smithi022 ceph-mon[31360]: pgmap v7768: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:27 smithi022 ceph-mon[40250]: pgmap v7768: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:27 smithi122 ceph-mon[32255]: pgmap v7768: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:28.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:29.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:29.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:29.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:29 smithi022 ceph-mon[31360]: pgmap v7769: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2857136410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:29 smithi022 ceph-mon[40250]: pgmap v7769: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2857136410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:30] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:17:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:29 smithi122 ceph-mon[32255]: pgmap v7769: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2857136410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:31 smithi022 ceph-mon[31360]: pgmap v7770: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:32 smithi022 ceph-mon[40250]: pgmap v7770: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:32 smithi122 ceph-mon[32255]: pgmap v7770: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:17:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:17:34.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[31360]: pgmap v7771: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:17:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:17:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:17:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:17:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[40250]: pgmap v7771: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:17:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:17:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:17:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:17:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:34 smithi122 ceph-mon[32255]: pgmap v7771: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:17:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:17:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:17:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:17:34.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:34.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:34.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4139274424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4139274424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4139274424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:36 smithi022 ceph-mon[31360]: pgmap v7772: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:36 smithi022 ceph-mon[40250]: pgmap v7772: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:36.327 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:36 smithi122 ceph-mon[32255]: pgmap v7772: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:36.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:17:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:17:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:17:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:38 smithi022 ceph-mon[31360]: pgmap v7773: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:38 smithi022 ceph-mon[40250]: pgmap v7773: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:38 smithi122 ceph-mon[32255]: pgmap v7773: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:39.538 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:39.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:39.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:39.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:40 smithi022 ceph-mon[31360]: pgmap v7774: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:40.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1436851348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:40 smithi022 ceph-mon[40250]: pgmap v7774: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1436851348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:40] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:17:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:40 smithi122 ceph-mon[32255]: pgmap v7774: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:40.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1436851348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:42 smithi022 ceph-mon[31360]: pgmap v7775: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:42 smithi022 ceph-mon[40250]: pgmap v7775: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:42 smithi122 ceph-mon[32255]: pgmap v7775: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:17:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:17:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:44 smithi022 ceph-mon[31360]: pgmap v7776: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:44 smithi022 ceph-mon[40250]: pgmap v7776: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:44 smithi122 ceph-mon[32255]: pgmap v7776: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:44.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:45.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:45.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:45.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:46 smithi022 ceph-mon[31360]: pgmap v7777: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/816114904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:46 smithi022 ceph-mon[40250]: pgmap v7777: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/816114904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:46.338 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:46 smithi122 ceph-mon[32255]: pgmap v7777: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:46.338 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/816114904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:46.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:17:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:17:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:17:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:48 smithi022 ceph-mon[31360]: pgmap v7778: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:48 smithi022 ceph-mon[40250]: pgmap v7778: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:48 smithi122 ceph-mon[32255]: pgmap v7778: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:50 smithi022 ceph-mon[31360]: pgmap v7779: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:50 smithi022 ceph-mon[40250]: pgmap v7779: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:50.282 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:50] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:17:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:50 smithi122 ceph-mon[32255]: pgmap v7779: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:17:50.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:50.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:50.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/626426553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/626426553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/626426553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:52 smithi122 ceph-mon[32255]: pgmap v7780: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:52 smithi022 ceph-mon[31360]: pgmap v7780: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:52 smithi022 ceph-mon[40250]: pgmap v7780: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:17:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:17:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:54 smithi122 ceph-mon[32255]: pgmap v7781: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:54 smithi022 ceph-mon[31360]: pgmap v7781: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:54 smithi022 ceph-mon[40250]: pgmap v7781: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:17:55.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:17:56.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:17:56.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:17:56.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:17:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:56 smithi022 ceph-mon[31360]: pgmap v7782: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T22:17:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4135073685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:56 smithi022 ceph-mon[40250]: pgmap v7782: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T22:17:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4135073685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:56.339 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:56 smithi122 ceph-mon[32255]: pgmap v7782: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T22:17:56.340 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4135073685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:17:56.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:17:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:17:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:17:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:17:58 smithi122 ceph-mon[32255]: pgmap v7783: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:17:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:17:58 smithi022 ceph-mon[31360]: pgmap v7783: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:17:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:17:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:17:58 smithi022 ceph-mon[40250]: pgmap v7783: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:18:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:00 smithi122 ceph-mon[32255]: pgmap v7784: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T22:18:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:00] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:18:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:00 smithi022 ceph-mon[31360]: pgmap v7784: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T22:18:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:00 smithi022 ceph-mon[40250]: pgmap v7784: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T22:18:01.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:01.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:01.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:01.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:02 smithi122 ceph-mon[32255]: pgmap v7785: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:18:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2851150902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:02 smithi022 ceph-mon[31360]: pgmap v7785: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:18:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2851150902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:02.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:02 smithi022 ceph-mon[40250]: pgmap v7785: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:18:02.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2851150902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:18:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:18:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:04 smithi122 ceph-mon[32255]: pgmap v7786: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:18:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:04 smithi022 ceph-mon[31360]: pgmap v7786: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:18:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:04 smithi022 ceph-mon[40250]: pgmap v7786: 132 pgs: 132 active+clean; 582 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T22:18:06.341 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:06 smithi122 ceph-mon[32255]: pgmap v7787: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T22:18:06.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:06.402 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:06 smithi022 ceph-mon[31360]: pgmap v7787: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T22:18:06.403 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:06 smithi022 ceph-mon[40250]: pgmap v7787: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-12-15T22:18:06.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:18:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:18:06.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:06.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:06.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396905888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396905888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396905888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:18:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:08 smithi122 ceph-mon[32255]: pgmap v7788: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:08 smithi022 ceph-mon[31360]: pgmap v7788: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:08 smithi022 ceph-mon[40250]: pgmap v7788: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:18:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:10 smithi122 ceph-mon[32255]: pgmap v7789: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:18:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:10 smithi022 ceph-mon[31360]: pgmap v7789: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:18:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:10 smithi022 ceph-mon[40250]: pgmap v7789: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:18:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:12.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:12.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:12.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:18:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:18:12.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:12 smithi122 ceph-mon[32255]: pgmap v7790: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:12.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:18:12.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:18:12.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:18:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[31360]: pgmap v7790: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:18:12.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:18:12.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:18:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[40250]: pgmap v7790: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:18:12.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:18:12.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:18:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1496080080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:18:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1496080080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:18:13.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1496080080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:13.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:18:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:14 smithi122 ceph-mon[32255]: pgmap v7791: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:14 smithi022 ceph-mon[31360]: pgmap v7791: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:14 smithi022 ceph-mon[40250]: pgmap v7791: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:18:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:16 smithi122 ceph-mon[32255]: pgmap v7792: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:18:16.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:18:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:18:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:16 smithi022 ceph-mon[31360]: pgmap v7792: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:18:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:16 smithi022 ceph-mon[40250]: pgmap v7792: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:18:17.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:18:17.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:17.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:17.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:18 smithi122 ceph-mon[32255]: pgmap v7793: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1209951326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:18 smithi022 ceph-mon[31360]: pgmap v7793: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1209951326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:18 smithi022 ceph-mon[40250]: pgmap v7793: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:18.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1209951326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:20] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:18:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:20 smithi122 ceph-mon[32255]: pgmap v7794: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:20 smithi022 ceph-mon[31360]: pgmap v7794: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:20 smithi022 ceph-mon[40250]: pgmap v7794: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:22 smithi022 ceph-mon[31360]: pgmap v7795: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:22 smithi022 ceph-mon[40250]: pgmap v7795: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:22.541 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:22 smithi122 ceph-mon[32255]: pgmap v7795: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:18:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:18:22.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:22.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:22.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1296657561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:23.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1296657561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1296657561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:24 smithi022 ceph-mon[31360]: pgmap v7796: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:24 smithi022 ceph-mon[40250]: pgmap v7796: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:24 smithi122 ceph-mon[32255]: pgmap v7796: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:26 smithi022 ceph-mon[31360]: pgmap v7797: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:26 smithi022 ceph-mon[40250]: pgmap v7797: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:26 smithi122 ceph-mon[32255]: pgmap v7797: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:26.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:18:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:18:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:18:27.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:28.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:28.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:28.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:28 smithi022 ceph-mon[31360]: pgmap v7798: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:28 smithi022 ceph-mon[40250]: pgmap v7798: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:28 smithi122 ceph-mon[32255]: pgmap v7798: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2382574184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2382574184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2382574184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:30 smithi122 ceph-mon[32255]: pgmap v7799: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:30] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:18:30.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:30 smithi022 ceph-mon[31360]: pgmap v7799: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:30 smithi022 ceph-mon[40250]: pgmap v7799: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:31 smithi022 ceph-mon[31360]: pgmap v7800: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:31 smithi022 ceph-mon[40250]: pgmap v7800: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:31 smithi122 ceph-mon[32255]: pgmap v7800: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:18:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:18:33.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:33.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:33.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:33.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[31360]: pgmap v7801: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:18:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:18:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:18:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:18:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/527802109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[40250]: pgmap v7801: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:18:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:18:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:18:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:18:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/527802109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:33 smithi122 ceph-mon[32255]: pgmap v7801: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:18:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:18:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:18:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:18:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/527802109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:35 smithi122 ceph-mon[32255]: pgmap v7802: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:35 smithi022 ceph-mon[31360]: pgmap v7802: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:35 smithi022 ceph-mon[40250]: pgmap v7802: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:36.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:18:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:18:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:18:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:37 smithi122 ceph-mon[32255]: pgmap v7803: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:37 smithi022 ceph-mon[40250]: pgmap v7803: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:37 smithi022 ceph-mon[31360]: pgmap v7803: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:38.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:39.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:39.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:39.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:39 smithi122 ceph-mon[32255]: pgmap v7804: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1013153694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:39 smithi022 ceph-mon[31360]: pgmap v7804: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1013153694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:39 smithi022 ceph-mon[40250]: pgmap v7804: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1013153694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:40] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:18:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:41 smithi122 ceph-mon[32255]: pgmap v7805: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:41 smithi022 ceph-mon[31360]: pgmap v7805: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:41 smithi022 ceph-mon[40250]: pgmap v7805: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:18:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:18:44.026 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:43 smithi122 ceph-mon[32255]: pgmap v7806: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:43 smithi022 ceph-mon[40250]: pgmap v7806: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:43 smithi022 ceph-mon[31360]: pgmap v7806: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:44.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:44.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:44.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:45.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1057272270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1057272270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:45.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1057272270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:45 smithi122 ceph-mon[32255]: pgmap v7807: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:45 smithi022 ceph-mon[31360]: pgmap v7807: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:45 smithi022 ceph-mon[40250]: pgmap v7807: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:46.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:18:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:18:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:18:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:47 smithi122 ceph-mon[32255]: pgmap v7808: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:47 smithi022 ceph-mon[31360]: pgmap v7808: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:47 smithi022 ceph-mon[40250]: pgmap v7808: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:49.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:49.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:49.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:49.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:49 smithi022 ceph-mon[31360]: pgmap v7809: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2803648350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:49 smithi022 ceph-mon[40250]: pgmap v7809: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2803648350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:49 smithi122 ceph-mon[32255]: pgmap v7809: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2803648350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:50] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:18:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:51 smithi122 ceph-mon[32255]: pgmap v7810: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:51 smithi022 ceph-mon[31360]: pgmap v7810: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:51 smithi022 ceph-mon[40250]: pgmap v7810: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:18:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:18:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:53 smithi122 ceph-mon[32255]: pgmap v7811: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:53 smithi022 ceph-mon[31360]: pgmap v7811: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:53 smithi022 ceph-mon[40250]: pgmap v7811: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:54.776 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:18:55.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:18:55.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:18:55.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:18:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:55 smithi122 ceph-mon[32255]: pgmap v7812: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3759941768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:55 smithi022 ceph-mon[31360]: pgmap v7812: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:56.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3759941768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:55 smithi022 ceph-mon[40250]: pgmap v7812: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:18:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3759941768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:18:56.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:18:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:18:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:18:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:57 smithi122 ceph-mon[32255]: pgmap v7813: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:57 smithi022 ceph-mon[31360]: pgmap v7813: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:18:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:18:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:57 smithi022 ceph-mon[40250]: pgmap v7813: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:00.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:00.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:18:59 smithi022 ceph-mon[31360]: pgmap v7814: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:00.149 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:18:59 smithi022 ceph-mon[40250]: pgmap v7814: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:18:59 smithi122 ceph-mon[32255]: pgmap v7814: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:00] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:19:00.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:00.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:00.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3826349891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3826349891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3826349891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:01 smithi122 ceph-mon[32255]: pgmap v7815: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:01 smithi022 ceph-mon[31360]: pgmap v7815: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:01 smithi022 ceph-mon[40250]: pgmap v7815: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:19:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:19:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:03 smithi122 ceph-mon[32255]: pgmap v7816: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:03 smithi022 ceph-mon[31360]: pgmap v7816: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:03 smithi022 ceph-mon[40250]: pgmap v7816: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:05.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:05.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:05.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:05.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:05 smithi122 ceph-mon[32255]: pgmap v7817: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:05 smithi022 ceph-mon[31360]: pgmap v7817: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:05 smithi022 ceph-mon[40250]: pgmap v7817: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:06.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:19:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:19:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1622833308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1622833308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1622833308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:19:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:07 smithi122 ceph-mon[32255]: pgmap v7818: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:07 smithi022 ceph-mon[31360]: pgmap v7818: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:07 smithi022 ceph-mon[40250]: pgmap v7818: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:09 smithi122 ceph-mon[32255]: pgmap v7819: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:09 smithi022 ceph-mon[31360]: pgmap v7819: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:09 smithi022 ceph-mon[40250]: pgmap v7819: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:10] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:19:10.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:11.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:11.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:11.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:11 smithi122 ceph-mon[32255]: pgmap v7820: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2405204171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:12.222 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:11 smithi022 ceph-mon[40250]: pgmap v7820: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:12.222 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2405204171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:12.223 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:11 smithi022 ceph-mon[31360]: pgmap v7820: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:12.223 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2405204171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:12.570 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:19:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:19:12.885 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:19:12.885 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:19:12.886 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:19:12.886 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:12.886 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:19:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:19:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:19:13.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:13.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:19:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:19:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:19:13.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:13.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:14 smithi022 ceph-mon[31360]: pgmap v7821: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:14 smithi022 ceph-mon[40250]: pgmap v7821: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:14 smithi122 ceph-mon[32255]: pgmap v7821: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:19:16.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:16.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:16 smithi122 ceph-mon[32255]: pgmap v7822: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:16 smithi022 ceph-mon[31360]: pgmap v7822: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:16 smithi022 ceph-mon[40250]: pgmap v7822: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:16.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:16.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:16.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:16.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:19:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:19:17.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3509895088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3509895088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3509895088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:19:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:18 smithi122 ceph-mon[32255]: pgmap v7823: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:18 smithi022 ceph-mon[31360]: pgmap v7823: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:18 smithi022 ceph-mon[40250]: pgmap v7823: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:20 smithi122 ceph-mon[32255]: pgmap v7824: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:19:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:20 smithi022 ceph-mon[31360]: pgmap v7824: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:20 smithi022 ceph-mon[40250]: pgmap v7824: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:21.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:21.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:21.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:21.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:22 smithi022 ceph-mon[31360]: pgmap v7825: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/833044989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:22 smithi022 ceph-mon[40250]: pgmap v7825: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/833044989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:22 smithi122 ceph-mon[32255]: pgmap v7825: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/833044989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:19:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:19:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:24 smithi122 ceph-mon[32255]: pgmap v7826: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:24 smithi022 ceph-mon[31360]: pgmap v7826: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:24 smithi022 ceph-mon[40250]: pgmap v7826: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:26.414 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:26 smithi122 ceph-mon[32255]: pgmap v7827: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:26 smithi022 ceph-mon[31360]: pgmap v7827: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:26 smithi022 ceph-mon[40250]: pgmap v7827: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:26.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:19:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:19:27.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:27.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:27.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:27.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:19:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:28 smithi122 ceph-mon[32255]: pgmap v7828: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/865266999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:28 smithi022 ceph-mon[31360]: pgmap v7828: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/865266999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:28 smithi022 ceph-mon[40250]: pgmap v7828: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/865266999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:30 smithi122 ceph-mon[32255]: pgmap v7829: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:19:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:30 smithi022 ceph-mon[31360]: pgmap v7829: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:30 smithi022 ceph-mon[40250]: pgmap v7829: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:32 smithi122 ceph-mon[32255]: pgmap v7830: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:32.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:32.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:32 smithi022 ceph-mon[40250]: pgmap v7830: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:32.373 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:32 smithi022 ceph-mon[31360]: pgmap v7830: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:19:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:19:32.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:32.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:32.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3870976269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3870976269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3870976269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:34 smithi122 ceph-mon[32255]: pgmap v7831: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:19:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:19:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:19:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:19:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[31360]: pgmap v7831: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:19:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:19:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:19:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[40250]: pgmap v7831: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:19:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:19:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:19:36.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:36 smithi122 ceph-mon[32255]: pgmap v7832: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:36 smithi022 ceph-mon[31360]: pgmap v7832: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:36 smithi022 ceph-mon[40250]: pgmap v7832: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:36.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:19:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:19:37.741 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:37.741 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:19:38.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:38.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:38.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:38 smithi122 ceph-mon[32255]: pgmap v7833: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:38 smithi022 ceph-mon[31360]: pgmap v7833: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:38 smithi022 ceph-mon[40250]: pgmap v7833: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2970798974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2970798974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2970798974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:19:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:40 smithi122 ceph-mon[32255]: pgmap v7834: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:40 smithi022 ceph-mon[31360]: pgmap v7834: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:40 smithi022 ceph-mon[40250]: pgmap v7834: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:42 smithi122 ceph-mon[32255]: pgmap v7835: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:19:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:19:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:42 smithi022 ceph-mon[31360]: pgmap v7835: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:42.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:42 smithi022 ceph-mon[40250]: pgmap v7835: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:43.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:43.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:43.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:43.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:44 smithi122 ceph-mon[32255]: pgmap v7836: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2719759326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:44 smithi022 ceph-mon[31360]: pgmap v7836: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2719759326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:44 smithi022 ceph-mon[40250]: pgmap v7836: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2719759326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:46.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:46 smithi122 ceph-mon[32255]: pgmap v7837: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:46 smithi022 ceph-mon[31360]: pgmap v7837: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:46 smithi022 ceph-mon[40250]: pgmap v7837: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:46.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:19:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:19:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:19:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:48 smithi122 ceph-mon[32255]: pgmap v7838: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:48.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:48.485 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:48.485 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:48 smithi022 ceph-mon[31360]: pgmap v7838: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:48.486 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:48.486 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:48 smithi022 ceph-mon[40250]: pgmap v7838: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:48.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:49 smithi122 ceph-mon[32255]: pgmap v7839: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:50.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/499623100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:49 smithi022 ceph-mon[31360]: pgmap v7839: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:50.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/499623100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:49 smithi022 ceph-mon[40250]: pgmap v7839: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:50.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/499623100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:50] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:19:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:51 smithi122 ceph-mon[32255]: pgmap v7840: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:51 smithi022 ceph-mon[31360]: pgmap v7840: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:51 smithi022 ceph-mon[40250]: pgmap v7840: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:19:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:19:53.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:53 smithi122 ceph-mon[32255]: pgmap v7841: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:54.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:54.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:54.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:19:54.225 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:53 smithi022 ceph-mon[40250]: pgmap v7841: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:54.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:53 smithi022 ceph-mon[31360]: pgmap v7841: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:55.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3153628994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3153628994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3153628994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:19:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:55 smithi122 ceph-mon[32255]: pgmap v7842: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:55 smithi022 ceph-mon[31360]: pgmap v7842: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:55 smithi022 ceph-mon[40250]: pgmap v7842: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:19:56.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:19:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:19:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:19:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:57 smithi122 ceph-mon[32255]: pgmap v7843: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:57 smithi022 ceph-mon[31360]: pgmap v7843: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:19:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:57 smithi022 ceph-mon[40250]: pgmap v7843: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:19:59.226 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:19:59.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:19:59.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:19:59.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:59 smithi022 ceph-mon[31360]: pgmap v7844: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:19:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/999607628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:59 smithi022 ceph-mon[40250]: pgmap v7844: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:19:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/999607628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:59 smithi122 ceph-mon[32255]: pgmap v7844: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:19:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/999607628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:00] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:20:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T22:20:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T22:20:01.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T22:20:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:01 smithi122 ceph-mon[32255]: pgmap v7845: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:01 smithi022 ceph-mon[40250]: pgmap v7845: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:02.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:01 smithi022 ceph-mon[31360]: pgmap v7845: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:20:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:20:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:03 smithi122 ceph-mon[32255]: pgmap v7846: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:03 smithi022 ceph-mon[31360]: pgmap v7846: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:03 smithi022 ceph-mon[40250]: pgmap v7846: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:04.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:04.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:04.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:04.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:05 smithi122 ceph-mon[32255]: pgmap v7847: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2940429748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:05 smithi022 ceph-mon[31360]: pgmap v7847: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2940429748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:05 smithi022 ceph-mon[40250]: pgmap v7847: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2940429748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:06.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:20:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:20:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:20:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:07 smithi122 ceph-mon[32255]: pgmap v7848: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:07 smithi022 ceph-mon[31360]: pgmap v7848: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:07 smithi022 ceph-mon[40250]: pgmap v7848: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:09.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:09 smithi122 ceph-mon[32255]: pgmap v7849: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:09 smithi022 ceph-mon[31360]: pgmap v7849: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:09 smithi022 ceph-mon[40250]: pgmap v7849: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:10.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:10.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:10.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:10.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:20:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1826762822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1826762822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1826762822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:11 smithi122 ceph-mon[32255]: pgmap v7850: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:11 smithi022 ceph-mon[31360]: pgmap v7850: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:11 smithi022 ceph-mon[40250]: pgmap v7850: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:20:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:20:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:13 smithi122 ceph-mon[32255]: pgmap v7851: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:20:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:20:14.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:20:14.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:20:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[31360]: pgmap v7851: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:20:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:20:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:20:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:20:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[40250]: pgmap v7851: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:14.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:20:14.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:20:14.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:20:14.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:20:15.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:15.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:15.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:15.710 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:15 smithi022 ceph-mon[31360]: pgmap v7852: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3832193992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:15 smithi022 ceph-mon[40250]: pgmap v7852: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3832193992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:15 smithi122 ceph-mon[32255]: pgmap v7852: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3832193992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:16.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:20:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:20:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:20:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:17 smithi122 ceph-mon[32255]: pgmap v7853: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:17 smithi022 ceph-mon[31360]: pgmap v7853: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:17 smithi022 ceph-mon[40250]: pgmap v7853: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:19 smithi122 ceph-mon[32255]: pgmap v7854: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:19 smithi022 ceph-mon[31360]: pgmap v7854: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:19 smithi022 ceph-mon[40250]: pgmap v7854: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:20] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:20:20.711 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:21.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:21.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:21.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:21 smithi122 ceph-mon[32255]: pgmap v7855: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4291677314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:21 smithi022 ceph-mon[31360]: pgmap v7855: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4291677314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:21 smithi022 ceph-mon[40250]: pgmap v7855: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4291677314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:20:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:20:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:23 smithi122 ceph-mon[32255]: pgmap v7856: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:23 smithi022 ceph-mon[31360]: pgmap v7856: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:23 smithi022 ceph-mon[40250]: pgmap v7856: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:26.083 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:25 smithi122 ceph-mon[32255]: pgmap v7857: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:25 smithi022 ceph-mon[31360]: pgmap v7857: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:25 smithi022 ceph-mon[40250]: pgmap v7857: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:26.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:20:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:20:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/180852711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/180852711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/180852711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:20:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:27 smithi122 ceph-mon[32255]: pgmap v7858: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:27 smithi022 ceph-mon[31360]: pgmap v7858: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:27 smithi022 ceph-mon[40250]: pgmap v7858: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:29 smithi122 ceph-mon[32255]: pgmap v7859: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:29 smithi022 ceph-mon[31360]: pgmap v7859: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:29 smithi022 ceph-mon[40250]: pgmap v7859: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:30] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:20:31.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:31.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:31.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:31.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:31 smithi122 ceph-mon[32255]: pgmap v7860: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1714426811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:31 smithi022 ceph-mon[31360]: pgmap v7860: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1714426811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:31 smithi022 ceph-mon[40250]: pgmap v7860: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1714426811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:20:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:20:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:33 smithi122 ceph-mon[32255]: pgmap v7861: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:20:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:20:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:20:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:20:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[31360]: pgmap v7861: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:20:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:20:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:20:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:20:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[40250]: pgmap v7861: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:20:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:20:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:20:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:20:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:35 smithi122 ceph-mon[32255]: pgmap v7862: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:35 smithi022 ceph-mon[31360]: pgmap v7862: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:35 smithi022 ceph-mon[40250]: pgmap v7862: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:36.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:36.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:20:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:20:37.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:37.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:37.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:20:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:37 smithi122 ceph-mon[32255]: pgmap v7863: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2984078813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:37 smithi022 ceph-mon[31360]: pgmap v7863: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2984078813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:37 smithi022 ceph-mon[40250]: pgmap v7863: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2984078813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:39 smithi122 ceph-mon[32255]: pgmap v7864: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:39 smithi022 ceph-mon[31360]: pgmap v7864: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:39 smithi022 ceph-mon[40250]: pgmap v7864: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:40] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:20:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:41 smithi122 ceph-mon[32255]: pgmap v7865: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:42.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:42.198 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:41 smithi022 ceph-mon[31360]: pgmap v7865: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:42.198 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:41 smithi022 ceph-mon[40250]: pgmap v7865: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:42.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:42.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:42.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:20:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:20:42.887 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2513696701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:42.888 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2513696701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:43.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2513696701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:43 smithi122 ceph-mon[32255]: pgmap v7866: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:43 smithi022 ceph-mon[31360]: pgmap v7866: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:43 smithi022 ceph-mon[40250]: pgmap v7866: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:45 smithi122 ceph-mon[32255]: pgmap v7867: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:45 smithi022 ceph-mon[31360]: pgmap v7867: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:45 smithi022 ceph-mon[40250]: pgmap v7867: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:46.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:20:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:20:47.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:20:47.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:47.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:47.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:47 smithi122 ceph-mon[32255]: pgmap v7868: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:47 smithi022 ceph-mon[31360]: pgmap v7868: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:47 smithi022 ceph-mon[40250]: pgmap v7868: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1397887258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1397887258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1397887258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:49 smithi122 ceph-mon[32255]: pgmap v7869: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:49 smithi022 ceph-mon[31360]: pgmap v7869: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:49 smithi022 ceph-mon[40250]: pgmap v7869: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:50] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:20:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:51 smithi122 ceph-mon[32255]: pgmap v7870: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:51 smithi022 ceph-mon[31360]: pgmap v7870: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:51 smithi022 ceph-mon[40250]: pgmap v7870: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:20:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:20:52.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:53.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:53.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:53.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:53 smithi122 ceph-mon[32255]: pgmap v7871: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3495657888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:53 smithi022 ceph-mon[31360]: pgmap v7871: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3495657888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:53 smithi022 ceph-mon[40250]: pgmap v7871: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3495657888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:55 smithi122 ceph-mon[32255]: pgmap v7872: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:55 smithi022 ceph-mon[31360]: pgmap v7872: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:55 smithi022 ceph-mon[40250]: pgmap v7872: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:20:56.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:20:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:20:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:20:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:57 smithi122 ceph-mon[32255]: pgmap v7873: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:57 smithi022 ceph-mon[31360]: pgmap v7873: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:20:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:57 smithi022 ceph-mon[40250]: pgmap v7873: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:20:58.310 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:20:58.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:20:58.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:20:58.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:20:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3813331201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3813331201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:20:59.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3813331201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:20:59 smithi122 ceph-mon[32255]: pgmap v7874: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:20:59 smithi022 ceph-mon[31360]: pgmap v7874: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:20:59 smithi022 ceph-mon[40250]: pgmap v7874: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:00] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:21:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:01 smithi122 ceph-mon[32255]: pgmap v7875: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:01 smithi022 ceph-mon[31360]: pgmap v7875: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:01 smithi022 ceph-mon[40250]: pgmap v7875: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:21:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:21:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:03 smithi022 ceph-mon[31360]: pgmap v7876: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:03 smithi022 ceph-mon[40250]: pgmap v7876: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:04.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:04.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:04.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:03 smithi122 ceph-mon[32255]: pgmap v7876: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3763263228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3763263228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3763263228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:05 smithi122 ceph-mon[32255]: pgmap v7877: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:05 smithi022 ceph-mon[31360]: pgmap v7877: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:05 smithi022 ceph-mon[40250]: pgmap v7877: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:06.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:21:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:21:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:21:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:07 smithi122 ceph-mon[32255]: pgmap v7878: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:07 smithi022 ceph-mon[31360]: pgmap v7878: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:07 smithi022 ceph-mon[40250]: pgmap v7878: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:09.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:09.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:09.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:09.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:10.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:09 smithi122 ceph-mon[32255]: pgmap v7879: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:10.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/767004932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:10.188 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:21:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:09 smithi022 ceph-mon[31360]: pgmap v7879: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/767004932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:09 smithi022 ceph-mon[40250]: pgmap v7879: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/767004932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:11 smithi022 ceph-mon[40250]: pgmap v7880: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:12.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:11 smithi022 ceph-mon[31360]: pgmap v7880: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:11 smithi122 ceph-mon[32255]: pgmap v7880: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:21:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:21:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:13 smithi122 ceph-mon[32255]: pgmap v7881: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:21:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:21:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:21:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[31360]: pgmap v7881: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:21:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:21:14.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:21:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[40250]: pgmap v7881: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:21:14.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:21:14.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:21:14.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:14.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:14.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:14.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:21:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/273461164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:21:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/273461164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:21:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/273461164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:15 smithi022 ceph-mon[31360]: pgmap v7882: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:15 smithi022 ceph-mon[40250]: pgmap v7882: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:15 smithi122 ceph-mon[32255]: pgmap v7882: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:16.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:21:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:21:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:21:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:17 smithi022 ceph-mon[31360]: pgmap v7883: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:17 smithi022 ceph-mon[40250]: pgmap v7883: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:17 smithi122 ceph-mon[32255]: pgmap v7883: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:19.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:20.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:20.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:20.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:20.159 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:19 smithi022 ceph-mon[31360]: pgmap v7884: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:20.159 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:19 smithi022 ceph-mon[40250]: pgmap v7884: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:19 smithi122 ceph-mon[32255]: pgmap v7884: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:21:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/518399154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:21.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/518399154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/518399154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:21 smithi022 ceph-mon[31360]: pgmap v7885: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:21 smithi022 ceph-mon[40250]: pgmap v7885: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:21 smithi122 ceph-mon[32255]: pgmap v7885: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:21:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:21:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:23 smithi022 ceph-mon[31360]: pgmap v7886: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:23 smithi022 ceph-mon[40250]: pgmap v7886: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:23 smithi122 ceph-mon[32255]: pgmap v7886: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:25.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:25.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:25.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:25.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:25 smithi022 ceph-mon[31360]: pgmap v7887: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1034634539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:25 smithi022 ceph-mon[40250]: pgmap v7887: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1034634539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:25 smithi122 ceph-mon[32255]: pgmap v7887: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1034634539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:26.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:21:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:21:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:21:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:27 smithi022 ceph-mon[40250]: pgmap v7888: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:28.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:28.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:27 smithi022 ceph-mon[31360]: pgmap v7888: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:27 smithi122 ceph-mon[32255]: pgmap v7888: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:29 smithi122 ceph-mon[32255]: pgmap v7889: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:29 smithi022 ceph-mon[31360]: pgmap v7889: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:29 smithi022 ceph-mon[40250]: pgmap v7889: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:21:30.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:30.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:30.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:30.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:31 smithi122 ceph-mon[32255]: pgmap v7890: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3043094697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:31 smithi022 ceph-mon[40250]: pgmap v7890: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3043094697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:32.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:31 smithi022 ceph-mon[31360]: pgmap v7890: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:32.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3043094697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:21:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:21:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:33 smithi122 ceph-mon[32255]: pgmap v7891: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:21:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:21:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:21:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:21:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[31360]: pgmap v7891: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:21:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:21:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:21:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:21:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[40250]: pgmap v7891: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:21:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:21:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:21:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:21:35.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:35 smithi122 ceph-mon[32255]: pgmap v7892: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:36.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:36.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:36.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:36.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:35 smithi022 ceph-mon[40250]: pgmap v7892: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:36.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:35 smithi022 ceph-mon[31360]: pgmap v7892: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:36.843 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:21:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:21:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3155489136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3155489136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3155489136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:37 smithi122 ceph-mon[32255]: pgmap v7893: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:37 smithi022 ceph-mon[31360]: pgmap v7893: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:38.281 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:21:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:37 smithi022 ceph-mon[40250]: pgmap v7893: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:39 smithi122 ceph-mon[32255]: pgmap v7894: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:39 smithi022 ceph-mon[31360]: pgmap v7894: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:39 smithi022 ceph-mon[40250]: pgmap v7894: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:40] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:21:41.277 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:41.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:41.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:41.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:41 smithi022 ceph-mon[31360]: pgmap v7895: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/75208666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:41 smithi022 ceph-mon[40250]: pgmap v7895: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/75208666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:41 smithi122 ceph-mon[32255]: pgmap v7895: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/75208666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:21:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:21:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:43 smithi122 ceph-mon[32255]: pgmap v7896: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:43 smithi022 ceph-mon[31360]: pgmap v7896: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:43 smithi022 ceph-mon[40250]: pgmap v7896: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:45 smithi122 ceph-mon[32255]: pgmap v7897: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:45 smithi022 ceph-mon[31360]: pgmap v7897: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:45 smithi022 ceph-mon[40250]: pgmap v7897: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:46.645 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:46.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:21:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:21:47.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:47.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:47.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:21:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:47 smithi122 ceph-mon[32255]: pgmap v7898: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2315100396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:47 smithi022 ceph-mon[31360]: pgmap v7898: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:48.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2315100396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:47 smithi022 ceph-mon[40250]: pgmap v7898: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2315100396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:49 smithi122 ceph-mon[32255]: pgmap v7899: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:49 smithi022 ceph-mon[31360]: pgmap v7899: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:49 smithi022 ceph-mon[40250]: pgmap v7899: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:50] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:21:52.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:51 smithi122 ceph-mon[32255]: pgmap v7900: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:51 smithi022 ceph-mon[31360]: pgmap v7900: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:51 smithi022 ceph-mon[40250]: pgmap v7900: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:52.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:52.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:52.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:21:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:21:53.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1422465651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1422465651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1422465651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:53 smithi122 ceph-mon[32255]: pgmap v7901: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:53 smithi022 ceph-mon[31360]: pgmap v7901: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:53 smithi022 ceph-mon[40250]: pgmap v7901: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:55 smithi122 ceph-mon[32255]: pgmap v7902: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:55 smithi022 ceph-mon[31360]: pgmap v7902: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:55 smithi022 ceph-mon[40250]: pgmap v7902: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:21:56.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:21:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:21:57.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:21:57.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:21:57.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:21:57.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:21:57.758 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:21:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:57 smithi022 ceph-mon[31360]: pgmap v7903: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301643235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:58.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:58.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:57 smithi022 ceph-mon[40250]: pgmap v7903: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:58.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301643235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:21:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:21:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:57 smithi122 ceph-mon[32255]: pgmap v7903: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:21:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301643235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:00.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:21:59 smithi122 ceph-mon[32255]: pgmap v7904: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:21:59 smithi022 ceph-mon[31360]: pgmap v7904: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:21:59 smithi022 ceph-mon[40250]: pgmap v7904: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:00] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:22:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:01 smithi122 ceph-mon[32255]: pgmap v7905: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:01 smithi022 ceph-mon[31360]: pgmap v7905: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:01 smithi022 ceph-mon[40250]: pgmap v7905: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:22:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:22:02.759 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:03.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:03.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:03.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:03 smithi122 ceph-mon[32255]: pgmap v7906: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2510848067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:03 smithi022 ceph-mon[31360]: pgmap v7906: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2510848067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:03 smithi022 ceph-mon[40250]: pgmap v7906: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2510848067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:05 smithi122 ceph-mon[32255]: pgmap v7907: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:05 smithi022 ceph-mon[31360]: pgmap v7907: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:05 smithi022 ceph-mon[40250]: pgmap v7907: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:06.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:22:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:22:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:22:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:08.131 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:08.131 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:07 smithi022 ceph-mon[31360]: pgmap v7908: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:08.132 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:08.132 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:07 smithi022 ceph-mon[40250]: pgmap v7908: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:07 smithi122 ceph-mon[32255]: pgmap v7908: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:08.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:08.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:08.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1835967456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1835967456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:09.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1835967456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:09 smithi122 ceph-mon[32255]: pgmap v7909: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:09 smithi022 ceph-mon[31360]: pgmap v7909: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:09 smithi022 ceph-mon[40250]: pgmap v7909: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:22:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:11 smithi122 ceph-mon[32255]: pgmap v7910: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:11 smithi022 ceph-mon[31360]: pgmap v7910: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:11 smithi022 ceph-mon[40250]: pgmap v7910: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:22:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:22:13.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:13.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:13 smithi122 ceph-mon[32255]: pgmap v7911: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:13 smithi022 ceph-mon[31360]: pgmap v7911: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:13 smithi022 ceph-mon[40250]: pgmap v7911: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2158463471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:22:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:22:15.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:22:15.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:22:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2158463471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:22:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:22:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:22:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:22:15.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2158463471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:15.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:22:15.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:22:15.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:22:15.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:22:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:15 smithi122 ceph-mon[32255]: pgmap v7912: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:15 smithi022 ceph-mon[31360]: pgmap v7912: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:15 smithi022 ceph-mon[40250]: pgmap v7912: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:16.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:22:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:22:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:22:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:17 smithi122 ceph-mon[32255]: pgmap v7913: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:17 smithi022 ceph-mon[31360]: pgmap v7913: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:17 smithi022 ceph-mon[40250]: pgmap v7913: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:18.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:19.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:19.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:19.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:19 smithi122 ceph-mon[32255]: pgmap v7914: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1880909609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:19 smithi022 ceph-mon[31360]: pgmap v7914: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1880909609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:19 smithi022 ceph-mon[40250]: pgmap v7914: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1880909609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:22:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:21 smithi122 ceph-mon[32255]: pgmap v7915: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:21 smithi022 ceph-mon[31360]: pgmap v7915: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:21 smithi022 ceph-mon[40250]: pgmap v7915: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:22:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:22:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:23 smithi122 ceph-mon[32255]: pgmap v7916: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:24.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:24.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:23 smithi022 ceph-mon[40250]: pgmap v7916: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:24.241 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:23 smithi022 ceph-mon[31360]: pgmap v7916: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:24.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:24.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:24.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2616588884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2616588884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2616588884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:25 smithi122 ceph-mon[32255]: pgmap v7917: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:25 smithi022 ceph-mon[31360]: pgmap v7917: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:25 smithi022 ceph-mon[40250]: pgmap v7917: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:26.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:22:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:22:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:22:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:27 smithi122 ceph-mon[32255]: pgmap v7918: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:27 smithi022 ceph-mon[31360]: pgmap v7918: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:27 smithi022 ceph-mon[40250]: pgmap v7918: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:29.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:29.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:29.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:29.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:29.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:29 smithi022 ceph-mon[31360]: pgmap v7919: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:29.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:29 smithi022 ceph-mon[40250]: pgmap v7919: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:29 smithi122 ceph-mon[32255]: pgmap v7919: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:22:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1714419068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:31.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1714419068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1714419068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:31 smithi122 ceph-mon[32255]: pgmap v7920: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:31 smithi022 ceph-mon[31360]: pgmap v7920: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:31 smithi022 ceph-mon[40250]: pgmap v7920: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:22:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:22:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:33 smithi122 ceph-mon[32255]: pgmap v7921: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:22:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:22:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:22:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:22:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[31360]: pgmap v7921: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:22:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:22:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:22:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:22:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[40250]: pgmap v7921: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:22:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:22:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:22:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:22:34.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:35.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:35.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:35.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:35 smithi122 ceph-mon[32255]: pgmap v7922: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1587558981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:35 smithi022 ceph-mon[31360]: pgmap v7922: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1587558981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:35 smithi022 ceph-mon[40250]: pgmap v7922: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1587558981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:36.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:22:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:22:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:22:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:37 smithi122 ceph-mon[32255]: pgmap v7923: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:37 smithi022 ceph-mon[31360]: pgmap v7923: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:37 smithi022 ceph-mon[40250]: pgmap v7923: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:39 smithi122 ceph-mon[32255]: pgmap v7924: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:39 smithi022 ceph-mon[31360]: pgmap v7924: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:39 smithi022 ceph-mon[40250]: pgmap v7924: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:40.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:40] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:22:40.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:40.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:40.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3581060987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3581060987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3581060987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:41 smithi122 ceph-mon[32255]: pgmap v7925: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:41 smithi022 ceph-mon[31360]: pgmap v7925: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:41 smithi022 ceph-mon[40250]: pgmap v7925: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:22:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:22:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:43 smithi122 ceph-mon[32255]: pgmap v7926: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:43 smithi022 ceph-mon[31360]: pgmap v7926: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:43 smithi022 ceph-mon[40250]: pgmap v7926: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:45.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:45 smithi022 ceph-mon[31360]: pgmap v7927: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:45 smithi022 ceph-mon[40250]: pgmap v7927: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:46.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:46.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:46.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:45 smithi122 ceph-mon[32255]: pgmap v7927: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:46.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:22:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:22:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3898358974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3898358974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:47.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3898358974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:22:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:48.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:47 smithi122 ceph-mon[32255]: pgmap v7928: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:47 smithi022 ceph-mon[31360]: pgmap v7928: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:47 smithi022 ceph-mon[40250]: pgmap v7928: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:49 smithi122 ceph-mon[32255]: pgmap v7929: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:49 smithi022 ceph-mon[31360]: pgmap v7929: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:49 smithi022 ceph-mon[40250]: pgmap v7929: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:50] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:22:51.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:51.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:51.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:51.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:51 smithi122 ceph-mon[32255]: pgmap v7930: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/975327021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:51 smithi022 ceph-mon[31360]: pgmap v7930: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/975327021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:51 smithi022 ceph-mon[40250]: pgmap v7930: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/975327021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:22:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:22:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:53 smithi122 ceph-mon[32255]: pgmap v7931: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:53 smithi022 ceph-mon[31360]: pgmap v7931: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:53 smithi022 ceph-mon[40250]: pgmap v7931: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:55 smithi122 ceph-mon[32255]: pgmap v7932: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:55 smithi022 ceph-mon[31360]: pgmap v7932: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:55 smithi022 ceph-mon[40250]: pgmap v7932: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:22:56.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:22:56.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:22:56.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:22:56.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:22:56.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:22:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:22:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1548717635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1548717635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1548717635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:22:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:22:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:57 smithi122 ceph-mon[32255]: pgmap v7933: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:57 smithi022 ceph-mon[31360]: pgmap v7933: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:22:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:22:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:57 smithi022 ceph-mon[40250]: pgmap v7933: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:22:59 smithi122 ceph-mon[32255]: pgmap v7934: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:22:59 smithi022 ceph-mon[31360]: pgmap v7934: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:22:59 smithi022 ceph-mon[40250]: pgmap v7934: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:00] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:23:01.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:01 smithi122 ceph-mon[32255]: pgmap v7935: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:02.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:02.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:02.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:02.200 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:01 smithi022 ceph-mon[31360]: pgmap v7935: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:02.200 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:01 smithi022 ceph-mon[40250]: pgmap v7935: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:23:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:23:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2540544011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2540544011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2540544011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:03 smithi122 ceph-mon[32255]: pgmap v7936: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:03 smithi022 ceph-mon[31360]: pgmap v7936: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:03 smithi022 ceph-mon[40250]: pgmap v7936: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:05 smithi122 ceph-mon[32255]: pgmap v7937: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:05 smithi022 ceph-mon[31360]: pgmap v7937: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:05 smithi022 ceph-mon[40250]: pgmap v7937: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:06.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:23:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:23:07.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:23:07.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:07.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:07.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:07 smithi122 ceph-mon[32255]: pgmap v7938: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/172817586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:07 smithi022 ceph-mon[31360]: pgmap v7938: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:08.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/172817586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:07 smithi022 ceph-mon[40250]: pgmap v7938: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/172817586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:09 smithi122 ceph-mon[32255]: pgmap v7939: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:09 smithi022 ceph-mon[31360]: pgmap v7939: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:09 smithi022 ceph-mon[40250]: pgmap v7939: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:10] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:23:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:11 smithi122 ceph-mon[32255]: pgmap v7940: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:11 smithi022 ceph-mon[31360]: pgmap v7940: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:11 smithi022 ceph-mon[40250]: pgmap v7940: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:12.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:23:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:23:12.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:12.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:12.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:13 smithi122 ceph-mon[32255]: pgmap v7941: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2323627099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:13 smithi022 ceph-mon[31360]: pgmap v7941: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2323627099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:13 smithi022 ceph-mon[40250]: pgmap v7941: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2323627099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:23:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:23:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:23:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:23:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:23:15.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:23:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:23:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:23:15.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:23:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:15 smithi122 ceph-mon[32255]: pgmap v7942: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:15 smithi022 ceph-mon[31360]: pgmap v7942: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:15 smithi022 ceph-mon[40250]: pgmap v7942: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:16.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:23:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:23:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:23:17.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:18.185 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:18.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:17 smithi022 ceph-mon[31360]: pgmap v7943: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:18.186 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:18.187 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:17 smithi022 ceph-mon[40250]: pgmap v7943: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:17 smithi122 ceph-mon[32255]: pgmap v7943: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:18.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:18.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:18.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3438255303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3438255303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:19.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3438255303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:19.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:23:19.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:19.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:20] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:23:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:20 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3198M 2021-12-15T22:23:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:20 smithi122 ceph-mon[32255]: pgmap v7944: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:20 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3198M 2021-12-15T22:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:20 smithi022 ceph-mon[31360]: pgmap v7944: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:20 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3198M 2021-12-15T22:23:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:20 smithi022 ceph-mon[40250]: pgmap v7944: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:23:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:22 smithi122 ceph-mon[32255]: pgmap v7945: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:23:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:23:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:22 smithi022 ceph-mon[31360]: pgmap v7945: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:22 smithi022 ceph-mon[40250]: pgmap v7945: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:23.305 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:23.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:23.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:23.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:24 smithi122 ceph-mon[32255]: pgmap v7946: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:24.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1640552399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:24 smithi022 ceph-mon[31360]: pgmap v7946: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1640552399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:24 smithi022 ceph-mon[40250]: pgmap v7946: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1640552399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:26 smithi122 ceph-mon[32255]: pgmap v7947: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:26 smithi022 ceph-mon[31360]: pgmap v7947: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:26 smithi022 ceph-mon[40250]: pgmap v7947: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:26.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:23:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:23:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:23:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:28 smithi122 ceph-mon[32255]: pgmap v7948: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:28 smithi022 ceph-mon[31360]: pgmap v7948: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:28 smithi022 ceph-mon[40250]: pgmap v7948: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:28.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:29.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:29.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:29 smithi122 ceph-mon[32255]: pgmap v7949: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1981633598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:29 smithi022 ceph-mon[31360]: pgmap v7949: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1981633598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:29 smithi022 ceph-mon[40250]: pgmap v7949: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1981633598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:30] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:23:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:31 smithi122 ceph-mon[32255]: pgmap v7950: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:31 smithi022 ceph-mon[31360]: pgmap v7950: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:31 smithi022 ceph-mon[40250]: pgmap v7950: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:23:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:23:34.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:33 smithi122 ceph-mon[32255]: pgmap v7951: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:23:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:23:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:23:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:23:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[31360]: pgmap v7951: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:23:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:23:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:23:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:23:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[40250]: pgmap v7951: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:23:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:23:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:23:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:23:34.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:34.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:34.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3015161078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3015161078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3015161078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:35 smithi122 ceph-mon[32255]: pgmap v7952: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:35 smithi022 ceph-mon[31360]: pgmap v7952: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:35 smithi022 ceph-mon[40250]: pgmap v7952: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:36.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:23:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:23:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:23:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:37 smithi122 ceph-mon[32255]: pgmap v7953: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:37 smithi022 ceph-mon[31360]: pgmap v7953: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:37 smithi022 ceph-mon[40250]: pgmap v7953: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:39.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:39.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:39.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:39.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:39 smithi022 ceph-mon[31360]: pgmap v7954: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/280587286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:39 smithi022 ceph-mon[40250]: pgmap v7954: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/280587286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:39 smithi122 ceph-mon[32255]: pgmap v7954: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/280587286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:40] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:23:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:41 smithi122 ceph-mon[32255]: pgmap v7955: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:41 smithi022 ceph-mon[31360]: pgmap v7955: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:41 smithi022 ceph-mon[40250]: pgmap v7955: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:23:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:23:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:43 smithi122 ceph-mon[32255]: pgmap v7956: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:43 smithi022 ceph-mon[31360]: pgmap v7956: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:43 smithi022 ceph-mon[40250]: pgmap v7956: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:44.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:45.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:45 smithi122 ceph-mon[32255]: pgmap v7957: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1334509404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:45 smithi022 ceph-mon[31360]: pgmap v7957: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:46.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1334509404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:45 smithi022 ceph-mon[40250]: pgmap v7957: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1334509404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:46.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:23:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:23:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:23:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:48.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:47 smithi122 ceph-mon[32255]: pgmap v7958: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:47 smithi022 ceph-mon[31360]: pgmap v7958: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:47 smithi022 ceph-mon[40250]: pgmap v7958: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:50.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:50.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:49 smithi022 ceph-mon[31360]: pgmap v7959: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:50.148 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:49 smithi022 ceph-mon[40250]: pgmap v7959: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:49 smithi122 ceph-mon[32255]: pgmap v7959: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:50] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:23:50.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:50.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:50.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:51.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1228832929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1228832929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1228832929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:52.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:51 smithi122 ceph-mon[32255]: pgmap v7960: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:51 smithi022 ceph-mon[31360]: pgmap v7960: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:51 smithi022 ceph-mon[40250]: pgmap v7960: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:23:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:23:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:53 smithi122 ceph-mon[32255]: pgmap v7961: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:53 smithi022 ceph-mon[31360]: pgmap v7961: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:53 smithi022 ceph-mon[40250]: pgmap v7961: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:55.516 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:23:55.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:23:55.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:23:55.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:23:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:55 smithi122 ceph-mon[32255]: pgmap v7962: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1956871208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:55 smithi022 ceph-mon[31360]: pgmap v7962: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1956871208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:55 smithi022 ceph-mon[40250]: pgmap v7962: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:23:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1956871208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:23:56.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:23:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:23:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:23:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:57 smithi122 ceph-mon[32255]: pgmap v7963: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:57 smithi022 ceph-mon[31360]: pgmap v7963: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:23:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:23:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:57 smithi022 ceph-mon[40250]: pgmap v7963: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:23:59 smithi122 ceph-mon[32255]: pgmap v7964: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:23:59 smithi022 ceph-mon[31360]: pgmap v7964: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:00.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:23:59 smithi022 ceph-mon[40250]: pgmap v7964: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:00] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:24:00.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:01.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:01 smithi122 ceph-mon[32255]: pgmap v7965: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3060017311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:01 smithi022 ceph-mon[31360]: pgmap v7965: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3060017311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:01 smithi022 ceph-mon[40250]: pgmap v7965: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3060017311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:24:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:24:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:03 smithi122 ceph-mon[32255]: pgmap v7966: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:03 smithi022 ceph-mon[31360]: pgmap v7966: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:03 smithi022 ceph-mon[40250]: pgmap v7966: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:05 smithi122 ceph-mon[32255]: pgmap v7967: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:06.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:06.260 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:05 smithi022 ceph-mon[31360]: pgmap v7967: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:06.260 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:05 smithi022 ceph-mon[40250]: pgmap v7967: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:06.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:06.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:06.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:06.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:24:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:24:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1992094253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1992094253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1992094253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:24:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:07 smithi122 ceph-mon[32255]: pgmap v7968: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:07 smithi022 ceph-mon[31360]: pgmap v7968: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:07 smithi022 ceph-mon[40250]: pgmap v7968: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:09 smithi122 ceph-mon[32255]: pgmap v7969: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:09 smithi022 ceph-mon[31360]: pgmap v7969: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:09 smithi022 ceph-mon[40250]: pgmap v7969: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:10] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T22:24:11.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:11.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:11.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:11.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:11 smithi122 ceph-mon[32255]: pgmap v7970: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:11 smithi022 ceph-mon[31360]: pgmap v7970: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:11 smithi022 ceph-mon[40250]: pgmap v7970: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:24:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:24:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/764600236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/764600236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/764600236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:13 smithi122 ceph-mon[32255]: pgmap v7971: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:13 smithi022 ceph-mon[31360]: pgmap v7971: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:13 smithi022 ceph-mon[40250]: pgmap v7971: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:15 smithi122 ceph-mon[32255]: pgmap v7972: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:15 smithi022 ceph-mon[31360]: pgmap v7972: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:15 smithi022 ceph-mon[40250]: pgmap v7972: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:16.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:24:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:24:16.999 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:17.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:17.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:17.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:24:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:17 smithi022 ceph-mon[31360]: pgmap v7973: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2803273257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:17 smithi022 ceph-mon[40250]: pgmap v7973: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2803273257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:17 smithi122 ceph-mon[32255]: pgmap v7973: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2803273257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:19 smithi122 ceph-mon[32255]: pgmap v7974: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:24:20.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:24:20.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:24:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[31360]: pgmap v7974: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:24:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:24:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:24:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[40250]: pgmap v7974: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:24:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:24:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:24:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:20] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T22:24:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:24:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:24:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:24:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:21 smithi122 ceph-mon[32255]: pgmap v7975: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:21 smithi022 ceph-mon[31360]: pgmap v7975: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:21 smithi022 ceph-mon[40250]: pgmap v7975: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:22.371 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:24:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:24:22.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:22.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:22.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1956972758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:23.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1956972758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1956972758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:23 smithi122 ceph-mon[32255]: pgmap v7976: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:23 smithi022 ceph-mon[31360]: pgmap v7976: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:23 smithi022 ceph-mon[40250]: pgmap v7976: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:25 smithi122 ceph-mon[32255]: pgmap v7977: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:25 smithi022 ceph-mon[31360]: pgmap v7977: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:25 smithi022 ceph-mon[40250]: pgmap v7977: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:26.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:24:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:24:27.740 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:27.740 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:24:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:27 smithi022 ceph-mon[31360]: pgmap v7978: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:28.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:27 smithi022 ceph-mon[40250]: pgmap v7978: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:28.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:28.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:28.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:27 smithi122 ceph-mon[32255]: pgmap v7978: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2296803500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2296803500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2296803500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:29 smithi022 ceph-mon[31360]: pgmap v7979: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:29 smithi022 ceph-mon[40250]: pgmap v7979: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:29 smithi122 ceph-mon[32255]: pgmap v7979: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:30] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T22:24:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:31 smithi022 ceph-mon[31360]: pgmap v7980: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:31 smithi022 ceph-mon[40250]: pgmap v7980: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:31 smithi122 ceph-mon[32255]: pgmap v7980: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:24:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:24:33.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:33.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:33.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:33.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[31360]: pgmap v7981: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:24:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:24:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:24:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:24:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3205111104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[40250]: pgmap v7981: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:24:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:24:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:24:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:24:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3205111104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:33 smithi122 ceph-mon[32255]: pgmap v7981: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:24:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:24:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:24:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:24:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3205111104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:35 smithi022 ceph-mon[31360]: pgmap v7982: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:35 smithi022 ceph-mon[40250]: pgmap v7982: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:35 smithi122 ceph-mon[32255]: pgmap v7982: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:36.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:24:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:24:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:24:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:37 smithi022 ceph-mon[31360]: pgmap v7983: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:37 smithi022 ceph-mon[40250]: pgmap v7983: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:37 smithi122 ceph-mon[32255]: pgmap v7983: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:38.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:38.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:38.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:38.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2469382453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:39.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2469382453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2469382453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:39 smithi022 ceph-mon[31360]: pgmap v7984: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:39 smithi022 ceph-mon[40250]: pgmap v7984: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:39 smithi122 ceph-mon[32255]: pgmap v7984: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:40] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:24:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:41 smithi022 ceph-mon[31360]: pgmap v7985: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:41 smithi022 ceph-mon[40250]: pgmap v7985: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:41 smithi122 ceph-mon[32255]: pgmap v7985: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:24:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:24:43.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:44.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:44.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:44.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:44.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:43 smithi022 ceph-mon[31360]: pgmap v7986: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:44.232 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:43 smithi022 ceph-mon[40250]: pgmap v7986: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:43 smithi122 ceph-mon[32255]: pgmap v7986: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2001498521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2001498521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2001498521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:45 smithi022 ceph-mon[31360]: pgmap v7987: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:45 smithi022 ceph-mon[40250]: pgmap v7987: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:45 smithi122 ceph-mon[32255]: pgmap v7987: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:46.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:24:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:24:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:24:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:47 smithi022 ceph-mon[31360]: pgmap v7988: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:47 smithi022 ceph-mon[40250]: pgmap v7988: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:47 smithi122 ceph-mon[32255]: pgmap v7988: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:49.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:49.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:49.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:49.596 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:49 smithi122 ceph-mon[32255]: pgmap v7989: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1205351058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:49 smithi022 ceph-mon[31360]: pgmap v7989: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1205351058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:49 smithi022 ceph-mon[40250]: pgmap v7989: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1205351058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:24:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:51 smithi022 ceph-mon[31360]: pgmap v7990: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:51 smithi022 ceph-mon[40250]: pgmap v7990: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:51 smithi122 ceph-mon[32255]: pgmap v7990: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:24:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:24:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:53 smithi022 ceph-mon[31360]: pgmap v7991: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:53 smithi022 ceph-mon[40250]: pgmap v7991: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:53 smithi122 ceph-mon[32255]: pgmap v7991: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:54.597 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:24:54.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:24:54.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:24:54.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:24:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:55 smithi022 ceph-mon[31360]: pgmap v7992: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1000081559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:55 smithi022 ceph-mon[40250]: pgmap v7992: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1000081559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:55 smithi122 ceph-mon[32255]: pgmap v7992: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:24:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1000081559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:24:56.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:24:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:24:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:24:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:57 smithi022 ceph-mon[40250]: pgmap v7993: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:58.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:57 smithi022 ceph-mon[31360]: pgmap v7993: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:24:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:57 smithi122 ceph-mon[32255]: pgmap v7993: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:24:59.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:24:59 smithi122 ceph-mon[32255]: pgmap v7994: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:24:59 smithi022 ceph-mon[31360]: pgmap v7994: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:24:59 smithi022 ceph-mon[40250]: pgmap v7994: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:00.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:00.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:00.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:25:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3363019113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3363019113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3363019113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:01 smithi122 ceph-mon[32255]: pgmap v7995: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:01 smithi022 ceph-mon[31360]: pgmap v7995: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:01 smithi022 ceph-mon[40250]: pgmap v7995: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:25:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:25:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:03 smithi022 ceph-mon[31360]: pgmap v7996: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:03 smithi022 ceph-mon[40250]: pgmap v7996: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:03 smithi122 ceph-mon[32255]: pgmap v7996: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:05.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:05.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:05.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:05 smithi022 ceph-mon[31360]: pgmap v7997: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/782130278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:05 smithi022 ceph-mon[40250]: pgmap v7997: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/782130278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:05 smithi122 ceph-mon[32255]: pgmap v7997: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/782130278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:06.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:25:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:25:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:25:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:07 smithi022 ceph-mon[31360]: pgmap v7998: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:07 smithi022 ceph-mon[40250]: pgmap v7998: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:07 smithi122 ceph-mon[32255]: pgmap v7998: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:09 smithi122 ceph-mon[32255]: pgmap v7999: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:09 smithi022 ceph-mon[31360]: pgmap v7999: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:09 smithi022 ceph-mon[40250]: pgmap v7999: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:25:10.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:11.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:11.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:11.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:11 smithi022 ceph-mon[31360]: pgmap v8000: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1100566320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:11 smithi022 ceph-mon[40250]: pgmap v8000: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1100566320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:11 smithi122 ceph-mon[32255]: pgmap v8000: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1100566320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:25:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:25:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:13 smithi022 ceph-mon[31360]: pgmap v8001: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:13 smithi022 ceph-mon[40250]: pgmap v8001: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:13 smithi122 ceph-mon[32255]: pgmap v8001: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:16.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:15 smithi022 ceph-mon[31360]: pgmap v8002: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:15 smithi022 ceph-mon[40250]: pgmap v8002: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:15 smithi122 ceph-mon[32255]: pgmap v8002: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:16.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:16.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:16.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:16.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:25:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:25:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1945175113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1945175113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1945175113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:25:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:17 smithi022 ceph-mon[31360]: pgmap v8003: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:17 smithi022 ceph-mon[40250]: pgmap v8003: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:17 smithi122 ceph-mon[32255]: pgmap v8003: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:19 smithi122 ceph-mon[32255]: pgmap v8004: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:20.174 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:20] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:25:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:19 smithi022 ceph-mon[31360]: pgmap v8004: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:19 smithi022 ceph-mon[40250]: pgmap v8004: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:25:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:25:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:25:21.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:25:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:25:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:25:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:25:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:25:21.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:25:21.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:25:21.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:25:21.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:25:21.452 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:21.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:21.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:21.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:21 smithi122 ceph-mon[32255]: pgmap v8005: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/543810263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:21 smithi022 ceph-mon[31360]: pgmap v8005: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/543810263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:21 smithi022 ceph-mon[40250]: pgmap v8005: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/543810263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:25:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:25:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:23 smithi022 ceph-mon[31360]: pgmap v8006: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:23 smithi022 ceph-mon[40250]: pgmap v8006: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:23 smithi122 ceph-mon[32255]: pgmap v8006: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:25 smithi022 ceph-mon[31360]: pgmap v8007: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:25 smithi022 ceph-mon[40250]: pgmap v8007: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:25 smithi122 ceph-mon[32255]: pgmap v8007: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:26.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:27.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:25:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:25:27.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:27.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:27.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:25:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:27 smithi022 ceph-mon[31360]: pgmap v8008: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3962155372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:27 smithi022 ceph-mon[40250]: pgmap v8008: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3962155372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:27 smithi122 ceph-mon[32255]: pgmap v8008: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:28.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3962155372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:29 smithi122 ceph-mon[32255]: pgmap v8009: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:29 smithi022 ceph-mon[31360]: pgmap v8009: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:29 smithi022 ceph-mon[40250]: pgmap v8009: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:30] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:25:32.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:31 smithi022 ceph-mon[31360]: pgmap v8010: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:31 smithi022 ceph-mon[40250]: pgmap v8010: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:31 smithi122 ceph-mon[32255]: pgmap v8010: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:25:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:25:33.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3511998930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3511998930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3511998930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[40250]: pgmap v8011: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:25:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:25:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:25:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:25:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[31360]: pgmap v8011: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:25:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:25:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:25:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:25:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:33 smithi122 ceph-mon[32255]: pgmap v8011: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:25:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:25:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:25:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:25:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:35 smithi022 ceph-mon[31360]: pgmap v8012: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:35 smithi022 ceph-mon[40250]: pgmap v8012: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:35 smithi122 ceph-mon[32255]: pgmap v8012: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:37.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:25:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:25:37.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:25:37.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:37.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:37.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:37 smithi022 ceph-mon[31360]: pgmap v8013: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:38.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/657882594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:37 smithi022 ceph-mon[40250]: pgmap v8013: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:38.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/657882594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:37 smithi122 ceph-mon[32255]: pgmap v8013: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:38.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/657882594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:39 smithi122 ceph-mon[32255]: pgmap v8014: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:39 smithi022 ceph-mon[31360]: pgmap v8014: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:39 smithi022 ceph-mon[40250]: pgmap v8014: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:40] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:25:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:41 smithi022 ceph-mon[31360]: pgmap v8015: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:41 smithi022 ceph-mon[40250]: pgmap v8015: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:41 smithi122 ceph-mon[32255]: pgmap v8015: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:25:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:25:42.933 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:43 smithi022 ceph-mon[31360]: pgmap v8016: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1424063511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:43 smithi022 ceph-mon[40250]: pgmap v8016: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1424063511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:43 smithi122 ceph-mon[32255]: pgmap v8016: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1424063511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:45 smithi022 ceph-mon[31360]: pgmap v8017: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:45 smithi022 ceph-mon[40250]: pgmap v8017: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:45 smithi122 ceph-mon[32255]: pgmap v8017: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:47.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:25:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:25:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:25:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:47 smithi022 ceph-mon[31360]: pgmap v8018: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:47 smithi022 ceph-mon[40250]: pgmap v8018: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:48.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:47 smithi122 ceph-mon[32255]: pgmap v8018: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:48.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:48.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:48.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:49.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1680280592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1680280592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1680280592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:49 smithi022 ceph-mon[31360]: pgmap v8019: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:49 smithi022 ceph-mon[40250]: pgmap v8019: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:49 smithi122 ceph-mon[32255]: pgmap v8019: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:50] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:25:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:51 smithi022 ceph-mon[31360]: pgmap v8020: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:51 smithi022 ceph-mon[40250]: pgmap v8020: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:51 smithi122 ceph-mon[32255]: pgmap v8020: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:25:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:25:53.677 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:54.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:54.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:54.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:25:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:53 smithi022 ceph-mon[31360]: pgmap v8021: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:53 smithi022 ceph-mon[40250]: pgmap v8021: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:53 smithi122 ceph-mon[32255]: pgmap v8021: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3442702804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3442702804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:55.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3442702804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:25:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:55 smithi022 ceph-mon[31360]: pgmap v8022: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:55 smithi022 ceph-mon[40250]: pgmap v8022: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:55 smithi122 ceph-mon[32255]: pgmap v8022: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:25:57.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:25:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:25:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:25:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:57 smithi022 ceph-mon[31360]: pgmap v8023: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:57 smithi022 ceph-mon[40250]: pgmap v8023: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:25:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:57 smithi122 ceph-mon[32255]: pgmap v8023: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:25:59.048 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:25:59.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:25:59.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:25:59.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:59 smithi122 ceph-mon[32255]: pgmap v8024: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:25:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4151435184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:59 smithi022 ceph-mon[40250]: pgmap v8024: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:25:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4151435184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:00.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:59 smithi022 ceph-mon[31360]: pgmap v8024: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:00.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:25:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4151435184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:00] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:26:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:01 smithi022 ceph-mon[31360]: pgmap v8025: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:01 smithi022 ceph-mon[40250]: pgmap v8025: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:01 smithi122 ceph-mon[32255]: pgmap v8025: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:26:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:26:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:03 smithi022 ceph-mon[31360]: pgmap v8026: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:03 smithi022 ceph-mon[40250]: pgmap v8026: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:04.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:03 smithi122 ceph-mon[32255]: pgmap v8026: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:04.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:04.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:04.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2295831748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2295831748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:05.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2295831748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:05 smithi022 ceph-mon[31360]: pgmap v8027: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:05 smithi022 ceph-mon[40250]: pgmap v8027: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:05 smithi122 ceph-mon[32255]: pgmap v8027: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:07.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:26:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:26:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:26:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:07 smithi022 ceph-mon[31360]: pgmap v8028: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:07 smithi022 ceph-mon[40250]: pgmap v8028: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:07 smithi122 ceph-mon[32255]: pgmap v8028: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:09.788 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:10.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:10.159 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:09 smithi022 ceph-mon[31360]: pgmap v8029: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:10.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:09 smithi022 ceph-mon[40250]: pgmap v8029: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:09 smithi122 ceph-mon[32255]: pgmap v8029: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:10] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:26:11.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/530261259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/530261259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/530261259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:11 smithi122 ceph-mon[32255]: pgmap v8030: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:11 smithi022 ceph-mon[31360]: pgmap v8030: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:11 smithi022 ceph-mon[40250]: pgmap v8030: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:26:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:26:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:13 smithi022 ceph-mon[31360]: pgmap v8031: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:13 smithi022 ceph-mon[40250]: pgmap v8031: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:13 smithi122 ceph-mon[32255]: pgmap v8031: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:15.160 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:15.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:15.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:15.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:15 smithi022 ceph-mon[31360]: pgmap v8032: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/983120822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:15 smithi022 ceph-mon[40250]: pgmap v8032: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/983120822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:15 smithi122 ceph-mon[32255]: pgmap v8032: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/983120822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:17.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:26:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:26:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:26:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:17 smithi022 ceph-mon[31360]: pgmap v8033: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:17 smithi022 ceph-mon[40250]: pgmap v8033: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:17 smithi122 ceph-mon[32255]: pgmap v8033: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:19 smithi122 ceph-mon[32255]: pgmap v8034: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:19 smithi022 ceph-mon[31360]: pgmap v8034: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:19 smithi022 ceph-mon[40250]: pgmap v8034: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:20.424 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:20] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:26:20.533 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:20.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:20.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:20.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:26:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:26:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:26:21.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:26:21.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/969937456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:26:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:26:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:26:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:26:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/969937456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:21.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:26:21.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:26:21.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:26:21.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:26:21.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/969937456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:21 smithi122 ceph-mon[32255]: pgmap v8035: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:21 smithi022 ceph-mon[31360]: pgmap v8035: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:21 smithi022 ceph-mon[40250]: pgmap v8035: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:26:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:26:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:23 smithi022 ceph-mon[31360]: pgmap v8036: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:23 smithi022 ceph-mon[40250]: pgmap v8036: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:23 smithi122 ceph-mon[32255]: pgmap v8036: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:25.901 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:26.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:26.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:26.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:26.275 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:25 smithi022 ceph-mon[31360]: pgmap v8037: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:26.275 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:25 smithi022 ceph-mon[40250]: pgmap v8037: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:25 smithi122 ceph-mon[32255]: pgmap v8037: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1437446163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:27.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:26:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:26:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1437446163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1437446163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:26:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:27 smithi022 ceph-mon[31360]: pgmap v8038: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:27 smithi022 ceph-mon[40250]: pgmap v8038: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:27 smithi122 ceph-mon[32255]: pgmap v8038: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:29 smithi122 ceph-mon[32255]: pgmap v8039: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:29 smithi022 ceph-mon[31360]: pgmap v8039: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:29 smithi022 ceph-mon[40250]: pgmap v8039: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:30] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:26:31.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:31.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:31.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:31.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:31 smithi022 ceph-mon[31360]: pgmap v8040: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2390621452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:31 smithi022 ceph-mon[40250]: pgmap v8040: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2390621452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:31 smithi122 ceph-mon[32255]: pgmap v8040: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2390621452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:26:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:26:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[31360]: pgmap v8041: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:26:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:26:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:26:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:26:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[40250]: pgmap v8041: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:26:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:26:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:26:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:26:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:33 smithi122 ceph-mon[32255]: pgmap v8041: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:26:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:26:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:26:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:26:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:35 smithi022 ceph-mon[31360]: pgmap v8042: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:35 smithi022 ceph-mon[40250]: pgmap v8042: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:35 smithi122 ceph-mon[32255]: pgmap v8042: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:36.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:37.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:37.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:37.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:37.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:26:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:26:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:26:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:37 smithi022 ceph-mon[31360]: pgmap v8043: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/618612105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:37 smithi022 ceph-mon[40250]: pgmap v8043: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/618612105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:37 smithi122 ceph-mon[32255]: pgmap v8043: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/618612105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:39 smithi122 ceph-mon[32255]: pgmap v8044: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:39 smithi022 ceph-mon[31360]: pgmap v8044: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:39 smithi022 ceph-mon[40250]: pgmap v8044: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:40] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:26:42.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:41 smithi022 ceph-mon[31360]: pgmap v8045: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:41 smithi022 ceph-mon[40250]: pgmap v8045: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:41 smithi122 ceph-mon[32255]: pgmap v8045: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:26:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:26:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3291604426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3291604426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3291604426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:43 smithi022 ceph-mon[31360]: pgmap v8046: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:43 smithi022 ceph-mon[40250]: pgmap v8046: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:43 smithi122 ceph-mon[32255]: pgmap v8046: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:45 smithi022 ceph-mon[31360]: pgmap v8047: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:45 smithi022 ceph-mon[40250]: pgmap v8047: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:45 smithi122 ceph-mon[32255]: pgmap v8047: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:47.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:26:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:26:47.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:47.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:47.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:47.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:47.758 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:26:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:47 smithi022 ceph-mon[31360]: pgmap v8048: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:48.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1226611547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:47 smithi022 ceph-mon[40250]: pgmap v8048: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1226611547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:47 smithi122 ceph-mon[32255]: pgmap v8048: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1226611547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:49 smithi122 ceph-mon[32255]: pgmap v8049: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:49 smithi022 ceph-mon[31360]: pgmap v8049: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:49 smithi022 ceph-mon[40250]: pgmap v8049: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:26:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:51 smithi022 ceph-mon[31360]: pgmap v8050: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:51 smithi022 ceph-mon[40250]: pgmap v8050: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:51 smithi122 ceph-mon[32255]: pgmap v8050: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:26:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:26:52.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:53.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:53.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:53.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:53 smithi022 ceph-mon[31360]: pgmap v8051: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2506346534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:53 smithi022 ceph-mon[40250]: pgmap v8051: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2506346534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:53 smithi122 ceph-mon[32255]: pgmap v8051: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2506346534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:55 smithi022 ceph-mon[31360]: pgmap v8052: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:55 smithi022 ceph-mon[40250]: pgmap v8052: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:55 smithi122 ceph-mon[32255]: pgmap v8052: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:26:57.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:26:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:26:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:26:58.130 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:26:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:57 smithi022 ceph-mon[31360]: pgmap v8053: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:57 smithi022 ceph-mon[40250]: pgmap v8053: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:26:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:57 smithi122 ceph-mon[32255]: pgmap v8053: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:26:58.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:26:58.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:26:58.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:26:59.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/57760858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/57760858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:26:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/57760858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:26:59 smithi022 ceph-mon[31360]: pgmap v8054: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:26:59 smithi022 ceph-mon[40250]: pgmap v8054: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:26:59 smithi122 ceph-mon[32255]: pgmap v8054: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:27:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:01 smithi022 ceph-mon[31360]: pgmap v8055: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:01 smithi022 ceph-mon[40250]: pgmap v8055: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:01 smithi122 ceph-mon[32255]: pgmap v8055: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:27:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:27:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:03.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:03.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:03.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:03 smithi022 ceph-mon[31360]: pgmap v8056: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3232678242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:03 smithi022 ceph-mon[40250]: pgmap v8056: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3232678242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:03 smithi122 ceph-mon[32255]: pgmap v8056: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3232678242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:05 smithi022 ceph-mon[31360]: pgmap v8057: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:05 smithi022 ceph-mon[40250]: pgmap v8057: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:05 smithi122 ceph-mon[32255]: pgmap v8057: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:07.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:27:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:27:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:27:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:07 smithi022 ceph-mon[31360]: pgmap v8058: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:07 smithi022 ceph-mon[40250]: pgmap v8058: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:07 smithi122 ceph-mon[32255]: pgmap v8058: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:08.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:09.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:09.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:09.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:09 smithi122 ceph-mon[32255]: pgmap v8059: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/464822389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:09 smithi022 ceph-mon[31360]: pgmap v8059: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/464822389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:09 smithi022 ceph-mon[40250]: pgmap v8059: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/464822389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:27:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:11 smithi022 ceph-mon[31360]: pgmap v8060: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:11 smithi022 ceph-mon[40250]: pgmap v8060: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:11 smithi122 ceph-mon[32255]: pgmap v8060: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:27:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:27:14.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:14.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:13 smithi022 ceph-mon[40250]: pgmap v8061: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:14.242 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:13 smithi022 ceph-mon[31360]: pgmap v8061: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:13 smithi122 ceph-mon[32255]: pgmap v8061: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:27:14.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:14.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:14.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4163927052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4163927052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4163927052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:15 smithi022 ceph-mon[31360]: pgmap v8062: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:15 smithi022 ceph-mon[40250]: pgmap v8062: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:15 smithi122 ceph-mon[32255]: pgmap v8062: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:17.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:27:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:27:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:27:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:17 smithi022 ceph-mon[31360]: pgmap v8063: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:17 smithi022 ceph-mon[40250]: pgmap v8063: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:17 smithi122 ceph-mon[32255]: pgmap v8063: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:19.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:19.905 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:19 smithi022 ceph-mon[40250]: pgmap v8064: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:19.906 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:19 smithi022 ceph-mon[31360]: pgmap v8064: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:19.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:19.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:19.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:19 smithi122 ceph-mon[32255]: pgmap v8064: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:20] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:27:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2286785884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:27:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:27:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:27:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2286785884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:27:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:27:21.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:27:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2286785884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:27:21.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:27:21.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:27:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:21 smithi122 ceph-mon[32255]: pgmap v8065: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:27:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:21 smithi022 ceph-mon[31360]: pgmap v8065: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:27:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:21 smithi022 ceph-mon[40250]: pgmap v8065: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:27:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:27:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:27:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:23 smithi022 ceph-mon[31360]: pgmap v8066: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:23 smithi022 ceph-mon[40250]: pgmap v8066: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:23 smithi122 ceph-mon[32255]: pgmap v8066: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:24.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:25.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:25.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:25.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:25 smithi022 ceph-mon[31360]: pgmap v8067: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:26.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3900383376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:25 smithi022 ceph-mon[40250]: pgmap v8067: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:26.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3900383376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:25 smithi122 ceph-mon[32255]: pgmap v8067: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3900383376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:27.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:27:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:27:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:27:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:27 smithi022 ceph-mon[31360]: pgmap v8068: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:27 smithi022 ceph-mon[40250]: pgmap v8068: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:27 smithi122 ceph-mon[32255]: pgmap v8068: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:29 smithi122 ceph-mon[32255]: pgmap v8069: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:29 smithi022 ceph-mon[31360]: pgmap v8069: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:29 smithi022 ceph-mon[40250]: pgmap v8069: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:30.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:30] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:27:30.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:30.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:30.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2812045809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2812045809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:31.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2812045809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:31 smithi122 ceph-mon[32255]: pgmap v8070: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:31 smithi022 ceph-mon[31360]: pgmap v8070: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:31 smithi022 ceph-mon[40250]: pgmap v8070: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:27:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:27:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[31360]: pgmap v8071: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:27:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:27:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:27:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:27:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[40250]: pgmap v8071: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:27:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:27:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:27:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:27:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:33 smithi122 ceph-mon[32255]: pgmap v8071: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:27:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:27:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:27:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:27:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:36.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:36.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:36.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:35 smithi022 ceph-mon[31360]: pgmap v8072: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:35 smithi022 ceph-mon[40250]: pgmap v8072: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:35 smithi122 ceph-mon[32255]: pgmap v8072: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1445165475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:37.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:27:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:27:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1445165475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1445165475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:27:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:38 smithi022 ceph-mon[31360]: pgmap v8073: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:38 smithi022 ceph-mon[40250]: pgmap v8073: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:38 smithi122 ceph-mon[32255]: pgmap v8073: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:39 smithi122 ceph-mon[32255]: pgmap v8074: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:39 smithi022 ceph-mon[31360]: pgmap v8074: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:39 smithi022 ceph-mon[40250]: pgmap v8074: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:40] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:27:41.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:41.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:41.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:41 smithi022 ceph-mon[31360]: pgmap v8075: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3129668497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:41 smithi022 ceph-mon[40250]: pgmap v8075: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:42.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3129668497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:41 smithi122 ceph-mon[32255]: pgmap v8075: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3129668497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:27:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:27:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:43 smithi022 ceph-mon[31360]: pgmap v8076: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:43 smithi022 ceph-mon[40250]: pgmap v8076: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:43 smithi122 ceph-mon[32255]: pgmap v8076: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:45 smithi022 ceph-mon[31360]: pgmap v8077: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:45 smithi022 ceph-mon[40250]: pgmap v8077: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:45 smithi122 ceph-mon[32255]: pgmap v8077: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:46.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:46.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:46.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:46.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3719965371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:47.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:27:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:27:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3719965371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3719965371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:27:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:48 smithi022 ceph-mon[31360]: pgmap v8078: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:48 smithi022 ceph-mon[40250]: pgmap v8078: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:48 smithi122 ceph-mon[32255]: pgmap v8078: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:49 smithi122 ceph-mon[32255]: pgmap v8079: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:49 smithi022 ceph-mon[31360]: pgmap v8079: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:49 smithi022 ceph-mon[40250]: pgmap v8079: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:27:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:27:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:52.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:52.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:52.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:52.208 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:51 smithi022 ceph-mon[31360]: pgmap v8080: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:52.208 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:51 smithi022 ceph-mon[40250]: pgmap v8080: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:51 smithi122 ceph-mon[32255]: pgmap v8080: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:27:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:27:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1224966388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1224966388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1224966388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:53 smithi022 ceph-mon[31360]: pgmap v8081: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:53 smithi022 ceph-mon[40250]: pgmap v8081: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:54 smithi122 ceph-mon[32255]: pgmap v8081: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:27:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:56 smithi022 ceph-mon[31360]: pgmap v8082: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:27:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:56 smithi022 ceph-mon[40250]: pgmap v8082: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:27:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:56 smithi122 ceph-mon[32255]: pgmap v8082: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:27:57.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:27:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:27:57.209 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:27:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:27:57.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:27:57.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:27:57.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:27:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:58 smithi022 ceph-mon[31360]: pgmap v8083: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:27:58.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3926466601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:58 smithi022 ceph-mon[40250]: pgmap v8083: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:27:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3926466601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:27:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:27:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:58 smithi122 ceph-mon[32255]: pgmap v8083: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:27:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3926466601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:27:59 smithi122 ceph-mon[32255]: pgmap v8084: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:27:59 smithi022 ceph-mon[31360]: pgmap v8084: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:27:59 smithi022 ceph-mon[40250]: pgmap v8084: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:28:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:01 smithi022 ceph-mon[31360]: pgmap v8085: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:01 smithi022 ceph-mon[40250]: pgmap v8085: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:01 smithi122 ceph-mon[32255]: pgmap v8085: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:02.583 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:28:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:28:02.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:02.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:02.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2781544062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2781544062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2781544062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:04 smithi022 ceph-mon[31360]: pgmap v8086: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:04 smithi022 ceph-mon[40250]: pgmap v8086: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:04 smithi122 ceph-mon[32255]: pgmap v8086: 132 pgs: 132 active+clean; 582 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:28:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:06 smithi022 ceph-mon[31360]: pgmap v8087: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:28:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:06 smithi022 ceph-mon[40250]: pgmap v8087: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:28:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:06 smithi122 ceph-mon[32255]: pgmap v8087: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:28:07.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:28:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:28:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:28:07.953 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:08 smithi022 ceph-mon[31360]: pgmap v8088: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:08 smithi022 ceph-mon[40250]: pgmap v8088: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:08 smithi122 ceph-mon[32255]: pgmap v8088: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:09.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2825622508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2825622508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2825622508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:10 smithi022 ceph-mon[31360]: pgmap v8089: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:28:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:10 smithi022 ceph-mon[40250]: pgmap v8089: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:28:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:10] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T22:28:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:10 smithi122 ceph-mon[32255]: pgmap v8089: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:28:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:12 smithi022 ceph-mon[31360]: pgmap v8090: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:12 smithi022 ceph-mon[40250]: pgmap v8090: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:12 smithi122 ceph-mon[32255]: pgmap v8090: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:28:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:28:13.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:13.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:13.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:13.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:14 smithi022 ceph-mon[31360]: pgmap v8091: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3746289523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:14 smithi022 ceph-mon[40250]: pgmap v8091: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3746289523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:14 smithi122 ceph-mon[32255]: pgmap v8091: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:28:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3746289523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:16 smithi022 ceph-mon[31360]: pgmap v8092: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:28:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:16 smithi022 ceph-mon[40250]: pgmap v8092: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:28:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:16 smithi122 ceph-mon[32255]: pgmap v8092: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:28:17.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:28:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:28:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:28:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:18 smithi022 ceph-mon[31360]: pgmap v8093: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:18 smithi022 ceph-mon[40250]: pgmap v8093: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:18 smithi122 ceph-mon[32255]: pgmap v8093: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:18.693 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:19.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:19.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:19.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:19 smithi122 ceph-mon[32255]: pgmap v8094: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2539017218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:19 smithi022 ceph-mon[31360]: pgmap v8094: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2539017218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:19 smithi022 ceph-mon[40250]: pgmap v8094: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2539017218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:28:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[31360]: pgmap v8095: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:28:22.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:28:22.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:28:22.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:28:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[40250]: pgmap v8095: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:28:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:28:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:28:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:28:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:22 smithi122 ceph-mon[32255]: pgmap v8095: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:28:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:28:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:28:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:28:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:28:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:28:24.063 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:24 smithi022 ceph-mon[31360]: pgmap v8096: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:24 smithi022 ceph-mon[40250]: pgmap v8096: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:24.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:24.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:24.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:24 smithi122 ceph-mon[32255]: pgmap v8096: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1359146530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1359146530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1359146530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:26 smithi022 ceph-mon[31360]: pgmap v8097: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:26 smithi022 ceph-mon[40250]: pgmap v8097: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:26 smithi122 ceph-mon[32255]: pgmap v8097: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:27.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:28:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:28:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:28:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:28 smithi022 ceph-mon[31360]: pgmap v8098: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:28 smithi022 ceph-mon[40250]: pgmap v8098: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:28 smithi122 ceph-mon[32255]: pgmap v8098: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:29.437 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:30.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:28:30.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:29 smithi122 ceph-mon[32255]: pgmap v8099: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:30.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/441244537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:29 smithi022 ceph-mon[31360]: pgmap v8099: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/441244537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:29 smithi022 ceph-mon[40250]: pgmap v8099: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/441244537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:32 smithi022 ceph-mon[31360]: pgmap v8100: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:32 smithi022 ceph-mon[40250]: pgmap v8100: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:32 smithi122 ceph-mon[32255]: pgmap v8100: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:28:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:28:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[31360]: pgmap v8101: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:28:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:28:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:28:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:28:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[40250]: pgmap v8101: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:28:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:28:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:28:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:28:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:34 smithi122 ceph-mon[32255]: pgmap v8101: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:28:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:28:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:28:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:28:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:35.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:35.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:35.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:36 smithi022 ceph-mon[31360]: pgmap v8102: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2275333402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:36 smithi022 ceph-mon[40250]: pgmap v8102: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2275333402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:36 smithi122 ceph-mon[32255]: pgmap v8102: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2275333402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:37.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:28:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:28:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:28:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:38 smithi022 ceph-mon[31360]: pgmap v8103: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:38 smithi022 ceph-mon[40250]: pgmap v8103: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:38 smithi122 ceph-mon[32255]: pgmap v8103: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:40.179 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:40.180 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:39 smithi022 ceph-mon[31360]: pgmap v8104: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:40.180 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:39 smithi022 ceph-mon[40250]: pgmap v8104: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:40.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:39 smithi122 ceph-mon[32255]: pgmap v8104: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:40.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:28:40.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:40.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:40.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1804148887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1804148887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1804148887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:42.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:41 smithi122 ceph-mon[32255]: pgmap v8105: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:41 smithi022 ceph-mon[31360]: pgmap v8105: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:41 smithi022 ceph-mon[40250]: pgmap v8105: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:28:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:28:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:44 smithi022 ceph-mon[31360]: pgmap v8106: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:44 smithi022 ceph-mon[40250]: pgmap v8106: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:44 smithi122 ceph-mon[32255]: pgmap v8106: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:45.549 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:45.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:45.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:45.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:46 smithi022 ceph-mon[31360]: pgmap v8107: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2688314539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:46 smithi022 ceph-mon[40250]: pgmap v8107: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2688314539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:46 smithi122 ceph-mon[32255]: pgmap v8107: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2688314539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:47.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:28:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:28:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:28:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:48 smithi022 ceph-mon[31360]: pgmap v8108: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:48 smithi022 ceph-mon[40250]: pgmap v8108: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:48 smithi122 ceph-mon[32255]: pgmap v8108: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:50.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:49 smithi122 ceph-mon[32255]: pgmap v8109: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:50.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:50] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:28:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:49 smithi022 ceph-mon[31360]: pgmap v8109: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:49 smithi022 ceph-mon[40250]: pgmap v8109: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:50.920 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:51.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:51.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:51.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:52 smithi022 ceph-mon[31360]: pgmap v8110: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4056706434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:52 smithi022 ceph-mon[40250]: pgmap v8110: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4056706434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:52 smithi122 ceph-mon[32255]: pgmap v8110: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4056706434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:28:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:28:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:54 smithi022 ceph-mon[31360]: pgmap v8111: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:54 smithi022 ceph-mon[40250]: pgmap v8111: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:54 smithi122 ceph-mon[32255]: pgmap v8111: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:56 smithi022 ceph-mon[31360]: pgmap v8112: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:56 smithi022 ceph-mon[40250]: pgmap v8112: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:56.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:28:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:56 smithi122 ceph-mon[32255]: pgmap v8112: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:28:56.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:28:56.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:28:56.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:28:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1547474564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:57.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:28:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:28:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1547474564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1547474564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:28:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:28:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:58 smithi022 ceph-mon[31360]: pgmap v8113: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:58 smithi022 ceph-mon[40250]: pgmap v8113: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:28:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:28:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:58 smithi122 ceph-mon[32255]: pgmap v8113: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:00.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:28:59 smithi122 ceph-mon[32255]: pgmap v8114: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:00.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:00] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:29:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:28:59 smithi022 ceph-mon[31360]: pgmap v8114: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:28:59 smithi022 ceph-mon[40250]: pgmap v8114: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:01.658 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:02.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:02 smithi022 ceph-mon[40250]: pgmap v8115: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:02.017 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/492912480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:02 smithi022 ceph-mon[31360]: pgmap v8115: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/492912480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:02 smithi122 ceph-mon[32255]: pgmap v8115: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/492912480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:29:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:29:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:04 smithi022 ceph-mon[40250]: pgmap v8116: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:04 smithi022 ceph-mon[31360]: pgmap v8116: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:04 smithi122 ceph-mon[32255]: pgmap v8116: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:06 smithi022 ceph-mon[31360]: pgmap v8117: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:06 smithi022 ceph-mon[40250]: pgmap v8117: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:06 smithi122 ceph-mon[32255]: pgmap v8117: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:07.028 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:07.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:29:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:29:07.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:07.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:07.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:29:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:08 smithi022 ceph-mon[31360]: pgmap v8118: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:08.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1435998095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:08 smithi022 ceph-mon[40250]: pgmap v8118: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1435998095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:08 smithi122 ceph-mon[32255]: pgmap v8118: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1435998095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:10.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:09 smithi122 ceph-mon[32255]: pgmap v8119: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:10.187 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:10] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:29:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:09 smithi022 ceph-mon[31360]: pgmap v8119: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:09 smithi022 ceph-mon[40250]: pgmap v8119: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:12 smithi022 ceph-mon[31360]: pgmap v8120: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:12 smithi022 ceph-mon[40250]: pgmap v8120: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:12 smithi122 ceph-mon[32255]: pgmap v8120: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:12.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:29:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:29:12.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:12.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:12.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1083661731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1083661731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1083661731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:14 smithi022 ceph-mon[31360]: pgmap v8121: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:14 smithi022 ceph-mon[40250]: pgmap v8121: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:14 smithi122 ceph-mon[32255]: pgmap v8121: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:16 smithi022 ceph-mon[31360]: pgmap v8122: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:16 smithi022 ceph-mon[40250]: pgmap v8122: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:16 smithi122 ceph-mon[32255]: pgmap v8122: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:17.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:29:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:29:17.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:17.767 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:29:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:18.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:18.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:18 smithi022 ceph-mon[31360]: pgmap v8123: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:18 smithi022 ceph-mon[40250]: pgmap v8123: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:18 smithi122 ceph-mon[32255]: pgmap v8123: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4158316110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4158316110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4158316110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:20 smithi122 ceph-mon[32255]: pgmap v8124: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:29:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:20 smithi022 ceph-mon[31360]: pgmap v8124: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:20 smithi022 ceph-mon[40250]: pgmap v8124: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:22.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:22 smithi122 ceph-mon[32255]: pgmap v8125: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:22.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:29:22.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:29:22.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:29:22.182 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[31360]: pgmap v8125: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:22.182 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:29:22.182 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:29:22.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:29:22.183 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[40250]: pgmap v8125: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:22.183 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:29:22.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:29:22.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:29:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:29:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:29:23.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:23.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:23.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:23.511 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.511 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.511 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.512 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.512 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:23.512 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:29:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:24 smithi122 ceph-mon[32255]: pgmap v8126: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1239767350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:24 smithi022 ceph-mon[40250]: pgmap v8126: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1239767350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:24 smithi022 ceph-mon[31360]: pgmap v8126: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1239767350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:26 smithi122 ceph-mon[32255]: pgmap v8127: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:26 smithi022 ceph-mon[31360]: pgmap v8127: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:26 smithi022 ceph-mon[40250]: pgmap v8127: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:27.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:29:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:29:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:29:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:28 smithi122 ceph-mon[32255]: pgmap v8128: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:28.512 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:28.513 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:28.513 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:28 smithi022 ceph-mon[40250]: pgmap v8128: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:28.513 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:28.514 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:28 smithi022 ceph-mon[31360]: pgmap v8128: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:28.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:28.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:28.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1300283073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1300283073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1300283073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:29:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:30 smithi122 ceph-mon[32255]: pgmap v8129: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:30 smithi022 ceph-mon[31360]: pgmap v8129: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:30 smithi022 ceph-mon[40250]: pgmap v8129: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:32 smithi022 ceph-mon[31360]: pgmap v8130: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:32 smithi022 ceph-mon[40250]: pgmap v8130: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:32 smithi122 ceph-mon[32255]: pgmap v8130: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:29:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:29:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:29:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:29:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:29:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:29:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:29:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:29:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:29:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:29:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:29:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:29:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:29:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:29:33.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:34.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:34.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:34.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:34 smithi022 ceph-mon[31360]: pgmap v8131: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:34 smithi022 ceph-mon[40250]: pgmap v8131: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:34 smithi122 ceph-mon[32255]: pgmap v8131: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1560434334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1560434334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:35.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1560434334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:36 smithi022 ceph-mon[31360]: pgmap v8132: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:36 smithi022 ceph-mon[40250]: pgmap v8132: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:36 smithi122 ceph-mon[32255]: pgmap v8132: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:37.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:29:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:29:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:29:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:38 smithi022 ceph-mon[40250]: pgmap v8133: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:38 smithi022 ceph-mon[31360]: pgmap v8133: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:38 smithi122 ceph-mon[32255]: pgmap v8133: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:39.261 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:39.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:39.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:39.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:39 smithi022 ceph-mon[31360]: pgmap v8134: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3861002178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:39 smithi022 ceph-mon[40250]: pgmap v8134: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3861002178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:39 smithi122 ceph-mon[32255]: pgmap v8134: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3861002178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:40] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:29:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:42 smithi022 ceph-mon[31360]: pgmap v8135: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:42 smithi022 ceph-mon[40250]: pgmap v8135: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:42 smithi122 ceph-mon[32255]: pgmap v8135: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:29:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:29:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:44 smithi122 ceph-mon[32255]: pgmap v8136: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:44 smithi022 ceph-mon[31360]: pgmap v8136: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:44 smithi022 ceph-mon[40250]: pgmap v8136: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:44.630 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:44.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:44.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:44.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3013703398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3013703398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:45.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3013703398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:46 smithi122 ceph-mon[32255]: pgmap v8137: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:46 smithi022 ceph-mon[31360]: pgmap v8137: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:46 smithi022 ceph-mon[40250]: pgmap v8137: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:47.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:29:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:29:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:29:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:48 smithi122 ceph-mon[32255]: pgmap v8138: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:48 smithi022 ceph-mon[31360]: pgmap v8138: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:48 smithi022 ceph-mon[40250]: pgmap v8138: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:49.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:49 smithi022 ceph-mon[31360]: pgmap v8139: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:49 smithi022 ceph-mon[40250]: pgmap v8139: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:50.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:50.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:50.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:50] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:29:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:49 smithi122 ceph-mon[32255]: pgmap v8139: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3873994484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3873994484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3873994484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:51 smithi022 ceph-mon[31360]: pgmap v8140: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:51 smithi022 ceph-mon[40250]: pgmap v8140: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:51 smithi122 ceph-mon[32255]: pgmap v8140: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:29:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:29:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:54 smithi122 ceph-mon[32255]: pgmap v8141: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:54 smithi022 ceph-mon[31360]: pgmap v8141: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:54 smithi022 ceph-mon[40250]: pgmap v8141: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:55.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:29:55.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:29:55.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:29:55.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:29:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:56 smithi122 ceph-mon[32255]: pgmap v8142: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2093276795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:56 smithi022 ceph-mon[31360]: pgmap v8142: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2093276795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:56 smithi022 ceph-mon[40250]: pgmap v8142: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:29:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2093276795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:29:57.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:29:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:29:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:29:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:58 smithi122 ceph-mon[32255]: pgmap v8143: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:58 smithi022 ceph-mon[31360]: pgmap v8143: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:29:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:29:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:58 smithi022 ceph-mon[40250]: pgmap v8143: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:29:59 smithi022 ceph-mon[31360]: pgmap v8144: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:29:59 smithi022 ceph-mon[40250]: pgmap v8144: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:29:59 smithi122 ceph-mon[32255]: pgmap v8144: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:00] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:30:00.740 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T22:30:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T22:30:01.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:01.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:01.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T22:30:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:01 smithi122 ceph-mon[32255]: pgmap v8145: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:02.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1851591718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:01 smithi022 ceph-mon[31360]: pgmap v8145: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1851591718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:01 smithi022 ceph-mon[40250]: pgmap v8145: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1851591718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:30:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:30:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:04 smithi122 ceph-mon[32255]: pgmap v8146: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:04 smithi022 ceph-mon[31360]: pgmap v8146: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:04 smithi022 ceph-mon[40250]: pgmap v8146: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:06.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:06 smithi122 ceph-mon[32255]: pgmap v8147: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:06.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:06.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:06.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:06.475 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:06 smithi022 ceph-mon[40250]: pgmap v8147: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:06.475 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:06 smithi022 ceph-mon[31360]: pgmap v8147: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:07.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:30:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:30:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2575112836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2575112836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2575112836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:30:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:08 smithi122 ceph-mon[32255]: pgmap v8148: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:08 smithi022 ceph-mon[31360]: pgmap v8148: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:08 smithi022 ceph-mon[40250]: pgmap v8148: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:09 smithi022 ceph-mon[31360]: pgmap v8149: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:09 smithi022 ceph-mon[40250]: pgmap v8149: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:30:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:09 smithi122 ceph-mon[32255]: pgmap v8149: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:11.476 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:11.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:11.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:11.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:12 smithi022 ceph-mon[31360]: pgmap v8150: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3326517095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:12 smithi022 ceph-mon[40250]: pgmap v8150: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:12.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3326517095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:12 smithi122 ceph-mon[32255]: pgmap v8150: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3326517095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:30:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:30:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:14 smithi122 ceph-mon[32255]: pgmap v8151: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:14 smithi022 ceph-mon[31360]: pgmap v8151: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:14 smithi022 ceph-mon[40250]: pgmap v8151: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:16 smithi122 ceph-mon[32255]: pgmap v8152: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:16 smithi022 ceph-mon[31360]: pgmap v8152: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:16 smithi022 ceph-mon[40250]: pgmap v8152: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:16.848 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:17.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:30:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:30:17.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:17.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:17.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:17.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:30:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:18 smithi122 ceph-mon[32255]: pgmap v8153: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/117045081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:18 smithi022 ceph-mon[31360]: pgmap v8153: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/117045081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:18 smithi022 ceph-mon[40250]: pgmap v8153: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/117045081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:19 smithi022 ceph-mon[31360]: pgmap v8154: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:19 smithi022 ceph-mon[40250]: pgmap v8154: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:19 smithi122 ceph-mon[32255]: pgmap v8154: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:20] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:30:22.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:22 smithi122 ceph-mon[32255]: pgmap v8155: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:22 smithi022 ceph-mon[31360]: pgmap v8155: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:22 smithi022 ceph-mon[40250]: pgmap v8155: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:22.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:22.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:22.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:22.664 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:30:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:30:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1980208288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:23.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:30:23.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:30:23.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:30:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1980208288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:30:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:30:23.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:30:23.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1980208288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:23.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:30:23.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:30:23.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:30:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:24 smithi122 ceph-mon[32255]: pgmap v8156: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:30:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:24 smithi022 ceph-mon[31360]: pgmap v8156: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:30:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:24 smithi022 ceph-mon[40250]: pgmap v8156: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:30:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:26 smithi122 ceph-mon[32255]: pgmap v8157: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:26 smithi022 ceph-mon[31360]: pgmap v8157: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:26 smithi022 ceph-mon[40250]: pgmap v8157: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:27.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:30:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:30:27.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:30:27.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:27.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:27.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:28 smithi022 ceph-mon[31360]: pgmap v8158: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3633813841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:28 smithi022 ceph-mon[40250]: pgmap v8158: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3633813841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:28 smithi122 ceph-mon[32255]: pgmap v8158: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3633813841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:29 smithi022 ceph-mon[31360]: pgmap v8159: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:29 smithi022 ceph-mon[40250]: pgmap v8159: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:30] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:30:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:29 smithi122 ceph-mon[32255]: pgmap v8159: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:32 smithi122 ceph-mon[32255]: pgmap v8160: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:32 smithi022 ceph-mon[31360]: pgmap v8160: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:32 smithi022 ceph-mon[40250]: pgmap v8160: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:30:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:30:32.957 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:33.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:33.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:33.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:34 smithi122 ceph-mon[32255]: pgmap v8161: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:30:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:30:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:30:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:30:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/895703943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[31360]: pgmap v8161: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:30:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:30:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:30:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:30:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/895703943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[40250]: pgmap v8161: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:30:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:30:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:30:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:30:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/895703943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:36 smithi122 ceph-mon[32255]: pgmap v8162: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:36 smithi022 ceph-mon[31360]: pgmap v8162: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:36 smithi022 ceph-mon[40250]: pgmap v8162: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:37.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:30:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:30:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:30:38.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:38.331 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:38.331 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:38 smithi022 ceph-mon[31360]: pgmap v8163: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:38.331 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:38.332 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:38 smithi022 ceph-mon[40250]: pgmap v8163: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:38 smithi122 ceph-mon[32255]: pgmap v8163: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:38.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:38.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:38.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3185135448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3185135448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3185135448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:40 smithi122 ceph-mon[32255]: pgmap v8164: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:40] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:30:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:40 smithi022 ceph-mon[31360]: pgmap v8164: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:40 smithi022 ceph-mon[40250]: pgmap v8164: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:42 smithi122 ceph-mon[32255]: pgmap v8165: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:42 smithi022 ceph-mon[31360]: pgmap v8165: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:42 smithi022 ceph-mon[40250]: pgmap v8165: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:30:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:30:43.702 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:44 smithi122 ceph-mon[32255]: pgmap v8166: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1765891573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:44 smithi022 ceph-mon[31360]: pgmap v8166: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1765891573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:44 smithi022 ceph-mon[40250]: pgmap v8166: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1765891573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:46 smithi122 ceph-mon[32255]: pgmap v8167: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:46 smithi022 ceph-mon[31360]: pgmap v8167: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:46 smithi022 ceph-mon[40250]: pgmap v8167: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:47.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:30:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:30:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:30:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:48 smithi122 ceph-mon[32255]: pgmap v8168: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:48 smithi022 ceph-mon[31360]: pgmap v8168: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:48 smithi022 ceph-mon[40250]: pgmap v8168: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:49.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:49.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:49.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:49.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:49 smithi022 ceph-mon[31360]: pgmap v8169: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/616302335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:49 smithi022 ceph-mon[40250]: pgmap v8169: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/616302335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:49 smithi122 ceph-mon[32255]: pgmap v8169: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/616302335' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:50] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:30:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:52 smithi122 ceph-mon[32255]: pgmap v8170: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:52 smithi022 ceph-mon[31360]: pgmap v8170: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:52 smithi022 ceph-mon[40250]: pgmap v8170: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:30:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:30:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:54 smithi122 ceph-mon[32255]: pgmap v8171: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:30:54.445 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:54 smithi022 ceph-mon[40250]: pgmap v8171: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:54.446 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:54 smithi022 ceph-mon[31360]: pgmap v8171: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:54.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:30:54.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:30:54.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:30:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2962092087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2962092087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:55.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2962092087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:30:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:56 smithi122 ceph-mon[32255]: pgmap v8172: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:56 smithi022 ceph-mon[31360]: pgmap v8172: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:56 smithi022 ceph-mon[40250]: pgmap v8172: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:30:57.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:30:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:30:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:30:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:58 smithi122 ceph-mon[32255]: pgmap v8173: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:58 smithi022 ceph-mon[31360]: pgmap v8173: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:30:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:58 smithi022 ceph-mon[40250]: pgmap v8173: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:30:59.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:00.188 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:30:59 smithi022 ceph-mon[31360]: pgmap v8174: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:00.189 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:30:59 smithi022 ceph-mon[40250]: pgmap v8174: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:30:59 smithi122 ceph-mon[32255]: pgmap v8174: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:00] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:31:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3907258167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3907258167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3907258167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:01 smithi022 ceph-mon[31360]: pgmap v8175: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:01 smithi022 ceph-mon[40250]: pgmap v8175: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:01 smithi122 ceph-mon[32255]: pgmap v8175: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:31:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:31:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:04 smithi122 ceph-mon[32255]: pgmap v8176: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:04 smithi022 ceph-mon[31360]: pgmap v8176: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:04 smithi022 ceph-mon[40250]: pgmap v8176: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:05.189 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:05.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:05.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:05.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:06 smithi122 ceph-mon[32255]: pgmap v8177: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4191232432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:06 smithi022 ceph-mon[31360]: pgmap v8177: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4191232432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:06 smithi022 ceph-mon[40250]: pgmap v8177: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4191232432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:07.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:31:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:31:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:31:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:08 smithi122 ceph-mon[32255]: pgmap v8178: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:08 smithi022 ceph-mon[31360]: pgmap v8178: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:08 smithi022 ceph-mon[40250]: pgmap v8178: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:09 smithi022 ceph-mon[31360]: pgmap v8179: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:09 smithi022 ceph-mon[40250]: pgmap v8179: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:31:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:09 smithi122 ceph-mon[32255]: pgmap v8179: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:10.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:10.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:10.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2212509749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2212509749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2212509749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:11 smithi022 ceph-mon[31360]: pgmap v8180: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:11 smithi022 ceph-mon[40250]: pgmap v8180: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:11 smithi122 ceph-mon[32255]: pgmap v8180: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:31:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:31:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:14 smithi122 ceph-mon[32255]: pgmap v8181: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:14.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:14 smithi022 ceph-mon[31360]: pgmap v8181: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:14 smithi022 ceph-mon[40250]: pgmap v8181: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:15.932 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:16.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:16 smithi022 ceph-mon[31360]: pgmap v8182: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:16 smithi022 ceph-mon[40250]: pgmap v8182: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:16.304 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:16.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:16.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:16 smithi122 ceph-mon[32255]: pgmap v8182: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:17.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1274934102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:17.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1274934102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:17.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:31:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:31:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1274934102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:31:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:18 smithi122 ceph-mon[32255]: pgmap v8183: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:18 smithi022 ceph-mon[40250]: pgmap v8183: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:18 smithi022 ceph-mon[31360]: pgmap v8183: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:19 smithi022 ceph-mon[31360]: pgmap v8184: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:19 smithi022 ceph-mon[40250]: pgmap v8184: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:20] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:31:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:19 smithi122 ceph-mon[32255]: pgmap v8184: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:21.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:21.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:21.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:21.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:22 smithi122 ceph-mon[32255]: pgmap v8185: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/960229328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:22 smithi022 ceph-mon[31360]: pgmap v8185: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/960229328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:22 smithi022 ceph-mon[40250]: pgmap v8185: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/960229328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:31:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:31:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:24 smithi122 ceph-mon[32255]: pgmap v8186: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:31:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:31:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:31:24.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:31:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[31360]: pgmap v8186: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:31:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:31:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:31:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:31:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[40250]: pgmap v8186: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:31:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:31:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:31:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:31:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:26 smithi122 ceph-mon[32255]: pgmap v8187: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:26 smithi022 ceph-mon[31360]: pgmap v8187: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:26 smithi022 ceph-mon[40250]: pgmap v8187: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:26.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:27.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:27.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:27.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1592649998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1592649998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1592649998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:27.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:31:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:31:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:31:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:28 smithi122 ceph-mon[32255]: pgmap v8188: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:28 smithi022 ceph-mon[31360]: pgmap v8188: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:28 smithi022 ceph-mon[40250]: pgmap v8188: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:29 smithi022 ceph-mon[31360]: pgmap v8189: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:29 smithi022 ceph-mon[40250]: pgmap v8189: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:30] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:31:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:29 smithi122 ceph-mon[32255]: pgmap v8189: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:32.042 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:32.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:32 smithi122 ceph-mon[32255]: pgmap v8190: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:32.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:32.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:32.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:32.414 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:32 smithi022 ceph-mon[31360]: pgmap v8190: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:32.415 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:32 smithi022 ceph-mon[40250]: pgmap v8190: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:31:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:31:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1811316688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1811316688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1811316688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:34 smithi122 ceph-mon[32255]: pgmap v8191: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:31:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:31:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:31:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:31:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[31360]: pgmap v8191: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:31:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:31:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:31:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:31:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[40250]: pgmap v8191: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:31:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:31:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:31:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:31:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:36 smithi122 ceph-mon[32255]: pgmap v8192: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:36 smithi022 ceph-mon[31360]: pgmap v8192: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:36 smithi022 ceph-mon[40250]: pgmap v8192: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:37.415 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:37.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:31:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:31:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:31:37.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:37.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:37.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:38 smithi122 ceph-mon[32255]: pgmap v8193: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2708454110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:38 smithi022 ceph-mon[31360]: pgmap v8193: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2708454110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:38 smithi022 ceph-mon[40250]: pgmap v8193: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2708454110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:39 smithi022 ceph-mon[31360]: pgmap v8194: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:39 smithi022 ceph-mon[40250]: pgmap v8194: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:31:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:39 smithi122 ceph-mon[32255]: pgmap v8194: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:42 smithi122 ceph-mon[32255]: pgmap v8195: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:31:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:31:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:42 smithi022 ceph-mon[31360]: pgmap v8195: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:42.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:42 smithi022 ceph-mon[40250]: pgmap v8195: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:42.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:43.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:43.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:43.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:44 smithi122 ceph-mon[32255]: pgmap v8196: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3897382591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:44 smithi022 ceph-mon[31360]: pgmap v8196: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3897382591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:44 smithi022 ceph-mon[40250]: pgmap v8196: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3897382591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:46 smithi122 ceph-mon[32255]: pgmap v8197: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:46 smithi022 ceph-mon[31360]: pgmap v8197: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:46 smithi022 ceph-mon[40250]: pgmap v8197: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:47.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:31:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:31:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:31:48.158 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:48 smithi122 ceph-mon[32255]: pgmap v8198: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:48 smithi022 ceph-mon[31360]: pgmap v8198: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:48 smithi022 ceph-mon[40250]: pgmap v8198: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/923870193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/923870193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/923870193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:50 smithi122 ceph-mon[32255]: pgmap v8199: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:31:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:50 smithi022 ceph-mon[31360]: pgmap v8199: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:50 smithi022 ceph-mon[40250]: pgmap v8199: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:31:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:31:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:52 smithi122 ceph-mon[32255]: pgmap v8200: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:52 smithi022 ceph-mon[31360]: pgmap v8200: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:52 smithi022 ceph-mon[40250]: pgmap v8200: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:53.529 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:53.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:53.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:53.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:31:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:54 smithi022 ceph-mon[31360]: pgmap v8201: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1909088280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:54 smithi022 ceph-mon[40250]: pgmap v8201: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1909088280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:54 smithi122 ceph-mon[32255]: pgmap v8201: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1909088280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:31:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:56 smithi122 ceph-mon[32255]: pgmap v8202: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:56 smithi022 ceph-mon[31360]: pgmap v8202: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:56 smithi022 ceph-mon[40250]: pgmap v8202: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:31:57.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:31:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:31:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:31:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:58 smithi122 ceph-mon[32255]: pgmap v8203: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:58 smithi022 ceph-mon[40250]: pgmap v8203: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:31:58.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:58 smithi022 ceph-mon[31360]: pgmap v8203: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:31:58.897 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:31:59.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:31:59.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:31:59.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:00.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:59 smithi022 ceph-mon[40250]: pgmap v8204: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:00.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:31:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2323295521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:00.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:59 smithi022 ceph-mon[31360]: pgmap v8204: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:00.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:31:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2323295521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:32:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:59 smithi122 ceph-mon[32255]: pgmap v8204: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:31:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2323295521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:32:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:32:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:02 smithi122 ceph-mon[32255]: pgmap v8205: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:02 smithi022 ceph-mon[31360]: pgmap v8205: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:02 smithi022 ceph-mon[40250]: pgmap v8205: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:04.269 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:04 smithi122 ceph-mon[32255]: pgmap v8206: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:04 smithi022 ceph-mon[31360]: pgmap v8206: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:04 smithi022 ceph-mon[40250]: pgmap v8206: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:04.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:04.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:04.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2597190421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2597190421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2597190421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:06 smithi122 ceph-mon[32255]: pgmap v8207: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:06 smithi022 ceph-mon[31360]: pgmap v8207: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:06 smithi022 ceph-mon[40250]: pgmap v8207: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:07.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:32:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:32:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:32:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:08 smithi122 ceph-mon[32255]: pgmap v8208: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:08 smithi022 ceph-mon[31360]: pgmap v8208: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:08 smithi022 ceph-mon[40250]: pgmap v8208: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:09.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:09.974 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:09 smithi022 ceph-mon[40250]: pgmap v8209: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:09.975 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:09 smithi022 ceph-mon[31360]: pgmap v8209: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:10.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:10.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:10.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:32:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:09 smithi122 ceph-mon[32255]: pgmap v8209: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:11.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2343885520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2343885520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2343885520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:11 smithi022 ceph-mon[31360]: pgmap v8210: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:11 smithi022 ceph-mon[40250]: pgmap v8210: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:11 smithi122 ceph-mon[32255]: pgmap v8210: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:32:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:32:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:14 smithi122 ceph-mon[32255]: pgmap v8211: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:14 smithi022 ceph-mon[31360]: pgmap v8211: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:14 smithi022 ceph-mon[40250]: pgmap v8211: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:15.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:16 smithi122 ceph-mon[32255]: pgmap v8212: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3650548576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:16 smithi022 ceph-mon[31360]: pgmap v8212: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3650548576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:16 smithi022 ceph-mon[40250]: pgmap v8212: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3650548576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:17.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:32:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:32:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:32:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:18 smithi122 ceph-mon[32255]: pgmap v8213: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:18 smithi022 ceph-mon[31360]: pgmap v8213: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:18 smithi022 ceph-mon[40250]: pgmap v8213: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:19 smithi022 ceph-mon[31360]: pgmap v8214: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:19 smithi022 ceph-mon[40250]: pgmap v8214: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:20.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:20] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:32:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:19 smithi122 ceph-mon[32255]: pgmap v8214: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:20.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:20.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:20.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1012327834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1012327834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1012327834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:21 smithi022 ceph-mon[31360]: pgmap v8215: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:21 smithi022 ceph-mon[40250]: pgmap v8215: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:21 smithi122 ceph-mon[32255]: pgmap v8215: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:32:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:32:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[31360]: pgmap v8216: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:32:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:32:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:32:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:32:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[40250]: pgmap v8216: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:32:24.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:32:24.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:32:24.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:32:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:24 smithi122 ceph-mon[32255]: pgmap v8216: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:32:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:32:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:32:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:26.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:26.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:26.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:26 smithi122 ceph-mon[32255]: pgmap v8217: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:26 smithi022 ceph-mon[31360]: pgmap v8217: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:26 smithi022 ceph-mon[40250]: pgmap v8217: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3867863281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3867863281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3867863281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:27.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:32:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:32:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:32:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:28 smithi122 ceph-mon[32255]: pgmap v8218: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:28 smithi022 ceph-mon[31360]: pgmap v8218: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:28 smithi022 ceph-mon[40250]: pgmap v8218: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:29 smithi022 ceph-mon[31360]: pgmap v8219: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:29 smithi022 ceph-mon[40250]: pgmap v8219: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:30] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:32:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:29 smithi122 ceph-mon[32255]: pgmap v8219: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:31.125 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:31.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:31.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:31.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:32 smithi122 ceph-mon[32255]: pgmap v8220: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/304389284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:32.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:32:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:32:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:32 smithi022 ceph-mon[31360]: pgmap v8220: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/304389284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:32 smithi022 ceph-mon[40250]: pgmap v8220: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/304389284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:34 smithi122 ceph-mon[32255]: pgmap v8221: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:32:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:32:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:32:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:32:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[31360]: pgmap v8221: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:32:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[40250]: pgmap v8221: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:32:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:32:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:32:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:32:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:36 smithi122 ceph-mon[32255]: pgmap v8222: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:36.493 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:36.494 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:36 smithi022 ceph-mon[31360]: pgmap v8222: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:36.494 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:36 smithi022 ceph-mon[40250]: pgmap v8222: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:36.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:36.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:36.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2554636109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2554636109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:37.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2554636109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:37.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:32:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:32:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:32:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:38 smithi122 ceph-mon[32255]: pgmap v8223: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:38 smithi022 ceph-mon[31360]: pgmap v8223: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:38 smithi022 ceph-mon[40250]: pgmap v8223: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:39 smithi022 ceph-mon[31360]: pgmap v8224: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:39 smithi022 ceph-mon[40250]: pgmap v8224: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:40] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:32:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:39 smithi122 ceph-mon[32255]: pgmap v8224: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:41.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:42.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:42.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:42.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:42.240 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:42 smithi022 ceph-mon[40250]: pgmap v8225: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:42.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:42 smithi022 ceph-mon[31360]: pgmap v8225: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:42 smithi122 ceph-mon[32255]: pgmap v8225: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:32:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:32:43.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/631615817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/631615817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/631615817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:44 smithi122 ceph-mon[32255]: pgmap v8226: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:44 smithi022 ceph-mon[31360]: pgmap v8226: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:44 smithi022 ceph-mon[40250]: pgmap v8226: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:46 smithi122 ceph-mon[32255]: pgmap v8227: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:46 smithi022 ceph-mon[31360]: pgmap v8227: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:46 smithi022 ceph-mon[40250]: pgmap v8227: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:47.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:47.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:32:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:32:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:32:47.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:47.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:47.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:48 smithi122 ceph-mon[32255]: pgmap v8228: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/266518990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:48 smithi022 ceph-mon[31360]: pgmap v8228: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/266518990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:48 smithi022 ceph-mon[40250]: pgmap v8228: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/266518990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:49 smithi022 ceph-mon[31360]: pgmap v8229: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:49 smithi022 ceph-mon[40250]: pgmap v8229: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:50] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:32:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:49 smithi122 ceph-mon[32255]: pgmap v8229: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:52 smithi122 ceph-mon[32255]: pgmap v8230: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:32:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:32:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:52 smithi022 ceph-mon[31360]: pgmap v8230: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:52 smithi022 ceph-mon[40250]: pgmap v8230: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:52.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:52.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:52.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:52.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3717043942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3717043942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3717043942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:54 smithi122 ceph-mon[32255]: pgmap v8231: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:54 smithi022 ceph-mon[31360]: pgmap v8231: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:54 smithi022 ceph-mon[40250]: pgmap v8231: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:56 smithi122 ceph-mon[32255]: pgmap v8232: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:56 smithi022 ceph-mon[31360]: pgmap v8232: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:56 smithi022 ceph-mon[40250]: pgmap v8232: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:32:57.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:32:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:32:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:32:57.986 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:32:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:58 smithi022 ceph-mon[31360]: pgmap v8233: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:58 smithi022 ceph-mon[40250]: pgmap v8233: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:58.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:32:58.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:32:58.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:32:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:58 smithi122 ceph-mon[32255]: pgmap v8233: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:32:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:32:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:32:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1510426512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:32:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1510426512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:32:59.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:32:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1510426512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:00] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:33:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:00 smithi122 ceph-mon[32255]: pgmap v8234: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:00 smithi022 ceph-mon[31360]: pgmap v8234: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:00 smithi022 ceph-mon[40250]: pgmap v8234: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:02 smithi122 ceph-mon[32255]: pgmap v8235: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:33:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:33:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:02 smithi022 ceph-mon[31360]: pgmap v8235: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:02 smithi022 ceph-mon[40250]: pgmap v8235: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:03.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:04 smithi122 ceph-mon[32255]: pgmap v8236: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2780778266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:04 smithi022 ceph-mon[31360]: pgmap v8236: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2780778266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:04 smithi022 ceph-mon[40250]: pgmap v8236: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2780778266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:06 smithi122 ceph-mon[32255]: pgmap v8237: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:06 smithi022 ceph-mon[31360]: pgmap v8237: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:06 smithi022 ceph-mon[40250]: pgmap v8237: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:07.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:33:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:33:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:33:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:08 smithi122 ceph-mon[32255]: pgmap v8238: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:08 smithi022 ceph-mon[31360]: pgmap v8238: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:08 smithi022 ceph-mon[40250]: pgmap v8238: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:08.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:09.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:09.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:09.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:09 smithi022 ceph-mon[31360]: pgmap v8239: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/826093698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:09 smithi022 ceph-mon[40250]: pgmap v8239: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/826093698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:10] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:33:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:09 smithi122 ceph-mon[32255]: pgmap v8239: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/826093698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:12 smithi122 ceph-mon[32255]: pgmap v8240: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:33:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:33:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:12 smithi022 ceph-mon[31360]: pgmap v8240: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:12 smithi022 ceph-mon[40250]: pgmap v8240: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:14.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:14 smithi122 ceph-mon[32255]: pgmap v8241: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:14.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:14.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:14.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:14.466 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:14 smithi022 ceph-mon[40250]: pgmap v8241: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:14.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:14 smithi022 ceph-mon[31360]: pgmap v8241: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2221488167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2221488167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2221488167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:16 smithi122 ceph-mon[32255]: pgmap v8242: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:16 smithi022 ceph-mon[31360]: pgmap v8242: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:16 smithi022 ceph-mon[40250]: pgmap v8242: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:17.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:33:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:33:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:33:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:18 smithi122 ceph-mon[32255]: pgmap v8243: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:18 smithi022 ceph-mon[31360]: pgmap v8243: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:18 smithi022 ceph-mon[40250]: pgmap v8243: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:19.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:19.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:19.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:19.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:19 smithi022 ceph-mon[31360]: pgmap v8244: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4223783034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:19 smithi022 ceph-mon[40250]: pgmap v8244: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4223783034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:33:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:19 smithi122 ceph-mon[32255]: pgmap v8244: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4223783034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:22 smithi122 ceph-mon[32255]: pgmap v8245: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:33:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:33:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:22 smithi022 ceph-mon[31360]: pgmap v8245: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:22 smithi022 ceph-mon[40250]: pgmap v8245: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[31360]: pgmap v8246: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:33:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:33:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:33:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[40250]: pgmap v8246: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:33:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:33:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:33:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:24 smithi122 ceph-mon[32255]: pgmap v8246: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:33:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:33:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:33:24.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:25.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:25.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:25.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:26 smithi122 ceph-mon[32255]: pgmap v8247: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1336660761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:26 smithi022 ceph-mon[31360]: pgmap v8247: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1336660761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:26 smithi022 ceph-mon[40250]: pgmap v8247: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1336660761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:27.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:33:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:33:27.676 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:33:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:28 smithi122 ceph-mon[32255]: pgmap v8248: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:28.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:28 smithi022 ceph-mon[31360]: pgmap v8248: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:28.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:28.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:28.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:28 smithi022 ceph-mon[40250]: pgmap v8248: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:28.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:28.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:29 smithi122 ceph-mon[32255]: pgmap v8249: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.692 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.692 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.692 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[31360]: pgmap v8249: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:29.693 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:29.693 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[40250]: pgmap v8249: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:29.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:30.205 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:30.340 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:33:30.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:30.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:30.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2891119511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:31.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2891119511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:31.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:31.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2891119511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:31.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:31.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:33:31.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:31 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3188M 2021-12-15T22:33:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:31 smithi122 ceph-mon[32255]: pgmap v8250: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3188M 2021-12-15T22:33:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[31360]: pgmap v8250: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:32.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:32.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3188M 2021-12-15T22:33:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[40250]: pgmap v8250: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:32.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:32.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:33:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:33:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:33:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:34 smithi122 ceph-mon[32255]: pgmap v8251: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:33:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:33:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:33:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[31360]: pgmap v8251: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:33:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:33:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:33:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[40250]: pgmap v8251: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:33:35.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:35.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:35.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:35.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:36 smithi022 ceph-mon[31360]: pgmap v8252: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2055331273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:36 smithi022 ceph-mon[40250]: pgmap v8252: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2055331273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:36 smithi122 ceph-mon[32255]: pgmap v8252: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2055331273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:37.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:33:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:33:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:33:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:38 smithi122 ceph-mon[32255]: pgmap v8253: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:38 smithi022 ceph-mon[31360]: pgmap v8253: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:38 smithi022 ceph-mon[40250]: pgmap v8253: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:39 smithi022 ceph-mon[31360]: pgmap v8254: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:39 smithi022 ceph-mon[40250]: pgmap v8254: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:40] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:33:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:39 smithi122 ceph-mon[32255]: pgmap v8254: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:40.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:41.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:41.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:41.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:42 smithi122 ceph-mon[32255]: pgmap v8255: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2972648606' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:42.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:33:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:33:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:42 smithi022 ceph-mon[31360]: pgmap v8255: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2972648606' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:42 smithi022 ceph-mon[40250]: pgmap v8255: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2972648606' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:44 smithi122 ceph-mon[32255]: pgmap v8256: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:44 smithi022 ceph-mon[31360]: pgmap v8256: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:44 smithi022 ceph-mon[40250]: pgmap v8256: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:46.308 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:46 smithi122 ceph-mon[32255]: pgmap v8257: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:46 smithi022 ceph-mon[31360]: pgmap v8257: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:46 smithi022 ceph-mon[40250]: pgmap v8257: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:46.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:46.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:46.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2844116387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2844116387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:47.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:33:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:33:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2844116387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:33:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:48 smithi122 ceph-mon[32255]: pgmap v8258: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:48 smithi022 ceph-mon[31360]: pgmap v8258: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:48 smithi022 ceph-mon[40250]: pgmap v8258: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:49 smithi022 ceph-mon[31360]: pgmap v8259: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:49 smithi022 ceph-mon[40250]: pgmap v8259: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:49 smithi122 ceph-mon[32255]: pgmap v8259: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:50] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:33:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:52.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:52.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:52.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:52 smithi122 ceph-mon[32255]: pgmap v8260: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2536205822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:52.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:33:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:33:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:52 smithi022 ceph-mon[31360]: pgmap v8260: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2536205822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:52 smithi022 ceph-mon[40250]: pgmap v8260: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2536205822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:54 smithi122 ceph-mon[32255]: pgmap v8261: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:54 smithi022 ceph-mon[31360]: pgmap v8261: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:54 smithi022 ceph-mon[40250]: pgmap v8261: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:56 smithi122 ceph-mon[32255]: pgmap v8262: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:56 smithi022 ceph-mon[31360]: pgmap v8262: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:56 smithi022 ceph-mon[40250]: pgmap v8262: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:33:57.053 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:33:57.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:33:57.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:33:57.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:33:57.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:33:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:33:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:33:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:58 smithi122 ceph-mon[32255]: pgmap v8263: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:58.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4264769833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:58 smithi022 ceph-mon[31360]: pgmap v8263: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:58.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4264769833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:33:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:58 smithi022 ceph-mon[40250]: pgmap v8263: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:33:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:33:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4264769833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:33:59 smithi022 ceph-mon[31360]: pgmap v8264: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:33:59 smithi022 ceph-mon[40250]: pgmap v8264: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:00] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:34:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:33:59 smithi122 ceph-mon[32255]: pgmap v8264: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:02.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:02.423 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:02 smithi022 ceph-mon[31360]: pgmap v8265: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:02.423 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:02 smithi022 ceph-mon[40250]: pgmap v8265: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:02 smithi122 ceph-mon[32255]: pgmap v8265: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:34:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:34:02.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:02.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:02.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:03.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2003877069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2003877069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2003877069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:04 smithi122 ceph-mon[32255]: pgmap v8266: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:04 smithi022 ceph-mon[31360]: pgmap v8266: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:04 smithi022 ceph-mon[40250]: pgmap v8266: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:06 smithi122 ceph-mon[32255]: pgmap v8267: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:06 smithi022 ceph-mon[31360]: pgmap v8267: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:06 smithi022 ceph-mon[40250]: pgmap v8267: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:07.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:34:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:34:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:34:07.793 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:08.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:08.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:08.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:08 smithi122 ceph-mon[32255]: pgmap v8268: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:08 smithi022 ceph-mon[31360]: pgmap v8268: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:08.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:08 smithi022 ceph-mon[40250]: pgmap v8268: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/804807102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/804807102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/804807102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:34:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:10 smithi122 ceph-mon[32255]: pgmap v8269: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:10 smithi022 ceph-mon[31360]: pgmap v8269: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:10 smithi022 ceph-mon[40250]: pgmap v8269: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:34:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:34:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:12 smithi122 ceph-mon[32255]: pgmap v8270: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:12 smithi022 ceph-mon[31360]: pgmap v8270: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:12 smithi022 ceph-mon[40250]: pgmap v8270: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:13.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:13.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:13.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:13.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:14 smithi122 ceph-mon[32255]: pgmap v8271: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2203046903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:14 smithi022 ceph-mon[31360]: pgmap v8271: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2203046903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:14 smithi022 ceph-mon[40250]: pgmap v8271: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2203046903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:16 smithi122 ceph-mon[32255]: pgmap v8272: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:16 smithi022 ceph-mon[31360]: pgmap v8272: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:16 smithi022 ceph-mon[40250]: pgmap v8272: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:17.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:34:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:34:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:34:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:18 smithi122 ceph-mon[32255]: pgmap v8273: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:18 smithi022 ceph-mon[31360]: pgmap v8273: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:18 smithi022 ceph-mon[40250]: pgmap v8273: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:18.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:18.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:18.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:18.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4006732799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4006732799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4006732799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:20 smithi122 ceph-mon[32255]: pgmap v8274: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:20] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:34:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:20 smithi022 ceph-mon[31360]: pgmap v8274: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:20 smithi022 ceph-mon[40250]: pgmap v8274: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:22 smithi122 ceph-mon[32255]: pgmap v8275: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:34:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:34:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:22 smithi022 ceph-mon[31360]: pgmap v8275: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:22 smithi022 ceph-mon[40250]: pgmap v8275: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:23.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:24 smithi022 ceph-mon[31360]: pgmap v8276: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:24 smithi022 ceph-mon[40250]: pgmap v8276: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:24.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:24.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:24.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:24 smithi122 ceph-mon[32255]: pgmap v8276: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3998145507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3998145507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3998145507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:26 smithi122 ceph-mon[32255]: pgmap v8277: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:26 smithi022 ceph-mon[31360]: pgmap v8277: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:26 smithi022 ceph-mon[40250]: pgmap v8277: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:27.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:34:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:34:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:34:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:28 smithi122 ceph-mon[32255]: pgmap v8278: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:28 smithi022 ceph-mon[40250]: pgmap v8278: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:28 smithi022 ceph-mon[31360]: pgmap v8278: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:29.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:29.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:29.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:29.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:30 smithi022 ceph-mon[31360]: pgmap v8279: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:30.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3483659859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:30 smithi022 ceph-mon[40250]: pgmap v8279: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:30.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3483659859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:30] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:34:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:30 smithi122 ceph-mon[32255]: pgmap v8279: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3483659859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:32.117 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[31360]: pgmap v8280: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:32.117 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:34:32.117 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:34:32.117 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:34:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:34:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:34:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:32 smithi122 ceph-mon[32255]: pgmap v8280: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:32.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:34:32.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:34:32.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:34:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[40250]: pgmap v8280: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:34:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:34:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:34:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:34:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:34:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:34:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:34:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:34:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:34:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:34:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:34:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:34:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:34:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:34:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:34:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:34:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:34:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:34:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:34 smithi022 ceph-mon[31360]: pgmap v8281: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:34 smithi022 ceph-mon[40250]: pgmap v8281: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:34.659 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:34 smithi122 ceph-mon[32255]: pgmap v8281: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:35.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:35.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:35.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4197858980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4197858980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4197858980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:36 smithi022 ceph-mon[31360]: pgmap v8282: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:36 smithi022 ceph-mon[40250]: pgmap v8282: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:36 smithi122 ceph-mon[32255]: pgmap v8282: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:37.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:34:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:34:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:34:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:38 smithi022 ceph-mon[31360]: pgmap v8283: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:38 smithi022 ceph-mon[40250]: pgmap v8283: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:38 smithi122 ceph-mon[32255]: pgmap v8283: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:40.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:40 smithi022 ceph-mon[31360]: pgmap v8284: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:40 smithi022 ceph-mon[40250]: pgmap v8284: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:40.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:40.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:40.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:40 smithi122 ceph-mon[32255]: pgmap v8284: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:34:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/500176747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:41.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/500176747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:41.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/500176747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:42 smithi022 ceph-mon[31360]: pgmap v8285: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:42 smithi022 ceph-mon[40250]: pgmap v8285: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:42 smithi122 ceph-mon[32255]: pgmap v8285: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:34:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:34:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:44 smithi122 ceph-mon[32255]: pgmap v8286: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:44 smithi022 ceph-mon[31360]: pgmap v8286: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:44 smithi022 ceph-mon[40250]: pgmap v8286: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:45.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:45.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:45.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:45.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:46 smithi122 ceph-mon[32255]: pgmap v8287: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2221523816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:46 smithi022 ceph-mon[31360]: pgmap v8287: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2221523816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:46 smithi022 ceph-mon[40250]: pgmap v8287: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2221523816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:47.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:34:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:34:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:34:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:48 smithi122 ceph-mon[32255]: pgmap v8288: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:48 smithi022 ceph-mon[31360]: pgmap v8288: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:48 smithi022 ceph-mon[40250]: pgmap v8288: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:50.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:50 smithi022 ceph-mon[31360]: pgmap v8289: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:50 smithi022 ceph-mon[40250]: pgmap v8289: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:50] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:34:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:50 smithi122 ceph-mon[32255]: pgmap v8289: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:50.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:51.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:51.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:51.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:34:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:34:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:52 smithi122 ceph-mon[32255]: pgmap v8290: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/39872627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:52 smithi022 ceph-mon[31360]: pgmap v8290: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/39872627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:52 smithi022 ceph-mon[40250]: pgmap v8290: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/39872627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:54 smithi122 ceph-mon[32255]: pgmap v8291: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:54 smithi022 ceph-mon[31360]: pgmap v8291: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:54 smithi022 ceph-mon[40250]: pgmap v8291: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:56.137 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:34:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:56 smithi122 ceph-mon[32255]: pgmap v8292: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:56.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:34:56.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:34:56.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:34:56.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:56 smithi022 ceph-mon[31360]: pgmap v8292: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:56.507 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:56 smithi022 ceph-mon[40250]: pgmap v8292: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:34:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/894016007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/894016007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/894016007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:34:57.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:34:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:34:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:34:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:58 smithi122 ceph-mon[32255]: pgmap v8293: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:34:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:58 smithi022 ceph-mon[40250]: pgmap v8293: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:34:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:34:58.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:58 smithi022 ceph-mon[31360]: pgmap v8293: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:34:58.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:34:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:00 smithi022 ceph-mon[40250]: pgmap v8294: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:00 smithi022 ceph-mon[31360]: pgmap v8294: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:00] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:35:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:00 smithi122 ceph-mon[32255]: pgmap v8294: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:01.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:01.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:01.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:01.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:02 smithi122 ceph-mon[32255]: pgmap v8295: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/767295867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:35:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:35:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:02 smithi022 ceph-mon[31360]: pgmap v8295: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/767295867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:02.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:02 smithi022 ceph-mon[40250]: pgmap v8295: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:02.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/767295867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:04 smithi122 ceph-mon[32255]: pgmap v8296: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:04 smithi022 ceph-mon[31360]: pgmap v8296: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:04 smithi022 ceph-mon[40250]: pgmap v8296: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:06 smithi122 ceph-mon[32255]: pgmap v8297: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:06 smithi022 ceph-mon[31360]: pgmap v8297: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:06 smithi022 ceph-mon[40250]: pgmap v8297: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:06.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:07.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:07.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:07.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:07.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:35:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:35:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:35:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:08 smithi122 ceph-mon[32255]: pgmap v8298: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3075740966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:08 smithi022 ceph-mon[31360]: pgmap v8298: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3075740966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:08 smithi022 ceph-mon[40250]: pgmap v8298: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3075740966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:10 smithi022 ceph-mon[31360]: pgmap v8299: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:10 smithi022 ceph-mon[40250]: pgmap v8299: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:35:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:10 smithi122 ceph-mon[32255]: pgmap v8299: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:12.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:12 smithi122 ceph-mon[32255]: pgmap v8300: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:35:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:35:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:12 smithi022 ceph-mon[31360]: pgmap v8300: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:12 smithi022 ceph-mon[40250]: pgmap v8300: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:12.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:12.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:12.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/516382521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/516382521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/516382521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:14 smithi122 ceph-mon[32255]: pgmap v8301: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:14 smithi022 ceph-mon[31360]: pgmap v8301: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:14 smithi022 ceph-mon[40250]: pgmap v8301: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:16 smithi122 ceph-mon[32255]: pgmap v8302: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:16 smithi022 ceph-mon[31360]: pgmap v8302: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:16 smithi022 ceph-mon[40250]: pgmap v8302: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:17.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:35:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:35:17.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:35:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:18 smithi022 ceph-mon[31360]: pgmap v8303: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2243125881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:18 smithi022 ceph-mon[40250]: pgmap v8303: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2243125881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:18 smithi122 ceph-mon[32255]: pgmap v8303: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2243125881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:20 smithi022 ceph-mon[31360]: pgmap v8304: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:20 smithi022 ceph-mon[40250]: pgmap v8304: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:20 smithi122 ceph-mon[32255]: pgmap v8304: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T22:35:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:35:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:35:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:22 smithi122 ceph-mon[32255]: pgmap v8305: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:22 smithi022 ceph-mon[31360]: pgmap v8305: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:22 smithi022 ceph-mon[40250]: pgmap v8305: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:22.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:23.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:23.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:23.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:24 smithi122 ceph-mon[32255]: pgmap v8306: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3265961597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:24 smithi022 ceph-mon[31360]: pgmap v8306: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3265961597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:24 smithi022 ceph-mon[40250]: pgmap v8306: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3265961597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:26 smithi122 ceph-mon[32255]: pgmap v8307: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:26 smithi022 ceph-mon[31360]: pgmap v8307: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:26 smithi022 ceph-mon[40250]: pgmap v8307: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:27.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:35:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:35:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:35:28.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:28 smithi122 ceph-mon[32255]: pgmap v8308: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:28 smithi022 ceph-mon[31360]: pgmap v8308: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:28 smithi022 ceph-mon[40250]: pgmap v8308: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:28.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:28.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:28.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3608149859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3608149859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3608149859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T22:35:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:30 smithi122 ceph-mon[32255]: pgmap v8309: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:30 smithi022 ceph-mon[31360]: pgmap v8309: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:30 smithi022 ceph-mon[40250]: pgmap v8309: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:32 smithi122 ceph-mon[32255]: pgmap v8310: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:35:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:35:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:32 smithi022 ceph-mon[31360]: pgmap v8310: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:32 smithi022 ceph-mon[40250]: pgmap v8310: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:35:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:35:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:35:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:35:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:35:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:35:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:35:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:35:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:35:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:35:33.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:34.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:34 smithi122 ceph-mon[32255]: pgmap v8311: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:35:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:35:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:35:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:35:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2699686423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[31360]: pgmap v8311: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:35:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:35:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:35:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:35:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2699686423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[40250]: pgmap v8311: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:35:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2699686423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:36 smithi122 ceph-mon[32255]: pgmap v8312: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:36 smithi022 ceph-mon[31360]: pgmap v8312: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:36 smithi022 ceph-mon[40250]: pgmap v8312: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:37.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:35:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:35:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:35:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:38 smithi122 ceph-mon[32255]: pgmap v8313: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:38 smithi022 ceph-mon[31360]: pgmap v8313: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:38 smithi022 ceph-mon[40250]: pgmap v8313: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:39.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:39.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:39.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:39.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:40 smithi022 ceph-mon[31360]: pgmap v8314: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:40.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2445607277' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:40 smithi022 ceph-mon[40250]: pgmap v8314: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:40.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2445607277' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:35:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:40 smithi122 ceph-mon[32255]: pgmap v8314: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2445607277' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:42 smithi122 ceph-mon[32255]: pgmap v8315: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:35:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:35:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:42 smithi022 ceph-mon[31360]: pgmap v8315: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:42 smithi022 ceph-mon[40250]: pgmap v8315: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:44 smithi122 ceph-mon[32255]: pgmap v8316: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:44.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:44.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:44 smithi022 ceph-mon[31360]: pgmap v8316: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:44.468 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:44 smithi022 ceph-mon[40250]: pgmap v8316: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:44.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:44.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:44.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1081170462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:45.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1081170462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1081170462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:46 smithi122 ceph-mon[32255]: pgmap v8317: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:46 smithi022 ceph-mon[31360]: pgmap v8317: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:46 smithi022 ceph-mon[40250]: pgmap v8317: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:47.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:35:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:35:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:35:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:48 smithi122 ceph-mon[32255]: pgmap v8318: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:48 smithi022 ceph-mon[31360]: pgmap v8318: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:48 smithi022 ceph-mon[40250]: pgmap v8318: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:50.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:50.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:50.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:50.208 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:50 smithi022 ceph-mon[40250]: pgmap v8319: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:50.208 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:50 smithi022 ceph-mon[31360]: pgmap v8319: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:50] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:35:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:50 smithi122 ceph-mon[32255]: pgmap v8319: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/359330768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/359330768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/359330768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:52 smithi022 ceph-mon[31360]: pgmap v8320: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:52 smithi022 ceph-mon[40250]: pgmap v8320: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:52 smithi122 ceph-mon[32255]: pgmap v8320: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:35:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:35:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:54 smithi122 ceph-mon[32255]: pgmap v8321: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:54 smithi022 ceph-mon[31360]: pgmap v8321: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:54 smithi022 ceph-mon[40250]: pgmap v8321: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:55.209 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:35:55.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:35:55.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:35:55.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:35:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:56 smithi122 ceph-mon[32255]: pgmap v8322: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/329342836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:56 smithi022 ceph-mon[31360]: pgmap v8322: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/329342836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:56 smithi022 ceph-mon[40250]: pgmap v8322: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:35:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/329342836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:35:57.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:35:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:35:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:35:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:58 smithi122 ceph-mon[32255]: pgmap v8323: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:35:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:58 smithi022 ceph-mon[31360]: pgmap v8323: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:35:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:35:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:58 smithi022 ceph-mon[40250]: pgmap v8323: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:35:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:35:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:00.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:00 smithi022 ceph-mon[31360]: pgmap v8324: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:00 smithi022 ceph-mon[40250]: pgmap v8324: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:00 smithi122 ceph-mon[32255]: pgmap v8324: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:00] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:36:00.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:00.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:00.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:00.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3047293913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3047293913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3047293913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:02 smithi122 ceph-mon[32255]: pgmap v8325: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:02 smithi022 ceph-mon[31360]: pgmap v8325: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:02 smithi022 ceph-mon[40250]: pgmap v8325: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:36:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:36:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:04 smithi122 ceph-mon[32255]: pgmap v8326: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:04 smithi022 ceph-mon[40250]: pgmap v8326: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:04 smithi022 ceph-mon[31360]: pgmap v8326: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:05.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:06 smithi022 ceph-mon[31360]: pgmap v8327: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:06 smithi022 ceph-mon[40250]: pgmap v8327: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:06.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:06.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:06.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:06 smithi122 ceph-mon[32255]: pgmap v8327: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:07.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:36:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:36:07.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1953186058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1953186058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:36:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1953186058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:08 smithi122 ceph-mon[32255]: pgmap v8328: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:08 smithi022 ceph-mon[31360]: pgmap v8328: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:08 smithi022 ceph-mon[40250]: pgmap v8328: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:10 smithi022 ceph-mon[31360]: pgmap v8329: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:10 smithi022 ceph-mon[40250]: pgmap v8329: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:10 smithi122 ceph-mon[32255]: pgmap v8329: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:36:11.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:11.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:11.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:11.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:12 smithi122 ceph-mon[32255]: pgmap v8330: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3142274694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:36:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:36:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:12 smithi022 ceph-mon[31360]: pgmap v8330: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3142274694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:12 smithi022 ceph-mon[40250]: pgmap v8330: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3142274694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:14 smithi122 ceph-mon[32255]: pgmap v8331: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:14 smithi022 ceph-mon[31360]: pgmap v8331: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:14 smithi022 ceph-mon[40250]: pgmap v8331: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:16 smithi122 ceph-mon[32255]: pgmap v8332: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:16 smithi022 ceph-mon[31360]: pgmap v8332: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:16 smithi022 ceph-mon[40250]: pgmap v8332: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:16.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:17.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2249431141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2249431141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:17.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:36:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:36:17.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2249431141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:36:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:18 smithi122 ceph-mon[32255]: pgmap v8333: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:18 smithi022 ceph-mon[31360]: pgmap v8333: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:18 smithi022 ceph-mon[40250]: pgmap v8333: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:20 smithi022 ceph-mon[31360]: pgmap v8334: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:20 smithi022 ceph-mon[40250]: pgmap v8334: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:20] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:36:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:20 smithi122 ceph-mon[32255]: pgmap v8334: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:22.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:22 smithi122 ceph-mon[32255]: pgmap v8335: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:36:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:36:22.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:22.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:22.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:22.444 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:22 smithi022 ceph-mon[31360]: pgmap v8335: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:22.444 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:22 smithi022 ceph-mon[40250]: pgmap v8335: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1229437413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1229437413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1229437413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:24 smithi122 ceph-mon[32255]: pgmap v8336: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:24 smithi022 ceph-mon[31360]: pgmap v8336: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:24 smithi022 ceph-mon[40250]: pgmap v8336: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:26 smithi122 ceph-mon[32255]: pgmap v8337: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:26 smithi022 ceph-mon[31360]: pgmap v8337: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:26 smithi022 ceph-mon[40250]: pgmap v8337: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:27.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:27.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:36:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:36:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:36:27.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:27.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:27.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:28 smithi122 ceph-mon[32255]: pgmap v8338: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3357297609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:28 smithi022 ceph-mon[31360]: pgmap v8338: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3357297609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:28 smithi022 ceph-mon[40250]: pgmap v8338: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3357297609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:30 smithi122 ceph-mon[32255]: pgmap v8339: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:30] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:36:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:30 smithi022 ceph-mon[31360]: pgmap v8339: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:30 smithi022 ceph-mon[40250]: pgmap v8339: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:32 smithi122 ceph-mon[32255]: pgmap v8340: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:36:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:36:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:32 smithi022 ceph-mon[31360]: pgmap v8340: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:32 smithi022 ceph-mon[40250]: pgmap v8340: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:32.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:33.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:33.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:33.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:36:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:36:33.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:36:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:36:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:36:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:36:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:36:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:36:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:36:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:34 smithi122 ceph-mon[32255]: pgmap v8341: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:36:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3803721430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:36:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:36:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:36:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:36:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[31360]: pgmap v8341: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:36:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3803721430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:36:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:36:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:36:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:36:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[40250]: pgmap v8341: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:36:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3803721430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:36:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:36:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:36:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:36:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:36 smithi122 ceph-mon[32255]: pgmap v8342: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:36 smithi022 ceph-mon[31360]: pgmap v8342: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:36 smithi022 ceph-mon[40250]: pgmap v8342: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:37.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:36:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:36:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:36:38.194 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:38 smithi122 ceph-mon[32255]: pgmap v8343: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:38 smithi022 ceph-mon[31360]: pgmap v8343: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:38 smithi022 ceph-mon[40250]: pgmap v8343: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:38.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:38.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:38.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2426842498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2426842498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2426842498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:36:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:40 smithi122 ceph-mon[32255]: pgmap v8344: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:40 smithi022 ceph-mon[31360]: pgmap v8344: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:40 smithi022 ceph-mon[40250]: pgmap v8344: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:42 smithi122 ceph-mon[32255]: pgmap v8345: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:36:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:36:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:42 smithi022 ceph-mon[31360]: pgmap v8345: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:42 smithi022 ceph-mon[40250]: pgmap v8345: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:43.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:43.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:43.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:44 smithi022 ceph-mon[31360]: pgmap v8346: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3005000195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:44 smithi022 ceph-mon[40250]: pgmap v8346: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3005000195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:44 smithi122 ceph-mon[32255]: pgmap v8346: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3005000195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:46 smithi122 ceph-mon[32255]: pgmap v8347: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:46 smithi022 ceph-mon[31360]: pgmap v8347: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:46 smithi022 ceph-mon[40250]: pgmap v8347: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:47.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:36:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:36:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:36:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:48 smithi122 ceph-mon[32255]: pgmap v8348: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:48 smithi022 ceph-mon[31360]: pgmap v8348: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:48 smithi022 ceph-mon[40250]: pgmap v8348: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:48.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:49.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:49.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:49.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:50] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:36:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:50 smithi122 ceph-mon[32255]: pgmap v8349: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/881222180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:50 smithi022 ceph-mon[31360]: pgmap v8349: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/881222180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:50 smithi022 ceph-mon[40250]: pgmap v8349: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/881222180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:52 smithi122 ceph-mon[32255]: pgmap v8350: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:36:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:36:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:52 smithi022 ceph-mon[31360]: pgmap v8350: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:52 smithi022 ceph-mon[40250]: pgmap v8350: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:54.304 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:36:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:54 smithi122 ceph-mon[32255]: pgmap v8351: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:54 smithi022 ceph-mon[31360]: pgmap v8351: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:54 smithi022 ceph-mon[40250]: pgmap v8351: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:54.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:36:54.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:36:54.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:36:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1640647748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:55.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1640647748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1640647748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:36:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:56 smithi122 ceph-mon[32255]: pgmap v8352: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:56 smithi022 ceph-mon[31360]: pgmap v8352: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:56 smithi022 ceph-mon[40250]: pgmap v8352: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:36:57.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:36:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:36:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:36:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:58 smithi122 ceph-mon[32255]: pgmap v8353: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:36:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:58 smithi022 ceph-mon[31360]: pgmap v8353: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:36:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:58 smithi022 ceph-mon[40250]: pgmap v8353: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:36:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:36:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:36:59.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:00.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:00.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:00.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:00] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:37:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:00 smithi122 ceph-mon[32255]: pgmap v8354: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4268535821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:00 smithi022 ceph-mon[31360]: pgmap v8354: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4268535821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:00 smithi022 ceph-mon[40250]: pgmap v8354: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4268535821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:37:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:37:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:02 smithi122 ceph-mon[32255]: pgmap v8355: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:02 smithi022 ceph-mon[31360]: pgmap v8355: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:02 smithi022 ceph-mon[40250]: pgmap v8355: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:04 smithi122 ceph-mon[32255]: pgmap v8356: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:04 smithi022 ceph-mon[31360]: pgmap v8356: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:04 smithi022 ceph-mon[40250]: pgmap v8356: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:05.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:05.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:05.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:05.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:06 smithi122 ceph-mon[32255]: pgmap v8357: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3899981200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:06 smithi022 ceph-mon[31360]: pgmap v8357: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3899981200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:06 smithi022 ceph-mon[40250]: pgmap v8357: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3899981200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:07.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:37:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:37:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:37:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:08 smithi122 ceph-mon[32255]: pgmap v8358: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:08 smithi022 ceph-mon[31360]: pgmap v8358: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:08 smithi022 ceph-mon[40250]: pgmap v8358: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:10.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:10.420 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:10 smithi022 ceph-mon[31360]: pgmap v8359: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:10.421 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:10 smithi022 ceph-mon[40250]: pgmap v8359: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:10] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:37:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:10 smithi122 ceph-mon[32255]: pgmap v8359: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:10.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:10.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:10.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3384961137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:11.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3384961137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3384961137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:12 smithi122 ceph-mon[32255]: pgmap v8360: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:12 smithi022 ceph-mon[31360]: pgmap v8360: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:12 smithi022 ceph-mon[40250]: pgmap v8360: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:37:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:37:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:14 smithi122 ceph-mon[32255]: pgmap v8361: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:14 smithi022 ceph-mon[31360]: pgmap v8361: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:14 smithi022 ceph-mon[40250]: pgmap v8361: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:15.785 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:16.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:16.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:16.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:16 smithi122 ceph-mon[32255]: pgmap v8362: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3389533245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:16 smithi022 ceph-mon[31360]: pgmap v8362: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3389533245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:16 smithi022 ceph-mon[40250]: pgmap v8362: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3389533245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:17.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:37:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:37:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:37:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:18 smithi122 ceph-mon[32255]: pgmap v8363: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:18 smithi022 ceph-mon[40250]: pgmap v8363: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:18 smithi022 ceph-mon[31360]: pgmap v8363: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:20 smithi122 ceph-mon[32255]: pgmap v8364: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:37:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:20 smithi022 ceph-mon[31360]: pgmap v8364: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:20 smithi022 ceph-mon[40250]: pgmap v8364: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:21.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:21.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:21.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:21.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:37:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:37:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:22 smithi122 ceph-mon[32255]: pgmap v8365: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/638507596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:22 smithi022 ceph-mon[31360]: pgmap v8365: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/638507596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:22 smithi022 ceph-mon[40250]: pgmap v8365: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/638507596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:24 smithi122 ceph-mon[32255]: pgmap v8366: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:24 smithi022 ceph-mon[31360]: pgmap v8366: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:24 smithi022 ceph-mon[40250]: pgmap v8366: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:37:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:26 smithi122 ceph-mon[32255]: pgmap v8367: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:26.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:26 smithi022 ceph-mon[40250]: pgmap v8367: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:26.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:26 smithi022 ceph-mon[31360]: pgmap v8367: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:26.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:26.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:26.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:27.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4096002631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:27.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:37:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:37:27.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4096002631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:27.531 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:37:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4096002631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:28 smithi122 ceph-mon[32255]: pgmap v8368: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:28 smithi022 ceph-mon[31360]: pgmap v8368: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:28 smithi022 ceph-mon[40250]: pgmap v8368: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:30 smithi122 ceph-mon[32255]: pgmap v8369: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:37:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:30 smithi022 ceph-mon[31360]: pgmap v8369: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:30 smithi022 ceph-mon[40250]: pgmap v8369: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:31.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:32.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:32.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:32.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:32 smithi122 ceph-mon[32255]: pgmap v8370: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:37:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:37:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:32 smithi022 ceph-mon[31360]: pgmap v8370: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:32 smithi022 ceph-mon[40250]: pgmap v8370: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2620917569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2620917569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2620917569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: pgmap v8371: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:37:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:37:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:37:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:37:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:37:34.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:37:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: pgmap v8371: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:37:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:37:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:37:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:37:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:37:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:37:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:37:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:37:34.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: pgmap v8371: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:34.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:37:34.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:37:34.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:37:34.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:37:34.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:37:34.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:37:34.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:37:34.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:37:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:36 smithi122 ceph-mon[32255]: pgmap v8372: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:36 smithi022 ceph-mon[31360]: pgmap v8372: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:36 smithi022 ceph-mon[40250]: pgmap v8372: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:37.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:37:37.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:37.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:37.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:37.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:37:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:37:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:38 smithi122 ceph-mon[32255]: pgmap v8373: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1413280584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:38 smithi022 ceph-mon[31360]: pgmap v8373: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1413280584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:38 smithi022 ceph-mon[40250]: pgmap v8373: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:38.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1413280584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:37:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:40 smithi122 ceph-mon[32255]: pgmap v8374: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:40 smithi022 ceph-mon[31360]: pgmap v8374: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:40 smithi022 ceph-mon[40250]: pgmap v8374: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:42 smithi122 ceph-mon[32255]: pgmap v8375: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:37:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:37:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:42 smithi022 ceph-mon[31360]: pgmap v8375: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:42 smithi022 ceph-mon[40250]: pgmap v8375: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:42.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:43.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:43.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:43.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:43.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3606554331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:43.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3606554331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3606554331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:44 smithi122 ceph-mon[32255]: pgmap v8376: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:44 smithi022 ceph-mon[31360]: pgmap v8376: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:44 smithi022 ceph-mon[40250]: pgmap v8376: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:46 smithi122 ceph-mon[32255]: pgmap v8377: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:46 smithi022 ceph-mon[31360]: pgmap v8377: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:46 smithi022 ceph-mon[40250]: pgmap v8377: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:47.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:37:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:37:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:37:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:48 smithi022 ceph-mon[31360]: pgmap v8378: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:48 smithi022 ceph-mon[40250]: pgmap v8378: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:48.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:48.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:48.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:48.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:48 smithi122 ceph-mon[32255]: pgmap v8378: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/367033533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/367033533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/367033533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:37:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:50 smithi122 ceph-mon[32255]: pgmap v8379: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:50 smithi022 ceph-mon[31360]: pgmap v8379: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:50 smithi022 ceph-mon[40250]: pgmap v8379: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:37:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:52 smithi022 ceph-mon[31360]: pgmap v8380: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:52 smithi022 ceph-mon[40250]: pgmap v8380: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:52 smithi122 ceph-mon[32255]: pgmap v8380: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:37:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:37:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:37:53.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:53.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:53.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:53.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:37:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:54 smithi022 ceph-mon[31360]: pgmap v8381: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:37:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/559155798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:54 smithi022 ceph-mon[40250]: pgmap v8381: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:37:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/559155798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:54 smithi122 ceph-mon[32255]: pgmap v8381: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:37:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/559155798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:37:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:56 smithi022 ceph-mon[31360]: pgmap v8382: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:37:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:56 smithi022 ceph-mon[40250]: pgmap v8382: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:37:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:56 smithi122 ceph-mon[32255]: pgmap v8382: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:37:57.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:37:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:37:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:37:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:58 smithi022 ceph-mon[31360]: pgmap v8383: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:37:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:37:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:58 smithi022 ceph-mon[40250]: pgmap v8383: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:37:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:37:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:58 smithi122 ceph-mon[32255]: pgmap v8383: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:37:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:37:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:37:58.748 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:37:59.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:37:59.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:37:59.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3694626379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:00 smithi122 ceph-mon[32255]: pgmap v8384: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:38:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:38:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3694626379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:00 smithi022 ceph-mon[31360]: pgmap v8384: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:38:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3694626379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:00 smithi022 ceph-mon[40250]: pgmap v8384: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:38:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:02 smithi122 ceph-mon[32255]: pgmap v8385: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:38:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:38:02.320Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:38:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:02 smithi022 ceph-mon[31360]: pgmap v8385: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:38:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:02 smithi022 ceph-mon[40250]: pgmap v8385: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:38:04.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:04 smithi122 ceph-mon[32255]: pgmap v8386: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-15T22:38:04.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:04.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:04.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:04.488 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:04 smithi022 ceph-mon[31360]: pgmap v8386: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-15T22:38:04.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:04 smithi022 ceph-mon[40250]: pgmap v8386: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-15T22:38:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/267329594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/267329594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/267329594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:06 smithi022 ceph-mon[31360]: pgmap v8387: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:38:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:06 smithi022 ceph-mon[40250]: pgmap v8387: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:38:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:06 smithi122 ceph-mon[32255]: pgmap v8387: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:38:07.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:38:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:38:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:38:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:08 smithi022 ceph-mon[31360]: pgmap v8388: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:38:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:08 smithi022 ceph-mon[40250]: pgmap v8388: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:38:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:08 smithi122 ceph-mon[32255]: pgmap v8388: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:38:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:09.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:09.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:09.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:10 smithi022 ceph-mon[31360]: pgmap v8389: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:38:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2140262491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:10 smithi022 ceph-mon[40250]: pgmap v8389: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:38:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2140262491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:10 smithi122 ceph-mon[32255]: pgmap v8389: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:38:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2140262491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:10] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:38:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:12 smithi122 ceph-mon[32255]: pgmap v8390: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:38:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:38:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:38:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:12 smithi022 ceph-mon[31360]: pgmap v8390: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:38:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:12 smithi022 ceph-mon[40250]: pgmap v8390: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:38:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:14 smithi122 ceph-mon[32255]: pgmap v8391: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:38:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:14 smithi022 ceph-mon[31360]: pgmap v8391: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:38:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:14 smithi022 ceph-mon[40250]: pgmap v8391: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:38:14.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:15.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:15.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:15.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:16 smithi022 ceph-mon[31360]: pgmap v8392: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T22:38:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3429621579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:16 smithi022 ceph-mon[40250]: pgmap v8392: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T22:38:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3429621579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:16 smithi122 ceph-mon[32255]: pgmap v8392: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T22:38:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3429621579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:17.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:38:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:38:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:38:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:18 smithi022 ceph-mon[31360]: pgmap v8393: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:18 smithi022 ceph-mon[40250]: pgmap v8393: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:18 smithi122 ceph-mon[32255]: pgmap v8393: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:20.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:20 smithi122 ceph-mon[32255]: pgmap v8394: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:38:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:20 smithi022 ceph-mon[31360]: pgmap v8394: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:20 smithi022 ceph-mon[40250]: pgmap v8394: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:20.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:20.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:20.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:21.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3432053225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3432053225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:21.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3432053225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:22 smithi122 ceph-mon[32255]: pgmap v8395: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:22 smithi022 ceph-mon[31360]: pgmap v8395: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:22 smithi022 ceph-mon[40250]: pgmap v8395: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:38:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:38:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:24 smithi022 ceph-mon[31360]: pgmap v8396: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:24 smithi022 ceph-mon[40250]: pgmap v8396: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:24 smithi122 ceph-mon[32255]: pgmap v8396: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:25.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:26 smithi022 ceph-mon[31360]: pgmap v8397: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744694975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:26 smithi022 ceph-mon[40250]: pgmap v8397: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744694975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:26 smithi122 ceph-mon[32255]: pgmap v8397: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744694975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:27.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:38:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:38:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:38:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:28 smithi022 ceph-mon[31360]: pgmap v8398: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:28 smithi022 ceph-mon[40250]: pgmap v8398: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:28 smithi122 ceph-mon[32255]: pgmap v8398: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:30 smithi122 ceph-mon[32255]: pgmap v8399: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:38:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:30 smithi022 ceph-mon[31360]: pgmap v8399: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:30 smithi022 ceph-mon[40250]: pgmap v8399: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:30.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:31.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:31.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:31.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:32 smithi022 ceph-mon[31360]: pgmap v8400: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/944807866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:32 smithi022 ceph-mon[40250]: pgmap v8400: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/944807866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:32 smithi122 ceph-mon[32255]: pgmap v8400: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/944807866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:38:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:38:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: pgmap v8401: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:38:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:38:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:38:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:38:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:38:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:38:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:38:34.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:38:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: pgmap v8401: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:34.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:38:34.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:38:34.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:38:34.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:38:34.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:38:34.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:38:34.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:38:34.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:38:34.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: pgmap v8401: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:34.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:38:34.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:38:34.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:38:34.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:38:34.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:38:34.537 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:38:34.537 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:38:34.537 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:38:36.349 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:36 smithi022 ceph-mon[31360]: pgmap v8402: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:36 smithi022 ceph-mon[40250]: pgmap v8402: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:36 smithi122 ceph-mon[32255]: pgmap v8402: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:36.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:38:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2935268829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:37.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2935268829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2935268829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:37.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:38:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:38:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:38 smithi022 ceph-mon[31360]: pgmap v8403: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:38 smithi022 ceph-mon[40250]: pgmap v8403: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:38 smithi122 ceph-mon[32255]: pgmap v8403: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:38:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:40 smithi122 ceph-mon[32255]: pgmap v8404: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:40 smithi022 ceph-mon[31360]: pgmap v8404: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:40 smithi022 ceph-mon[40250]: pgmap v8404: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:41.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:42.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:42.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:42 smithi022 ceph-mon[31360]: pgmap v8405: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3818826634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:42.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:42 smithi022 ceph-mon[40250]: pgmap v8405: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:42.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3818826634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:38:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:38:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:42 smithi122 ceph-mon[32255]: pgmap v8405: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3818826634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:44 smithi022 ceph-mon[31360]: pgmap v8406: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:44 smithi022 ceph-mon[40250]: pgmap v8406: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:44 smithi122 ceph-mon[32255]: pgmap v8406: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:46 smithi022 ceph-mon[31360]: pgmap v8407: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:46 smithi022 ceph-mon[40250]: pgmap v8407: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:46 smithi122 ceph-mon[32255]: pgmap v8407: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:47.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:47.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:47.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:47.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:47.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:38:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:38:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:38:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:48 smithi022 ceph-mon[31360]: pgmap v8408: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1750234135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:48 smithi022 ceph-mon[40250]: pgmap v8408: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1750234135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:48 smithi122 ceph-mon[32255]: pgmap v8408: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1750234135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:50 smithi122 ceph-mon[32255]: pgmap v8409: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:50] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:38:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:50 smithi022 ceph-mon[31360]: pgmap v8409: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:50 smithi022 ceph-mon[40250]: pgmap v8409: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:52.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:52.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:52 smithi022 ceph-mon[31360]: pgmap v8410: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:52.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:52 smithi022 ceph-mon[40250]: pgmap v8410: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:52 smithi122 ceph-mon[32255]: pgmap v8410: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:38:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:38:52.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:52.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:52.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1397270145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1397270145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:53.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1397270145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:54 smithi022 ceph-mon[31360]: pgmap v8411: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:54 smithi022 ceph-mon[40250]: pgmap v8411: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:54 smithi122 ceph-mon[32255]: pgmap v8411: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:38:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:56 smithi022 ceph-mon[31360]: pgmap v8412: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:56 smithi022 ceph-mon[40250]: pgmap v8412: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:56 smithi122 ceph-mon[32255]: pgmap v8412: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:57.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:38:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:38:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:38:57.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:38:58.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:38:58.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:38:58.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:38:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:58 smithi022 ceph-mon[31360]: pgmap v8413: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:38:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/968490440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:58 smithi022 ceph-mon[40250]: pgmap v8413: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:38:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/968490440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:38:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:58 smithi122 ceph-mon[32255]: pgmap v8413: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:38:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:38:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:38:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/968490440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:00 smithi122 ceph-mon[32255]: pgmap v8414: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:00] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:39:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:00 smithi022 ceph-mon[31360]: pgmap v8414: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:00 smithi022 ceph-mon[40250]: pgmap v8414: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:02 smithi022 ceph-mon[31360]: pgmap v8415: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:02.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:02 smithi022 ceph-mon[40250]: pgmap v8415: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:02 smithi122 ceph-mon[32255]: pgmap v8415: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:39:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:39:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:03.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:03.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:03.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:04 smithi022 ceph-mon[31360]: pgmap v8416: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:04.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2102403538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:04 smithi022 ceph-mon[40250]: pgmap v8416: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2102403538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:04 smithi122 ceph-mon[32255]: pgmap v8416: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2102403538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:06 smithi022 ceph-mon[31360]: pgmap v8417: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:06 smithi022 ceph-mon[40250]: pgmap v8417: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:06 smithi122 ceph-mon[32255]: pgmap v8417: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:07.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:39:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:39:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:39:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:08 smithi022 ceph-mon[31360]: pgmap v8418: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:08 smithi022 ceph-mon[40250]: pgmap v8418: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:08.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:08 smithi122 ceph-mon[32255]: pgmap v8418: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:08.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:08.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:08.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3641497462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3641497462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3641497462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:10] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:39:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:10 smithi122 ceph-mon[32255]: pgmap v8419: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:10 smithi022 ceph-mon[31360]: pgmap v8419: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:10 smithi022 ceph-mon[40250]: pgmap v8419: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:12 smithi022 ceph-mon[31360]: pgmap v8420: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:12 smithi022 ceph-mon[40250]: pgmap v8420: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:12 smithi122 ceph-mon[32255]: pgmap v8420: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:39:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:39:13.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:14.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:14.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:14.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:14 smithi022 ceph-mon[31360]: pgmap v8421: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:14 smithi022 ceph-mon[40250]: pgmap v8421: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:14 smithi122 ceph-mon[32255]: pgmap v8421: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:15.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2299308687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2299308687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2299308687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:16 smithi022 ceph-mon[31360]: pgmap v8422: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:16 smithi022 ceph-mon[40250]: pgmap v8422: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:16 smithi122 ceph-mon[32255]: pgmap v8422: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:17.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:39:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:39:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:39:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:18 smithi022 ceph-mon[31360]: pgmap v8423: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:18 smithi022 ceph-mon[40250]: pgmap v8423: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:18 smithi122 ceph-mon[32255]: pgmap v8423: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:19.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:19.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:19.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:19.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:39:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:20 smithi122 ceph-mon[32255]: pgmap v8424: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1639785312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:20 smithi022 ceph-mon[31360]: pgmap v8424: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1639785312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:20 smithi022 ceph-mon[40250]: pgmap v8424: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1639785312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:22 smithi022 ceph-mon[31360]: pgmap v8425: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:22 smithi022 ceph-mon[40250]: pgmap v8425: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:22 smithi122 ceph-mon[32255]: pgmap v8425: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:39:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:39:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:24 smithi022 ceph-mon[31360]: pgmap v8426: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:24 smithi022 ceph-mon[40250]: pgmap v8426: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:24.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:24 smithi122 ceph-mon[32255]: pgmap v8426: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:25.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:25.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:25.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/885366941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/885366941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:25.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/885366941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:26 smithi022 ceph-mon[31360]: pgmap v8427: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:26 smithi022 ceph-mon[40250]: pgmap v8427: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:26 smithi122 ceph-mon[32255]: pgmap v8427: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:27.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:39:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:39:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:39:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:28 smithi022 ceph-mon[31360]: pgmap v8428: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:28 smithi022 ceph-mon[40250]: pgmap v8428: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:28 smithi122 ceph-mon[32255]: pgmap v8428: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:30.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:30.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:30.434 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:30 smithi022 ceph-mon[31360]: pgmap v8429: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:30.435 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:30 smithi022 ceph-mon[40250]: pgmap v8429: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:30 smithi122 ceph-mon[32255]: pgmap v8429: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:39:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2474780165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2474780165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2474780165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:32 smithi122 ceph-mon[32255]: pgmap v8430: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:39:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:39:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:32 smithi022 ceph-mon[31360]: pgmap v8430: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:32 smithi022 ceph-mon[40250]: pgmap v8430: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:39:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:39:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:39:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:39:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:39:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:39:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:39:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:39:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:39:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:39:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:39:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:39:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:34 smithi122 ceph-mon[32255]: pgmap v8431: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:39:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:39:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:39:34.494 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[31360]: pgmap v8431: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:34.494 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:39:34.495 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:39:34.495 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:39:34.495 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[40250]: pgmap v8431: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:34.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:39:34.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:39:34.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:39:35.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[31360]: pgmap v8432: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:35 smithi022 ceph-mon[40250]: pgmap v8432: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:35.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:35.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:35.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:39:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:35 smithi122 ceph-mon[32255]: pgmap v8432: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2623605417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2623605417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2623605417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:37 smithi122 ceph-mon[32255]: pgmap v8433: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:37.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:37.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:39:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:39:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:39:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:37 smithi022 ceph-mon[31360]: pgmap v8433: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:37.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:37 smithi022 ceph-mon[40250]: pgmap v8433: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:39:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:40 smithi122 ceph-mon[32255]: pgmap v8434: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:40 smithi022 ceph-mon[31360]: pgmap v8434: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:40 smithi022 ceph-mon[40250]: pgmap v8434: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:40.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:41.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:41.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:41.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3525712102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3525712102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3525712102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:42 smithi022 ceph-mon[31360]: pgmap v8435: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:42 smithi022 ceph-mon[40250]: pgmap v8435: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:42 smithi122 ceph-mon[32255]: pgmap v8435: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:39:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:39:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:44 smithi022 ceph-mon[31360]: pgmap v8436: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:44 smithi022 ceph-mon[40250]: pgmap v8436: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:44 smithi122 ceph-mon[32255]: pgmap v8436: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:46.166 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:46 smithi022 ceph-mon[31360]: pgmap v8437: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:46 smithi022 ceph-mon[40250]: pgmap v8437: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:46.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:46.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:46.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:46 smithi122 ceph-mon[32255]: pgmap v8437: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:47.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1093357437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:39:47.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1093357437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1093357437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:47.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:39:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:39:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:48 smithi022 ceph-mon[31360]: pgmap v8438: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:48 smithi022 ceph-mon[40250]: pgmap v8438: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:48 smithi122 ceph-mon[32255]: pgmap v8438: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:50 smithi122 ceph-mon[32255]: pgmap v8439: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:50] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:39:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:50 smithi022 ceph-mon[31360]: pgmap v8439: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:50 smithi022 ceph-mon[40250]: pgmap v8439: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:51.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:51.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:51.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:51.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:52 smithi022 ceph-mon[31360]: pgmap v8440: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2198855874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:52 smithi022 ceph-mon[40250]: pgmap v8440: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2198855874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:39:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:39:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:52 smithi122 ceph-mon[32255]: pgmap v8440: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2198855874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:54 smithi022 ceph-mon[31360]: pgmap v8441: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:54 smithi022 ceph-mon[40250]: pgmap v8441: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:54 smithi122 ceph-mon[32255]: pgmap v8441: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:39:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:56 smithi022 ceph-mon[31360]: pgmap v8442: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:56 smithi022 ceph-mon[40250]: pgmap v8442: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:56 smithi122 ceph-mon[32255]: pgmap v8442: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:56.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:39:57.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:39:57.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:39:57.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:39:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:39:57.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:39:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:39:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:58 smithi022 ceph-mon[31360]: pgmap v8443: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1559520870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:58 smithi022 ceph-mon[40250]: pgmap v8443: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1559520870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:58 smithi122 ceph-mon[32255]: pgmap v8443: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:39:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1559520870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:39:59.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:39:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:59.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:39:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:39:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:39:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:00 smithi122 ceph-mon[32255]: pgmap v8444: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T22:40:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:00] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:40:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:00 smithi022 ceph-mon[31360]: pgmap v8444: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T22:40:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:00 smithi022 ceph-mon[40250]: pgmap v8444: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T22:40:02.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:02 smithi022 ceph-mon[31360]: pgmap v8445: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:02 smithi022 ceph-mon[40250]: pgmap v8445: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:02 smithi122 ceph-mon[32255]: pgmap v8445: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:40:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:40:03.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3026432807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3026432807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3026432807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:04 smithi022 ceph-mon[31360]: pgmap v8446: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:04 smithi022 ceph-mon[40250]: pgmap v8446: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:04 smithi122 ceph-mon[32255]: pgmap v8446: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:06 smithi022 ceph-mon[31360]: pgmap v8447: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:06 smithi022 ceph-mon[40250]: pgmap v8447: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:06 smithi122 ceph-mon[32255]: pgmap v8447: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:07.643 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:07.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:40:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:40:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:40:08.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:08.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:08.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:08 smithi022 ceph-mon[31360]: pgmap v8448: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/46341838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:08 smithi022 ceph-mon[40250]: pgmap v8448: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:08.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/46341838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:08 smithi122 ceph-mon[32255]: pgmap v8448: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/46341838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:10 smithi122 ceph-mon[32255]: pgmap v8449: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:40:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:10 smithi022 ceph-mon[31360]: pgmap v8449: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:10 smithi022 ceph-mon[40250]: pgmap v8449: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:12 smithi022 ceph-mon[31360]: pgmap v8450: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:12 smithi022 ceph-mon[40250]: pgmap v8450: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:12 smithi122 ceph-mon[32255]: pgmap v8450: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:40:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:40:13.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:13.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:13.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:13.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:14 smithi022 ceph-mon[31360]: pgmap v8451: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2087782208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:14 smithi022 ceph-mon[40250]: pgmap v8451: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2087782208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:14 smithi122 ceph-mon[32255]: pgmap v8451: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2087782208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:16 smithi022 ceph-mon[31360]: pgmap v8452: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:16 smithi022 ceph-mon[40250]: pgmap v8452: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:16 smithi122 ceph-mon[32255]: pgmap v8452: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:40:17.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:40:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:40:18.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:18 smithi022 ceph-mon[31360]: pgmap v8453: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:18 smithi022 ceph-mon[40250]: pgmap v8453: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:18 smithi122 ceph-mon[32255]: pgmap v8453: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3988083147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3988083147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3988083147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:20 smithi122 ceph-mon[32255]: pgmap v8454: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:20] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:40:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:20 smithi022 ceph-mon[31360]: pgmap v8454: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:20 smithi022 ceph-mon[40250]: pgmap v8454: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:22 smithi022 ceph-mon[31360]: pgmap v8455: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:22 smithi022 ceph-mon[40250]: pgmap v8455: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:22 smithi122 ceph-mon[32255]: pgmap v8455: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:40:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:40:23.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:24.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:24.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:24.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:24 smithi022 ceph-mon[31360]: pgmap v8456: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3352961790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:24 smithi022 ceph-mon[40250]: pgmap v8456: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3352961790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:24 smithi122 ceph-mon[32255]: pgmap v8456: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3352961790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:26 smithi022 ceph-mon[31360]: pgmap v8457: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:26 smithi022 ceph-mon[40250]: pgmap v8457: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:26 smithi122 ceph-mon[32255]: pgmap v8457: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:40:27.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:40:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:40:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:28 smithi022 ceph-mon[31360]: pgmap v8458: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:28 smithi022 ceph-mon[40250]: pgmap v8458: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:28 smithi122 ceph-mon[32255]: pgmap v8458: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:29.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:29.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:29.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:29.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:29.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:29.505 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:30] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:40:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:30 smithi122 ceph-mon[32255]: pgmap v8459: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3081448822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:30 smithi022 ceph-mon[31360]: pgmap v8459: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3081448822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:30 smithi022 ceph-mon[40250]: pgmap v8459: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3081448822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:32 smithi022 ceph-mon[31360]: pgmap v8460: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:32 smithi022 ceph-mon[40250]: pgmap v8460: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:32 smithi122 ceph-mon[32255]: pgmap v8460: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:40:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:40:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:40:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:40:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:40:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:40:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:40:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:40:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:40:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:40:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:40:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:40:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:40:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:40:34.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:34 smithi022 ceph-mon[31360]: pgmap v8461: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:34 smithi022 ceph-mon[40250]: pgmap v8461: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:34 smithi122 ceph-mon[32255]: pgmap v8461: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:34.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:34.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:34.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:35.284 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3904369785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:35.284 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:40:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:40:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:40:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3904369785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:40:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:40:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:40:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3904369785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:40:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:40:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:40:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:36 smithi122 ceph-mon[32255]: pgmap v8462: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:40:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:36 smithi022 ceph-mon[31360]: pgmap v8462: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:40:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:36 smithi022 ceph-mon[40250]: pgmap v8462: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:40:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:40:37.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:40:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:40:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:38 smithi122 ceph-mon[32255]: pgmap v8463: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:38 smithi022 ceph-mon[31360]: pgmap v8463: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:38.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:38 smithi022 ceph-mon[40250]: pgmap v8463: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:39.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:40.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:40.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:40.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:40:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:40 smithi122 ceph-mon[32255]: pgmap v8464: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1583163205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:40 smithi022 ceph-mon[31360]: pgmap v8464: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1583163205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:40 smithi022 ceph-mon[40250]: pgmap v8464: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1583163205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:42 smithi122 ceph-mon[32255]: pgmap v8465: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:40:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:40:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:42 smithi022 ceph-mon[31360]: pgmap v8465: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:42 smithi022 ceph-mon[40250]: pgmap v8465: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:44 smithi122 ceph-mon[32255]: pgmap v8466: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:44 smithi022 ceph-mon[31360]: pgmap v8466: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:44 smithi022 ceph-mon[40250]: pgmap v8466: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:45.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:45.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:45.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:45.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:46 smithi122 ceph-mon[32255]: pgmap v8467: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3016145434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:46 smithi022 ceph-mon[31360]: pgmap v8467: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3016145434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:46 smithi022 ceph-mon[40250]: pgmap v8467: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3016145434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:40:47.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:40:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:40:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:48 smithi122 ceph-mon[32255]: pgmap v8468: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:48 smithi022 ceph-mon[31360]: pgmap v8468: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:48 smithi022 ceph-mon[40250]: pgmap v8468: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:50 smithi122 ceph-mon[32255]: pgmap v8469: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:50] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:40:50.628 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:50 smithi022 ceph-mon[31360]: pgmap v8469: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:50 smithi022 ceph-mon[40250]: pgmap v8469: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:50.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:50.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:50.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2165116950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2165116950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2165116950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:52 smithi122 ceph-mon[32255]: pgmap v8470: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:40:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:40:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:52 smithi022 ceph-mon[31360]: pgmap v8470: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:52 smithi022 ceph-mon[40250]: pgmap v8470: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:54 smithi122 ceph-mon[32255]: pgmap v8471: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:54 smithi022 ceph-mon[31360]: pgmap v8471: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:54 smithi022 ceph-mon[40250]: pgmap v8471: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:40:55.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:40:56.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:40:56.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:40:56.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:40:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:56 smithi122 ceph-mon[32255]: pgmap v8472: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1948651736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:56 smithi022 ceph-mon[31360]: pgmap v8472: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1948651736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:56 smithi022 ceph-mon[40250]: pgmap v8472: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1948651736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:40:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:40:57.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:40:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:40:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:58 smithi122 ceph-mon[32255]: pgmap v8473: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:58 smithi022 ceph-mon[31360]: pgmap v8473: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:58 smithi022 ceph-mon[40250]: pgmap v8473: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:40:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:40:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:40:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:40:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:40:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:00.430 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:00] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:41:00.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:00 smithi122 ceph-mon[32255]: pgmap v8474: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:00 smithi022 ceph-mon[31360]: pgmap v8474: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:00 smithi022 ceph-mon[40250]: pgmap v8474: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:01.368 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:01.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:01.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:01.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:02 smithi122 ceph-mon[32255]: pgmap v8475: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2068078726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:41:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:41:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:02 smithi022 ceph-mon[31360]: pgmap v8475: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2068078726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:02 smithi022 ceph-mon[40250]: pgmap v8475: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2068078726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:04 smithi122 ceph-mon[32255]: pgmap v8476: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:04 smithi022 ceph-mon[31360]: pgmap v8476: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:04 smithi022 ceph-mon[40250]: pgmap v8476: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:06.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:06.739 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:06 smithi022 ceph-mon[31360]: pgmap v8477: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:06.740 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:06 smithi022 ceph-mon[40250]: pgmap v8477: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:06 smithi122 ceph-mon[32255]: pgmap v8477: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:07.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:07.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:07.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:41:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1000293943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:07 smithi022 ceph-mon[31360]: pgmap v8478: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1000293943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:07 smithi022 ceph-mon[40250]: pgmap v8478: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:07.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:41:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:41:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1000293943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:07 smithi122 ceph-mon[32255]: pgmap v8478: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:10] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:41:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:10 smithi122 ceph-mon[32255]: pgmap v8479: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:10 smithi022 ceph-mon[31360]: pgmap v8479: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:10 smithi022 ceph-mon[40250]: pgmap v8479: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:12.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:12.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:12.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:12.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:12.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:12 smithi022 ceph-mon[40250]: pgmap v8480: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:12.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:12 smithi022 ceph-mon[31360]: pgmap v8480: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:12 smithi122 ceph-mon[32255]: pgmap v8480: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:41:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:41:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3077433681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3077433681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3077433681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:14 smithi022 ceph-mon[31360]: pgmap v8481: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:14 smithi022 ceph-mon[40250]: pgmap v8481: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:14 smithi122 ceph-mon[32255]: pgmap v8481: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:16 smithi022 ceph-mon[31360]: pgmap v8482: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:16 smithi022 ceph-mon[40250]: pgmap v8482: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:16 smithi122 ceph-mon[32255]: pgmap v8482: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:17.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:41:17.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:17.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:17.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:17.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:41:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:41:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:18 smithi022 ceph-mon[31360]: pgmap v8483: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2001549401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:18 smithi022 ceph-mon[40250]: pgmap v8483: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2001549401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:18 smithi122 ceph-mon[32255]: pgmap v8483: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2001549401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T22:41:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:20 smithi122 ceph-mon[32255]: pgmap v8484: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:20 smithi022 ceph-mon[31360]: pgmap v8484: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:20 smithi022 ceph-mon[40250]: pgmap v8484: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:22 smithi022 ceph-mon[31360]: pgmap v8485: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:22 smithi022 ceph-mon[40250]: pgmap v8485: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:22 smithi122 ceph-mon[32255]: pgmap v8485: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:41:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:41:22.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:23.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:23.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:23.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:23.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2159095458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2159095458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2159095458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:24 smithi022 ceph-mon[31360]: pgmap v8486: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:24 smithi022 ceph-mon[40250]: pgmap v8486: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:24 smithi122 ceph-mon[32255]: pgmap v8486: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:26 smithi022 ceph-mon[31360]: pgmap v8487: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:26 smithi022 ceph-mon[40250]: pgmap v8487: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:26 smithi122 ceph-mon[32255]: pgmap v8487: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:41:27.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:41:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:41:28.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:28 smithi022 ceph-mon[31360]: pgmap v8488: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:28 smithi022 ceph-mon[40250]: pgmap v8488: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:28.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:28.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:28 smithi122 ceph-mon[32255]: pgmap v8488: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1722093949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1722093949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1722093949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T22:41:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:30 smithi122 ceph-mon[32255]: pgmap v8489: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:30 smithi022 ceph-mon[31360]: pgmap v8489: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:30 smithi022 ceph-mon[40250]: pgmap v8489: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:32 smithi022 ceph-mon[31360]: pgmap v8490: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:32 smithi022 ceph-mon[40250]: pgmap v8490: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:32 smithi122 ceph-mon[32255]: pgmap v8490: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:41:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:41:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:41:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:41:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:41:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:41:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:41:33.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:41:33.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:41:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:41:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:41:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:41:33.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:33.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:34 smithi022 ceph-mon[31360]: pgmap v8491: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1669995803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:34 smithi022 ceph-mon[40250]: pgmap v8491: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1669995803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:34 smithi122 ceph-mon[32255]: pgmap v8491: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1669995803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:36 smithi122 ceph-mon[32255]: pgmap v8492: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:41:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:41:36.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:41:36.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:41:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[31360]: pgmap v8492: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:41:36.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:41:36.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:41:36.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:41:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[40250]: pgmap v8492: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:36.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:41:36.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:41:36.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:41:36.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:41:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:41:37.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:41:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:41:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:38 smithi122 ceph-mon[32255]: pgmap v8493: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:38 smithi022 ceph-mon[31360]: pgmap v8493: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:38 smithi022 ceph-mon[40250]: pgmap v8493: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:38.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:39.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:39.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:39.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:40] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:41:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:40 smithi122 ceph-mon[32255]: pgmap v8494: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:40.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1195596857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:40 smithi022 ceph-mon[31360]: pgmap v8494: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1195596857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:40 smithi022 ceph-mon[40250]: pgmap v8494: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1195596857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:41:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:41:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:42 smithi122 ceph-mon[32255]: pgmap v8495: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:42 smithi022 ceph-mon[31360]: pgmap v8495: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:42 smithi022 ceph-mon[40250]: pgmap v8495: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:44.335 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:44 smithi122 ceph-mon[32255]: pgmap v8496: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:44.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:44.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:44.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:44.706 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:44 smithi022 ceph-mon[31360]: pgmap v8496: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:44.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:44 smithi022 ceph-mon[40250]: pgmap v8496: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3902084936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3902084936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3902084936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:46 smithi122 ceph-mon[32255]: pgmap v8497: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:46 smithi022 ceph-mon[31360]: pgmap v8497: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:46 smithi022 ceph-mon[40250]: pgmap v8497: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:41:47.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:41:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:41:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:48 smithi122 ceph-mon[32255]: pgmap v8498: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:48 smithi022 ceph-mon[31360]: pgmap v8498: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:48 smithi022 ceph-mon[40250]: pgmap v8498: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:49.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:49.708 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:49.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:50.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:50.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:50.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:50] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:41:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:50 smithi122 ceph-mon[32255]: pgmap v8499: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301556605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:50 smithi022 ceph-mon[31360]: pgmap v8499: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301556605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:50 smithi022 ceph-mon[40250]: pgmap v8499: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301556605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:52 smithi122 ceph-mon[32255]: pgmap v8500: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:41:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:41:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:52 smithi022 ceph-mon[31360]: pgmap v8500: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:52 smithi022 ceph-mon[40250]: pgmap v8500: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:54 smithi122 ceph-mon[32255]: pgmap v8501: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:54 smithi022 ceph-mon[31360]: pgmap v8501: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:54 smithi022 ceph-mon[40250]: pgmap v8501: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:41:55.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:41:55.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:41:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:41:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:41:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:56 smithi122 ceph-mon[32255]: pgmap v8502: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/994648093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:56 smithi022 ceph-mon[31360]: pgmap v8502: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/994648093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:56 smithi022 ceph-mon[40250]: pgmap v8502: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/994648093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:41:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:41:57.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:41:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:41:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:58 smithi122 ceph-mon[32255]: pgmap v8503: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:58 smithi022 ceph-mon[31360]: pgmap v8503: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:58 smithi022 ceph-mon[40250]: pgmap v8503: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:41:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:41:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:41:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:41:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:41:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:00] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:42:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:00 smithi122 ceph-mon[32255]: pgmap v8504: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:00 smithi022 ceph-mon[31360]: pgmap v8504: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:00 smithi022 ceph-mon[40250]: pgmap v8504: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:00.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:00.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:00.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/80231709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/80231709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/80231709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:02 smithi122 ceph-mon[32255]: pgmap v8505: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:42:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:42:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:02 smithi022 ceph-mon[31360]: pgmap v8505: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:02 smithi022 ceph-mon[40250]: pgmap v8505: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:04 smithi122 ceph-mon[32255]: pgmap v8506: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:04 smithi022 ceph-mon[31360]: pgmap v8506: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:04 smithi022 ceph-mon[40250]: pgmap v8506: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:05.821 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:06.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:06.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:06.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:06 smithi022 ceph-mon[31360]: pgmap v8507: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2307037073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:06 smithi022 ceph-mon[40250]: pgmap v8507: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2307037073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:06 smithi122 ceph-mon[32255]: pgmap v8507: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2307037073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:42:07.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:42:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:42:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:08 smithi122 ceph-mon[32255]: pgmap v8508: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:08 smithi022 ceph-mon[31360]: pgmap v8508: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:08 smithi022 ceph-mon[40250]: pgmap v8508: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:09.846 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:42:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:10 smithi122 ceph-mon[32255]: pgmap v8509: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:10 smithi022 ceph-mon[31360]: pgmap v8509: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:10 smithi022 ceph-mon[40250]: pgmap v8509: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:11.189 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:11.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:11.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:11.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:12 smithi122 ceph-mon[32255]: pgmap v8510: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/337525668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:42:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:42:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:12 smithi022 ceph-mon[31360]: pgmap v8510: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/337525668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:12 smithi022 ceph-mon[40250]: pgmap v8510: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/337525668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:14 smithi122 ceph-mon[32255]: pgmap v8511: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:14 smithi022 ceph-mon[31360]: pgmap v8511: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:14 smithi022 ceph-mon[40250]: pgmap v8511: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:16.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:16 smithi122 ceph-mon[32255]: pgmap v8512: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:16 smithi022 ceph-mon[31360]: pgmap v8512: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:16 smithi022 ceph-mon[40250]: pgmap v8512: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:16.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:16.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:16.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:17.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:42:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:42:17.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2942876269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2942876269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:42:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2942876269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:18 smithi122 ceph-mon[32255]: pgmap v8513: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:18 smithi022 ceph-mon[31360]: pgmap v8513: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:18 smithi022 ceph-mon[40250]: pgmap v8513: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:42:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:20 smithi122 ceph-mon[32255]: pgmap v8514: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:20 smithi022 ceph-mon[40250]: pgmap v8514: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:20 smithi022 ceph-mon[31360]: pgmap v8514: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:21.932 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:22.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:22.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:22.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:42:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:42:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:22 smithi122 ceph-mon[32255]: pgmap v8515: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:22.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/485445514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:22 smithi022 ceph-mon[40250]: pgmap v8515: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/485445514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:22.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:22 smithi022 ceph-mon[31360]: pgmap v8515: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:22.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/485445514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:24 smithi122 ceph-mon[32255]: pgmap v8516: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:24 smithi022 ceph-mon[31360]: pgmap v8516: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:24 smithi022 ceph-mon[40250]: pgmap v8516: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:26 smithi122 ceph-mon[32255]: pgmap v8517: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:26 smithi022 ceph-mon[31360]: pgmap v8517: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:26 smithi022 ceph-mon[40250]: pgmap v8517: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:27.303 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:27.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:27.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:27.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:27.672 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:42:27.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:42:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:42:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:28 smithi122 ceph-mon[32255]: pgmap v8518: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3077233806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:28 smithi022 ceph-mon[31360]: pgmap v8518: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3077233806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:28 smithi022 ceph-mon[40250]: pgmap v8518: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3077233806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:42:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:30 smithi122 ceph-mon[32255]: pgmap v8519: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:30 smithi022 ceph-mon[31360]: pgmap v8519: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:30 smithi022 ceph-mon[40250]: pgmap v8519: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:32.673 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:32.673 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:32 smithi022 ceph-mon[31360]: pgmap v8520: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:32.674 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:32 smithi022 ceph-mon[40250]: pgmap v8520: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:42:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:42:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:32 smithi122 ceph-mon[32255]: pgmap v8520: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:33.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:33.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:33.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:33.372 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3045212794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:33.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:42:33.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:42:33.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:42:33.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:42:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3045212794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:42:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:42:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:42:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3045212794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:42:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:42:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:42:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:34 smithi122 ceph-mon[32255]: pgmap v8521: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:34 smithi022 ceph-mon[31360]: pgmap v8521: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:34 smithi022 ceph-mon[40250]: pgmap v8521: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:36 smithi122 ceph-mon[32255]: pgmap v8522: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:42:36.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:42:36.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:42:36.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:42:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[40250]: pgmap v8522: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:42:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:42:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:42:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:42:36.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[31360]: pgmap v8522: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:36.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:42:36.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:42:36.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:42:36.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:42:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:42:37.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:42:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:42:38.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:38.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:38.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:38.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:38 smithi122 ceph-mon[32255]: pgmap v8523: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:38 smithi022 ceph-mon[40250]: pgmap v8523: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:38 smithi022 ceph-mon[31360]: pgmap v8523: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1384926280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1384926280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1384926280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:42:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:40 smithi122 ceph-mon[32255]: pgmap v8524: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:40 smithi022 ceph-mon[31360]: pgmap v8524: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:40 smithi022 ceph-mon[40250]: pgmap v8524: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:42:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:42:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:42 smithi122 ceph-mon[32255]: pgmap v8525: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:42 smithi022 ceph-mon[31360]: pgmap v8525: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:42 smithi022 ceph-mon[40250]: pgmap v8525: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:43.417 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:43.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:43.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:43.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:44 smithi122 ceph-mon[32255]: pgmap v8526: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/260047173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:44 smithi022 ceph-mon[31360]: pgmap v8526: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/260047173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:44 smithi022 ceph-mon[40250]: pgmap v8526: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/260047173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:46 smithi122 ceph-mon[32255]: pgmap v8527: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:46 smithi022 ceph-mon[31360]: pgmap v8527: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:46 smithi022 ceph-mon[40250]: pgmap v8527: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:42:47.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:42:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:42:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:48 smithi122 ceph-mon[32255]: pgmap v8528: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:48 smithi022 ceph-mon[31360]: pgmap v8528: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:48 smithi022 ceph-mon[40250]: pgmap v8528: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:48.788 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:49.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:49.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1809708488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1809708488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1809708488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:50 smithi122 ceph-mon[32255]: pgmap v8529: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:50] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:42:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:50 smithi022 ceph-mon[31360]: pgmap v8529: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:50 smithi022 ceph-mon[40250]: pgmap v8529: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:52 smithi122 ceph-mon[32255]: pgmap v8530: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:42:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:42:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:52 smithi022 ceph-mon[40250]: pgmap v8530: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:52 smithi022 ceph-mon[31360]: pgmap v8530: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:54.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:54 smithi022 ceph-mon[31360]: pgmap v8531: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:54 smithi022 ceph-mon[40250]: pgmap v8531: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:54.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:54.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:54.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:54 smithi122 ceph-mon[32255]: pgmap v8531: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:42:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3409510906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3409510906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3409510906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:42:56.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:56 smithi122 ceph-mon[32255]: pgmap v8532: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:56 smithi022 ceph-mon[31360]: pgmap v8532: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:56 smithi022 ceph-mon[40250]: pgmap v8532: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:42:57.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:42:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:42:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:58 smithi122 ceph-mon[32255]: pgmap v8533: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:58 smithi022 ceph-mon[31360]: pgmap v8533: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:58 smithi022 ceph-mon[40250]: pgmap v8533: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:42:59.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:42:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:42:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:42:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:42:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:42:59.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:42:59.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:42:59.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:00] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:43:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:00 smithi122 ceph-mon[32255]: pgmap v8534: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/778589765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:00 smithi022 ceph-mon[31360]: pgmap v8534: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/778589765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:00 smithi022 ceph-mon[40250]: pgmap v8534: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/778589765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:02 smithi122 ceph-mon[32255]: pgmap v8535: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:43:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:43:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:02 smithi022 ceph-mon[31360]: pgmap v8535: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:02 smithi022 ceph-mon[40250]: pgmap v8535: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:04 smithi122 ceph-mon[32255]: pgmap v8536: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:04 smithi022 ceph-mon[31360]: pgmap v8536: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:04 smithi022 ceph-mon[40250]: pgmap v8536: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:04.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:05.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:05.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:05.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/661420500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/661420500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/661420500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:06 smithi122 ceph-mon[32255]: pgmap v8537: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:06 smithi022 ceph-mon[31360]: pgmap v8537: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:06 smithi022 ceph-mon[40250]: pgmap v8537: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:43:07.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:43:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:43:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:08 smithi122 ceph-mon[32255]: pgmap v8538: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:08.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:08 smithi022 ceph-mon[31360]: pgmap v8538: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:08.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:08 smithi022 ceph-mon[40250]: pgmap v8538: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:10.431 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:10] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:43:10.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:10.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:10.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:10.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:10 smithi022 ceph-mon[31360]: pgmap v8539: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:10.653 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:10 smithi022 ceph-mon[40250]: pgmap v8539: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:10.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:10 smithi122 ceph-mon[32255]: pgmap v8539: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:11.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/894109620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/894109620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/894109620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:12 smithi122 ceph-mon[32255]: pgmap v8540: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:43:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:43:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:12 smithi022 ceph-mon[31360]: pgmap v8540: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:12 smithi022 ceph-mon[40250]: pgmap v8540: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:14 smithi022 ceph-mon[31360]: pgmap v8541: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:14 smithi022 ceph-mon[40250]: pgmap v8541: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:14 smithi122 ceph-mon[32255]: pgmap v8541: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:15.653 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:16.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:16.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:16.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:16 smithi022 ceph-mon[31360]: pgmap v8542: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3977883893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:16 smithi022 ceph-mon[40250]: pgmap v8542: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3977883893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:16 smithi122 ceph-mon[32255]: pgmap v8542: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3977883893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:43:17.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:43:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:43:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:18 smithi022 ceph-mon[31360]: pgmap v8543: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:18 smithi022 ceph-mon[40250]: pgmap v8543: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:18 smithi122 ceph-mon[32255]: pgmap v8543: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:20] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:43:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:20 smithi022 ceph-mon[31360]: pgmap v8544: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:20 smithi022 ceph-mon[40250]: pgmap v8544: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:20 smithi122 ceph-mon[32255]: pgmap v8544: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:21.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:21.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:21.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:21.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/830971569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/830971569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/830971569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:43:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:43:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:22 smithi122 ceph-mon[32255]: pgmap v8545: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:22 smithi022 ceph-mon[31360]: pgmap v8545: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:22 smithi022 ceph-mon[40250]: pgmap v8545: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:24 smithi022 ceph-mon[31360]: pgmap v8546: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:24 smithi022 ceph-mon[40250]: pgmap v8546: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:24 smithi122 ceph-mon[32255]: pgmap v8546: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:26.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:26.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:26.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:26.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:26.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:26 smithi022 ceph-mon[31360]: pgmap v8547: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:26.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:26 smithi022 ceph-mon[40250]: pgmap v8547: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:26 smithi122 ceph-mon[32255]: pgmap v8547: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1586553129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:43:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1586553129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1586553129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:27.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:43:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:43:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:28 smithi022 ceph-mon[31360]: pgmap v8548: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:28 smithi022 ceph-mon[40250]: pgmap v8548: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:28 smithi122 ceph-mon[32255]: pgmap v8548: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:30 smithi122 ceph-mon[32255]: pgmap v8549: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:30] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:43:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:30 smithi022 ceph-mon[31360]: pgmap v8549: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:30 smithi022 ceph-mon[40250]: pgmap v8549: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:31.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:32.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:32.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:32.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:32 smithi022 ceph-mon[31360]: pgmap v8550: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1387481393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:32 smithi022 ceph-mon[40250]: pgmap v8550: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1387481393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:43:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:43:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:32 smithi122 ceph-mon[32255]: pgmap v8550: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1387481393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:43:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:43:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:43:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:43:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:43:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:43:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:43:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:43:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:43:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:43:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:43:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:43:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:34 smithi122 ceph-mon[32255]: pgmap v8551: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:34 smithi022 ceph-mon[31360]: pgmap v8551: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:34 smithi022 ceph-mon[40250]: pgmap v8551: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:36 smithi122 ceph-mon[32255]: pgmap v8552: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:36 smithi022 ceph-mon[31360]: pgmap v8552: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:36 smithi022 ceph-mon[40250]: pgmap v8552: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:37.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:37.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:43:37.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:43:37.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:43:37.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:43:37.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:43:37.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:43:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:37.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:37.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:37.599 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:43:37.599 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:43:37.599 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:43:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:43:37.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:43:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:43:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:38 smithi122 ceph-mon[32255]: pgmap v8553: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1500760126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:38 smithi022 ceph-mon[31360]: pgmap v8553: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1500760126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:38.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:38 smithi022 ceph-mon[40250]: pgmap v8553: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:38.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1500760126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:43:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:40 smithi122 ceph-mon[32255]: pgmap v8554: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:40 smithi022 ceph-mon[31360]: pgmap v8554: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:40 smithi022 ceph-mon[40250]: pgmap v8554: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3178M 2021-12-15T22:43:41.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:41.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3178M 2021-12-15T22:43:41.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:41.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:41.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:43:41.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3178M 2021-12-15T22:43:41.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:41.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:42.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:42 smithi122 ceph-mon[32255]: pgmap v8555: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:43:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:43:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:42 smithi022 ceph-mon[31360]: pgmap v8555: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:42 smithi022 ceph-mon[40250]: pgmap v8555: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:43:42.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:42.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:42.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/367820428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/367820428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/367820428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:44 smithi122 ceph-mon[32255]: pgmap v8556: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:44 smithi022 ceph-mon[31360]: pgmap v8556: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:44 smithi022 ceph-mon[40250]: pgmap v8556: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:46 smithi122 ceph-mon[32255]: pgmap v8557: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:46 smithi022 ceph-mon[31360]: pgmap v8557: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:46 smithi022 ceph-mon[40250]: pgmap v8557: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:43:47.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:47.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:43:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:43:48.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:48.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:48.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:48 smithi022 ceph-mon[31360]: pgmap v8558: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2799241764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:48 smithi022 ceph-mon[40250]: pgmap v8558: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2799241764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:48 smithi122 ceph-mon[32255]: pgmap v8558: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2799241764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:50] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:43:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:50 smithi122 ceph-mon[32255]: pgmap v8559: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:50 smithi022 ceph-mon[31360]: pgmap v8559: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:50 smithi022 ceph-mon[40250]: pgmap v8559: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:43:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:43:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:52 smithi122 ceph-mon[32255]: pgmap v8560: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:52 smithi022 ceph-mon[31360]: pgmap v8560: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:52 smithi022 ceph-mon[40250]: pgmap v8560: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:53.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:53.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:53.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:53.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:54 smithi122 ceph-mon[32255]: pgmap v8561: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3160475391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:54 smithi022 ceph-mon[40250]: pgmap v8561: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3160475391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:54.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:54 smithi022 ceph-mon[31360]: pgmap v8561: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:43:54.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3160475391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:56 smithi122 ceph-mon[32255]: pgmap v8562: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:56 smithi022 ceph-mon[31360]: pgmap v8562: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:56 smithi022 ceph-mon[40250]: pgmap v8562: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:43:57.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:43:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:43:58.628 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:43:58.629 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:58 smithi022 ceph-mon[40250]: pgmap v8563: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:58.629 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:58 smithi022 ceph-mon[31360]: pgmap v8563: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:58 smithi122 ceph-mon[32255]: pgmap v8563: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:43:58.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:43:58.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:43:58.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:43:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:43:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3994969535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:43:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3994969535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:43:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:43:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:43:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3994969535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:00] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:44:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:00 smithi122 ceph-mon[32255]: pgmap v8564: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:00 smithi022 ceph-mon[31360]: pgmap v8564: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:00 smithi022 ceph-mon[40250]: pgmap v8564: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:02 smithi122 ceph-mon[32255]: pgmap v8565: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:44:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:44:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:02 smithi022 ceph-mon[31360]: pgmap v8565: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:02 smithi022 ceph-mon[40250]: pgmap v8565: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:03.999 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:04.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:04.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:04.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:04 smithi122 ceph-mon[32255]: pgmap v8566: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:04.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3869672864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:04 smithi022 ceph-mon[31360]: pgmap v8566: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3869672864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:04 smithi022 ceph-mon[40250]: pgmap v8566: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3869672864' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:06 smithi122 ceph-mon[32255]: pgmap v8567: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:06 smithi022 ceph-mon[31360]: pgmap v8567: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:06 smithi022 ceph-mon[40250]: pgmap v8567: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:44:07.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:44:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:44:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:08 smithi122 ceph-mon[32255]: pgmap v8568: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:08 smithi022 ceph-mon[31360]: pgmap v8568: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:08 smithi022 ceph-mon[40250]: pgmap v8568: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:09.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:09.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:09.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:09.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:09.744 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:09.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:10] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:44:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:10 smithi122 ceph-mon[32255]: pgmap v8569: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3353650835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:10 smithi022 ceph-mon[31360]: pgmap v8569: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3353650835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:10 smithi022 ceph-mon[40250]: pgmap v8569: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3353650835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:12 smithi122 ceph-mon[32255]: pgmap v8570: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:44:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:44:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:12 smithi022 ceph-mon[31360]: pgmap v8570: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:12 smithi022 ceph-mon[40250]: pgmap v8570: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:14 smithi122 ceph-mon[32255]: pgmap v8571: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:14.744 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:14.745 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:14 smithi022 ceph-mon[31360]: pgmap v8571: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:14.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:14 smithi022 ceph-mon[40250]: pgmap v8571: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:15.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:15.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:15.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:15.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2815183109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2815183109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2815183109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:16 smithi122 ceph-mon[32255]: pgmap v8572: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:16 smithi022 ceph-mon[31360]: pgmap v8572: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:16 smithi022 ceph-mon[40250]: pgmap v8572: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:44:17.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:44:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:44:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:18 smithi122 ceph-mon[32255]: pgmap v8573: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:18 smithi022 ceph-mon[31360]: pgmap v8573: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:18 smithi022 ceph-mon[40250]: pgmap v8573: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:20.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:20 smithi122 ceph-mon[32255]: pgmap v8574: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:20] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T22:44:20.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:20.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:20.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:20 smithi022 ceph-mon[31360]: pgmap v8574: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:20 smithi022 ceph-mon[40250]: pgmap v8574: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4030803411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:21.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4030803411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:21.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4030803411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:22 smithi122 ceph-mon[32255]: pgmap v8575: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:44:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:44:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:22 smithi022 ceph-mon[31360]: pgmap v8575: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:22 smithi022 ceph-mon[40250]: pgmap v8575: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:24 smithi122 ceph-mon[32255]: pgmap v8576: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:24 smithi022 ceph-mon[31360]: pgmap v8576: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:24 smithi022 ceph-mon[40250]: pgmap v8576: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:25.485 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:26 smithi122 ceph-mon[32255]: pgmap v8577: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2087890973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:26 smithi022 ceph-mon[31360]: pgmap v8577: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2087890973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:26.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:26 smithi022 ceph-mon[40250]: pgmap v8577: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:26.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2087890973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:44:27.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:44:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:44:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:28 smithi122 ceph-mon[32255]: pgmap v8578: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:28 smithi022 ceph-mon[31360]: pgmap v8578: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:28 smithi022 ceph-mon[40250]: pgmap v8578: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:30 smithi122 ceph-mon[32255]: pgmap v8579: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:30] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T22:44:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:30 smithi022 ceph-mon[31360]: pgmap v8579: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:30 smithi022 ceph-mon[40250]: pgmap v8579: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:30.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2096658021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2096658021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2096658021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:32 smithi122 ceph-mon[32255]: pgmap v8580: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:44:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:44:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:32 smithi022 ceph-mon[31360]: pgmap v8580: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:32 smithi022 ceph-mon[40250]: pgmap v8580: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:44:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:44:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:44:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:44:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:44:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:44:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:44:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:44:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:44:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:44:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:44:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:44:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:34 smithi122 ceph-mon[32255]: pgmap v8581: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:34 smithi022 ceph-mon[31360]: pgmap v8581: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:34 smithi022 ceph-mon[40250]: pgmap v8581: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:36.219 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:36 smithi022 ceph-mon[31360]: pgmap v8582: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:36 smithi022 ceph-mon[40250]: pgmap v8582: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:36.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:36.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:36.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:36 smithi122 ceph-mon[32255]: pgmap v8582: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2846388311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:44:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2846388311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:37.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:44:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:44:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2846388311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:38 smithi022 ceph-mon[31360]: pgmap v8583: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:38 smithi022 ceph-mon[40250]: pgmap v8583: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:38 smithi122 ceph-mon[32255]: pgmap v8583: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:44:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:40 smithi022 ceph-mon[31360]: pgmap v8584: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:40 smithi022 ceph-mon[40250]: pgmap v8584: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:40 smithi122 ceph-mon[32255]: pgmap v8584: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:41.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:44:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:44:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:44:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:44:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:44:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:44:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:44:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:44:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:44:41.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:41.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:41.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:42 smithi122 ceph-mon[32255]: pgmap v8585: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:44:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2747079961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:44:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:44:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:42 smithi022 ceph-mon[31360]: pgmap v8585: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:44:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2747079961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:42 smithi022 ceph-mon[40250]: pgmap v8585: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:44:42.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2747079961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:44 smithi022 ceph-mon[31360]: pgmap v8586: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:44 smithi022 ceph-mon[40250]: pgmap v8586: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:44 smithi122 ceph-mon[32255]: pgmap v8586: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:46 smithi022 ceph-mon[31360]: pgmap v8587: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:46 smithi022 ceph-mon[40250]: pgmap v8587: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:46 smithi122 ceph-mon[32255]: pgmap v8587: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:46.970 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:47.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:47.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:47.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2787718069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:44:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2787718069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:47.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:44:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:44:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2787718069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:48 smithi022 ceph-mon[31360]: pgmap v8588: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:48 smithi022 ceph-mon[40250]: pgmap v8588: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:48 smithi122 ceph-mon[32255]: pgmap v8588: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:50] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:44:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:50 smithi022 ceph-mon[31360]: pgmap v8589: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:50 smithi022 ceph-mon[40250]: pgmap v8589: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:50 smithi122 ceph-mon[32255]: pgmap v8589: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:52.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:44:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:44:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:52 smithi122 ceph-mon[32255]: pgmap v8590: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:52.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:52.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:52.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:52.710 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:52 smithi022 ceph-mon[31360]: pgmap v8590: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:52.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:52 smithi022 ceph-mon[40250]: pgmap v8590: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2210459794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2210459794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2210459794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:54 smithi022 ceph-mon[31360]: pgmap v8591: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:54 smithi022 ceph-mon[40250]: pgmap v8591: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:54 smithi122 ceph-mon[32255]: pgmap v8591: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:44:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:55 smithi022 ceph-mon[31360]: pgmap v8592: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:55 smithi022 ceph-mon[40250]: pgmap v8592: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:55 smithi122 ceph-mon[32255]: pgmap v8592: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:57.711 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:44:57.711 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:44:57.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:44:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:44:58.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:44:58.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:44:58.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:44:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:58 smithi022 ceph-mon[31360]: pgmap v8593: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2408254713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:58 smithi022 ceph-mon[40250]: pgmap v8593: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2408254713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:58 smithi122 ceph-mon[32255]: pgmap v8593: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:44:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2408254713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:44:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:44:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:44:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:44:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:44:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:00] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:45:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:00 smithi122 ceph-mon[32255]: pgmap v8594: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:00 smithi022 ceph-mon[31360]: pgmap v8594: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:00 smithi022 ceph-mon[40250]: pgmap v8594: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:02 smithi122 ceph-mon[32255]: pgmap v8595: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:45:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:45:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:02 smithi022 ceph-mon[31360]: pgmap v8595: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:02 smithi022 ceph-mon[40250]: pgmap v8595: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:03.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:03.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:04 smithi122 ceph-mon[32255]: pgmap v8596: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1942144398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:04 smithi022 ceph-mon[31360]: pgmap v8596: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1942144398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:04 smithi022 ceph-mon[40250]: pgmap v8596: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1942144398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:06 smithi122 ceph-mon[32255]: pgmap v8597: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:06 smithi022 ceph-mon[31360]: pgmap v8597: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:06 smithi022 ceph-mon[40250]: pgmap v8597: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:45:07.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:45:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:45:08.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:08 smithi122 ceph-mon[32255]: pgmap v8598: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:08 smithi022 ceph-mon[31360]: pgmap v8598: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:08 smithi022 ceph-mon[40250]: pgmap v8598: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:08.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:08.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:08.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:09.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1523779254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:09.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1523779254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1523779254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:10 smithi122 ceph-mon[32255]: pgmap v8599: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:45:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:10 smithi022 ceph-mon[31360]: pgmap v8599: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:10 smithi022 ceph-mon[40250]: pgmap v8599: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:12 smithi122 ceph-mon[32255]: pgmap v8600: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:45:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:45:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:12 smithi022 ceph-mon[31360]: pgmap v8600: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:12 smithi022 ceph-mon[40250]: pgmap v8600: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:13.823 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:14.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:14.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:14.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:14 smithi022 ceph-mon[31360]: pgmap v8601: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3544134476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:14 smithi022 ceph-mon[40250]: pgmap v8601: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3544134476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:14 smithi122 ceph-mon[32255]: pgmap v8601: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3544134476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:16 smithi122 ceph-mon[32255]: pgmap v8602: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:16 smithi022 ceph-mon[31360]: pgmap v8602: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:16 smithi022 ceph-mon[40250]: pgmap v8602: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:45:17.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:45:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:45:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:18 smithi122 ceph-mon[32255]: pgmap v8603: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:18 smithi022 ceph-mon[31360]: pgmap v8603: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:18 smithi022 ceph-mon[40250]: pgmap v8603: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:19.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:19.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:19.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:19.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:19.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:20 smithi122 ceph-mon[32255]: pgmap v8604: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2057472980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:20] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:45:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:20 smithi022 ceph-mon[31360]: pgmap v8604: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2057472980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:20 smithi022 ceph-mon[40250]: pgmap v8604: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2057472980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:22 smithi122 ceph-mon[32255]: pgmap v8605: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:45:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:45:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:22 smithi022 ceph-mon[31360]: pgmap v8605: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:22 smithi022 ceph-mon[40250]: pgmap v8605: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:24.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:24 smithi122 ceph-mon[32255]: pgmap v8606: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:24 smithi022 ceph-mon[31360]: pgmap v8606: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:24 smithi022 ceph-mon[40250]: pgmap v8606: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:24.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:24.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:24.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:25.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1040944555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:25.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1040944555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1040944555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:26 smithi122 ceph-mon[32255]: pgmap v8607: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:26 smithi022 ceph-mon[31360]: pgmap v8607: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:26 smithi022 ceph-mon[40250]: pgmap v8607: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:45:28.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:45:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:45:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:28 smithi122 ceph-mon[32255]: pgmap v8608: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:28 smithi022 ceph-mon[31360]: pgmap v8608: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:28 smithi022 ceph-mon[40250]: pgmap v8608: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:29.936 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:30.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:30.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:30.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:30] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:45:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:30 smithi122 ceph-mon[32255]: pgmap v8609: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/816187386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:30 smithi022 ceph-mon[31360]: pgmap v8609: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/816187386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:30 smithi022 ceph-mon[40250]: pgmap v8609: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/816187386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:32 smithi122 ceph-mon[32255]: pgmap v8610: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:45:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:45:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:32 smithi022 ceph-mon[31360]: pgmap v8610: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:32 smithi022 ceph-mon[40250]: pgmap v8610: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:45:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:45:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:45:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:45:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:45:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:45:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:45:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:34 smithi122 ceph-mon[32255]: pgmap v8611: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:34 smithi022 ceph-mon[31360]: pgmap v8611: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:34 smithi022 ceph-mon[40250]: pgmap v8611: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:35.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:35.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:35.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:36 smithi122 ceph-mon[32255]: pgmap v8612: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2145449813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:36 smithi022 ceph-mon[31360]: pgmap v8612: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2145449813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:36 smithi022 ceph-mon[40250]: pgmap v8612: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2145449813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:45:38.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:45:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:45:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:38 smithi122 ceph-mon[32255]: pgmap v8613: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:38 smithi022 ceph-mon[31360]: pgmap v8613: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:38 smithi022 ceph-mon[40250]: pgmap v8613: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:40 smithi122 ceph-mon[32255]: pgmap v8614: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:40] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:45:40.680 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:40.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:40 smithi022 ceph-mon[31360]: pgmap v8614: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:40.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:40 smithi022 ceph-mon[40250]: pgmap v8614: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:41.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:41.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:41.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396934342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396934342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396934342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:45:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:45:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:42 smithi122 ceph-mon[32255]: pgmap v8615: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:45:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:45:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:45:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:45:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[40250]: pgmap v8615: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:45:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:45:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:45:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:45:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[31360]: pgmap v8615: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:45:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:45:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:45:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:45:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:44 smithi122 ceph-mon[32255]: pgmap v8616: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:44 smithi022 ceph-mon[31360]: pgmap v8616: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:44 smithi022 ceph-mon[40250]: pgmap v8616: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:46.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:46.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:46.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:46.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:46 smithi122 ceph-mon[32255]: pgmap v8617: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:46 smithi022 ceph-mon[31360]: pgmap v8617: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:46 smithi022 ceph-mon[40250]: pgmap v8617: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:47.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/145392404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/145392404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:45:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/145392404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:48.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:45:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:45:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:48 smithi122 ceph-mon[32255]: pgmap v8618: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:48 smithi022 ceph-mon[31360]: pgmap v8618: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:48 smithi022 ceph-mon[40250]: pgmap v8618: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:50 smithi122 ceph-mon[32255]: pgmap v8619: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:50] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:45:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:50 smithi022 ceph-mon[31360]: pgmap v8619: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:50 smithi022 ceph-mon[40250]: pgmap v8619: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:51.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:51.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:51.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:51.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:45:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:45:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:52 smithi122 ceph-mon[32255]: pgmap v8620: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/897664958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:52 smithi022 ceph-mon[31360]: pgmap v8620: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/897664958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:52 smithi022 ceph-mon[40250]: pgmap v8620: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/897664958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:54 smithi122 ceph-mon[32255]: pgmap v8621: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:54 smithi022 ceph-mon[31360]: pgmap v8621: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:54 smithi022 ceph-mon[40250]: pgmap v8621: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:45:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:56 smithi122 ceph-mon[32255]: pgmap v8622: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:56 smithi022 ceph-mon[31360]: pgmap v8622: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:56 smithi022 ceph-mon[40250]: pgmap v8622: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:56.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:45:57.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:45:57.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:45:57.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:45:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/273385341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:45:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/273385341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/273385341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:45:58.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:45:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:45:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:58 smithi122 ceph-mon[32255]: pgmap v8623: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:58 smithi022 ceph-mon[31360]: pgmap v8623: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:58 smithi022 ceph-mon[40250]: pgmap v8623: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:45:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:45:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:45:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:45:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:45:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:00] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:46:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:00 smithi122 ceph-mon[32255]: pgmap v8624: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:00 smithi022 ceph-mon[31360]: pgmap v8624: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:00 smithi022 ceph-mon[40250]: pgmap v8624: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:02.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:02.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:02.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:02.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:02 smithi122 ceph-mon[32255]: pgmap v8625: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:46:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:46:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:02 smithi022 ceph-mon[31360]: pgmap v8625: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:02 smithi022 ceph-mon[40250]: pgmap v8625: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3571992402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3571992402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3571992402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:04 smithi122 ceph-mon[32255]: pgmap v8626: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:04 smithi022 ceph-mon[31360]: pgmap v8626: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:04 smithi022 ceph-mon[40250]: pgmap v8626: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:06 smithi122 ceph-mon[32255]: pgmap v8627: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:06 smithi022 ceph-mon[31360]: pgmap v8627: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:06 smithi022 ceph-mon[40250]: pgmap v8627: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:07.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:46:07.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:07.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:07.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:08.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:46:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:46:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:08 smithi122 ceph-mon[32255]: pgmap v8628: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3735452884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:08 smithi022 ceph-mon[31360]: pgmap v8628: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3735452884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:08 smithi022 ceph-mon[40250]: pgmap v8628: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3735452884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:10.427 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:46:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:10 smithi122 ceph-mon[32255]: pgmap v8629: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:10 smithi022 ceph-mon[31360]: pgmap v8629: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:10 smithi022 ceph-mon[40250]: pgmap v8629: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:12 smithi122 ceph-mon[32255]: pgmap v8630: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:46:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:46:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:12 smithi022 ceph-mon[31360]: pgmap v8630: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:12 smithi022 ceph-mon[40250]: pgmap v8630: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:12.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:13.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:13.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:13.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1019113472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1019113472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1019113472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:14 smithi122 ceph-mon[32255]: pgmap v8631: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:14 smithi022 ceph-mon[31360]: pgmap v8631: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:14 smithi022 ceph-mon[40250]: pgmap v8631: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:16 smithi022 ceph-mon[31360]: pgmap v8632: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:16 smithi022 ceph-mon[40250]: pgmap v8632: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:16 smithi122 ceph-mon[32255]: pgmap v8632: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:46:18.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:46:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:46:18.268 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:18 smithi022 ceph-mon[31360]: pgmap v8633: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:18 smithi022 ceph-mon[40250]: pgmap v8633: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:18.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:18.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:18.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:18 smithi122 ceph-mon[32255]: pgmap v8633: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/60636459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/60636459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/60636459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:20] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:46:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:20 smithi022 ceph-mon[31360]: pgmap v8634: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:20 smithi022 ceph-mon[40250]: pgmap v8634: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:20 smithi122 ceph-mon[32255]: pgmap v8634: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:22 smithi122 ceph-mon[32255]: pgmap v8635: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:46:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:46:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:22 smithi022 ceph-mon[31360]: pgmap v8635: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:22 smithi022 ceph-mon[40250]: pgmap v8635: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:23.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:24.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:24.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:24.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:24 smithi022 ceph-mon[31360]: pgmap v8636: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2399599191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:24.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:24 smithi022 ceph-mon[40250]: pgmap v8636: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:24.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2399599191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:24 smithi122 ceph-mon[32255]: pgmap v8636: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2399599191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:26 smithi022 ceph-mon[31360]: pgmap v8637: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:26 smithi022 ceph-mon[40250]: pgmap v8637: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:26 smithi122 ceph-mon[32255]: pgmap v8637: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:46:28.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:46:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:46:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:28 smithi022 ceph-mon[40250]: pgmap v8638: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:28 smithi022 ceph-mon[31360]: pgmap v8638: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:28 smithi122 ceph-mon[32255]: pgmap v8638: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:29.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:29.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:29.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:29.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2161532014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:29.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2161532014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2161532014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:30] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:46:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:30 smithi022 ceph-mon[31360]: pgmap v8639: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:30 smithi022 ceph-mon[40250]: pgmap v8639: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:30 smithi122 ceph-mon[32255]: pgmap v8639: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:32 smithi122 ceph-mon[32255]: pgmap v8640: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:46:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:46:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:32 smithi022 ceph-mon[31360]: pgmap v8640: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:32 smithi022 ceph-mon[40250]: pgmap v8640: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:46:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:46:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:46:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:46:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:46:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:46:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:46:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:46:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:46:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:46:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:46:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:46:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:34.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:34.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:34.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:34.749 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:34 smithi022 ceph-mon[31360]: pgmap v8641: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:34.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:34 smithi022 ceph-mon[40250]: pgmap v8641: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:34 smithi122 ceph-mon[32255]: pgmap v8641: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1949996082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1949996082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1949996082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:36 smithi022 ceph-mon[31360]: pgmap v8642: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:36 smithi022 ceph-mon[40250]: pgmap v8642: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:36 smithi122 ceph-mon[32255]: pgmap v8642: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:46:38.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:46:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:46:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:38 smithi022 ceph-mon[31360]: pgmap v8643: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:38 smithi022 ceph-mon[40250]: pgmap v8643: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:38 smithi122 ceph-mon[32255]: pgmap v8643: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:39.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:39.751 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:39.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:40.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:40.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:40.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:46:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:40 smithi022 ceph-mon[31360]: pgmap v8644: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3190844218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:40 smithi022 ceph-mon[40250]: pgmap v8644: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3190844218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:40 smithi122 ceph-mon[32255]: pgmap v8644: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3190844218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:46:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:46:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:42 smithi122 ceph-mon[32255]: pgmap v8645: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:46:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:46:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:46:42.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[31360]: pgmap v8645: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:42.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:46:42.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:46:42.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:46:42.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[40250]: pgmap v8645: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:42.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:46:42.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:46:42.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:46:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:46:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:43 smithi022 ceph-mon[31360]: pgmap v8646: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:46:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:43 smithi022 ceph-mon[40250]: pgmap v8646: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:46:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:43 smithi122 ceph-mon[32255]: pgmap v8646: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:45.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:45.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:45.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:45.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:46 smithi122 ceph-mon[32255]: pgmap v8647: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2957563603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:46 smithi022 ceph-mon[40250]: pgmap v8647: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2957563603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:46 smithi022 ceph-mon[31360]: pgmap v8647: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2957563603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:46:48.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:46:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:46:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:48 smithi122 ceph-mon[32255]: pgmap v8648: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:48 smithi022 ceph-mon[40250]: pgmap v8648: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:48 smithi022 ceph-mon[31360]: pgmap v8648: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:46:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:50 smithi122 ceph-mon[32255]: pgmap v8649: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:50.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:50 smithi022 ceph-mon[31360]: pgmap v8649: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:50 smithi022 ceph-mon[40250]: pgmap v8649: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:50.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4218430318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4218430318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4218430318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:52 smithi122 ceph-mon[32255]: pgmap v8650: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:46:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:46:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:52 smithi022 ceph-mon[31360]: pgmap v8650: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:52 smithi022 ceph-mon[40250]: pgmap v8650: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:54 smithi122 ceph-mon[32255]: pgmap v8651: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:54 smithi022 ceph-mon[31360]: pgmap v8651: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:54 smithi022 ceph-mon[40250]: pgmap v8651: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:46:55.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:46:56.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:46:56.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:46:56.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:46:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:56 smithi022 ceph-mon[31360]: pgmap v8652: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4045677912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:56 smithi022 ceph-mon[40250]: pgmap v8652: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4045677912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:56 smithi122 ceph-mon[32255]: pgmap v8652: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4045677912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:46:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:46:58.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:46:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:46:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:58 smithi122 ceph-mon[32255]: pgmap v8653: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:58 smithi022 ceph-mon[31360]: pgmap v8653: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:58 smithi022 ceph-mon[40250]: pgmap v8653: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:46:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:46:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:46:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:46:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:46:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:47:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:00 smithi122 ceph-mon[32255]: pgmap v8654: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:00 smithi022 ceph-mon[31360]: pgmap v8654: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:00 smithi022 ceph-mon[40250]: pgmap v8654: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:01.226 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:01.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:01.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:01.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:02 smithi122 ceph-mon[32255]: pgmap v8655: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2047753895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:47:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:47:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:02 smithi022 ceph-mon[31360]: pgmap v8655: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2047753895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:02 smithi022 ceph-mon[40250]: pgmap v8655: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2047753895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:04 smithi122 ceph-mon[32255]: pgmap v8656: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:04 smithi022 ceph-mon[31360]: pgmap v8656: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:04 smithi022 ceph-mon[40250]: pgmap v8656: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:06.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:06 smithi122 ceph-mon[32255]: pgmap v8657: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:06 smithi022 ceph-mon[31360]: pgmap v8657: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:06 smithi022 ceph-mon[40250]: pgmap v8657: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:06.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:06.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:06.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:07.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3048017947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3048017947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:47:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3048017947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:08.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:47:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:47:08.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:08 smithi122 ceph-mon[32255]: pgmap v8658: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:08 smithi022 ceph-mon[31360]: pgmap v8658: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:08 smithi022 ceph-mon[40250]: pgmap v8658: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:10] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:47:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:10 smithi122 ceph-mon[32255]: pgmap v8659: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:10 smithi022 ceph-mon[31360]: pgmap v8659: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:10 smithi022 ceph-mon[40250]: pgmap v8659: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:11.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:12.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:12.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:12.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:12 smithi122 ceph-mon[32255]: pgmap v8660: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1596362126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:47:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:47:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:12 smithi022 ceph-mon[31360]: pgmap v8660: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1596362126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:12 smithi022 ceph-mon[40250]: pgmap v8660: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1596362126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:14 smithi122 ceph-mon[32255]: pgmap v8661: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:14 smithi022 ceph-mon[31360]: pgmap v8661: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:14 smithi022 ceph-mon[40250]: pgmap v8661: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:16 smithi122 ceph-mon[32255]: pgmap v8662: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:16 smithi022 ceph-mon[31360]: pgmap v8662: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:16 smithi022 ceph-mon[40250]: pgmap v8662: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:17.336 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:17.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:17.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:17.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:17.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:47:18.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:47:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:47:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:18 smithi122 ceph-mon[32255]: pgmap v8663: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/531217969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:18 smithi022 ceph-mon[31360]: pgmap v8663: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/531217969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:18 smithi022 ceph-mon[40250]: pgmap v8663: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/531217969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:20] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:47:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:20 smithi122 ceph-mon[32255]: pgmap v8664: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:20.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:20 smithi022 ceph-mon[31360]: pgmap v8664: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:20.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:20 smithi022 ceph-mon[40250]: pgmap v8664: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:22 smithi122 ceph-mon[32255]: pgmap v8665: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:47:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:47:22.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:22.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:22 smithi022 ceph-mon[40250]: pgmap v8665: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:22.708 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:22 smithi022 ceph-mon[31360]: pgmap v8665: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:23.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:23.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:23.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:23.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/291512400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/291512400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/291512400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:24 smithi122 ceph-mon[32255]: pgmap v8666: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:24 smithi022 ceph-mon[31360]: pgmap v8666: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:24 smithi022 ceph-mon[40250]: pgmap v8666: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:26.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:26 smithi122 ceph-mon[32255]: pgmap v8667: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:26 smithi022 ceph-mon[31360]: pgmap v8667: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:26 smithi022 ceph-mon[40250]: pgmap v8667: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:47:28.074 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:28.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:47:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:47:28.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:28.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:28.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:28 smithi122 ceph-mon[32255]: pgmap v8668: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:28 smithi022 ceph-mon[31360]: pgmap v8668: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:28 smithi022 ceph-mon[40250]: pgmap v8668: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:47:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:29.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3644851630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3644851630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3644851630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:30 smithi122 ceph-mon[32255]: pgmap v8669: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:30] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:47:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:30 smithi022 ceph-mon[31360]: pgmap v8669: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:30 smithi022 ceph-mon[40250]: pgmap v8669: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:32 smithi122 ceph-mon[32255]: pgmap v8670: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:47:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:47:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:32 smithi022 ceph-mon[31360]: pgmap v8670: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:32 smithi022 ceph-mon[40250]: pgmap v8670: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:33.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:47:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:47:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:47:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:47:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:47:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:47:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:47:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:47:33.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:33.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:33.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:34 smithi122 ceph-mon[32255]: pgmap v8671: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4078050024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:34 smithi022 ceph-mon[40250]: pgmap v8671: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4078050024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:34 smithi022 ceph-mon[31360]: pgmap v8671: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4078050024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:36 smithi122 ceph-mon[32255]: pgmap v8672: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:36 smithi022 ceph-mon[31360]: pgmap v8672: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:36 smithi022 ceph-mon[40250]: pgmap v8672: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:47:38.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:47:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:38 smithi122 ceph-mon[32255]: pgmap v8673: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:38 smithi022 ceph-mon[31360]: pgmap v8673: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:38.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:38 smithi022 ceph-mon[40250]: pgmap v8673: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:38.818 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:39.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:39.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:39.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:39.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3662059787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:39.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3662059787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:39.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3662059787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:40] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:47:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:40 smithi122 ceph-mon[32255]: pgmap v8674: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:40 smithi022 ceph-mon[31360]: pgmap v8674: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:40 smithi022 ceph-mon[40250]: pgmap v8674: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:42 smithi122 ceph-mon[32255]: pgmap v8675: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:47:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:47:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:42 smithi022 ceph-mon[31360]: pgmap v8675: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:42 smithi022 ceph-mon[40250]: pgmap v8675: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:47:43.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:47:43.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:47:43.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:47:43.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:47:43.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:47:43.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:47:43.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:47:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:47:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:47:43.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:47:43.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:47:44.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:44 smithi022 ceph-mon[31360]: pgmap v8676: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:44 smithi022 ceph-mon[40250]: pgmap v8676: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:44.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:44.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:44.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:44 smithi122 ceph-mon[32255]: pgmap v8676: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3175711553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3175711553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3175711553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:46 smithi122 ceph-mon[32255]: pgmap v8677: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:46 smithi022 ceph-mon[31360]: pgmap v8677: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:46 smithi022 ceph-mon[40250]: pgmap v8677: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:47:48.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:47:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:47:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:48 smithi122 ceph-mon[32255]: pgmap v8678: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:48 smithi022 ceph-mon[31360]: pgmap v8678: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:48 smithi022 ceph-mon[40250]: pgmap v8678: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:49.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:49.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:49.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:49.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:50] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:47:50.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:50 smithi122 ceph-mon[32255]: pgmap v8679: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:50.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1484961476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:50 smithi022 ceph-mon[31360]: pgmap v8679: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1484961476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:50 smithi022 ceph-mon[40250]: pgmap v8679: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:47:50.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1484961476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:52 smithi122 ceph-mon[32255]: pgmap v8680: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:47:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:47:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:52 smithi022 ceph-mon[31360]: pgmap v8680: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:52 smithi022 ceph-mon[40250]: pgmap v8680: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:47:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:54 smithi022 ceph-mon[31360]: pgmap v8681: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:54 smithi022 ceph-mon[40250]: pgmap v8681: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:54 smithi122 ceph-mon[32255]: pgmap v8681: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:54.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:47:55.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:47:55.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:47:55.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:47:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4009246341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4009246341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4009246341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:47:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:56 smithi022 ceph-mon[31360]: pgmap v8682: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:56 smithi022 ceph-mon[40250]: pgmap v8682: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:56 smithi122 ceph-mon[32255]: pgmap v8682: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:47:58.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:47:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:47:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:58 smithi022 ceph-mon[31360]: pgmap v8683: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:58 smithi022 ceph-mon[40250]: pgmap v8683: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:58 smithi122 ceph-mon[32255]: pgmap v8683: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:47:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:47:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:47:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:47:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:47:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:00.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:00] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T22:48:00.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:00.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:00.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:00.678 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:00 smithi022 ceph-mon[31360]: pgmap v8684: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:48:00.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:00 smithi022 ceph-mon[40250]: pgmap v8684: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:48:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:00 smithi122 ceph-mon[32255]: pgmap v8684: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:48:01.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2957657367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2957657367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2957657367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:48:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:48:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:02 smithi122 ceph-mon[32255]: pgmap v8685: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:48:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:02 smithi022 ceph-mon[31360]: pgmap v8685: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:48:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:02 smithi022 ceph-mon[40250]: pgmap v8685: 132 pgs: 132 active+clean; 582 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:48:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:04 smithi022 ceph-mon[31360]: pgmap v8686: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:48:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:04 smithi022 ceph-mon[40250]: pgmap v8686: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:48:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:04 smithi122 ceph-mon[32255]: pgmap v8686: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:48:05.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:06.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:06.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:06.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:06 smithi022 ceph-mon[31360]: pgmap v8687: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:48:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2244678465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:06 smithi022 ceph-mon[40250]: pgmap v8687: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:48:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2244678465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:06 smithi122 ceph-mon[32255]: pgmap v8687: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:48:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2244678465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:48:08.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:48:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:48:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:08 smithi022 ceph-mon[31360]: pgmap v8688: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:48:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:08 smithi022 ceph-mon[40250]: pgmap v8688: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:48:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:08 smithi122 ceph-mon[32255]: pgmap v8688: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:48:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:48:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:10 smithi022 ceph-mon[31360]: pgmap v8689: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:48:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:10 smithi022 ceph-mon[40250]: pgmap v8689: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:48:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:10 smithi122 ceph-mon[32255]: pgmap v8689: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:48:11.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1401945429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1401945429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1401945429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:48:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:48:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:12 smithi122 ceph-mon[32255]: pgmap v8690: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:48:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:12 smithi022 ceph-mon[31360]: pgmap v8690: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:48:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:12 smithi022 ceph-mon[40250]: pgmap v8690: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:48:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:14 smithi022 ceph-mon[31360]: pgmap v8691: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:48:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:14 smithi022 ceph-mon[40250]: pgmap v8691: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:48:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:14 smithi122 ceph-mon[32255]: pgmap v8691: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:48:16.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:16 smithi022 ceph-mon[31360]: pgmap v8692: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:16 smithi022 ceph-mon[40250]: pgmap v8692: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:16.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:16.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:16.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:16 smithi122 ceph-mon[32255]: pgmap v8692: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/970917181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:48:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/970917181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:17.826 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/970917181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:18.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:48:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:48:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:18 smithi022 ceph-mon[31360]: pgmap v8693: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:18 smithi022 ceph-mon[40250]: pgmap v8693: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:18 smithi122 ceph-mon[32255]: pgmap v8693: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:20] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:48:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:20 smithi022 ceph-mon[31360]: pgmap v8694: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:20 smithi022 ceph-mon[40250]: pgmap v8694: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:20 smithi122 ceph-mon[32255]: pgmap v8694: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:21.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:22.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:22.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:22.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:22 smithi122 ceph-mon[32255]: pgmap v8695: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1115272508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:22.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:48:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:48:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:22 smithi022 ceph-mon[31360]: pgmap v8695: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1115272508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:22 smithi022 ceph-mon[40250]: pgmap v8695: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:22.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1115272508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:24 smithi022 ceph-mon[31360]: pgmap v8696: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:24 smithi022 ceph-mon[40250]: pgmap v8696: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:24 smithi122 ceph-mon[32255]: pgmap v8696: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:26 smithi022 ceph-mon[31360]: pgmap v8697: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:26 smithi022 ceph-mon[40250]: pgmap v8697: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:26 smithi122 ceph-mon[32255]: pgmap v8697: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:27.163 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:48:27.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:27.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:27.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:28.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:48:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:48:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:28 smithi022 ceph-mon[31360]: pgmap v8698: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2400172664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:28 smithi022 ceph-mon[40250]: pgmap v8698: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2400172664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:28 smithi122 ceph-mon[32255]: pgmap v8698: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2400172664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:30] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:48:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:30 smithi022 ceph-mon[31360]: pgmap v8699: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:30 smithi022 ceph-mon[40250]: pgmap v8699: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:30 smithi122 ceph-mon[32255]: pgmap v8699: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:32.532 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:32 smithi122 ceph-mon[32255]: pgmap v8700: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:48:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:48:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:32 smithi022 ceph-mon[31360]: pgmap v8700: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:32 smithi022 ceph-mon[40250]: pgmap v8700: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:32.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:32.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:32.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2721708283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:33.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:48:33.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:48:33.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:48:33.824 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:48:33.825 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2721708283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:33.825 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:48:33.825 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:48:33.825 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:48:33.826 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:48:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2721708283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:48:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:48:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:48:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:48:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:34 smithi022 ceph-mon[31360]: pgmap v8701: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:34 smithi022 ceph-mon[40250]: pgmap v8701: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:34 smithi122 ceph-mon[32255]: pgmap v8701: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:36 smithi022 ceph-mon[31360]: pgmap v8702: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:36 smithi022 ceph-mon[40250]: pgmap v8702: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:36 smithi122 ceph-mon[32255]: pgmap v8702: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:48:37.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:38.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:48:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:48:38.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:38.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:38.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:38 smithi022 ceph-mon[31360]: pgmap v8703: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3413753577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:38 smithi022 ceph-mon[40250]: pgmap v8703: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3413753577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:38 smithi122 ceph-mon[32255]: pgmap v8703: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3413753577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:48:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:40 smithi022 ceph-mon[31360]: pgmap v8704: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:40 smithi022 ceph-mon[40250]: pgmap v8704: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:40 smithi122 ceph-mon[32255]: pgmap v8704: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:41 smithi022 ceph-mon[31360]: pgmap v8705: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:41 smithi022 ceph-mon[40250]: pgmap v8705: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:41 smithi122 ceph-mon[32255]: pgmap v8705: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:48:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:48:43.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:48:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:48:43.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:48:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:48:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:48:43.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:48:43.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:43.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:43.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:43.657 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:48:43.657 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:48:43.657 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:48:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:44 smithi122 ceph-mon[32255]: pgmap v8706: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4132127592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:44.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:48:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:44 smithi022 ceph-mon[31360]: pgmap v8706: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4132127592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:48:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:44 smithi022 ceph-mon[40250]: pgmap v8706: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4132127592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:48:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:46 smithi122 ceph-mon[32255]: pgmap v8707: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:46 smithi022 ceph-mon[31360]: pgmap v8707: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:46 smithi022 ceph-mon[40250]: pgmap v8707: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:48:48.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:48:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:48:48.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:48.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:48 smithi022 ceph-mon[31360]: pgmap v8708: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:48.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:48 smithi022 ceph-mon[40250]: pgmap v8708: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:48 smithi122 ceph-mon[32255]: pgmap v8708: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:49.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:49.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:49.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1233860902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1233860902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1233860902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:48:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:50 smithi122 ceph-mon[32255]: pgmap v8709: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:50 smithi022 ceph-mon[31360]: pgmap v8709: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:50 smithi022 ceph-mon[40250]: pgmap v8709: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:52 smithi122 ceph-mon[32255]: pgmap v8710: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:48:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:48:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:52 smithi022 ceph-mon[31360]: pgmap v8710: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:52 smithi022 ceph-mon[40250]: pgmap v8710: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:54.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:54.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:54.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:54.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:54 smithi122 ceph-mon[32255]: pgmap v8711: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/856736174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:54 smithi022 ceph-mon[31360]: pgmap v8711: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/856736174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:54 smithi022 ceph-mon[40250]: pgmap v8711: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:48:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/856736174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:48:56.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:56 smithi122 ceph-mon[32255]: pgmap v8712: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:56 smithi022 ceph-mon[31360]: pgmap v8712: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:56 smithi022 ceph-mon[40250]: pgmap v8712: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:48:58.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:48:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:48:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:58 smithi122 ceph-mon[32255]: pgmap v8713: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:58 smithi022 ceph-mon[31360]: pgmap v8713: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:58 smithi022 ceph-mon[40250]: pgmap v8713: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:48:59.394 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:48:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:48:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:59.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:48:59.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:48:59.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:48:59.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:48:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:48:59.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:48:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:49:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:00 smithi122 ceph-mon[32255]: pgmap v8714: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/947359636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:00 smithi022 ceph-mon[31360]: pgmap v8714: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:00.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/947359636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:00 smithi022 ceph-mon[40250]: pgmap v8714: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:00.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/947359636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:02 smithi122 ceph-mon[32255]: pgmap v8715: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:49:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:49:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:02 smithi022 ceph-mon[31360]: pgmap v8715: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:02 smithi022 ceph-mon[40250]: pgmap v8715: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:04 smithi122 ceph-mon[32255]: pgmap v8716: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:04.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:04.761 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:04 smithi022 ceph-mon[40250]: pgmap v8716: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:04.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:04 smithi022 ceph-mon[31360]: pgmap v8716: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:05.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:05.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:05.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:05.400 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2030786450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:05.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2030786450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2030786450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:06 smithi122 ceph-mon[32255]: pgmap v8717: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:06 smithi022 ceph-mon[31360]: pgmap v8717: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:06 smithi022 ceph-mon[40250]: pgmap v8717: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:49:08.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:49:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:49:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:08 smithi122 ceph-mon[32255]: pgmap v8718: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:08 smithi022 ceph-mon[31360]: pgmap v8718: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:08 smithi022 ceph-mon[40250]: pgmap v8718: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:10.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:49:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:10 smithi122 ceph-mon[32255]: pgmap v8719: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:10.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:10.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:10.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:10 smithi022 ceph-mon[31360]: pgmap v8719: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:10 smithi022 ceph-mon[40250]: pgmap v8719: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1501778356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1501778356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1501778356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:12 smithi122 ceph-mon[32255]: pgmap v8720: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:49:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:49:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:12 smithi022 ceph-mon[31360]: pgmap v8720: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:12 smithi022 ceph-mon[40250]: pgmap v8720: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:14 smithi122 ceph-mon[32255]: pgmap v8721: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:14 smithi022 ceph-mon[31360]: pgmap v8721: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:14 smithi022 ceph-mon[40250]: pgmap v8721: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:15.504 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:15.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:15.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:15.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:16 smithi122 ceph-mon[32255]: pgmap v8722: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744237291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:16 smithi022 ceph-mon[31360]: pgmap v8722: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744237291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:16 smithi022 ceph-mon[40250]: pgmap v8722: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744237291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:49:17.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:49:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:49:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:18 smithi122 ceph-mon[32255]: pgmap v8723: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:18 smithi022 ceph-mon[31360]: pgmap v8723: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:18 smithi022 ceph-mon[40250]: pgmap v8723: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:20] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:49:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:20 smithi122 ceph-mon[32255]: pgmap v8724: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:20 smithi022 ceph-mon[31360]: pgmap v8724: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:20 smithi022 ceph-mon[40250]: pgmap v8724: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:20.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:21.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:21.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:21.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:21.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4107591992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4107591992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:21.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4107591992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:22 smithi122 ceph-mon[32255]: pgmap v8725: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:49:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:49:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:22 smithi022 ceph-mon[31360]: pgmap v8725: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:22 smithi022 ceph-mon[40250]: pgmap v8725: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:24.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:24 smithi122 ceph-mon[32255]: pgmap v8726: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:24 smithi022 ceph-mon[31360]: pgmap v8726: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:24 smithi022 ceph-mon[40250]: pgmap v8726: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:26.247 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:26 smithi022 ceph-mon[31360]: pgmap v8727: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:26 smithi022 ceph-mon[40250]: pgmap v8727: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:26.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:26.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:26.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:26 smithi122 ceph-mon[32255]: pgmap v8727: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:27.715 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1312254725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1312254725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:49:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1312254725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:28.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:49:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:49:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:28 smithi022 ceph-mon[31360]: pgmap v8728: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:28 smithi022 ceph-mon[40250]: pgmap v8728: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:28 smithi122 ceph-mon[32255]: pgmap v8728: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:30] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:49:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:30 smithi022 ceph-mon[31360]: pgmap v8729: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:30 smithi022 ceph-mon[40250]: pgmap v8729: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:30 smithi122 ceph-mon[32255]: pgmap v8729: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:31.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:31.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:31.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:31.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:49:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:49:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:32 smithi122 ceph-mon[32255]: pgmap v8730: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3870864462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:32 smithi022 ceph-mon[31360]: pgmap v8730: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3870864462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:32 smithi022 ceph-mon[40250]: pgmap v8730: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3870864462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:49:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:49:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:49:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:49:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:49:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:49:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:49:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:49:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:49:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:49:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:49:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:49:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:34 smithi022 ceph-mon[31360]: pgmap v8731: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:34 smithi022 ceph-mon[40250]: pgmap v8731: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:34 smithi122 ceph-mon[32255]: pgmap v8731: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:36 smithi022 ceph-mon[31360]: pgmap v8732: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:36 smithi022 ceph-mon[40250]: pgmap v8732: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:36 smithi122 ceph-mon[32255]: pgmap v8732: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:37.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:37.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:37.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4109929582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:49:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4109929582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:37.898 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4109929582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:38.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:49:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:49:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:38 smithi022 ceph-mon[31360]: pgmap v8733: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:38 smithi022 ceph-mon[40250]: pgmap v8733: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:38 smithi122 ceph-mon[32255]: pgmap v8733: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:40] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:49:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:40 smithi022 ceph-mon[31360]: pgmap v8734: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:40 smithi022 ceph-mon[40250]: pgmap v8734: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:40 smithi122 ceph-mon[32255]: pgmap v8734: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:42 smithi122 ceph-mon[32255]: pgmap v8735: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:49:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:49:42.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:42.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:42.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:42.728 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:42 smithi022 ceph-mon[31360]: pgmap v8735: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:42.728 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:42 smithi022 ceph-mon[40250]: pgmap v8735: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:43.740 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/477977006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:43.745 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/477977006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:43.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/477977006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:44.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[40250]: pgmap v8736: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:44.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:49:44.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:49:44.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:49:44.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:44.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[31360]: pgmap v8736: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:49:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:49:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:49:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:44.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:44 smithi122 ceph-mon[32255]: pgmap v8736: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:49:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:49:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:49:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:45 smithi122 ceph-mon[32255]: pgmap v8737: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:45 smithi022 ceph-mon[31360]: pgmap v8737: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:49:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:45 smithi022 ceph-mon[40250]: pgmap v8737: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:47.730 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:47.730 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:49:48.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:48.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:48.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:48.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:49:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:49:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:48 smithi122 ceph-mon[32255]: pgmap v8738: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2814565657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:48 smithi022 ceph-mon[31360]: pgmap v8738: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2814565657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:48 smithi022 ceph-mon[40250]: pgmap v8738: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T22:49:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2814565657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:49.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:50] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:49:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:50 smithi122 ceph-mon[32255]: pgmap v8739: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:50 smithi022 ceph-mon[31360]: pgmap v8739: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:50 smithi022 ceph-mon[40250]: pgmap v8739: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T22:49:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:52 smithi122 ceph-mon[32255]: pgmap v8740: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:49:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:49:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:52 smithi022 ceph-mon[31360]: pgmap v8740: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:52 smithi022 ceph-mon[40250]: pgmap v8740: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:53.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:53.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:53.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:54 smithi122 ceph-mon[32255]: pgmap v8741: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3308264600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:54 smithi022 ceph-mon[31360]: pgmap v8741: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3308264600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:54 smithi022 ceph-mon[40250]: pgmap v8741: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:49:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3308264600' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:56 smithi122 ceph-mon[32255]: pgmap v8742: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:56 smithi022 ceph-mon[31360]: pgmap v8742: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:56 smithi022 ceph-mon[40250]: pgmap v8742: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:49:58.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:49:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:49:58.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:49:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:58 smithi122 ceph-mon[32255]: pgmap v8743: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:58 smithi022 ceph-mon[31360]: pgmap v8743: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:58 smithi022 ceph-mon[40250]: pgmap v8743: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:49:58.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:49:58.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:49:58.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:49:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:59.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:49:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3161528632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:49:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3161528632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:49:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:49:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:49:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3161528632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:00] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:50:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:00 smithi122 ceph-mon[32255]: pgmap v8744: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T22:50:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:00 smithi022 ceph-mon[31360]: pgmap v8744: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T22:50:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:00 smithi022 ceph-mon[40250]: pgmap v8744: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T22:50:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:02 smithi122 ceph-mon[32255]: pgmap v8745: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:50:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:50:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:02 smithi022 ceph-mon[31360]: pgmap v8745: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:02 smithi022 ceph-mon[40250]: pgmap v8745: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:03.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:04.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:04.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:04.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:04 smithi022 ceph-mon[31360]: pgmap v8746: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/776210817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:04 smithi022 ceph-mon[40250]: pgmap v8746: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/776210817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:04 smithi122 ceph-mon[32255]: pgmap v8746: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/776210817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:06 smithi122 ceph-mon[32255]: pgmap v8747: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:06 smithi022 ceph-mon[31360]: pgmap v8747: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:06 smithi022 ceph-mon[40250]: pgmap v8747: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:50:08.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:50:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:08 smithi122 ceph-mon[32255]: pgmap v8748: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:08 smithi022 ceph-mon[31360]: pgmap v8748: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:08 smithi022 ceph-mon[40250]: pgmap v8748: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:09.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:09.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:09.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:09.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:10 smithi122 ceph-mon[32255]: pgmap v8749: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3558695890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:50:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:10 smithi022 ceph-mon[31360]: pgmap v8749: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3558695890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:10 smithi022 ceph-mon[40250]: pgmap v8749: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3558695890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:12 smithi122 ceph-mon[32255]: pgmap v8750: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:50:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:50:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:12 smithi022 ceph-mon[31360]: pgmap v8750: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:12 smithi022 ceph-mon[40250]: pgmap v8750: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:14.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:14 smithi122 ceph-mon[32255]: pgmap v8751: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:14 smithi022 ceph-mon[31360]: pgmap v8751: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:14 smithi022 ceph-mon[40250]: pgmap v8751: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:14.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:14.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:14.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4230936513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4230936513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4230936513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:16 smithi122 ceph-mon[32255]: pgmap v8752: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:16 smithi022 ceph-mon[31360]: pgmap v8752: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:16 smithi022 ceph-mon[40250]: pgmap v8752: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:50:18.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:50:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:50:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:18 smithi122 ceph-mon[32255]: pgmap v8753: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:18 smithi022 ceph-mon[31360]: pgmap v8753: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:18 smithi022 ceph-mon[40250]: pgmap v8753: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:19.959 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:20.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:20.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:20.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:20] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:50:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:20 smithi022 ceph-mon[31360]: pgmap v8754: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2191851546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:20 smithi022 ceph-mon[40250]: pgmap v8754: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2191851546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:20 smithi122 ceph-mon[32255]: pgmap v8754: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2191851546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:22 smithi122 ceph-mon[32255]: pgmap v8755: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:50:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:50:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:22 smithi022 ceph-mon[31360]: pgmap v8755: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:22 smithi022 ceph-mon[40250]: pgmap v8755: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:24 smithi022 ceph-mon[31360]: pgmap v8756: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:24 smithi022 ceph-mon[40250]: pgmap v8756: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:24 smithi122 ceph-mon[32255]: pgmap v8756: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:25.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:25.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:25.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:25.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:26 smithi022 ceph-mon[31360]: pgmap v8757: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2132515956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:26 smithi022 ceph-mon[40250]: pgmap v8757: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:26.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2132515956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:26 smithi122 ceph-mon[32255]: pgmap v8757: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2132515956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:50:28.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:50:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:50:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:28 smithi022 ceph-mon[31360]: pgmap v8758: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:28 smithi022 ceph-mon[40250]: pgmap v8758: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:28 smithi122 ceph-mon[32255]: pgmap v8758: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:30] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:50:30.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:30 smithi022 ceph-mon[31360]: pgmap v8759: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:30 smithi022 ceph-mon[40250]: pgmap v8759: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:30 smithi122 ceph-mon[32255]: pgmap v8759: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:31.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:31.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:31.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1247262665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1247262665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1247262665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:50:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:50:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:32 smithi122 ceph-mon[32255]: pgmap v8760: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:32 smithi022 ceph-mon[31360]: pgmap v8760: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:32 smithi022 ceph-mon[40250]: pgmap v8760: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:50:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:50:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:50:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:50:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:50:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:50:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:50:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:50:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:50:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:50:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:50:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:50:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:34 smithi022 ceph-mon[31360]: pgmap v8761: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:34 smithi022 ceph-mon[40250]: pgmap v8761: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:34 smithi122 ceph-mon[32255]: pgmap v8761: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:36.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:36.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:36.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:36.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:36 smithi022 ceph-mon[31360]: pgmap v8762: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/464134972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:36 smithi022 ceph-mon[40250]: pgmap v8762: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/464134972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:36 smithi122 ceph-mon[32255]: pgmap v8762: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/464134972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:50:38.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:50:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:50:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:38 smithi022 ceph-mon[31360]: pgmap v8763: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:38 smithi022 ceph-mon[40250]: pgmap v8763: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:38 smithi122 ceph-mon[32255]: pgmap v8763: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:40] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:50:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:40 smithi022 ceph-mon[31360]: pgmap v8764: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:40 smithi022 ceph-mon[40250]: pgmap v8764: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:40 smithi122 ceph-mon[32255]: pgmap v8764: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:41.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:41.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:41.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:41.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:42 smithi122 ceph-mon[32255]: pgmap v8765: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/504840419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:50:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:50:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:42 smithi022 ceph-mon[31360]: pgmap v8765: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/504840419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:42 smithi022 ceph-mon[40250]: pgmap v8765: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/504840419' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:44 smithi022 ceph-mon[31360]: pgmap v8766: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:44 smithi022 ceph-mon[40250]: pgmap v8766: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:44 smithi122 ceph-mon[32255]: pgmap v8766: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:50:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:50:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:50:45.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:50:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:50:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:50:45.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:50:45.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:50:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:50:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:50:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:50:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:50:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:46 smithi022 ceph-mon[31360]: pgmap v8767: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:46 smithi022 ceph-mon[40250]: pgmap v8767: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:46.812 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:46 smithi122 ceph-mon[32255]: pgmap v8767: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:47.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:47.483 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:50:47.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3791095133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3791095133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3791095133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:48.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:50:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:50:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:48 smithi022 ceph-mon[31360]: pgmap v8768: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:48 smithi022 ceph-mon[40250]: pgmap v8768: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:48 smithi122 ceph-mon[32255]: pgmap v8768: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:50] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:50:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:50 smithi022 ceph-mon[31360]: pgmap v8769: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:50 smithi022 ceph-mon[40250]: pgmap v8769: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:50 smithi122 ceph-mon[32255]: pgmap v8769: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:52.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:52.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:52.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:52.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:52 smithi122 ceph-mon[32255]: pgmap v8770: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:50:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:50:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:52 smithi022 ceph-mon[31360]: pgmap v8770: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:52 smithi022 ceph-mon[40250]: pgmap v8770: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/949400049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/949400049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/949400049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:54 smithi022 ceph-mon[31360]: pgmap v8771: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:54 smithi022 ceph-mon[40250]: pgmap v8771: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:54 smithi122 ceph-mon[32255]: pgmap v8771: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:50:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:56 smithi022 ceph-mon[31360]: pgmap v8772: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:56 smithi022 ceph-mon[40250]: pgmap v8772: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:56 smithi122 ceph-mon[32255]: pgmap v8772: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:57.555 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:50:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:50:57.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:50:57.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:50:57.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:50:58.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:50:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:50:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:58 smithi022 ceph-mon[31360]: pgmap v8773: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2024892953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:58 smithi022 ceph-mon[40250]: pgmap v8773: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2024892953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:58 smithi122 ceph-mon[32255]: pgmap v8773: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:50:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2024892953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:50:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:50:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:50:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:50:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:50:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:00] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:51:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:00 smithi022 ceph-mon[31360]: pgmap v8774: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:00 smithi022 ceph-mon[40250]: pgmap v8774: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:00 smithi122 ceph-mon[32255]: pgmap v8774: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:02 smithi122 ceph-mon[32255]: pgmap v8775: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:51:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:51:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:02 smithi022 ceph-mon[31360]: pgmap v8775: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:02 smithi022 ceph-mon[40250]: pgmap v8775: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:03.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:03.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:03.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/55081053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/55081053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/55081053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:04 smithi022 ceph-mon[31360]: pgmap v8776: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:04 smithi022 ceph-mon[40250]: pgmap v8776: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:04 smithi122 ceph-mon[32255]: pgmap v8776: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:06 smithi022 ceph-mon[31360]: pgmap v8777: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:06 smithi022 ceph-mon[40250]: pgmap v8777: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:06 smithi122 ceph-mon[32255]: pgmap v8777: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:51:08.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:51:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:51:08.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:08.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:08.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:08.666 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:08 smithi022 ceph-mon[31360]: pgmap v8778: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:08.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:08 smithi022 ceph-mon[40250]: pgmap v8778: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:08 smithi122 ceph-mon[32255]: pgmap v8778: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2161550332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2161550332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2161550332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:51:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:10 smithi022 ceph-mon[31360]: pgmap v8779: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:10 smithi022 ceph-mon[40250]: pgmap v8779: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:10 smithi122 ceph-mon[32255]: pgmap v8779: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:12 smithi122 ceph-mon[32255]: pgmap v8780: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:51:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:51:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:12 smithi022 ceph-mon[31360]: pgmap v8780: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:12 smithi022 ceph-mon[40250]: pgmap v8780: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:13.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:14.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:14.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:14.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:14 smithi022 ceph-mon[40250]: pgmap v8781: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3755872280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:14.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:14 smithi022 ceph-mon[31360]: pgmap v8781: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:14.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3755872280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:14 smithi122 ceph-mon[32255]: pgmap v8781: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3755872280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:16 smithi022 ceph-mon[31360]: pgmap v8782: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:16 smithi022 ceph-mon[40250]: pgmap v8782: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:16 smithi122 ceph-mon[32255]: pgmap v8782: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:51:18.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:51:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:51:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:18 smithi122 ceph-mon[32255]: pgmap v8783: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:18 smithi022 ceph-mon[31360]: pgmap v8783: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:18 smithi022 ceph-mon[40250]: pgmap v8783: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:19.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:19.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:19.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:19.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2578428396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2578428396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2578428396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:51:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:20 smithi122 ceph-mon[32255]: pgmap v8784: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:20 smithi022 ceph-mon[31360]: pgmap v8784: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:20 smithi022 ceph-mon[40250]: pgmap v8784: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:22 smithi122 ceph-mon[32255]: pgmap v8785: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:51:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:51:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:22 smithi022 ceph-mon[31360]: pgmap v8785: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:22 smithi022 ceph-mon[40250]: pgmap v8785: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:24 smithi022 ceph-mon[31360]: pgmap v8786: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:24 smithi022 ceph-mon[40250]: pgmap v8786: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:24.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:24 smithi122 ceph-mon[32255]: pgmap v8786: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/870761923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/870761923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/870761923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:26 smithi122 ceph-mon[32255]: pgmap v8787: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:26 smithi022 ceph-mon[31360]: pgmap v8787: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:26 smithi022 ceph-mon[40250]: pgmap v8787: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:27 smithi022 ceph-mon[31360]: pgmap v8788: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:51:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:27 smithi022 ceph-mon[40250]: pgmap v8788: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:27 smithi122 ceph-mon[32255]: pgmap v8788: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:28.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:51:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:51:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:29.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:30.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:30.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:30.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:30 smithi122 ceph-mon[32255]: pgmap v8789: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4030196555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:51:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:30 smithi022 ceph-mon[31360]: pgmap v8789: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4030196555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:30 smithi022 ceph-mon[40250]: pgmap v8789: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4030196555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:32 smithi122 ceph-mon[32255]: pgmap v8790: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:51:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:51:32.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:32 smithi022 ceph-mon[31360]: pgmap v8790: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:32.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:32 smithi022 ceph-mon[40250]: pgmap v8790: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:51:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:51:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:51:33.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:51:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:51:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:51:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:51:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:51:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:51:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:51:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:51:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:51:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:34 smithi122 ceph-mon[32255]: pgmap v8791: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:34 smithi022 ceph-mon[31360]: pgmap v8791: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:34 smithi022 ceph-mon[40250]: pgmap v8791: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:35.154 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:35.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:36 smithi122 ceph-mon[32255]: pgmap v8792: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3824696205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:36 smithi022 ceph-mon[31360]: pgmap v8792: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3824696205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:36 smithi022 ceph-mon[40250]: pgmap v8792: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3824696205' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:51:38.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:51:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:51:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:38 smithi122 ceph-mon[32255]: pgmap v8793: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:38 smithi022 ceph-mon[31360]: pgmap v8793: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:38 smithi022 ceph-mon[40250]: pgmap v8793: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:51:40.525 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:40 smithi122 ceph-mon[32255]: pgmap v8794: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:40 smithi022 ceph-mon[31360]: pgmap v8794: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:40 smithi022 ceph-mon[40250]: pgmap v8794: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:40.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:40.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:40.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:41.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/837215727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/837215727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/837215727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:42 smithi122 ceph-mon[32255]: pgmap v8795: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:51:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:51:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:42 smithi022 ceph-mon[31360]: pgmap v8795: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:42 smithi022 ceph-mon[40250]: pgmap v8795: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:44 smithi022 ceph-mon[31360]: pgmap v8796: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:44 smithi022 ceph-mon[40250]: pgmap v8796: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:44 smithi122 ceph-mon[32255]: pgmap v8796: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:45.449 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:51:45.450 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:51:45.450 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:51:45.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:51:45.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:51:45.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:51:45.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:51:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:51:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:51:46.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:46.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:46.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:46 smithi022 ceph-mon[31360]: pgmap v8797: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:51:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3459474963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:46 smithi022 ceph-mon[40250]: pgmap v8797: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:51:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3459474963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:46 smithi122 ceph-mon[32255]: pgmap v8797: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:51:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3459474963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:51:48.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:51:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:51:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:48 smithi022 ceph-mon[31360]: pgmap v8798: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:48 smithi022 ceph-mon[40250]: pgmap v8798: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:48 smithi122 ceph-mon[32255]: pgmap v8798: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:50] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:51:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:50 smithi022 ceph-mon[31360]: pgmap v8799: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:50 smithi022 ceph-mon[40250]: pgmap v8799: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:50 smithi122 ceph-mon[32255]: pgmap v8799: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:51.258 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:51.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:52 smithi122 ceph-mon[32255]: pgmap v8800: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/893195565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:51:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:51:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:52 smithi022 ceph-mon[31360]: pgmap v8800: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/893195565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:52 smithi022 ceph-mon[40250]: pgmap v8800: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/893195565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:54 smithi022 ceph-mon[31360]: pgmap v8801: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:54 smithi022 ceph-mon[40250]: pgmap v8801: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:54 smithi122 ceph-mon[32255]: pgmap v8801: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:51:56.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:51:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:56 smithi022 ceph-mon[31360]: pgmap v8802: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:56 smithi022 ceph-mon[40250]: pgmap v8802: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:56 smithi122 ceph-mon[32255]: pgmap v8802: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:56.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:51:56.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:51:56.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:51:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4071569257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:51:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4071569257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4071569257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:51:58.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:51:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:51:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:58 smithi022 ceph-mon[31360]: pgmap v8803: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:58 smithi022 ceph-mon[40250]: pgmap v8803: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:58 smithi122 ceph-mon[32255]: pgmap v8803: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:51:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:51:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:51:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:51:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:51:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:00] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:52:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:00 smithi022 ceph-mon[31360]: pgmap v8804: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:00 smithi022 ceph-mon[40250]: pgmap v8804: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:00 smithi122 ceph-mon[32255]: pgmap v8804: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:02.000 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:02.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:02.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:02.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:52:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:52:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:02 smithi122 ceph-mon[32255]: pgmap v8805: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3619728645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:02 smithi022 ceph-mon[31360]: pgmap v8805: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3619728645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:02 smithi022 ceph-mon[40250]: pgmap v8805: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3619728645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:04 smithi022 ceph-mon[31360]: pgmap v8806: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:04 smithi022 ceph-mon[40250]: pgmap v8806: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:04 smithi122 ceph-mon[32255]: pgmap v8806: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:06 smithi022 ceph-mon[31360]: pgmap v8807: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:06 smithi022 ceph-mon[40250]: pgmap v8807: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:06 smithi122 ceph-mon[32255]: pgmap v8807: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:07.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:07.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:07.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:07.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:07.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:52:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:52:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:52:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:08 smithi022 ceph-mon[31360]: pgmap v8808: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1555799249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:08 smithi022 ceph-mon[40250]: pgmap v8808: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1555799249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:08 smithi122 ceph-mon[32255]: pgmap v8808: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1555799249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T22:52:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:10 smithi022 ceph-mon[31360]: pgmap v8809: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:10 smithi022 ceph-mon[40250]: pgmap v8809: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:10 smithi122 ceph-mon[32255]: pgmap v8809: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:12 smithi122 ceph-mon[32255]: pgmap v8810: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:52:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:52:12.744 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:12 smithi022 ceph-mon[31360]: pgmap v8810: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:12 smithi022 ceph-mon[40250]: pgmap v8810: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:13.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:13.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:13.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2852888152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2852888152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2852888152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:14 smithi022 ceph-mon[31360]: pgmap v8811: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:14 smithi022 ceph-mon[40250]: pgmap v8811: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:14 smithi122 ceph-mon[32255]: pgmap v8811: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:16 smithi022 ceph-mon[31360]: pgmap v8812: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:16 smithi022 ceph-mon[40250]: pgmap v8812: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:16 smithi122 ceph-mon[32255]: pgmap v8812: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:52:18.113 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:18.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:52:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:52:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:18 smithi022 ceph-mon[31360]: pgmap v8813: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2242899388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:18 smithi022 ceph-mon[40250]: pgmap v8813: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2242899388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:18 smithi122 ceph-mon[32255]: pgmap v8813: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2242899388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:52:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:20 smithi022 ceph-mon[31360]: pgmap v8814: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:20 smithi022 ceph-mon[40250]: pgmap v8814: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:20 smithi122 ceph-mon[32255]: pgmap v8814: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:22 smithi122 ceph-mon[32255]: pgmap v8815: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:52:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:52:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:22 smithi022 ceph-mon[31360]: pgmap v8815: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:22 smithi022 ceph-mon[40250]: pgmap v8815: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:23.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:23.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:23.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:23.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:24 smithi022 ceph-mon[31360]: pgmap v8816: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1034538765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:24 smithi022 ceph-mon[40250]: pgmap v8816: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1034538765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:24 smithi122 ceph-mon[32255]: pgmap v8816: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1034538765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:26 smithi022 ceph-mon[31360]: pgmap v8817: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:26 smithi022 ceph-mon[40250]: pgmap v8817: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:26 smithi122 ceph-mon[32255]: pgmap v8817: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:52:28.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:52:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:52:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:28 smithi022 ceph-mon[31360]: pgmap v8818: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:28 smithi022 ceph-mon[40250]: pgmap v8818: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:28.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:28 smithi122 ceph-mon[32255]: pgmap v8818: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:29.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:29.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:29.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4055944995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4055944995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4055944995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:52:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:30 smithi022 ceph-mon[31360]: pgmap v8819: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:30 smithi022 ceph-mon[40250]: pgmap v8819: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:30 smithi122 ceph-mon[32255]: pgmap v8819: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:32 smithi122 ceph-mon[32255]: pgmap v8820: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:52:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:52:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:32 smithi022 ceph-mon[31360]: pgmap v8820: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:32 smithi022 ceph-mon[40250]: pgmap v8820: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:52:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:52:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:52:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:52:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:52:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:52:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:52:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:52:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:52:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:52:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:52:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:52:34.229 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:34.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:34.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:34.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:34 smithi122 ceph-mon[32255]: pgmap v8821: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:34 smithi022 ceph-mon[31360]: pgmap v8821: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:34 smithi022 ceph-mon[40250]: pgmap v8821: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2127305175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2127305175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2127305175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:36 smithi122 ceph-mon[32255]: pgmap v8822: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:36 smithi022 ceph-mon[31360]: pgmap v8822: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:36 smithi022 ceph-mon[40250]: pgmap v8822: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:52:38.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:52:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:52:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:38 smithi122 ceph-mon[32255]: pgmap v8823: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:38 smithi022 ceph-mon[31360]: pgmap v8823: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:38 smithi022 ceph-mon[40250]: pgmap v8823: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:39.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:39.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:39.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:39.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:39.970 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:39.971 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:52:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:40 smithi122 ceph-mon[32255]: pgmap v8824: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2223452421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:40 smithi022 ceph-mon[31360]: pgmap v8824: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2223452421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:40 smithi022 ceph-mon[40250]: pgmap v8824: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2223452421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:42 smithi122 ceph-mon[32255]: pgmap v8825: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:52:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:52:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:42 smithi022 ceph-mon[31360]: pgmap v8825: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:42 smithi022 ceph-mon[40250]: pgmap v8825: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:44 smithi122 ceph-mon[32255]: pgmap v8826: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:44.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:44.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:44 smithi022 ceph-mon[31360]: pgmap v8826: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:44.972 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:44 smithi022 ceph-mon[40250]: pgmap v8826: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1694897840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:52:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:52:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:52:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1694897840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:52:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:52:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:52:45.920 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1694897840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:45.921 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:52:45.921 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:52:45.921 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:52:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:46 smithi122 ceph-mon[32255]: pgmap v8827: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:52:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:46 smithi022 ceph-mon[31360]: pgmap v8827: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:52:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:46 smithi022 ceph-mon[40250]: pgmap v8827: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:52:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:52:48.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:52:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:52:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:48 smithi122 ceph-mon[32255]: pgmap v8828: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:48 smithi022 ceph-mon[31360]: pgmap v8828: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:48 smithi022 ceph-mon[40250]: pgmap v8828: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:49 smithi122 ceph-mon[32255]: pgmap v8829: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:49 smithi022 ceph-mon[31360]: pgmap v8829: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:49 smithi022 ceph-mon[40250]: pgmap v8829: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:50.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:50] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:52:50.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:50.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:50.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4003303866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4003303866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4003303866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:52 smithi022 ceph-mon[31360]: pgmap v8830: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:52 smithi022 ceph-mon[40250]: pgmap v8830: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:52:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:52:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:52 smithi122 ceph-mon[32255]: pgmap v8830: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:54 smithi022 ceph-mon[31360]: pgmap v8831: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:54 smithi022 ceph-mon[40250]: pgmap v8831: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:54 smithi122 ceph-mon[32255]: pgmap v8831: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:52:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:52:56.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:52:56.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:52:56.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:52:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:56 smithi022 ceph-mon[31360]: pgmap v8832: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3296191088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:56 smithi022 ceph-mon[40250]: pgmap v8832: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3296191088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:56 smithi122 ceph-mon[32255]: pgmap v8832: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3296191088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:52:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:52:58.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:52:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:52:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:58 smithi022 ceph-mon[31360]: pgmap v8833: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:58 smithi022 ceph-mon[40250]: pgmap v8833: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:58 smithi122 ceph-mon[32255]: pgmap v8833: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:52:59.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:52:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:52:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:52:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:52:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:00] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:53:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:00 smithi022 ceph-mon[31360]: pgmap v8834: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:00 smithi022 ceph-mon[40250]: pgmap v8834: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:00 smithi122 ceph-mon[32255]: pgmap v8834: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:01.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:01.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:01.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:01.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/614901756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/614901756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/614901756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:02 smithi122 ceph-mon[32255]: pgmap v8835: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:53:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:53:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:02 smithi022 ceph-mon[31360]: pgmap v8835: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:02 smithi022 ceph-mon[40250]: pgmap v8835: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:04 smithi022 ceph-mon[31360]: pgmap v8836: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:04 smithi022 ceph-mon[40250]: pgmap v8836: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:04 smithi122 ceph-mon[32255]: pgmap v8836: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:06.450 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:06 smithi022 ceph-mon[31360]: pgmap v8837: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:06 smithi022 ceph-mon[40250]: pgmap v8837: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:06.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:06.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:06.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:06 smithi122 ceph-mon[32255]: pgmap v8837: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:53:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/23844054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/23844054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/23844054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:53:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:53:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:08 smithi022 ceph-mon[31360]: pgmap v8838: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:08 smithi022 ceph-mon[40250]: pgmap v8838: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:08 smithi122 ceph-mon[32255]: pgmap v8838: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:53:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:10 smithi022 ceph-mon[31360]: pgmap v8839: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:10 smithi022 ceph-mon[40250]: pgmap v8839: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:10 smithi122 ceph-mon[32255]: pgmap v8839: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:11.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:12.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:12.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:12.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:12 smithi122 ceph-mon[32255]: pgmap v8840: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1763533988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:53:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:53:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:12 smithi022 ceph-mon[31360]: pgmap v8840: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1763533988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:12 smithi022 ceph-mon[40250]: pgmap v8840: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1763533988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:14 smithi022 ceph-mon[31360]: pgmap v8841: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:14 smithi022 ceph-mon[40250]: pgmap v8841: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:14 smithi122 ceph-mon[32255]: pgmap v8841: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:16 smithi022 ceph-mon[31360]: pgmap v8842: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:16 smithi022 ceph-mon[40250]: pgmap v8842: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:16 smithi122 ceph-mon[32255]: pgmap v8842: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:17.194 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:53:17.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:17.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:17.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:18.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:53:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:53:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:18 smithi022 ceph-mon[31360]: pgmap v8843: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2109520901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:18 smithi022 ceph-mon[40250]: pgmap v8843: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2109520901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:18 smithi122 ceph-mon[32255]: pgmap v8843: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2109520901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:53:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:20 smithi022 ceph-mon[31360]: pgmap v8844: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:20 smithi022 ceph-mon[40250]: pgmap v8844: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:20 smithi122 ceph-mon[32255]: pgmap v8844: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:22.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:22 smithi122 ceph-mon[32255]: pgmap v8845: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:53:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:53:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:22 smithi022 ceph-mon[31360]: pgmap v8845: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:22 smithi022 ceph-mon[40250]: pgmap v8845: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:22.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:22.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:22.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/885218701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/885218701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/885218701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:24 smithi022 ceph-mon[31360]: pgmap v8846: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:24 smithi022 ceph-mon[40250]: pgmap v8846: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:24 smithi122 ceph-mon[32255]: pgmap v8846: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:26 smithi022 ceph-mon[31360]: pgmap v8847: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:26 smithi022 ceph-mon[40250]: pgmap v8847: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:26 smithi122 ceph-mon[32255]: pgmap v8847: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:53:27.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:28.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:53:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:53:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:28 smithi022 ceph-mon[31360]: pgmap v8848: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/627260475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:28 smithi022 ceph-mon[40250]: pgmap v8848: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/627260475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:28 smithi122 ceph-mon[32255]: pgmap v8848: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/627260475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:53:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:30 smithi022 ceph-mon[31360]: pgmap v8849: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:30.799 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:30 smithi022 ceph-mon[40250]: pgmap v8849: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:30 smithi122 ceph-mon[32255]: pgmap v8849: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:32 smithi122 ceph-mon[32255]: pgmap v8850: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:53:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:53:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:32 smithi022 ceph-mon[31360]: pgmap v8850: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:32 smithi022 ceph-mon[40250]: pgmap v8850: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:33.303 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:33.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:33.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:33.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:33.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:53:33.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:53:33.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:53:33.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:53:33.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:53:33.677 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:53:33.677 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:53:33.677 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:53:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:53:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:53:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:53:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:53:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:34 smithi022 ceph-mon[31360]: pgmap v8851: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/448346776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:34 smithi022 ceph-mon[40250]: pgmap v8851: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/448346776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:34 smithi122 ceph-mon[32255]: pgmap v8851: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/448346776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:36 smithi022 ceph-mon[31360]: pgmap v8852: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:36 smithi022 ceph-mon[40250]: pgmap v8852: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:36 smithi122 ceph-mon[32255]: pgmap v8852: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:53:38.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:53:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:53:38.676 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:38 smithi022 ceph-mon[31360]: pgmap v8853: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:38 smithi022 ceph-mon[40250]: pgmap v8853: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:38 smithi122 ceph-mon[32255]: pgmap v8853: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:39.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:39.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:39.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2490709242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:39.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2490709242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2490709242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T22:53:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:40 smithi022 ceph-mon[31360]: pgmap v8854: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:40 smithi022 ceph-mon[40250]: pgmap v8854: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:40 smithi122 ceph-mon[32255]: pgmap v8854: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:42 smithi122 ceph-mon[32255]: pgmap v8855: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:53:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:53:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:42 smithi022 ceph-mon[31360]: pgmap v8855: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:42 smithi022 ceph-mon[40250]: pgmap v8855: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:44.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:44.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:44.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:44.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:44 smithi022 ceph-mon[31360]: pgmap v8856: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1771870816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:44 smithi022 ceph-mon[40250]: pgmap v8856: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1771870816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:44 smithi122 ceph-mon[32255]: pgmap v8856: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1771870816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[31360]: pgmap v8857: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:53:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:53:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:53:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[40250]: pgmap v8857: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:53:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:53:46.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:53:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:46 smithi122 ceph-mon[32255]: pgmap v8857: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:53:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:53:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:53:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:53:48.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:53:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:53:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:48 smithi022 ceph-mon[31360]: pgmap v8858: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:48 smithi022 ceph-mon[40250]: pgmap v8858: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:48 smithi122 ceph-mon[32255]: pgmap v8858: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:49.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:49.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:49.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:49.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:53:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:53:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: pgmap v8859: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/647226669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: pgmap v8859: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/647226669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: pgmap v8859: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/647226669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T22:53:51.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:51.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:51 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3168M 2021-12-15T22:53:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:51 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3168M 2021-12-15T22:53:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:51 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3168M 2021-12-15T22:53:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:53:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:52 smithi122 ceph-mon[32255]: pgmap v8860: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:53:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:53:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:52 smithi022 ceph-mon[31360]: pgmap v8860: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:52 smithi022 ceph-mon[40250]: pgmap v8860: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:54.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:54 smithi022 ceph-mon[40250]: pgmap v8861: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:54.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:54 smithi022 ceph-mon[31360]: pgmap v8861: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:54.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:53:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:54 smithi122 ceph-mon[32255]: pgmap v8861: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:53:55.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:53:55.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:53:55.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:53:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2444493805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2444493805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2444493805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:53:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:56 smithi122 ceph-mon[32255]: pgmap v8862: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:56 smithi022 ceph-mon[31360]: pgmap v8862: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:56 smithi022 ceph-mon[40250]: pgmap v8862: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:53:58.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:53:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:53:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:58 smithi122 ceph-mon[32255]: pgmap v8863: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:58 smithi022 ceph-mon[31360]: pgmap v8863: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:58 smithi022 ceph-mon[40250]: pgmap v8863: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:53:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:53:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:53:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:53:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:00.161 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T22:54:00.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:00.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:00.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:00 smithi122 ceph-mon[32255]: pgmap v8864: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/141276838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:00 smithi022 ceph-mon[31360]: pgmap v8864: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/141276838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:00 smithi022 ceph-mon[40250]: pgmap v8864: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/141276838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:02 smithi122 ceph-mon[32255]: pgmap v8865: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:54:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:54:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:02 smithi022 ceph-mon[31360]: pgmap v8865: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:02 smithi022 ceph-mon[40250]: pgmap v8865: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:04 smithi122 ceph-mon[32255]: pgmap v8866: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:04 smithi022 ceph-mon[31360]: pgmap v8866: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:04 smithi022 ceph-mon[40250]: pgmap v8866: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:05.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:05.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:05.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:05.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:06 smithi122 ceph-mon[32255]: pgmap v8867: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/138761939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:06 smithi022 ceph-mon[31360]: pgmap v8867: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/138761939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:06 smithi022 ceph-mon[40250]: pgmap v8867: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/138761939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:54:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:54:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:54:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:08 smithi122 ceph-mon[32255]: pgmap v8868: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:08 smithi022 ceph-mon[31360]: pgmap v8868: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:08 smithi022 ceph-mon[40250]: pgmap v8868: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:10.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:10.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:10] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:54:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:10.908 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:10 smithi022 ceph-mon[31360]: pgmap v8869: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:10.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:10 smithi022 ceph-mon[40250]: pgmap v8869: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:10 smithi122 ceph-mon[32255]: pgmap v8869: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:11.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:11.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:11.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2478542429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2478542429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2478542429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:12.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:54:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:54:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:12 smithi122 ceph-mon[32255]: pgmap v8870: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:12 smithi022 ceph-mon[31360]: pgmap v8870: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:12 smithi022 ceph-mon[40250]: pgmap v8870: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:14 smithi122 ceph-mon[32255]: pgmap v8871: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:14 smithi022 ceph-mon[31360]: pgmap v8871: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:14 smithi022 ceph-mon[40250]: pgmap v8871: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:16.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:16.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:16.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:16.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:16 smithi122 ceph-mon[32255]: pgmap v8872: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:16 smithi022 ceph-mon[31360]: pgmap v8872: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:16 smithi022 ceph-mon[40250]: pgmap v8872: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:54:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/582934168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:17.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:17 smithi022 ceph-mon[31360]: pgmap v8873: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/582934168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:17 smithi022 ceph-mon[40250]: pgmap v8873: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/582934168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:17.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:17 smithi122 ceph-mon[32255]: pgmap v8873: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:18.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:54:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:54:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:54:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:20 smithi122 ceph-mon[32255]: pgmap v8874: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:20 smithi022 ceph-mon[31360]: pgmap v8874: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:20 smithi022 ceph-mon[40250]: pgmap v8874: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:21.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:22.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:22.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:22.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:22 smithi122 ceph-mon[32255]: pgmap v8875: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2888356206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:54:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:54:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:22 smithi022 ceph-mon[31360]: pgmap v8875: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2888356206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:22 smithi022 ceph-mon[40250]: pgmap v8875: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2888356206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:24 smithi022 ceph-mon[31360]: pgmap v8876: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:24 smithi022 ceph-mon[40250]: pgmap v8876: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:24 smithi122 ceph-mon[32255]: pgmap v8876: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:26 smithi022 ceph-mon[31360]: pgmap v8877: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:26 smithi022 ceph-mon[40250]: pgmap v8877: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:26 smithi122 ceph-mon[32255]: pgmap v8877: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:27.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:27.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:27.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:27.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/106496477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:54:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/106496477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/106496477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:28.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:54:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:54:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:28 smithi022 ceph-mon[31360]: pgmap v8878: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:28 smithi022 ceph-mon[40250]: pgmap v8878: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:28 smithi122 ceph-mon[32255]: pgmap v8878: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:54:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:30 smithi022 ceph-mon[31360]: pgmap v8879: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:30 smithi022 ceph-mon[40250]: pgmap v8879: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:30 smithi122 ceph-mon[32255]: pgmap v8879: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:32.398 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:32 smithi122 ceph-mon[32255]: pgmap v8880: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:54:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:54:32.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:32.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:32.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:32.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:32 smithi022 ceph-mon[31360]: pgmap v8880: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:32.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:32 smithi022 ceph-mon[40250]: pgmap v8880: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1641579851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:54:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:54:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:54:33.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:54:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1641579851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:54:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:54:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:54:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:54:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1641579851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:54:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:54:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:34 smithi022 ceph-mon[31360]: pgmap v8881: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:34 smithi022 ceph-mon[40250]: pgmap v8881: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:34 smithi122 ceph-mon[32255]: pgmap v8881: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:36 smithi022 ceph-mon[31360]: pgmap v8882: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:36 smithi022 ceph-mon[40250]: pgmap v8882: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:36 smithi122 ceph-mon[32255]: pgmap v8882: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:37.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:37.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:54:38.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:38.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:38.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:38.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:54:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:54:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:38 smithi022 ceph-mon[31360]: pgmap v8883: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1428886802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:38 smithi022 ceph-mon[40250]: pgmap v8883: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:38.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1428886802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:38 smithi122 ceph-mon[32255]: pgmap v8883: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1428886802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:40] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T22:54:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:40 smithi022 ceph-mon[31360]: pgmap v8884: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:40 smithi022 ceph-mon[40250]: pgmap v8884: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:40 smithi122 ceph-mon[32255]: pgmap v8884: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:42 smithi122 ceph-mon[32255]: pgmap v8885: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:54:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:54:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:42 smithi022 ceph-mon[31360]: pgmap v8885: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:42 smithi022 ceph-mon[40250]: pgmap v8885: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:43.142 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:43.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/796635183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:43.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/796635183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:43.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:43.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:43.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/796635183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:44 smithi022 ceph-mon[31360]: pgmap v8886: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:44 smithi022 ceph-mon[40250]: pgmap v8886: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:44 smithi122 ceph-mon[32255]: pgmap v8886: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:46 smithi022 ceph-mon[31360]: pgmap v8887: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:46 smithi022 ceph-mon[40250]: pgmap v8887: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:46 smithi122 ceph-mon[32255]: pgmap v8887: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:54:48.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:54:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:54:48.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:48 smithi022 ceph-mon[31360]: pgmap v8888: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:48 smithi022 ceph-mon[40250]: pgmap v8888: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:48.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:48.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:48.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:48 smithi122 ceph-mon[32255]: pgmap v8888: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/79235441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:49.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/79235441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:54:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/79235441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:50] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:54:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:50 smithi022 ceph-mon[31360]: pgmap v8889: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:50 smithi022 ceph-mon[40250]: pgmap v8889: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:50.806 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:50 smithi122 ceph-mon[32255]: pgmap v8889: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:54:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:54:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:54:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:54:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:54:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:54:51.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:54:51.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:54:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:54:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:54:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:54:51.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:54:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:52 smithi122 ceph-mon[32255]: pgmap v8890: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:54:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:54:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:52 smithi022 ceph-mon[31360]: pgmap v8890: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:52 smithi022 ceph-mon[40250]: pgmap v8890: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:53.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:54.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:54.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:54.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:54 smithi022 ceph-mon[31360]: pgmap v8891: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1193402260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:54 smithi022 ceph-mon[40250]: pgmap v8891: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1193402260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:54 smithi122 ceph-mon[32255]: pgmap v8891: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:54:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1193402260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:54:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:56 smithi122 ceph-mon[32255]: pgmap v8892: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:56 smithi022 ceph-mon[31360]: pgmap v8892: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:56 smithi022 ceph-mon[40250]: pgmap v8892: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:54:58.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:54:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:54:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:58 smithi122 ceph-mon[32255]: pgmap v8893: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:58 smithi022 ceph-mon[31360]: pgmap v8893: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:58 smithi022 ceph-mon[40250]: pgmap v8893: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:54:59.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:54:59.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:54:59.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:54:59.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:54:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:54:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:54:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:54:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:00] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T22:55:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:00 smithi122 ceph-mon[32255]: pgmap v8894: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3088951540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:00 smithi022 ceph-mon[31360]: pgmap v8894: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3088951540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:00 smithi022 ceph-mon[40250]: pgmap v8894: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3088951540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:02 smithi122 ceph-mon[32255]: pgmap v8895: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:55:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:55:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:02 smithi022 ceph-mon[31360]: pgmap v8895: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:02 smithi022 ceph-mon[40250]: pgmap v8895: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:04.635 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:04 smithi122 ceph-mon[32255]: pgmap v8896: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:05.004 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:04 smithi022 ceph-mon[31360]: pgmap v8896: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:05.004 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:04 smithi022 ceph-mon[40250]: pgmap v8896: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1789595768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1789595768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1789595768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:06 smithi122 ceph-mon[32255]: pgmap v8897: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:06 smithi022 ceph-mon[31360]: pgmap v8897: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:06 smithi022 ceph-mon[40250]: pgmap v8897: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:55:08.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:55:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:55:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:08 smithi122 ceph-mon[32255]: pgmap v8898: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:08 smithi022 ceph-mon[31360]: pgmap v8898: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:08 smithi022 ceph-mon[40250]: pgmap v8898: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:10.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:10.005 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:10.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:10.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:10.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:10.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:10] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:55:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:10 smithi022 ceph-mon[31360]: pgmap v8899: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3559059030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:10 smithi022 ceph-mon[40250]: pgmap v8899: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3559059030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:10 smithi122 ceph-mon[32255]: pgmap v8899: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3559059030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:12 smithi122 ceph-mon[32255]: pgmap v8900: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:55:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:55:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:12 smithi022 ceph-mon[31360]: pgmap v8900: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:12 smithi022 ceph-mon[40250]: pgmap v8900: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:14 smithi122 ceph-mon[32255]: pgmap v8901: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:14 smithi022 ceph-mon[31360]: pgmap v8901: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:14 smithi022 ceph-mon[40250]: pgmap v8901: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:15.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:15.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:15.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:16 smithi122 ceph-mon[32255]: pgmap v8902: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/553403873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:16 smithi022 ceph-mon[31360]: pgmap v8902: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/553403873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:16 smithi022 ceph-mon[40250]: pgmap v8902: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/553403873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:55:18.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:55:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:55:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:18 smithi122 ceph-mon[32255]: pgmap v8903: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:18 smithi022 ceph-mon[31360]: pgmap v8903: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:18 smithi022 ceph-mon[40250]: pgmap v8903: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:20] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:55:20.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:20 smithi122 ceph-mon[32255]: pgmap v8904: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:20 smithi022 ceph-mon[31360]: pgmap v8904: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:20 smithi022 ceph-mon[40250]: pgmap v8904: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:21.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:21.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:21.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4110912522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4110912522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4110912522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:22.572 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:55:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:55:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:22 smithi122 ceph-mon[32255]: pgmap v8905: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:22 smithi022 ceph-mon[31360]: pgmap v8905: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:22 smithi022 ceph-mon[40250]: pgmap v8905: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:24 smithi122 ceph-mon[32255]: pgmap v8906: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:24 smithi022 ceph-mon[31360]: pgmap v8906: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:24 smithi022 ceph-mon[40250]: pgmap v8906: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:26.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:26.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:26.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:26.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:26 smithi022 ceph-mon[31360]: pgmap v8907: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:26.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/76269398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:26 smithi022 ceph-mon[40250]: pgmap v8907: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:26.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/76269398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:26 smithi122 ceph-mon[32255]: pgmap v8907: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/76269398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:55:28.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:55:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:55:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:28 smithi122 ceph-mon[32255]: pgmap v8908: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:28 smithi022 ceph-mon[31360]: pgmap v8908: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:28 smithi022 ceph-mon[40250]: pgmap v8908: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:30] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T22:55:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:30 smithi122 ceph-mon[32255]: pgmap v8909: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:30 smithi022 ceph-mon[31360]: pgmap v8909: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:30 smithi022 ceph-mon[40250]: pgmap v8909: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:31 smithi022 ceph-mon[31360]: pgmap v8910: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:31 smithi022 ceph-mon[40250]: pgmap v8910: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:31.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:31.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:31.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:31 smithi122 ceph-mon[32255]: pgmap v8910: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:32.636 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:55:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:55:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2685448138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2685448138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2685448138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:55:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:55:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:55:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:55:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:33 smithi122 ceph-mon[32255]: pgmap v8911: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:55:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:55:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:55:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:55:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[31360]: pgmap v8911: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:55:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:55:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:55:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:55:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:33 smithi022 ceph-mon[40250]: pgmap v8911: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:36 smithi022 ceph-mon[31360]: pgmap v8912: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:36 smithi022 ceph-mon[40250]: pgmap v8912: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:36.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:36 smithi122 ceph-mon[32255]: pgmap v8912: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:37.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:37.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:37.500 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:55:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2653381669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2653381669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2653381669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:38.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:55:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:55:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:38 smithi022 ceph-mon[31360]: pgmap v8913: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:38.852 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:38 smithi022 ceph-mon[40250]: pgmap v8913: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:38 smithi122 ceph-mon[32255]: pgmap v8913: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:40] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T22:55:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:40 smithi022 ceph-mon[31360]: pgmap v8914: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:40 smithi022 ceph-mon[40250]: pgmap v8914: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:40 smithi122 ceph-mon[32255]: pgmap v8914: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:42.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:42.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:42.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:42.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:42 smithi122 ceph-mon[32255]: pgmap v8915: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:55:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:55:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:42 smithi022 ceph-mon[31360]: pgmap v8915: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:42 smithi022 ceph-mon[40250]: pgmap v8915: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2983937367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2983937367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2983937367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:44 smithi022 ceph-mon[31360]: pgmap v8916: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:44 smithi022 ceph-mon[40250]: pgmap v8916: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:44 smithi122 ceph-mon[32255]: pgmap v8916: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:46 smithi022 ceph-mon[31360]: pgmap v8917: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:46 smithi022 ceph-mon[40250]: pgmap v8917: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:46 smithi122 ceph-mon[32255]: pgmap v8917: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:47.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:48.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:55:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:55:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:48 smithi022 ceph-mon[31360]: pgmap v8918: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/39949423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:48 smithi022 ceph-mon[40250]: pgmap v8918: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:48.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/39949423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:48 smithi122 ceph-mon[32255]: pgmap v8918: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/39949423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:50] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:55:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:50 smithi022 ceph-mon[31360]: pgmap v8919: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:50 smithi022 ceph-mon[40250]: pgmap v8919: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:50 smithi122 ceph-mon[32255]: pgmap v8919: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:55:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:55:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:55:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:55:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:55:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:55:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:55:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:55:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:55:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:52 smithi122 ceph-mon[32255]: pgmap v8920: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:55:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:55:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:55:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:52 smithi022 ceph-mon[31360]: pgmap v8920: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:55:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:52 smithi022 ceph-mon[40250]: pgmap v8920: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:55:52.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:53.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:53.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:53.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3019985064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3019985064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3019985064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:55:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:54 smithi122 ceph-mon[32255]: pgmap v8921: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:54 smithi022 ceph-mon[31360]: pgmap v8921: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:54 smithi022 ceph-mon[40250]: pgmap v8921: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:55:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:56 smithi122 ceph-mon[32255]: pgmap v8922: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:56 smithi022 ceph-mon[31360]: pgmap v8922: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:56 smithi022 ceph-mon[40250]: pgmap v8922: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:55:58.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:55:58.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:55:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:55:58.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:55:58.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:55:58.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:55:58.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:58 smithi022 ceph-mon[40250]: pgmap v8923: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:58.716 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:58 smithi022 ceph-mon[31360]: pgmap v8923: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:58 smithi122 ceph-mon[32255]: pgmap v8923: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:55:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:55:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:55:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2963548372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:55:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2963548372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:55:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2963548372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:00] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:56:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:00 smithi122 ceph-mon[32255]: pgmap v8924: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:00 smithi022 ceph-mon[31360]: pgmap v8924: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:00 smithi022 ceph-mon[40250]: pgmap v8924: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:02 smithi122 ceph-mon[32255]: pgmap v8925: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:56:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:56:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:02 smithi022 ceph-mon[31360]: pgmap v8925: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:02 smithi022 ceph-mon[40250]: pgmap v8925: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:03.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:04.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:04.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:04.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:04 smithi122 ceph-mon[32255]: pgmap v8926: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2493981686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:04 smithi022 ceph-mon[31360]: pgmap v8926: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2493981686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:04 smithi022 ceph-mon[40250]: pgmap v8926: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2493981686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:06 smithi122 ceph-mon[32255]: pgmap v8927: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:06 smithi022 ceph-mon[31360]: pgmap v8927: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:06 smithi022 ceph-mon[40250]: pgmap v8927: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:56:08.550 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:56:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:56:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:08 smithi122 ceph-mon[32255]: pgmap v8928: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:08 smithi022 ceph-mon[31360]: pgmap v8928: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:08 smithi022 ceph-mon[40250]: pgmap v8928: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:09.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1439531156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1439531156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1439531156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:56:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:10 smithi122 ceph-mon[32255]: pgmap v8929: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:10 smithi022 ceph-mon[31360]: pgmap v8929: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:10 smithi022 ceph-mon[40250]: pgmap v8929: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:12 smithi122 ceph-mon[32255]: pgmap v8930: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:56:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:56:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:12 smithi022 ceph-mon[31360]: pgmap v8930: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:12 smithi022 ceph-mon[40250]: pgmap v8930: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:14.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:14 smithi022 ceph-mon[31360]: pgmap v8931: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:14 smithi022 ceph-mon[40250]: pgmap v8931: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:14.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:14.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:14.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:14 smithi122 ceph-mon[32255]: pgmap v8931: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4184858655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4184858655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4184858655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:16 smithi122 ceph-mon[32255]: pgmap v8932: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:16 smithi022 ceph-mon[31360]: pgmap v8932: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:16 smithi022 ceph-mon[40250]: pgmap v8932: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:56:18.567 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:56:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:56:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:18 smithi122 ceph-mon[32255]: pgmap v8933: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:18 smithi022 ceph-mon[31360]: pgmap v8933: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:18 smithi022 ceph-mon[40250]: pgmap v8933: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:19.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:19.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:19.832 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:20.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:20.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:20.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:56:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:20 smithi122 ceph-mon[32255]: pgmap v8934: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2231291831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:20 smithi022 ceph-mon[31360]: pgmap v8934: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2231291831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:20 smithi022 ceph-mon[40250]: pgmap v8934: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2231291831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:22.572 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:56:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:56:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:22 smithi122 ceph-mon[32255]: pgmap v8935: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:22 smithi022 ceph-mon[31360]: pgmap v8935: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:22 smithi022 ceph-mon[40250]: pgmap v8935: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:24 smithi122 ceph-mon[32255]: pgmap v8936: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:24 smithi022 ceph-mon[31360]: pgmap v8936: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:24 smithi022 ceph-mon[40250]: pgmap v8936: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:25.202 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:25.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3759378259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3759378259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3759378259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:26 smithi122 ceph-mon[32255]: pgmap v8937: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:26 smithi022 ceph-mon[31360]: pgmap v8937: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:26 smithi022 ceph-mon[40250]: pgmap v8937: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:56:28.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:56:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:56:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:28 smithi122 ceph-mon[32255]: pgmap v8938: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:28 smithi022 ceph-mon[31360]: pgmap v8938: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:28 smithi022 ceph-mon[40250]: pgmap v8938: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T22:56:30.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:30 smithi122 ceph-mon[32255]: pgmap v8939: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:30.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:30.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:30.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:30.945 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:30 smithi022 ceph-mon[31360]: pgmap v8939: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:30.945 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:30 smithi022 ceph-mon[40250]: pgmap v8939: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1957039168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1957039168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1957039168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:32.593 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:56:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:56:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:32 smithi122 ceph-mon[32255]: pgmap v8940: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:32 smithi022 ceph-mon[31360]: pgmap v8940: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:32 smithi022 ceph-mon[40250]: pgmap v8940: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:56:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:56:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:56:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:56:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:56:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:56:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:56:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:56:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:56:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:34 smithi122 ceph-mon[32255]: pgmap v8941: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:34 smithi022 ceph-mon[31360]: pgmap v8941: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:35.162 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:34 smithi022 ceph-mon[40250]: pgmap v8941: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:35 smithi122 ceph-mon[32255]: pgmap v8942: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:35.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:35.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:35 smithi022 ceph-mon[40250]: pgmap v8942: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:35.947 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:35 smithi022 ceph-mon[31360]: pgmap v8942: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:36.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:36.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:36.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:36.606 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2068462134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2068462134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2068462134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:37 smithi022 ceph-mon[31360]: pgmap v8943: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:56:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:37 smithi022 ceph-mon[40250]: pgmap v8943: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:37 smithi122 ceph-mon[32255]: pgmap v8943: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:38.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:56:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:56:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:40 smithi122 ceph-mon[32255]: pgmap v8944: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:56:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:40 smithi022 ceph-mon[31360]: pgmap v8944: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:40 smithi022 ceph-mon[40250]: pgmap v8944: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:41.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:41.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:41.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:41.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:56:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:56:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:42 smithi122 ceph-mon[32255]: pgmap v8945: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/900616619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:42 smithi022 ceph-mon[31360]: pgmap v8945: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/900616619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:42 smithi022 ceph-mon[40250]: pgmap v8945: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/900616619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:44 smithi022 ceph-mon[31360]: pgmap v8946: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:44 smithi022 ceph-mon[40250]: pgmap v8946: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:44 smithi122 ceph-mon[32255]: pgmap v8946: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:46.688 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:46 smithi022 ceph-mon[31360]: pgmap v8947: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:46 smithi022 ceph-mon[40250]: pgmap v8947: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:46 smithi122 ceph-mon[32255]: pgmap v8947: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:47.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:47.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:47.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2826625208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:56:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2826625208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2826625208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:48.528 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:56:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:56:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:48 smithi022 ceph-mon[31360]: pgmap v8948: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:48 smithi022 ceph-mon[40250]: pgmap v8948: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:48 smithi122 ceph-mon[32255]: pgmap v8948: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:56:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:50] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:56:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:50 smithi122 ceph-mon[32255]: pgmap v8949: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:50 smithi022 ceph-mon[31360]: pgmap v8949: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:50 smithi022 ceph-mon[40250]: pgmap v8949: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:52.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:52.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:52.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:52.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:56:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:56:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[31360]: pgmap v8950: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:56:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:56:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:56:52.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:56:52.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3679977760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[40250]: pgmap v8950: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:52.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:56:52.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:56:52.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:56:52.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:56:52.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3679977760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:52 smithi122 ceph-mon[32255]: pgmap v8950: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:56:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:56:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:56:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:56:52.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3679977760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:54 smithi122 ceph-mon[32255]: pgmap v8951: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:54 smithi022 ceph-mon[31360]: pgmap v8951: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:54 smithi022 ceph-mon[40250]: pgmap v8951: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:56:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:56 smithi122 ceph-mon[32255]: pgmap v8952: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:56 smithi022 ceph-mon[31360]: pgmap v8952: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:56 smithi022 ceph-mon[40250]: pgmap v8952: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:57.421 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:56:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:56:57.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:56:57.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:56:57.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:56:58.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:56:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:56:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:58 smithi122 ceph-mon[32255]: pgmap v8953: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2465250766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:58 smithi022 ceph-mon[31360]: pgmap v8953: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2465250766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:58 smithi022 ceph-mon[40250]: pgmap v8953: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:56:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2465250766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:56:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:56:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:56:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:56:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:00] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:57:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:00 smithi122 ceph-mon[32255]: pgmap v8954: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:00 smithi022 ceph-mon[31360]: pgmap v8954: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:00 smithi022 ceph-mon[40250]: pgmap v8954: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:02 smithi122 ceph-mon[32255]: pgmap v8955: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:57:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:57:02.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:02 smithi022 ceph-mon[31360]: pgmap v8955: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:02 smithi022 ceph-mon[40250]: pgmap v8955: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:03.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:03.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:03.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/287488817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/287488817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/287488817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:04 smithi122 ceph-mon[32255]: pgmap v8956: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:04 smithi022 ceph-mon[31360]: pgmap v8956: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:04 smithi022 ceph-mon[40250]: pgmap v8956: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:06 smithi122 ceph-mon[32255]: pgmap v8957: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:06 smithi022 ceph-mon[31360]: pgmap v8957: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:06 smithi022 ceph-mon[40250]: pgmap v8957: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:57:08.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:08.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:08.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:08.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:08.557 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:57:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:57:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:08 smithi022 ceph-mon[31360]: pgmap v8958: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/674260004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:08 smithi022 ceph-mon[40250]: pgmap v8958: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/674260004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:08 smithi122 ceph-mon[32255]: pgmap v8958: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/674260004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:10] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:57:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:10 smithi122 ceph-mon[32255]: pgmap v8959: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:10 smithi022 ceph-mon[31360]: pgmap v8959: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:10 smithi022 ceph-mon[40250]: pgmap v8959: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:12 smithi122 ceph-mon[32255]: pgmap v8960: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:57:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:57:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:12 smithi022 ceph-mon[31360]: pgmap v8960: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:12 smithi022 ceph-mon[40250]: pgmap v8960: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:13.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:13.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:13.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:13.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:14 smithi122 ceph-mon[32255]: pgmap v8961: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4012282408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:14 smithi022 ceph-mon[31360]: pgmap v8961: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4012282408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:14 smithi022 ceph-mon[40250]: pgmap v8961: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4012282408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:16 smithi122 ceph-mon[32255]: pgmap v8962: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:16 smithi022 ceph-mon[31360]: pgmap v8962: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:16 smithi022 ceph-mon[40250]: pgmap v8962: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:57:18.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:57:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:57:18.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:18.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:18 smithi022 ceph-mon[31360]: pgmap v8963: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:18.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:18 smithi022 ceph-mon[40250]: pgmap v8963: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:18 smithi122 ceph-mon[32255]: pgmap v8963: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:19.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:19.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:19.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3253501692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3253501692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3253501692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T22:57:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:20 smithi122 ceph-mon[32255]: pgmap v8964: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:20 smithi022 ceph-mon[31360]: pgmap v8964: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:20 smithi022 ceph-mon[40250]: pgmap v8964: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:22.576 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:57:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:57:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:22 smithi122 ceph-mon[32255]: pgmap v8965: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:22 smithi022 ceph-mon[31360]: pgmap v8965: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:22 smithi022 ceph-mon[40250]: pgmap v8965: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:24.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:24.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:24.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:24.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:24 smithi122 ceph-mon[32255]: pgmap v8966: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:24 smithi022 ceph-mon[31360]: pgmap v8966: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:24 smithi022 ceph-mon[40250]: pgmap v8966: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2124781032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2124781032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2124781032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:26 smithi122 ceph-mon[32255]: pgmap v8967: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:26 smithi022 ceph-mon[31360]: pgmap v8967: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:26 smithi022 ceph-mon[40250]: pgmap v8967: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:57:28.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:57:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:57:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:28 smithi122 ceph-mon[32255]: pgmap v8968: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:28 smithi022 ceph-mon[31360]: pgmap v8968: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:28 smithi022 ceph-mon[40250]: pgmap v8968: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:29.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:30.016 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:30.017 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T22:57:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:30 smithi122 ceph-mon[32255]: pgmap v8969: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4128602163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:30 smithi022 ceph-mon[31360]: pgmap v8969: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4128602163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:30 smithi022 ceph-mon[40250]: pgmap v8969: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4128602163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:32.597 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:57:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:57:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:32 smithi122 ceph-mon[32255]: pgmap v8970: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:32 smithi022 ceph-mon[31360]: pgmap v8970: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:32 smithi022 ceph-mon[40250]: pgmap v8970: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:57:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:57:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:57:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:57:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:57:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:57:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:57:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:57:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:57:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:34 smithi122 ceph-mon[32255]: pgmap v8971: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:35.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:35.018 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:34 smithi022 ceph-mon[31360]: pgmap v8971: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:35.019 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:34 smithi022 ceph-mon[40250]: pgmap v8971: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:35.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:35.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:35.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2605778391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2605778391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2605778391' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:36 smithi122 ceph-mon[32255]: pgmap v8972: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:36 smithi022 ceph-mon[31360]: pgmap v8972: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:36 smithi022 ceph-mon[40250]: pgmap v8972: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:37 smithi022 ceph-mon[31360]: pgmap v8973: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:57:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:37 smithi022 ceph-mon[40250]: pgmap v8973: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:37 smithi122 ceph-mon[32255]: pgmap v8973: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:38.615 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:57:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:57:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:40.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T22:57:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:40 smithi122 ceph-mon[32255]: pgmap v8974: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:40.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:40.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:40.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:40.756 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:40 smithi022 ceph-mon[31360]: pgmap v8974: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:40.757 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:40 smithi022 ceph-mon[40250]: pgmap v8974: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4211649946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4211649946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4211649946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:57:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:57:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:42 smithi122 ceph-mon[32255]: pgmap v8975: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:42 smithi022 ceph-mon[31360]: pgmap v8975: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:42 smithi022 ceph-mon[40250]: pgmap v8975: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:44 smithi122 ceph-mon[32255]: pgmap v8976: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:44 smithi022 ceph-mon[31360]: pgmap v8976: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:44 smithi022 ceph-mon[40250]: pgmap v8976: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:45.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:46.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:46.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:46.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:46 smithi122 ceph-mon[32255]: pgmap v8977: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1610964118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:46 smithi022 ceph-mon[31360]: pgmap v8977: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1610964118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:46 smithi022 ceph-mon[40250]: pgmap v8977: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1610964118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:57:48.539 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:57:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:57:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:48 smithi122 ceph-mon[32255]: pgmap v8978: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:48 smithi022 ceph-mon[31360]: pgmap v8978: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:48 smithi022 ceph-mon[40250]: pgmap v8978: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:57:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:50] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:57:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:50 smithi122 ceph-mon[32255]: pgmap v8979: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:50 smithi022 ceph-mon[31360]: pgmap v8979: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:50 smithi022 ceph-mon[40250]: pgmap v8979: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:57:51.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:51.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:51.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:51.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/836449147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/836449147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/836449147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:52.547 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:57:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:57:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:52 smithi122 ceph-mon[32255]: pgmap v8980: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:57:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:57:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:57:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[40250]: pgmap v8980: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:57:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:57:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:57:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[31360]: pgmap v8980: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:57:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:57:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:57:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:57:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:57:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:53 smithi122 ceph-mon[32255]: pgmap v8981: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:57:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:53 smithi022 ceph-mon[31360]: pgmap v8981: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:57:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:53 smithi022 ceph-mon[40250]: pgmap v8981: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:56.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:57:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:56 smithi022 ceph-mon[31360]: pgmap v8982: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:56 smithi022 ceph-mon[40250]: pgmap v8982: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:56.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:57:56.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:57:56.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:57:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:56 smithi122 ceph-mon[32255]: pgmap v8982: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/504933231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:57:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/504933231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/504933231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:57:58.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:57:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:57:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:58 smithi122 ceph-mon[32255]: pgmap v8983: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:58 smithi022 ceph-mon[31360]: pgmap v8983: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:58 smithi022 ceph-mon[40250]: pgmap v8983: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2021-12-15T22:57:59.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:57:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:57:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:57:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:00] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T22:58:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:00 smithi122 ceph-mon[32255]: pgmap v8984: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:58:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:00 smithi022 ceph-mon[31360]: pgmap v8984: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:58:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:00 smithi022 ceph-mon[40250]: pgmap v8984: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:58:01.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:02.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:02.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:02.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:58:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:58:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:02 smithi122 ceph-mon[32255]: pgmap v8985: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:58:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/407109552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:02 smithi022 ceph-mon[31360]: pgmap v8985: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:58:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/407109552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:02 smithi022 ceph-mon[40250]: pgmap v8985: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T22:58:03.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/407109552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:04 smithi122 ceph-mon[32255]: pgmap v8986: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:58:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:04 smithi022 ceph-mon[31360]: pgmap v8986: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:58:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:04 smithi022 ceph-mon[40250]: pgmap v8986: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T22:58:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:06 smithi122 ceph-mon[32255]: pgmap v8987: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:58:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:06 smithi022 ceph-mon[31360]: pgmap v8987: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:58:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:06 smithi022 ceph-mon[40250]: pgmap v8987: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:58:07.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:58:07.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:07.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:07.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:08.557 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:58:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:58:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:08 smithi122 ceph-mon[32255]: pgmap v8988: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:58:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2668560070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:08 smithi022 ceph-mon[31360]: pgmap v8988: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:58:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2668560070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:08 smithi022 ceph-mon[40250]: pgmap v8988: 132 pgs: 132 active+clean; 582 KiB data, 126 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2021-12-15T22:58:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2668560070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:58:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:10 smithi122 ceph-mon[32255]: pgmap v8989: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:58:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:10 smithi022 ceph-mon[31360]: pgmap v8989: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:58:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:10 smithi022 ceph-mon[40250]: pgmap v8989: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2021-12-15T22:58:12.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:12 smithi122 ceph-mon[32255]: pgmap v8990: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:58:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:58:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:58:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:12.980 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:12 smithi022 ceph-mon[31360]: pgmap v8990: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:58:12.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:12 smithi022 ceph-mon[40250]: pgmap v8990: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T22:58:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3240906307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3240906307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3240906307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:14 smithi122 ceph-mon[32255]: pgmap v8991: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:58:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:14 smithi022 ceph-mon[31360]: pgmap v8991: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:58:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:14 smithi022 ceph-mon[40250]: pgmap v8991: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T22:58:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:16 smithi122 ceph-mon[32255]: pgmap v8992: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:16 smithi022 ceph-mon[31360]: pgmap v8992: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:16 smithi022 ceph-mon[40250]: pgmap v8992: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:58:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:18.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:18.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:18.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:18.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:58:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:58:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:18 smithi022 ceph-mon[31360]: pgmap v8993: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/848826586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:18 smithi022 ceph-mon[40250]: pgmap v8993: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/848826586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:18 smithi122 ceph-mon[32255]: pgmap v8993: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/848826586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:58:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:20 smithi122 ceph-mon[32255]: pgmap v8994: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:20 smithi022 ceph-mon[31360]: pgmap v8994: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:20 smithi022 ceph-mon[40250]: pgmap v8994: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:22.579 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:58:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:58:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:22 smithi122 ceph-mon[32255]: pgmap v8995: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:22 smithi022 ceph-mon[31360]: pgmap v8995: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:22 smithi022 ceph-mon[40250]: pgmap v8995: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:23.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:23.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:23.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:23.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:24 smithi122 ceph-mon[32255]: pgmap v8996: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3698345192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:24 smithi022 ceph-mon[31360]: pgmap v8996: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3698345192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:24 smithi022 ceph-mon[40250]: pgmap v8996: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3698345192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:26 smithi122 ceph-mon[32255]: pgmap v8997: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:26 smithi022 ceph-mon[31360]: pgmap v8997: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:26 smithi022 ceph-mon[40250]: pgmap v8997: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:58:28.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:58:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:58:28.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:28 smithi122 ceph-mon[32255]: pgmap v8998: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:28 smithi022 ceph-mon[31360]: pgmap v8998: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:28 smithi022 ceph-mon[40250]: pgmap v8998: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:29.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:29.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:29.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1953795430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1953795430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1953795430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T22:58:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:30 smithi122 ceph-mon[32255]: pgmap v8999: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:30 smithi022 ceph-mon[31360]: pgmap v8999: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:30 smithi022 ceph-mon[40250]: pgmap v8999: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:32.593 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:58:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:58:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:32 smithi122 ceph-mon[32255]: pgmap v9000: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:32 smithi022 ceph-mon[31360]: pgmap v9000: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:32 smithi022 ceph-mon[40250]: pgmap v9000: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:58:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:58:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:58:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:58:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:58:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:58:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:58:34.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:34.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:34.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:34.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:34 smithi022 ceph-mon[31360]: pgmap v9001: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2726378852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:34 smithi022 ceph-mon[40250]: pgmap v9001: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2726378852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:34 smithi122 ceph-mon[32255]: pgmap v9001: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2726378852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:36 smithi122 ceph-mon[32255]: pgmap v9002: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:36 smithi022 ceph-mon[31360]: pgmap v9002: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:36 smithi022 ceph-mon[40250]: pgmap v9002: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:58:38.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:58:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:58:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:38 smithi122 ceph-mon[32255]: pgmap v9003: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:38 smithi022 ceph-mon[31360]: pgmap v9003: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:38 smithi022 ceph-mon[40250]: pgmap v9003: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T22:58:39.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:39.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:39.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:39.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:40] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T22:58:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:40 smithi122 ceph-mon[32255]: pgmap v9004: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3278469779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:40 smithi022 ceph-mon[31360]: pgmap v9004: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3278469779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:40 smithi022 ceph-mon[40250]: pgmap v9004: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3278469779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:42.609 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:58:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:58:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:42 smithi122 ceph-mon[32255]: pgmap v9005: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:42 smithi022 ceph-mon[31360]: pgmap v9005: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:42 smithi022 ceph-mon[40250]: pgmap v9005: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:44.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:44 smithi122 ceph-mon[32255]: pgmap v9006: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:44 smithi022 ceph-mon[31360]: pgmap v9006: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:44 smithi022 ceph-mon[40250]: pgmap v9006: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:45.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:45.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:45.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/319590517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/319590517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/319590517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:46 smithi122 ceph-mon[32255]: pgmap v9007: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:46 smithi022 ceph-mon[31360]: pgmap v9007: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:46 smithi022 ceph-mon[40250]: pgmap v9007: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:47.668 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:58:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:47 smithi122 ceph-mon[32255]: pgmap v9008: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:47 smithi022 ceph-mon[31360]: pgmap v9008: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:47 smithi022 ceph-mon[40250]: pgmap v9008: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:48.670 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:58:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:58:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:58:50.197 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:50] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:58:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:50 smithi122 ceph-mon[32255]: pgmap v9009: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:50 smithi022 ceph-mon[31360]: pgmap v9009: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:50 smithi022 ceph-mon[40250]: pgmap v9009: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:50.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:50.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:50.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2391510333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2391510333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2391510333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:52 smithi122 ceph-mon[32255]: pgmap v9010: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:58:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:58:52.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:52 smithi022 ceph-mon[31360]: pgmap v9010: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:52.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:52 smithi022 ceph-mon[40250]: pgmap v9010: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:58:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:58:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:58:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:58:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:58:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:58:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:58:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:58:53.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:58:53.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:58:53.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:58:53.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:58:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:54 smithi122 ceph-mon[32255]: pgmap v9011: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:54 smithi022 ceph-mon[31360]: pgmap v9011: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:54 smithi022 ceph-mon[40250]: pgmap v9011: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:58:55.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:58:55.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:58:55.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:58:55.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:58:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:56 smithi122 ceph-mon[32255]: pgmap v9012: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2533710493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:56 smithi022 ceph-mon[31360]: pgmap v9012: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2533710493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:56 smithi022 ceph-mon[40250]: pgmap v9012: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2533710493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:58:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:58:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:58 smithi122 ceph-mon[32255]: pgmap v9013: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:58.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:58:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:58:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:58 smithi022 ceph-mon[31360]: pgmap v9013: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:58 smithi022 ceph-mon[40250]: pgmap v9013: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:58:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:58:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:58:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:58:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:00] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:59:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:00 smithi122 ceph-mon[32255]: pgmap v9014: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:00.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:00.939 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:00 smithi022 ceph-mon[31360]: pgmap v9014: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:00.940 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:00 smithi022 ceph-mon[40250]: pgmap v9014: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:01.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:01.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:01.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:01.567 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2741392708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2741392708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2741392708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:02.568 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:59:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:59:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:02 smithi122 ceph-mon[32255]: pgmap v9015: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:02 smithi022 ceph-mon[31360]: pgmap v9015: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:02 smithi022 ceph-mon[40250]: pgmap v9015: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:04 smithi122 ceph-mon[32255]: pgmap v9016: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:04 smithi022 ceph-mon[31360]: pgmap v9016: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:04 smithi022 ceph-mon[40250]: pgmap v9016: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:06.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:06.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:06.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:06.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:06 smithi122 ceph-mon[32255]: pgmap v9017: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:06 smithi022 ceph-mon[31360]: pgmap v9017: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:06 smithi022 ceph-mon[40250]: pgmap v9017: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:59:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3982803588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3982803588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3982803588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:08 smithi122 ceph-mon[32255]: pgmap v9018: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:08.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:59:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:59:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:08 smithi022 ceph-mon[31360]: pgmap v9018: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:08 smithi022 ceph-mon[40250]: pgmap v9018: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:59:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:10 smithi122 ceph-mon[32255]: pgmap v9019: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:10 smithi022 ceph-mon[31360]: pgmap v9019: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:10 smithi022 ceph-mon[40250]: pgmap v9019: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:11.681 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:12.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:12.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:12.584 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:59:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:59:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:12 smithi122 ceph-mon[32255]: pgmap v9020: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/219933247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:12 smithi022 ceph-mon[31360]: pgmap v9020: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/219933247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:12 smithi022 ceph-mon[40250]: pgmap v9020: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/219933247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:14 smithi122 ceph-mon[32255]: pgmap v9021: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:14 smithi022 ceph-mon[31360]: pgmap v9021: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:14 smithi022 ceph-mon[40250]: pgmap v9021: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:16 smithi122 ceph-mon[32255]: pgmap v9022: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:16 smithi022 ceph-mon[31360]: pgmap v9022: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:16 smithi022 ceph-mon[40250]: pgmap v9022: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:17.048 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:17.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:17.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:17.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2254141437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:59:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2254141437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2254141437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:18 smithi122 ceph-mon[32255]: pgmap v9023: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:18.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:59:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:59:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:18 smithi022 ceph-mon[31360]: pgmap v9023: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:18 smithi022 ceph-mon[40250]: pgmap v9023: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:20] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:59:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:20 smithi122 ceph-mon[32255]: pgmap v9024: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:20 smithi022 ceph-mon[31360]: pgmap v9024: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:20 smithi022 ceph-mon[40250]: pgmap v9024: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:22.601 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:59:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:59:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:22 smithi022 ceph-mon[31360]: pgmap v9025: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:22 smithi022 ceph-mon[40250]: pgmap v9025: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:22 smithi122 ceph-mon[32255]: pgmap v9025: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/265265066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/265265066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/265265066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:24 smithi122 ceph-mon[32255]: pgmap v9026: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:24 smithi022 ceph-mon[31360]: pgmap v9026: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:24 smithi022 ceph-mon[40250]: pgmap v9026: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:26 smithi122 ceph-mon[32255]: pgmap v9027: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:26 smithi022 ceph-mon[31360]: pgmap v9027: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:26 smithi022 ceph-mon[40250]: pgmap v9027: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:59:27.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:28.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:28.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:28.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:28.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:59:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:59:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:28 smithi122 ceph-mon[32255]: pgmap v9028: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3894353078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:28 smithi022 ceph-mon[31360]: pgmap v9028: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3894353078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:28 smithi022 ceph-mon[40250]: pgmap v9028: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3894353078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:30] "GET /metrics HTTP/1.1" 200 202376 "" "Prometheus/2.18.1" 2021-12-15T22:59:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:30 smithi122 ceph-mon[32255]: pgmap v9029: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:30 smithi022 ceph-mon[31360]: pgmap v9029: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:30 smithi022 ceph-mon[40250]: pgmap v9029: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:32.618 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:59:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:59:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:32 smithi122 ceph-mon[32255]: pgmap v9030: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:32 smithi022 ceph-mon[31360]: pgmap v9030: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:32 smithi022 ceph-mon[40250]: pgmap v9030: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:33.163 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:33.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:33.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:33.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:59:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:59:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:59:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:59:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2593513927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:59:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:59:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:59:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:59:34.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2593513927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:59:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T22:59:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:59:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T22:59:34.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2593513927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:34 smithi122 ceph-mon[32255]: pgmap v9031: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:34 smithi022 ceph-mon[31360]: pgmap v9031: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:34 smithi022 ceph-mon[40250]: pgmap v9031: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:36 smithi122 ceph-mon[32255]: pgmap v9032: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:36 smithi022 ceph-mon[31360]: pgmap v9032: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:36 smithi022 ceph-mon[40250]: pgmap v9032: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:37 smithi022 ceph-mon[31360]: pgmap v9033: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:59:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:37 smithi022 ceph-mon[40250]: pgmap v9033: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:37 smithi122 ceph-mon[32255]: pgmap v9033: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:38.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:38.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:59:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:59:38.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:38.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:38.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:38.916 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:38.916 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2189960850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:39 smithi122 ceph-mon[32255]: pgmap v9034: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2189960850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:39 smithi022 ceph-mon[31360]: pgmap v9034: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2189960850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:39 smithi022 ceph-mon[40250]: pgmap v9034: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:40] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:59:42.571 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:59:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:59:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:42 smithi122 ceph-mon[32255]: pgmap v9035: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:42 smithi022 ceph-mon[31360]: pgmap v9035: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:42 smithi022 ceph-mon[40250]: pgmap v9035: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:43.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:44.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:44.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:44.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:44 smithi122 ceph-mon[32255]: pgmap v9036: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2600361719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:44 smithi022 ceph-mon[31360]: pgmap v9036: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2600361719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:45.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:44 smithi022 ceph-mon[40250]: pgmap v9036: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:45.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2600361719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:46 smithi122 ceph-mon[32255]: pgmap v9037: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:46 smithi022 ceph-mon[31360]: pgmap v9037: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:46 smithi022 ceph-mon[40250]: pgmap v9037: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:59:48.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:59:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:59:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:48 smithi122 ceph-mon[32255]: pgmap v9038: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:48 smithi022 ceph-mon[31360]: pgmap v9038: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:48 smithi022 ceph-mon[40250]: pgmap v9038: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:49.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:49.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:49.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:49.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T22:59:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 22:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:50] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T22:59:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:50 smithi122 ceph-mon[32255]: pgmap v9039: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3204567273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:50 smithi022 ceph-mon[31360]: pgmap v9039: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3204567273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:50 smithi022 ceph-mon[40250]: pgmap v9039: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:51.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3204567273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:52.584 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 22:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T22:59:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T22:59:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:52 smithi122 ceph-mon[32255]: pgmap v9040: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:52 smithi022 ceph-mon[31360]: pgmap v9040: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:52 smithi022 ceph-mon[40250]: pgmap v9040: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:59:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:59:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:59:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:53.962 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:59:53.962 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:59:53.962 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:59:53.963 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:53.963 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:53.963 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T22:59:53.963 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T22:59:53.964 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T22:59:53.964 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:53.964 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:54.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T22:59:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:54 smithi122 ceph-mon[32255]: pgmap v9041: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T22:59:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T22:59:55.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T22:59:55.023 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:54 smithi022 ceph-mon[31360]: pgmap v9041: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:55.023 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:55.024 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:54 smithi022 ceph-mon[40250]: pgmap v9041: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T22:59:55.025 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T22:59:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/764993733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/764993733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/764993733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T22:59:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:56 smithi122 ceph-mon[32255]: pgmap v9042: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:56 smithi022 ceph-mon[31360]: pgmap v9042: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:56 smithi022 ceph-mon[40250]: pgmap v9042: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 22:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:22:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T22:59:58.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 22:59:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T22:59:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:58 smithi122 ceph-mon[32255]: pgmap v9043: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:58 smithi022 ceph-mon[31360]: pgmap v9043: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:58 smithi022 ceph-mon[40250]: pgmap v9043: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T22:59:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 22:59:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:00.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:00.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 22:59:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:00.025 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 22:59:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:00.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:00.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:00.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:00] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:00:00.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:00 smithi122 conmon[74079]: level=info ts=2021-12-15T23:00:00.229Z caller=compact.go:495 component=tsdb msg="write block" mint=1639598400000 maxt=1639605600000 ulid=01FQ055VHJ7R39RPBMRTBXAVYA duration=50.868048ms 2021-12-15T23:00:00.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:00 smithi122 conmon[74079]: level=info ts=2021-12-15T23:00:00.239Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=2.893541ms 2021-12-15T23:00:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:00 smithi022 ceph-mon[31360]: pgmap v9044: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T23:00:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2978785949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:00 smithi022 ceph-mon[40250]: pgmap v9044: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T23:00:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2978785949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:00 smithi122 ceph-mon[32255]: pgmap v9044: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T23:00:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2978785949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:02.604 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:00:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:00:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:02 smithi122 ceph-mon[32255]: pgmap v9045: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:02 smithi022 ceph-mon[31360]: pgmap v9045: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:02 smithi022 ceph-mon[40250]: pgmap v9045: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:04 smithi122 ceph-mon[32255]: pgmap v9046: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:04 smithi022 ceph-mon[31360]: pgmap v9046: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:04 smithi022 ceph-mon[40250]: pgmap v9046: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:05.397 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:05.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:05.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:05.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:06 smithi122 ceph-mon[32255]: pgmap v9047: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1325151663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:06 smithi022 ceph-mon[31360]: pgmap v9047: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1325151663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:06 smithi022 ceph-mon[40250]: pgmap v9047: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1325151663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:00:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:08 smithi122 ceph-mon[32255]: pgmap v9048: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:08.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:00:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:00:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:08 smithi022 ceph-mon[31360]: pgmap v9048: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:08 smithi022 ceph-mon[40250]: pgmap v9048: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:00:10.769 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:10 smithi122 ceph-mon[32255]: pgmap v9049: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:10 smithi022 ceph-mon[31360]: pgmap v9049: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:10 smithi022 ceph-mon[40250]: pgmap v9049: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:11.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:11.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:11.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/579386633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/579386633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/579386633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:12.622 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:00:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:00:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:12 smithi122 ceph-mon[32255]: pgmap v9050: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:12 smithi022 ceph-mon[31360]: pgmap v9050: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:12 smithi022 ceph-mon[40250]: pgmap v9050: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:14 smithi122 ceph-mon[32255]: pgmap v9051: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:14 smithi022 ceph-mon[31360]: pgmap v9051: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:14 smithi022 ceph-mon[40250]: pgmap v9051: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:16.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:16.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:16 smithi022 ceph-mon[31360]: pgmap v9052: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1786121095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:16 smithi022 ceph-mon[40250]: pgmap v9052: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1786121095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:16 smithi122 ceph-mon[32255]: pgmap v9052: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1786121095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:00:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:18 smithi122 ceph-mon[32255]: pgmap v9053: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:18.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:00:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:00:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:18 smithi022 ceph-mon[31360]: pgmap v9053: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:18 smithi022 ceph-mon[40250]: pgmap v9053: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:20] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:00:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:20 smithi122 ceph-mon[32255]: pgmap v9054: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:20 smithi022 ceph-mon[31360]: pgmap v9054: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:20 smithi022 ceph-mon[40250]: pgmap v9054: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:21.512 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:21 smithi022 ceph-mon[31360]: pgmap v9055: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:21 smithi022 ceph-mon[40250]: pgmap v9055: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:21.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:21.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:21.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:21 smithi122 ceph-mon[32255]: pgmap v9055: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:00:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:00:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2225829338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2225829338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2225829338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:23 smithi022 ceph-mon[31360]: pgmap v9056: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:23 smithi022 ceph-mon[40250]: pgmap v9056: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:23 smithi122 ceph-mon[32255]: pgmap v9056: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:26.884 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:26.884 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:26 smithi022 ceph-mon[40250]: pgmap v9057: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:26.885 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:26 smithi022 ceph-mon[31360]: pgmap v9057: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:26 smithi122 ceph-mon[32255]: pgmap v9057: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:27.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:27.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:27.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:00:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4043371971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4043371971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4043371971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:28.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:00:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:00:28.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:28 smithi122 ceph-mon[32255]: pgmap v9058: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:28 smithi022 ceph-mon[31360]: pgmap v9058: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:28 smithi022 ceph-mon[40250]: pgmap v9058: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:30] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:00:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:30 smithi122 ceph-mon[32255]: pgmap v9059: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:30 smithi022 ceph-mon[31360]: pgmap v9059: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:30 smithi022 ceph-mon[40250]: pgmap v9059: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:32.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:32.597 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:00:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:00:32.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:32.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:32.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:32 smithi122 ceph-mon[32255]: pgmap v9060: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:32 smithi022 ceph-mon[31360]: pgmap v9060: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:32 smithi022 ceph-mon[40250]: pgmap v9060: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3747144259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:00:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:00:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:00:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:00:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3747144259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:00:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:00:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:00:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:00:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3747144259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:00:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:00:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:00:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:00:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:34 smithi122 ceph-mon[32255]: pgmap v9061: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:34 smithi022 ceph-mon[31360]: pgmap v9061: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:34 smithi022 ceph-mon[40250]: pgmap v9061: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:36 smithi122 ceph-mon[32255]: pgmap v9062: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:36 smithi022 ceph-mon[31360]: pgmap v9062: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:36 smithi022 ceph-mon[40250]: pgmap v9062: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:37.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:00:37.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:37.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:37.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:38.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:00:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:00:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:38 smithi122 ceph-mon[32255]: pgmap v9063: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2604077867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:38 smithi022 ceph-mon[31360]: pgmap v9063: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2604077867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:38 smithi022 ceph-mon[40250]: pgmap v9063: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2604077867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:40] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:00:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:40 smithi122 ceph-mon[32255]: pgmap v9064: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:40 smithi022 ceph-mon[31360]: pgmap v9064: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:40 smithi022 ceph-mon[40250]: pgmap v9064: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:42.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:00:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:00:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:42 smithi122 ceph-mon[32255]: pgmap v9065: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:42.993 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:42.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:42 smithi022 ceph-mon[31360]: pgmap v9065: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:42.995 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:42 smithi022 ceph-mon[40250]: pgmap v9065: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:43.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:43.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:43.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3348980012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3348980012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3348980012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:44 smithi122 ceph-mon[32255]: pgmap v9066: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:44 smithi022 ceph-mon[31360]: pgmap v9066: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:44 smithi022 ceph-mon[40250]: pgmap v9066: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:46 smithi122 ceph-mon[32255]: pgmap v9067: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:46 smithi022 ceph-mon[31360]: pgmap v9067: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:46 smithi022 ceph-mon[40250]: pgmap v9067: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:00:48.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:48 smithi122 ceph-mon[32255]: pgmap v9068: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:48.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:00:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:00:48.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:48.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:48.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:48 smithi022 ceph-mon[31360]: pgmap v9068: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:48 smithi022 ceph-mon[40250]: pgmap v9068: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3094947225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3094947225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:50.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3094947225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:00:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:50 smithi122 ceph-mon[32255]: pgmap v9069: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:50 smithi022 ceph-mon[31360]: pgmap v9069: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:50 smithi022 ceph-mon[40250]: pgmap v9069: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:52.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:00:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:00:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:52 smithi122 ceph-mon[32255]: pgmap v9070: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:52 smithi022 ceph-mon[31360]: pgmap v9070: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:52 smithi022 ceph-mon[40250]: pgmap v9070: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:53.744 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:54.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:54.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:54.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:54 smithi122 ceph-mon[32255]: pgmap v9071: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:00:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:00:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:00:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1239765613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:00:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[31360]: pgmap v9071: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:00:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:00:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:00:54.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1239765613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:54.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:00:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[40250]: pgmap v9071: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:00:54.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:00:54.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:00:54.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:00:54.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1239765613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:00:54.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:00:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:56 smithi122 ceph-mon[32255]: pgmap v9072: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:56 smithi022 ceph-mon[31360]: pgmap v9072: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:56 smithi022 ceph-mon[40250]: pgmap v9072: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:57.690 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:00:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:57 smithi022 ceph-mon[31360]: pgmap v9073: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:57 smithi022 ceph-mon[40250]: pgmap v9073: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:57 smithi122 ceph-mon[32255]: pgmap v9073: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:00:58.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:00:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:00:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:00:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:00:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:59.119 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:00:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:00:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:00:59.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:00:59.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:00:59.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:01:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/531545629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:00 smithi122 ceph-mon[32255]: pgmap v9074: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/531545629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:00 smithi022 ceph-mon[31360]: pgmap v9074: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/531545629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:00 smithi022 ceph-mon[40250]: pgmap v9074: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:02.597 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:01:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:01:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:02 smithi122 ceph-mon[32255]: pgmap v9075: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:02 smithi022 ceph-mon[31360]: pgmap v9075: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:02 smithi022 ceph-mon[40250]: pgmap v9075: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:04 smithi022 ceph-mon[31360]: pgmap v9076: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:04 smithi022 ceph-mon[40250]: pgmap v9076: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:04.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:04.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:04.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:04 smithi122 ceph-mon[32255]: pgmap v9076: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1666438837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1666438837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1666438837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:06 smithi122 ceph-mon[32255]: pgmap v9077: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:06 smithi022 ceph-mon[31360]: pgmap v9077: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:06 smithi022 ceph-mon[40250]: pgmap v9077: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:01:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:08 smithi122 ceph-mon[32255]: pgmap v9078: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:08.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:01:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:01:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:08 smithi022 ceph-mon[31360]: pgmap v9078: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:08 smithi022 ceph-mon[40250]: pgmap v9078: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:09.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:09.871 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:09.872 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:10.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:10.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:10] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:01:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:10 smithi122 ceph-mon[32255]: pgmap v9079: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4213646334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:10 smithi022 ceph-mon[31360]: pgmap v9079: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4213646334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:10 smithi022 ceph-mon[40250]: pgmap v9079: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4213646334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:12.613 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:01:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:01:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:12 smithi022 ceph-mon[31360]: pgmap v9080: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:12 smithi022 ceph-mon[40250]: pgmap v9080: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:12 smithi122 ceph-mon[32255]: pgmap v9080: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:14 smithi122 ceph-mon[32255]: pgmap v9081: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:14 smithi022 ceph-mon[31360]: pgmap v9081: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:14 smithi022 ceph-mon[40250]: pgmap v9081: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:15.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:15.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:15.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1658538665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1658538665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1658538665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:16 smithi122 ceph-mon[32255]: pgmap v9082: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:16 smithi022 ceph-mon[31360]: pgmap v9082: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:16 smithi022 ceph-mon[40250]: pgmap v9082: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:01:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:18 smithi122 ceph-mon[32255]: pgmap v9083: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:18.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:01:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:01:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:18 smithi022 ceph-mon[31360]: pgmap v9083: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:18 smithi022 ceph-mon[40250]: pgmap v9083: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:01:20.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:20 smithi122 ceph-mon[32255]: pgmap v9084: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:20.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:20.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:20.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:20.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:20 smithi022 ceph-mon[40250]: pgmap v9084: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:20.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:20 smithi022 ceph-mon[31360]: pgmap v9084: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/973627682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/973627682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/973627682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:22.632 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:01:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:01:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:22 smithi122 ceph-mon[32255]: pgmap v9085: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:22 smithi022 ceph-mon[31360]: pgmap v9085: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:22 smithi022 ceph-mon[40250]: pgmap v9085: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:24 smithi122 ceph-mon[32255]: pgmap v9086: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:24 smithi022 ceph-mon[31360]: pgmap v9086: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:24 smithi022 ceph-mon[40250]: pgmap v9086: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:25.982 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:26.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:26.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:26.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:26 smithi122 ceph-mon[32255]: pgmap v9087: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3911225176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:26 smithi022 ceph-mon[31360]: pgmap v9087: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3911225176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:26 smithi022 ceph-mon[40250]: pgmap v9087: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3911225176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:01:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:28 smithi122 ceph-mon[32255]: pgmap v9088: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:28.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:01:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:01:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:28 smithi022 ceph-mon[31360]: pgmap v9088: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:28 smithi022 ceph-mon[40250]: pgmap v9088: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:01:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:30 smithi122 ceph-mon[32255]: pgmap v9089: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:30 smithi022 ceph-mon[31360]: pgmap v9089: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:30 smithi022 ceph-mon[40250]: pgmap v9089: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:31.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:31.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:31.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:31.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:32.646 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:01:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:01:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:32 smithi122 ceph-mon[32255]: pgmap v9090: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2341617405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:32 smithi022 ceph-mon[31360]: pgmap v9090: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2341617405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:32 smithi022 ceph-mon[40250]: pgmap v9090: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2341617405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:01:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:01:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:01:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:01:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:33 smithi122 ceph-mon[32255]: pgmap v9091: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:01:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[31360]: pgmap v9091: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:01:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:01:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:01:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:01:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:33 smithi022 ceph-mon[40250]: pgmap v9091: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:36 smithi122 ceph-mon[32255]: pgmap v9092: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:36 smithi022 ceph-mon[31360]: pgmap v9092: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:36 smithi022 ceph-mon[40250]: pgmap v9092: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:37.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:37.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:37.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/588339096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:01:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/588339096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/588339096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:38.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:01:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:01:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:38 smithi122 ceph-mon[32255]: pgmap v9093: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:38 smithi022 ceph-mon[31360]: pgmap v9093: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:38 smithi022 ceph-mon[40250]: pgmap v9093: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:40] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:01:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:40 smithi122 ceph-mon[32255]: pgmap v9094: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:40 smithi022 ceph-mon[31360]: pgmap v9094: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:40 smithi022 ceph-mon[40250]: pgmap v9094: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:42.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:42.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:42.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:42.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:42.620 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:01:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:01:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:42 smithi022 ceph-mon[31360]: pgmap v9095: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2635666173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:42 smithi022 ceph-mon[40250]: pgmap v9095: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2635666173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:42 smithi122 ceph-mon[32255]: pgmap v9095: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2635666173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:44 smithi122 ceph-mon[32255]: pgmap v9096: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:44 smithi022 ceph-mon[31360]: pgmap v9096: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:44 smithi022 ceph-mon[40250]: pgmap v9096: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:46 smithi122 ceph-mon[32255]: pgmap v9097: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:46 smithi022 ceph-mon[31360]: pgmap v9097: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:46 smithi022 ceph-mon[40250]: pgmap v9097: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:47.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:01:47.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:47.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:47.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:48.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:01:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:01:48.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:48 smithi122 ceph-mon[32255]: pgmap v9098: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:48.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/584662786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:48 smithi022 ceph-mon[31360]: pgmap v9098: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/584662786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:48 smithi022 ceph-mon[40250]: pgmap v9098: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/584662786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:01:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:01:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:50 smithi122 ceph-mon[32255]: pgmap v9099: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:50 smithi022 ceph-mon[31360]: pgmap v9099: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:50 smithi022 ceph-mon[40250]: pgmap v9099: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:52.632 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:01:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:01:52.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:52 smithi122 ceph-mon[32255]: pgmap v9100: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:52 smithi022 ceph-mon[31360]: pgmap v9100: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:52 smithi022 ceph-mon[40250]: pgmap v9100: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:53.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:53.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:53.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3212186706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3212186706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3212186706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[31360]: pgmap v9101: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:01:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:01:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:01:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[40250]: pgmap v9101: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:01:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:01:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:01:54.874 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:54 smithi122 ceph-mon[32255]: pgmap v9101: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:01:54.874 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:01:54.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:01:54.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:01:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:01:56.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:55 smithi122 ceph-mon[32255]: pgmap v9102: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:01:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:55 smithi022 ceph-mon[31360]: pgmap v9102: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:01:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:55 smithi022 ceph-mon[40250]: pgmap v9102: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:01:58.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:01:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:01:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:01:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:01:58.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:01:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:01:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:58 smithi122 ceph-mon[32255]: pgmap v9103: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3615384827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:58 smithi022 ceph-mon[31360]: pgmap v9103: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3615384827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:58 smithi022 ceph-mon[40250]: pgmap v9103: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:01:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3615384827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:01:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:01:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:01:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:02:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:00 smithi122 ceph-mon[32255]: pgmap v9104: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:00 smithi022 ceph-mon[31360]: pgmap v9104: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:00 smithi022 ceph-mon[40250]: pgmap v9104: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:02.620 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:02:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:02:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:02 smithi122 ceph-mon[32255]: pgmap v9105: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:02 smithi022 ceph-mon[31360]: pgmap v9105: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:02 smithi022 ceph-mon[40250]: pgmap v9105: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:03.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:03.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:04 smithi122 ceph-mon[32255]: pgmap v9106: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3635906322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:04 smithi022 ceph-mon[31360]: pgmap v9106: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3635906322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:04 smithi022 ceph-mon[40250]: pgmap v9106: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3635906322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:06 smithi122 ceph-mon[32255]: pgmap v9107: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:06 smithi022 ceph-mon[31360]: pgmap v9107: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:06 smithi022 ceph-mon[40250]: pgmap v9107: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:02:08.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:02:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:02:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:08 smithi122 ceph-mon[32255]: pgmap v9108: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:08.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:08.943 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:08 smithi022 ceph-mon[31360]: pgmap v9108: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:08.944 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:08 smithi022 ceph-mon[40250]: pgmap v9108: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:09.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:09.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:09.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2202882884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2202882884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:10.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2202882884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:02:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:10 smithi122 ceph-mon[32255]: pgmap v9109: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:10 smithi022 ceph-mon[31360]: pgmap v9109: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:10 smithi022 ceph-mon[40250]: pgmap v9109: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:12.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:02:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:02:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:12 smithi122 ceph-mon[32255]: pgmap v9110: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:12 smithi022 ceph-mon[40250]: pgmap v9110: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:12 smithi022 ceph-mon[31360]: pgmap v9110: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:14.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:14.635 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:14 smithi022 ceph-mon[31360]: pgmap v9111: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:14.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:14.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:14.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:14 smithi122 ceph-mon[32255]: pgmap v9111: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:14 smithi022 ceph-mon[40250]: pgmap v9111: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4082284739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4082284739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4082284739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:16 smithi122 ceph-mon[32255]: pgmap v9112: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:16 smithi022 ceph-mon[31360]: pgmap v9112: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:16 smithi022 ceph-mon[40250]: pgmap v9112: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:02:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:18 smithi122 ceph-mon[32255]: pgmap v9113: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:18.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:02:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:02:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:18 smithi022 ceph-mon[31360]: pgmap v9113: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:18 smithi022 ceph-mon[40250]: pgmap v9113: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:19.688 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:20.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:20.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:20.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:20] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T23:02:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:20 smithi122 ceph-mon[32255]: pgmap v9114: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2060044577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:20 smithi022 ceph-mon[31360]: pgmap v9114: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2060044577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:20 smithi022 ceph-mon[40250]: pgmap v9114: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2060044577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:22.649 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:02:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:02:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:22 smithi122 ceph-mon[32255]: pgmap v9115: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:22 smithi022 ceph-mon[31360]: pgmap v9115: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:22 smithi022 ceph-mon[40250]: pgmap v9115: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:24 smithi122 ceph-mon[32255]: pgmap v9116: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:24 smithi022 ceph-mon[31360]: pgmap v9116: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:24 smithi022 ceph-mon[40250]: pgmap v9116: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:25.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:25.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:25.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:25.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1295324067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1295324067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1295324067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:26 smithi122 ceph-mon[32255]: pgmap v9117: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:26 smithi022 ceph-mon[31360]: pgmap v9117: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:26 smithi022 ceph-mon[40250]: pgmap v9117: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:27.662 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:02:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:27 smithi122 ceph-mon[32255]: pgmap v9118: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:27 smithi022 ceph-mon[31360]: pgmap v9118: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:27 smithi022 ceph-mon[40250]: pgmap v9118: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:28.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:02:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:02:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:30.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:30 smithi122 ceph-mon[32255]: pgmap v9119: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:30] "GET /metrics HTTP/1.1" 200 202372 "" "Prometheus/2.18.1" 2021-12-15T23:02:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:30 smithi022 ceph-mon[31360]: pgmap v9119: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:30 smithi022 ceph-mon[40250]: pgmap v9119: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:30.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:30.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:30.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/809816123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:31.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/809816123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/809816123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:32 smithi122 ceph-mon[32255]: pgmap v9120: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:02:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:02:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:32 smithi022 ceph-mon[31360]: pgmap v9120: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:32 smithi022 ceph-mon[40250]: pgmap v9120: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:02:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:02:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:02:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:02:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:02:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:02:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:02:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:02:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:02:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:02:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:02:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:34 smithi122 ceph-mon[32255]: pgmap v9121: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:34 smithi022 ceph-mon[31360]: pgmap v9121: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:34 smithi022 ceph-mon[40250]: pgmap v9121: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:35.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:36.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:36.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:36.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:36 smithi122 ceph-mon[32255]: pgmap v9122: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3089096244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:36 smithi022 ceph-mon[31360]: pgmap v9122: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3089096244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:36 smithi022 ceph-mon[40250]: pgmap v9122: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3089096244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:02:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:38 smithi122 ceph-mon[32255]: pgmap v9123: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:38.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:02:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:02:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:38 smithi022 ceph-mon[31360]: pgmap v9123: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:38 smithi022 ceph-mon[40250]: pgmap v9123: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:40] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T23:02:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:40 smithi122 ceph-mon[32255]: pgmap v9124: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:40 smithi022 ceph-mon[31360]: pgmap v9124: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:40 smithi022 ceph-mon[40250]: pgmap v9124: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:41.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:41.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:41.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:41.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1077422097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1077422097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1077422097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:42.637 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:02:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:02:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:42 smithi122 ceph-mon[32255]: pgmap v9125: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:42 smithi022 ceph-mon[31360]: pgmap v9125: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:42 smithi022 ceph-mon[40250]: pgmap v9125: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:44 smithi122 ceph-mon[32255]: pgmap v9126: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:44 smithi022 ceph-mon[31360]: pgmap v9126: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:44 smithi022 ceph-mon[40250]: pgmap v9126: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:46.538 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:46.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:46.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:46.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:46.906 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:46 smithi022 ceph-mon[40250]: pgmap v9127: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:46.907 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:46 smithi022 ceph-mon[31360]: pgmap v9127: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:46 smithi122 ceph-mon[32255]: pgmap v9127: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:47.647 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:02:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4265948528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4265948528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4265948528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:48 smithi122 ceph-mon[32255]: pgmap v9128: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:48.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:02:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:02:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:48 smithi022 ceph-mon[31360]: pgmap v9128: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:48 smithi022 ceph-mon[40250]: pgmap v9128: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:50] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T23:02:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:50 smithi122 ceph-mon[32255]: pgmap v9129: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:50 smithi022 ceph-mon[31360]: pgmap v9129: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:50 smithi022 ceph-mon[40250]: pgmap v9129: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:51.907 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:52.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:52.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:52.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:52.654 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:02:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:02:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:52 smithi122 ceph-mon[32255]: pgmap v9130: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3108205738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:52 smithi022 ceph-mon[31360]: pgmap v9130: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3108205738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:52 smithi022 ceph-mon[40250]: pgmap v9130: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3108205738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:54.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:54 smithi022 ceph-mon[40250]: pgmap v9131: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:54.933 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:54 smithi022 ceph-mon[31360]: pgmap v9131: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:54 smithi122 ceph-mon[32255]: pgmap v9131: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:02:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:02:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:02:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:02:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:02:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:02:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:02:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:02:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:02:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:02:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:02:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:02:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:02:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:56 smithi122 ceph-mon[32255]: pgmap v9132: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:56 smithi022 ceph-mon[31360]: pgmap v9132: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:56 smithi022 ceph-mon[40250]: pgmap v9132: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:57.278 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:02:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:02:57.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:02:57.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:02:57.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:02:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4278421590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4278421590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4278421590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:02:58.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:02:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:02:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:58 smithi122 ceph-mon[32255]: pgmap v9133: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:02:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:58 smithi022 ceph-mon[31360]: pgmap v9133: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:02:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:02:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:58 smithi022 ceph-mon[40250]: pgmap v9133: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:02:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:02:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:00] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T23:03:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:00 smithi122 ceph-mon[32255]: pgmap v9134: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:00 smithi022 ceph-mon[40250]: pgmap v9134: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:00.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:00 smithi022 ceph-mon[31360]: pgmap v9134: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:02.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:03:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:03:02.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:02 smithi122 ceph-mon[32255]: pgmap v9135: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:03.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:03.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:03.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:03.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:02 smithi022 ceph-mon[31360]: pgmap v9135: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:03.017 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:02 smithi022 ceph-mon[40250]: pgmap v9135: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3914092676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3914092676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3914092676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:04 smithi122 ceph-mon[32255]: pgmap v9136: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:04 smithi022 ceph-mon[31360]: pgmap v9136: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:04 smithi022 ceph-mon[40250]: pgmap v9136: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:06 smithi122 ceph-mon[32255]: pgmap v9137: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:06 smithi022 ceph-mon[31360]: pgmap v9137: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:06 smithi022 ceph-mon[40250]: pgmap v9137: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:03:08.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:08.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:08.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:08.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:08.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:03:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:03:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:08 smithi122 ceph-mon[32255]: pgmap v9138: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508803572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:08 smithi022 ceph-mon[31360]: pgmap v9138: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508803572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:08 smithi022 ceph-mon[40250]: pgmap v9138: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:09.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508803572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:03:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:10 smithi122 ceph-mon[32255]: pgmap v9139: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:10 smithi022 ceph-mon[31360]: pgmap v9139: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:10 smithi022 ceph-mon[40250]: pgmap v9139: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:12.651 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:03:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:03:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:12 smithi122 ceph-mon[32255]: pgmap v9140: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:12 smithi022 ceph-mon[31360]: pgmap v9140: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:12 smithi022 ceph-mon[40250]: pgmap v9140: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:13.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:13.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:13.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:13.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:14 smithi122 ceph-mon[32255]: pgmap v9141: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1789614892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:14 smithi022 ceph-mon[40250]: pgmap v9141: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1789614892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:14 smithi022 ceph-mon[31360]: pgmap v9141: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1789614892' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:16 smithi122 ceph-mon[32255]: pgmap v9142: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:16 smithi022 ceph-mon[31360]: pgmap v9142: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:16 smithi022 ceph-mon[40250]: pgmap v9142: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:03:18.765 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:18.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:03:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:03:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:18 smithi122 ceph-mon[32255]: pgmap v9143: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:18 smithi022 ceph-mon[31360]: pgmap v9143: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:18 smithi022 ceph-mon[40250]: pgmap v9143: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:19.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:19.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:19.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2876237653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2876237653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2876237653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:20] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:03:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:20 smithi122 ceph-mon[32255]: pgmap v9144: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:20 smithi022 ceph-mon[31360]: pgmap v9144: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:20 smithi022 ceph-mon[40250]: pgmap v9144: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:21 smithi122 ceph-mon[32255]: pgmap v9145: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:21 smithi022 ceph-mon[31360]: pgmap v9145: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:21 smithi022 ceph-mon[40250]: pgmap v9145: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:03:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:03:24.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:24.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:24 smithi022 ceph-mon[31360]: pgmap v9146: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2513519075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:24 smithi022 ceph-mon[40250]: pgmap v9146: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2513519075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:24 smithi122 ceph-mon[32255]: pgmap v9146: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2513519075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:26 smithi122 ceph-mon[32255]: pgmap v9147: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:26 smithi022 ceph-mon[31360]: pgmap v9147: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:26 smithi022 ceph-mon[40250]: pgmap v9147: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:03:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:28 smithi122 ceph-mon[32255]: pgmap v9148: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:28.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:03:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:03:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:28 smithi022 ceph-mon[31360]: pgmap v9148: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:28 smithi022 ceph-mon[40250]: pgmap v9148: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:29.502 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:29.841 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:29.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:29.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:30] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:03:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:30 smithi122 ceph-mon[32255]: pgmap v9149: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/153098617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:30 smithi022 ceph-mon[31360]: pgmap v9149: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/153098617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:30 smithi022 ceph-mon[40250]: pgmap v9149: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/153098617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:32.650 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:03:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:03:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:32 smithi122 ceph-mon[32255]: pgmap v9150: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:32 smithi022 ceph-mon[31360]: pgmap v9150: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:32 smithi022 ceph-mon[40250]: pgmap v9150: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:03:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:03:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:03:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:03:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:03:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:03:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:03:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:03:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:03:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:03:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:03:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:03:34.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:34 smithi122 ceph-mon[32255]: pgmap v9151: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:34 smithi022 ceph-mon[31360]: pgmap v9151: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:34 smithi022 ceph-mon[40250]: pgmap v9151: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:35.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:35.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:35.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/967222274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/967222274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/967222274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:36 smithi122 ceph-mon[32255]: pgmap v9152: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:36 smithi022 ceph-mon[31360]: pgmap v9152: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:36 smithi022 ceph-mon[40250]: pgmap v9152: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:03:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:38 smithi122 ceph-mon[32255]: pgmap v9153: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:38.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:03:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:03:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:38 smithi022 ceph-mon[31360]: pgmap v9153: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:38 smithi022 ceph-mon[40250]: pgmap v9153: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:40.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:40.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:40] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T23:03:40.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:40.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:40.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:40 smithi122 ceph-mon[32255]: pgmap v9154: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1889935504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:40 smithi022 ceph-mon[31360]: pgmap v9154: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1889935504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:40 smithi022 ceph-mon[40250]: pgmap v9154: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1889935504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:41 smithi122 ceph-mon[32255]: pgmap v9155: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:41 smithi022 ceph-mon[31360]: pgmap v9155: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:41 smithi022 ceph-mon[40250]: pgmap v9155: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:03:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:03:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:44 smithi122 ceph-mon[32255]: pgmap v9156: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:44 smithi022 ceph-mon[31360]: pgmap v9156: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:44 smithi022 ceph-mon[40250]: pgmap v9156: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:45.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:45.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:45.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:45.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:46 smithi122 ceph-mon[32255]: pgmap v9157: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/704210093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:46 smithi022 ceph-mon[31360]: pgmap v9157: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/704210093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:46 smithi022 ceph-mon[40250]: pgmap v9157: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/704210093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:03:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:48 smithi122 ceph-mon[32255]: pgmap v9158: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:48.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:03:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:03:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:48 smithi022 ceph-mon[31360]: pgmap v9158: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:48 smithi022 ceph-mon[40250]: pgmap v9158: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:50] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T23:03:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:50 smithi122 ceph-mon[32255]: pgmap v9159: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:50.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:50.992 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:50 smithi022 ceph-mon[31360]: pgmap v9159: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:50.993 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:50 smithi022 ceph-mon[40250]: pgmap v9159: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:51.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:51.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:51.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2200658878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2200658878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2200658878' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:52.660 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:03:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:03:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:52 smithi122 ceph-mon[32255]: pgmap v9160: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:52 smithi022 ceph-mon[31360]: pgmap v9160: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:52 smithi022 ceph-mon[40250]: pgmap v9160: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:54 smithi122 ceph-mon[32255]: pgmap v9161: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:54 smithi022 ceph-mon[31360]: pgmap v9161: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:54 smithi022 ceph-mon[40250]: pgmap v9161: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:03:55.666 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:03:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:03:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:03:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:03:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:03:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:03:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:03:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:03:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:03:56.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:03:56.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:56 smithi022 ceph-mon[40250]: pgmap v9162: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:56.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:03:56.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:03:56.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:03:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:56 smithi122 ceph-mon[32255]: pgmap v9162: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:56 smithi022 ceph-mon[31360]: pgmap v9162: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:57.670 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:03:57.671 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/389152482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/389152482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/389152482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:03:58.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:03:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:03:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:58 smithi122 ceph-mon[32255]: pgmap v9163: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:58.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:58 smithi022 ceph-mon[31360]: pgmap v9163: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:58.952 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:03:58.953 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:58 smithi022 ceph-mon[40250]: pgmap v9163: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:03:58.953 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[40250]: pgmap v9164: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[31360]: pgmap v9164: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:00.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:03:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.174 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:00] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T23:04:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:59 smithi122 ceph-mon[32255]: pgmap v9164: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:03:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:01.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:01.840 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:01.840 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:01.908 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:02.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:02.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:02.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:04:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:04:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: pgmap v9165: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4049503305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:02.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: pgmap v9165: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4049503305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: pgmap v9165: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4049503305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:04:03.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:03 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3158M 2021-12-15T23:04:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:03 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3158M 2021-12-15T23:04:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:03 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3158M 2021-12-15T23:04:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:04:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:04 smithi122 ceph-mon[32255]: pgmap v9166: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:04 smithi022 ceph-mon[31360]: pgmap v9166: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:04 smithi022 ceph-mon[40250]: pgmap v9166: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:06 smithi122 ceph-mon[32255]: pgmap v9167: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:06 smithi022 ceph-mon[31360]: pgmap v9167: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:06 smithi022 ceph-mon[40250]: pgmap v9167: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:07.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:07.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:07.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:07.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:07.655 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:04:07.655 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1914886111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1914886111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1914886111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:08 smithi122 ceph-mon[32255]: pgmap v9168: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:08.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:04:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:04:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:08 smithi022 ceph-mon[31360]: pgmap v9168: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:08 smithi022 ceph-mon[40250]: pgmap v9168: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:04:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:10 smithi122 ceph-mon[32255]: pgmap v9169: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:10 smithi022 ceph-mon[31360]: pgmap v9169: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:10 smithi022 ceph-mon[40250]: pgmap v9169: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:12.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:12.662 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:04:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:04:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:12 smithi022 ceph-mon[31360]: pgmap v9170: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:12 smithi022 ceph-mon[40250]: pgmap v9170: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:12.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:12.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:12.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:12 smithi122 ceph-mon[32255]: pgmap v9170: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2494209512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2494209512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2494209512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:14 smithi122 ceph-mon[32255]: pgmap v9171: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:14 smithi022 ceph-mon[31360]: pgmap v9171: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:14 smithi022 ceph-mon[40250]: pgmap v9171: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:16 smithi122 ceph-mon[32255]: pgmap v9172: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:16 smithi022 ceph-mon[31360]: pgmap v9172: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:16 smithi022 ceph-mon[40250]: pgmap v9172: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:04:17.843 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:18.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:18.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:18.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:18.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:04:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:04:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:18 smithi122 ceph-mon[32255]: pgmap v9173: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1593129665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:18 smithi022 ceph-mon[31360]: pgmap v9173: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1593129665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:18 smithi022 ceph-mon[40250]: pgmap v9173: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1593129665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:19 smithi122 ceph-mon[32255]: pgmap v9174: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:19 smithi022 ceph-mon[31360]: pgmap v9174: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:19 smithi022 ceph-mon[40250]: pgmap v9174: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:20] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:04:22.656 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:04:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:04:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:22 smithi122 ceph-mon[32255]: pgmap v9175: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:22 smithi022 ceph-mon[31360]: pgmap v9175: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:22 smithi022 ceph-mon[40250]: pgmap v9175: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:23.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:23.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:23.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:23.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3357399308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3357399308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3357399308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:24 smithi122 ceph-mon[32255]: pgmap v9176: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:24 smithi022 ceph-mon[31360]: pgmap v9176: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:24 smithi022 ceph-mon[40250]: pgmap v9176: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:26 smithi122 ceph-mon[32255]: pgmap v9177: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:26 smithi022 ceph-mon[31360]: pgmap v9177: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:26 smithi022 ceph-mon[40250]: pgmap v9177: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:04:28.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:28 smithi122 ceph-mon[32255]: pgmap v9178: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:28.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:04:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:04:28.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:28.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:28.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:28.963 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:28 smithi022 ceph-mon[31360]: pgmap v9178: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:28.963 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:28 smithi022 ceph-mon[40250]: pgmap v9178: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/282332343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/282332343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/282332343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:30] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:04:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:30 smithi122 ceph-mon[32255]: pgmap v9179: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:30 smithi022 ceph-mon[31360]: pgmap v9179: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:30 smithi022 ceph-mon[40250]: pgmap v9179: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:32.673 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:04:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:04:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:32 smithi122 ceph-mon[32255]: pgmap v9180: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:32 smithi022 ceph-mon[31360]: pgmap v9180: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:32 smithi022 ceph-mon[40250]: pgmap v9180: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:04:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:04:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:04:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:04:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:33.964 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:04:33.965 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:04:33.965 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:04:33.965 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:04:33.965 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:04:33.965 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:04:33.966 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:04:33.966 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:04:34.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:34.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:34.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:34 smithi122 ceph-mon[32255]: pgmap v9181: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3158198140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:34 smithi022 ceph-mon[31360]: pgmap v9181: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3158198140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:34 smithi022 ceph-mon[40250]: pgmap v9181: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3158198140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:35 smithi122 ceph-mon[32255]: pgmap v9182: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:35 smithi022 ceph-mon[31360]: pgmap v9182: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:35 smithi022 ceph-mon[40250]: pgmap v9182: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:04:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:38 smithi122 ceph-mon[32255]: pgmap v9183: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:38.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:04:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:04:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:38 smithi022 ceph-mon[31360]: pgmap v9183: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:38 smithi022 ceph-mon[40250]: pgmap v9183: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:39.335 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:39.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:39.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:39.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:40] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:04:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:40 smithi122 ceph-mon[32255]: pgmap v9184: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1195271713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:40 smithi022 ceph-mon[31360]: pgmap v9184: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1195271713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:40 smithi022 ceph-mon[40250]: pgmap v9184: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1195271713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:42.666 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:04:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:04:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:42 smithi122 ceph-mon[32255]: pgmap v9185: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:42 smithi022 ceph-mon[31360]: pgmap v9185: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:42 smithi022 ceph-mon[40250]: pgmap v9185: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:44.710 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:44 smithi122 ceph-mon[32255]: pgmap v9186: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:44 smithi022 ceph-mon[31360]: pgmap v9186: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:44 smithi022 ceph-mon[40250]: pgmap v9186: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:45.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:45.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:45.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1291338017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1291338017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1291338017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:46 smithi122 ceph-mon[32255]: pgmap v9187: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:46 smithi022 ceph-mon[31360]: pgmap v9187: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:46 smithi022 ceph-mon[40250]: pgmap v9187: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:04:48.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:04:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:04:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:48 smithi122 ceph-mon[32255]: pgmap v9188: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:48 smithi022 ceph-mon[31360]: pgmap v9188: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:48 smithi022 ceph-mon[40250]: pgmap v9188: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:49 smithi122 ceph-mon[32255]: pgmap v9189: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:49 smithi022 ceph-mon[31360]: pgmap v9189: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:04:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:49 smithi022 ceph-mon[40250]: pgmap v9189: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:50.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:50] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T23:04:50.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:50.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:50.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2671166383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2671166383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2671166383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:52 smithi122 ceph-mon[32255]: pgmap v9190: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:04:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:04:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:52 smithi022 ceph-mon[31360]: pgmap v9190: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:52 smithi022 ceph-mon[40250]: pgmap v9190: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:54 smithi122 ceph-mon[32255]: pgmap v9191: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:54 smithi022 ceph-mon[31360]: pgmap v9191: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:54 smithi022 ceph-mon[40250]: pgmap v9191: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:04:55.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:04:55.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:04:55.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:04:55.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:04:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:56 smithi122 ceph-mon[32255]: pgmap v9192: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2627319655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:56 smithi022 ceph-mon[31360]: pgmap v9192: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2627319655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:56 smithi022 ceph-mon[40250]: pgmap v9192: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2627319655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:04:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:04:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:58 smithi122 ceph-mon[32255]: pgmap v9193: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:58.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:04:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:04:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:58 smithi022 ceph-mon[31360]: pgmap v9193: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:58 smithi022 ceph-mon[40250]: pgmap v9193: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:04:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:04:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:04:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:04:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:00] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T23:05:00.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:00 smithi122 ceph-mon[32255]: pgmap v9194: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:00 smithi022 ceph-mon[31360]: pgmap v9194: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:00 smithi022 ceph-mon[40250]: pgmap v9194: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:01.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:01.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:01.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3098881780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3098881780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3098881780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:02.680 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:05:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:05:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:02 smithi122 ceph-mon[32255]: pgmap v9195: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:02 smithi022 ceph-mon[31360]: pgmap v9195: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:02 smithi022 ceph-mon[40250]: pgmap v9195: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:05:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:05:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:05:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:05:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:03 smithi122 ceph-mon[32255]: pgmap v9196: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:05:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:05:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:05:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:05:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[31360]: pgmap v9196: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:05:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:05:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:05:04.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:05:04.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:03 smithi022 ceph-mon[40250]: pgmap v9196: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:06.198 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:06.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:06.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:06.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:06 smithi122 ceph-mon[32255]: pgmap v9197: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2605906338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:06 smithi022 ceph-mon[31360]: pgmap v9197: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2605906338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:06 smithi022 ceph-mon[40250]: pgmap v9197: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2605906338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:05:08.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:05:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:05:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:08 smithi122 ceph-mon[32255]: pgmap v9198: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:08 smithi022 ceph-mon[31360]: pgmap v9198: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:08 smithi022 ceph-mon[40250]: pgmap v9198: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:05:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:10 smithi122 ceph-mon[32255]: pgmap v9199: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:10 smithi022 ceph-mon[31360]: pgmap v9199: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:10 smithi022 ceph-mon[40250]: pgmap v9199: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:11.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:11.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:11.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:11.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:12.677 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:05:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:05:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:12 smithi122 ceph-mon[32255]: pgmap v9200: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4278894211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:12 smithi022 ceph-mon[31360]: pgmap v9200: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4278894211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:12 smithi022 ceph-mon[40250]: pgmap v9200: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4278894211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:14 smithi122 ceph-mon[32255]: pgmap v9201: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:14 smithi022 ceph-mon[31360]: pgmap v9201: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:14 smithi022 ceph-mon[40250]: pgmap v9201: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:16 smithi122 ceph-mon[32255]: pgmap v9202: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:16.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:16.945 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:16 smithi022 ceph-mon[40250]: pgmap v9202: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:16.946 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:16 smithi022 ceph-mon[31360]: pgmap v9202: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:17.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:17.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:17.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:17.687 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:05:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4069671284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:17 smithi022 ceph-mon[31360]: pgmap v9203: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4069671284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:17 smithi022 ceph-mon[40250]: pgmap v9203: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4069671284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:17 smithi122 ceph-mon[32255]: pgmap v9203: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:18.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:05:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:05:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:20 smithi122 ceph-mon[32255]: pgmap v9204: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:20] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T23:05:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:20 smithi022 ceph-mon[31360]: pgmap v9204: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:20 smithi022 ceph-mon[40250]: pgmap v9204: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:22.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:22.674 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:05:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:05:22.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:22 smithi022 ceph-mon[40250]: pgmap v9205: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:22.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/421586242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:22.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:22 smithi022 ceph-mon[31360]: pgmap v9205: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:22.676 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/421586242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:22.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:22.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:22.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:22 smithi122 ceph-mon[32255]: pgmap v9205: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/421586242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:24 smithi122 ceph-mon[32255]: pgmap v9206: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:24 smithi022 ceph-mon[31360]: pgmap v9206: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:24 smithi022 ceph-mon[40250]: pgmap v9206: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:26 smithi122 ceph-mon[32255]: pgmap v9207: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:26 smithi022 ceph-mon[31360]: pgmap v9207: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:26 smithi022 ceph-mon[40250]: pgmap v9207: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:27.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:27.683 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:05:28.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:28.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:28.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:28.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:05:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:05:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:28 smithi122 ceph-mon[32255]: pgmap v9208: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4038136650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:28 smithi022 ceph-mon[31360]: pgmap v9208: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4038136650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:28 smithi022 ceph-mon[40250]: pgmap v9208: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4038136650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:30] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T23:05:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:30 smithi122 ceph-mon[32255]: pgmap v9209: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:30 smithi022 ceph-mon[31360]: pgmap v9209: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:30 smithi022 ceph-mon[40250]: pgmap v9209: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:31 smithi022 ceph-mon[31360]: pgmap v9210: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:31 smithi022 ceph-mon[40250]: pgmap v9210: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:31 smithi122 ceph-mon[32255]: pgmap v9210: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:05:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:05:33.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:05:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:05:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:05:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:05:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:05:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:05:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:05:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:05:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:05:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:05:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:05:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:05:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1059699665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:34 smithi122 ceph-mon[32255]: pgmap v9211: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1059699665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:34 smithi022 ceph-mon[31360]: pgmap v9211: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1059699665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:34 smithi022 ceph-mon[40250]: pgmap v9211: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:36 smithi122 ceph-mon[32255]: pgmap v9212: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:36 smithi022 ceph-mon[31360]: pgmap v9212: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:36 smithi022 ceph-mon[40250]: pgmap v9212: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:05:38.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:38.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:38.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:38.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:38 smithi122 ceph-mon[32255]: pgmap v9213: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:38.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:05:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:05:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:38 smithi022 ceph-mon[31360]: pgmap v9213: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:38 smithi022 ceph-mon[40250]: pgmap v9213: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1447641699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1447641699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1447641699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:40] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:05:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:40 smithi122 ceph-mon[32255]: pgmap v9214: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:40 smithi022 ceph-mon[31360]: pgmap v9214: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:40 smithi022 ceph-mon[40250]: pgmap v9214: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:05:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:05:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:42 smithi022 ceph-mon[31360]: pgmap v9215: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:42 smithi022 ceph-mon[40250]: pgmap v9215: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:42 smithi122 ceph-mon[32255]: pgmap v9215: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:43.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:43 smithi022 ceph-mon[31360]: pgmap v9216: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:43 smithi022 ceph-mon[40250]: pgmap v9216: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:44.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:44.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:44.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:43 smithi122 ceph-mon[32255]: pgmap v9216: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2746125978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2746125978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2746125978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:45 smithi022 ceph-mon[31360]: pgmap v9217: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:45 smithi022 ceph-mon[40250]: pgmap v9217: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:45 smithi122 ceph-mon[32255]: pgmap v9217: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:05:48.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:05:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:05:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:48 smithi122 ceph-mon[32255]: pgmap v9218: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:48 smithi022 ceph-mon[31360]: pgmap v9218: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:48 smithi022 ceph-mon[40250]: pgmap v9218: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:49.160 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:49.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:49.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:49.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:49.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3601580364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3601580364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:05:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3601580364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:50] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T23:05:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:50 smithi122 ceph-mon[32255]: pgmap v9219: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:50 smithi022 ceph-mon[31360]: pgmap v9219: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:50 smithi022 ceph-mon[40250]: pgmap v9219: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:05:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:05:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:52 smithi022 ceph-mon[31360]: pgmap v9220: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:52 smithi022 ceph-mon[40250]: pgmap v9220: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:52 smithi122 ceph-mon[32255]: pgmap v9220: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:53 smithi022 ceph-mon[31360]: pgmap v9221: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:53 smithi022 ceph-mon[40250]: pgmap v9221: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:53 smithi122 ceph-mon[32255]: pgmap v9221: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:05:54.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:54.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:05:54.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:05:54.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:05:55.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3288826943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3288826943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3288826943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:05:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:56 smithi122 ceph-mon[32255]: pgmap v9222: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:56 smithi022 ceph-mon[31360]: pgmap v9222: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:56 smithi022 ceph-mon[40250]: pgmap v9222: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:05:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:58 smithi122 ceph-mon[32255]: pgmap v9223: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:58.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:05:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:05:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:58 smithi022 ceph-mon[31360]: pgmap v9223: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:58 smithi022 ceph-mon[40250]: pgmap v9223: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:05:59.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:05:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:05:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:05:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:05:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:00.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:00.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:00.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:00] "GET /metrics HTTP/1.1" 200 202377 "" "Prometheus/2.18.1" 2021-12-15T23:06:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:00 smithi022 ceph-mon[40250]: pgmap v9224: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/965061101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:00 smithi022 ceph-mon[31360]: pgmap v9224: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/965061101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:00 smithi122 ceph-mon[32255]: pgmap v9224: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/965061101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:06:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:06:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:02 smithi022 ceph-mon[31360]: pgmap v9225: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:02 smithi022 ceph-mon[40250]: pgmap v9225: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:02 smithi122 ceph-mon[32255]: pgmap v9225: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:06:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:06:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:06:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:03 smithi122 ceph-mon[32255]: pgmap v9226: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:06:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:06:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:06:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[31360]: pgmap v9226: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:06:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:06:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:06:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:03 smithi022 ceph-mon[40250]: pgmap v9226: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:05.269 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:06:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:06:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:06:05.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:05.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:05.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3704686939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:06 smithi122 ceph-mon[32255]: pgmap v9227: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3704686939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:06 smithi022 ceph-mon[31360]: pgmap v9227: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3704686939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:06 smithi022 ceph-mon[40250]: pgmap v9227: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:06:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:08 smithi122 ceph-mon[32255]: pgmap v9228: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:08.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:06:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:06:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:08 smithi022 ceph-mon[31360]: pgmap v9228: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:08 smithi022 ceph-mon[40250]: pgmap v9228: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:10] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T23:06:10.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:11.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:11.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:11.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:10 smithi022 ceph-mon[40250]: pgmap v9229: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:11.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:10 smithi022 ceph-mon[31360]: pgmap v9229: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:10 smithi122 ceph-mon[32255]: pgmap v9229: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/114135881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:11 smithi022 ceph-mon[31360]: pgmap v9230: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/114135881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:12.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:11 smithi022 ceph-mon[40250]: pgmap v9230: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/114135881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:11 smithi122 ceph-mon[32255]: pgmap v9230: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:06:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:06:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:14 smithi022 ceph-mon[31360]: pgmap v9231: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:14 smithi022 ceph-mon[40250]: pgmap v9231: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:14 smithi122 ceph-mon[32255]: pgmap v9231: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:16.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:16.018 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:15 smithi022 ceph-mon[31360]: pgmap v9232: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:16.018 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:15 smithi022 ceph-mon[40250]: pgmap v9232: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:15 smithi122 ceph-mon[32255]: pgmap v9232: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:16.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:16.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:16.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1002043698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1002043698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1002043698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:17.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:06:17.702 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:17 smithi022 ceph-mon[40250]: pgmap v9233: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:17 smithi022 ceph-mon[31360]: pgmap v9233: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:17 smithi122 ceph-mon[32255]: pgmap v9233: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:18.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:06:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:06:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:20.433 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:20] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T23:06:20.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:20 smithi122 ceph-mon[32255]: pgmap v9234: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:20 smithi022 ceph-mon[31360]: pgmap v9234: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:20 smithi022 ceph-mon[40250]: pgmap v9234: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:21.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:21.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:21.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:21.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:06:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:06:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:22 smithi022 ceph-mon[31360]: pgmap v9235: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:23.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3844493835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:22 smithi022 ceph-mon[40250]: pgmap v9235: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3844493835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:22 smithi122 ceph-mon[32255]: pgmap v9235: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3844493835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:23 smithi022 ceph-mon[31360]: pgmap v9236: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:23 smithi022 ceph-mon[40250]: pgmap v9236: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:23 smithi122 ceph-mon[32255]: pgmap v9236: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:25 smithi022 ceph-mon[31360]: pgmap v9237: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:25 smithi022 ceph-mon[40250]: pgmap v9237: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:25 smithi122 ceph-mon[32255]: pgmap v9237: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:26.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:27.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:27.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:27.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1372678930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1372678930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1372678930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:06:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:28 smithi122 ceph-mon[32255]: pgmap v9238: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:28 smithi022 ceph-mon[31360]: pgmap v9238: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:28 smithi022 ceph-mon[40250]: pgmap v9238: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:28.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:06:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:06:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:30.433 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:30] "GET /metrics HTTP/1.1" 200 202373 "" "Prometheus/2.18.1" 2021-12-15T23:06:30.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:30 smithi122 ceph-mon[32255]: pgmap v9239: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:30 smithi022 ceph-mon[31360]: pgmap v9239: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:30 smithi022 ceph-mon[40250]: pgmap v9239: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:31 smithi022 ceph-mon[31360]: pgmap v9240: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:31 smithi022 ceph-mon[40250]: pgmap v9240: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:32.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:31 smithi122 ceph-mon[32255]: pgmap v9240: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:32.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:32.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:32.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:06:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:06:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4151932855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4151932855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4151932855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:06:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:06:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:06:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:06:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[31360]: pgmap v9241: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:06:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:06:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:06:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:06:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:33 smithi022 ceph-mon[40250]: pgmap v9241: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:06:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:06:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:33 smithi122 ceph-mon[32255]: pgmap v9241: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:35 smithi022 ceph-mon[31360]: pgmap v9242: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:35 smithi022 ceph-mon[40250]: pgmap v9242: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:35 smithi122 ceph-mon[32255]: pgmap v9242: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:37.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:37.745 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:06:37.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:37.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:37.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:37 smithi022 ceph-mon[31360]: pgmap v9243: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:38.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:37 smithi022 ceph-mon[40250]: pgmap v9243: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:37 smithi122 ceph-mon[32255]: pgmap v9243: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:38.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:06:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:06:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1367079216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1367079216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1367079216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:40] "GET /metrics HTTP/1.1" 200 202374 "" "Prometheus/2.18.1" 2021-12-15T23:06:40.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:40 smithi122 ceph-mon[32255]: pgmap v9244: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:40 smithi022 ceph-mon[31360]: pgmap v9244: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:40 smithi022 ceph-mon[40250]: pgmap v9244: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:41 smithi022 ceph-mon[31360]: pgmap v9245: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:41 smithi022 ceph-mon[40250]: pgmap v9245: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:41 smithi122 ceph-mon[32255]: pgmap v9245: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:06:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:06:42.878 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:43.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:43.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:43.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2618815551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2618815551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2618815551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:44 smithi122 ceph-mon[32255]: pgmap v9246: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:44 smithi022 ceph-mon[31360]: pgmap v9246: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:44 smithi022 ceph-mon[40250]: pgmap v9246: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:45 smithi022 ceph-mon[31360]: pgmap v9247: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:45 smithi022 ceph-mon[40250]: pgmap v9247: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:45 smithi122 ceph-mon[32255]: pgmap v9247: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:47.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:06:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:47 smithi022 ceph-mon[31360]: pgmap v9248: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:47 smithi022 ceph-mon[40250]: pgmap v9248: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:47 smithi122 ceph-mon[32255]: pgmap v9248: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:48.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:48.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:48.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:48.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:48.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:06:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:06:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/306567274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/306567274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/306567274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:50] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:06:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:50 smithi022 ceph-mon[31360]: pgmap v9249: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:50 smithi022 ceph-mon[40250]: pgmap v9249: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:50 smithi122 ceph-mon[32255]: pgmap v9249: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:51 smithi022 ceph-mon[31360]: pgmap v9250: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:51 smithi022 ceph-mon[40250]: pgmap v9250: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:51 smithi122 ceph-mon[32255]: pgmap v9250: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:06:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:06:53.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:53.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:53.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:53.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:06:53.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:53 smithi022 ceph-mon[31360]: pgmap v9251: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:53.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:53 smithi022 ceph-mon[40250]: pgmap v9251: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:53 smithi122 ceph-mon[32255]: pgmap v9251: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:06:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2355526249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2355526249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2355526249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:06:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:55 smithi022 ceph-mon[31360]: pgmap v9252: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:55 smithi022 ceph-mon[40250]: pgmap v9252: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:55 smithi122 ceph-mon[32255]: pgmap v9252: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:57.748 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:06:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:57 smithi022 ceph-mon[31360]: pgmap v9253: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:57 smithi022 ceph-mon[40250]: pgmap v9253: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:57 smithi122 ceph-mon[32255]: pgmap v9253: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:06:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:58.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:06:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:06:58.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:06:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:06:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:06:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:06:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2217211056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:06:59 smithi022 ceph-mon[40250]: pgmap v9254: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2217211056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:06:59 smithi022 ceph-mon[31360]: pgmap v9254: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2217211056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:06:59 smithi122 ceph-mon[32255]: pgmap v9254: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:00] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:07:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:01 smithi022 ceph-mon[31360]: pgmap v9255: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:01 smithi022 ceph-mon[40250]: pgmap v9255: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:01 smithi122 ceph-mon[32255]: pgmap v9255: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:07:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:07:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:03 smithi022 ceph-mon[31360]: pgmap v9256: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:03 smithi022 ceph-mon[40250]: pgmap v9256: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:04.107 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:03 smithi122 ceph-mon[32255]: pgmap v9256: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:04.360 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:04.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:04.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:04.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:07:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:07:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:07:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:07:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3896496142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:07:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:07:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:07:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:07:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3896496142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:07:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:07:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:07:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:07:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3896496142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:05 smithi022 ceph-mon[31360]: pgmap v9257: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:05 smithi022 ceph-mon[40250]: pgmap v9257: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:05 smithi122 ceph-mon[32255]: pgmap v9257: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:07.750 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:07:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:07 smithi022 ceph-mon[31360]: pgmap v9258: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:07 smithi022 ceph-mon[40250]: pgmap v9258: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:07 smithi122 ceph-mon[32255]: pgmap v9258: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:07:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:07:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:09.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:10.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:10.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:10.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:10] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T23:07:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:10 smithi022 ceph-mon[31360]: pgmap v9259: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3717154145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:10 smithi022 ceph-mon[40250]: pgmap v9259: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3717154145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:10 smithi122 ceph-mon[32255]: pgmap v9259: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3717154145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:11 smithi022 ceph-mon[31360]: pgmap v9260: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:11 smithi022 ceph-mon[40250]: pgmap v9260: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:11 smithi122 ceph-mon[32255]: pgmap v9260: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:07:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:07:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:13 smithi022 ceph-mon[31360]: pgmap v9261: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:13 smithi022 ceph-mon[40250]: pgmap v9261: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:13 smithi122 ceph-mon[32255]: pgmap v9261: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:15.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1012989124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1012989124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1012989124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:16 smithi022 ceph-mon[31360]: pgmap v9262: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:16 smithi022 ceph-mon[40250]: pgmap v9262: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:16 smithi122 ceph-mon[32255]: pgmap v9262: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:17.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:07:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:17 smithi022 ceph-mon[31360]: pgmap v9263: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:17 smithi022 ceph-mon[40250]: pgmap v9263: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:17 smithi122 ceph-mon[32255]: pgmap v9263: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:19.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:07:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:07:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:20] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:07:20.468 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:20 smithi022 ceph-mon[31360]: pgmap v9264: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:20 smithi022 ceph-mon[40250]: pgmap v9264: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:20.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:20.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:20.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:20 smithi122 ceph-mon[32255]: pgmap v9264: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3298699256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3298699256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3298699256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:22.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:22 smithi122 ceph-mon[32255]: pgmap v9265: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:07:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:07:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:22 smithi022 ceph-mon[31360]: pgmap v9265: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:22 smithi022 ceph-mon[40250]: pgmap v9265: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:23 smithi022 ceph-mon[31360]: pgmap v9266: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:23 smithi022 ceph-mon[40250]: pgmap v9266: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:23 smithi122 ceph-mon[32255]: pgmap v9266: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:25 smithi022 ceph-mon[31360]: pgmap v9267: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:25 smithi022 ceph-mon[40250]: pgmap v9267: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:25 smithi122 ceph-mon[32255]: pgmap v9267: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:26.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:26.209 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:26.209 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1070738303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1070738303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1070738303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:27.760 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:07:27.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:27 smithi022 ceph-mon[31360]: pgmap v9268: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:27 smithi022 ceph-mon[40250]: pgmap v9268: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:27 smithi122 ceph-mon[32255]: pgmap v9268: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:07:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:07:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:30] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:07:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:30 smithi022 ceph-mon[31360]: pgmap v9269: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:30 smithi022 ceph-mon[40250]: pgmap v9269: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:30 smithi122 ceph-mon[32255]: pgmap v9269: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:31.210 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:31.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:31.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:31.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2727305073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2727305073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2727305073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:32.623 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:07:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:07:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:32 smithi122 ceph-mon[32255]: pgmap v9270: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:32 smithi022 ceph-mon[31360]: pgmap v9270: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:32 smithi022 ceph-mon[40250]: pgmap v9270: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:07:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:07:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:07:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:07:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:07:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:07:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:07:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:07:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:07:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:07:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:07:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:07:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:34 smithi122 ceph-mon[32255]: pgmap v9271: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:34 smithi022 ceph-mon[31360]: pgmap v9271: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:34 smithi022 ceph-mon[40250]: pgmap v9271: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:35 smithi022 ceph-mon[31360]: pgmap v9272: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:35 smithi022 ceph-mon[40250]: pgmap v9272: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:35 smithi122 ceph-mon[32255]: pgmap v9272: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:36.577 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:36.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:36.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:36.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2211658478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2211658478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2211658478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:07:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:37 smithi022 ceph-mon[31360]: pgmap v9273: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:37 smithi022 ceph-mon[40250]: pgmap v9273: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:37 smithi122 ceph-mon[32255]: pgmap v9273: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:07:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:07:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:40] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T23:07:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:40 smithi022 ceph-mon[31360]: pgmap v9274: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:40 smithi022 ceph-mon[40250]: pgmap v9274: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:40 smithi122 ceph-mon[32255]: pgmap v9274: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:41.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:41 smithi022 ceph-mon[31360]: pgmap v9275: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:41 smithi022 ceph-mon[40250]: pgmap v9275: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:41 smithi122 ceph-mon[32255]: pgmap v9275: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:42.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:42.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:42.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:07:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:07:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3464009031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3464009031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3464009031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:43 smithi022 ceph-mon[31360]: pgmap v9276: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:43 smithi022 ceph-mon[40250]: pgmap v9276: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:43 smithi122 ceph-mon[32255]: pgmap v9276: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:45 smithi022 ceph-mon[31360]: pgmap v9277: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:45 smithi022 ceph-mon[40250]: pgmap v9277: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:45 smithi122 ceph-mon[32255]: pgmap v9277: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:47.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:47.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:47.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:47.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:47.685 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:07:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:47 smithi022 ceph-mon[31360]: pgmap v9278: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/838675564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:47 smithi022 ceph-mon[40250]: pgmap v9278: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/838675564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:47 smithi122 ceph-mon[32255]: pgmap v9278: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/838675564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:07:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:07:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:50] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T23:07:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:50 smithi022 ceph-mon[31360]: pgmap v9279: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:50 smithi022 ceph-mon[40250]: pgmap v9279: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:07:50.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:50 smithi122 ceph-mon[32255]: pgmap v9279: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:51 smithi022 ceph-mon[31360]: pgmap v9280: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:51 smithi022 ceph-mon[40250]: pgmap v9280: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:51 smithi122 ceph-mon[32255]: pgmap v9280: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:52.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:07:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:07:53.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:53.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:53.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3029580642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3029580642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3029580642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:54 smithi122 ceph-mon[32255]: pgmap v9281: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:54 smithi022 ceph-mon[31360]: pgmap v9281: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:54 smithi022 ceph-mon[40250]: pgmap v9281: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:07:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:55 smithi022 ceph-mon[31360]: pgmap v9282: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:55 smithi022 ceph-mon[40250]: pgmap v9282: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:55 smithi122 ceph-mon[32255]: pgmap v9282: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:57.759 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:07:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:57 smithi022 ceph-mon[31360]: pgmap v9283: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:57 smithi022 ceph-mon[40250]: pgmap v9283: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:58.056 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:07:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:57 smithi122 ceph-mon[32255]: pgmap v9283: 132 pgs: 132 active+clean; 582 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:07:58.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:07:58.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:07:58.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:07:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:07:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/218147102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:07:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/218147102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:07:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/218147102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:07:59.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:07:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:08:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:00] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T23:08:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:00 smithi022 ceph-mon[31360]: pgmap v9284: 132 pgs: 132 active+clean; 582 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:08:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:00 smithi022 ceph-mon[40250]: pgmap v9284: 132 pgs: 132 active+clean; 582 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:08:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:00 smithi122 ceph-mon[32255]: pgmap v9284: 132 pgs: 132 active+clean; 582 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:08:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:01 smithi022 ceph-mon[31360]: pgmap v9285: 132 pgs: 132 active+clean; 582 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:08:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:01 smithi022 ceph-mon[40250]: pgmap v9285: 132 pgs: 132 active+clean; 582 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:08:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:01 smithi122 ceph-mon[32255]: pgmap v9285: 132 pgs: 132 active+clean; 582 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:08:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:08:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:08:03.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:03.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:03.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:03.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:03 smithi022 ceph-mon[31360]: pgmap v9286: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:03 smithi022 ceph-mon[40250]: pgmap v9286: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:03 smithi122 ceph-mon[32255]: pgmap v9286: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1529809567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:08:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:08:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:08:04.984 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1529809567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:04.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:08:04.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:08:04.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:08:04.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1529809567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:04.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:08:04.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:08:04.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:08:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:08:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:05 smithi022 ceph-mon[31360]: pgmap v9287: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:08:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:05 smithi022 ceph-mon[40250]: pgmap v9287: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:08:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:05 smithi122 ceph-mon[32255]: pgmap v9287: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:07.761 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:08:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:07 smithi022 ceph-mon[31360]: pgmap v9288: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:07 smithi022 ceph-mon[40250]: pgmap v9288: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:07 smithi122 ceph-mon[32255]: pgmap v9288: 132 pgs: 132 active+clean; 582 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:08.803 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:09.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:09.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:09.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:08:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:08:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4013508963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4013508963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4013508963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:10] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:08:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:10 smithi022 ceph-mon[31360]: pgmap v9289: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:10 smithi022 ceph-mon[40250]: pgmap v9289: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:10 smithi122 ceph-mon[32255]: pgmap v9289: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:08:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:11 smithi022 ceph-mon[31360]: pgmap v9290: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:08:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:11 smithi022 ceph-mon[40250]: pgmap v9290: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:08:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:11 smithi122 ceph-mon[32255]: pgmap v9290: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:08:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:08:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:08:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:13 smithi022 ceph-mon[31360]: pgmap v9291: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:08:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:13 smithi022 ceph-mon[40250]: pgmap v9291: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:08:14.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:13 smithi122 ceph-mon[32255]: pgmap v9291: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:08:14.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:14.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:14.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4243567524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4243567524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4243567524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:16 smithi022 ceph-mon[40250]: pgmap v9292: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:16 smithi022 ceph-mon[31360]: pgmap v9292: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:16 smithi122 ceph-mon[32255]: pgmap v9292: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:17.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:08:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:17 smithi022 ceph-mon[31360]: pgmap v9293: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:17 smithi022 ceph-mon[40250]: pgmap v9293: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:17 smithi122 ceph-mon[32255]: pgmap v9293: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:19.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:08:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:08:19.545 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:19.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:19.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:19.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:08:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:20 smithi022 ceph-mon[31360]: pgmap v9294: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4174526498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:20 smithi022 ceph-mon[40250]: pgmap v9294: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4174526498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:20 smithi122 ceph-mon[32255]: pgmap v9294: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4174526498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:21 smithi022 ceph-mon[31360]: pgmap v9295: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:21 smithi022 ceph-mon[40250]: pgmap v9295: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:21 smithi122 ceph-mon[32255]: pgmap v9295: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:08:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:08:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:23 smithi022 ceph-mon[31360]: pgmap v9296: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:23 smithi022 ceph-mon[40250]: pgmap v9296: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:23 smithi122 ceph-mon[32255]: pgmap v9296: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:24.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:25.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:25.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:25.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2645714104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2645714104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2645714104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:26 smithi022 ceph-mon[31360]: pgmap v9297: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:26 smithi022 ceph-mon[40250]: pgmap v9297: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:26 smithi122 ceph-mon[32255]: pgmap v9297: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:27.764 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:08:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:27 smithi022 ceph-mon[31360]: pgmap v9298: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:27 smithi022 ceph-mon[40250]: pgmap v9298: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:27 smithi122 ceph-mon[32255]: pgmap v9298: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:08:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:08:30.292 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:08:30.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:30.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:30.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:30.662 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:30.662 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:30 smithi022 ceph-mon[31360]: pgmap v9299: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:30.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:30.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:30 smithi022 ceph-mon[40250]: pgmap v9299: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:30 smithi122 ceph-mon[32255]: pgmap v9299: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2788288470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2788288470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2788288470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:32 smithi122 ceph-mon[32255]: pgmap v9300: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:08:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:08:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:32 smithi022 ceph-mon[31360]: pgmap v9300: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:32 smithi022 ceph-mon[40250]: pgmap v9300: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:08:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:08:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:08:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:08:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:08:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:08:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:08:33.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:08:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:08:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:08:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:08:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:08:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:34 smithi022 ceph-mon[31360]: pgmap v9301: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:34 smithi022 ceph-mon[40250]: pgmap v9301: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:34 smithi122 ceph-mon[32255]: pgmap v9301: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:35.663 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:35 smithi022 ceph-mon[31360]: pgmap v9302: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:35 smithi022 ceph-mon[40250]: pgmap v9302: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:36.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:36.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:36.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:35 smithi122 ceph-mon[32255]: pgmap v9302: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3250309196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3250309196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3250309196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:37.771 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:08:37.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:37 smithi022 ceph-mon[40250]: pgmap v9303: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:37.772 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:37 smithi022 ceph-mon[31360]: pgmap v9303: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:37 smithi122 ceph-mon[32255]: pgmap v9303: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:08:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:08:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:40] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:08:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:40 smithi022 ceph-mon[31360]: pgmap v9304: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:40 smithi022 ceph-mon[40250]: pgmap v9304: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:40 smithi122 ceph-mon[32255]: pgmap v9304: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:41.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:41.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:41.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:41.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1316993625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1316993625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1316993625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:42 smithi122 ceph-mon[32255]: pgmap v9305: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:08:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:08:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:42 smithi022 ceph-mon[31360]: pgmap v9305: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:42 smithi022 ceph-mon[40250]: pgmap v9305: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:43 smithi022 ceph-mon[31360]: pgmap v9306: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:43 smithi022 ceph-mon[40250]: pgmap v9306: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:43 smithi122 ceph-mon[32255]: pgmap v9306: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:45 smithi022 ceph-mon[31360]: pgmap v9307: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:45 smithi022 ceph-mon[40250]: pgmap v9307: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:45 smithi122 ceph-mon[32255]: pgmap v9307: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:46.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:46.769 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3663737251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:46.769 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3663737251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3663737251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:47.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:08:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:47 smithi022 ceph-mon[31360]: pgmap v9308: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:47 smithi022 ceph-mon[40250]: pgmap v9308: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:47 smithi122 ceph-mon[32255]: pgmap v9308: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:08:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:08:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:08:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:50] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:08:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:50 smithi022 ceph-mon[31360]: pgmap v9309: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:50 smithi022 ceph-mon[40250]: pgmap v9309: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:08:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:50 smithi122 ceph-mon[32255]: pgmap v9309: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:51.770 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:51 smithi022 ceph-mon[31360]: pgmap v9310: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:51 smithi022 ceph-mon[40250]: pgmap v9310: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:52.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:52.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:52.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:51 smithi122 ceph-mon[32255]: pgmap v9310: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:08:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:08:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/252838757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/252838757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/252838757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:53 smithi022 ceph-mon[31360]: pgmap v9311: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:53 smithi022 ceph-mon[40250]: pgmap v9311: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:53 smithi122 ceph-mon[32255]: pgmap v9311: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:08:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:55 smithi022 ceph-mon[31360]: pgmap v9312: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:55 smithi022 ceph-mon[40250]: pgmap v9312: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:55 smithi122 ceph-mon[32255]: pgmap v9312: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:57.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:08:57.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:08:57.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:08:57.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:08:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/264558164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:08:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/264558164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/264558164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:08:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:08:58 smithi122 ceph-mon[32255]: pgmap v9313: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:58.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:08:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:08:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:08:58 smithi022 ceph-mon[31360]: pgmap v9313: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:08:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:08:58 smithi022 ceph-mon[40250]: pgmap v9313: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:09:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:00] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:09:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:00 smithi022 ceph-mon[31360]: pgmap v9314: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:00 smithi022 ceph-mon[40250]: pgmap v9314: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:00 smithi122 ceph-mon[32255]: pgmap v9314: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:01 smithi022 ceph-mon[31360]: pgmap v9315: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:01 smithi022 ceph-mon[40250]: pgmap v9315: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:01 smithi122 ceph-mon[32255]: pgmap v9315: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:02.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:09:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:09:02.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:02.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:02.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/746535081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/746535081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/746535081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:03 smithi122 ceph-mon[32255]: pgmap v9316: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:03 smithi022 ceph-mon[31360]: pgmap v9316: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:03 smithi022 ceph-mon[40250]: pgmap v9316: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:09:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:09:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:09:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:09:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:09:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:09:05.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:09:05.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:09:05.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:09:05.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:09:05.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:09:05.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:09:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:06 smithi022 ceph-mon[31360]: pgmap v9317: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:06 smithi022 ceph-mon[40250]: pgmap v9317: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:06 smithi122 ceph-mon[32255]: pgmap v9317: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:07.772 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:09:07.878 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:07 smithi022 ceph-mon[31360]: pgmap v9318: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:07 smithi022 ceph-mon[40250]: pgmap v9318: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:07 smithi122 ceph-mon[32255]: pgmap v9318: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:08.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:08.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:08.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3215378397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3215378397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3215378397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:09.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:09:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:09:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:10] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:09:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:10 smithi022 ceph-mon[31360]: pgmap v9319: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:10 smithi022 ceph-mon[40250]: pgmap v9319: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:10 smithi122 ceph-mon[32255]: pgmap v9319: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:11 smithi022 ceph-mon[31360]: pgmap v9320: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:11 smithi022 ceph-mon[40250]: pgmap v9320: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:11 smithi122 ceph-mon[32255]: pgmap v9320: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:09:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:09:13.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:13.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:13.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:13.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1123738616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1123738616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1123738616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:14 smithi122 ceph-mon[32255]: pgmap v9321: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:14 smithi022 ceph-mon[40250]: pgmap v9321: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:14 smithi022 ceph-mon[31360]: pgmap v9321: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:15 smithi022 ceph-mon[31360]: pgmap v9322: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:15 smithi022 ceph-mon[40250]: pgmap v9322: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:15 smithi122 ceph-mon[32255]: pgmap v9322: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:17.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:09:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:17 smithi022 ceph-mon[31360]: pgmap v9323: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:17 smithi022 ceph-mon[40250]: pgmap v9323: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:17 smithi122 ceph-mon[32255]: pgmap v9323: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:18.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:18.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3968604617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:19.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:09:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:09:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3968604617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3968604617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:20] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:09:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:20 smithi022 ceph-mon[31360]: pgmap v9324: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:20 smithi022 ceph-mon[40250]: pgmap v9324: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:20 smithi122 ceph-mon[32255]: pgmap v9324: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:21 smithi022 ceph-mon[31360]: pgmap v9325: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:21 smithi022 ceph-mon[40250]: pgmap v9325: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:21 smithi122 ceph-mon[32255]: pgmap v9325: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:09:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:09:23.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:23 smithi022 ceph-mon[31360]: pgmap v9326: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:23 smithi022 ceph-mon[40250]: pgmap v9326: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:23 smithi122 ceph-mon[32255]: pgmap v9326: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:24.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:24.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:24.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1585405128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1585405128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1585405128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:25 smithi122 ceph-mon[32255]: pgmap v9327: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:25 smithi022 ceph-mon[31360]: pgmap v9327: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:25 smithi022 ceph-mon[40250]: pgmap v9327: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:27.775 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:09:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:27 smithi022 ceph-mon[31360]: pgmap v9328: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:27 smithi022 ceph-mon[40250]: pgmap v9328: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:27 smithi122 ceph-mon[32255]: pgmap v9328: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:09:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:09:29.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:29.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:29.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:29.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:30] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:09:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:30 smithi022 ceph-mon[31360]: pgmap v9329: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3447684871' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:30 smithi022 ceph-mon[40250]: pgmap v9329: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3447684871' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:30 smithi122 ceph-mon[32255]: pgmap v9329: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3447684871' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:31 smithi022 ceph-mon[31360]: pgmap v9330: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:31 smithi022 ceph-mon[40250]: pgmap v9330: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:31 smithi122 ceph-mon[32255]: pgmap v9330: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:09:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:09:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:09:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:09:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:09:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:09:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:09:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:09:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:09:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:09:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:09:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:09:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:09:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:09:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:34 smithi122 ceph-mon[32255]: pgmap v9331: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:34.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:34.724 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:34 smithi022 ceph-mon[31360]: pgmap v9331: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:34.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:34 smithi022 ceph-mon[40250]: pgmap v9331: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:35.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:35.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:35.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3865449274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3865449274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3865449274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:36 smithi022 ceph-mon[31360]: pgmap v9332: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:36 smithi022 ceph-mon[40250]: pgmap v9332: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:36 smithi122 ceph-mon[32255]: pgmap v9332: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:37.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:09:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:37 smithi022 ceph-mon[31360]: pgmap v9333: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:37 smithi022 ceph-mon[40250]: pgmap v9333: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:37 smithi122 ceph-mon[32255]: pgmap v9333: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:09:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:09:40.095 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:09:40.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:40.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:40.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:40 smithi022 ceph-mon[31360]: pgmap v9334: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1014991202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:40 smithi022 ceph-mon[40250]: pgmap v9334: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1014991202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:40 smithi122 ceph-mon[32255]: pgmap v9334: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1014991202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:41 smithi022 ceph-mon[31360]: pgmap v9335: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:41 smithi022 ceph-mon[40250]: pgmap v9335: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:41 smithi122 ceph-mon[32255]: pgmap v9335: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:09:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:09:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:43 smithi022 ceph-mon[31360]: pgmap v9336: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:43 smithi022 ceph-mon[40250]: pgmap v9336: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:43 smithi122 ceph-mon[32255]: pgmap v9336: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:45.471 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:45.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:45.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:45.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:45 smithi022 ceph-mon[31360]: pgmap v9337: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:45 smithi022 ceph-mon[40250]: pgmap v9337: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:45 smithi122 ceph-mon[32255]: pgmap v9337: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3713434595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3713434595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3713434595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:09:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:47 smithi122 ceph-mon[32255]: pgmap v9338: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:47 smithi022 ceph-mon[31360]: pgmap v9338: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:47 smithi022 ceph-mon[40250]: pgmap v9338: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:09:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:09:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:50] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:09:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:50 smithi022 ceph-mon[31360]: pgmap v9339: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:50.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:50 smithi022 ceph-mon[40250]: pgmap v9339: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:50.843 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:09:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:50 smithi122 ceph-mon[32255]: pgmap v9339: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:51.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:51.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:51.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/389173787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/389173787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/389173787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:52 smithi122 ceph-mon[32255]: pgmap v9340: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:09:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:09:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:52 smithi022 ceph-mon[31360]: pgmap v9340: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:52 smithi022 ceph-mon[40250]: pgmap v9340: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:53 smithi122 ceph-mon[32255]: pgmap v9341: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:53 smithi022 ceph-mon[31360]: pgmap v9341: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:53 smithi022 ceph-mon[40250]: pgmap v9341: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:09:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:55 smithi122 ceph-mon[32255]: pgmap v9342: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:56.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:09:56.212 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:55 smithi022 ceph-mon[31360]: pgmap v9342: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:56.213 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:55 smithi022 ceph-mon[40250]: pgmap v9342: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:56.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:09:56.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:09:56.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:09:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/830670523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/830670523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/830670523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:09:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:09:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:09:57 smithi122 ceph-mon[32255]: pgmap v9343: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:09:57 smithi022 ceph-mon[31360]: pgmap v9343: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:09:57 smithi022 ceph-mon[40250]: pgmap v9343: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:09:59.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:09:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:10:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:00] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:10:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:00 smithi022 ceph-mon[31360]: pgmap v9344: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T23:10:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:00 smithi022 ceph-mon[40250]: pgmap v9344: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T23:10:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:00 smithi122 ceph-mon[32255]: pgmap v9344: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T23:10:01.586 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:01.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:01.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:01.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:01.956 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:01 smithi022 ceph-mon[31360]: pgmap v9345: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:01.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:01 smithi022 ceph-mon[40250]: pgmap v9345: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:01 smithi122 ceph-mon[32255]: pgmap v9345: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:10:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:10:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/918656536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/918656536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/918656536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:03 smithi122 ceph-mon[32255]: pgmap v9346: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:03 smithi022 ceph-mon[31360]: pgmap v9346: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:03 smithi022 ceph-mon[40250]: pgmap v9346: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:10:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:10:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:10:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:10:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:10:05.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:10:05.842 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:10:05.843 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:10:05.843 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:10:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:06 smithi122 ceph-mon[32255]: pgmap v9347: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[31360]: pgmap v9347: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[40250]: pgmap v9347: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:10:06.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:07.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:07.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:07.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2299636834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:10:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2299636834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2299636834' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:08 smithi022 ceph-mon[31360]: pgmap v9348: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:08 smithi022 ceph-mon[40250]: pgmap v9348: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:08.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:08 smithi122 ceph-mon[32255]: pgmap v9348: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:10:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:10:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:10] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:10:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:10 smithi022 ceph-mon[31360]: pgmap v9349: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:10 smithi022 ceph-mon[40250]: pgmap v9349: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:10 smithi122 ceph-mon[32255]: pgmap v9349: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:11 smithi122 ceph-mon[32255]: pgmap v9350: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:11 smithi022 ceph-mon[31360]: pgmap v9350: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:11 smithi022 ceph-mon[40250]: pgmap v9350: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:12.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:10:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:10:12.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:12.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:12.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1413479235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1413479235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1413479235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:13 smithi122 ceph-mon[32255]: pgmap v9351: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:13 smithi022 ceph-mon[31360]: pgmap v9351: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:13 smithi022 ceph-mon[40250]: pgmap v9351: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:15 smithi122 ceph-mon[32255]: pgmap v9352: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:15 smithi022 ceph-mon[31360]: pgmap v9352: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:16.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:15 smithi022 ceph-mon[40250]: pgmap v9352: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:17.700 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:10:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:17 smithi022 ceph-mon[31360]: pgmap v9353: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:17 smithi022 ceph-mon[40250]: pgmap v9353: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:18.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:18.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:18.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:17 smithi122 ceph-mon[32255]: pgmap v9353: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/112784597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:19.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:10:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:10:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/112784597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/112784597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:20] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:10:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:20 smithi022 ceph-mon[31360]: pgmap v9354: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:20 smithi022 ceph-mon[40250]: pgmap v9354: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:20 smithi122 ceph-mon[32255]: pgmap v9354: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:21 smithi122 ceph-mon[32255]: pgmap v9355: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:21 smithi022 ceph-mon[31360]: pgmap v9355: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:21 smithi022 ceph-mon[40250]: pgmap v9355: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:10:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:10:23.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:23.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:23.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:23.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2817010505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2817010505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2817010505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:24 smithi022 ceph-mon[31360]: pgmap v9356: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:24 smithi022 ceph-mon[40250]: pgmap v9356: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:24 smithi122 ceph-mon[32255]: pgmap v9356: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:25 smithi122 ceph-mon[32255]: pgmap v9357: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:25 smithi022 ceph-mon[31360]: pgmap v9357: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:25 smithi022 ceph-mon[40250]: pgmap v9357: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:10:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:27 smithi122 ceph-mon[32255]: pgmap v9358: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:27 smithi022 ceph-mon[31360]: pgmap v9358: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:27 smithi022 ceph-mon[40250]: pgmap v9358: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:28.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:28.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:28.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:28.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1500486498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:29.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:10:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:10:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1500486498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:29.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1500486498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:30] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:10:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:30 smithi022 ceph-mon[31360]: pgmap v9359: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:30 smithi022 ceph-mon[40250]: pgmap v9359: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:30 smithi122 ceph-mon[32255]: pgmap v9359: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:31 smithi122 ceph-mon[32255]: pgmap v9360: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:31 smithi022 ceph-mon[31360]: pgmap v9360: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:31 smithi022 ceph-mon[40250]: pgmap v9360: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:10:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:10:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:10:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:10:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:10:33.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:10:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:10:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:10:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:10:33.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:10:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:10:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:10:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:10:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:10:33.812 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:34.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:34.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:34.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:34 smithi022 ceph-mon[31360]: pgmap v9361: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1480261015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:34 smithi022 ceph-mon[40250]: pgmap v9361: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1480261015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:34 smithi122 ceph-mon[32255]: pgmap v9361: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1480261015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:35 smithi122 ceph-mon[32255]: pgmap v9362: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:35 smithi022 ceph-mon[31360]: pgmap v9362: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:35 smithi022 ceph-mon[40250]: pgmap v9362: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:10:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:37 smithi122 ceph-mon[32255]: pgmap v9363: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:37 smithi022 ceph-mon[31360]: pgmap v9363: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:37 smithi022 ceph-mon[40250]: pgmap v9363: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:39.180 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:10:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:10:39.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:39.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:39.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:40] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:10:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/889170480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:40 smithi022 ceph-mon[31360]: pgmap v9364: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/889170480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:40 smithi022 ceph-mon[40250]: pgmap v9364: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/889170480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:40 smithi122 ceph-mon[32255]: pgmap v9364: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:41 smithi122 ceph-mon[32255]: pgmap v9365: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:41 smithi022 ceph-mon[31360]: pgmap v9365: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:41 smithi022 ceph-mon[40250]: pgmap v9365: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:10:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:10:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:43 smithi122 ceph-mon[32255]: pgmap v9366: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:43 smithi022 ceph-mon[31360]: pgmap v9366: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:43 smithi022 ceph-mon[40250]: pgmap v9366: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:44.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:44.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:44.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:44.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:45.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2911713512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:45.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2911713512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2911713512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:46 smithi022 ceph-mon[31360]: pgmap v9367: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:46 smithi022 ceph-mon[40250]: pgmap v9367: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:46 smithi122 ceph-mon[32255]: pgmap v9367: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:10:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:47 smithi122 ceph-mon[32255]: pgmap v9368: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:47 smithi022 ceph-mon[40250]: pgmap v9368: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:47 smithi022 ceph-mon[31360]: pgmap v9368: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:10:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:10:49.918 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:50.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:50.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:50.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:50] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:10:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:50 smithi022 ceph-mon[31360]: pgmap v9369: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2937844253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:50 smithi022 ceph-mon[40250]: pgmap v9369: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:50.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2937844253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:10:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:50 smithi122 ceph-mon[32255]: pgmap v9369: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2937844253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:51 smithi122 ceph-mon[32255]: pgmap v9370: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:51 smithi022 ceph-mon[31360]: pgmap v9370: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:51 smithi022 ceph-mon[40250]: pgmap v9370: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:10:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:10:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:53 smithi122 ceph-mon[32255]: pgmap v9371: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:53 smithi022 ceph-mon[31360]: pgmap v9371: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:53 smithi022 ceph-mon[40250]: pgmap v9371: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:10:55.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:10:55.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:10:55.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:10:55.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:10:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2247790886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2247790886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2247790886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:10:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:56 smithi022 ceph-mon[31360]: pgmap v9372: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:56 smithi022 ceph-mon[40250]: pgmap v9372: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:56 smithi122 ceph-mon[32255]: pgmap v9372: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:10:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:10:57 smithi122 ceph-mon[32255]: pgmap v9373: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:10:57 smithi022 ceph-mon[31360]: pgmap v9373: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:10:57 smithi022 ceph-mon[40250]: pgmap v9373: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:10:59.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:10:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:11:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:00] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:11:00.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:00 smithi022 ceph-mon[31360]: pgmap v9374: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:00 smithi022 ceph-mon[40250]: pgmap v9374: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:00 smithi122 ceph-mon[32255]: pgmap v9374: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:01.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/932459115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/932459115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/932459115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:02 smithi122 ceph-mon[32255]: pgmap v9375: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:11:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:11:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:02 smithi022 ceph-mon[31360]: pgmap v9375: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:02 smithi022 ceph-mon[40250]: pgmap v9375: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:03 smithi122 ceph-mon[32255]: pgmap v9376: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:03 smithi022 ceph-mon[31360]: pgmap v9376: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:03 smithi022 ceph-mon[40250]: pgmap v9376: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:06.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:05 smithi122 ceph-mon[32255]: pgmap v9377: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:05 smithi022 ceph-mon[31360]: pgmap v9377: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:05 smithi022 ceph-mon[40250]: pgmap v9377: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:06.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:06.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:06.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:11:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:11:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:11:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3126587278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:11:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:11:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:11:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:11:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3126587278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:11:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:11:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:11:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:11:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3126587278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:11:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:11:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:07 smithi122 ceph-mon[32255]: pgmap v9378: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:07 smithi022 ceph-mon[31360]: pgmap v9378: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:07 smithi022 ceph-mon[40250]: pgmap v9378: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:09.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:11:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:11:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:10] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:11:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:10 smithi022 ceph-mon[31360]: pgmap v9379: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:10 smithi022 ceph-mon[40250]: pgmap v9379: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:10 smithi122 ceph-mon[32255]: pgmap v9379: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:11.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:11.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:11.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:11 smithi022 ceph-mon[31360]: pgmap v9380: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1081945296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:11 smithi022 ceph-mon[40250]: pgmap v9380: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1081945296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:11 smithi122 ceph-mon[32255]: pgmap v9380: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1081945296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:11:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:11:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:13 smithi122 ceph-mon[32255]: pgmap v9381: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:13 smithi022 ceph-mon[31360]: pgmap v9381: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:13 smithi022 ceph-mon[40250]: pgmap v9381: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:15 smithi122 ceph-mon[32255]: pgmap v9382: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:15 smithi022 ceph-mon[31360]: pgmap v9382: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:15 smithi022 ceph-mon[40250]: pgmap v9382: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:16.769 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:17.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:17.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:17.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/14715893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/14715893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/14715893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:11:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:18 smithi122 ceph-mon[32255]: pgmap v9383: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:18 smithi022 ceph-mon[31360]: pgmap v9383: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:18 smithi022 ceph-mon[40250]: pgmap v9383: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:19.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:11:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:11:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:20] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T23:11:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:20 smithi022 ceph-mon[31360]: pgmap v9384: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:20 smithi022 ceph-mon[40250]: pgmap v9384: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:20 smithi122 ceph-mon[32255]: pgmap v9384: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:22.137 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:22.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:21 smithi022 ceph-mon[31360]: pgmap v9385: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:22.138 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:21 smithi022 ceph-mon[40250]: pgmap v9385: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:21 smithi122 ceph-mon[32255]: pgmap v9385: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:11:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:11:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1211909169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1211909169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1211909169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:23 smithi122 ceph-mon[32255]: pgmap v9386: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:23 smithi022 ceph-mon[31360]: pgmap v9386: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:23 smithi022 ceph-mon[40250]: pgmap v9386: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:25 smithi122 ceph-mon[32255]: pgmap v9387: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:25 smithi022 ceph-mon[31360]: pgmap v9387: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:25 smithi022 ceph-mon[40250]: pgmap v9387: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:27.512 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:11:27.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:27.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:27.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:27 smithi122 ceph-mon[32255]: pgmap v9388: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:27 smithi022 ceph-mon[31360]: pgmap v9388: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:27 smithi022 ceph-mon[40250]: pgmap v9388: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:29.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:11:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:11:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1025328672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1025328672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1025328672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:30] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T23:11:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:30 smithi022 ceph-mon[31360]: pgmap v9389: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:30 smithi022 ceph-mon[40250]: pgmap v9389: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:30 smithi122 ceph-mon[32255]: pgmap v9389: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:31 smithi122 ceph-mon[32255]: pgmap v9390: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:31 smithi022 ceph-mon[31360]: pgmap v9390: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:31 smithi022 ceph-mon[40250]: pgmap v9390: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:11:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:11:32.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:33.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:33.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:33.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3060799145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3060799145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3060799145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:11:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:11:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:11:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:11:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:34 smithi122 ceph-mon[32255]: pgmap v9391: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[40250]: pgmap v9391: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:11:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:11:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:11:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:11:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:34 smithi022 ceph-mon[31360]: pgmap v9391: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:35 smithi122 ceph-mon[32255]: pgmap v9392: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:35 smithi022 ceph-mon[31360]: pgmap v9392: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:35 smithi022 ceph-mon[40250]: pgmap v9392: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:11:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:37 smithi122 ceph-mon[32255]: pgmap v9393: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:38.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:38.255 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:37 smithi022 ceph-mon[40250]: pgmap v9393: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:38.255 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:37 smithi022 ceph-mon[31360]: pgmap v9393: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:38.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:38.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:38.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2256075881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2256075881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2256075881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:39.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:11:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:11:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:11:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:40 smithi022 ceph-mon[31360]: pgmap v9394: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:40 smithi022 ceph-mon[40250]: pgmap v9394: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:40 smithi122 ceph-mon[32255]: pgmap v9394: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:41 smithi122 ceph-mon[32255]: pgmap v9395: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:41 smithi022 ceph-mon[31360]: pgmap v9395: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:41 smithi022 ceph-mon[40250]: pgmap v9395: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:11:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:11:43.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:44.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:44.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:44.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:44.004 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:43 smithi022 ceph-mon[31360]: pgmap v9396: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:44.005 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:43 smithi022 ceph-mon[40250]: pgmap v9396: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:43 smithi122 ceph-mon[32255]: pgmap v9396: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1979846186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1979846186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1979846186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:45 smithi122 ceph-mon[32255]: pgmap v9397: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:45 smithi022 ceph-mon[31360]: pgmap v9397: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:45 smithi022 ceph-mon[40250]: pgmap v9397: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:11:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:47 smithi122 ceph-mon[32255]: pgmap v9398: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:47 smithi022 ceph-mon[31360]: pgmap v9398: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:47 smithi022 ceph-mon[40250]: pgmap v9398: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:49.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:49.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:11:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:11:49.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:49.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:49.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/200397039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/200397039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/200397039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:11:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:50 smithi022 ceph-mon[31360]: pgmap v9399: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:50 smithi022 ceph-mon[40250]: pgmap v9399: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:11:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:50 smithi122 ceph-mon[32255]: pgmap v9399: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:51 smithi122 ceph-mon[32255]: pgmap v9400: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:51 smithi022 ceph-mon[31360]: pgmap v9400: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:51 smithi022 ceph-mon[40250]: pgmap v9400: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:11:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:11:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:53 smithi122 ceph-mon[32255]: pgmap v9401: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:53 smithi022 ceph-mon[31360]: pgmap v9401: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:53 smithi022 ceph-mon[40250]: pgmap v9401: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:11:54.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:11:54.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:11:54.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:11:54.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:11:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1448758514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1448758514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1448758514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:11:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:56 smithi022 ceph-mon[31360]: pgmap v9402: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:56 smithi022 ceph-mon[40250]: pgmap v9402: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:56 smithi122 ceph-mon[32255]: pgmap v9402: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:11:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:11:57 smithi122 ceph-mon[32255]: pgmap v9403: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:11:57 smithi022 ceph-mon[31360]: pgmap v9403: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:11:57 smithi022 ceph-mon[40250]: pgmap v9403: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:11:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:11:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:11:59.743 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:00.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:00.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:00.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:12:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:00 smithi022 ceph-mon[31360]: pgmap v9404: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4133216902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:00 smithi022 ceph-mon[40250]: pgmap v9404: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4133216902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:00 smithi122 ceph-mon[32255]: pgmap v9404: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4133216902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:01 smithi122 ceph-mon[32255]: pgmap v9405: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:01 smithi022 ceph-mon[31360]: pgmap v9405: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:01 smithi022 ceph-mon[40250]: pgmap v9405: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:12:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:12:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:03 smithi122 ceph-mon[32255]: pgmap v9406: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:03 smithi022 ceph-mon[31360]: pgmap v9406: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:03 smithi022 ceph-mon[40250]: pgmap v9406: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:05.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:05.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:05.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4160691062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4160691062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4160691062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:06 smithi022 ceph-mon[31360]: pgmap v9407: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:06 smithi022 ceph-mon[40250]: pgmap v9407: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:06.804 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:06 smithi122 ceph-mon[32255]: pgmap v9407: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:07.509 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:12:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:12:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:12:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:12:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:12:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:12:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:12:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:12:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:12:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:12:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:08 smithi022 ceph-mon[31360]: pgmap v9408: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:08 smithi022 ceph-mon[40250]: pgmap v9408: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:08 smithi122 ceph-mon[32255]: pgmap v9408: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:12:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:12:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:12:10.482 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:10 smithi022 ceph-mon[31360]: pgmap v9409: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:10 smithi022 ceph-mon[40250]: pgmap v9409: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:10.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:10.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:10.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:10 smithi122 ceph-mon[32255]: pgmap v9409: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/841040995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/841040995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/841040995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:12 smithi122 ceph-mon[32255]: pgmap v9410: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:12:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:12:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:12 smithi022 ceph-mon[31360]: pgmap v9410: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:12 smithi022 ceph-mon[40250]: pgmap v9410: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:13 smithi122 ceph-mon[32255]: pgmap v9411: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:13 smithi022 ceph-mon[31360]: pgmap v9411: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:13 smithi022 ceph-mon[40250]: pgmap v9411: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:15 smithi122 ceph-mon[32255]: pgmap v9412: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:16.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:16.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:16.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:16.229 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:15 smithi022 ceph-mon[31360]: pgmap v9412: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:16.229 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:15 smithi022 ceph-mon[40250]: pgmap v9412: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4194082816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4194082816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4194082816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:12:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:17 smithi122 ceph-mon[32255]: pgmap v9413: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:17 smithi022 ceph-mon[31360]: pgmap v9413: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:17 smithi022 ceph-mon[40250]: pgmap v9413: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:12:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:12:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:20 smithi022 ceph-mon[31360]: pgmap v9414: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:20 smithi022 ceph-mon[40250]: pgmap v9414: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:20 smithi122 ceph-mon[32255]: pgmap v9414: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:21.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:21.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:21.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:21.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3733785469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3733785469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3733785469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:22.647 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:12:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:12:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:22 smithi122 ceph-mon[32255]: pgmap v9415: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:22 smithi022 ceph-mon[31360]: pgmap v9415: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:22 smithi022 ceph-mon[40250]: pgmap v9415: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:23 smithi122 ceph-mon[32255]: pgmap v9416: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:23 smithi022 ceph-mon[31360]: pgmap v9416: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:23 smithi022 ceph-mon[40250]: pgmap v9416: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:25 smithi122 ceph-mon[32255]: pgmap v9417: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:25 smithi022 ceph-mon[31360]: pgmap v9417: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:25 smithi022 ceph-mon[40250]: pgmap v9417: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1433785999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1433785999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1433785999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:12:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:28 smithi022 ceph-mon[31360]: pgmap v9418: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:28 smithi022 ceph-mon[40250]: pgmap v9418: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:28 smithi122 ceph-mon[32255]: pgmap v9418: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:12:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:12:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:12:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:30 smithi022 ceph-mon[31360]: pgmap v9419: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:30 smithi022 ceph-mon[40250]: pgmap v9419: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:30 smithi122 ceph-mon[32255]: pgmap v9419: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:31.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:31 smithi122 ceph-mon[32255]: pgmap v9420: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:31 smithi022 ceph-mon[31360]: pgmap v9420: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:31 smithi022 ceph-mon[40250]: pgmap v9420: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:32.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:32.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:12:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:12:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/751784045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/751784045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/751784045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:12:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:12:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:12:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:12:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:33 smithi122 ceph-mon[32255]: pgmap v9421: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:12:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:12:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:12:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:12:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[31360]: pgmap v9421: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:12:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:12:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:12:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:12:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:33 smithi022 ceph-mon[40250]: pgmap v9421: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:35 smithi122 ceph-mon[32255]: pgmap v9422: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:35 smithi022 ceph-mon[31360]: pgmap v9422: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:35 smithi022 ceph-mon[40250]: pgmap v9422: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:37.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:37.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:37.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:37.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:37.713 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:12:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3019281974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3019281974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3019281974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:12:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:12:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:39 smithi022 ceph-mon[31360]: pgmap v9423: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:39 smithi022 ceph-mon[40250]: pgmap v9423: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:39 smithi122 ceph-mon[32255]: pgmap v9423: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:40] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:12:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:41 smithi022 ceph-mon[31360]: pgmap v9424: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:41 smithi022 ceph-mon[40250]: pgmap v9424: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:41 smithi122 ceph-mon[32255]: pgmap v9424: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:12:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:12:42.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:42 smithi022 ceph-mon[31360]: pgmap v9425: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:42 smithi022 ceph-mon[40250]: pgmap v9425: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:43.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:43.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:43.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:42 smithi122 ceph-mon[32255]: pgmap v9425: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/448348380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/448348380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/448348380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:45 smithi022 ceph-mon[31360]: pgmap v9426: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:45 smithi022 ceph-mon[40250]: pgmap v9426: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:45 smithi122 ceph-mon[32255]: pgmap v9426: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:12:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:47 smithi022 ceph-mon[40250]: pgmap v9427: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:47 smithi022 ceph-mon[31360]: pgmap v9427: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:47 smithi122 ceph-mon[32255]: pgmap v9427: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:48.085 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:48.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:48.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2194454450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2194454450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2194454450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:49.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:12:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:12:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:49 smithi022 ceph-mon[31360]: pgmap v9428: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:49 smithi022 ceph-mon[40250]: pgmap v9428: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:49 smithi122 ceph-mon[32255]: pgmap v9428: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:12:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:12:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:51 smithi022 ceph-mon[31360]: pgmap v9429: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:51 smithi022 ceph-mon[40250]: pgmap v9429: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:51 smithi122 ceph-mon[32255]: pgmap v9429: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:12:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:12:53.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:53 smithi022 ceph-mon[31360]: pgmap v9430: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:53 smithi022 ceph-mon[40250]: pgmap v9430: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:53.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:53.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:53.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:53 smithi122 ceph-mon[32255]: pgmap v9430: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1824967938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1824967938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1824967938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:12:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:55 smithi122 ceph-mon[32255]: pgmap v9431: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:55 smithi022 ceph-mon[31360]: pgmap v9431: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:55 smithi022 ceph-mon[40250]: pgmap v9431: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:12:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:57 smithi022 ceph-mon[31360]: pgmap v9432: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:12:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:57 smithi022 ceph-mon[40250]: pgmap v9432: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:57 smithi122 ceph-mon[32255]: pgmap v9432: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:58.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:12:59.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:12:59.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:12:59.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:12:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:12:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:12:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:59 smithi122 ceph-mon[32255]: pgmap v9433: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:12:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:12:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1590118071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:59 smithi022 ceph-mon[31360]: pgmap v9433: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:12:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1590118071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:59 smithi022 ceph-mon[40250]: pgmap v9433: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:12:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1590118071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:13:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:01 smithi122 ceph-mon[32255]: pgmap v9434: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:01 smithi022 ceph-mon[31360]: pgmap v9434: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:01 smithi022 ceph-mon[40250]: pgmap v9434: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:13:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:13:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:03 smithi122 ceph-mon[32255]: pgmap v9435: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:03 smithi022 ceph-mon[31360]: pgmap v9435: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:03 smithi022 ceph-mon[40250]: pgmap v9435: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:04.199 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:04.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:04.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:04.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:05 smithi122 ceph-mon[32255]: pgmap v9436: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1469142689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:05 smithi022 ceph-mon[31360]: pgmap v9436: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1469142689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:05 smithi022 ceph-mon[40250]: pgmap v9436: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1469142689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:13:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:07 smithi122 ceph-mon[32255]: pgmap v9437: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:13:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:13:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:13:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[31360]: pgmap v9437: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:13:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:13:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:13:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[40250]: pgmap v9437: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:13:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:13:08.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:13:08.999 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:13:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:13:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:13:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:13:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:13:09.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:09.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:09.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:09.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:09.942 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:09 smithi022 ceph-mon[31360]: pgmap v9438: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:09.942 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:09 smithi022 ceph-mon[40250]: pgmap v9438: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:09 smithi122 ceph-mon[32255]: pgmap v9438: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:13:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2684663259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2684663259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2684663259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:11 smithi022 ceph-mon[31360]: pgmap v9439: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:11 smithi022 ceph-mon[40250]: pgmap v9439: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:11 smithi122 ceph-mon[32255]: pgmap v9439: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:13:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:13:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:13 smithi022 ceph-mon[31360]: pgmap v9440: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:13 smithi022 ceph-mon[40250]: pgmap v9440: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:13 smithi122 ceph-mon[32255]: pgmap v9440: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:14.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:15.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:15.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:15.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:15 smithi022 ceph-mon[31360]: pgmap v9441: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2474698206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:15 smithi022 ceph-mon[40250]: pgmap v9441: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2474698206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:15 smithi122 ceph-mon[32255]: pgmap v9441: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2474698206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:17.715 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:13:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:17 smithi022 ceph-mon[31360]: pgmap v9442: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:17 smithi022 ceph-mon[40250]: pgmap v9442: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:17 smithi122 ceph-mon[32255]: pgmap v9442: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:13:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:13:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:19 smithi022 ceph-mon[31360]: pgmap v9443: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:19 smithi022 ceph-mon[40250]: pgmap v9443: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:19 smithi122 ceph-mon[32255]: pgmap v9443: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:20.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:13:20.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:20.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:20.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2884769007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2884769007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2884769007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:21 smithi022 ceph-mon[31360]: pgmap v9444: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:21 smithi022 ceph-mon[40250]: pgmap v9444: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:21 smithi122 ceph-mon[32255]: pgmap v9444: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:13:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:13:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:23 smithi022 ceph-mon[31360]: pgmap v9445: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:23 smithi022 ceph-mon[40250]: pgmap v9445: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:23 smithi122 ceph-mon[32255]: pgmap v9445: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:25.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:25 smithi022 ceph-mon[31360]: pgmap v9446: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:25 smithi022 ceph-mon[40250]: pgmap v9446: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:25 smithi122 ceph-mon[32255]: pgmap v9446: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/976338374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/976338374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/976338374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:27.735 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:13:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:27 smithi022 ceph-mon[31360]: pgmap v9447: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:27 smithi022 ceph-mon[40250]: pgmap v9447: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:27 smithi122 ceph-mon[32255]: pgmap v9447: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:13:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:13:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:29 smithi022 ceph-mon[31360]: pgmap v9448: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:29 smithi022 ceph-mon[40250]: pgmap v9448: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:29 smithi122 ceph-mon[32255]: pgmap v9448: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:13:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:31.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:31.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:31.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:31.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:31 smithi022 ceph-mon[31360]: pgmap v9449: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/571941737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:31 smithi022 ceph-mon[40250]: pgmap v9449: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/571941737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:31 smithi122 ceph-mon[32255]: pgmap v9449: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/571941737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:13:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:13:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[31360]: pgmap v9450: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:13:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:13:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:13:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:13:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[40250]: pgmap v9450: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:13:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:13:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:13:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:13:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:33 smithi122 ceph-mon[32255]: pgmap v9450: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:13:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:13:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:13:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:13:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:35 smithi022 ceph-mon[31360]: pgmap v9451: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:35 smithi022 ceph-mon[40250]: pgmap v9451: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:35 smithi122 ceph-mon[32255]: pgmap v9451: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:36.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:36.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:36.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:37.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:13:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:37 smithi022 ceph-mon[31360]: pgmap v9452: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/64189873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:37 smithi022 ceph-mon[40250]: pgmap v9452: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/64189873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:37 smithi122 ceph-mon[32255]: pgmap v9452: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/64189873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:13:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:13:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:39 smithi022 ceph-mon[31360]: pgmap v9453: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:39 smithi022 ceph-mon[40250]: pgmap v9453: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:39 smithi122 ceph-mon[32255]: pgmap v9453: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:13:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:41.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:41 smithi022 ceph-mon[31360]: pgmap v9454: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:41 smithi022 ceph-mon[40250]: pgmap v9454: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:42.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:42.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:42.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:41 smithi122 ceph-mon[32255]: pgmap v9454: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:13:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:13:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/345153839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/345153839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/345153839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:43 smithi022 ceph-mon[31360]: pgmap v9455: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:43 smithi022 ceph-mon[40250]: pgmap v9455: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:43 smithi122 ceph-mon[32255]: pgmap v9455: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:45 smithi022 ceph-mon[31360]: pgmap v9456: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:45 smithi022 ceph-mon[40250]: pgmap v9456: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:45 smithi122 ceph-mon[32255]: pgmap v9456: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:47.166 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:47.526 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:13:47.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:47.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:47.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:47 smithi022 ceph-mon[31360]: pgmap v9457: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/737138664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:47 smithi022 ceph-mon[40250]: pgmap v9457: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/737138664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:47 smithi122 ceph-mon[32255]: pgmap v9457: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/737138664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:49.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:13:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:13:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:49 smithi022 ceph-mon[31360]: pgmap v9458: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:49 smithi022 ceph-mon[40250]: pgmap v9458: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:49 smithi122 ceph-mon[32255]: pgmap v9458: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:50] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T23:13:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:13:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:51 smithi122 ceph-mon[32255]: pgmap v9459: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:51 smithi022 ceph-mon[31360]: pgmap v9459: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:51 smithi022 ceph-mon[40250]: pgmap v9459: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:52.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:13:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:13:52.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:52.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:52.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:53 smithi122 ceph-mon[32255]: pgmap v9460: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:53.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2212290829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:53 smithi022 ceph-mon[31360]: pgmap v9460: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2212290829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:53 smithi022 ceph-mon[40250]: pgmap v9460: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2212290829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:55 smithi122 ceph-mon[32255]: pgmap v9461: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:55 smithi022 ceph-mon[31360]: pgmap v9461: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:55 smithi022 ceph-mon[40250]: pgmap v9461: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:13:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:57 smithi022 ceph-mon[31360]: pgmap v9462: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:13:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:57 smithi022 ceph-mon[40250]: pgmap v9462: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:57.904 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:13:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:57 smithi122 ceph-mon[32255]: pgmap v9462: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:13:58.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:13:58.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:13:58.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:13:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/949219616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/949219616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/949219616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:13:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:13:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:13:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:13:59 smithi122 ceph-mon[32255]: pgmap v9463: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:13:59 smithi022 ceph-mon[31360]: pgmap v9463: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:13:59 smithi022 ceph-mon[40250]: pgmap v9463: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:00] "GET /metrics HTTP/1.1" 200 202381 "" "Prometheus/2.18.1" 2021-12-15T23:14:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:01 smithi122 ceph-mon[32255]: pgmap v9464: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:01 smithi022 ceph-mon[31360]: pgmap v9464: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:01 smithi022 ceph-mon[40250]: pgmap v9464: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:14:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:14:03.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:03.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:03.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:03.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:03 smithi122 ceph-mon[32255]: pgmap v9465: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:03 smithi022 ceph-mon[31360]: pgmap v9465: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:03 smithi022 ceph-mon[40250]: pgmap v9465: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3899310816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3899310816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3899310816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:05 smithi122 ceph-mon[32255]: pgmap v9466: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:05 smithi022 ceph-mon[31360]: pgmap v9466: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:05 smithi022 ceph-mon[40250]: pgmap v9466: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:07.733 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:14:07.734 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:07 smithi022 ceph-mon[31360]: pgmap v9467: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:07.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:07 smithi022 ceph-mon[40250]: pgmap v9467: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:07 smithi122 ceph-mon[32255]: pgmap v9467: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:08.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:14:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:14:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:14:09.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:09.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:09.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:09.018 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:14:09.018 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:14:09.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:14:09.019 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:14:09.019 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:14:09.019 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:14:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:14:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:14:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:09 smithi122 ceph-mon[32255]: pgmap v9468: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1247037395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:09 smithi022 ceph-mon[31360]: pgmap v9468: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1247037395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:09 smithi022 ceph-mon[40250]: pgmap v9468: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1247037395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:10] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:14:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:11 smithi022 ceph-mon[40250]: pgmap v9469: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:11 smithi022 ceph-mon[31360]: pgmap v9469: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:11 smithi122 ceph-mon[32255]: pgmap v9469: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:12.583 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:14:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:14:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:12.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:12.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:12.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:13.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:13.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:13.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:14:13.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:13.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:13.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:14:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:13 smithi122 ceph-mon[32255]: pgmap v9470: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:13 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3149M 2021-12-15T23:14:14.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:13 smithi022 ceph-mon[31360]: pgmap v9470: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:14.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:13 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3149M 2021-12-15T23:14:14.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:13 smithi022 ceph-mon[40250]: pgmap v9470: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:14.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:13 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3149M 2021-12-15T23:14:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:14.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:14.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:14.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3118619492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3118619492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3118619492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:15 smithi122 ceph-mon[32255]: pgmap v9471: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:15 smithi022 ceph-mon[31360]: pgmap v9471: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:15 smithi022 ceph-mon[40250]: pgmap v9471: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:17 smithi022 ceph-mon[31360]: pgmap v9472: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:14:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:17 smithi022 ceph-mon[40250]: pgmap v9472: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:17 smithi122 ceph-mon[32255]: pgmap v9472: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:19.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:14:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:14:19.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:19.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:19.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:19.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:19 smithi022 ceph-mon[40250]: pgmap v9473: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:19.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:19 smithi022 ceph-mon[31360]: pgmap v9473: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:19 smithi122 ceph-mon[32255]: pgmap v9473: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:20] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:14:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/547173214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/547173214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/547173214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:21 smithi122 ceph-mon[32255]: pgmap v9474: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:21 smithi022 ceph-mon[31360]: pgmap v9474: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:21 smithi022 ceph-mon[40250]: pgmap v9474: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:14:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:14:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:23 smithi122 ceph-mon[32255]: pgmap v9475: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:23 smithi022 ceph-mon[31360]: pgmap v9475: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:23 smithi022 ceph-mon[40250]: pgmap v9475: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:24.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:25.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:25.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:25.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:25 smithi122 ceph-mon[32255]: pgmap v9476: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1735698637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:25 smithi022 ceph-mon[31360]: pgmap v9476: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1735698637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:25 smithi022 ceph-mon[40250]: pgmap v9476: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1735698637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:27 smithi022 ceph-mon[31360]: pgmap v9477: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:14:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:27 smithi022 ceph-mon[40250]: pgmap v9477: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:27 smithi122 ceph-mon[32255]: pgmap v9477: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:14:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:14:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:29 smithi122 ceph-mon[32255]: pgmap v9478: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:29 smithi022 ceph-mon[31360]: pgmap v9478: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:29 smithi022 ceph-mon[40250]: pgmap v9478: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:30.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:30] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:14:30.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:30.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:30.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/971835575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/971835575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/971835575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:31 smithi122 ceph-mon[32255]: pgmap v9479: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:31 smithi022 ceph-mon[31360]: pgmap v9479: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:31 smithi022 ceph-mon[40250]: pgmap v9479: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:14:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:14:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:33 smithi122 ceph-mon[32255]: pgmap v9480: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:14:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:14:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:14:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:14:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[31360]: pgmap v9480: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:14:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:14:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:14:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:14:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[40250]: pgmap v9480: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:14:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:14:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:14:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:14:35.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:35 smithi022 ceph-mon[31360]: pgmap v9481: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:35 smithi022 ceph-mon[40250]: pgmap v9481: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:35.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:35.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:35.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:35 smithi122 ceph-mon[32255]: pgmap v9481: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1565220011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1565220011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1565220011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:37 smithi022 ceph-mon[31360]: pgmap v9482: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:14:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:37 smithi022 ceph-mon[40250]: pgmap v9482: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:37 smithi122 ceph-mon[32255]: pgmap v9482: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:14:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:14:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:39 smithi122 ceph-mon[32255]: pgmap v9483: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:39 smithi022 ceph-mon[31360]: pgmap v9483: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:39 smithi022 ceph-mon[40250]: pgmap v9483: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:14:40.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:41.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:41.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:41 smithi122 ceph-mon[32255]: pgmap v9484: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/721707320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:41 smithi022 ceph-mon[31360]: pgmap v9484: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/721707320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:41 smithi022 ceph-mon[40250]: pgmap v9484: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/721707320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:14:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:14:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:43 smithi122 ceph-mon[32255]: pgmap v9485: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:43 smithi022 ceph-mon[31360]: pgmap v9485: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:43 smithi022 ceph-mon[40250]: pgmap v9485: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:45 smithi122 ceph-mon[32255]: pgmap v9486: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:45 smithi022 ceph-mon[31360]: pgmap v9486: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:45 smithi022 ceph-mon[40250]: pgmap v9486: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:46.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:46.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:46.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:46.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2343962950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2343962950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2343962950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:47.652 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:14:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:47 smithi122 ceph-mon[32255]: pgmap v9487: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:47 smithi022 ceph-mon[31360]: pgmap v9487: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:47 smithi022 ceph-mon[40250]: pgmap v9487: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:49.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:14:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:14:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:49 smithi122 ceph-mon[32255]: pgmap v9488: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:49 smithi022 ceph-mon[31360]: pgmap v9488: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:49 smithi022 ceph-mon[40250]: pgmap v9488: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:14:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:14:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:51 smithi122 ceph-mon[32255]: pgmap v9489: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:51.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:51.997 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:51 smithi022 ceph-mon[40250]: pgmap v9489: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:51.997 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:51 smithi022 ceph-mon[31360]: pgmap v9489: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:52.665 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:14:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:14:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1804772922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1804772922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1804772922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:53 smithi122 ceph-mon[32255]: pgmap v9490: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:53 smithi022 ceph-mon[31360]: pgmap v9490: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:53 smithi022 ceph-mon[40250]: pgmap v9490: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:55 smithi122 ceph-mon[32255]: pgmap v9491: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:55 smithi022 ceph-mon[31360]: pgmap v9491: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:55 smithi022 ceph-mon[40250]: pgmap v9491: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:14:56.998 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:14:57.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:14:57.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:14:57.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:14:57.672 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:14:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:57 smithi122 ceph-mon[32255]: pgmap v9492: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2174857146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:57 smithi022 ceph-mon[31360]: pgmap v9492: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2174857146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:57 smithi022 ceph-mon[40250]: pgmap v9492: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:14:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2174857146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:14:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:14:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:14:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:14:59 smithi122 ceph-mon[32255]: pgmap v9493: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:14:59 smithi022 ceph-mon[31360]: pgmap v9493: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:14:59 smithi022 ceph-mon[40250]: pgmap v9493: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:15:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:01 smithi022 ceph-mon[31360]: pgmap v9494: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:01 smithi022 ceph-mon[40250]: pgmap v9494: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:01 smithi122 ceph-mon[32255]: pgmap v9494: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:02.371 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:15:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:15:02.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:02.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:02.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:03 smithi022 ceph-mon[31360]: pgmap v9495: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/282944592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:03 smithi022 ceph-mon[40250]: pgmap v9495: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/282944592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:03 smithi122 ceph-mon[32255]: pgmap v9495: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/282944592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:05 smithi022 ceph-mon[31360]: pgmap v9496: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:05 smithi022 ceph-mon[40250]: pgmap v9496: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:05 smithi122 ceph-mon[32255]: pgmap v9496: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:07.695 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:15:07.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:07 smithi022 ceph-mon[31360]: pgmap v9497: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:07 smithi022 ceph-mon[40250]: pgmap v9497: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:08.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:08.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:08.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:07 smithi122 ceph-mon[32255]: pgmap v9497: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/379932974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/379932974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:09.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/379932974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:15:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:15:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:09 smithi022 ceph-mon[31360]: pgmap v9498: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:09 smithi022 ceph-mon[40250]: pgmap v9498: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:09 smithi122 ceph-mon[32255]: pgmap v9498: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:15:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:11 smithi022 ceph-mon[31360]: pgmap v9499: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:11 smithi022 ceph-mon[40250]: pgmap v9499: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:11 smithi122 ceph-mon[32255]: pgmap v9499: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:15:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:15:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:15:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:15:12.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:15:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:15:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:15:12.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:15:13.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:15:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:15:13.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:15:13.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:13.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:13.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:13 smithi022 ceph-mon[31360]: pgmap v9500: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:15:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3192312982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:13 smithi022 ceph-mon[40250]: pgmap v9500: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:15:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3192312982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:13 smithi122 ceph-mon[32255]: pgmap v9500: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:15:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3192312982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:15 smithi022 ceph-mon[31360]: pgmap v9501: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:15 smithi022 ceph-mon[40250]: pgmap v9501: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:15 smithi122 ceph-mon[32255]: pgmap v9501: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:17.718 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:15:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:17 smithi022 ceph-mon[31360]: pgmap v9502: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:17 smithi022 ceph-mon[40250]: pgmap v9502: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:17 smithi122 ceph-mon[32255]: pgmap v9502: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:18.478 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:18.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:18.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:18.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:19.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:15:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:15:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:19 smithi022 ceph-mon[31360]: pgmap v9503: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1402813178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:19 smithi022 ceph-mon[40250]: pgmap v9503: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1402813178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:20.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:19 smithi122 ceph-mon[32255]: pgmap v9503: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:20.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1402813178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:20] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:15:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:21 smithi022 ceph-mon[31360]: pgmap v9504: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:21 smithi022 ceph-mon[40250]: pgmap v9504: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:21 smithi122 ceph-mon[32255]: pgmap v9504: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:15:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:15:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:23 smithi022 ceph-mon[31360]: pgmap v9505: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:23 smithi022 ceph-mon[40250]: pgmap v9505: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:23 smithi122 ceph-mon[32255]: pgmap v9505: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:24.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:24.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:24.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2463866541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2463866541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2463866541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:25 smithi022 ceph-mon[31360]: pgmap v9506: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:25 smithi022 ceph-mon[40250]: pgmap v9506: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:25 smithi122 ceph-mon[32255]: pgmap v9506: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:27.740 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:15:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:27 smithi022 ceph-mon[31360]: pgmap v9507: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:27 smithi022 ceph-mon[40250]: pgmap v9507: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:27 smithi122 ceph-mon[32255]: pgmap v9507: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:29.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:15:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:15:29.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:29.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:29.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:29 smithi022 ceph-mon[31360]: pgmap v9508: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1143774565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:29 smithi022 ceph-mon[40250]: pgmap v9508: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1143774565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:29 smithi122 ceph-mon[32255]: pgmap v9508: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1143774565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:30] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:15:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:31 smithi022 ceph-mon[31360]: pgmap v9509: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:31 smithi022 ceph-mon[40250]: pgmap v9509: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:31 smithi122 ceph-mon[32255]: pgmap v9509: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:15:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:15:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[31360]: pgmap v9510: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:15:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:15:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:15:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:15:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[40250]: pgmap v9510: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:15:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:15:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:15:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:15:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:33 smithi122 ceph-mon[32255]: pgmap v9510: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:15:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:15:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:15:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:15:34.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:34.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:34.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:34.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:35 smithi022 ceph-mon[31360]: pgmap v9511: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/548765865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:35 smithi022 ceph-mon[40250]: pgmap v9511: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/548765865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:35 smithi122 ceph-mon[32255]: pgmap v9511: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/548765865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:37.756 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:15:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:37 smithi022 ceph-mon[31360]: pgmap v9512: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:37 smithi022 ceph-mon[40250]: pgmap v9512: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:37 smithi122 ceph-mon[32255]: pgmap v9512: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:15:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:15:39.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:39 smithi022 ceph-mon[31360]: pgmap v9513: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:39 smithi022 ceph-mon[40250]: pgmap v9513: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:39 smithi122 ceph-mon[32255]: pgmap v9513: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:40.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:40.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:40.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:40] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:15:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/938692481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/938692481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/938692481' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:41 smithi022 ceph-mon[31360]: pgmap v9514: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:41 smithi022 ceph-mon[40250]: pgmap v9514: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:41 smithi122 ceph-mon[32255]: pgmap v9514: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:15:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:15:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:43 smithi022 ceph-mon[31360]: pgmap v9515: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:43 smithi022 ceph-mon[40250]: pgmap v9515: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:43 smithi122 ceph-mon[32255]: pgmap v9515: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:45.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:45.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:45.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:45.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:45 smithi022 ceph-mon[31360]: pgmap v9516: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4055029942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:45 smithi022 ceph-mon[40250]: pgmap v9516: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4055029942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:45 smithi122 ceph-mon[32255]: pgmap v9516: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4055029942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:47.773 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:15:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:47 smithi022 ceph-mon[31360]: pgmap v9517: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:47 smithi022 ceph-mon[40250]: pgmap v9517: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:47 smithi122 ceph-mon[32255]: pgmap v9517: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:49.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:15:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:15:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:49 smithi022 ceph-mon[31360]: pgmap v9518: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:49 smithi022 ceph-mon[40250]: pgmap v9518: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:49 smithi122 ceph-mon[32255]: pgmap v9518: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:50] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:15:50.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:51.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:51.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:51.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:15:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:51 smithi122 ceph-mon[32255]: pgmap v9519: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1106034036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:51 smithi022 ceph-mon[31360]: pgmap v9519: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1106034036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:51 smithi022 ceph-mon[40250]: pgmap v9519: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1106034036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:15:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:15:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:53 smithi122 ceph-mon[32255]: pgmap v9520: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:53 smithi022 ceph-mon[31360]: pgmap v9520: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:53 smithi022 ceph-mon[40250]: pgmap v9520: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:56.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:15:56.071 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:55 smithi022 ceph-mon[31360]: pgmap v9521: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:56.071 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:55 smithi022 ceph-mon[40250]: pgmap v9521: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:55 smithi122 ceph-mon[32255]: pgmap v9521: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:15:56.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:15:56.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:15:56.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:15:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2929925103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2929925103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2929925103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:15:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:15:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:57 smithi122 ceph-mon[32255]: pgmap v9522: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:57 smithi022 ceph-mon[31360]: pgmap v9522: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:57 smithi022 ceph-mon[40250]: pgmap v9522: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:59.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:15:58 smithi122 ceph-mon[32255]: pgmap v9523: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:15:58 smithi022 ceph-mon[31360]: pgmap v9523: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:59.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:15:58 smithi022 ceph-mon[40250]: pgmap v9523: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:15:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:15:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:16:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:00] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:16:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:01.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:01 smithi022 ceph-mon[31360]: pgmap v9524: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:01 smithi022 ceph-mon[40250]: pgmap v9524: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:01.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:01.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:01.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:01 smithi122 ceph-mon[32255]: pgmap v9524: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1921510692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:16:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:16:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1921510692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1921510692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:03 smithi122 ceph-mon[32255]: pgmap v9525: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:03 smithi022 ceph-mon[31360]: pgmap v9525: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:03 smithi022 ceph-mon[40250]: pgmap v9525: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:05 smithi122 ceph-mon[32255]: pgmap v9526: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:05 smithi022 ceph-mon[31360]: pgmap v9526: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:05 smithi022 ceph-mon[40250]: pgmap v9526: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:06.811 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:07.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:07.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:07.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:16:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:07 smithi122 ceph-mon[32255]: pgmap v9527: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/125253935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:07 smithi022 ceph-mon[31360]: pgmap v9527: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/125253935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:07 smithi022 ceph-mon[40250]: pgmap v9527: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/125253935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:09.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:16:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:16:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:09 smithi122 ceph-mon[32255]: pgmap v9528: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:09 smithi022 ceph-mon[31360]: pgmap v9528: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:09 smithi022 ceph-mon[40250]: pgmap v9528: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:16:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:11 smithi122 ceph-mon[32255]: pgmap v9529: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:11 smithi022 ceph-mon[31360]: pgmap v9529: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:11 smithi022 ceph-mon[40250]: pgmap v9529: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:12.179 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:12.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:12.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:12.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:12.585 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:16:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:16:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2274359071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:12.942 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2274359071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:12.942 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2274359071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:13 smithi122 ceph-mon[32255]: pgmap v9530: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:16:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:16:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:16:13.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:16:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[31360]: pgmap v9530: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:16:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:16:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:16:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:16:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[40250]: pgmap v9530: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:16:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:16:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:16:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:16:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:15 smithi122 ceph-mon[32255]: pgmap v9531: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:15 smithi022 ceph-mon[31360]: pgmap v9531: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:15 smithi022 ceph-mon[40250]: pgmap v9531: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:17.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:17 smithi022 ceph-mon[31360]: pgmap v9532: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:16:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:17 smithi022 ceph-mon[40250]: pgmap v9532: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:17.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:17.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:17.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:17 smithi122 ceph-mon[32255]: pgmap v9532: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/467293029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/467293029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/467293029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:16:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:16:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:19 smithi122 ceph-mon[32255]: pgmap v9533: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:19 smithi022 ceph-mon[31360]: pgmap v9533: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:19 smithi022 ceph-mon[40250]: pgmap v9533: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:16:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:21 smithi122 ceph-mon[32255]: pgmap v9534: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:21 smithi022 ceph-mon[31360]: pgmap v9534: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:21 smithi022 ceph-mon[40250]: pgmap v9534: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:16:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:16:22.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:23.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:23.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:23.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:23 smithi122 ceph-mon[32255]: pgmap v9535: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1708761114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:23 smithi022 ceph-mon[31360]: pgmap v9535: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1708761114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:23 smithi022 ceph-mon[40250]: pgmap v9535: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1708761114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:25 smithi122 ceph-mon[32255]: pgmap v9536: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:25 smithi022 ceph-mon[31360]: pgmap v9536: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:25 smithi022 ceph-mon[40250]: pgmap v9536: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:27 smithi022 ceph-mon[31360]: pgmap v9537: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:16:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:27 smithi022 ceph-mon[40250]: pgmap v9537: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:27 smithi122 ceph-mon[32255]: pgmap v9537: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:28.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:28.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:28.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:16:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:16:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:29 smithi122 ceph-mon[32255]: pgmap v9538: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4167179274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:29 smithi022 ceph-mon[31360]: pgmap v9538: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4167179274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:29 smithi022 ceph-mon[40250]: pgmap v9538: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4167179274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:16:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:31 smithi122 ceph-mon[32255]: pgmap v9539: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:31 smithi022 ceph-mon[31360]: pgmap v9539: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:31 smithi022 ceph-mon[40250]: pgmap v9539: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:16:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:16:33.667 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:33 smithi122 ceph-mon[32255]: pgmap v9540: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:16:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:16:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:16:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:16:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[31360]: pgmap v9540: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:16:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:16:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:16:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:16:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[40250]: pgmap v9540: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:16:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:16:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:16:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:16:34.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:34.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:34.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3746688625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3746688625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3746688625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:35 smithi122 ceph-mon[32255]: pgmap v9541: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:35 smithi022 ceph-mon[31360]: pgmap v9541: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:35 smithi022 ceph-mon[40250]: pgmap v9541: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:37 smithi022 ceph-mon[31360]: pgmap v9542: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:16:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:37 smithi022 ceph-mon[40250]: pgmap v9542: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:37 smithi122 ceph-mon[32255]: pgmap v9542: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:39.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:39.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:39.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:39.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:39.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:16:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:16:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:39 smithi022 ceph-mon[31360]: pgmap v9543: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:39.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/47185196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:39 smithi022 ceph-mon[40250]: pgmap v9543: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/47185196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:39 smithi122 ceph-mon[32255]: pgmap v9543: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/47185196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:40] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:16:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:41 smithi122 ceph-mon[32255]: pgmap v9544: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:41 smithi022 ceph-mon[31360]: pgmap v9544: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:41 smithi022 ceph-mon[40250]: pgmap v9544: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:16:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:16:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:43 smithi122 ceph-mon[32255]: pgmap v9545: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:43 smithi022 ceph-mon[31360]: pgmap v9545: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:43 smithi022 ceph-mon[40250]: pgmap v9545: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:44.405 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:44.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:44.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:44.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:45 smithi122 ceph-mon[32255]: pgmap v9546: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2618764396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:45 smithi022 ceph-mon[31360]: pgmap v9546: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2618764396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:45 smithi022 ceph-mon[40250]: pgmap v9546: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2618764396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:47.646 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:16:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:47 smithi122 ceph-mon[32255]: pgmap v9547: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:47 smithi022 ceph-mon[31360]: pgmap v9547: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:47 smithi022 ceph-mon[40250]: pgmap v9547: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:49.649 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:16:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:16:49.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:49 smithi122 ceph-mon[32255]: pgmap v9548: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:49 smithi022 ceph-mon[31360]: pgmap v9548: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:49 smithi022 ceph-mon[40250]: pgmap v9548: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:50.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:50.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:50.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:50] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T23:16:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2360760852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2360760852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:16:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2360760852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:51 smithi122 ceph-mon[32255]: pgmap v9549: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:51 smithi022 ceph-mon[31360]: pgmap v9549: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:51 smithi022 ceph-mon[40250]: pgmap v9549: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:16:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:16:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:53 smithi122 ceph-mon[32255]: pgmap v9550: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:53 smithi022 ceph-mon[31360]: pgmap v9550: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:53 smithi022 ceph-mon[40250]: pgmap v9550: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:55.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:16:55.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:16:55.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:16:55.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:16:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:55 smithi022 ceph-mon[31360]: pgmap v9551: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:55.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1860128365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:55 smithi022 ceph-mon[40250]: pgmap v9551: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:55.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1860128365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:55 smithi122 ceph-mon[32255]: pgmap v9551: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:16:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1860128365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:16:57.663 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:16:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:57 smithi122 ceph-mon[32255]: pgmap v9552: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:57 smithi022 ceph-mon[31360]: pgmap v9552: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:57 smithi022 ceph-mon[40250]: pgmap v9552: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:16:59.667 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:16:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:16:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:16:59 smithi122 ceph-mon[32255]: pgmap v9553: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:16:59 smithi022 ceph-mon[31360]: pgmap v9553: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:16:59 smithi022 ceph-mon[40250]: pgmap v9553: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:00] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T23:17:00.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:00.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:00.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:00.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:01 smithi122 ceph-mon[32255]: pgmap v9554: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3536253300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:01 smithi022 ceph-mon[31360]: pgmap v9554: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3536253300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:01 smithi022 ceph-mon[40250]: pgmap v9554: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3536253300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:17:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:17:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:03 smithi122 ceph-mon[32255]: pgmap v9555: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:03 smithi022 ceph-mon[31360]: pgmap v9555: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:03 smithi022 ceph-mon[40250]: pgmap v9555: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:05.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:05 smithi122 ceph-mon[32255]: pgmap v9556: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:05 smithi022 ceph-mon[31360]: pgmap v9556: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:05 smithi022 ceph-mon[40250]: pgmap v9556: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:06.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:06.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:06.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1048036025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1048036025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1048036025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:07.684 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:17:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:07 smithi122 ceph-mon[32255]: pgmap v9557: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:07 smithi022 ceph-mon[31360]: pgmap v9557: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:07 smithi022 ceph-mon[40250]: pgmap v9557: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:17:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:17:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:09 smithi022 ceph-mon[31360]: pgmap v9558: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:09 smithi022 ceph-mon[40250]: pgmap v9558: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:09 smithi122 ceph-mon[32255]: pgmap v9558: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:10] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:17:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:11.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:11.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:11.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:11.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:11 smithi022 ceph-mon[31360]: pgmap v9559: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3532615710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:11 smithi022 ceph-mon[40250]: pgmap v9559: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3532615710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:11 smithi122 ceph-mon[32255]: pgmap v9559: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3532615710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:17:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:17:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[31360]: pgmap v9560: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:17:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:17:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:17:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[40250]: pgmap v9560: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:17:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:17:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:17:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:13 smithi122 ceph-mon[32255]: pgmap v9560: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:17:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:17:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:17:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:17:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:14 smithi122 ceph-mon[32255]: pgmap v9561: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:17:15.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:14 smithi022 ceph-mon[31360]: pgmap v9561: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:17:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:14 smithi022 ceph-mon[40250]: pgmap v9561: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:16.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:16.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:16.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:16.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:16 smithi122 ceph-mon[32255]: pgmap v9562: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:16 smithi022 ceph-mon[31360]: pgmap v9562: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:16 smithi022 ceph-mon[40250]: pgmap v9562: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:17:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/344120737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/344120737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/344120737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:18 smithi122 ceph-mon[32255]: pgmap v9563: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:18 smithi022 ceph-mon[31360]: pgmap v9563: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:18 smithi022 ceph-mon[40250]: pgmap v9563: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:17:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:17:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:20] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T23:17:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:21 smithi122 ceph-mon[32255]: pgmap v9564: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:21.995 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:21 smithi022 ceph-mon[31360]: pgmap v9564: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:21.995 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:21 smithi022 ceph-mon[40250]: pgmap v9564: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:22.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:22.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:22.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:22.581 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:17:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:17:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2477249685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2477249685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2477249685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:23 smithi122 ceph-mon[32255]: pgmap v9565: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:23 smithi022 ceph-mon[31360]: pgmap v9565: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:23 smithi022 ceph-mon[40250]: pgmap v9565: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:25 smithi122 ceph-mon[32255]: pgmap v9566: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:25 smithi022 ceph-mon[31360]: pgmap v9566: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:25 smithi022 ceph-mon[40250]: pgmap v9566: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:27.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:27.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:27.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:27.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:27.732 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:17:27.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:27 smithi022 ceph-mon[40250]: pgmap v9567: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:27.733 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:27 smithi022 ceph-mon[31360]: pgmap v9567: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:27 smithi122 ceph-mon[32255]: pgmap v9567: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3708910086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3708910086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3708910086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:29.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:17:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:17:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:29 smithi122 ceph-mon[32255]: pgmap v9568: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:29 smithi022 ceph-mon[31360]: pgmap v9568: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:29 smithi022 ceph-mon[40250]: pgmap v9568: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:30] "GET /metrics HTTP/1.1" 200 202375 "" "Prometheus/2.18.1" 2021-12-15T23:17:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:31 smithi122 ceph-mon[32255]: pgmap v9569: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:31 smithi022 ceph-mon[31360]: pgmap v9569: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:31 smithi022 ceph-mon[40250]: pgmap v9569: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:17:32.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:17:32.733 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:33.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:33 smithi122 ceph-mon[32255]: pgmap v9570: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2350774550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:17:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:17:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[31360]: pgmap v9570: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2350774550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:17:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:17:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:17:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:17:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[40250]: pgmap v9570: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2350774550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:17:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:17:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:17:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:17:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:35 smithi122 ceph-mon[32255]: pgmap v9571: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:35 smithi022 ceph-mon[31360]: pgmap v9571: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:35 smithi022 ceph-mon[40250]: pgmap v9571: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:37 smithi022 ceph-mon[31360]: pgmap v9572: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:17:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:37 smithi022 ceph-mon[40250]: pgmap v9572: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:37 smithi122 ceph-mon[32255]: pgmap v9572: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:38.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:38.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:38.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:38.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2820995752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2820995752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2820995752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:39.614 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:17:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:17:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:39 smithi122 ceph-mon[32255]: pgmap v9573: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:39 smithi022 ceph-mon[31360]: pgmap v9573: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:39 smithi022 ceph-mon[40250]: pgmap v9573: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:17:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:41 smithi122 ceph-mon[32255]: pgmap v9574: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:41 smithi022 ceph-mon[31360]: pgmap v9574: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:41 smithi022 ceph-mon[40250]: pgmap v9574: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:17:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:17:43.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:43 smithi022 ceph-mon[31360]: pgmap v9575: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:43 smithi022 ceph-mon[40250]: pgmap v9575: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:43.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:43.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:43.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:43 smithi122 ceph-mon[32255]: pgmap v9575: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/778642128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/778642128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/778642128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:45 smithi122 ceph-mon[32255]: pgmap v9576: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:45 smithi022 ceph-mon[31360]: pgmap v9576: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:45 smithi022 ceph-mon[40250]: pgmap v9576: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:47 smithi022 ceph-mon[31360]: pgmap v9577: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:17:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:47 smithi022 ceph-mon[40250]: pgmap v9577: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:47 smithi122 ceph-mon[32255]: pgmap v9577: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:49.209 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:49.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:49.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:49.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:17:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:17:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:49 smithi122 ceph-mon[32255]: pgmap v9578: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2633524316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:49 smithi022 ceph-mon[31360]: pgmap v9578: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2633524316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:49 smithi022 ceph-mon[40250]: pgmap v9578: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2633524316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:50] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:17:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:17:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:51 smithi122 ceph-mon[32255]: pgmap v9579: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:51 smithi022 ceph-mon[31360]: pgmap v9579: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:51 smithi022 ceph-mon[40250]: pgmap v9579: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:17:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:17:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:17:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:53 smithi122 ceph-mon[32255]: pgmap v9580: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:53 smithi022 ceph-mon[31360]: pgmap v9580: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:53 smithi022 ceph-mon[40250]: pgmap v9580: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:17:54.211 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:54.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:54.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:54.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3043719468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3043719468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3043719468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:17:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:55 smithi122 ceph-mon[32255]: pgmap v9581: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:55 smithi022 ceph-mon[31360]: pgmap v9581: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:55 smithi022 ceph-mon[40250]: pgmap v9581: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:57.655 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:17:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:57 smithi122 ceph-mon[32255]: pgmap v9582: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:57 smithi022 ceph-mon[31360]: pgmap v9582: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:57 smithi022 ceph-mon[40250]: pgmap v9582: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:59.586 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:17:59.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:17:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:17:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:17:59 smithi122 ceph-mon[32255]: pgmap v9583: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:17:59.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:17:59.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:17:59.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:17:59 smithi022 ceph-mon[40250]: pgmap v9583: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:17:59.957 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:17:59 smithi022 ceph-mon[31360]: pgmap v9583: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-15T23:18:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:00] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:18:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/360727434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/360727434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/360727434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:01 smithi122 ceph-mon[32255]: pgmap v9584: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T23:18:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:01 smithi022 ceph-mon[31360]: pgmap v9584: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T23:18:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:01 smithi022 ceph-mon[40250]: pgmap v9584: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-15T23:18:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:18:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:18:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:03 smithi122 ceph-mon[32255]: pgmap v9585: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T23:18:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:03 smithi022 ceph-mon[31360]: pgmap v9585: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T23:18:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:03 smithi022 ceph-mon[40250]: pgmap v9585: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-15T23:18:04.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:05.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:05.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:05.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:05 smithi122 ceph-mon[32255]: pgmap v9586: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2021-12-15T23:18:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1148418197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:05 smithi022 ceph-mon[31360]: pgmap v9586: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2021-12-15T23:18:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1148418197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:05 smithi022 ceph-mon[40250]: pgmap v9586: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2021-12-15T23:18:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1148418197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:07.672 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:18:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:07 smithi122 ceph-mon[32255]: pgmap v9587: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2021-12-15T23:18:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:07 smithi022 ceph-mon[40250]: pgmap v9587: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2021-12-15T23:18:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:07 smithi022 ceph-mon[31360]: pgmap v9587: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2021-12-15T23:18:09.675 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:18:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:18:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:09 smithi122 ceph-mon[32255]: pgmap v9588: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2021-12-15T23:18:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:09 smithi022 ceph-mon[31360]: pgmap v9588: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2021-12-15T23:18:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:09 smithi022 ceph-mon[40250]: pgmap v9588: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2021-12-15T23:18:10.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:18:10.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:10.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:10.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:10.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:10.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:11 smithi122 ceph-mon[32255]: pgmap v9589: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:18:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/807237946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:11 smithi022 ceph-mon[31360]: pgmap v9589: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:18:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/807237946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:11 smithi022 ceph-mon[40250]: pgmap v9589: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:18:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/807237946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:18:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:18:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:13 smithi122 ceph-mon[32255]: pgmap v9590: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:18:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:13 smithi022 ceph-mon[31360]: pgmap v9590: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:18:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:13 smithi022 ceph-mon[40250]: pgmap v9590: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:18:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:18:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:18:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:18:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:18:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:18:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:18:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:18:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:18:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:18:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:18:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:18:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:18:15.706 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:15 smithi022 ceph-mon[40250]: pgmap v9591: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:18:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:15 smithi022 ceph-mon[31360]: pgmap v9591: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:18:16.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:16.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:16.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:15 smithi122 ceph-mon[32255]: pgmap v9591: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:18:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/584431936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/584431936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/584431936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:17.696 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:18:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:17 smithi022 ceph-mon[40250]: pgmap v9592: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:17 smithi022 ceph-mon[31360]: pgmap v9592: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:17 smithi122 ceph-mon[32255]: pgmap v9592: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:18:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:18:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:19 smithi022 ceph-mon[31360]: pgmap v9593: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:19 smithi022 ceph-mon[40250]: pgmap v9593: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:19 smithi122 ceph-mon[32255]: pgmap v9593: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:18:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:21.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:21.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:21.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:21.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:21 smithi022 ceph-mon[31360]: pgmap v9594: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/521894475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:21 smithi022 ceph-mon[40250]: pgmap v9594: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/521894475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:21 smithi122 ceph-mon[32255]: pgmap v9594: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 9 op/s 2021-12-15T23:18:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/521894475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:18:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:18:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:23 smithi022 ceph-mon[31360]: pgmap v9595: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:23 smithi022 ceph-mon[40250]: pgmap v9595: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:23 smithi122 ceph-mon[32255]: pgmap v9595: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:25 smithi022 ceph-mon[31360]: pgmap v9596: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:25 smithi022 ceph-mon[40250]: pgmap v9596: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:25 smithi122 ceph-mon[32255]: pgmap v9596: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:26.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:26.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:26.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:27.714 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:18:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:27 smithi022 ceph-mon[31360]: pgmap v9597: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/38928292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:27 smithi022 ceph-mon[40250]: pgmap v9597: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/38928292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:27 smithi122 ceph-mon[32255]: pgmap v9597: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/38928292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:18:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:18:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:29 smithi022 ceph-mon[31360]: pgmap v9598: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:29 smithi022 ceph-mon[40250]: pgmap v9598: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:29 smithi122 ceph-mon[32255]: pgmap v9598: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:18:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:31 smithi022 ceph-mon[31360]: pgmap v9599: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:31 smithi022 ceph-mon[40250]: pgmap v9599: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:32.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:32.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:32.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:31 smithi122 ceph-mon[32255]: pgmap v9599: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:18:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:18:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1174081084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1174081084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1174081084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[31360]: pgmap v9600: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:18:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:18:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:18:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:18:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[40250]: pgmap v9600: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:18:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:18:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:18:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:18:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:33 smithi122 ceph-mon[32255]: pgmap v9600: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:18:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:18:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:18:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:18:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:35 smithi022 ceph-mon[31360]: pgmap v9601: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:35 smithi022 ceph-mon[40250]: pgmap v9601: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:35 smithi122 ceph-mon[32255]: pgmap v9601: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:37.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:18:37.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:37.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:37.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:37 smithi022 ceph-mon[31360]: pgmap v9602: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4102579675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:37 smithi022 ceph-mon[40250]: pgmap v9602: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4102579675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:37 smithi122 ceph-mon[32255]: pgmap v9602: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4102579675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:18:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:18:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:39 smithi022 ceph-mon[31360]: pgmap v9603: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:39 smithi022 ceph-mon[40250]: pgmap v9603: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:39 smithi122 ceph-mon[32255]: pgmap v9603: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:18:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:41 smithi022 ceph-mon[31360]: pgmap v9604: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:41 smithi022 ceph-mon[40250]: pgmap v9604: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:41 smithi122 ceph-mon[32255]: pgmap v9604: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:42.556 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:18:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:18:42.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:42.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:42.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:43 smithi022 ceph-mon[31360]: pgmap v9605: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1233856521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:43 smithi022 ceph-mon[40250]: pgmap v9605: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1233856521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:43 smithi122 ceph-mon[32255]: pgmap v9605: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1233856521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:45 smithi022 ceph-mon[31360]: pgmap v9606: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:45 smithi022 ceph-mon[40250]: pgmap v9606: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:45 smithi122 ceph-mon[32255]: pgmap v9606: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:47.751 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:18:47.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:47 smithi022 ceph-mon[31360]: pgmap v9607: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:47 smithi022 ceph-mon[40250]: pgmap v9607: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:47 smithi122 ceph-mon[32255]: pgmap v9607: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:48.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:48.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:48.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2269585707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2269585707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2269585707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:18:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:18:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:49 smithi022 ceph-mon[31360]: pgmap v9608: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:49 smithi022 ceph-mon[40250]: pgmap v9608: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:49 smithi122 ceph-mon[32255]: pgmap v9608: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:50] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:18:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:18:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:51 smithi022 ceph-mon[31360]: pgmap v9609: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:51 smithi022 ceph-mon[40250]: pgmap v9609: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:51 smithi122 ceph-mon[32255]: pgmap v9609: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:18:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:18:53.299 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:53 smithi022 ceph-mon[31360]: pgmap v9610: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/80575068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:53 smithi022 ceph-mon[40250]: pgmap v9610: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/80575068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:53 smithi122 ceph-mon[32255]: pgmap v9610: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/80575068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:18:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:55 smithi022 ceph-mon[31360]: pgmap v9611: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:55 smithi022 ceph-mon[40250]: pgmap v9611: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:55 smithi122 ceph-mon[32255]: pgmap v9611: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:18:57.771 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:18:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:57 smithi022 ceph-mon[31360]: pgmap v9612: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:57 smithi022 ceph-mon[40250]: pgmap v9612: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:57 smithi122 ceph-mon[32255]: pgmap v9612: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:18:58.667 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:18:59.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:18:59.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:18:59.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:18:59.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:18:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:19:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:59 smithi022 ceph-mon[31360]: pgmap v9613: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:18:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/491121489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:59 smithi022 ceph-mon[40250]: pgmap v9613: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:18:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/491121489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:59 smithi122 ceph-mon[32255]: pgmap v9613: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:18:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/491121489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:00] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:19:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:01 smithi022 ceph-mon[31360]: pgmap v9614: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:01 smithi022 ceph-mon[40250]: pgmap v9614: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:01 smithi122 ceph-mon[32255]: pgmap v9614: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:19:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:19:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:04.040 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:03 smithi022 ceph-mon[31360]: pgmap v9615: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:04.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:03 smithi022 ceph-mon[40250]: pgmap v9615: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:03 smithi122 ceph-mon[32255]: pgmap v9615: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:04.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1472716876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1472716876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1472716876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:05 smithi122 ceph-mon[32255]: pgmap v9616: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:05 smithi022 ceph-mon[31360]: pgmap v9616: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:05 smithi022 ceph-mon[40250]: pgmap v9616: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:19:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:07 smithi122 ceph-mon[32255]: pgmap v9617: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:07 smithi022 ceph-mon[31360]: pgmap v9617: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:07 smithi022 ceph-mon[40250]: pgmap v9617: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:09.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:19:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:19:09.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:09.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:09.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:09 smithi022 ceph-mon[31360]: pgmap v9618: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:10.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2126356334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:09 smithi022 ceph-mon[40250]: pgmap v9618: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2126356334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:09 smithi122 ceph-mon[32255]: pgmap v9618: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:19:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2126356334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:10] "GET /metrics HTTP/1.1" 200 202378 "" "Prometheus/2.18.1" 2021-12-15T23:19:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:11 smithi122 ceph-mon[32255]: pgmap v9619: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:11 smithi022 ceph-mon[31360]: pgmap v9619: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:11 smithi022 ceph-mon[40250]: pgmap v9619: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:19:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:19:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:13 smithi122 ceph-mon[32255]: pgmap v9620: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:13 smithi022 ceph-mon[31360]: pgmap v9620: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:13 smithi022 ceph-mon[40250]: pgmap v9620: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:14.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:19:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:19:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:19:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:19:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:19:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:19:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:19:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:19:15.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:15.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:15.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:19:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:19:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:19:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:19:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:15 smithi122 ceph-mon[32255]: pgmap v9621: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396901827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:15 smithi022 ceph-mon[31360]: pgmap v9621: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396901827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:15 smithi022 ceph-mon[40250]: pgmap v9621: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396901827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:19:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:17 smithi122 ceph-mon[32255]: pgmap v9622: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:17 smithi022 ceph-mon[31360]: pgmap v9622: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:17 smithi022 ceph-mon[40250]: pgmap v9622: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:19:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:19:20.137 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:20.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:19 smithi022 ceph-mon[31360]: pgmap v9623: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:20.138 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:19 smithi022 ceph-mon[40250]: pgmap v9623: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:19 smithi122 ceph-mon[32255]: pgmap v9623: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:20] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:19:20.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:20.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:20.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2015673462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:21.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2015673462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2015673462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:21 smithi122 ceph-mon[32255]: pgmap v9624: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:21 smithi022 ceph-mon[31360]: pgmap v9624: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:21 smithi022 ceph-mon[40250]: pgmap v9624: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:19:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:19:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:23 smithi122 ceph-mon[32255]: pgmap v9625: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:23 smithi022 ceph-mon[31360]: pgmap v9625: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:23 smithi022 ceph-mon[40250]: pgmap v9625: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:25.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:25 smithi122 ceph-mon[32255]: pgmap v9626: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:25 smithi022 ceph-mon[31360]: pgmap v9626: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:25 smithi022 ceph-mon[40250]: pgmap v9626: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3603141645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3603141645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3603141645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:19:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:27 smithi122 ceph-mon[32255]: pgmap v9627: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:27 smithi022 ceph-mon[31360]: pgmap v9627: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:27 smithi022 ceph-mon[40250]: pgmap v9627: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:28 smithi122 ceph-mon[32255]: pgmap v9628: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:28 smithi022 ceph-mon[31360]: pgmap v9628: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:28 smithi022 ceph-mon[40250]: pgmap v9628: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:19:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:19:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:30] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:19:30.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:30.877 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:30.877 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:31.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:31.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:31.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:31 smithi122 ceph-mon[32255]: pgmap v9629: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2405915300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:31 smithi022 ceph-mon[31360]: pgmap v9629: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2405915300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:31 smithi022 ceph-mon[40250]: pgmap v9629: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2405915300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:19:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:19:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:33 smithi122 ceph-mon[32255]: pgmap v9630: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:19:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:19:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:19:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:19:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[31360]: pgmap v9630: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:19:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:19:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:19:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:19:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[40250]: pgmap v9630: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:19:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:19:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:19:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:19:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:35 smithi122 ceph-mon[32255]: pgmap v9631: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:35 smithi022 ceph-mon[31360]: pgmap v9631: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:35 smithi022 ceph-mon[40250]: pgmap v9631: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:36.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:36.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:36.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:36.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4004250691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4004250691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4004250691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:37 smithi022 ceph-mon[31360]: pgmap v9632: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:19:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:37 smithi022 ceph-mon[40250]: pgmap v9632: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:37 smithi122 ceph-mon[32255]: pgmap v9632: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:39.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:19:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:19:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:39 smithi122 ceph-mon[32255]: pgmap v9633: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:39 smithi022 ceph-mon[31360]: pgmap v9633: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:39 smithi022 ceph-mon[40250]: pgmap v9633: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:19:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:41.613 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:41 smithi122 ceph-mon[32255]: pgmap v9634: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:41.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:41.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:41.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:41.983 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:41 smithi022 ceph-mon[31360]: pgmap v9634: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:41.984 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:41 smithi022 ceph-mon[40250]: pgmap v9634: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:42.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:19:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:19:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3320735340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3320735340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3320735340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:43 smithi122 ceph-mon[32255]: pgmap v9635: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:43 smithi022 ceph-mon[31360]: pgmap v9635: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:43 smithi022 ceph-mon[40250]: pgmap v9635: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:45 smithi122 ceph-mon[32255]: pgmap v9636: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:45 smithi022 ceph-mon[31360]: pgmap v9636: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:45 smithi022 ceph-mon[40250]: pgmap v9636: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:46.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:47.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:47.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:47.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:47.639 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:19:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:47 smithi122 ceph-mon[32255]: pgmap v9637: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/475363418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:47 smithi022 ceph-mon[31360]: pgmap v9637: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/475363418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:47 smithi022 ceph-mon[40250]: pgmap v9637: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/475363418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:49.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:19:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:19:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:49 smithi122 ceph-mon[32255]: pgmap v9638: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:49 smithi022 ceph-mon[31360]: pgmap v9638: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:49 smithi022 ceph-mon[40250]: pgmap v9638: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:50] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:19:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:19:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:51 smithi122 ceph-mon[32255]: pgmap v9639: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:51 smithi022 ceph-mon[31360]: pgmap v9639: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:51 smithi022 ceph-mon[40250]: pgmap v9639: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:52.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:19:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:19:52.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:52.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:52.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:53 smithi122 ceph-mon[32255]: pgmap v9640: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/691223749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:53 smithi022 ceph-mon[31360]: pgmap v9640: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/691223749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:53 smithi022 ceph-mon[40250]: pgmap v9640: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/691223749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:55 smithi122 ceph-mon[32255]: pgmap v9641: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:55 smithi022 ceph-mon[31360]: pgmap v9641: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:55 smithi022 ceph-mon[40250]: pgmap v9641: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:19:57.655 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:19:57.727 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:19:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:57 smithi122 ceph-mon[32255]: pgmap v9642: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:57 smithi022 ceph-mon[31360]: pgmap v9642: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:57 smithi022 ceph-mon[40250]: pgmap v9642: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:19:58.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:19:58.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:19:58.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:19:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/592064697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/592064697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/592064697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:19:59.662 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:19:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:19:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:19:59 smithi122 ceph-mon[32255]: pgmap v9643: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:19:59 smithi022 ceph-mon[31360]: pgmap v9643: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:19:59 smithi022 ceph-mon[40250]: pgmap v9643: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:00] "GET /metrics HTTP/1.1" 200 202383 "" "Prometheus/2.18.1" 2021-12-15T23:20:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T23:20:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T23:20:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T23:20:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:01 smithi122 ceph-mon[32255]: pgmap v9644: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:01 smithi022 ceph-mon[31360]: pgmap v9644: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:01 smithi022 ceph-mon[40250]: pgmap v9644: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:20:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:20:03.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:03 smithi022 ceph-mon[31360]: pgmap v9645: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2619431080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:03 smithi022 ceph-mon[40250]: pgmap v9645: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2619431080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:03 smithi122 ceph-mon[32255]: pgmap v9645: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2619431080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:05 smithi122 ceph-mon[32255]: pgmap v9646: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:05 smithi022 ceph-mon[31360]: pgmap v9646: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:05 smithi022 ceph-mon[40250]: pgmap v9646: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:07.676 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:20:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:07 smithi122 ceph-mon[32255]: pgmap v9647: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:07 smithi022 ceph-mon[31360]: pgmap v9647: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:07 smithi022 ceph-mon[40250]: pgmap v9647: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:08.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:08.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:08.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:08.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:09.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:20:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:20:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:09 smithi122 ceph-mon[32255]: pgmap v9648: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1016772903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:09 smithi022 ceph-mon[31360]: pgmap v9648: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1016772903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:09 smithi022 ceph-mon[40250]: pgmap v9648: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:10.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1016772903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:10] "GET /metrics HTTP/1.1" 200 202380 "" "Prometheus/2.18.1" 2021-12-15T23:20:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:11 smithi022 ceph-mon[31360]: pgmap v9649: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:11 smithi022 ceph-mon[40250]: pgmap v9649: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:11 smithi122 ceph-mon[32255]: pgmap v9649: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:20:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:20:13.842 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:13 smithi022 ceph-mon[31360]: pgmap v9650: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:13 smithi022 ceph-mon[40250]: pgmap v9650: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:13 smithi122 ceph-mon[32255]: pgmap v9650: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:14.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:14.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:14.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/500423024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/500423024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:15.139 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/500423024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:15 smithi122 ceph-mon[32255]: pgmap v9651: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:20:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:20:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:20:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[31360]: pgmap v9651: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:20:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:20:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:20:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:16.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[40250]: pgmap v9651: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:20:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:20:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:20:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:20:17.698 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:20:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:17 smithi022 ceph-mon[31360]: pgmap v9652: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:17 smithi022 ceph-mon[40250]: pgmap v9652: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:17 smithi122 ceph-mon[32255]: pgmap v9652: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:19.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:19.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:19.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:19.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:19.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:20:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:20:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:19 smithi022 ceph-mon[31360]: pgmap v9653: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2800351597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:19 smithi022 ceph-mon[40250]: pgmap v9653: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2800351597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:19 smithi122 ceph-mon[32255]: pgmap v9653: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2800351597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:20] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:20:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:21 smithi022 ceph-mon[31360]: pgmap v9654: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:21 smithi022 ceph-mon[40250]: pgmap v9654: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:21 smithi122 ceph-mon[32255]: pgmap v9654: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:20:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:20:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:23 smithi022 ceph-mon[31360]: pgmap v9655: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:23 smithi022 ceph-mon[40250]: pgmap v9655: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:23 smithi122 ceph-mon[32255]: pgmap v9655: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:24.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:25 smithi022 ceph-mon[31360]: pgmap v9656: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/651588232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:25 smithi022 ceph-mon[40250]: pgmap v9656: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/651588232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:25 smithi122 ceph-mon[32255]: pgmap v9656: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/651588232' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:27.715 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:20:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:27 smithi022 ceph-mon[31360]: pgmap v9657: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:27 smithi022 ceph-mon[40250]: pgmap v9657: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:27 smithi122 ceph-mon[32255]: pgmap v9657: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:29.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:20:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:20:29.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:29 smithi022 ceph-mon[31360]: pgmap v9658: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:29 smithi022 ceph-mon[40250]: pgmap v9658: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:29 smithi122 ceph-mon[32255]: pgmap v9658: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:30.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:30.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:30.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:30] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:20:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1334238887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1334238887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1334238887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:31 smithi022 ceph-mon[31360]: pgmap v9659: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:31 smithi022 ceph-mon[40250]: pgmap v9659: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:31 smithi122 ceph-mon[32255]: pgmap v9659: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:20:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:20:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[31360]: pgmap v9660: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:20:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:20:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:20:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:20:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[40250]: pgmap v9660: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:20:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:20:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:20:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:20:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:33 smithi122 ceph-mon[32255]: pgmap v9660: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:20:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:20:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:20:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:20:35.336 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:35.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:35 smithi022 ceph-mon[40250]: pgmap v9661: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508605256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:35 smithi022 ceph-mon[31360]: pgmap v9661: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508605256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:35 smithi122 ceph-mon[32255]: pgmap v9661: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508605256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:37.733 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:20:37.734 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:37 smithi022 ceph-mon[31360]: pgmap v9662: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:37 smithi022 ceph-mon[40250]: pgmap v9662: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:37 smithi122 ceph-mon[32255]: pgmap v9662: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:39.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:20:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:20:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:39 smithi022 ceph-mon[31360]: pgmap v9663: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:39 smithi022 ceph-mon[40250]: pgmap v9663: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:39 smithi122 ceph-mon[32255]: pgmap v9663: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:40] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:20:40.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:41.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:41.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:41.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:41 smithi022 ceph-mon[31360]: pgmap v9664: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3707311004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:41 smithi022 ceph-mon[40250]: pgmap v9664: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3707311004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:41 smithi122 ceph-mon[32255]: pgmap v9664: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3707311004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:20:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:20:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:43 smithi022 ceph-mon[31360]: pgmap v9665: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:43 smithi022 ceph-mon[40250]: pgmap v9665: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:43 smithi122 ceph-mon[32255]: pgmap v9665: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:45 smithi022 ceph-mon[31360]: pgmap v9666: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:45 smithi022 ceph-mon[40250]: pgmap v9666: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:46.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:45 smithi122 ceph-mon[32255]: pgmap v9666: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3419761595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3419761595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3419761595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:47.754 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:20:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:47 smithi022 ceph-mon[31360]: pgmap v9667: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:47 smithi022 ceph-mon[40250]: pgmap v9667: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:47 smithi122 ceph-mon[32255]: pgmap v9667: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:20:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:20:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:49 smithi022 ceph-mon[31360]: pgmap v9668: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:49 smithi022 ceph-mon[40250]: pgmap v9668: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:49 smithi122 ceph-mon[32255]: pgmap v9668: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:20:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:20:51.450 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:51.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:51.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:51.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:51 smithi022 ceph-mon[31360]: pgmap v9669: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:51 smithi022 ceph-mon[40250]: pgmap v9669: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:51 smithi122 ceph-mon[32255]: pgmap v9669: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:20:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:20:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1407040743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1407040743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1407040743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:53 smithi022 ceph-mon[31360]: pgmap v9670: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:53 smithi022 ceph-mon[40250]: pgmap v9670: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:53 smithi122 ceph-mon[32255]: pgmap v9670: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:55 smithi022 ceph-mon[31360]: pgmap v9671: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:55 smithi022 ceph-mon[40250]: pgmap v9671: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:55 smithi122 ceph-mon[32255]: pgmap v9671: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:20:56.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:20:57.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:20:57.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:20:57.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:20:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:20:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:57 smithi022 ceph-mon[31360]: pgmap v9672: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3027182143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:57 smithi022 ceph-mon[40250]: pgmap v9672: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3027182143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:57 smithi122 ceph-mon[32255]: pgmap v9672: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:20:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3027182143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:20:59.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:20:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:21:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:20:59 smithi022 ceph-mon[31360]: pgmap v9673: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:20:59 smithi022 ceph-mon[40250]: pgmap v9673: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:20:59 smithi122 ceph-mon[32255]: pgmap v9673: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:21:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:01 smithi122 ceph-mon[32255]: pgmap v9674: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:02.198 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:02.199 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:01 smithi022 ceph-mon[31360]: pgmap v9674: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:02.199 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:01 smithi022 ceph-mon[40250]: pgmap v9674: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:02.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:02.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:02.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:21:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:21:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/893781343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/893781343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/893781343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:03 smithi122 ceph-mon[32255]: pgmap v9675: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:03 smithi022 ceph-mon[31360]: pgmap v9675: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:03 smithi022 ceph-mon[40250]: pgmap v9675: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:05 smithi122 ceph-mon[32255]: pgmap v9676: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:05 smithi022 ceph-mon[31360]: pgmap v9676: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:05 smithi022 ceph-mon[40250]: pgmap v9676: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:07.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:21:07.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:07.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:07.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:07 smithi122 ceph-mon[32255]: pgmap v9677: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:07 smithi022 ceph-mon[31360]: pgmap v9677: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:07 smithi022 ceph-mon[40250]: pgmap v9677: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1440480204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1440480204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1440480204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:09.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:21:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:21:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:09 smithi122 ceph-mon[32255]: pgmap v9678: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:09 smithi022 ceph-mon[31360]: pgmap v9678: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:09 smithi022 ceph-mon[40250]: pgmap v9678: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:10] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:21:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:11 smithi122 ceph-mon[32255]: pgmap v9679: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:11 smithi022 ceph-mon[31360]: pgmap v9679: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:11 smithi022 ceph-mon[40250]: pgmap v9679: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:21:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:21:12.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:13.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:13.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:13.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:13 smithi122 ceph-mon[32255]: pgmap v9680: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/765032913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:13 smithi022 ceph-mon[31360]: pgmap v9680: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/765032913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:13 smithi022 ceph-mon[40250]: pgmap v9680: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/765032913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:15 smithi122 ceph-mon[32255]: pgmap v9681: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:21:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:21:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:21:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[31360]: pgmap v9681: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:21:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:21:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:21:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[40250]: pgmap v9681: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:21:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:21:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:21:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:17 smithi022 ceph-mon[31360]: pgmap v9682: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:17.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:21:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:17 smithi022 ceph-mon[40250]: pgmap v9682: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:21:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:17 smithi122 ceph-mon[32255]: pgmap v9682: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:21:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:21:18.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:18.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:18.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:18.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:19 smithi122 ceph-mon[32255]: pgmap v9683: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1476675041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:19 smithi022 ceph-mon[31360]: pgmap v9683: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1476675041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:19 smithi022 ceph-mon[40250]: pgmap v9683: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1476675041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:19.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:21:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:21:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:21:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:21 smithi122 ceph-mon[32255]: pgmap v9684: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:21 smithi022 ceph-mon[31360]: pgmap v9684: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:21 smithi022 ceph-mon[40250]: pgmap v9684: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:21:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:21:23.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:23 smithi122 ceph-mon[32255]: pgmap v9685: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:23 smithi022 ceph-mon[31360]: pgmap v9685: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:23 smithi022 ceph-mon[40250]: pgmap v9685: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3806595952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3806595952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3806595952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:25 smithi122 ceph-mon[32255]: pgmap v9686: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:25 smithi022 ceph-mon[31360]: pgmap v9686: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:25 smithi022 ceph-mon[40250]: pgmap v9686: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:27.644 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:21:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:27 smithi122 ceph-mon[32255]: pgmap v9687: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:27 smithi022 ceph-mon[31360]: pgmap v9687: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:27 smithi022 ceph-mon[40250]: pgmap v9687: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:29.056 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:29.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:29.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:29.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:29 smithi022 ceph-mon[31360]: pgmap v9688: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/11932659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:29 smithi022 ceph-mon[40250]: pgmap v9688: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/11932659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:29.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:21:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:21:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:29 smithi122 ceph-mon[32255]: pgmap v9688: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:29.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/11932659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:21:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:31 smithi122 ceph-mon[32255]: pgmap v9689: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:31 smithi022 ceph-mon[31360]: pgmap v9689: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:31 smithi022 ceph-mon[40250]: pgmap v9689: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:21:32.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:21:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:33 smithi122 ceph-mon[32255]: pgmap v9690: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:21:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:21:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:21:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:21:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[31360]: pgmap v9690: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:21:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:21:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:21:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:21:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[40250]: pgmap v9690: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:21:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:21:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:21:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:21:34.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:34.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:34.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:34.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:35 smithi122 ceph-mon[32255]: pgmap v9691: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3312970838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:35 smithi022 ceph-mon[31360]: pgmap v9691: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3312970838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:35 smithi022 ceph-mon[40250]: pgmap v9691: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3312970838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:37.661 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:21:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:37 smithi122 ceph-mon[32255]: pgmap v9692: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:37 smithi022 ceph-mon[31360]: pgmap v9692: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:37 smithi022 ceph-mon[40250]: pgmap v9692: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:39.796 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:39 smithi122 ceph-mon[32255]: pgmap v9693: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:39.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:21:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:21:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:39 smithi022 ceph-mon[31360]: pgmap v9693: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:39 smithi022 ceph-mon[40250]: pgmap v9693: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:40.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:40.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:40.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:40] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:21:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1836979867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1836979867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1836979867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:41 smithi122 ceph-mon[32255]: pgmap v9694: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:41 smithi022 ceph-mon[31360]: pgmap v9694: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:41 smithi022 ceph-mon[40250]: pgmap v9694: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:21:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:21:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:43 smithi122 ceph-mon[32255]: pgmap v9695: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:43 smithi022 ceph-mon[31360]: pgmap v9695: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:43 smithi022 ceph-mon[40250]: pgmap v9695: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:45.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:45.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:45.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:45 smithi122 ceph-mon[32255]: pgmap v9696: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1491197848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:45 smithi022 ceph-mon[31360]: pgmap v9696: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1491197848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:45 smithi022 ceph-mon[40250]: pgmap v9696: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1491197848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:47.679 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:21:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:47 smithi122 ceph-mon[32255]: pgmap v9697: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:47 smithi022 ceph-mon[31360]: pgmap v9697: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:47 smithi022 ceph-mon[40250]: pgmap v9697: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:21:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:21:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:49 smithi122 ceph-mon[32255]: pgmap v9698: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:49 smithi022 ceph-mon[31360]: pgmap v9698: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:49 smithi022 ceph-mon[40250]: pgmap v9698: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:50] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:21:50.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:50.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:50.903 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:50.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:21:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:51 smithi022 ceph-mon[31360]: pgmap v9699: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3592678974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:51 smithi022 ceph-mon[40250]: pgmap v9699: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3592678974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:51 smithi122 ceph-mon[32255]: pgmap v9699: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3592678974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:21:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:21:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:53 smithi022 ceph-mon[31360]: pgmap v9700: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:53 smithi022 ceph-mon[40250]: pgmap v9700: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:53 smithi122 ceph-mon[32255]: pgmap v9700: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:55.904 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:21:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:55 smithi022 ceph-mon[31360]: pgmap v9701: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:55 smithi022 ceph-mon[40250]: pgmap v9701: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:55 smithi122 ceph-mon[32255]: pgmap v9701: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:21:56.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:21:56.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:21:56.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:21:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1356111836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1356111836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1356111836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:21:57.700 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:21:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:57 smithi022 ceph-mon[31360]: pgmap v9702: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:57 smithi022 ceph-mon[40250]: pgmap v9702: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:57 smithi122 ceph-mon[32255]: pgmap v9702: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:21:59 smithi122 ceph-mon[32255]: pgmap v9703: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:21:59.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:21:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:22:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:21:59 smithi022 ceph-mon[31360]: pgmap v9703: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:21:59 smithi022 ceph-mon[40250]: pgmap v9703: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:00] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:22:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:01.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:01.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:01.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:01.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:01 smithi022 ceph-mon[31360]: pgmap v9704: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3551547749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:01 smithi022 ceph-mon[40250]: pgmap v9704: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3551547749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:01 smithi122 ceph-mon[32255]: pgmap v9704: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3551547749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:22:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:22:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:03 smithi022 ceph-mon[31360]: pgmap v9705: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:03 smithi022 ceph-mon[40250]: pgmap v9705: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:03 smithi122 ceph-mon[32255]: pgmap v9705: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:05 smithi022 ceph-mon[31360]: pgmap v9706: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:05 smithi022 ceph-mon[40250]: pgmap v9706: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:05 smithi122 ceph-mon[32255]: pgmap v9706: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:06.643 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:07.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:07.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:07.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:07.717 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:22:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:07 smithi022 ceph-mon[31360]: pgmap v9707: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/632631446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:07 smithi022 ceph-mon[40250]: pgmap v9707: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/632631446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:07 smithi122 ceph-mon[32255]: pgmap v9707: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/632631446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:09 smithi122 ceph-mon[32255]: pgmap v9708: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:09.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:22:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:22:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:09 smithi022 ceph-mon[31360]: pgmap v9708: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:09 smithi022 ceph-mon[40250]: pgmap v9708: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:10] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:22:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:12.016 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:12.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:11 smithi022 ceph-mon[31360]: pgmap v9709: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:12.018 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:11 smithi022 ceph-mon[40250]: pgmap v9709: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:11 smithi122 ceph-mon[32255]: pgmap v9709: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:12.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:12.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:12.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:22:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:22:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2904325880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2904325880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2904325880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:13 smithi022 ceph-mon[31360]: pgmap v9710: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:13 smithi022 ceph-mon[40250]: pgmap v9710: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:13 smithi122 ceph-mon[32255]: pgmap v9710: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:15 smithi022 ceph-mon[31360]: pgmap v9711: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:15 smithi022 ceph-mon[40250]: pgmap v9711: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:16.109 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:15 smithi122 ceph-mon[32255]: pgmap v9711: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:22:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:22:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:22:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:22:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:22:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:22:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:22:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:22:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:22:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:22:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:22:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:22:17.387 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:17.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:22:17.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:17.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:17.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:17 smithi022 ceph-mon[31360]: pgmap v9712: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:17 smithi022 ceph-mon[40250]: pgmap v9712: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:17 smithi122 ceph-mon[32255]: pgmap v9712: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2988189725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2988189725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2988189725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:19.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:22:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:22:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:19 smithi022 ceph-mon[31360]: pgmap v9713: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:19 smithi022 ceph-mon[40250]: pgmap v9713: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:19 smithi122 ceph-mon[32255]: pgmap v9713: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:20] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:22:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:21 smithi022 ceph-mon[31360]: pgmap v9714: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:21 smithi022 ceph-mon[40250]: pgmap v9714: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:21 smithi122 ceph-mon[32255]: pgmap v9714: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:22:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:22:22.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:23.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:23.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:23.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:23 smithi022 ceph-mon[31360]: pgmap v9715: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:24.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1017240678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:23 smithi022 ceph-mon[40250]: pgmap v9715: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1017240678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:23 smithi122 ceph-mon[32255]: pgmap v9715: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1017240678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:25 smithi022 ceph-mon[31360]: pgmap v9716: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:25 smithi022 ceph-mon[40250]: pgmap v9716: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:25 smithi122 ceph-mon[32255]: pgmap v9716: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:27.762 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:22:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:27 smithi022 ceph-mon[31360]: pgmap v9717: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:27 smithi022 ceph-mon[40250]: pgmap v9717: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:28.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:27 smithi122 ceph-mon[32255]: pgmap v9717: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:28.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:28.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:28.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1617982579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1617982579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1617982579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:29.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:22:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:22:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:29 smithi022 ceph-mon[31360]: pgmap v9718: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:29 smithi022 ceph-mon[40250]: pgmap v9718: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:29 smithi122 ceph-mon[32255]: pgmap v9718: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:30] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:22:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:31 smithi022 ceph-mon[31360]: pgmap v9719: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:31 smithi022 ceph-mon[40250]: pgmap v9719: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:31 smithi122 ceph-mon[32255]: pgmap v9719: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:22:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:22:33.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:33.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:33.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:33.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[31360]: pgmap v9720: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:22:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:22:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:22:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:22:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[40250]: pgmap v9720: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:22:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:22:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:22:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:22:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:33 smithi122 ceph-mon[32255]: pgmap v9720: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:22:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:22:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:22:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:22:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2756692396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2756692396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2756692396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:35 smithi122 ceph-mon[32255]: pgmap v9721: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:35 smithi022 ceph-mon[31360]: pgmap v9721: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:35 smithi022 ceph-mon[40250]: pgmap v9721: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:22:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:37 smithi122 ceph-mon[32255]: pgmap v9722: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:37 smithi022 ceph-mon[31360]: pgmap v9722: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:37 smithi022 ceph-mon[40250]: pgmap v9722: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:38.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:39.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:39.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:39.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:39.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:22:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:22:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:39 smithi122 ceph-mon[32255]: pgmap v9723: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3283783501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:39 smithi022 ceph-mon[31360]: pgmap v9723: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3283783501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:39 smithi022 ceph-mon[40250]: pgmap v9723: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3283783501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:40] "GET /metrics HTTP/1.1" 200 202382 "" "Prometheus/2.18.1" 2021-12-15T23:22:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:41 smithi122 ceph-mon[32255]: pgmap v9724: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:41 smithi022 ceph-mon[31360]: pgmap v9724: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:41 smithi022 ceph-mon[40250]: pgmap v9724: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:22:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:22:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:43 smithi122 ceph-mon[32255]: pgmap v9725: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:44.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:44.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:43 smithi022 ceph-mon[31360]: pgmap v9725: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:44.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:43 smithi022 ceph-mon[40250]: pgmap v9725: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:44.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:44.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:44.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1446670606' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1446670606' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1446670606' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:45 smithi122 ceph-mon[32255]: pgmap v9726: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:45 smithi022 ceph-mon[31360]: pgmap v9726: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:45 smithi022 ceph-mon[40250]: pgmap v9726: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:22:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:47 smithi122 ceph-mon[32255]: pgmap v9727: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:47 smithi022 ceph-mon[31360]: pgmap v9727: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:47 smithi022 ceph-mon[40250]: pgmap v9727: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:49.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:49.808 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:22:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:22:49.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:49.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:49.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:49.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:49 smithi022 ceph-mon[31360]: pgmap v9728: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:49.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:49 smithi022 ceph-mon[40250]: pgmap v9728: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:49 smithi122 ceph-mon[32255]: pgmap v9728: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:50] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T23:22:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:51.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3400889109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3400889109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:22:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3400889109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:51 smithi122 ceph-mon[32255]: pgmap v9729: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:51 smithi022 ceph-mon[31360]: pgmap v9729: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:51 smithi022 ceph-mon[40250]: pgmap v9729: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:22:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:22:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:53 smithi122 ceph-mon[32255]: pgmap v9730: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:53 smithi022 ceph-mon[31360]: pgmap v9730: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:53 smithi022 ceph-mon[40250]: pgmap v9730: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:54.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:22:55.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:22:55.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:22:55.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:22:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:55 smithi122 ceph-mon[32255]: pgmap v9731: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1607827132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:55 smithi022 ceph-mon[31360]: pgmap v9731: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1607827132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:55 smithi022 ceph-mon[40250]: pgmap v9731: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:22:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1607827132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:22:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:22:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:57 smithi122 ceph-mon[32255]: pgmap v9732: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:57 smithi022 ceph-mon[31360]: pgmap v9732: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:57 smithi022 ceph-mon[40250]: pgmap v9732: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:22:59.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:22:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:23:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:22:59 smithi122 ceph-mon[32255]: pgmap v9733: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:22:59 smithi022 ceph-mon[31360]: pgmap v9733: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:22:59 smithi022 ceph-mon[40250]: pgmap v9733: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:00.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:00] "GET /metrics HTTP/1.1" 200 202379 "" "Prometheus/2.18.1" 2021-12-15T23:23:00.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:00.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:00.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/903527170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/903527170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/903527170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:01 smithi122 ceph-mon[32255]: pgmap v9734: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:01 smithi022 ceph-mon[31360]: pgmap v9734: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:01 smithi022 ceph-mon[40250]: pgmap v9734: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:23:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:23:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:03 smithi122 ceph-mon[32255]: pgmap v9735: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:03 smithi022 ceph-mon[31360]: pgmap v9735: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:03 smithi022 ceph-mon[40250]: pgmap v9735: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:05.719 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:05 smithi022 ceph-mon[31360]: pgmap v9736: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:05 smithi022 ceph-mon[40250]: pgmap v9736: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:06.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:06.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:06.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:05 smithi122 ceph-mon[32255]: pgmap v9736: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3906831622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3906831622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3906831622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:23:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:07 smithi122 ceph-mon[32255]: pgmap v9737: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:07 smithi022 ceph-mon[31360]: pgmap v9737: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:07 smithi022 ceph-mon[40250]: pgmap v9737: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:09.844 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:23:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:23:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:09 smithi122 ceph-mon[32255]: pgmap v9738: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:09 smithi022 ceph-mon[31360]: pgmap v9738: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:09 smithi022 ceph-mon[40250]: pgmap v9738: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:10] "GET /metrics HTTP/1.1" 200 202384 "" "Prometheus/2.18.1" 2021-12-15T23:23:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:11.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:11.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:11.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:11 smithi122 ceph-mon[32255]: pgmap v9739: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1001866244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:11 smithi022 ceph-mon[31360]: pgmap v9739: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1001866244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:11 smithi022 ceph-mon[40250]: pgmap v9739: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1001866244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:23:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:23:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:13 smithi122 ceph-mon[32255]: pgmap v9740: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:13 smithi022 ceph-mon[31360]: pgmap v9740: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:13 smithi022 ceph-mon[40250]: pgmap v9740: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:15 smithi122 ceph-mon[32255]: pgmap v9741: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:15 smithi022 ceph-mon[31360]: pgmap v9741: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:15 smithi022 ceph-mon[40250]: pgmap v9741: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:16.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:16.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:16.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:16.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:23:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:23:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:23:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2810738569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:23:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:23:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:23:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2810738569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:23:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:23:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:23:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2810738569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:23:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:17 smithi122 ceph-mon[32255]: pgmap v9742: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:23:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:17 smithi022 ceph-mon[40250]: pgmap v9742: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:23:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:17 smithi022 ceph-mon[31360]: pgmap v9742: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:23:19.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:23:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:23:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:19 smithi122 ceph-mon[32255]: pgmap v9743: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:19 smithi022 ceph-mon[31360]: pgmap v9743: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:19 smithi022 ceph-mon[40250]: pgmap v9743: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:20] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:23:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:21.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:21 smithi122 ceph-mon[32255]: pgmap v9744: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:22.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:22.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:22.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:22.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:21 smithi022 ceph-mon[40250]: pgmap v9744: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:22.207 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:21 smithi022 ceph-mon[31360]: pgmap v9744: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:23:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:23:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/724236306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/724236306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/724236306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:23 smithi122 ceph-mon[32255]: pgmap v9745: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:23 smithi022 ceph-mon[31360]: pgmap v9745: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:23 smithi022 ceph-mon[40250]: pgmap v9745: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:25 smithi122 ceph-mon[32255]: pgmap v9746: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:26.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:25 smithi022 ceph-mon[31360]: pgmap v9746: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:26.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:25 smithi022 ceph-mon[40250]: pgmap v9746: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:27.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:23:27.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:27.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:27.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:27 smithi122 ceph-mon[32255]: pgmap v9747: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:27.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/141000543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:27 smithi022 ceph-mon[31360]: pgmap v9747: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:28.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/141000543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:27 smithi022 ceph-mon[40250]: pgmap v9747: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/141000543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:29 smithi122 ceph-mon[32255]: pgmap v9748: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:29.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:23:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:23:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:29 smithi022 ceph-mon[31360]: pgmap v9748: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:29 smithi022 ceph-mon[40250]: pgmap v9748: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:30] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:23:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:31.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:31 smithi122 ceph-mon[32255]: pgmap v9749: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:31 smithi022 ceph-mon[31360]: pgmap v9749: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:31 smithi022 ceph-mon[40250]: pgmap v9749: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:32.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:23:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:23:32.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:32.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:32.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:33 smithi122 ceph-mon[32255]: pgmap v9750: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/419812795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:23:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:23:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:23:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:23:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[31360]: pgmap v9750: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/419812795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:23:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:23:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:23:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:23:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[40250]: pgmap v9750: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/419812795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:23:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:23:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:23:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:23:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:35 smithi122 ceph-mon[32255]: pgmap v9751: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:35 smithi022 ceph-mon[31360]: pgmap v9751: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:35 smithi022 ceph-mon[40250]: pgmap v9751: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:37.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:23:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:37 smithi122 ceph-mon[32255]: pgmap v9752: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:37.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:37.946 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:37 smithi022 ceph-mon[31360]: pgmap v9752: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:37.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:37 smithi022 ceph-mon[40250]: pgmap v9752: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:38.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:38.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:38.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2848350890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2848350890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2848350890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:39 smithi122 ceph-mon[32255]: pgmap v9753: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:39.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:23:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:23:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:39 smithi022 ceph-mon[31360]: pgmap v9753: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:39 smithi022 ceph-mon[40250]: pgmap v9753: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:23:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:41.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:41 smithi122 ceph-mon[32255]: pgmap v9754: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:41 smithi022 ceph-mon[31360]: pgmap v9754: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:41 smithi022 ceph-mon[40250]: pgmap v9754: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:23:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:23:43.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:43.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:43.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:43.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:43 smithi022 ceph-mon[31360]: pgmap v9755: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3736930448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:43 smithi022 ceph-mon[40250]: pgmap v9755: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3736930448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:43 smithi122 ceph-mon[32255]: pgmap v9755: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3736930448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:45 smithi022 ceph-mon[31360]: pgmap v9756: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:45 smithi022 ceph-mon[40250]: pgmap v9756: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:45 smithi122 ceph-mon[32255]: pgmap v9756: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:47.693 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:23:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:47 smithi022 ceph-mon[40250]: pgmap v9757: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:47 smithi022 ceph-mon[31360]: pgmap v9757: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:47 smithi122 ceph-mon[32255]: pgmap v9757: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:48.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:49.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:49.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:49.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:23:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:23:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:49 smithi122 ceph-mon[32255]: pgmap v9758: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2435209495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:49 smithi022 ceph-mon[31360]: pgmap v9758: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2435209495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:49 smithi022 ceph-mon[40250]: pgmap v9758: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2435209495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:50] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:23:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:23:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:51 smithi022 ceph-mon[40250]: pgmap v9759: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:51 smithi022 ceph-mon[31360]: pgmap v9759: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:51 smithi122 ceph-mon[32255]: pgmap v9759: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:23:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:23:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:53 smithi022 ceph-mon[40250]: pgmap v9760: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:53 smithi022 ceph-mon[31360]: pgmap v9760: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:54.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:53 smithi122 ceph-mon[32255]: pgmap v9760: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:54.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:54.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:54.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2568601055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2568601055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2568601055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:23:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:55 smithi022 ceph-mon[31360]: pgmap v9761: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:55 smithi022 ceph-mon[40250]: pgmap v9761: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:55 smithi122 ceph-mon[32255]: pgmap v9761: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:23:57.720 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:23:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:57 smithi022 ceph-mon[31360]: pgmap v9762: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:57 smithi022 ceph-mon[40250]: pgmap v9762: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:57 smithi122 ceph-mon[32255]: pgmap v9762: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:59.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:23:59.723 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:23:59 smithi022 ceph-mon[40250]: pgmap v9763: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:59.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:23:59.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:23:59.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:23:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:23:59 smithi122 ceph-mon[32255]: pgmap v9763: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:23:59.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:23:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:24:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:23:59 smithi022 ceph-mon[31360]: pgmap v9763: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:00] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:24:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/403412195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/403412195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/403412195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:01 smithi022 ceph-mon[31360]: pgmap v9764: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:01 smithi022 ceph-mon[40250]: pgmap v9764: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:01 smithi122 ceph-mon[32255]: pgmap v9764: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:24:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:24:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:03 smithi022 ceph-mon[31360]: pgmap v9765: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:03 smithi022 ceph-mon[40250]: pgmap v9765: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:03 smithi122 ceph-mon[32255]: pgmap v9765: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:04.804 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:05.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:05.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:05.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:05 smithi022 ceph-mon[31360]: pgmap v9766: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1808591058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:05 smithi022 ceph-mon[40250]: pgmap v9766: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1808591058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:05 smithi122 ceph-mon[32255]: pgmap v9766: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1808591058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:07.737 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:24:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:07 smithi022 ceph-mon[31360]: pgmap v9767: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:07 smithi022 ceph-mon[40250]: pgmap v9767: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:07 smithi122 ceph-mon[32255]: pgmap v9767: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:09 smithi122 ceph-mon[32255]: pgmap v9768: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:09.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:24:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:24:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:09 smithi022 ceph-mon[31360]: pgmap v9768: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:09 smithi022 ceph-mon[40250]: pgmap v9768: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:10.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:10] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:24:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4277291513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4277291513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4277291513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:11 smithi022 ceph-mon[31360]: pgmap v9769: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:11 smithi022 ceph-mon[40250]: pgmap v9769: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:11 smithi122 ceph-mon[32255]: pgmap v9769: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:24:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:24:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:13 smithi022 ceph-mon[31360]: pgmap v9770: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:13 smithi022 ceph-mon[40250]: pgmap v9770: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:13 smithi122 ceph-mon[32255]: pgmap v9770: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:15.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:15.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:15.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:15.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:15.912 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:15 smithi022 ceph-mon[31360]: pgmap v9771: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:15.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:15 smithi022 ceph-mon[40250]: pgmap v9771: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:15 smithi122 ceph-mon[32255]: pgmap v9771: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:17.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3373685765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3373685765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3373685765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:17.757 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:24:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[31360]: pgmap v9772: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:24:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:24:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:24:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[40250]: pgmap v9772: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:24:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:24:18.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:24:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:17 smithi122 ceph-mon[32255]: pgmap v9772: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:24:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:24:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:24:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:19 smithi122 ceph-mon[32255]: pgmap v9773: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:19.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:24:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:24:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:19 smithi022 ceph-mon[31360]: pgmap v9773: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:19 smithi022 ceph-mon[40250]: pgmap v9773: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:24:20.913 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:20.913 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:20.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:20.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:20.915 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:21.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:21.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:21.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: pgmap v9774: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:21.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:21.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1270250732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:21.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: pgmap v9774: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1270250732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:22.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: pgmap v9774: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:22.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:24:22.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:22.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1270250732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:22.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:24:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:24:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:22 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3138M 2021-12-15T23:24:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:22 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3138M 2021-12-15T23:24:23.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:22 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3138M 2021-12-15T23:24:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:24:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:23 smithi022 ceph-mon[31360]: pgmap v9775: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:23 smithi022 ceph-mon[40250]: pgmap v9775: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:23 smithi122 ceph-mon[32255]: pgmap v9775: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:25 smithi022 ceph-mon[31360]: pgmap v9776: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:25 smithi022 ceph-mon[40250]: pgmap v9776: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:25 smithi122 ceph-mon[32255]: pgmap v9776: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:26.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3523186289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3523186289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3523186289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:24:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:27 smithi122 ceph-mon[32255]: pgmap v9777: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:27 smithi022 ceph-mon[31360]: pgmap v9777: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:27 smithi022 ceph-mon[40250]: pgmap v9777: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:29 smithi122 ceph-mon[32255]: pgmap v9778: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:29.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:24:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:24:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:29 smithi022 ceph-mon[31360]: pgmap v9778: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:29 smithi022 ceph-mon[40250]: pgmap v9778: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:24:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:31.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:32.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:32.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:32.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:32.016 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:31 smithi022 ceph-mon[31360]: pgmap v9779: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:32.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:31 smithi022 ceph-mon[40250]: pgmap v9779: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:31 smithi122 ceph-mon[32255]: pgmap v9779: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:24:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:24:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3138863887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3138863887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3138863887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:33 smithi122 ceph-mon[32255]: pgmap v9780: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:24:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:24:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:24:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:24:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[31360]: pgmap v9780: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:24:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:24:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:24:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:24:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[40250]: pgmap v9780: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:24:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:24:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:24:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:24:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:35 smithi122 ceph-mon[32255]: pgmap v9781: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:35 smithi022 ceph-mon[31360]: pgmap v9781: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:35 smithi022 ceph-mon[40250]: pgmap v9781: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:37.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:37.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:37.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:37.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:24:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:37 smithi122 ceph-mon[32255]: pgmap v9782: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1004203031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:37 smithi022 ceph-mon[31360]: pgmap v9782: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1004203031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:37 smithi022 ceph-mon[40250]: pgmap v9782: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1004203031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:39 smithi122 ceph-mon[32255]: pgmap v9783: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:39.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:24:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:24:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:39 smithi022 ceph-mon[31360]: pgmap v9783: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:39 smithi022 ceph-mon[40250]: pgmap v9783: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:40] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:24:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:41 smithi122 ceph-mon[32255]: pgmap v9784: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:41 smithi022 ceph-mon[31360]: pgmap v9784: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:41 smithi022 ceph-mon[40250]: pgmap v9784: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:24:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:24:42.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:42.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:42.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/913288334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/913288334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/913288334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:43 smithi122 ceph-mon[32255]: pgmap v9785: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:43 smithi022 ceph-mon[31360]: pgmap v9785: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:43 smithi022 ceph-mon[40250]: pgmap v9785: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:45 smithi122 ceph-mon[32255]: pgmap v9786: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:45 smithi022 ceph-mon[31360]: pgmap v9786: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:45 smithi022 ceph-mon[40250]: pgmap v9786: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:47.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:47.756 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:24:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:47 smithi022 ceph-mon[31360]: pgmap v9787: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:47 smithi022 ceph-mon[40250]: pgmap v9787: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:48.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:48.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:48.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:47 smithi122 ceph-mon[32255]: pgmap v9787: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2169554846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2169554846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2169554846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:49.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:24:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:24:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:49 smithi122 ceph-mon[32255]: pgmap v9788: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:49 smithi022 ceph-mon[31360]: pgmap v9788: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:49 smithi022 ceph-mon[40250]: pgmap v9788: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:50] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:24:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:24:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:51 smithi122 ceph-mon[32255]: pgmap v9789: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:51 smithi022 ceph-mon[31360]: pgmap v9789: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:51 smithi022 ceph-mon[40250]: pgmap v9789: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:24:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:24:53.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:53.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:53.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:53.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:53 smithi122 ceph-mon[32255]: pgmap v9790: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2922043485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:53 smithi022 ceph-mon[31360]: pgmap v9790: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2922043485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:53 smithi022 ceph-mon[40250]: pgmap v9790: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2922043485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:24:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:55 smithi122 ceph-mon[32255]: pgmap v9791: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:55 smithi022 ceph-mon[31360]: pgmap v9791: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:55 smithi022 ceph-mon[40250]: pgmap v9791: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:24:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:24:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:57 smithi122 ceph-mon[32255]: pgmap v9792: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:57 smithi022 ceph-mon[31360]: pgmap v9792: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:57 smithi022 ceph-mon[40250]: pgmap v9792: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:24:58.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:24:58.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:24:58.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:24:58.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:24:59.841 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:24:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:25:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:59 smithi122 ceph-mon[32255]: pgmap v9793: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:24:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3777331397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:59 smithi022 ceph-mon[31360]: pgmap v9793: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:00.335 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:24:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3777331397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:00.335 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:59 smithi022 ceph-mon[40250]: pgmap v9793: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:00.336 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:24:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3777331397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:00] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:25:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:01 smithi122 ceph-mon[32255]: pgmap v9794: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:01 smithi022 ceph-mon[31360]: pgmap v9794: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:01 smithi022 ceph-mon[40250]: pgmap v9794: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:25:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:25:03.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:03 smithi122 ceph-mon[32255]: pgmap v9795: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:04.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:04.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:04.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:04.236 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:03 smithi022 ceph-mon[31360]: pgmap v9795: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:04.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:03 smithi022 ceph-mon[40250]: pgmap v9795: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2507985901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2507985901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2507985901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:05 smithi122 ceph-mon[32255]: pgmap v9796: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:05 smithi022 ceph-mon[31360]: pgmap v9796: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:05 smithi022 ceph-mon[40250]: pgmap v9796: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:25:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:07 smithi122 ceph-mon[32255]: pgmap v9797: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:07 smithi022 ceph-mon[31360]: pgmap v9797: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:07 smithi022 ceph-mon[40250]: pgmap v9797: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:09.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:09.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:09.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:09.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:09.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:25:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:25:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:09 smithi122 ceph-mon[32255]: pgmap v9798: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2796242442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:09 smithi022 ceph-mon[31360]: pgmap v9798: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2796242442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:09 smithi022 ceph-mon[40250]: pgmap v9798: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2796242442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:10] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:25:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:11 smithi122 ceph-mon[32255]: pgmap v9799: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:11 smithi022 ceph-mon[31360]: pgmap v9799: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:11 smithi022 ceph-mon[40250]: pgmap v9799: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:25:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:25:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:13 smithi122 ceph-mon[32255]: pgmap v9800: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:13 smithi022 ceph-mon[31360]: pgmap v9800: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:13 smithi022 ceph-mon[40250]: pgmap v9800: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:14.609 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:14.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:14.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:14.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:15 smithi122 ceph-mon[32255]: pgmap v9801: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1903114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:15 smithi022 ceph-mon[31360]: pgmap v9801: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1903114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:15 smithi022 ceph-mon[40250]: pgmap v9801: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1903114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:25:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:17 smithi122 ceph-mon[32255]: pgmap v9802: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:17 smithi022 ceph-mon[31360]: pgmap v9802: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:17 smithi022 ceph-mon[40250]: pgmap v9802: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:19.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:25:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:25:19.986 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:19 smithi122 ceph-mon[32255]: pgmap v9803: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:19 smithi022 ceph-mon[31360]: pgmap v9803: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:19 smithi022 ceph-mon[40250]: pgmap v9803: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:20] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:25:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1965768350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1965768350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1965768350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:22.135 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:21 smithi022 ceph-mon[31360]: pgmap v9804: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:22.136 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:21 smithi022 ceph-mon[40250]: pgmap v9804: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:21 smithi122 ceph-mon[32255]: pgmap v9804: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:25:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:25:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:25:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:25:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:25:23.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:25:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:25:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:25:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:25:23.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:25:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:25:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:25:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:25:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:25:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:23 smithi122 ceph-mon[32255]: pgmap v9805: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:23 smithi022 ceph-mon[31360]: pgmap v9805: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:23 smithi022 ceph-mon[40250]: pgmap v9805: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:25.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:25.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:25.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:25.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:25 smithi122 ceph-mon[32255]: pgmap v9806: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:25 smithi022 ceph-mon[31360]: pgmap v9806: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:25 smithi022 ceph-mon[40250]: pgmap v9806: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4069840614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4069840614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4069840614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:27.688 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:25:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:27 smithi022 ceph-mon[31360]: pgmap v9807: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:27 smithi022 ceph-mon[40250]: pgmap v9807: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:27 smithi122 ceph-mon[32255]: pgmap v9807: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:29 smithi122 ceph-mon[32255]: pgmap v9808: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:29.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:25:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:25:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:29 smithi022 ceph-mon[31360]: pgmap v9808: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:29 smithi022 ceph-mon[40250]: pgmap v9808: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:30] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:25:30.728 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:31.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:31.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:31.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:31 smithi022 ceph-mon[31360]: pgmap v9809: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2123048962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:31 smithi022 ceph-mon[40250]: pgmap v9809: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2123048962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:31 smithi122 ceph-mon[32255]: pgmap v9809: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2123048962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:25:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:25:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[31360]: pgmap v9810: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:25:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:25:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:25:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:25:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[40250]: pgmap v9810: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:25:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:25:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:25:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:25:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:33 smithi122 ceph-mon[32255]: pgmap v9810: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:25:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:25:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:25:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:25:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:35 smithi022 ceph-mon[31360]: pgmap v9811: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:35 smithi022 ceph-mon[40250]: pgmap v9811: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:36.097 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:35 smithi122 ceph-mon[32255]: pgmap v9811: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3059871508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3059871508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3059871508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:37.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:25:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:37 smithi022 ceph-mon[31360]: pgmap v9812: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:37 smithi022 ceph-mon[40250]: pgmap v9812: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:37 smithi122 ceph-mon[32255]: pgmap v9812: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:39.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:25:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:25:39.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:39 smithi122 ceph-mon[32255]: pgmap v9813: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:39 smithi022 ceph-mon[31360]: pgmap v9813: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:39 smithi022 ceph-mon[40250]: pgmap v9813: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:40] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:25:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:41 smithi022 ceph-mon[31360]: pgmap v9814: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:41 smithi022 ceph-mon[40250]: pgmap v9814: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:41.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:41.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:41.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:41 smithi122 ceph-mon[32255]: pgmap v9814: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:25:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:25:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2428285581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2428285581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2428285581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:43 smithi022 ceph-mon[31360]: pgmap v9815: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:43 smithi022 ceph-mon[40250]: pgmap v9815: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:43 smithi122 ceph-mon[32255]: pgmap v9815: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:45 smithi022 ceph-mon[31360]: pgmap v9816: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:45 smithi022 ceph-mon[40250]: pgmap v9816: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:45 smithi122 ceph-mon[32255]: pgmap v9816: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:46.851 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:47.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:47.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:47.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:25:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:47 smithi022 ceph-mon[31360]: pgmap v9817: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2164601850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:47 smithi022 ceph-mon[40250]: pgmap v9817: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2164601850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:47 smithi122 ceph-mon[32255]: pgmap v9817: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2164601850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:49 smithi122 ceph-mon[32255]: pgmap v9818: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:25:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:25:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:49 smithi022 ceph-mon[31360]: pgmap v9818: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:49 smithi022 ceph-mon[40250]: pgmap v9818: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:50] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:25:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:25:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:51 smithi022 ceph-mon[31360]: pgmap v9819: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:51 smithi022 ceph-mon[40250]: pgmap v9819: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:51 smithi122 ceph-mon[32255]: pgmap v9819: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:52.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:52.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:52.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:52.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:25:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:25:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3464673854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3464673854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3464673854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:53 smithi022 ceph-mon[31360]: pgmap v9820: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:53 smithi022 ceph-mon[40250]: pgmap v9820: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:53 smithi122 ceph-mon[32255]: pgmap v9820: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:55 smithi022 ceph-mon[31360]: pgmap v9821: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:55 smithi022 ceph-mon[40250]: pgmap v9821: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:55 smithi122 ceph-mon[32255]: pgmap v9821: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:25:57.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:25:57.748 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:25:57.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:25:57.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:25:57.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:25:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:57 smithi022 ceph-mon[31360]: pgmap v9822: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:57 smithi022 ceph-mon[40250]: pgmap v9822: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:57 smithi122 ceph-mon[32255]: pgmap v9822: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4225567316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4225567316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4225567316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:25:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:25:59 smithi122 ceph-mon[32255]: pgmap v9823: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:25:59.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:25:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:26:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:25:59 smithi022 ceph-mon[31360]: pgmap v9823: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:25:59 smithi022 ceph-mon[40250]: pgmap v9823: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:00] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:26:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:01 smithi022 ceph-mon[31360]: pgmap v9824: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:01 smithi022 ceph-mon[40250]: pgmap v9824: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:01 smithi122 ceph-mon[32255]: pgmap v9824: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:26:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:26:02.966 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:03.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:03.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:03.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:03 smithi022 ceph-mon[31360]: pgmap v9825: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4258679221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:03 smithi022 ceph-mon[40250]: pgmap v9825: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4258679221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:03 smithi122 ceph-mon[32255]: pgmap v9825: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4258679221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:05 smithi022 ceph-mon[31360]: pgmap v9826: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:05 smithi022 ceph-mon[40250]: pgmap v9826: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:05 smithi122 ceph-mon[32255]: pgmap v9826: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:07.769 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:26:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:07 smithi022 ceph-mon[31360]: pgmap v9827: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:07 smithi022 ceph-mon[40250]: pgmap v9827: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:07 smithi122 ceph-mon[32255]: pgmap v9827: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:08.336 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:08.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:08.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:08.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/213342640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/213342640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/213342640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:09.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:26:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:26:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:09 smithi122 ceph-mon[32255]: pgmap v9828: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:09 smithi022 ceph-mon[31360]: pgmap v9828: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:09 smithi022 ceph-mon[40250]: pgmap v9828: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:10] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:26:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:11 smithi122 ceph-mon[32255]: pgmap v9829: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:11 smithi022 ceph-mon[31360]: pgmap v9829: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:11 smithi022 ceph-mon[40250]: pgmap v9829: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:26:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:26:13.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:13 smithi022 ceph-mon[40250]: pgmap v9830: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:13 smithi022 ceph-mon[31360]: pgmap v9830: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:14.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:14.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:13 smithi122 ceph-mon[32255]: pgmap v9830: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/837497229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/837497229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/837497229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:15 smithi122 ceph-mon[32255]: pgmap v9831: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:15 smithi022 ceph-mon[31360]: pgmap v9831: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:15 smithi022 ceph-mon[40250]: pgmap v9831: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:26:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:17 smithi122 ceph-mon[32255]: pgmap v9832: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:17 smithi022 ceph-mon[31360]: pgmap v9832: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:17 smithi022 ceph-mon[40250]: pgmap v9832: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:19.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:19.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:19.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:19.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:19.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:26:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:26:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:19 smithi122 ceph-mon[32255]: pgmap v9833: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:19.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1318227072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:19 smithi022 ceph-mon[31360]: pgmap v9833: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:20.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1318227072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:19 smithi022 ceph-mon[40250]: pgmap v9833: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1318227072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:26:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:21 smithi122 ceph-mon[32255]: pgmap v9834: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:21 smithi022 ceph-mon[31360]: pgmap v9834: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:21 smithi022 ceph-mon[40250]: pgmap v9834: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:22.566 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:26:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:26:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:26:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:26:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:26:22.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:26:22.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:26:22.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:26:22.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:26:22.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:26:22.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:26:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:23 smithi122 ceph-mon[32255]: pgmap v9835: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:26:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:23 smithi022 ceph-mon[31360]: pgmap v9835: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:26:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:23 smithi022 ceph-mon[40250]: pgmap v9835: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:26:24.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:24.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:24.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:24.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:25 smithi122 ceph-mon[32255]: pgmap v9836: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2295481536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:25 smithi022 ceph-mon[31360]: pgmap v9836: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2295481536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:25 smithi022 ceph-mon[40250]: pgmap v9836: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2295481536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:26:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:27 smithi122 ceph-mon[32255]: pgmap v9837: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:27 smithi022 ceph-mon[31360]: pgmap v9837: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:27 smithi022 ceph-mon[40250]: pgmap v9837: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:29.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:29 smithi122 ceph-mon[32255]: pgmap v9838: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:29.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:26:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:26:30.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:30.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:30.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:30.194 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:29 smithi022 ceph-mon[31360]: pgmap v9838: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:30.194 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:29 smithi022 ceph-mon[40250]: pgmap v9838: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:26:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3590413758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3590413758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3590413758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:31 smithi122 ceph-mon[32255]: pgmap v9839: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:31 smithi022 ceph-mon[31360]: pgmap v9839: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:31 smithi022 ceph-mon[40250]: pgmap v9839: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:26:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:26:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:33 smithi122 ceph-mon[32255]: pgmap v9840: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:26:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:26:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:26:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:26:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[31360]: pgmap v9840: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:26:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:26:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:26:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:26:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[40250]: pgmap v9840: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:26:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:26:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:26:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:26:35.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:35.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:35.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:35.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:35 smithi122 ceph-mon[32255]: pgmap v9841: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3632071280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:35 smithi022 ceph-mon[31360]: pgmap v9841: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3632071280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:35 smithi022 ceph-mon[40250]: pgmap v9841: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3632071280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:26:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:37 smithi122 ceph-mon[32255]: pgmap v9842: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:37 smithi022 ceph-mon[31360]: pgmap v9842: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:37 smithi022 ceph-mon[40250]: pgmap v9842: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:39 smithi122 ceph-mon[32255]: pgmap v9843: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:39.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:26:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:26:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:39 smithi022 ceph-mon[31360]: pgmap v9843: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:39 smithi022 ceph-mon[40250]: pgmap v9843: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:40] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:26:40.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:40.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:40.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:40.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:40.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:41 smithi122 ceph-mon[32255]: pgmap v9844: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4101723327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:41 smithi022 ceph-mon[31360]: pgmap v9844: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4101723327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:41 smithi022 ceph-mon[40250]: pgmap v9844: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4101723327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:26:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:26:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:43 smithi122 ceph-mon[32255]: pgmap v9845: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:43 smithi022 ceph-mon[31360]: pgmap v9845: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:43 smithi022 ceph-mon[40250]: pgmap v9845: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:45.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:45 smithi122 ceph-mon[32255]: pgmap v9846: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:45 smithi022 ceph-mon[31360]: pgmap v9846: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:45 smithi022 ceph-mon[40250]: pgmap v9846: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:46.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:46.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:46.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1191791097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1191791097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1191791097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:26:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:47 smithi122 ceph-mon[32255]: pgmap v9847: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:47 smithi022 ceph-mon[31360]: pgmap v9847: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:47 smithi022 ceph-mon[40250]: pgmap v9847: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:49 smithi122 ceph-mon[32255]: pgmap v9848: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:49.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:26:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:26:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:49 smithi022 ceph-mon[31360]: pgmap v9848: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:49 smithi022 ceph-mon[40250]: pgmap v9848: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:26:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:26:51.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:51 smithi022 ceph-mon[31360]: pgmap v9849: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:52.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2822453111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:51 smithi022 ceph-mon[40250]: pgmap v9849: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2822453111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:51 smithi122 ceph-mon[32255]: pgmap v9849: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2822453111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:26:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:26:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:53 smithi122 ceph-mon[32255]: pgmap v9850: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:53 smithi022 ceph-mon[31360]: pgmap v9850: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:53 smithi022 ceph-mon[40250]: pgmap v9850: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:55 smithi122 ceph-mon[32255]: pgmap v9851: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:55 smithi022 ceph-mon[31360]: pgmap v9851: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:55 smithi022 ceph-mon[40250]: pgmap v9851: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:26:56.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:26:57.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:26:57.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:26:57.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:26:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:26:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:57 smithi122 ceph-mon[32255]: pgmap v9852: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/156674542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:57 smithi022 ceph-mon[31360]: pgmap v9852: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/156674542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:57 smithi022 ceph-mon[40250]: pgmap v9852: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/156674542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:26:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:26:59 smithi122 ceph-mon[32255]: pgmap v9853: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:26:59.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:26:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:27:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:26:59 smithi022 ceph-mon[31360]: pgmap v9853: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:26:59 smithi022 ceph-mon[40250]: pgmap v9853: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:27:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:02.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:01 smithi122 ceph-mon[32255]: pgmap v9854: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:01 smithi022 ceph-mon[31360]: pgmap v9854: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:01 smithi022 ceph-mon[40250]: pgmap v9854: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:02.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:02.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:02.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:27:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:27:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2042229206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2042229206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2042229206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:03 smithi122 ceph-mon[32255]: pgmap v9855: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:03 smithi022 ceph-mon[31360]: pgmap v9855: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:03 smithi022 ceph-mon[40250]: pgmap v9855: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:05 smithi122 ceph-mon[32255]: pgmap v9856: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:05 smithi022 ceph-mon[31360]: pgmap v9856: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:05 smithi022 ceph-mon[40250]: pgmap v9856: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:07.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:27:07.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:07.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:07.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:07 smithi122 ceph-mon[32255]: pgmap v9857: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1577791893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:07 smithi022 ceph-mon[31360]: pgmap v9857: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1577791893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:07 smithi022 ceph-mon[40250]: pgmap v9857: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1577791893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:09.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:09 smithi122 ceph-mon[32255]: pgmap v9858: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:09.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:27:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:27:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:09 smithi022 ceph-mon[31360]: pgmap v9858: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:09 smithi022 ceph-mon[40250]: pgmap v9858: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:10] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:27:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:11 smithi122 ceph-mon[32255]: pgmap v9859: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:11 smithi022 ceph-mon[31360]: pgmap v9859: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:11 smithi022 ceph-mon[40250]: pgmap v9859: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:27:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:27:12.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:13.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:13.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:13.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:13 smithi122 ceph-mon[32255]: pgmap v9860: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:14.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3580061440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:13 smithi022 ceph-mon[31360]: pgmap v9860: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:14.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3580061440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:13 smithi022 ceph-mon[40250]: pgmap v9860: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3580061440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:15 smithi122 ceph-mon[32255]: pgmap v9861: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:15 smithi022 ceph-mon[31360]: pgmap v9861: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:15 smithi022 ceph-mon[40250]: pgmap v9861: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:27:18.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:18.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:17 smithi022 ceph-mon[31360]: pgmap v9862: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:18.168 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:17 smithi022 ceph-mon[40250]: pgmap v9862: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:17 smithi122 ceph-mon[32255]: pgmap v9862: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:18.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:18.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:18.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/172747817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/172747817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/172747817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:19 smithi122 ceph-mon[32255]: pgmap v9863: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:19.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:27:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:27:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:19 smithi022 ceph-mon[31360]: pgmap v9863: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:19 smithi022 ceph-mon[40250]: pgmap v9863: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:20] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:27:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:21 smithi122 ceph-mon[32255]: pgmap v9864: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:21 smithi022 ceph-mon[31360]: pgmap v9864: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:21 smithi022 ceph-mon[40250]: pgmap v9864: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:27:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:27:23.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:23.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:23.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:23.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:23 smithi122 ceph-mon[32255]: pgmap v9865: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:27:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:27:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:27:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:27:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1183903917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[31360]: pgmap v9865: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:27:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:27:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:27:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:27:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1183903917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[40250]: pgmap v9865: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:27:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:27:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:27:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:27:24.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1183903917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:25 smithi022 ceph-mon[31360]: pgmap v9866: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:25 smithi022 ceph-mon[40250]: pgmap v9866: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:25 smithi122 ceph-mon[32255]: pgmap v9866: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:27.711 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:27:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:27 smithi022 ceph-mon[31360]: pgmap v9867: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:27 smithi022 ceph-mon[40250]: pgmap v9867: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:27 smithi122 ceph-mon[32255]: pgmap v9867: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:28.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:29 smithi022 ceph-mon[31360]: pgmap v9868: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/314248102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:29 smithi022 ceph-mon[40250]: pgmap v9868: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/314248102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:29 smithi122 ceph-mon[32255]: pgmap v9868: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/314248102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:30.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:27:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:27:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:30] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:27:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:31 smithi022 ceph-mon[31360]: pgmap v9869: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:31 smithi022 ceph-mon[40250]: pgmap v9869: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:31 smithi122 ceph-mon[32255]: pgmap v9869: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:27:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:27:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[31360]: pgmap v9870: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:27:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:27:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:27:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:27:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[40250]: pgmap v9870: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:27:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:27:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:27:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:27:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:33 smithi122 ceph-mon[32255]: pgmap v9870: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:27:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:27:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:27:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:27:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:34.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:34.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:34.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/508014069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/508014069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/508014069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:35 smithi022 ceph-mon[31360]: pgmap v9871: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:35 smithi022 ceph-mon[40250]: pgmap v9871: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:35 smithi122 ceph-mon[32255]: pgmap v9871: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:37.730 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:27:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:37 smithi022 ceph-mon[31360]: pgmap v9872: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:38.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:37 smithi022 ceph-mon[40250]: pgmap v9872: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:37 smithi122 ceph-mon[32255]: pgmap v9872: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:39.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:40.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:40.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:40.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:40.013 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:39 smithi022 ceph-mon[31360]: pgmap v9873: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:40.013 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:39 smithi022 ceph-mon[40250]: pgmap v9873: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:40.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:27:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:27:40.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:39 smithi122 ceph-mon[32255]: pgmap v9873: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:40] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:27:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/279008662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/279008662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/279008662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:41 smithi022 ceph-mon[31360]: pgmap v9874: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:41 smithi022 ceph-mon[40250]: pgmap v9874: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:41 smithi122 ceph-mon[32255]: pgmap v9874: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:27:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:27:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:43 smithi022 ceph-mon[31360]: pgmap v9875: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:43 smithi022 ceph-mon[40250]: pgmap v9875: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:43 smithi122 ceph-mon[32255]: pgmap v9875: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:45.014 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:45.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:45.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:45.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:45 smithi022 ceph-mon[31360]: pgmap v9876: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3320769421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:45 smithi022 ceph-mon[40250]: pgmap v9876: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3320769421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:45 smithi122 ceph-mon[32255]: pgmap v9876: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3320769421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:47.748 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:27:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:47 smithi022 ceph-mon[31360]: pgmap v9877: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:47 smithi022 ceph-mon[40250]: pgmap v9877: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:47 smithi122 ceph-mon[32255]: pgmap v9877: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:49 smithi022 ceph-mon[31360]: pgmap v9878: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:49 smithi022 ceph-mon[40250]: pgmap v9878: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:49 smithi122 ceph-mon[32255]: pgmap v9878: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:50.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:27:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:27:50.387 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:50] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:27:50.757 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:50.757 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2054310127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:50.757 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:50.758 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2054310127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:50.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:50.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:50.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:27:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2054310127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:51 smithi022 ceph-mon[31360]: pgmap v9879: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:51 smithi022 ceph-mon[40250]: pgmap v9879: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:51 smithi122 ceph-mon[32255]: pgmap v9879: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:27:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:27:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:53 smithi022 ceph-mon[31360]: pgmap v9880: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:53 smithi022 ceph-mon[40250]: pgmap v9880: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:53 smithi122 ceph-mon[32255]: pgmap v9880: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:55.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:27:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:55 smithi022 ceph-mon[40250]: pgmap v9881: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:55 smithi022 ceph-mon[31360]: pgmap v9881: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:56.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:27:56.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:27:56.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:27:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:55 smithi122 ceph-mon[32255]: pgmap v9881: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:27:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1083887293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1083887293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1083887293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:27:57.768 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:27:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:57 smithi022 ceph-mon[40250]: pgmap v9882: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:57 smithi022 ceph-mon[31360]: pgmap v9882: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:27:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:57 smithi122 ceph-mon[32255]: pgmap v9882: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:27:59 smithi022 ceph-mon[31360]: pgmap v9883: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:27:59 smithi022 ceph-mon[40250]: pgmap v9883: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:27:59 smithi122 ceph-mon[32255]: pgmap v9883: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:00.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:27:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:28:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:00] "GET /metrics HTTP/1.1" 200 202389 "" "Prometheus/2.18.1" 2021-12-15T23:28:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:01.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:01.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:01.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:01.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:01 smithi022 ceph-mon[31360]: pgmap v9884: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:28:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3519812343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:01 smithi022 ceph-mon[40250]: pgmap v9884: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:28:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3519812343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:01 smithi122 ceph-mon[32255]: pgmap v9884: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:28:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3519812343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:28:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:28:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:03 smithi122 ceph-mon[32255]: pgmap v9885: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:28:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:03 smithi022 ceph-mon[31360]: pgmap v9885: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:28:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:03 smithi022 ceph-mon[40250]: pgmap v9885: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:28:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:05 smithi122 ceph-mon[32255]: pgmap v9886: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:05 smithi022 ceph-mon[31360]: pgmap v9886: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:05 smithi022 ceph-mon[40250]: pgmap v9886: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:06.503 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:06.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:06.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:06.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:28:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:07 smithi122 ceph-mon[32255]: pgmap v9887: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4133542827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:07 smithi022 ceph-mon[31360]: pgmap v9887: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:08.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4133542827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:07 smithi022 ceph-mon[40250]: pgmap v9887: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4133542827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:09 smithi122 ceph-mon[32255]: pgmap v9888: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:10.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:28:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:28:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:09 smithi022 ceph-mon[31360]: pgmap v9888: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:09 smithi022 ceph-mon[40250]: pgmap v9888: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2021-12-15T23:28:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:10] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-15T23:28:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:11.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:11 smithi122 ceph-mon[32255]: pgmap v9889: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:28:12.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:12.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:12.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:12.245 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:11 smithi022 ceph-mon[31360]: pgmap v9889: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:28:12.245 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:11 smithi022 ceph-mon[40250]: pgmap v9889: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:28:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:28:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:28:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/861390280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/861390280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/861390280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:13 smithi122 ceph-mon[32255]: pgmap v9890: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:28:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:13 smithi022 ceph-mon[31360]: pgmap v9890: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:28:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:13 smithi022 ceph-mon[40250]: pgmap v9890: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:28:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:15 smithi122 ceph-mon[32255]: pgmap v9891: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:28:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:15 smithi022 ceph-mon[31360]: pgmap v9891: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:28:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:15 smithi022 ceph-mon[40250]: pgmap v9891: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:28:17.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:28:17.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:17.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:17.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:17 smithi022 ceph-mon[31360]: pgmap v9892: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T23:28:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/571853981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:17 smithi022 ceph-mon[40250]: pgmap v9892: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T23:28:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/571853981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:17 smithi122 ceph-mon[32255]: pgmap v9892: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T23:28:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/571853981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:19 smithi122 ceph-mon[32255]: pgmap v9893: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T23:28:20.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:28:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:28:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:19 smithi022 ceph-mon[31360]: pgmap v9893: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T23:28:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:19 smithi022 ceph-mon[40250]: pgmap v9893: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-15T23:28:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:20] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:28:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:21 smithi122 ceph-mon[32255]: pgmap v9894: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2021-12-15T23:28:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:21 smithi022 ceph-mon[31360]: pgmap v9894: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2021-12-15T23:28:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:21 smithi022 ceph-mon[40250]: pgmap v9894: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2021-12-15T23:28:22.613 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:28:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:28:22.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:22.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:22.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:23 smithi122 ceph-mon[32255]: pgmap v9895: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3494539613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:28:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:28:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:28:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[31360]: pgmap v9895: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3494539613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:28:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:28:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:28:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[40250]: pgmap v9895: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3494539613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:28:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:28:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:28:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:28:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:28:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:28:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:25 smithi122 ceph-mon[32255]: pgmap v9896: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:25 smithi022 ceph-mon[31360]: pgmap v9896: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:25 smithi022 ceph-mon[40250]: pgmap v9896: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:28:27.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:27 smithi122 ceph-mon[32255]: pgmap v9897: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:27 smithi022 ceph-mon[31360]: pgmap v9897: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:27 smithi022 ceph-mon[40250]: pgmap v9897: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:28.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:28.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:28.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1269343596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1269343596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1269343596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:29 smithi122 ceph-mon[32255]: pgmap v9898: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:30.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:28:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:28:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:29 smithi022 ceph-mon[31360]: pgmap v9898: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:29 smithi022 ceph-mon[40250]: pgmap v9898: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:30] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:28:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:31 smithi122 ceph-mon[32255]: pgmap v9899: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:31 smithi022 ceph-mon[31360]: pgmap v9899: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:31 smithi022 ceph-mon[40250]: pgmap v9899: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:28:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:28:33.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:33.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:33.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:33.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[31360]: pgmap v9900: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:28:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:28:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:28:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:28:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3717304690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[40250]: pgmap v9900: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:28:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:28:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:28:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:28:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3717304690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:33 smithi122 ceph-mon[32255]: pgmap v9900: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:28:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:28:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:28:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:28:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3717304690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:35 smithi022 ceph-mon[31360]: pgmap v9901: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:35 smithi022 ceph-mon[40250]: pgmap v9901: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:35 smithi122 ceph-mon[32255]: pgmap v9901: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:37.727 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:28:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:37 smithi022 ceph-mon[31360]: pgmap v9902: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:37 smithi022 ceph-mon[40250]: pgmap v9902: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:37 smithi122 ceph-mon[32255]: pgmap v9902: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:38.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:39.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:39.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:39.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:39 smithi022 ceph-mon[31360]: pgmap v9903: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3270418913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:39 smithi022 ceph-mon[40250]: pgmap v9903: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3270418913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:39 smithi122 ceph-mon[32255]: pgmap v9903: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3270418913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:40.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:28:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:28:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:40] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:28:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:41 smithi022 ceph-mon[31360]: pgmap v9904: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:41 smithi022 ceph-mon[40250]: pgmap v9904: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:41 smithi122 ceph-mon[32255]: pgmap v9904: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:28:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:28:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:43 smithi022 ceph-mon[31360]: pgmap v9905: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:43 smithi022 ceph-mon[40250]: pgmap v9905: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:43 smithi122 ceph-mon[32255]: pgmap v9905: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:44.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:44.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:44.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:44.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3105452922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3105452922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:45.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3105452922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:45 smithi022 ceph-mon[31360]: pgmap v9906: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:45 smithi022 ceph-mon[40250]: pgmap v9906: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:45 smithi122 ceph-mon[32255]: pgmap v9906: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:47.745 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:28:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:47 smithi022 ceph-mon[31360]: pgmap v9907: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:47 smithi022 ceph-mon[40250]: pgmap v9907: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:47 smithi122 ceph-mon[32255]: pgmap v9907: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:49.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:49 smithi022 ceph-mon[31360]: pgmap v9908: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:49 smithi022 ceph-mon[40250]: pgmap v9908: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:49 smithi122 ceph-mon[32255]: pgmap v9908: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:50.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:28:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:28:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:50] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:28:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2841070776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2841070776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:28:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2841070776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:51 smithi022 ceph-mon[31360]: pgmap v9909: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:51 smithi022 ceph-mon[40250]: pgmap v9909: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:51 smithi122 ceph-mon[32255]: pgmap v9909: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:28:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:28:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:53 smithi022 ceph-mon[31360]: pgmap v9910: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:53 smithi022 ceph-mon[40250]: pgmap v9910: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:53 smithi122 ceph-mon[32255]: pgmap v9910: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:54.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:28:55.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:28:55.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:28:55.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:28:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:55 smithi022 ceph-mon[31360]: pgmap v9911: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/27625762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:55 smithi022 ceph-mon[40250]: pgmap v9911: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/27625762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:55 smithi122 ceph-mon[32255]: pgmap v9911: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:28:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/27625762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:28:57.762 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:28:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:57 smithi022 ceph-mon[31360]: pgmap v9912: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:57 smithi022 ceph-mon[40250]: pgmap v9912: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:28:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:57 smithi122 ceph-mon[32255]: pgmap v9912: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:28:59 smithi022 ceph-mon[31360]: pgmap v9913: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:28:59 smithi022 ceph-mon[40250]: pgmap v9913: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:28:59 smithi122 ceph-mon[32255]: pgmap v9913: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:00.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:28:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:29:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:00] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:29:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/612620611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/612620611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/612620611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:01 smithi022 ceph-mon[31360]: pgmap v9914: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:01 smithi022 ceph-mon[40250]: pgmap v9914: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:01 smithi122 ceph-mon[32255]: pgmap v9914: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:29:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:29:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:03 smithi022 ceph-mon[31360]: pgmap v9915: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:03 smithi022 ceph-mon[40250]: pgmap v9915: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:03 smithi122 ceph-mon[32255]: pgmap v9915: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:05.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:05.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:05.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:05.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:05.946 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:05 smithi022 ceph-mon[40250]: pgmap v9916: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:05.946 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:05 smithi022 ceph-mon[31360]: pgmap v9916: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:05 smithi122 ceph-mon[32255]: pgmap v9916: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/861208953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/861208953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/861208953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:29:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:07 smithi122 ceph-mon[32255]: pgmap v9917: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:07 smithi022 ceph-mon[31360]: pgmap v9917: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:07 smithi022 ceph-mon[40250]: pgmap v9917: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:09 smithi122 ceph-mon[32255]: pgmap v9918: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:10.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:29:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:29:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:09 smithi022 ceph-mon[31360]: pgmap v9918: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:09 smithi022 ceph-mon[40250]: pgmap v9918: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:10] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:29:10.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:11.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:11.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:11.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:11 smithi122 ceph-mon[32255]: pgmap v9919: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1315472308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:11 smithi022 ceph-mon[31360]: pgmap v9919: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1315472308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:11 smithi022 ceph-mon[40250]: pgmap v9919: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1315472308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:29:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:29:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:13 smithi122 ceph-mon[32255]: pgmap v9920: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:13 smithi022 ceph-mon[31360]: pgmap v9920: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:13 smithi022 ceph-mon[40250]: pgmap v9920: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:15 smithi122 ceph-mon[32255]: pgmap v9921: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:15 smithi022 ceph-mon[31360]: pgmap v9921: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:16.333 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:15 smithi022 ceph-mon[40250]: pgmap v9921: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:16.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:16.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:16.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:16.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3885255828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3885255828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3885255828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:29:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:17 smithi122 ceph-mon[32255]: pgmap v9922: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:17 smithi022 ceph-mon[40250]: pgmap v9922: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:17 smithi022 ceph-mon[31360]: pgmap v9922: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:19 smithi122 ceph-mon[32255]: pgmap v9923: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:20.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:29:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:29:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:19 smithi022 ceph-mon[31360]: pgmap v9923: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:19 smithi022 ceph-mon[40250]: pgmap v9923: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:20] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:29:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:21.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:21 smithi022 ceph-mon[31360]: pgmap v9924: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:21 smithi022 ceph-mon[40250]: pgmap v9924: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:22.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:22.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:22.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:21 smithi122 ceph-mon[32255]: pgmap v9924: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:29:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:29:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/817267559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/817267559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/817267559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:23 smithi122 ceph-mon[32255]: pgmap v9925: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:23 smithi022 ceph-mon[31360]: pgmap v9925: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:23 smithi022 ceph-mon[40250]: pgmap v9925: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:29:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:29:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:29:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:29:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:29:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:29:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:29:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:29:25.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:29:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:29:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:29:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:29:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:25 smithi122 ceph-mon[32255]: pgmap v9926: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:25 smithi022 ceph-mon[31360]: pgmap v9926: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:25 smithi022 ceph-mon[40250]: pgmap v9926: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:27.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:27.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:27.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:27.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:29:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:27 smithi122 ceph-mon[32255]: pgmap v9927: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/686592361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:27 smithi022 ceph-mon[31360]: pgmap v9927: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/686592361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:27 smithi022 ceph-mon[40250]: pgmap v9927: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/686592361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:30.178 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:29:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:29:30.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:29 smithi122 ceph-mon[32255]: pgmap v9928: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:29 smithi022 ceph-mon[31360]: pgmap v9928: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:29 smithi022 ceph-mon[40250]: pgmap v9928: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:29:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:30] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:29:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:31 smithi122 ceph-mon[32255]: pgmap v9929: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:31 smithi022 ceph-mon[31360]: pgmap v9929: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:31 smithi022 ceph-mon[40250]: pgmap v9929: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:29:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:29:32.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:32.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:32.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2063046288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2063046288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2063046288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:33 smithi122 ceph-mon[32255]: pgmap v9930: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:29:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:29:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:29:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:29:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[31360]: pgmap v9930: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:29:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:29:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:29:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:29:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[40250]: pgmap v9930: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:29:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:29:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:29:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:29:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:35 smithi122 ceph-mon[32255]: pgmap v9931: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:35 smithi022 ceph-mon[31360]: pgmap v9931: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:35 smithi022 ceph-mon[40250]: pgmap v9931: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:29:37.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:38.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:38.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:38.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:38.169 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:37 smithi022 ceph-mon[31360]: pgmap v9932: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:38.169 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:37 smithi022 ceph-mon[40250]: pgmap v9932: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:37 smithi122 ceph-mon[32255]: pgmap v9932: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4208821669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4208821669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4208821669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:29:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:29:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:39 smithi122 ceph-mon[32255]: pgmap v9933: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:39 smithi022 ceph-mon[31360]: pgmap v9933: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:39 smithi022 ceph-mon[40250]: pgmap v9933: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:40] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:29:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:41 smithi122 ceph-mon[32255]: pgmap v9934: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:41 smithi022 ceph-mon[31360]: pgmap v9934: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:41 smithi022 ceph-mon[40250]: pgmap v9934: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:29:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:29:43.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:43.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:43.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:43.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:43 smithi122 ceph-mon[32255]: pgmap v9935: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3341598162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:43 smithi022 ceph-mon[31360]: pgmap v9935: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3341598162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:43 smithi022 ceph-mon[40250]: pgmap v9935: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3341598162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:45 smithi122 ceph-mon[32255]: pgmap v9936: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:45 smithi022 ceph-mon[31360]: pgmap v9936: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:45 smithi022 ceph-mon[40250]: pgmap v9936: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:29:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:47 smithi122 ceph-mon[32255]: pgmap v9937: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:47 smithi022 ceph-mon[31360]: pgmap v9937: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:47 smithi022 ceph-mon[40250]: pgmap v9937: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:48.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:48.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:48.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:29:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:29:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:49 smithi122 ceph-mon[32255]: pgmap v9938: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3588754679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:49 smithi022 ceph-mon[31360]: pgmap v9938: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3588754679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:49 smithi022 ceph-mon[40250]: pgmap v9938: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3588754679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:50] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:29:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:29:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:51 smithi122 ceph-mon[32255]: pgmap v9939: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:51 smithi022 ceph-mon[31360]: pgmap v9939: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:51 smithi022 ceph-mon[40250]: pgmap v9939: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:29:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:29:53.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:53 smithi122 ceph-mon[32255]: pgmap v9940: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:53 smithi022 ceph-mon[31360]: pgmap v9940: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:53 smithi022 ceph-mon[40250]: pgmap v9940: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:54.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:54.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:54.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:29:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2928492814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2928492814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2928492814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:29:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:55 smithi122 ceph-mon[32255]: pgmap v9941: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:55 smithi022 ceph-mon[31360]: pgmap v9941: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:55 smithi022 ceph-mon[40250]: pgmap v9941: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:29:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:29:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:57 smithi122 ceph-mon[32255]: pgmap v9942: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:57 smithi022 ceph-mon[31360]: pgmap v9942: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:57 smithi022 ceph-mon[40250]: pgmap v9942: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:29:59.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:29:59.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:29:59.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:29:59.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:59 smithi022 ceph-mon[31360]: pgmap v9943: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:29:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1517149617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:59 smithi022 ceph-mon[40250]: pgmap v9943: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:29:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1517149617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:00.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:29:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:30:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:59 smithi122 ceph-mon[32255]: pgmap v9943: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:29:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1517149617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:00] "GET /metrics HTTP/1.1" 200 202385 "" "Prometheus/2.18.1" 2021-12-15T23:30:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T23:30:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T23:30:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T23:30:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:01 smithi122 ceph-mon[32255]: pgmap v9944: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:01 smithi022 ceph-mon[31360]: pgmap v9944: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:01 smithi022 ceph-mon[40250]: pgmap v9944: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:30:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:30:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:03 smithi122 ceph-mon[32255]: pgmap v9945: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:03 smithi022 ceph-mon[31360]: pgmap v9945: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:03 smithi022 ceph-mon[40250]: pgmap v9945: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:04.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:05 smithi122 ceph-mon[32255]: pgmap v9946: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2068694933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:05 smithi022 ceph-mon[31360]: pgmap v9946: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2068694933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:05 smithi022 ceph-mon[40250]: pgmap v9946: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2068694933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:30:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:07 smithi122 ceph-mon[32255]: pgmap v9947: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:07 smithi022 ceph-mon[31360]: pgmap v9947: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:07 smithi022 ceph-mon[40250]: pgmap v9947: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:10.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:09 smithi122 ceph-mon[32255]: pgmap v9948: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:10.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:30:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:30:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:09 smithi022 ceph-mon[31360]: pgmap v9948: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:09 smithi022 ceph-mon[40250]: pgmap v9948: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:10.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:10.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:10.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:10] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:30:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/324158838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/324158838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/324158838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:11 smithi122 ceph-mon[32255]: pgmap v9949: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:11 smithi022 ceph-mon[31360]: pgmap v9949: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:11 smithi022 ceph-mon[40250]: pgmap v9949: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:30:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:30:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:13 smithi122 ceph-mon[32255]: pgmap v9950: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:13 smithi022 ceph-mon[31360]: pgmap v9950: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:13 smithi022 ceph-mon[40250]: pgmap v9950: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:15.407 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:15.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:15.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:15.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:15 smithi022 ceph-mon[31360]: pgmap v9951: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/906251829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:15 smithi022 ceph-mon[40250]: pgmap v9951: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/906251829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:15 smithi122 ceph-mon[32255]: pgmap v9951: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/906251829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:30:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:17 smithi122 ceph-mon[32255]: pgmap v9952: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:17 smithi022 ceph-mon[31360]: pgmap v9952: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:17 smithi022 ceph-mon[40250]: pgmap v9952: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:19 smithi122 ceph-mon[32255]: pgmap v9953: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:20.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:30:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:30:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:19 smithi022 ceph-mon[31360]: pgmap v9953: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:19 smithi022 ceph-mon[40250]: pgmap v9953: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:20] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:30:20.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:21.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:21.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:21.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:21 smithi122 ceph-mon[32255]: pgmap v9954: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1458073206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:21 smithi022 ceph-mon[31360]: pgmap v9954: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1458073206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:21 smithi022 ceph-mon[40250]: pgmap v9954: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1458073206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:30:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:30:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:23 smithi022 ceph-mon[31360]: pgmap v9955: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:23 smithi022 ceph-mon[40250]: pgmap v9955: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:24.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:23 smithi122 ceph-mon[32255]: pgmap v9955: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:30:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:30:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:30:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:25.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:30:25.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:30:25.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:30:25.227 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:25.227 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:25.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:30:25.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:30:25.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:30:25.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:25.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:26.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:25 smithi022 ceph-mon[31360]: pgmap v9956: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:25 smithi022 ceph-mon[40250]: pgmap v9956: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:25 smithi122 ceph-mon[32255]: pgmap v9956: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:30:26.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:26.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:26.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/14073645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/14073645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/14073645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:30:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:27 smithi022 ceph-mon[31360]: pgmap v9957: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:27 smithi022 ceph-mon[40250]: pgmap v9957: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:27 smithi122 ceph-mon[32255]: pgmap v9957: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:28 smithi022 ceph-mon[31360]: pgmap v9958: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:28 smithi022 ceph-mon[40250]: pgmap v9958: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:28 smithi122 ceph-mon[32255]: pgmap v9958: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:30:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:30:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:30] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:30:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:31 smithi022 ceph-mon[31360]: pgmap v9959: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:31 smithi022 ceph-mon[40250]: pgmap v9959: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:31.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:31.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:31.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:31 smithi122 ceph-mon[32255]: pgmap v9959: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:30:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:30:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/271823894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/271823894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/271823894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[31360]: pgmap v9960: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:30:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:30:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:30:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:30:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[40250]: pgmap v9960: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:30:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:30:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:30:34.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:30:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:33 smithi122 ceph-mon[32255]: pgmap v9960: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:30:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:30:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:30:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:30:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:35 smithi022 ceph-mon[31360]: pgmap v9961: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:35 smithi022 ceph-mon[40250]: pgmap v9961: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:35 smithi122 ceph-mon[32255]: pgmap v9961: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:36.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:30:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:37 smithi022 ceph-mon[31360]: pgmap v9962: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2037558404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:38.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:37 smithi022 ceph-mon[40250]: pgmap v9962: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:38.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2037558404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:37 smithi122 ceph-mon[32255]: pgmap v9962: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2037558404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:39 smithi022 ceph-mon[31360]: pgmap v9963: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:39 smithi022 ceph-mon[40250]: pgmap v9963: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:30:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:30:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:39 smithi122 ceph-mon[32255]: pgmap v9963: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:40] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:30:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:41 smithi122 ceph-mon[32255]: pgmap v9964: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:42.258 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:42.259 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:41 smithi022 ceph-mon[31360]: pgmap v9964: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:42.259 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:41 smithi022 ceph-mon[40250]: pgmap v9964: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:42.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:42.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:42.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:30:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:30:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3067595366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3067595366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3067595366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:43 smithi122 ceph-mon[32255]: pgmap v9965: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:43 smithi022 ceph-mon[31360]: pgmap v9965: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:43 smithi022 ceph-mon[40250]: pgmap v9965: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:45 smithi122 ceph-mon[32255]: pgmap v9966: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:45 smithi022 ceph-mon[31360]: pgmap v9966: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:45 smithi022 ceph-mon[40250]: pgmap v9966: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:47.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:30:48.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:48.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:47 smithi122 ceph-mon[32255]: pgmap v9967: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:47 smithi022 ceph-mon[31360]: pgmap v9967: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:47 smithi022 ceph-mon[40250]: pgmap v9967: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3506985278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3506985278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3506985278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:30:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:30:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:49 smithi122 ceph-mon[32255]: pgmap v9968: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:49 smithi022 ceph-mon[31360]: pgmap v9968: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:49 smithi022 ceph-mon[40250]: pgmap v9968: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:50] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:30:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:30:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:51 smithi122 ceph-mon[32255]: pgmap v9969: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:51 smithi022 ceph-mon[31360]: pgmap v9969: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:51 smithi022 ceph-mon[40250]: pgmap v9969: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:30:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:30:53.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:53.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:53.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:53.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:53 smithi122 ceph-mon[32255]: pgmap v9970: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2476570244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:53 smithi022 ceph-mon[31360]: pgmap v9970: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2476570244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:53 smithi022 ceph-mon[40250]: pgmap v9970: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2476570244' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:55 smithi122 ceph-mon[32255]: pgmap v9971: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:55 smithi022 ceph-mon[31360]: pgmap v9971: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:55 smithi022 ceph-mon[40250]: pgmap v9971: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:30:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:30:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:57 smithi122 ceph-mon[32255]: pgmap v9972: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:57 smithi022 ceph-mon[31360]: pgmap v9972: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:57 smithi022 ceph-mon[40250]: pgmap v9972: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:30:58.377 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:30:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:30:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:30:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:30:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2276657287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2276657287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:30:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2276657287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:30:59 smithi122 ceph-mon[32255]: pgmap v9973: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:00.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:30:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:31:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:30:59 smithi022 ceph-mon[31360]: pgmap v9973: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:30:59 smithi022 ceph-mon[40250]: pgmap v9973: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:00] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:31:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:01 smithi122 ceph-mon[32255]: pgmap v9974: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:01 smithi022 ceph-mon[31360]: pgmap v9974: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:01 smithi022 ceph-mon[40250]: pgmap v9974: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:31:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:31:03.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:03 smithi022 ceph-mon[31360]: pgmap v9975: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:03 smithi022 ceph-mon[40250]: pgmap v9975: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:04.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:04.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:04.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:03 smithi122 ceph-mon[32255]: pgmap v9975: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2863446932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2863446932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2863446932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:05 smithi122 ceph-mon[32255]: pgmap v9976: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:05 smithi022 ceph-mon[31360]: pgmap v9976: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:05 smithi022 ceph-mon[40250]: pgmap v9976: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:31:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:07 smithi122 ceph-mon[32255]: pgmap v9977: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:07 smithi022 ceph-mon[31360]: pgmap v9977: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:07 smithi022 ceph-mon[40250]: pgmap v9977: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:09.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:09.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:09.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:09.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:09 smithi122 ceph-mon[32255]: pgmap v9978: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4083660923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:10.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:31:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:31:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:09 smithi022 ceph-mon[31360]: pgmap v9978: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4083660923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:09 smithi022 ceph-mon[40250]: pgmap v9978: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4083660923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:10] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:31:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:11 smithi122 ceph-mon[32255]: pgmap v9979: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:11 smithi022 ceph-mon[31360]: pgmap v9979: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:11 smithi022 ceph-mon[40250]: pgmap v9979: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:31:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:31:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:13 smithi122 ceph-mon[32255]: pgmap v9980: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:13 smithi022 ceph-mon[31360]: pgmap v9980: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:13 smithi022 ceph-mon[40250]: pgmap v9980: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:14.486 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:14.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:14.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:14.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:15 smithi022 ceph-mon[31360]: pgmap v9981: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2252333822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:15 smithi022 ceph-mon[40250]: pgmap v9981: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2252333822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:15 smithi122 ceph-mon[32255]: pgmap v9981: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2252333822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:17.764 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:31:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:17 smithi022 ceph-mon[31360]: pgmap v9982: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:17 smithi022 ceph-mon[40250]: pgmap v9982: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:17 smithi122 ceph-mon[32255]: pgmap v9982: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:19.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:19 smithi022 ceph-mon[31360]: pgmap v9983: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:19 smithi022 ceph-mon[40250]: pgmap v9983: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:20.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:31:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:31:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:19 smithi122 ceph-mon[32255]: pgmap v9983: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:20.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:20.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:20.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:20] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:31:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4256744291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4256744291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4256744291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:21 smithi022 ceph-mon[31360]: pgmap v9984: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:21 smithi022 ceph-mon[40250]: pgmap v9984: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:21 smithi122 ceph-mon[32255]: pgmap v9984: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:31:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:31:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:23 smithi022 ceph-mon[31360]: pgmap v9985: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:23 smithi022 ceph-mon[40250]: pgmap v9985: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:23 smithi122 ceph-mon[32255]: pgmap v9985: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:25.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:25.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:25.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:25.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:25 smithi122 ceph-mon[32255]: pgmap v9986: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:31:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:31:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:31:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4067199653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:31:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[40250]: pgmap v9986: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:31:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:31:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:31:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4067199653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:31:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[31360]: pgmap v9986: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:31:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:31:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:31:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4067199653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:26.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:31:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:31:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:27 smithi122 ceph-mon[32255]: pgmap v9987: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:27 smithi022 ceph-mon[31360]: pgmap v9987: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:27 smithi022 ceph-mon[40250]: pgmap v9987: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:31:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:31:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:29 smithi122 ceph-mon[32255]: pgmap v9988: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:29 smithi022 ceph-mon[31360]: pgmap v9988: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:29 smithi022 ceph-mon[40250]: pgmap v9988: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:30] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:31:30.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:30.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:30.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:30.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:30.987 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:30.988 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:31 smithi122 ceph-mon[32255]: pgmap v9989: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3904486227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:31 smithi022 ceph-mon[31360]: pgmap v9989: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3904486227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:31 smithi022 ceph-mon[40250]: pgmap v9989: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3904486227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:31:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:31:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:33 smithi122 ceph-mon[32255]: pgmap v9990: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:31:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:31:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:31:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:31:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[31360]: pgmap v9990: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:31:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:31:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:31:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:31:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[40250]: pgmap v9990: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:31:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:31:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:31:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:31:35.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:35 smithi122 ceph-mon[32255]: pgmap v9991: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:35 smithi022 ceph-mon[31360]: pgmap v9991: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:35 smithi022 ceph-mon[40250]: pgmap v9991: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/843334854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/843334854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/843334854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:31:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:37 smithi122 ceph-mon[32255]: pgmap v9992: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:37 smithi022 ceph-mon[31360]: pgmap v9992: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:37 smithi022 ceph-mon[40250]: pgmap v9992: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:31:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:31:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:39 smithi122 ceph-mon[32255]: pgmap v9993: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:39 smithi022 ceph-mon[31360]: pgmap v9993: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:39 smithi022 ceph-mon[40250]: pgmap v9993: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:40] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:31:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:41.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:41.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:41.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:41.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:41 smithi022 ceph-mon[31360]: pgmap v9994: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1878948656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:41 smithi022 ceph-mon[40250]: pgmap v9994: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1878948656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:41 smithi122 ceph-mon[32255]: pgmap v9994: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1878948656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:31:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:31:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:43 smithi122 ceph-mon[32255]: pgmap v9995: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:43 smithi022 ceph-mon[31360]: pgmap v9995: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:43 smithi022 ceph-mon[40250]: pgmap v9995: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:45 smithi122 ceph-mon[32255]: pgmap v9996: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:45 smithi022 ceph-mon[31360]: pgmap v9996: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:45 smithi022 ceph-mon[40250]: pgmap v9996: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:47.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:47.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:47.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:31:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:47 smithi122 ceph-mon[32255]: pgmap v9997: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/928894954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:47 smithi022 ceph-mon[31360]: pgmap v9997: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/928894954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:47 smithi022 ceph-mon[40250]: pgmap v9997: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/928894954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:49 smithi122 ceph-mon[32255]: pgmap v9998: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:50.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:31:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:31:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:49 smithi022 ceph-mon[31360]: pgmap v9998: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:49 smithi022 ceph-mon[40250]: pgmap v9998: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:50] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:31:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:31:52.097 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:52.098 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:51 smithi022 ceph-mon[40250]: pgmap v9999: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:52.098 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:51 smithi022 ceph-mon[31360]: pgmap v9999: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:51 smithi122 ceph-mon[32255]: pgmap v9999: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:52.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:52.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:52.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:31:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:31:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/128722007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/128722007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/128722007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:53 smithi122 ceph-mon[32255]: pgmap v10000: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:53 smithi022 ceph-mon[31360]: pgmap v10000: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:53 smithi022 ceph-mon[40250]: pgmap v10000: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:55 smithi122 ceph-mon[32255]: pgmap v10001: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:55 smithi022 ceph-mon[31360]: pgmap v10001: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:55 smithi022 ceph-mon[40250]: pgmap v10001: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:31:57.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:31:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:31:57.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:31:57.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:31:57.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:31:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:57 smithi122 ceph-mon[32255]: pgmap v10002: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3718285705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:57 smithi022 ceph-mon[31360]: pgmap v10002: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3718285705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:31:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:57 smithi022 ceph-mon[40250]: pgmap v10002: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:31:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3718285705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:31:59 smithi122 ceph-mon[32255]: pgmap v10003: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:00.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:31:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:32:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:31:59 smithi022 ceph-mon[31360]: pgmap v10003: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:31:59 smithi022 ceph-mon[40250]: pgmap v10003: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:00] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:32:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:01 smithi122 ceph-mon[32255]: pgmap v10004: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:01 smithi022 ceph-mon[31360]: pgmap v10004: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:01 smithi022 ceph-mon[40250]: pgmap v10004: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:32:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:32:02.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:03.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:03.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:03.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:03 smithi122 ceph-mon[32255]: pgmap v10005: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/164062300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:03 smithi022 ceph-mon[31360]: pgmap v10005: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/164062300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:03 smithi022 ceph-mon[40250]: pgmap v10005: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/164062300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:05 smithi122 ceph-mon[32255]: pgmap v10006: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:05 smithi022 ceph-mon[31360]: pgmap v10006: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:05 smithi022 ceph-mon[40250]: pgmap v10006: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:32:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:07 smithi122 ceph-mon[32255]: pgmap v10007: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:08.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:08.205 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:07 smithi022 ceph-mon[31360]: pgmap v10007: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:08.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:07 smithi022 ceph-mon[40250]: pgmap v10007: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:08.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:08.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:08.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/552675839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/552675839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/552675839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:09 smithi122 ceph-mon[32255]: pgmap v10008: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:10.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:32:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:32:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:09 smithi022 ceph-mon[31360]: pgmap v10008: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:09 smithi022 ceph-mon[40250]: pgmap v10008: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:10] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:32:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:11 smithi122 ceph-mon[32255]: pgmap v10009: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:11 smithi022 ceph-mon[31360]: pgmap v10009: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:11 smithi022 ceph-mon[40250]: pgmap v10009: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:32:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:32:13.581 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:13.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:13.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:13.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:13 smithi122 ceph-mon[32255]: pgmap v10010: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:13 smithi022 ceph-mon[31360]: pgmap v10010: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:13 smithi022 ceph-mon[40250]: pgmap v10010: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3849853003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3849853003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:15.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3849853003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:15 smithi122 ceph-mon[32255]: pgmap v10011: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:15 smithi022 ceph-mon[31360]: pgmap v10011: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:15 smithi022 ceph-mon[40250]: pgmap v10011: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:32:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:17 smithi122 ceph-mon[32255]: pgmap v10012: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:17 smithi022 ceph-mon[31360]: pgmap v10012: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:17 smithi022 ceph-mon[40250]: pgmap v10012: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:18.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:19.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:19.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:19.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:20.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:32:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:32:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:19 smithi122 ceph-mon[32255]: pgmap v10013: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1728968931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:19 smithi022 ceph-mon[31360]: pgmap v10013: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1728968931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:19 smithi022 ceph-mon[40250]: pgmap v10013: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1728968931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:20] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:32:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:21 smithi122 ceph-mon[32255]: pgmap v10014: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:21 smithi022 ceph-mon[31360]: pgmap v10014: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:21 smithi022 ceph-mon[40250]: pgmap v10014: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:32:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:32:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:23 smithi122 ceph-mon[32255]: pgmap v10015: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:23 smithi022 ceph-mon[31360]: pgmap v10015: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:23 smithi022 ceph-mon[40250]: pgmap v10015: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:24.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:24.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:24.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:24.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3060581010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3060581010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3060581010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[31360]: pgmap v10016: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:32:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:32:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:32:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[40250]: pgmap v10016: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:32:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:32:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:32:26.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:25 smithi122 ceph-mon[32255]: pgmap v10016: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:26.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:32:26.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:32:26.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:32:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:27 smithi122 ceph-mon[32255]: pgmap v10017: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:32:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:32:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:27 smithi022 ceph-mon[31360]: pgmap v10017: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:27.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:32:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:27 smithi022 ceph-mon[40250]: pgmap v10017: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:32:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:29 smithi122 ceph-mon[32255]: pgmap v10018: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:29 smithi022 ceph-mon[31360]: pgmap v10018: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:29 smithi022 ceph-mon[40250]: pgmap v10018: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:29.692 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:30.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:30.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:30.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:30] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:32:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:32:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:32:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2195388628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2195388628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2195388628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:31 smithi122 ceph-mon[32255]: pgmap v10019: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:31 smithi022 ceph-mon[31360]: pgmap v10019: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:31 smithi022 ceph-mon[40250]: pgmap v10019: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:32:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:32:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:33 smithi122 ceph-mon[32255]: pgmap v10020: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:32:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:32:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:32:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:32:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[31360]: pgmap v10020: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:32:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:32:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:32:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:32:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[40250]: pgmap v10020: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:32:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:32:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:32:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:32:35.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:35.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:35.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:35.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:35 smithi122 ceph-mon[32255]: pgmap v10021: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3354170167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:35 smithi022 ceph-mon[31360]: pgmap v10021: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3354170167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:35 smithi022 ceph-mon[40250]: pgmap v10021: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3354170167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:32:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:37 smithi122 ceph-mon[32255]: pgmap v10022: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:37 smithi022 ceph-mon[31360]: pgmap v10022: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:37 smithi022 ceph-mon[40250]: pgmap v10022: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:32:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:32:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:39 smithi122 ceph-mon[32255]: pgmap v10023: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:39 smithi022 ceph-mon[31360]: pgmap v10023: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:39 smithi022 ceph-mon[40250]: pgmap v10023: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:40.430 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:40] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:32:40.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:40.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:40.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3674294150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3674294150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3674294150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:41 smithi122 ceph-mon[32255]: pgmap v10024: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:41 smithi022 ceph-mon[31360]: pgmap v10024: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:41 smithi022 ceph-mon[40250]: pgmap v10024: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:32:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:32:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:43 smithi122 ceph-mon[32255]: pgmap v10025: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:43 smithi022 ceph-mon[31360]: pgmap v10025: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:43 smithi022 ceph-mon[40250]: pgmap v10025: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:45.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:46.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:46.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:46.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:46.168 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:45 smithi022 ceph-mon[40250]: pgmap v10026: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:46.169 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:45 smithi022 ceph-mon[31360]: pgmap v10026: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:45 smithi122 ceph-mon[32255]: pgmap v10026: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/123933073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/123933073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/123933073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:32:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:47 smithi122 ceph-mon[32255]: pgmap v10027: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:47 smithi022 ceph-mon[31360]: pgmap v10027: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:47 smithi022 ceph-mon[40250]: pgmap v10027: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:32:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:32:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:49 smithi122 ceph-mon[32255]: pgmap v10028: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:49 smithi022 ceph-mon[31360]: pgmap v10028: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:49 smithi022 ceph-mon[40250]: pgmap v10028: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:50] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:32:51.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:51.170 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:51.170 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:32:51.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:51.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:51.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:51 smithi122 ceph-mon[32255]: pgmap v10029: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1693421439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:51 smithi022 ceph-mon[31360]: pgmap v10029: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1693421439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:51 smithi022 ceph-mon[40250]: pgmap v10029: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1693421439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:32:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:32:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:53 smithi122 ceph-mon[32255]: pgmap v10030: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:53 smithi022 ceph-mon[31360]: pgmap v10030: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:53 smithi022 ceph-mon[40250]: pgmap v10030: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:55 smithi122 ceph-mon[32255]: pgmap v10031: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:55 smithi022 ceph-mon[31360]: pgmap v10031: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:55 smithi022 ceph-mon[40250]: pgmap v10031: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:32:56.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:32:56.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:32:56.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:32:56.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:32:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:32:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:57 smithi122 ceph-mon[32255]: pgmap v10032: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1789933265' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:57 smithi022 ceph-mon[31360]: pgmap v10032: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1789933265' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:32:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:57 smithi022 ceph-mon[40250]: pgmap v10032: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:32:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1789933265' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:32:59 smithi122 ceph-mon[32255]: pgmap v10033: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:00.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:32:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:33:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:32:59 smithi022 ceph-mon[31360]: pgmap v10033: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:32:59 smithi022 ceph-mon[40250]: pgmap v10033: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:00] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:33:01.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:01 smithi122 ceph-mon[32255]: pgmap v10034: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:02.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:02.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:02.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:02.275 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:02.275 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:01 smithi022 ceph-mon[31360]: pgmap v10034: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:02.275 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:02.276 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:01 smithi022 ceph-mon[40250]: pgmap v10034: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:33:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:33:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3617455601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3617455601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3617455601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:03 smithi122 ceph-mon[32255]: pgmap v10035: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:03 smithi022 ceph-mon[31360]: pgmap v10035: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:03 smithi022 ceph-mon[40250]: pgmap v10035: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:05 smithi122 ceph-mon[32255]: pgmap v10036: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:05 smithi022 ceph-mon[31360]: pgmap v10036: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:05 smithi022 ceph-mon[40250]: pgmap v10036: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:07.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:33:07.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:07.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:07.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:07 smithi022 ceph-mon[40250]: pgmap v10037: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3214241886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:07 smithi022 ceph-mon[31360]: pgmap v10037: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:08.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3214241886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:07 smithi122 ceph-mon[32255]: pgmap v10037: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3214241886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:09 smithi122 ceph-mon[32255]: pgmap v10038: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:10.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:33:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:33:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:09 smithi022 ceph-mon[31360]: pgmap v10038: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:09 smithi022 ceph-mon[40250]: pgmap v10038: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:10] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:33:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:11 smithi122 ceph-mon[32255]: pgmap v10039: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:11 smithi022 ceph-mon[31360]: pgmap v10039: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:11 smithi022 ceph-mon[40250]: pgmap v10039: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:12.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:33:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:33:13.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:13.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:13.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:13 smithi122 ceph-mon[32255]: pgmap v10040: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3435350211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:13 smithi022 ceph-mon[31360]: pgmap v10040: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3435350211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:13 smithi022 ceph-mon[40250]: pgmap v10040: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3435350211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:15 smithi122 ceph-mon[32255]: pgmap v10041: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:15 smithi022 ceph-mon[31360]: pgmap v10041: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:15 smithi022 ceph-mon[40250]: pgmap v10041: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:33:18.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:17 smithi122 ceph-mon[32255]: pgmap v10042: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:17 smithi022 ceph-mon[31360]: pgmap v10042: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:17 smithi022 ceph-mon[40250]: pgmap v10042: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:18.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:18.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:18.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4000858508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4000858508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4000858508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:19 smithi122 ceph-mon[32255]: pgmap v10043: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:20.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:33:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:33:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:19 smithi022 ceph-mon[31360]: pgmap v10043: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:19 smithi022 ceph-mon[40250]: pgmap v10043: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:20] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:33:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:21 smithi122 ceph-mon[32255]: pgmap v10044: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:21 smithi022 ceph-mon[31360]: pgmap v10044: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:21 smithi022 ceph-mon[40250]: pgmap v10044: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:33:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:33:23.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:23.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:23.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:23.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:23 smithi022 ceph-mon[31360]: pgmap v10045: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/126865122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:23 smithi022 ceph-mon[40250]: pgmap v10045: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/126865122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:23 smithi122 ceph-mon[32255]: pgmap v10045: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/126865122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:25 smithi122 ceph-mon[32255]: pgmap v10046: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:26.209 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:25 smithi022 ceph-mon[31360]: pgmap v10046: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:26.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:25 smithi022 ceph-mon[40250]: pgmap v10046: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:33:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:33:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:33:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:33:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:33:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:33:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:33:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:33:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:33:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:33:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:33:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:33:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:33:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:27 smithi122 ceph-mon[32255]: pgmap v10047: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:27 smithi022 ceph-mon[31360]: pgmap v10047: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:27 smithi022 ceph-mon[40250]: pgmap v10047: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:28.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:29.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:29.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:29.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:33:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:33:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:29 smithi122 ceph-mon[32255]: pgmap v10048: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1209452028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:29 smithi022 ceph-mon[31360]: pgmap v10048: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1209452028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:29 smithi022 ceph-mon[40250]: pgmap v10048: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1209452028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:30] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:33:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:31 smithi122 ceph-mon[32255]: pgmap v10049: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:32.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:31 smithi022 ceph-mon[31360]: pgmap v10049: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:32.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:32.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:31 smithi022 ceph-mon[40250]: pgmap v10049: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:33:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:33:34.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:33 smithi122 ceph-mon[32255]: pgmap v10050: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:33:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:33:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:33:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:33:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[31360]: pgmap v10050: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:33:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:33:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:33:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:33:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[40250]: pgmap v10050: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:33:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:33:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:33:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:33:34.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:34.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:34.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3007533770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3007533770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3007533770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:35 smithi122 ceph-mon[32255]: pgmap v10051: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:35 smithi022 ceph-mon[31360]: pgmap v10051: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:35 smithi022 ceph-mon[40250]: pgmap v10051: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:33:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:37 smithi122 ceph-mon[32255]: pgmap v10052: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:37 smithi022 ceph-mon[31360]: pgmap v10052: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:37 smithi022 ceph-mon[40250]: pgmap v10052: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:39.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:39.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:39.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:39.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:40.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:33:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:33:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:39 smithi122 ceph-mon[32255]: pgmap v10053: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2919828031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:39 smithi022 ceph-mon[31360]: pgmap v10053: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2919828031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:39 smithi022 ceph-mon[40250]: pgmap v10053: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2919828031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:40] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:33:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:41 smithi122 ceph-mon[32255]: pgmap v10054: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:41 smithi022 ceph-mon[31360]: pgmap v10054: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:41 smithi022 ceph-mon[40250]: pgmap v10054: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:33:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:33:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:43 smithi122 ceph-mon[32255]: pgmap v10055: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:43 smithi022 ceph-mon[31360]: pgmap v10055: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:43 smithi022 ceph-mon[40250]: pgmap v10055: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:44.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:45.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:45.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:45.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:45 smithi122 ceph-mon[32255]: pgmap v10056: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3809922363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:45 smithi022 ceph-mon[31360]: pgmap v10056: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3809922363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:45 smithi022 ceph-mon[40250]: pgmap v10056: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3809922363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:33:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:47 smithi122 ceph-mon[32255]: pgmap v10057: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:47 smithi022 ceph-mon[31360]: pgmap v10057: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:47 smithi022 ceph-mon[40250]: pgmap v10057: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:33:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:33:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:49 smithi122 ceph-mon[32255]: pgmap v10058: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:50.246 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:49 smithi022 ceph-mon[31360]: pgmap v10058: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:50.246 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:49 smithi022 ceph-mon[40250]: pgmap v10058: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:50] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:33:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:50.920 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/818674438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/818674438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/818674438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:51 smithi122 ceph-mon[32255]: pgmap v10059: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:51 smithi022 ceph-mon[31360]: pgmap v10059: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:33:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:51 smithi022 ceph-mon[40250]: pgmap v10059: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:33:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:33:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:53 smithi122 ceph-mon[32255]: pgmap v10060: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:53 smithi022 ceph-mon[31360]: pgmap v10060: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:53 smithi022 ceph-mon[40250]: pgmap v10060: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:55.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:33:55.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:33:55.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:33:55.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:33:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:55 smithi122 ceph-mon[32255]: pgmap v10061: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:55 smithi022 ceph-mon[31360]: pgmap v10061: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:55 smithi022 ceph-mon[40250]: pgmap v10061: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:33:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2047100628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2047100628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2047100628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:33:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:33:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:57 smithi122 ceph-mon[32255]: pgmap v10062: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:57 smithi022 ceph-mon[31360]: pgmap v10062: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:33:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:57 smithi022 ceph-mon[40250]: pgmap v10062: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:33:59 smithi022 ceph-mon[31360]: pgmap v10063: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:33:59 smithi022 ceph-mon[40250]: pgmap v10063: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:33:59 smithi122 ceph-mon[32255]: pgmap v10063: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:00] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:34:00.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:34:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:34:00.991 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:01.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:01.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:01.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:01 smithi022 ceph-mon[31360]: pgmap v10064: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2302628201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:01 smithi022 ceph-mon[40250]: pgmap v10064: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2302628201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:01 smithi122 ceph-mon[32255]: pgmap v10064: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2302628201' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:34:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:34:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:03 smithi022 ceph-mon[31360]: pgmap v10065: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:03 smithi022 ceph-mon[40250]: pgmap v10065: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:03 smithi122 ceph-mon[32255]: pgmap v10065: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:05 smithi022 ceph-mon[31360]: pgmap v10066: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:05 smithi022 ceph-mon[40250]: pgmap v10066: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:06.360 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:05 smithi122 ceph-mon[32255]: pgmap v10066: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:06.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:06.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:06.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1954279327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1954279327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1954279327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:34:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:07 smithi022 ceph-mon[31360]: pgmap v10067: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:07 smithi022 ceph-mon[40250]: pgmap v10067: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:07 smithi122 ceph-mon[32255]: pgmap v10067: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:09 smithi022 ceph-mon[31360]: pgmap v10068: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:09 smithi022 ceph-mon[40250]: pgmap v10068: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:10] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:34:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:09 smithi122 ceph-mon[32255]: pgmap v10068: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:10.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:34:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:34:11.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:11 smithi022 ceph-mon[31360]: pgmap v10069: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:11 smithi022 ceph-mon[40250]: pgmap v10069: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:12.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:12.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:12.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:11 smithi122 ceph-mon[32255]: pgmap v10069: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:34:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:34:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/635563114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/635563114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/635563114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:13 smithi022 ceph-mon[31360]: pgmap v10070: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:13 smithi022 ceph-mon[40250]: pgmap v10070: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:13 smithi122 ceph-mon[32255]: pgmap v10070: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:15 smithi022 ceph-mon[31360]: pgmap v10071: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:15 smithi022 ceph-mon[40250]: pgmap v10071: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:15 smithi122 ceph-mon[32255]: pgmap v10071: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:17.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:17.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:17.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:17.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:34:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:17 smithi022 ceph-mon[31360]: pgmap v10072: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/416647482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:17 smithi022 ceph-mon[40250]: pgmap v10072: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/416647482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:17 smithi122 ceph-mon[32255]: pgmap v10072: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/416647482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:19 smithi022 ceph-mon[31360]: pgmap v10073: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:19 smithi022 ceph-mon[40250]: pgmap v10073: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:34:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:19 smithi122 ceph-mon[32255]: pgmap v10073: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:34:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:34:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:21 smithi022 ceph-mon[31360]: pgmap v10074: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:21 smithi022 ceph-mon[40250]: pgmap v10074: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:21 smithi122 ceph-mon[32255]: pgmap v10074: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:22.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:34:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:34:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:22.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:22.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2602704010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2602704010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2602704010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:23 smithi022 ceph-mon[31360]: pgmap v10075: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:23 smithi022 ceph-mon[40250]: pgmap v10075: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:23 smithi122 ceph-mon[32255]: pgmap v10075: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:25 smithi022 ceph-mon[31360]: pgmap v10076: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:25 smithi022 ceph-mon[40250]: pgmap v10076: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:25 smithi122 ceph-mon[32255]: pgmap v10076: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:26.985 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:34:26.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:34:26.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:34:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:34:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:34:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:34:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:34:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:34:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:34:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:34:27.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:28.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:28.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:28.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:28.221 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:27 smithi022 ceph-mon[40250]: pgmap v10077: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:28.221 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:27 smithi022 ceph-mon[31360]: pgmap v10077: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:27 smithi122 ceph-mon[32255]: pgmap v10077: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:28 smithi022 ceph-mon[31360]: pgmap v10078: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1559799027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:28 smithi022 ceph-mon[40250]: pgmap v10078: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1559799027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:28 smithi122 ceph-mon[32255]: pgmap v10078: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1559799027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:34:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:34:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:34:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:31 smithi122 ceph-mon[32255]: pgmap v10079: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[31360]: pgmap v10079: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[40250]: pgmap v10079: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:31.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:31.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:32.585 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:34:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:34:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:33.222 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:33.223 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:33.223 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:33.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:33.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:33.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: pgmap v10080: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:34:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:34:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:34:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:34:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4011354433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: pgmap v10080: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:34:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:34:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:34:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:34:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4011354433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: pgmap v10080: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:34:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:34:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:34:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:34:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4011354433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:34:34.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:34 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3129M 2021-12-15T23:34:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:34 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3129M 2021-12-15T23:34:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:34 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3129M 2021-12-15T23:34:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:34:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:35 smithi122 ceph-mon[32255]: pgmap v10081: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:35 smithi022 ceph-mon[31360]: pgmap v10081: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:35 smithi022 ceph-mon[40250]: pgmap v10081: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:34:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:37 smithi122 ceph-mon[32255]: pgmap v10082: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:37 smithi022 ceph-mon[31360]: pgmap v10082: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:37 smithi022 ceph-mon[40250]: pgmap v10082: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:38.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:38.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:38.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:38.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:40.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:40] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:34:40.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:34:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:34:40.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:39 smithi122 ceph-mon[32255]: pgmap v10083: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:40.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2306192702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:39 smithi022 ceph-mon[31360]: pgmap v10083: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2306192702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:39 smithi022 ceph-mon[40250]: pgmap v10083: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2306192702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:41 smithi122 ceph-mon[32255]: pgmap v10084: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:41 smithi022 ceph-mon[31360]: pgmap v10084: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:41 smithi022 ceph-mon[40250]: pgmap v10084: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:34:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:34:43.979 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:43 smithi022 ceph-mon[31360]: pgmap v10085: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:43 smithi022 ceph-mon[40250]: pgmap v10085: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:44.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:44.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:44.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:43 smithi122 ceph-mon[32255]: pgmap v10085: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3399731867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3399731867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3399731867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:45 smithi022 ceph-mon[31360]: pgmap v10086: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:45 smithi022 ceph-mon[40250]: pgmap v10086: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:45 smithi122 ceph-mon[32255]: pgmap v10086: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:34:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:47 smithi022 ceph-mon[31360]: pgmap v10087: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:47 smithi022 ceph-mon[40250]: pgmap v10087: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:47 smithi122 ceph-mon[32255]: pgmap v10087: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:49.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:49.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:49.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:49.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:49 smithi022 ceph-mon[31360]: pgmap v10088: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3102254079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:49 smithi022 ceph-mon[40250]: pgmap v10088: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3102254079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:50] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:34:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:49 smithi122 ceph-mon[32255]: pgmap v10088: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3102254079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:34:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:34:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:51 smithi022 ceph-mon[31360]: pgmap v10089: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:51 smithi022 ceph-mon[40250]: pgmap v10089: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:34:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:51 smithi122 ceph-mon[32255]: pgmap v10089: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:34:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:34:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:53 smithi022 ceph-mon[31360]: pgmap v10090: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:53 smithi022 ceph-mon[40250]: pgmap v10090: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:53 smithi122 ceph-mon[32255]: pgmap v10090: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:54.719 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:34:55.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:34:55.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:34:55.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:34:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:55 smithi022 ceph-mon[31360]: pgmap v10091: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2945133465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:55 smithi022 ceph-mon[40250]: pgmap v10091: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2945133465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:55 smithi122 ceph-mon[32255]: pgmap v10091: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:34:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2945133465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:34:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:34:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:57 smithi022 ceph-mon[31360]: pgmap v10092: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:57 smithi022 ceph-mon[40250]: pgmap v10092: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:34:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:57 smithi122 ceph-mon[32255]: pgmap v10092: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:00.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:34:59 smithi022 ceph-mon[31360]: pgmap v10093: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:34:59 smithi022 ceph-mon[40250]: pgmap v10093: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:35:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:35:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:34:59 smithi122 ceph-mon[32255]: pgmap v10093: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:00] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:35:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/709830765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/709830765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/709830765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:01 smithi022 ceph-mon[31360]: pgmap v10094: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:01 smithi022 ceph-mon[40250]: pgmap v10094: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:01 smithi122 ceph-mon[32255]: pgmap v10094: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:35:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:35:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:03 smithi022 ceph-mon[31360]: pgmap v10095: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:03 smithi022 ceph-mon[40250]: pgmap v10095: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:03 smithi122 ceph-mon[32255]: pgmap v10095: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:05.465 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:05.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:05.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:05.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:05 smithi022 ceph-mon[31360]: pgmap v10096: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/323869700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:05 smithi022 ceph-mon[40250]: pgmap v10096: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/323869700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:05 smithi122 ceph-mon[32255]: pgmap v10096: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/323869700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:35:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:07 smithi022 ceph-mon[31360]: pgmap v10097: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:07 smithi022 ceph-mon[40250]: pgmap v10097: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:07 smithi122 ceph-mon[32255]: pgmap v10097: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:09 smithi022 ceph-mon[31360]: pgmap v10098: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:09 smithi022 ceph-mon[40250]: pgmap v10098: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:10] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:35:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:35:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:35:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:09 smithi122 ceph-mon[32255]: pgmap v10098: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:10.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:11.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:11.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:11.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:11 smithi022 ceph-mon[31360]: pgmap v10099: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3883538886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:11 smithi022 ceph-mon[40250]: pgmap v10099: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3883538886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:11 smithi122 ceph-mon[32255]: pgmap v10099: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3883538886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:35:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:35:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:13 smithi022 ceph-mon[31360]: pgmap v10100: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:14.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:13 smithi022 ceph-mon[40250]: pgmap v10100: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:13 smithi122 ceph-mon[32255]: pgmap v10100: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:16.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:15 smithi022 ceph-mon[31360]: pgmap v10101: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:15 smithi022 ceph-mon[40250]: pgmap v10101: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:15 smithi122 ceph-mon[32255]: pgmap v10101: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:16.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:16.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:16.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3107126284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3107126284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3107126284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:35:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:17 smithi022 ceph-mon[31360]: pgmap v10102: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:17 smithi022 ceph-mon[40250]: pgmap v10102: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:17 smithi122 ceph-mon[32255]: pgmap v10102: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:19 smithi022 ceph-mon[31360]: pgmap v10103: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:19 smithi022 ceph-mon[40250]: pgmap v10103: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:19 smithi122 ceph-mon[32255]: pgmap v10103: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:20] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:35:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:35:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:35:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:20 smithi022 ceph-mon[31360]: pgmap v10104: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:20 smithi022 ceph-mon[40250]: pgmap v10104: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:20 smithi122 ceph-mon[32255]: pgmap v10104: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:21.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:21.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:21.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:21.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2163432122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2163432122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2163432122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:35:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:35:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:23 smithi022 ceph-mon[31360]: pgmap v10105: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:23 smithi022 ceph-mon[40250]: pgmap v10105: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:23 smithi122 ceph-mon[32255]: pgmap v10105: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:25 smithi122 ceph-mon[32255]: pgmap v10106: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:25 smithi022 ceph-mon[31360]: pgmap v10106: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:25 smithi022 ceph-mon[40250]: pgmap v10106: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:26.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:27.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:27.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:27.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:35:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:27 smithi122 ceph-mon[32255]: pgmap v10107: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2190483504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:27 smithi022 ceph-mon[31360]: pgmap v10107: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2190483504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:27 smithi022 ceph-mon[40250]: pgmap v10107: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2190483504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:30.080 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:29 smithi122 ceph-mon[32255]: pgmap v10108: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:29 smithi022 ceph-mon[31360]: pgmap v10108: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:29 smithi022 ceph-mon[40250]: pgmap v10108: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:30] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:35:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:35:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:35:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:31 smithi122 ceph-mon[32255]: pgmap v10109: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:31 smithi022 ceph-mon[31360]: pgmap v10109: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:31 smithi022 ceph-mon[40250]: pgmap v10109: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:32.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:32.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:32.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:32.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:35:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:35:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1655150117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1655150117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1655150117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:33 smithi122 ceph-mon[32255]: pgmap v10110: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:35:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:35:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:35:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:35:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[31360]: pgmap v10110: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:35:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:35:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:35:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:35:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[40250]: pgmap v10110: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:35:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:35:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:35:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:35:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:35:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:35:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:35:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:35:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:35:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:35:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:35:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:35:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:35:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:35 smithi022 ceph-mon[31360]: pgmap v10111: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:35:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:35 smithi022 ceph-mon[40250]: pgmap v10111: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:35:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:35 smithi122 ceph-mon[32255]: pgmap v10111: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:35:37.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:37.687 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:35:37.979 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:37 smithi022 ceph-mon[31360]: pgmap v10112: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:38.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:38.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:38.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:37 smithi022 ceph-mon[40250]: pgmap v10112: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:37 smithi122 ceph-mon[32255]: pgmap v10112: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2065826499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2065826499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2065826499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:39 smithi022 ceph-mon[31360]: pgmap v10113: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:39 smithi022 ceph-mon[40250]: pgmap v10113: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:39 smithi122 ceph-mon[32255]: pgmap v10113: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:40] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:35:40.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:35:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:35:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:41 smithi022 ceph-mon[31360]: pgmap v10114: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:41 smithi022 ceph-mon[40250]: pgmap v10114: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:41 smithi122 ceph-mon[32255]: pgmap v10114: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:35:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:35:43.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:43.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:43.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:43.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:43 smithi022 ceph-mon[31360]: pgmap v10115: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2939690585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:43 smithi022 ceph-mon[40250]: pgmap v10115: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2939690585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:43 smithi122 ceph-mon[32255]: pgmap v10115: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2939690585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:45 smithi022 ceph-mon[31360]: pgmap v10116: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:45 smithi022 ceph-mon[40250]: pgmap v10116: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:45 smithi122 ceph-mon[32255]: pgmap v10116: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:35:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:47 smithi022 ceph-mon[31360]: pgmap v10117: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:47 smithi022 ceph-mon[40250]: pgmap v10117: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:48.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:47 smithi122 ceph-mon[32255]: pgmap v10117: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:48.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:48.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:48.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1879010347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1879010347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1879010347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:50 smithi022 ceph-mon[31360]: pgmap v10118: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:50 smithi022 ceph-mon[40250]: pgmap v10118: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:50] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:35:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:35:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:35:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:50 smithi122 ceph-mon[32255]: pgmap v10118: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:51 smithi022 ceph-mon[31360]: pgmap v10119: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:51 smithi022 ceph-mon[40250]: pgmap v10119: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:35:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:51 smithi122 ceph-mon[32255]: pgmap v10119: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:35:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:35:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:53 smithi022 ceph-mon[31360]: pgmap v10120: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:53 smithi022 ceph-mon[40250]: pgmap v10120: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:53 smithi122 ceph-mon[32255]: pgmap v10120: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:53.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:54.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:54.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:54.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:35:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1392478860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1392478860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1392478860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:35:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:55 smithi122 ceph-mon[32255]: pgmap v10121: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:55 smithi022 ceph-mon[31360]: pgmap v10121: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:55 smithi022 ceph-mon[40250]: pgmap v10121: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:35:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:35:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:57 smithi122 ceph-mon[32255]: pgmap v10122: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:57 smithi022 ceph-mon[31360]: pgmap v10122: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:57 smithi022 ceph-mon[40250]: pgmap v10122: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:35:59.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:35:59.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:35:59.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:35:59.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:59 smithi122 ceph-mon[32255]: pgmap v10123: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:35:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3191201160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:59 smithi022 ceph-mon[31360]: pgmap v10123: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:35:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3191201160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:59 smithi022 ceph-mon[40250]: pgmap v10123: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:35:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3191201160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:00] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:36:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:36:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:36:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:01 smithi122 ceph-mon[32255]: pgmap v10124: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:01 smithi022 ceph-mon[31360]: pgmap v10124: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:01 smithi022 ceph-mon[40250]: pgmap v10124: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:36:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:36:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:03 smithi122 ceph-mon[32255]: pgmap v10125: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:03 smithi022 ceph-mon[31360]: pgmap v10125: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:03 smithi022 ceph-mon[40250]: pgmap v10125: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:04.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:04.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:04.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:04.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:05 smithi122 ceph-mon[32255]: pgmap v10126: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/615236908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:05 smithi022 ceph-mon[31360]: pgmap v10126: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/615236908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:05 smithi022 ceph-mon[40250]: pgmap v10126: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/615236908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:36:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:07 smithi122 ceph-mon[32255]: pgmap v10127: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:07 smithi022 ceph-mon[31360]: pgmap v10127: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:07 smithi022 ceph-mon[40250]: pgmap v10127: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:09.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:10.105 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:09 smithi122 ceph-mon[32255]: pgmap v10128: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:09 smithi022 ceph-mon[31360]: pgmap v10128: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:09 smithi022 ceph-mon[40250]: pgmap v10128: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:10.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:10.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:10] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:36:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:36:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:36:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1777382920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1777382920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1777382920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:11 smithi122 ceph-mon[32255]: pgmap v10129: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:11 smithi022 ceph-mon[31360]: pgmap v10129: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:11 smithi022 ceph-mon[40250]: pgmap v10129: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:36:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:36:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:13 smithi122 ceph-mon[32255]: pgmap v10130: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:13 smithi022 ceph-mon[31360]: pgmap v10130: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:13 smithi022 ceph-mon[40250]: pgmap v10130: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:15.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:15 smithi022 ceph-mon[31360]: pgmap v10131: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/810357564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:15 smithi022 ceph-mon[40250]: pgmap v10131: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/810357564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:15 smithi122 ceph-mon[32255]: pgmap v10131: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/810357564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:36:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:17 smithi122 ceph-mon[32255]: pgmap v10132: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:17 smithi022 ceph-mon[31360]: pgmap v10132: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:17 smithi022 ceph-mon[40250]: pgmap v10132: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:20.115 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:19 smithi122 ceph-mon[32255]: pgmap v10133: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:19 smithi022 ceph-mon[31360]: pgmap v10133: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:19 smithi022 ceph-mon[40250]: pgmap v10133: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:36:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:36:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:20] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:36:20.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:21.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:21.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:21.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:21 smithi122 ceph-mon[32255]: pgmap v10134: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3388775000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:21 smithi022 ceph-mon[31360]: pgmap v10134: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3388775000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:21 smithi022 ceph-mon[40250]: pgmap v10134: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3388775000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:36:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:36:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:23 smithi122 ceph-mon[32255]: pgmap v10135: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:23 smithi022 ceph-mon[31360]: pgmap v10135: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:23 smithi022 ceph-mon[40250]: pgmap v10135: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:26.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:25 smithi122 ceph-mon[32255]: pgmap v10136: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:25 smithi022 ceph-mon[31360]: pgmap v10136: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:25 smithi022 ceph-mon[40250]: pgmap v10136: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:26.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:26.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:26.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3749797663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3749797663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3749797663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:36:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:27 smithi122 ceph-mon[32255]: pgmap v10137: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:27 smithi022 ceph-mon[31360]: pgmap v10137: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:27 smithi022 ceph-mon[40250]: pgmap v10137: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:29 smithi122 ceph-mon[32255]: pgmap v10138: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:30.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:36:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:36:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:29 smithi022 ceph-mon[31360]: pgmap v10138: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:29 smithi022 ceph-mon[40250]: pgmap v10138: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:30] "GET /metrics HTTP/1.1" 200 202388 "" "Prometheus/2.18.1" 2021-12-15T23:36:31.392 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:31.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:31.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:31.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:31 smithi022 ceph-mon[31360]: pgmap v10139: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/256865109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:31 smithi022 ceph-mon[40250]: pgmap v10139: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/256865109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:31 smithi122 ceph-mon[32255]: pgmap v10139: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/256865109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:36:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:36:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:33 smithi122 ceph-mon[32255]: pgmap v10140: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:36:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:36:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:36:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:36:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[31360]: pgmap v10140: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:36:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:36:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:36:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:36:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[40250]: pgmap v10140: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:36:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:36:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:36:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:36:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:35 smithi122 ceph-mon[32255]: pgmap v10141: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:36:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:36:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:36:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:36:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[31360]: pgmap v10141: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:36:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:36:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:36:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:36:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[40250]: pgmap v10141: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:36:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:36:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:36:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:36:36.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:37.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:37.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:37.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:36:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:37 smithi122 ceph-mon[32255]: pgmap v10142: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4011002271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:37 smithi022 ceph-mon[31360]: pgmap v10142: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4011002271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:37 smithi022 ceph-mon[40250]: pgmap v10142: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4011002271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:39 smithi122 ceph-mon[32255]: pgmap v10143: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:40.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:36:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:36:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:39 smithi022 ceph-mon[31360]: pgmap v10143: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:39 smithi022 ceph-mon[40250]: pgmap v10143: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:40] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:36:42.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:41 smithi122 ceph-mon[32255]: pgmap v10144: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:41 smithi022 ceph-mon[31360]: pgmap v10144: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:41 smithi022 ceph-mon[40250]: pgmap v10144: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:42.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:42.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:42.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:36:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:36:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3395500110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3395500110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3395500110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:43 smithi122 ceph-mon[32255]: pgmap v10145: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:43 smithi022 ceph-mon[31360]: pgmap v10145: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:43 smithi022 ceph-mon[40250]: pgmap v10145: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:45 smithi122 ceph-mon[32255]: pgmap v10146: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:45 smithi022 ceph-mon[31360]: pgmap v10146: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:45 smithi022 ceph-mon[40250]: pgmap v10146: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:47.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:36:47.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:47.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:47.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:47 smithi122 ceph-mon[32255]: pgmap v10147: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2938994430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:47 smithi022 ceph-mon[31360]: pgmap v10147: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2938994430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:47 smithi022 ceph-mon[40250]: pgmap v10147: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2938994430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:49 smithi122 ceph-mon[32255]: pgmap v10148: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:36:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:36:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:49 smithi022 ceph-mon[31360]: pgmap v10148: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:49 smithi022 ceph-mon[40250]: pgmap v10148: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:50] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:36:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:51 smithi122 ceph-mon[32255]: pgmap v10149: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:51 smithi022 ceph-mon[31360]: pgmap v10149: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:36:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:51 smithi022 ceph-mon[40250]: pgmap v10149: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:36:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:36:52.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:53.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:53.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:53.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:53 smithi122 ceph-mon[32255]: pgmap v10150: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3219865566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:53 smithi022 ceph-mon[31360]: pgmap v10150: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3219865566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:53 smithi022 ceph-mon[40250]: pgmap v10150: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3219865566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:55 smithi122 ceph-mon[32255]: pgmap v10151: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:55 smithi022 ceph-mon[31360]: pgmap v10151: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:55 smithi022 ceph-mon[40250]: pgmap v10151: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:36:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:36:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:57 smithi122 ceph-mon[32255]: pgmap v10152: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:58.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:36:58.248 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:57 smithi022 ceph-mon[31360]: pgmap v10152: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:58.248 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:57 smithi022 ceph-mon[40250]: pgmap v10152: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:36:58.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:36:58.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:36:58.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:36:59.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/671896181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/671896181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:36:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/671896181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:00.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:37:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:37:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:36:59 smithi122 ceph-mon[32255]: pgmap v10153: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:36:59 smithi022 ceph-mon[31360]: pgmap v10153: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:36:59 smithi022 ceph-mon[40250]: pgmap v10153: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:00] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:37:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:01 smithi122 ceph-mon[32255]: pgmap v10154: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:01 smithi022 ceph-mon[31360]: pgmap v10154: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:01 smithi022 ceph-mon[40250]: pgmap v10154: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:37:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:37:03.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:03.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:03.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:03.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:03 smithi122 ceph-mon[32255]: pgmap v10155: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:03 smithi022 ceph-mon[31360]: pgmap v10155: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:03 smithi022 ceph-mon[40250]: pgmap v10155: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1005597241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1005597241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1005597241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:05 smithi122 ceph-mon[32255]: pgmap v10156: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:05 smithi022 ceph-mon[31360]: pgmap v10156: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:05 smithi022 ceph-mon[40250]: pgmap v10156: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:37:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:07 smithi122 ceph-mon[32255]: pgmap v10157: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:07 smithi022 ceph-mon[31360]: pgmap v10157: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:07 smithi022 ceph-mon[40250]: pgmap v10157: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:08.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:09.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:09.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:09.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:09 smithi022 ceph-mon[31360]: pgmap v10158: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3122725603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:09 smithi022 ceph-mon[40250]: pgmap v10158: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3122725603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:10] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:37:10.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:37:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:37:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:09 smithi122 ceph-mon[32255]: pgmap v10158: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3122725603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:11 smithi022 ceph-mon[31360]: pgmap v10159: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:11 smithi022 ceph-mon[40250]: pgmap v10159: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:11 smithi122 ceph-mon[32255]: pgmap v10159: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:37:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:37:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:13 smithi022 ceph-mon[31360]: pgmap v10160: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:13 smithi022 ceph-mon[40250]: pgmap v10160: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:14.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:13 smithi122 ceph-mon[32255]: pgmap v10160: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:14.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:14.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:14.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2049349699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2049349699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2049349699' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:15 smithi022 ceph-mon[31360]: pgmap v10161: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:15 smithi022 ceph-mon[40250]: pgmap v10161: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:15 smithi122 ceph-mon[32255]: pgmap v10161: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:37:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:17 smithi022 ceph-mon[31360]: pgmap v10162: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:17 smithi022 ceph-mon[40250]: pgmap v10162: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:17 smithi122 ceph-mon[32255]: pgmap v10162: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:19.741 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:19 smithi022 ceph-mon[31360]: pgmap v10163: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:19 smithi022 ceph-mon[40250]: pgmap v10163: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:20.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:20.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:20.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:20] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:37:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:19 smithi122 ceph-mon[32255]: pgmap v10163: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:37:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:37:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1466739405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1466739405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1466739405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:21 smithi022 ceph-mon[31360]: pgmap v10164: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:21 smithi022 ceph-mon[40250]: pgmap v10164: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:21 smithi122 ceph-mon[32255]: pgmap v10164: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:37:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:37:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:23 smithi022 ceph-mon[31360]: pgmap v10165: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:23 smithi022 ceph-mon[40250]: pgmap v10165: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:23 smithi122 ceph-mon[32255]: pgmap v10165: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:25.114 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:25.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:25.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:25.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:25 smithi022 ceph-mon[31360]: pgmap v10166: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1788329001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:25 smithi022 ceph-mon[40250]: pgmap v10166: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1788329001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:25 smithi122 ceph-mon[32255]: pgmap v10166: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1788329001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:37:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:27 smithi022 ceph-mon[31360]: pgmap v10167: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:27 smithi022 ceph-mon[40250]: pgmap v10167: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:27 smithi122 ceph-mon[32255]: pgmap v10167: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:29 smithi022 ceph-mon[31360]: pgmap v10168: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:29 smithi022 ceph-mon[40250]: pgmap v10168: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:30] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:37:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:37:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:37:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:29 smithi122 ceph-mon[32255]: pgmap v10168: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:30.483 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:30.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:30.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:30.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4063282104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4063282104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4063282104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:31 smithi022 ceph-mon[31360]: pgmap v10169: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:31 smithi022 ceph-mon[40250]: pgmap v10169: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:31 smithi122 ceph-mon[32255]: pgmap v10169: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:37:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:37:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[31360]: pgmap v10170: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:37:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:37:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:37:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:37:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[40250]: pgmap v10170: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:37:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:37:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:37:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:37:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:33 smithi122 ceph-mon[32255]: pgmap v10170: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:37:35.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:35 smithi122 ceph-mon[32255]: pgmap v10171: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:37:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:37:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:37:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:37:36.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:36.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:36.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:36.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[31360]: pgmap v10171: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:36.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:37:36.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:37:36.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:37:36.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:37:36.226 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[40250]: pgmap v10171: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:36.226 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:37:36.226 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:37:36.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:37:36.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:37:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2104136240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2104136240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2104136240' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:37:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:37 smithi022 ceph-mon[31360]: pgmap v10172: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:37 smithi022 ceph-mon[40250]: pgmap v10172: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:37 smithi122 ceph-mon[32255]: pgmap v10172: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:39 smithi022 ceph-mon[31360]: pgmap v10173: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:39 smithi022 ceph-mon[40250]: pgmap v10173: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:40] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:37:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:39 smithi122 ceph-mon[32255]: pgmap v10173: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:40.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:37:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:37:41.226 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:41.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:41.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:41.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:41 smithi022 ceph-mon[31360]: pgmap v10174: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1583964311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:41 smithi022 ceph-mon[40250]: pgmap v10174: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1583964311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:41 smithi122 ceph-mon[32255]: pgmap v10174: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1583964311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:37:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:37:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:43 smithi022 ceph-mon[31360]: pgmap v10175: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:43 smithi022 ceph-mon[40250]: pgmap v10175: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:43 smithi122 ceph-mon[32255]: pgmap v10175: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:45 smithi022 ceph-mon[31360]: pgmap v10176: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:45 smithi022 ceph-mon[40250]: pgmap v10176: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:45 smithi122 ceph-mon[32255]: pgmap v10176: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:46.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:46.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:46.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:46.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3609579886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3609579886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3609579886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:37:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:48 smithi022 ceph-mon[31360]: pgmap v10177: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:48 smithi022 ceph-mon[40250]: pgmap v10177: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:48 smithi122 ceph-mon[32255]: pgmap v10177: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:50 smithi022 ceph-mon[31360]: pgmap v10178: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:50 smithi022 ceph-mon[40250]: pgmap v10178: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:50 smithi122 ceph-mon[32255]: pgmap v10178: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:50] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:37:50.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:37:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:37:51.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:52 smithi022 ceph-mon[31360]: pgmap v10179: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:52 smithi022 ceph-mon[40250]: pgmap v10179: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:37:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:52 smithi122 ceph-mon[32255]: pgmap v10179: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:52.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:52.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:52.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:37:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:37:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2367329285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2367329285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2367329285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:54 smithi022 ceph-mon[31360]: pgmap v10180: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:54 smithi022 ceph-mon[40250]: pgmap v10180: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:54 smithi122 ceph-mon[32255]: pgmap v10180: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:56 smithi022 ceph-mon[31360]: pgmap v10181: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:56 smithi022 ceph-mon[40250]: pgmap v10181: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:56 smithi122 ceph-mon[32255]: pgmap v10181: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:37:57.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:37:57.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:37:57.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:37:57.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:37:57.699 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:37:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:58 smithi022 ceph-mon[31360]: pgmap v10182: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/136414979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:58 smithi022 ceph-mon[40250]: pgmap v10182: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/136414979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:58 smithi122 ceph-mon[32255]: pgmap v10182: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:37:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/136414979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:37:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:37:59 smithi022 ceph-mon[31360]: pgmap v10183: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:37:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:37:59 smithi022 ceph-mon[40250]: pgmap v10183: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:37:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:37:59 smithi122 ceph-mon[32255]: pgmap v10183: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:00] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:38:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:38:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:38:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:01 smithi122 ceph-mon[32255]: pgmap v10184: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:01 smithi022 ceph-mon[40250]: pgmap v10184: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:01 smithi022 ceph-mon[31360]: pgmap v10184: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:38:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:38:02.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:03.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:03.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:03.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:03 smithi122 ceph-mon[32255]: pgmap v10185: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1351147095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:03 smithi022 ceph-mon[31360]: pgmap v10185: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1351147095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:03 smithi022 ceph-mon[40250]: pgmap v10185: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1351147095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:05 smithi122 ceph-mon[32255]: pgmap v10186: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:05 smithi022 ceph-mon[31360]: pgmap v10186: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:05 smithi022 ceph-mon[40250]: pgmap v10186: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:38:08.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:07 smithi122 ceph-mon[32255]: pgmap v10187: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:07 smithi022 ceph-mon[31360]: pgmap v10187: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:07 smithi022 ceph-mon[40250]: pgmap v10187: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:38:08.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:08.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:08.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3250060638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3250060638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3250060638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:09 smithi122 ceph-mon[32255]: pgmap v10188: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:38:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:09 smithi022 ceph-mon[31360]: pgmap v10188: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:38:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:09 smithi022 ceph-mon[40250]: pgmap v10188: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:38:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:10] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:38:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:38:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:38:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:11 smithi122 ceph-mon[32255]: pgmap v10189: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:11 smithi022 ceph-mon[31360]: pgmap v10189: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:11 smithi022 ceph-mon[40250]: pgmap v10189: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:38:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:38:13.442 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:13.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:13.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:13.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:13 smithi122 ceph-mon[32255]: pgmap v10190: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1942395327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:13 smithi022 ceph-mon[31360]: pgmap v10190: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1942395327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:13 smithi022 ceph-mon[40250]: pgmap v10190: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1942395327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:15 smithi122 ceph-mon[32255]: pgmap v10191: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:38:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:15 smithi022 ceph-mon[31360]: pgmap v10191: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:38:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:15 smithi022 ceph-mon[40250]: pgmap v10191: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:38:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:38:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:17 smithi122 ceph-mon[32255]: pgmap v10192: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:17 smithi022 ceph-mon[31360]: pgmap v10192: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:17 smithi022 ceph-mon[40250]: pgmap v10192: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:38:18.812 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:19.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:19.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:19.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:19 smithi122 ceph-mon[32255]: pgmap v10193: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:38:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1906106599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:19 smithi022 ceph-mon[31360]: pgmap v10193: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:38:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1906106599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:19 smithi022 ceph-mon[40250]: pgmap v10193: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:38:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1906106599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:20] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:38:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:38:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:38:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:21 smithi122 ceph-mon[32255]: pgmap v10194: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:21 smithi022 ceph-mon[31360]: pgmap v10194: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:21 smithi022 ceph-mon[40250]: pgmap v10194: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:38:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:38:24.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:24.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:23 smithi022 ceph-mon[31360]: pgmap v10195: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:24.186 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:23 smithi022 ceph-mon[40250]: pgmap v10195: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:23 smithi122 ceph-mon[32255]: pgmap v10195: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:24.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:24.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:24.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:24.891 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3080424268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3080424268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3080424268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:25 smithi122 ceph-mon[32255]: pgmap v10196: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:25 smithi022 ceph-mon[31360]: pgmap v10196: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:25 smithi022 ceph-mon[40250]: pgmap v10196: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:38:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:27 smithi122 ceph-mon[32255]: pgmap v10197: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:27 smithi022 ceph-mon[31360]: pgmap v10197: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:27 smithi022 ceph-mon[40250]: pgmap v10197: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:29.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:29.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:29.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:29.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:29 smithi122 ceph-mon[32255]: pgmap v10198: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:29 smithi022 ceph-mon[31360]: pgmap v10198: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:29 smithi022 ceph-mon[40250]: pgmap v10198: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:30] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:38:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:38:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:38:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2054544384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2054544384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2054544384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:31 smithi122 ceph-mon[32255]: pgmap v10199: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:31 smithi022 ceph-mon[31360]: pgmap v10199: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:31 smithi022 ceph-mon[40250]: pgmap v10199: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:38:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:38:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:33 smithi122 ceph-mon[32255]: pgmap v10200: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:38:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:38:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:38:34.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:38:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[31360]: pgmap v10200: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:38:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:38:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:38:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:38:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[40250]: pgmap v10200: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:38:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:38:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:38:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:38:34.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:35.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:35.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:35.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:35 smithi122 ceph-mon[32255]: pgmap v10201: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/362244812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:35 smithi022 ceph-mon[31360]: pgmap v10201: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/362244812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:35 smithi022 ceph-mon[40250]: pgmap v10201: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/362244812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:38:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:38:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:38:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:38:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:38:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:38:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:38:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:38:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:38:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:38:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:38:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:38:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:38:38.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:37 smithi122 ceph-mon[32255]: pgmap v10202: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:37 smithi022 ceph-mon[31360]: pgmap v10202: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:37 smithi022 ceph-mon[40250]: pgmap v10202: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:39 smithi122 ceph-mon[32255]: pgmap v10203: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:39 smithi022 ceph-mon[31360]: pgmap v10203: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:39 smithi022 ceph-mon[40250]: pgmap v10203: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:40.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:40] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:38:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:38:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:38:40.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:40.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:40.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2101075841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2101075841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2101075841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:41 smithi122 ceph-mon[32255]: pgmap v10204: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:41 smithi022 ceph-mon[31360]: pgmap v10204: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:41 smithi022 ceph-mon[40250]: pgmap v10204: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:38:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:38:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:43 smithi122 ceph-mon[32255]: pgmap v10205: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:43 smithi022 ceph-mon[31360]: pgmap v10205: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:43 smithi022 ceph-mon[40250]: pgmap v10205: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:45.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:45.928 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:45 smithi022 ceph-mon[31360]: pgmap v10206: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:46.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:46.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:46.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:45 smithi122 ceph-mon[32255]: pgmap v10206: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:45 smithi022 ceph-mon[40250]: pgmap v10206: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/353460437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/353460437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/353460437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:38:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:47 smithi122 ceph-mon[32255]: pgmap v10207: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:47 smithi022 ceph-mon[31360]: pgmap v10207: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:47 smithi022 ceph-mon[40250]: pgmap v10207: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:50.255 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:50] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:38:50.255 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:49 smithi122 ceph-mon[32255]: pgmap v10208: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:49 smithi022 ceph-mon[31360]: pgmap v10208: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:49 smithi022 ceph-mon[40250]: pgmap v10208: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:50.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:38:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:38:51.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:51.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:51.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:51.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:51 smithi022 ceph-mon[40250]: pgmap v10209: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3623959848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:51 smithi022 ceph-mon[31360]: pgmap v10209: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3623959848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:51 smithi122 ceph-mon[32255]: pgmap v10209: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:38:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3623959848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:38:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:38:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:53 smithi022 ceph-mon[31360]: pgmap v10210: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:53 smithi022 ceph-mon[40250]: pgmap v10210: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:53 smithi122 ceph-mon[32255]: pgmap v10210: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:55 smithi022 ceph-mon[31360]: pgmap v10211: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:55 smithi022 ceph-mon[40250]: pgmap v10211: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:56.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:38:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:55 smithi122 ceph-mon[32255]: pgmap v10211: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:38:56.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:38:56.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:38:56.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:38:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1859092726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1859092726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1859092726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:38:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:38:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:57 smithi022 ceph-mon[31360]: pgmap v10212: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:57 smithi022 ceph-mon[40250]: pgmap v10212: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:38:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:57 smithi122 ceph-mon[32255]: pgmap v10212: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:00.265 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:00] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:39:00.266 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:38:59 smithi122 ceph-mon[32255]: pgmap v10213: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:38:59 smithi022 ceph-mon[31360]: pgmap v10213: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:38:59 smithi022 ceph-mon[40250]: pgmap v10213: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:00.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:39:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:39:01.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:02.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:02.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:02.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:02.152 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:01 smithi022 ceph-mon[40250]: pgmap v10214: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:02.152 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:02.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:01 smithi022 ceph-mon[31360]: pgmap v10214: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:02.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:01 smithi122 ceph-mon[32255]: pgmap v10214: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:39:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:39:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4059548738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4059548738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4059548738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:03 smithi022 ceph-mon[31360]: pgmap v10215: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:03 smithi022 ceph-mon[40250]: pgmap v10215: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:03 smithi122 ceph-mon[32255]: pgmap v10215: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:05 smithi022 ceph-mon[31360]: pgmap v10216: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:05 smithi022 ceph-mon[40250]: pgmap v10216: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:05 smithi122 ceph-mon[32255]: pgmap v10216: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:07.153 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:07.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:07.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:07.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:07.523 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:39:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:07 smithi022 ceph-mon[31360]: pgmap v10217: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2581070584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:07 smithi022 ceph-mon[40250]: pgmap v10217: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2581070584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:07 smithi122 ceph-mon[32255]: pgmap v10217: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2581070584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:10.276 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:10] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:39:10.277 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:09 smithi122 ceph-mon[32255]: pgmap v10218: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:09 smithi022 ceph-mon[31360]: pgmap v10218: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:09 smithi022 ceph-mon[40250]: pgmap v10218: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:10.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:39:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:39:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:11 smithi022 ceph-mon[31360]: pgmap v10219: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:11 smithi022 ceph-mon[40250]: pgmap v10219: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:11 smithi122 ceph-mon[32255]: pgmap v10219: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:12.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:39:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:39:12.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:12.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:12.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/996364549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/996364549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/996364549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:13 smithi022 ceph-mon[31360]: pgmap v10220: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:13 smithi022 ceph-mon[40250]: pgmap v10220: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:13 smithi122 ceph-mon[32255]: pgmap v10220: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:15 smithi022 ceph-mon[31360]: pgmap v10221: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:15 smithi022 ceph-mon[40250]: pgmap v10221: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:15 smithi122 ceph-mon[32255]: pgmap v10221: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:39:17.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:18.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:18.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:18.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:18.263 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:17 smithi022 ceph-mon[40250]: pgmap v10222: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:18.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:17 smithi022 ceph-mon[31360]: pgmap v10222: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:17 smithi122 ceph-mon[32255]: pgmap v10222: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2667642118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2667642118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2667642118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:19 smithi022 ceph-mon[31360]: pgmap v10223: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:19 smithi022 ceph-mon[40250]: pgmap v10223: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:20.286 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:20] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:39:20.287 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:19 smithi122 ceph-mon[32255]: pgmap v10223: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:20.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:39:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:39:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:21 smithi022 ceph-mon[31360]: pgmap v10224: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:21 smithi022 ceph-mon[40250]: pgmap v10224: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:21 smithi122 ceph-mon[32255]: pgmap v10224: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:39:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:23.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:23.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:23.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:23 smithi022 ceph-mon[31360]: pgmap v10225: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3582814890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:23 smithi022 ceph-mon[40250]: pgmap v10225: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3582814890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:23 smithi122 ceph-mon[32255]: pgmap v10225: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3582814890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:26 smithi022 ceph-mon[31360]: pgmap v10226: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:26 smithi022 ceph-mon[40250]: pgmap v10226: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:26 smithi122 ceph-mon[32255]: pgmap v10226: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:39:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:28 smithi022 ceph-mon[31360]: pgmap v10227: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:28 smithi022 ceph-mon[40250]: pgmap v10227: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:28 smithi122 ceph-mon[32255]: pgmap v10227: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:28.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:29.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:29.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:29.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:29.009 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2861974593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:29.010 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2861974593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2861974593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:30 smithi022 ceph-mon[31360]: pgmap v10228: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:30 smithi022 ceph-mon[40250]: pgmap v10228: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:30.297 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:30] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:39:30.297 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:30 smithi122 ceph-mon[32255]: pgmap v10228: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:30.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:39:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:39:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:32 smithi022 ceph-mon[31360]: pgmap v10229: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:32 smithi022 ceph-mon[40250]: pgmap v10229: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:32 smithi122 ceph-mon[32255]: pgmap v10229: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:39:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:39:34.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[31360]: pgmap v10230: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:39:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:39:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:39:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:39:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[40250]: pgmap v10230: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:39:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:39:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:39:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:39:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:34 smithi122 ceph-mon[32255]: pgmap v10230: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:39:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:39:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:39:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:39:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4116068426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4116068426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4116068426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:36 smithi022 ceph-mon[31360]: pgmap v10231: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:36 smithi022 ceph-mon[40250]: pgmap v10231: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:36.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:36 smithi122 ceph-mon[32255]: pgmap v10231: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:39:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:39:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:39:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:39:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:39:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:39:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:39:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:39:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:39:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:39:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:39:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:39:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:39:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:38 smithi022 ceph-mon[31360]: pgmap v10232: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:38 smithi022 ceph-mon[40250]: pgmap v10232: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:38 smithi122 ceph-mon[32255]: pgmap v10232: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:39.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:39.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:39.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:39.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:40.307 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:40] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:39:40.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:40 smithi122 ceph-mon[32255]: pgmap v10233: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:40.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3924151397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:40 smithi022 ceph-mon[31360]: pgmap v10233: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3924151397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:40 smithi022 ceph-mon[40250]: pgmap v10233: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3924151397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:40.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:39:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:39:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:42 smithi122 ceph-mon[32255]: pgmap v10234: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:42 smithi022 ceph-mon[31360]: pgmap v10234: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:42 smithi022 ceph-mon[40250]: pgmap v10234: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:39:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:39:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:44 smithi122 ceph-mon[32255]: pgmap v10235: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:44 smithi022 ceph-mon[31360]: pgmap v10235: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:44 smithi022 ceph-mon[40250]: pgmap v10235: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:44.748 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:45.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:45.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:45.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:46 smithi122 ceph-mon[32255]: pgmap v10236: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4159837170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:46 smithi022 ceph-mon[31360]: pgmap v10236: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4159837170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:46 smithi022 ceph-mon[40250]: pgmap v10236: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4159837170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:39:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:48 smithi122 ceph-mon[32255]: pgmap v10237: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:48 smithi022 ceph-mon[31360]: pgmap v10237: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:48 smithi022 ceph-mon[40250]: pgmap v10237: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:49 smithi122 ceph-mon[32255]: pgmap v10238: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:49 smithi022 ceph-mon[31360]: pgmap v10238: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:49 smithi022 ceph-mon[40250]: pgmap v10238: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:50.119 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:50] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:39:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:39:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:39:50.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:50.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:50.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2352815979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2352815979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2352815979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:51 smithi122 ceph-mon[32255]: pgmap v10239: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:51 smithi022 ceph-mon[31360]: pgmap v10239: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:51 smithi022 ceph-mon[40250]: pgmap v10239: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:39:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:39:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:39:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:53 smithi122 ceph-mon[32255]: pgmap v10240: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:53 smithi022 ceph-mon[31360]: pgmap v10240: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:53 smithi022 ceph-mon[40250]: pgmap v10240: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:55.489 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:39:55.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:39:55.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:39:55.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:39:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:55 smithi122 ceph-mon[32255]: pgmap v10241: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1118752404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:55 smithi022 ceph-mon[31360]: pgmap v10241: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1118752404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:55 smithi022 ceph-mon[40250]: pgmap v10241: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:39:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1118752404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:39:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:39:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:57 smithi122 ceph-mon[32255]: pgmap v10242: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:57 smithi022 ceph-mon[31360]: pgmap v10242: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:39:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:57 smithi022 ceph-mon[40250]: pgmap v10242: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:39:59 smithi122 ceph-mon[32255]: pgmap v10243: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:39:59 smithi022 ceph-mon[31360]: pgmap v10243: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:39:59 smithi022 ceph-mon[40250]: pgmap v10243: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:00] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:40:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:40:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:40:00.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T23:40:01.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:01.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:01.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:01.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T23:40:01.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T23:40:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:01 smithi122 ceph-mon[32255]: pgmap v10244: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4023046442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:01 smithi022 ceph-mon[31360]: pgmap v10244: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4023046442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:01 smithi022 ceph-mon[40250]: pgmap v10244: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4023046442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:40:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:40:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:03 smithi122 ceph-mon[32255]: pgmap v10245: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:03 smithi022 ceph-mon[31360]: pgmap v10245: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:03 smithi022 ceph-mon[40250]: pgmap v10245: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:05 smithi122 ceph-mon[32255]: pgmap v10246: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:06.228 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:06.229 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:05 smithi022 ceph-mon[31360]: pgmap v10246: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:06.229 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:05 smithi022 ceph-mon[40250]: pgmap v10246: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:06.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:06.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:06.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:06.907 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3286480084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3286480084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3286480084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:40:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:07 smithi122 ceph-mon[32255]: pgmap v10247: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:07 smithi022 ceph-mon[31360]: pgmap v10247: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:07 smithi022 ceph-mon[40250]: pgmap v10247: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:09 smithi122 ceph-mon[32255]: pgmap v10248: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:09 smithi022 ceph-mon[31360]: pgmap v10248: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:09 smithi022 ceph-mon[40250]: pgmap v10248: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:10] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:40:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:40:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:40:11.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:11.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:11.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:11.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:11 smithi122 ceph-mon[32255]: pgmap v10249: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:11 smithi022 ceph-mon[31360]: pgmap v10249: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:11 smithi022 ceph-mon[40250]: pgmap v10249: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:40:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:40:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2800939517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2800939517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2800939517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:13 smithi122 ceph-mon[32255]: pgmap v10250: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:13 smithi022 ceph-mon[31360]: pgmap v10250: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:13 smithi022 ceph-mon[40250]: pgmap v10250: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:15 smithi122 ceph-mon[32255]: pgmap v10251: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:15 smithi022 ceph-mon[31360]: pgmap v10251: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:15 smithi022 ceph-mon[40250]: pgmap v10251: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:16.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:17.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:17.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:17.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:40:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:17 smithi122 ceph-mon[32255]: pgmap v10252: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1413892199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:17 smithi022 ceph-mon[31360]: pgmap v10252: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1413892199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:17 smithi022 ceph-mon[40250]: pgmap v10252: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1413892199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:20.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:19 smithi122 ceph-mon[32255]: pgmap v10253: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:20.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:40:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:19 smithi022 ceph-mon[31360]: pgmap v10253: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:19 smithi022 ceph-mon[40250]: pgmap v10253: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:20.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:40:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:40:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:21 smithi122 ceph-mon[32255]: pgmap v10254: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:21 smithi022 ceph-mon[31360]: pgmap v10254: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:21 smithi022 ceph-mon[40250]: pgmap v10254: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:22.347 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:40:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:40:22.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:22.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:22.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4131617990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4131617990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:23.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4131617990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:23 smithi022 ceph-mon[31360]: pgmap v10255: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:23 smithi022 ceph-mon[40250]: pgmap v10255: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:23 smithi122 ceph-mon[32255]: pgmap v10255: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:25 smithi022 ceph-mon[31360]: pgmap v10256: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:25 smithi022 ceph-mon[40250]: pgmap v10256: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:25 smithi122 ceph-mon[32255]: pgmap v10256: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:27.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:27.716 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:40:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:27 smithi022 ceph-mon[31360]: pgmap v10257: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:27 smithi022 ceph-mon[40250]: pgmap v10257: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:28.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:28.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:28.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:27 smithi122 ceph-mon[32255]: pgmap v10257: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4125458262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4125458262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4125458262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:29 smithi022 ceph-mon[31360]: pgmap v10258: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:29 smithi022 ceph-mon[40250]: pgmap v10258: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:30.360 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:40:30.360 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:29 smithi122 ceph-mon[32255]: pgmap v10258: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:30.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:40:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:40:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:31 smithi022 ceph-mon[31360]: pgmap v10259: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:31 smithi022 ceph-mon[40250]: pgmap v10259: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:31 smithi122 ceph-mon[32255]: pgmap v10259: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:40:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:40:33.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:33.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:33.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:33.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[31360]: pgmap v10260: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:40:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:40:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:40:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:40:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3924371225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[40250]: pgmap v10260: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:40:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:40:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:40:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:40:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3924371225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:33 smithi122 ceph-mon[32255]: pgmap v10260: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:40:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:40:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:40:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:40:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3924371225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:35 smithi022 ceph-mon[31360]: pgmap v10261: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:35 smithi022 ceph-mon[40250]: pgmap v10261: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:35 smithi122 ceph-mon[32255]: pgmap v10261: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:40:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:40:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:40:37.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:40:37.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:40:37.270 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:40:37.270 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:40:37.270 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:40:37.271 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:40:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:40:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[31360]: pgmap v10262: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[40250]: pgmap v10262: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:37 smithi122 ceph-mon[32255]: pgmap v10262: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:40:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:40:38.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:38.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:38.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:38.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2358530922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2358530922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2358530922' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:39 smithi022 ceph-mon[31360]: pgmap v10263: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:39 smithi022 ceph-mon[40250]: pgmap v10263: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:40.366 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:40] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:40:40.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:39 smithi122 ceph-mon[32255]: pgmap v10263: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:40.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:40:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:40:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:41 smithi022 ceph-mon[31360]: pgmap v10264: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:41 smithi022 ceph-mon[40250]: pgmap v10264: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:41 smithi122 ceph-mon[32255]: pgmap v10264: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:40:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:40:43.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:44.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:44.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:44.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:44.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:43 smithi022 ceph-mon[40250]: pgmap v10265: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:44.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:43 smithi022 ceph-mon[31360]: pgmap v10265: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:43 smithi122 ceph-mon[32255]: pgmap v10265: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3719588037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3719588037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3719588037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:45 smithi022 ceph-mon[31360]: pgmap v10266: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:45 smithi022 ceph-mon[40250]: pgmap v10266: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:45 smithi122 ceph-mon[32255]: pgmap v10266: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:40:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:47 smithi022 ceph-mon[31360]: pgmap v10267: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:47 smithi022 ceph-mon[40250]: pgmap v10267: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:47 smithi122 ceph-mon[32255]: pgmap v10267: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:49 smithi022 ceph-mon[31360]: pgmap v10268: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/373641236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:49 smithi022 ceph-mon[40250]: pgmap v10268: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/373641236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:50.375 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:50] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:40:50.375 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:49 smithi122 ceph-mon[32255]: pgmap v10268: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:50.375 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/373641236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:50.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:40:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:40:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:51 smithi022 ceph-mon[31360]: pgmap v10269: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:51 smithi022 ceph-mon[40250]: pgmap v10269: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:51 smithi122 ceph-mon[32255]: pgmap v10269: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:40:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:40:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:40:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:53 smithi022 ceph-mon[31360]: pgmap v10270: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:53 smithi022 ceph-mon[40250]: pgmap v10270: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:53 smithi122 ceph-mon[32255]: pgmap v10270: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:54.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:40:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:40:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:40:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:40:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:55 smithi122 ceph-mon[32255]: pgmap v10271: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/515124704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:55 smithi022 ceph-mon[31360]: pgmap v10271: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/515124704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:55 smithi022 ceph-mon[40250]: pgmap v10271: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:40:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/515124704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:40:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:40:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:57 smithi122 ceph-mon[32255]: pgmap v10272: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:57 smithi022 ceph-mon[31360]: pgmap v10272: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:57 smithi022 ceph-mon[40250]: pgmap v10272: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:40:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:40:59 smithi122 ceph-mon[32255]: pgmap v10273: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:40:59 smithi022 ceph-mon[31360]: pgmap v10273: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:40:59 smithi022 ceph-mon[40250]: pgmap v10273: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:00.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:00.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:00.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:41:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:41:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:00] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:41:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/392178420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/392178420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/392178420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:01 smithi122 ceph-mon[32255]: pgmap v10274: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:01 smithi022 ceph-mon[31360]: pgmap v10274: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:01 smithi022 ceph-mon[40250]: pgmap v10274: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:41:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:41:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:03 smithi122 ceph-mon[32255]: pgmap v10275: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:03 smithi022 ceph-mon[31360]: pgmap v10275: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:03 smithi022 ceph-mon[40250]: pgmap v10275: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:05.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:05.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:05.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:05.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:05 smithi022 ceph-mon[31360]: pgmap v10276: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1407569551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:05 smithi022 ceph-mon[40250]: pgmap v10276: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1407569551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:05 smithi122 ceph-mon[32255]: pgmap v10276: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1407569551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:41:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:07 smithi122 ceph-mon[32255]: pgmap v10277: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:07 smithi022 ceph-mon[31360]: pgmap v10277: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:07 smithi022 ceph-mon[40250]: pgmap v10277: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:09 smithi122 ceph-mon[32255]: pgmap v10278: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:09 smithi022 ceph-mon[31360]: pgmap v10278: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:09 smithi022 ceph-mon[40250]: pgmap v10278: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:10] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:41:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:41:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:41:10.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:11.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:11.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:11.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:11 smithi122 ceph-mon[32255]: pgmap v10279: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/996583764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:11 smithi022 ceph-mon[40250]: pgmap v10279: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/996583764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:11 smithi022 ceph-mon[31360]: pgmap v10279: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/996583764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:41:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:41:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:13 smithi122 ceph-mon[32255]: pgmap v10280: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:13 smithi022 ceph-mon[40250]: pgmap v10280: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:13 smithi022 ceph-mon[31360]: pgmap v10280: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:16.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:15 smithi122 ceph-mon[32255]: pgmap v10281: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:15 smithi022 ceph-mon[31360]: pgmap v10281: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:15 smithi022 ceph-mon[40250]: pgmap v10281: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:16.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:16.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:16.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:17.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3401027830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3401027830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3401027830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:41:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:17 smithi122 ceph-mon[32255]: pgmap v10282: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:18.318 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:17 smithi022 ceph-mon[31360]: pgmap v10282: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:18.319 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:17 smithi022 ceph-mon[40250]: pgmap v10282: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:20.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:19 smithi122 ceph-mon[32255]: pgmap v10283: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:20.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:20] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:41:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:19 smithi022 ceph-mon[31360]: pgmap v10283: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:19 smithi022 ceph-mon[40250]: pgmap v10283: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:20.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:41:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:41:21.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:21.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:21.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:21.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:21 smithi122 ceph-mon[32255]: pgmap v10284: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:22.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3525466275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:21 smithi022 ceph-mon[31360]: pgmap v10284: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:22.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3525466275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:21 smithi022 ceph-mon[40250]: pgmap v10284: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3525466275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:41:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:41:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:23 smithi122 ceph-mon[32255]: pgmap v10285: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:23 smithi022 ceph-mon[31360]: pgmap v10285: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:23 smithi022 ceph-mon[40250]: pgmap v10285: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:25 smithi022 ceph-mon[31360]: pgmap v10286: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:25 smithi022 ceph-mon[40250]: pgmap v10286: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:25 smithi122 ceph-mon[32255]: pgmap v10286: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:26.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:27.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:27.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:27.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:41:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:27 smithi022 ceph-mon[31360]: pgmap v10287: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3310299413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:27 smithi022 ceph-mon[40250]: pgmap v10287: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3310299413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:27 smithi122 ceph-mon[32255]: pgmap v10287: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3310299413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:29 smithi022 ceph-mon[31360]: pgmap v10288: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:29 smithi022 ceph-mon[40250]: pgmap v10288: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:30.409 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:30] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:41:30.409 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:29 smithi122 ceph-mon[32255]: pgmap v10288: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:30.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:41:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:41:32.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:31 smithi022 ceph-mon[31360]: pgmap v10289: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:31 smithi022 ceph-mon[40250]: pgmap v10289: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:31 smithi122 ceph-mon[32255]: pgmap v10289: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:32.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:32.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:32.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:41:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:41:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3175545777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3175545777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3175545777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[31360]: pgmap v10290: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:41:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:41:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:41:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:41:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[40250]: pgmap v10290: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:41:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:41:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:41:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:41:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:33 smithi122 ceph-mon[32255]: pgmap v10290: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:41:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:41:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:41:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:41:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:35 smithi022 ceph-mon[31360]: pgmap v10291: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:35 smithi022 ceph-mon[40250]: pgmap v10291: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:35 smithi122 ceph-mon[32255]: pgmap v10291: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:37.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:37.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:41:37.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:37.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:37.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[31360]: pgmap v10292: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:41:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:41:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:41:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/331615328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[40250]: pgmap v10292: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:41:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:41:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:41:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/331615328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:38.162 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:37 smithi122 ceph-mon[32255]: pgmap v10292: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:38.162 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:41:38.162 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:41:38.163 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:41:38.163 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/331615328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:41:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:39 smithi122 ceph-mon[32255]: pgmap v10293: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:41:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:39 smithi022 ceph-mon[31360]: pgmap v10293: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:41:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:39 smithi022 ceph-mon[40250]: pgmap v10293: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:41:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:41:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:41:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:41 smithi122 ceph-mon[32255]: pgmap v10294: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:41 smithi022 ceph-mon[31360]: pgmap v10294: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:41 smithi022 ceph-mon[40250]: pgmap v10294: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:41:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:41:42.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:43 smithi122 ceph-mon[32255]: pgmap v10295: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2158692776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:43 smithi022 ceph-mon[31360]: pgmap v10295: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2158692776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:43 smithi022 ceph-mon[40250]: pgmap v10295: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2158692776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:45 smithi122 ceph-mon[32255]: pgmap v10296: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:45 smithi022 ceph-mon[31360]: pgmap v10296: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:45 smithi022 ceph-mon[40250]: pgmap v10296: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:41:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:47 smithi122 ceph-mon[32255]: pgmap v10297: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:47 smithi022 ceph-mon[31360]: pgmap v10297: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:47 smithi022 ceph-mon[40250]: pgmap v10297: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:48.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:48.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:48.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:48.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:48.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/946025932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:49.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/946025932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/946025932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:49 smithi122 ceph-mon[32255]: pgmap v10298: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:49 smithi022 ceph-mon[31360]: pgmap v10298: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:49 smithi022 ceph-mon[40250]: pgmap v10298: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:50.430 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:50] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:41:50.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:41:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:41:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:51 smithi122 ceph-mon[32255]: pgmap v10299: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:51 smithi022 ceph-mon[31360]: pgmap v10299: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:51 smithi022 ceph-mon[40250]: pgmap v10299: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:41:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:41:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:41:53.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:54.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:54.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:54.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:41:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:53 smithi122 ceph-mon[32255]: pgmap v10300: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:53 smithi022 ceph-mon[31360]: pgmap v10300: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:53 smithi022 ceph-mon[40250]: pgmap v10300: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2592199896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2592199896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2592199896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:41:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:55 smithi122 ceph-mon[32255]: pgmap v10301: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:55 smithi022 ceph-mon[31360]: pgmap v10301: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:55 smithi022 ceph-mon[40250]: pgmap v10301: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:41:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:41:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:57 smithi122 ceph-mon[32255]: pgmap v10302: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:57 smithi022 ceph-mon[31360]: pgmap v10302: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:57 smithi022 ceph-mon[40250]: pgmap v10302: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:41:59.034 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:41:59.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:41:59.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:41:59.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:59 smithi022 ceph-mon[31360]: pgmap v10303: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:41:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2069601714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:59 smithi022 ceph-mon[40250]: pgmap v10303: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:41:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2069601714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:00.433 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:00] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:42:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:59 smithi122 ceph-mon[32255]: pgmap v10303: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:41:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2069601714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:00.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:42:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:42:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:01 smithi022 ceph-mon[31360]: pgmap v10304: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:01 smithi022 ceph-mon[40250]: pgmap v10304: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:01 smithi122 ceph-mon[32255]: pgmap v10304: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:42:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:42:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:03 smithi022 ceph-mon[31360]: pgmap v10305: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:03 smithi022 ceph-mon[40250]: pgmap v10305: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:03 smithi122 ceph-mon[32255]: pgmap v10305: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:04.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:04.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:04.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2284760303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2284760303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2284760303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:05 smithi022 ceph-mon[31360]: pgmap v10306: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:05 smithi022 ceph-mon[40250]: pgmap v10306: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:05 smithi122 ceph-mon[32255]: pgmap v10306: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:42:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:07 smithi022 ceph-mon[31360]: pgmap v10307: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:07 smithi022 ceph-mon[40250]: pgmap v10307: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:07 smithi122 ceph-mon[32255]: pgmap v10307: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:09.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:09 smithi022 ceph-mon[31360]: pgmap v10308: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:09 smithi022 ceph-mon[40250]: pgmap v10308: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:10.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:10.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:10.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:09 smithi122 ceph-mon[32255]: pgmap v10308: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:10] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:42:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:42:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:42:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1694505611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1694505611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1694505611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:11 smithi022 ceph-mon[31360]: pgmap v10309: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:11 smithi022 ceph-mon[40250]: pgmap v10309: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:11 smithi122 ceph-mon[32255]: pgmap v10309: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:42:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:42:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:13 smithi022 ceph-mon[31360]: pgmap v10310: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:13 smithi022 ceph-mon[40250]: pgmap v10310: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:13 smithi122 ceph-mon[32255]: pgmap v10310: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:15.145 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:15.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:15.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:15.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:15 smithi022 ceph-mon[31360]: pgmap v10311: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/931478756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:15 smithi022 ceph-mon[40250]: pgmap v10311: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/931478756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:15 smithi122 ceph-mon[32255]: pgmap v10311: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/931478756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:42:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:17 smithi022 ceph-mon[31360]: pgmap v10312: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:17 smithi022 ceph-mon[40250]: pgmap v10312: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:17 smithi122 ceph-mon[32255]: pgmap v10312: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:19 smithi022 ceph-mon[31360]: pgmap v10313: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:19 smithi022 ceph-mon[40250]: pgmap v10313: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:20] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:42:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:19 smithi122 ceph-mon[32255]: pgmap v10313: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:20.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:20.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:20.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:20.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:42:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:42:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1630821129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1630821129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1630821129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:21 smithi022 ceph-mon[31360]: pgmap v10314: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:21 smithi022 ceph-mon[40250]: pgmap v10314: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:21 smithi122 ceph-mon[32255]: pgmap v10314: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:42:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:42:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:23 smithi022 ceph-mon[31360]: pgmap v10315: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:23 smithi022 ceph-mon[40250]: pgmap v10315: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:23 smithi122 ceph-mon[32255]: pgmap v10315: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:25.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:26.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:26.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:26.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:26.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:25 smithi022 ceph-mon[31360]: pgmap v10316: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:26.255 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:25 smithi022 ceph-mon[40250]: pgmap v10316: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:25 smithi122 ceph-mon[32255]: pgmap v10316: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1787338443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1787338443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1787338443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:42:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:27 smithi022 ceph-mon[31360]: pgmap v10317: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:27 smithi022 ceph-mon[40250]: pgmap v10317: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:27 smithi122 ceph-mon[32255]: pgmap v10317: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:29 smithi022 ceph-mon[31360]: pgmap v10318: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:29 smithi022 ceph-mon[40250]: pgmap v10318: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:29 smithi122 ceph-mon[32255]: pgmap v10318: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:30] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:42:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:42:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:42:31.255 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:31.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:31.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:31.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:31 smithi022 ceph-mon[31360]: pgmap v10319: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2188621031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:31 smithi022 ceph-mon[40250]: pgmap v10319: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2188621031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:31 smithi122 ceph-mon[32255]: pgmap v10319: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2188621031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:42:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:42:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[40250]: pgmap v10320: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:42:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:42:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:42:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:42:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[31360]: pgmap v10320: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:42:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:42:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:42:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:42:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:33 smithi122 ceph-mon[32255]: pgmap v10320: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:42:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:42:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:42:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:42:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:35 smithi022 ceph-mon[31360]: pgmap v10321: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:35 smithi022 ceph-mon[40250]: pgmap v10321: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:35 smithi122 ceph-mon[32255]: pgmap v10321: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:36.625 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2509982731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2509982731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2509982731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:42:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:38 smithi022 ceph-mon[31360]: pgmap v10322: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:38 smithi022 ceph-mon[40250]: pgmap v10322: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:38 smithi122 ceph-mon[32255]: pgmap v10322: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:42:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:42:39.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:42:39.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:42:39.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:42:39.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:42:39.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:42:39.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:42:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:42:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:42:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:42:39.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:42:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:40 smithi022 ceph-mon[31360]: pgmap v10323: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:40 smithi022 ceph-mon[40250]: pgmap v10323: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:40] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:42:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:40 smithi122 ceph-mon[32255]: pgmap v10323: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:42:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:42:41.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:42 smithi022 ceph-mon[31360]: pgmap v10324: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:42 smithi022 ceph-mon[40250]: pgmap v10324: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:42 smithi122 ceph-mon[32255]: pgmap v10324: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:42.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:42.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:42.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:42:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:42:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1858992106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1858992106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1858992106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:44 smithi022 ceph-mon[31360]: pgmap v10325: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:44 smithi022 ceph-mon[40250]: pgmap v10325: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:44 smithi122 ceph-mon[32255]: pgmap v10325: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:46 smithi022 ceph-mon[31360]: pgmap v10326: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:46 smithi022 ceph-mon[40250]: pgmap v10326: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:46 smithi122 ceph-mon[32255]: pgmap v10326: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:47.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:47.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:47.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:47.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:47.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:42:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:48 smithi022 ceph-mon[31360]: pgmap v10327: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1369762430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:48 smithi022 ceph-mon[40250]: pgmap v10327: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1369762430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:48 smithi122 ceph-mon[32255]: pgmap v10327: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1369762430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:50] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:42:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:50 smithi122 ceph-mon[32255]: pgmap v10328: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:50 smithi022 ceph-mon[31360]: pgmap v10328: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:50 smithi022 ceph-mon[40250]: pgmap v10328: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:42:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:42:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:52 smithi122 ceph-mon[32255]: pgmap v10329: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:52 smithi022 ceph-mon[31360]: pgmap v10329: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:52 smithi022 ceph-mon[40250]: pgmap v10329: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:42:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:42:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:42:52.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:53.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:53.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:53.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:54 smithi122 ceph-mon[32255]: pgmap v10330: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4241495398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:54 smithi022 ceph-mon[31360]: pgmap v10330: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4241495398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:54 smithi022 ceph-mon[40250]: pgmap v10330: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4241495398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:56 smithi122 ceph-mon[32255]: pgmap v10331: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:56 smithi022 ceph-mon[31360]: pgmap v10331: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:56 smithi022 ceph-mon[40250]: pgmap v10331: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:42:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:42:58.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:42:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:58 smithi122 ceph-mon[32255]: pgmap v10332: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:58.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:42:58.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:42:58.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:42:58.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:58 smithi022 ceph-mon[31360]: pgmap v10332: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:58.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:58 smithi022 ceph-mon[40250]: pgmap v10332: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:42:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:42:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/554516035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:42:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/554516035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:42:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:42:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/554516035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:00 smithi122 ceph-mon[32255]: pgmap v10333: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:00] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:43:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:00 smithi022 ceph-mon[31360]: pgmap v10333: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:00 smithi022 ceph-mon[40250]: pgmap v10333: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:00.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:43:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:43:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:02 smithi122 ceph-mon[32255]: pgmap v10334: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:02 smithi022 ceph-mon[31360]: pgmap v10334: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:02 smithi022 ceph-mon[40250]: pgmap v10334: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:43:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:43:03.478 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:03.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:03.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:03.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:04 smithi022 ceph-mon[31360]: pgmap v10335: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1326998035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:04 smithi022 ceph-mon[40250]: pgmap v10335: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1326998035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:04 smithi122 ceph-mon[32255]: pgmap v10335: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1326998035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:06 smithi122 ceph-mon[32255]: pgmap v10336: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:06 smithi022 ceph-mon[31360]: pgmap v10336: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:06 smithi022 ceph-mon[40250]: pgmap v10336: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:43:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:08 smithi122 ceph-mon[32255]: pgmap v10337: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:08.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:08 smithi022 ceph-mon[31360]: pgmap v10337: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:08.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:08 smithi022 ceph-mon[40250]: pgmap v10337: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:08.848 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:09.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:09.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:09.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:10] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:43:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:10 smithi122 ceph-mon[32255]: pgmap v10338: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1272916535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:10 smithi022 ceph-mon[31360]: pgmap v10338: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1272916535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:10 smithi022 ceph-mon[40250]: pgmap v10338: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1272916535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:43:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:43:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:12 smithi122 ceph-mon[32255]: pgmap v10339: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:12 smithi022 ceph-mon[31360]: pgmap v10339: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:12 smithi022 ceph-mon[40250]: pgmap v10339: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:43:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:43:14.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:14 smithi122 ceph-mon[32255]: pgmap v10340: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:14 smithi022 ceph-mon[31360]: pgmap v10340: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:14 smithi022 ceph-mon[40250]: pgmap v10340: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:14.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:14.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:14.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1332968863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1332968863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1332968863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:16 smithi122 ceph-mon[32255]: pgmap v10341: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:16 smithi022 ceph-mon[31360]: pgmap v10341: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:16 smithi022 ceph-mon[40250]: pgmap v10341: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:43:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:18 smithi122 ceph-mon[32255]: pgmap v10342: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:18 smithi022 ceph-mon[31360]: pgmap v10342: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:18 smithi022 ceph-mon[40250]: pgmap v10342: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:19 smithi122 ceph-mon[32255]: pgmap v10343: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:19 smithi022 ceph-mon[31360]: pgmap v10343: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:19 smithi022 ceph-mon[40250]: pgmap v10343: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:19.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:19.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:19.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:19.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:20] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:43:20.925 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:43:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:43:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3017513094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3017513094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3017513094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:21 smithi122 ceph-mon[32255]: pgmap v10344: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:21 smithi022 ceph-mon[31360]: pgmap v10344: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:21 smithi022 ceph-mon[40250]: pgmap v10344: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:43:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:43:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:23 smithi122 ceph-mon[32255]: pgmap v10345: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:23 smithi022 ceph-mon[31360]: pgmap v10345: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:23 smithi022 ceph-mon[40250]: pgmap v10345: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:25.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:25.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:25.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:25 smithi122 ceph-mon[32255]: pgmap v10346: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/307629760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:25 smithi022 ceph-mon[31360]: pgmap v10346: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/307629760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:25 smithi022 ceph-mon[40250]: pgmap v10346: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/307629760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:43:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:27 smithi122 ceph-mon[32255]: pgmap v10347: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:27 smithi022 ceph-mon[31360]: pgmap v10347: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:27 smithi022 ceph-mon[40250]: pgmap v10347: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:29 smithi022 ceph-mon[31360]: pgmap v10348: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:29 smithi022 ceph-mon[40250]: pgmap v10348: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:30.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:30] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:43:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:29 smithi122 ceph-mon[32255]: pgmap v10348: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:30.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:30.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:30.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:43:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:43:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1896830710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1896830710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1896830710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:31 smithi022 ceph-mon[31360]: pgmap v10349: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:31 smithi022 ceph-mon[40250]: pgmap v10349: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:31 smithi122 ceph-mon[32255]: pgmap v10349: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:43:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:43:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[31360]: pgmap v10350: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:43:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:43:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:43:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:43:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[40250]: pgmap v10350: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:43:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:43:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:43:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:43:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:33 smithi122 ceph-mon[32255]: pgmap v10350: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:43:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:43:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:43:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:43:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:35 smithi022 ceph-mon[31360]: pgmap v10351: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:35 smithi022 ceph-mon[40250]: pgmap v10351: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:36.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:36.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:36.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:35 smithi122 ceph-mon[32255]: pgmap v10351: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2892551424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2892551424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2892551424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:43:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:37 smithi022 ceph-mon[31360]: pgmap v10352: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:37 smithi022 ceph-mon[40250]: pgmap v10352: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:37 smithi122 ceph-mon[32255]: pgmap v10352: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:43:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:43:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:43:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:43:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:43:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:43:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:43:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:43:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:43:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:39 smithi022 ceph-mon[31360]: pgmap v10353: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:43:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:39 smithi022 ceph-mon[40250]: pgmap v10353: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:40.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:43:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:40] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:43:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:39 smithi122 ceph-mon[32255]: pgmap v10353: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:43:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:43:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:43:41.076 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:41.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:41.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:41.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:41 smithi022 ceph-mon[31360]: pgmap v10354: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/524101638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:41 smithi022 ceph-mon[40250]: pgmap v10354: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/524101638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:41 smithi122 ceph-mon[32255]: pgmap v10354: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/524101638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:43:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:43:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:43 smithi022 ceph-mon[31360]: pgmap v10355: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:43 smithi022 ceph-mon[40250]: pgmap v10355: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:43 smithi122 ceph-mon[32255]: pgmap v10355: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:45 smithi022 ceph-mon[31360]: pgmap v10356: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:45 smithi022 ceph-mon[40250]: pgmap v10356: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:45 smithi122 ceph-mon[32255]: pgmap v10356: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:46.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:46.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:46.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:46.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/268096937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/268096937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/268096937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:43:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:47 smithi022 ceph-mon[31360]: pgmap v10357: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:47 smithi022 ceph-mon[40250]: pgmap v10357: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:47 smithi122 ceph-mon[32255]: pgmap v10357: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:49 smithi022 ceph-mon[31360]: pgmap v10358: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:49 smithi022 ceph-mon[40250]: pgmap v10358: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:49 smithi122 ceph-mon[32255]: pgmap v10358: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:50] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:43:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:43:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:43:51.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:52.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:52.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:52.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:52.188 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:51 smithi022 ceph-mon[40250]: pgmap v10359: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:52.188 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:52.188 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:51 smithi022 ceph-mon[31360]: pgmap v10359: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:52.189 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:51 smithi122 ceph-mon[32255]: pgmap v10359: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:43:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:43:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:43:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/372190452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/372190452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/372190452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:53 smithi022 ceph-mon[31360]: pgmap v10360: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:53 smithi022 ceph-mon[40250]: pgmap v10360: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:53 smithi122 ceph-mon[32255]: pgmap v10360: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:55 smithi022 ceph-mon[31360]: pgmap v10361: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:55 smithi022 ceph-mon[40250]: pgmap v10361: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:55 smithi122 ceph-mon[32255]: pgmap v10361: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:43:57.189 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:43:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:43:57.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:43:57.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:43:57.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:43:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:57 smithi022 ceph-mon[31360]: pgmap v10362: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1133434683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:57 smithi022 ceph-mon[40250]: pgmap v10362: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1133434683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:43:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:57 smithi122 ceph-mon[32255]: pgmap v10362: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:43:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1133434683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:43:59 smithi022 ceph-mon[31360]: pgmap v10363: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:43:59 smithi022 ceph-mon[40250]: pgmap v10363: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:00] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:44:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:43:59 smithi122 ceph-mon[32255]: pgmap v10363: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:00.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:44:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:44:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:01 smithi022 ceph-mon[31360]: pgmap v10364: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:01 smithi022 ceph-mon[40250]: pgmap v10364: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:01 smithi122 ceph-mon[32255]: pgmap v10364: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:02.558 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:44:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:44:02.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:02.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:02.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/64885436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/64885436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/64885436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:04 smithi022 ceph-mon[31360]: pgmap v10365: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:04 smithi022 ceph-mon[40250]: pgmap v10365: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:04 smithi122 ceph-mon[32255]: pgmap v10365: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:06 smithi022 ceph-mon[31360]: pgmap v10366: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:06 smithi022 ceph-mon[40250]: pgmap v10366: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:06 smithi122 ceph-mon[32255]: pgmap v10366: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:44:07.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:08 smithi022 ceph-mon[31360]: pgmap v10367: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:08 smithi022 ceph-mon[40250]: pgmap v10367: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:08.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:08.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:08.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:08 smithi122 ceph-mon[32255]: pgmap v10367: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2733817173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2733817173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2733817173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:10 smithi022 ceph-mon[31360]: pgmap v10368: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:10 smithi022 ceph-mon[40250]: pgmap v10368: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:10 smithi122 ceph-mon[32255]: pgmap v10368: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:10] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:44:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:44:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:44:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:12 smithi022 ceph-mon[31360]: pgmap v10369: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:12 smithi022 ceph-mon[40250]: pgmap v10369: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:12 smithi122 ceph-mon[32255]: pgmap v10369: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:44:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:44:13.297 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:13.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:13.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:13.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:14 smithi022 ceph-mon[31360]: pgmap v10370: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/570634452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:14 smithi022 ceph-mon[40250]: pgmap v10370: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/570634452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:14 smithi122 ceph-mon[32255]: pgmap v10370: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/570634452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:16 smithi022 ceph-mon[31360]: pgmap v10371: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:16 smithi022 ceph-mon[40250]: pgmap v10371: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:16 smithi122 ceph-mon[32255]: pgmap v10371: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:44:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:18 smithi022 ceph-mon[31360]: pgmap v10372: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:18 smithi022 ceph-mon[40250]: pgmap v10372: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:18 smithi122 ceph-mon[32255]: pgmap v10372: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:18.665 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:19.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2164470997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:19.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:19.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:19.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2164470997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2164470997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:20 smithi022 ceph-mon[31360]: pgmap v10373: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:20 smithi022 ceph-mon[40250]: pgmap v10373: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:20 smithi122 ceph-mon[32255]: pgmap v10373: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:44:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:44:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:44:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:22 smithi122 ceph-mon[32255]: pgmap v10374: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:22 smithi022 ceph-mon[31360]: pgmap v10374: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:22 smithi022 ceph-mon[40250]: pgmap v10374: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:44:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:44:24.036 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:24.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:24.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:24.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:24.402 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:24 smithi022 ceph-mon[31360]: pgmap v10375: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:24.403 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:24 smithi022 ceph-mon[40250]: pgmap v10375: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:24 smithi122 ceph-mon[32255]: pgmap v10375: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/53302058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/53302058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/53302058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:26 smithi122 ceph-mon[32255]: pgmap v10376: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:26 smithi022 ceph-mon[31360]: pgmap v10376: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:26 smithi022 ceph-mon[40250]: pgmap v10376: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:44:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:28 smithi122 ceph-mon[32255]: pgmap v10377: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:28 smithi022 ceph-mon[31360]: pgmap v10377: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:28 smithi022 ceph-mon[40250]: pgmap v10377: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:29.403 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:29.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:29.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:29.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:30 smithi122 ceph-mon[32255]: pgmap v10378: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3678519705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:44:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:30 smithi022 ceph-mon[31360]: pgmap v10378: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3678519705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:30 smithi022 ceph-mon[40250]: pgmap v10378: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3678519705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:44:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:44:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:32 smithi122 ceph-mon[32255]: pgmap v10379: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:32 smithi022 ceph-mon[31360]: pgmap v10379: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:32 smithi022 ceph-mon[40250]: pgmap v10379: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:44:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:44:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:34 smithi122 ceph-mon[32255]: pgmap v10380: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:44:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:44:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:44:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:44:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[31360]: pgmap v10380: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:44:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:44:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:44:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:44:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[40250]: pgmap v10380: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:44:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:44:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:44:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:44:34.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:35.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:35.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:35.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:36 smithi122 ceph-mon[32255]: pgmap v10381: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3751853816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:36 smithi022 ceph-mon[31360]: pgmap v10381: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3751853816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:36 smithi022 ceph-mon[40250]: pgmap v10381: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3751853816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:44:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:38 smithi122 ceph-mon[32255]: pgmap v10382: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:38 smithi022 ceph-mon[31360]: pgmap v10382: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:38 smithi022 ceph-mon[40250]: pgmap v10382: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:40.145 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:40] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:44:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:40 smithi122 ceph-mon[32255]: pgmap v10383: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:44:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:44:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:44:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:40.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[31360]: pgmap v10383: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:40.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:44:40.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:44:40.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:44:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[40250]: pgmap v10383: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:44:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:44:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:44:40.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:44:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:44:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/925401093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/925401093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/925401093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:42 smithi122 ceph-mon[32255]: pgmap v10384: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:42 smithi022 ceph-mon[31360]: pgmap v10384: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:42 smithi022 ceph-mon[40250]: pgmap v10384: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:44:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:44:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: pgmap v10385: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: pgmap v10385: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: pgmap v10385: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:44.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:44:44.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:44.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:45 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3119M 2021-12-15T23:44:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:45 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3119M 2021-12-15T23:44:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:45 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3119M 2021-12-15T23:44:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:44:45.529 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:45.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:45.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:45.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:46 smithi022 ceph-mon[40250]: pgmap v10386: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/69752733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:46 smithi022 ceph-mon[31360]: pgmap v10386: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/69752733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:46 smithi122 ceph-mon[32255]: pgmap v10386: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/69752733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:44:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:48 smithi022 ceph-mon[31360]: pgmap v10387: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:48 smithi022 ceph-mon[40250]: pgmap v10387: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:48 smithi122 ceph-mon[32255]: pgmap v10387: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:50 smithi022 ceph-mon[31360]: pgmap v10388: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:50 smithi022 ceph-mon[40250]: pgmap v10388: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:50] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:44:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:50 smithi122 ceph-mon[32255]: pgmap v10388: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:50.897 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:44:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:44:51.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:51.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:51.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:52 smithi022 ceph-mon[31360]: pgmap v10389: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:52.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1319061394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:52 smithi022 ceph-mon[40250]: pgmap v10389: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1319061394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:52 smithi122 ceph-mon[32255]: pgmap v10389: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:44:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1319061394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:44:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:44:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:54 smithi022 ceph-mon[31360]: pgmap v10390: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:54 smithi022 ceph-mon[40250]: pgmap v10390: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:54 smithi122 ceph-mon[32255]: pgmap v10390: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:56.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:44:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:56 smithi022 ceph-mon[31360]: pgmap v10391: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:56 smithi022 ceph-mon[40250]: pgmap v10391: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:56 smithi122 ceph-mon[32255]: pgmap v10391: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:44:56.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:44:56.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:44:56.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:44:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3302635972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3302635972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3302635972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:44:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:44:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:44:58 smithi022 ceph-mon[31360]: pgmap v10392: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:44:58 smithi022 ceph-mon[40250]: pgmap v10392: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:44:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:44:58 smithi122 ceph-mon[32255]: pgmap v10392: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:00 smithi122 ceph-mon[32255]: pgmap v10393: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:00] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:45:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:00 smithi022 ceph-mon[31360]: pgmap v10393: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:00 smithi022 ceph-mon[40250]: pgmap v10393: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:00.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:45:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:45:01.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:02.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:02.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:02.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:02 smithi022 ceph-mon[31360]: pgmap v10394: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/315384802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:02 smithi022 ceph-mon[40250]: pgmap v10394: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/315384802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:02 smithi122 ceph-mon[32255]: pgmap v10394: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/315384802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:45:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:45:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:04 smithi122 ceph-mon[32255]: pgmap v10395: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:04 smithi022 ceph-mon[31360]: pgmap v10395: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:04 smithi022 ceph-mon[40250]: pgmap v10395: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:06 smithi122 ceph-mon[32255]: pgmap v10396: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:06 smithi022 ceph-mon[31360]: pgmap v10396: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:06 smithi022 ceph-mon[40250]: pgmap v10396: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:07.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:07.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:07.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:07.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:45:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:08 smithi122 ceph-mon[32255]: pgmap v10397: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4100804701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:08 smithi022 ceph-mon[31360]: pgmap v10397: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4100804701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:08 smithi022 ceph-mon[40250]: pgmap v10397: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4100804701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:10] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:45:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:10 smithi122 ceph-mon[32255]: pgmap v10398: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:10 smithi022 ceph-mon[31360]: pgmap v10398: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:10 smithi022 ceph-mon[40250]: pgmap v10398: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:45:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:45:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:12 smithi122 ceph-mon[32255]: pgmap v10399: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:12.382 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:12 smithi022 ceph-mon[31360]: pgmap v10399: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:12.382 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:12.383 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:12 smithi022 ceph-mon[40250]: pgmap v10399: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:12.383 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:12.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:45:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:45:12.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:12.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:12.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/91584234' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/91584234' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/91584234' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:14 smithi122 ceph-mon[32255]: pgmap v10400: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:14 smithi022 ceph-mon[31360]: pgmap v10400: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:14 smithi022 ceph-mon[40250]: pgmap v10400: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-15T23:45:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:16 smithi122 ceph-mon[32255]: pgmap v10401: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:16 smithi022 ceph-mon[31360]: pgmap v10401: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:16 smithi022 ceph-mon[40250]: pgmap v10401: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-15T23:45:17.755 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:45:17.756 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:18.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:18.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:18.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:18 smithi122 ceph-mon[32255]: pgmap v10402: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:18 smithi022 ceph-mon[31360]: pgmap v10402: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:18 smithi022 ceph-mon[40250]: pgmap v10402: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3167148462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3167148462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3167148462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:20 smithi122 ceph-mon[32255]: pgmap v10403: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:20] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:45:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:20 smithi022 ceph-mon[31360]: pgmap v10403: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:20 smithi022 ceph-mon[40250]: pgmap v10403: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:45:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:45:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:45:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:45:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:22 smithi122 ceph-mon[32255]: pgmap v10404: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:22 smithi022 ceph-mon[31360]: pgmap v10404: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:22 smithi022 ceph-mon[40250]: pgmap v10404: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:23.125 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:23.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:23.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:23.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:24 smithi122 ceph-mon[32255]: pgmap v10405: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1129679590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:24 smithi022 ceph-mon[31360]: pgmap v10405: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1129679590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:24 smithi022 ceph-mon[40250]: pgmap v10405: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1129679590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:26 smithi122 ceph-mon[32255]: pgmap v10406: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:26 smithi022 ceph-mon[31360]: pgmap v10406: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:26 smithi022 ceph-mon[40250]: pgmap v10406: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:45:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:28 smithi122 ceph-mon[32255]: pgmap v10407: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:28.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:28.500 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:28 smithi022 ceph-mon[40250]: pgmap v10407: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:28.500 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:28 smithi022 ceph-mon[31360]: pgmap v10407: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:28.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:28.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:28.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2970262547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2970262547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2970262547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:30 smithi122 ceph-mon[32255]: pgmap v10408: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:30] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:45:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:30 smithi022 ceph-mon[31360]: pgmap v10408: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:30 smithi022 ceph-mon[40250]: pgmap v10408: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:45:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:45:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:32 smithi122 ceph-mon[32255]: pgmap v10409: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:45:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:45:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:32 smithi022 ceph-mon[31360]: pgmap v10409: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:32 smithi022 ceph-mon[40250]: pgmap v10409: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:33.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:34.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:34.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:34.242 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[31360]: pgmap v10410: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:34.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:45:34.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:45:34.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:45:34.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:45:34.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[40250]: pgmap v10410: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:34.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:45:34.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:45:34.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:45:34.245 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:45:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:34 smithi122 ceph-mon[32255]: pgmap v10410: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:34.554 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:45:34.554 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:45:34.554 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:45:34.555 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:45:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1753032223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1753032223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1753032223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:36 smithi122 ceph-mon[32255]: pgmap v10411: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:36 smithi022 ceph-mon[31360]: pgmap v10411: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:36 smithi022 ceph-mon[40250]: pgmap v10411: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:45:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:38 smithi122 ceph-mon[32255]: pgmap v10412: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:38 smithi022 ceph-mon[31360]: pgmap v10412: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:38 smithi022 ceph-mon[40250]: pgmap v10412: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:39.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:39.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:39.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:39.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:40] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:45:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:40 smithi122 ceph-mon[32255]: pgmap v10413: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2651030050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:40 smithi022 ceph-mon[31360]: pgmap v10413: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2651030050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:40 smithi022 ceph-mon[40250]: pgmap v10413: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2651030050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:45:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:45:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:45:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:45:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:42 smithi122 ceph-mon[32255]: pgmap v10414: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:42 smithi022 ceph-mon[31360]: pgmap v10414: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:42 smithi022 ceph-mon[40250]: pgmap v10414: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:43 smithi122 ceph-mon[32255]: pgmap v10415: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:43 smithi022 ceph-mon[31360]: pgmap v10415: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:43 smithi022 ceph-mon[40250]: pgmap v10415: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:45:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:45:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:45:44.475 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:45:44.475 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:45:44.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:45:44.476 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:45:44.476 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:45:44.476 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:45:44.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:44.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:44.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:44.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:45 smithi022 ceph-mon[31360]: pgmap v10416: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:45:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/71974515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:45 smithi022 ceph-mon[40250]: pgmap v10416: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:45:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/71974515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:45 smithi122 ceph-mon[32255]: pgmap v10416: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:45:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/71974515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:47 smithi022 ceph-mon[31360]: pgmap v10417: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:45:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:47 smithi022 ceph-mon[40250]: pgmap v10417: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:47 smithi122 ceph-mon[32255]: pgmap v10417: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:49 smithi022 ceph-mon[31360]: pgmap v10418: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:49 smithi022 ceph-mon[40250]: pgmap v10418: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:49 smithi122 ceph-mon[32255]: pgmap v10418: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:49.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:50.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:50.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:50.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:50] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:45:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:45:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:45:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1537563048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1537563048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1537563048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:51 smithi022 ceph-mon[31360]: pgmap v10419: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:51 smithi022 ceph-mon[40250]: pgmap v10419: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:51 smithi122 ceph-mon[32255]: pgmap v10419: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:45:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:45:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:45:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:53 smithi022 ceph-mon[31360]: pgmap v10420: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:53 smithi022 ceph-mon[40250]: pgmap v10420: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:53 smithi122 ceph-mon[32255]: pgmap v10420: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:55.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:45:55.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:45:55.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:45:55.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:45:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:55 smithi022 ceph-mon[31360]: pgmap v10421: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2229785223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:55 smithi022 ceph-mon[40250]: pgmap v10421: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2229785223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:55 smithi122 ceph-mon[32255]: pgmap v10421: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:45:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2229785223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:45:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:45:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:57 smithi022 ceph-mon[31360]: pgmap v10422: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:57 smithi022 ceph-mon[40250]: pgmap v10422: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:45:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:57 smithi122 ceph-mon[32255]: pgmap v10422: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:45:59 smithi022 ceph-mon[31360]: pgmap v10423: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:45:59 smithi022 ceph-mon[40250]: pgmap v10423: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:45:59 smithi122 ceph-mon[32255]: pgmap v10423: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:00] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:46:00.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:00.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:46:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:46:01.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:01.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:01.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:01 smithi022 ceph-mon[31360]: pgmap v10424: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2603243183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:01 smithi022 ceph-mon[40250]: pgmap v10424: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2603243183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:01 smithi122 ceph-mon[32255]: pgmap v10424: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2603243183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:46:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:46:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:03 smithi022 ceph-mon[31360]: pgmap v10425: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:03 smithi022 ceph-mon[40250]: pgmap v10425: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:03 smithi122 ceph-mon[32255]: pgmap v10425: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:06.084 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:05 smithi022 ceph-mon[31360]: pgmap v10426: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:05 smithi022 ceph-mon[40250]: pgmap v10426: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:05 smithi122 ceph-mon[32255]: pgmap v10426: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:06.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:06.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:06.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/509287395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/509287395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/509287395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:46:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:07 smithi022 ceph-mon[31360]: pgmap v10427: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:07 smithi022 ceph-mon[40250]: pgmap v10427: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:07 smithi122 ceph-mon[32255]: pgmap v10427: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:09 smithi022 ceph-mon[31360]: pgmap v10428: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:09 smithi022 ceph-mon[40250]: pgmap v10428: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:10] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:46:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:09 smithi122 ceph-mon[32255]: pgmap v10428: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:10.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:46:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:46:11.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:11.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:11.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:11.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:11 smithi022 ceph-mon[40250]: pgmap v10429: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2469148931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:11 smithi022 ceph-mon[31360]: pgmap v10429: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:12.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:12.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2469148931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:11 smithi122 ceph-mon[32255]: pgmap v10429: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2469148931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:46:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:46:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:14 smithi022 ceph-mon[31360]: pgmap v10430: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:14 smithi022 ceph-mon[40250]: pgmap v10430: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:14 smithi122 ceph-mon[32255]: pgmap v10430: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:16 smithi022 ceph-mon[31360]: pgmap v10431: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:16 smithi022 ceph-mon[40250]: pgmap v10431: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:16 smithi122 ceph-mon[32255]: pgmap v10431: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:16.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:17.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:17.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:17.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:46:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:18 smithi022 ceph-mon[31360]: pgmap v10432: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1119935691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:18 smithi022 ceph-mon[40250]: pgmap v10432: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1119935691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:18 smithi122 ceph-mon[32255]: pgmap v10432: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1119935691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:20 smithi022 ceph-mon[31360]: pgmap v10433: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:20 smithi022 ceph-mon[40250]: pgmap v10433: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:46:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:20 smithi122 ceph-mon[32255]: pgmap v10433: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:46:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:46:22.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:22 smithi022 ceph-mon[31360]: pgmap v10434: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:22 smithi022 ceph-mon[40250]: pgmap v10434: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:22 smithi122 ceph-mon[32255]: pgmap v10434: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:22.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:22.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:22.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:46:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:46:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1102761848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1102761848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1102761848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:24 smithi022 ceph-mon[31360]: pgmap v10435: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:24 smithi022 ceph-mon[40250]: pgmap v10435: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:24 smithi122 ceph-mon[32255]: pgmap v10435: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:26 smithi022 ceph-mon[31360]: pgmap v10436: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:26 smithi022 ceph-mon[40250]: pgmap v10436: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:26 smithi122 ceph-mon[32255]: pgmap v10436: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:27.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:46:27.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:27.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:27.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:28 smithi022 ceph-mon[31360]: pgmap v10437: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/431926820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:28 smithi022 ceph-mon[40250]: pgmap v10437: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/431926820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:28 smithi122 ceph-mon[32255]: pgmap v10437: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/431926820' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:30 smithi022 ceph-mon[31360]: pgmap v10438: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:30 smithi022 ceph-mon[40250]: pgmap v10438: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:46:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:30 smithi122 ceph-mon[32255]: pgmap v10438: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:46:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:46:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:32 smithi022 ceph-mon[31360]: pgmap v10439: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:32 smithi022 ceph-mon[40250]: pgmap v10439: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:32 smithi122 ceph-mon[32255]: pgmap v10439: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:46:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:46:32.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:33.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:33.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:33.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:34 smithi122 ceph-mon[32255]: pgmap v10440: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2226723255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:46:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:46:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:46:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:46:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[31360]: pgmap v10440: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2226723255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:46:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:46:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:46:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:46:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[40250]: pgmap v10440: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2226723255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:46:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:46:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:46:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:46:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:36 smithi122 ceph-mon[32255]: pgmap v10441: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:36 smithi022 ceph-mon[31360]: pgmap v10441: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:36 smithi022 ceph-mon[40250]: pgmap v10441: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:46:38.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:38.312 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:38 smithi022 ceph-mon[31360]: pgmap v10442: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:38.313 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:38 smithi022 ceph-mon[40250]: pgmap v10442: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:38 smithi122 ceph-mon[32255]: pgmap v10442: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:38.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:38.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:38.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2489819512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2489819512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2489819512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:40 smithi122 ceph-mon[32255]: pgmap v10443: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:40] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:46:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:40 smithi022 ceph-mon[31360]: pgmap v10443: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:40 smithi022 ceph-mon[40250]: pgmap v10443: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:40.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:46:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:46:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:42 smithi122 ceph-mon[32255]: pgmap v10444: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:42 smithi022 ceph-mon[31360]: pgmap v10444: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:42 smithi022 ceph-mon[40250]: pgmap v10444: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:46:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:46:43.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:44 smithi122 ceph-mon[32255]: pgmap v10445: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/437089536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:44.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:44 smithi022 ceph-mon[31360]: pgmap v10445: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:44.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/437089536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:44.519 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:44 smithi022 ceph-mon[40250]: pgmap v10445: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:44.519 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/437089536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:46:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:46:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:46:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:46:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:46:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:46:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:46:45.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:46:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:46:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:46:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:46:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:46:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:46 smithi122 ceph-mon[32255]: pgmap v10446: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:46 smithi022 ceph-mon[31360]: pgmap v10446: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:46 smithi022 ceph-mon[40250]: pgmap v10446: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:46:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:48 smithi122 ceph-mon[32255]: pgmap v10447: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:48 smithi022 ceph-mon[31360]: pgmap v10447: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:48 smithi022 ceph-mon[40250]: pgmap v10447: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:49.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:49.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:49.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:49.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:50] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:46:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:50 smithi122 ceph-mon[32255]: pgmap v10448: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3850931250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:50 smithi022 ceph-mon[31360]: pgmap v10448: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3850931250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:50 smithi022 ceph-mon[40250]: pgmap v10448: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3850931250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:50.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:46:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:46:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:52 smithi122 ceph-mon[32255]: pgmap v10449: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:52 smithi022 ceph-mon[31360]: pgmap v10449: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:52 smithi022 ceph-mon[40250]: pgmap v10449: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:46:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:46:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:46:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:46:54.422 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:54 smithi022 ceph-mon[31360]: pgmap v10450: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:54.422 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:54 smithi022 ceph-mon[40250]: pgmap v10450: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:54 smithi122 ceph-mon[32255]: pgmap v10450: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:46:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:46:54.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:46:55.066 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2895170704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2895170704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2895170704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:46:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:56 smithi122 ceph-mon[32255]: pgmap v10451: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:56 smithi022 ceph-mon[31360]: pgmap v10451: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:56 smithi022 ceph-mon[40250]: pgmap v10451: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:46:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:46:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:46:58 smithi122 ceph-mon[32255]: pgmap v10452: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:46:58 smithi022 ceph-mon[31360]: pgmap v10452: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:46:58 smithi022 ceph-mon[40250]: pgmap v10452: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:46:59.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:00.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:00.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:00.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:00] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:47:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:00 smithi122 ceph-mon[32255]: pgmap v10453: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:00 smithi022 ceph-mon[31360]: pgmap v10453: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:00 smithi022 ceph-mon[40250]: pgmap v10453: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:00.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:47:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:47:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/379624034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/379624034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/379624034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:02 smithi122 ceph-mon[32255]: pgmap v10454: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:47:02.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:47:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:02 smithi022 ceph-mon[31360]: pgmap v10454: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:02 smithi022 ceph-mon[40250]: pgmap v10454: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:04 smithi122 ceph-mon[32255]: pgmap v10455: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:04 smithi022 ceph-mon[31360]: pgmap v10455: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:04 smithi022 ceph-mon[40250]: pgmap v10455: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:05.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:05.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:05.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:05.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:06 smithi122 ceph-mon[32255]: pgmap v10456: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/100728832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:06 smithi022 ceph-mon[31360]: pgmap v10456: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/100728832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:06 smithi022 ceph-mon[40250]: pgmap v10456: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/100728832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:47:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:08 smithi122 ceph-mon[32255]: pgmap v10457: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:08 smithi022 ceph-mon[31360]: pgmap v10457: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:08 smithi022 ceph-mon[40250]: pgmap v10457: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:10] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:47:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:10 smithi122 ceph-mon[32255]: pgmap v10458: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:10 smithi022 ceph-mon[31360]: pgmap v10458: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:10 smithi022 ceph-mon[40250]: pgmap v10458: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:10.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:10.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:10.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:10.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:10.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:47:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:47:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1603446698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1603446698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1603446698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:12 smithi122 ceph-mon[32255]: pgmap v10459: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:47:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:47:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:12 smithi022 ceph-mon[31360]: pgmap v10459: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:12 smithi022 ceph-mon[40250]: pgmap v10459: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:14 smithi122 ceph-mon[32255]: pgmap v10460: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:14 smithi022 ceph-mon[31360]: pgmap v10460: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:14 smithi022 ceph-mon[40250]: pgmap v10460: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:15.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:16.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:16.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:16.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:16.272 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:16 smithi022 ceph-mon[31360]: pgmap v10461: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:16.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:16 smithi022 ceph-mon[40250]: pgmap v10461: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:16 smithi122 ceph-mon[32255]: pgmap v10461: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/193692615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/193692615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/193692615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:47:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:18 smithi122 ceph-mon[32255]: pgmap v10462: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:18 smithi022 ceph-mon[31360]: pgmap v10462: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:18 smithi022 ceph-mon[40250]: pgmap v10462: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:20 smithi122 ceph-mon[32255]: pgmap v10463: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:20] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:47:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:20 smithi022 ceph-mon[31360]: pgmap v10463: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:20 smithi022 ceph-mon[40250]: pgmap v10463: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:20.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:47:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:47:21.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:21.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:21.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:21.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:47:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:47:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:22 smithi122 ceph-mon[32255]: pgmap v10464: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1174817345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:22 smithi022 ceph-mon[31360]: pgmap v10464: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1174817345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:22 smithi022 ceph-mon[40250]: pgmap v10464: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1174817345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:23.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:23 smithi122 ceph-mon[32255]: pgmap v10465: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:23 smithi022 ceph-mon[31360]: pgmap v10465: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:23 smithi022 ceph-mon[40250]: pgmap v10465: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:25 smithi022 ceph-mon[31360]: pgmap v10466: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:25 smithi022 ceph-mon[40250]: pgmap v10466: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:25 smithi122 ceph-mon[32255]: pgmap v10466: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:26.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:27.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:27.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:27.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:47:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:27 smithi022 ceph-mon[31360]: pgmap v10467: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2910470353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:27 smithi022 ceph-mon[40250]: pgmap v10467: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2910470353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:27 smithi122 ceph-mon[32255]: pgmap v10467: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2910470353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:29 smithi022 ceph-mon[31360]: pgmap v10468: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:29 smithi022 ceph-mon[40250]: pgmap v10468: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:30] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:47:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:29 smithi122 ceph-mon[32255]: pgmap v10468: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:30.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:47:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:47:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:31 smithi022 ceph-mon[31360]: pgmap v10469: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:31 smithi022 ceph-mon[40250]: pgmap v10469: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:31 smithi122 ceph-mon[32255]: pgmap v10469: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:32.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:32.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:32.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:47:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:47:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1510879097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1510879097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1510879097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[31360]: pgmap v10470: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:47:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:47:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:47:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:47:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[40250]: pgmap v10470: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:47:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:47:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:47:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:47:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:33 smithi122 ceph-mon[32255]: pgmap v10470: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:47:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:47:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:47:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:47:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:35 smithi022 ceph-mon[31360]: pgmap v10471: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:35 smithi022 ceph-mon[40250]: pgmap v10471: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:35 smithi122 ceph-mon[32255]: pgmap v10471: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:37.385 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:37.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:37.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:37.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:37.755 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:47:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:37 smithi022 ceph-mon[31360]: pgmap v10472: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1133194777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:37 smithi022 ceph-mon[40250]: pgmap v10472: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1133194777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:37 smithi122 ceph-mon[32255]: pgmap v10472: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1133194777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:39 smithi022 ceph-mon[31360]: pgmap v10473: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:39 smithi022 ceph-mon[40250]: pgmap v10473: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:39 smithi122 ceph-mon[32255]: pgmap v10473: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:40] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:47:41.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:47:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:47:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:42 smithi022 ceph-mon[31360]: pgmap v10474: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:42 smithi022 ceph-mon[40250]: pgmap v10474: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:42 smithi122 ceph-mon[32255]: pgmap v10474: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:47:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:47:42.756 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:43.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:43.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:43.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:44 smithi022 ceph-mon[31360]: pgmap v10475: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1701777351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:44 smithi022 ceph-mon[40250]: pgmap v10475: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1701777351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:44 smithi122 ceph-mon[32255]: pgmap v10475: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1701777351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:47:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:47:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:47:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:47:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:47:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:47:45.390 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:47:45.390 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:47:45.391 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:47:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:46 smithi022 ceph-mon[31360]: pgmap v10476: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:47:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:46 smithi022 ceph-mon[40250]: pgmap v10476: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:47:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:46 smithi122 ceph-mon[32255]: pgmap v10476: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:47:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:47:48.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:48 smithi022 ceph-mon[31360]: pgmap v10477: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:48 smithi022 ceph-mon[40250]: pgmap v10477: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:48 smithi122 ceph-mon[32255]: pgmap v10477: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:48.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:48.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:48.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1754636088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1754636088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1754636088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:50 smithi022 ceph-mon[31360]: pgmap v10478: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:50 smithi022 ceph-mon[40250]: pgmap v10478: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:50] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:47:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:50 smithi122 ceph-mon[32255]: pgmap v10478: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:47:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:47:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:52 smithi022 ceph-mon[31360]: pgmap v10479: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:52 smithi022 ceph-mon[40250]: pgmap v10479: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:52 smithi122 ceph-mon[32255]: pgmap v10479: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:47:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:47:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:47:53.503 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:53.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:53.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:53.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:47:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:54 smithi022 ceph-mon[31360]: pgmap v10480: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4020825824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:54 smithi022 ceph-mon[40250]: pgmap v10480: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4020825824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:54 smithi122 ceph-mon[32255]: pgmap v10480: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4020825824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:47:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:56 smithi022 ceph-mon[31360]: pgmap v10481: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:56 smithi022 ceph-mon[40250]: pgmap v10481: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:56 smithi122 ceph-mon[32255]: pgmap v10481: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:47:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:47:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:47:58 smithi122 ceph-mon[32255]: pgmap v10482: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:47:58 smithi022 ceph-mon[31360]: pgmap v10482: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:47:58 smithi022 ceph-mon[40250]: pgmap v10482: 132 pgs: 132 active+clean; 582 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:47:58.868 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:47:59.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:47:59.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:47:59.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:00] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:48:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:00 smithi122 ceph-mon[32255]: pgmap v10483: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1427673326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:00 smithi022 ceph-mon[31360]: pgmap v10483: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1427673326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:00 smithi022 ceph-mon[40250]: pgmap v10483: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1427673326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:48:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:48:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:02 smithi122 ceph-mon[32255]: pgmap v10484: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:02 smithi022 ceph-mon[31360]: pgmap v10484: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:02 smithi022 ceph-mon[40250]: pgmap v10484: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:48:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:48:04.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:04 smithi122 ceph-mon[32255]: pgmap v10485: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:04 smithi022 ceph-mon[31360]: pgmap v10485: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:04 smithi022 ceph-mon[40250]: pgmap v10485: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:04.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:04.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:04.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/546275064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/546275064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/546275064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:06 smithi122 ceph-mon[32255]: pgmap v10486: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:06 smithi022 ceph-mon[31360]: pgmap v10486: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:06 smithi022 ceph-mon[40250]: pgmap v10486: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:48:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:08 smithi122 ceph-mon[32255]: pgmap v10487: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:08 smithi022 ceph-mon[31360]: pgmap v10487: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:08 smithi022 ceph-mon[40250]: pgmap v10487: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:48:09.599 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:09.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:09.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:09.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:10 smithi022 ceph-mon[31360]: pgmap v10488: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:48:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/607994215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:10 smithi022 ceph-mon[40250]: pgmap v10488: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:48:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/607994215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:10] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:48:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:10 smithi122 ceph-mon[32255]: pgmap v10488: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:48:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/607994215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:48:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:48:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:12 smithi122 ceph-mon[32255]: pgmap v10489: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:12 smithi022 ceph-mon[31360]: pgmap v10489: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:12 smithi022 ceph-mon[40250]: pgmap v10489: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:48:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:48:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:14 smithi122 ceph-mon[32255]: pgmap v10490: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:14 smithi022 ceph-mon[31360]: pgmap v10490: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:14 smithi022 ceph-mon[40250]: pgmap v10490: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:14.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:15.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:15.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:15.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:16 smithi122 ceph-mon[32255]: pgmap v10491: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:48:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3342797883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:16 smithi022 ceph-mon[31360]: pgmap v10491: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:48:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3342797883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:16 smithi022 ceph-mon[40250]: pgmap v10491: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:48:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3342797883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:48:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:18 smithi122 ceph-mon[32255]: pgmap v10492: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:18 smithi022 ceph-mon[31360]: pgmap v10492: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:18 smithi022 ceph-mon[40250]: pgmap v10492: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:48:20.338 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:20.339 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:20 smithi022 ceph-mon[31360]: pgmap v10493: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:48:20.339 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:20 smithi022 ceph-mon[40250]: pgmap v10493: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:48:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:20 smithi122 ceph-mon[32255]: pgmap v10493: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:48:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:20] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:48:20.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:20.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:20.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:21.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:48:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:48:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/100908169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/100908169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/100908169' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:48:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:48:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:22 smithi122 ceph-mon[32255]: pgmap v10494: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:22 smithi022 ceph-mon[31360]: pgmap v10494: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:22 smithi022 ceph-mon[40250]: pgmap v10494: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:24 smithi122 ceph-mon[32255]: pgmap v10495: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:24 smithi022 ceph-mon[31360]: pgmap v10495: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:24 smithi022 ceph-mon[40250]: pgmap v10495: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:25.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:26.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:26.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:26.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:26 smithi122 ceph-mon[32255]: pgmap v10496: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1333633451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:26 smithi022 ceph-mon[31360]: pgmap v10496: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1333633451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:26 smithi022 ceph-mon[40250]: pgmap v10496: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1333633451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:48:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:28 smithi122 ceph-mon[32255]: pgmap v10497: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:28 smithi022 ceph-mon[31360]: pgmap v10497: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:28 smithi022 ceph-mon[40250]: pgmap v10497: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:30] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:48:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:30 smithi122 ceph-mon[32255]: pgmap v10498: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:30 smithi022 ceph-mon[31360]: pgmap v10498: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:30 smithi022 ceph-mon[40250]: pgmap v10498: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:31.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:31.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:48:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:48:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:32 smithi122 ceph-mon[32255]: pgmap v10499: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1041362656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:48:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:48:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:32 smithi022 ceph-mon[31360]: pgmap v10499: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1041362656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:32 smithi022 ceph-mon[40250]: pgmap v10499: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1041362656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:34 smithi122 ceph-mon[32255]: pgmap v10500: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:48:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:48:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:48:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:48:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[31360]: pgmap v10500: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:48:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:48:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:48:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:48:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[40250]: pgmap v10500: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:48:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:48:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:48:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:48:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:36 smithi122 ceph-mon[32255]: pgmap v10501: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:36.447 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:36.447 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:36 smithi022 ceph-mon[40250]: pgmap v10501: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:36.448 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:36 smithi022 ceph-mon[31360]: pgmap v10501: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:36.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:36.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:36.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:37.096 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1994055676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1994055676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1994055676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:48:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:38 smithi122 ceph-mon[32255]: pgmap v10502: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:38 smithi022 ceph-mon[40250]: pgmap v10502: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:38 smithi022 ceph-mon[31360]: pgmap v10502: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:40 smithi122 ceph-mon[32255]: pgmap v10503: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:40] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:48:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:40 smithi022 ceph-mon[40250]: pgmap v10503: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:40 smithi022 ceph-mon[31360]: pgmap v10503: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:48:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:48:41.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:42.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:42.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:42.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:42 smithi122 ceph-mon[32255]: pgmap v10504: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:48:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:48:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:42 smithi022 ceph-mon[31360]: pgmap v10504: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:42 smithi022 ceph-mon[40250]: pgmap v10504: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3952389536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3952389536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3952389536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:44 smithi122 ceph-mon[32255]: pgmap v10505: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:44 smithi022 ceph-mon[31360]: pgmap v10505: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:44 smithi022 ceph-mon[40250]: pgmap v10505: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:46 smithi122 ceph-mon[32255]: pgmap v10506: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:48:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:48:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:48:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:48:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[31360]: pgmap v10506: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:48:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:48:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:48:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:48:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[40250]: pgmap v10506: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:48:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:48:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:48:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:48:47.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:48:47.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:47.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:47.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:48 smithi122 ceph-mon[32255]: pgmap v10507: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1212941321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:48 smithi022 ceph-mon[31360]: pgmap v10507: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1212941321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:48 smithi022 ceph-mon[40250]: pgmap v10507: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1212941321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:50] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:48:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:50 smithi122 ceph-mon[32255]: pgmap v10508: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:50 smithi022 ceph-mon[31360]: pgmap v10508: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:50 smithi022 ceph-mon[40250]: pgmap v10508: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:48:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:48:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:48:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:48:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:52 smithi122 ceph-mon[32255]: pgmap v10509: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:52.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:52 smithi022 ceph-mon[31360]: pgmap v10509: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:52 smithi022 ceph-mon[40250]: pgmap v10509: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:48:52.556 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:52.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:52.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:52.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3357959830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3357959830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3357959830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:54 smithi122 ceph-mon[32255]: pgmap v10510: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:54 smithi022 ceph-mon[31360]: pgmap v10510: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:54 smithi022 ceph-mon[40250]: pgmap v10510: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:56 smithi122 ceph-mon[32255]: pgmap v10511: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:56 smithi022 ceph-mon[31360]: pgmap v10511: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:56 smithi022 ceph-mon[40250]: pgmap v10511: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:48:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:48:57.928 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:48:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:58 smithi022 ceph-mon[31360]: pgmap v10512: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:58 smithi022 ceph-mon[40250]: pgmap v10512: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:58.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:48:58.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:48:58.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:48:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:58 smithi122 ceph-mon[32255]: pgmap v10512: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:48:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:48:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/95207800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:48:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/95207800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:48:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:48:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/95207800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:00 smithi122 ceph-mon[32255]: pgmap v10513: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:00] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:49:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:00 smithi022 ceph-mon[31360]: pgmap v10513: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:00 smithi022 ceph-mon[40250]: pgmap v10513: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:01.142 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:49:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:49:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:01 smithi122 ceph-mon[32255]: pgmap v10514: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:01 smithi022 ceph-mon[31360]: pgmap v10514: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:01 smithi022 ceph-mon[40250]: pgmap v10514: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:49:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:49:03.296 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:03.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:03.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:03.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:03.667 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:03 smithi022 ceph-mon[31360]: pgmap v10515: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:03.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:03 smithi022 ceph-mon[40250]: pgmap v10515: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:03 smithi122 ceph-mon[32255]: pgmap v10515: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1759525153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1759525153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1759525153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:05 smithi022 ceph-mon[31360]: pgmap v10516: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:06 smithi022 ceph-mon[40250]: pgmap v10516: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:06 smithi122 ceph-mon[32255]: pgmap v10516: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:49:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:08 smithi022 ceph-mon[31360]: pgmap v10517: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:08 smithi022 ceph-mon[40250]: pgmap v10517: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:08 smithi122 ceph-mon[32255]: pgmap v10517: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:08.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:09.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:09.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:09.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:10 smithi022 ceph-mon[40250]: pgmap v10518: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3803133210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:10 smithi022 ceph-mon[31360]: pgmap v10518: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:10.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3803133210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:10] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:49:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:10 smithi122 ceph-mon[32255]: pgmap v10518: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3803133210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:49:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:49:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:12 smithi022 ceph-mon[31360]: pgmap v10519: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:12 smithi022 ceph-mon[40250]: pgmap v10519: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:12 smithi122 ceph-mon[32255]: pgmap v10519: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:49:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:49:14.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:14 smithi022 ceph-mon[31360]: pgmap v10520: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:14 smithi022 ceph-mon[40250]: pgmap v10520: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:14.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:14.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:14.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:14 smithi122 ceph-mon[32255]: pgmap v10520: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1433169665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1433169665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1433169665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:16 smithi022 ceph-mon[31360]: pgmap v10521: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:16 smithi022 ceph-mon[40250]: pgmap v10521: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:16 smithi122 ceph-mon[32255]: pgmap v10521: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:49:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:18 smithi022 ceph-mon[31360]: pgmap v10522: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:18 smithi022 ceph-mon[40250]: pgmap v10522: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:18 smithi122 ceph-mon[32255]: pgmap v10522: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:19.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:19.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:19.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:19.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:20 smithi022 ceph-mon[31360]: pgmap v10523: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2067818067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:20 smithi022 ceph-mon[40250]: pgmap v10523: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2067818067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:49:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:20 smithi122 ceph-mon[32255]: pgmap v10523: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2067818067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:21.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:49:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:49:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:22 smithi022 ceph-mon[31360]: pgmap v10524: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:22 smithi022 ceph-mon[40250]: pgmap v10524: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:22 smithi122 ceph-mon[32255]: pgmap v10524: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:49:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:49:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:24 smithi022 ceph-mon[31360]: pgmap v10525: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:24 smithi022 ceph-mon[40250]: pgmap v10525: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:24 smithi122 ceph-mon[32255]: pgmap v10525: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:24.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:25.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:25.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:25.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:26 smithi022 ceph-mon[31360]: pgmap v10526: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/163654966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:26 smithi022 ceph-mon[40250]: pgmap v10526: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/163654966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:26 smithi122 ceph-mon[32255]: pgmap v10526: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/163654966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:49:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:28 smithi122 ceph-mon[32255]: pgmap v10527: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:28 smithi022 ceph-mon[31360]: pgmap v10527: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:28 smithi022 ceph-mon[40250]: pgmap v10527: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:30.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:30 smithi122 ceph-mon[32255]: pgmap v10528: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:49:30.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:30.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:30.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:30.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:30 smithi022 ceph-mon[31360]: pgmap v10528: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:30.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:30 smithi022 ceph-mon[40250]: pgmap v10528: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:31.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:49:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:49:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4249324356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4249324356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4249324356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:32 smithi122 ceph-mon[32255]: pgmap v10529: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:32 smithi022 ceph-mon[31360]: pgmap v10529: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:32 smithi022 ceph-mon[40250]: pgmap v10529: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:49:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:49:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:34 smithi122 ceph-mon[32255]: pgmap v10530: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:49:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:49:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:49:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:49:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[31360]: pgmap v10530: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:49:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:49:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:49:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:49:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[40250]: pgmap v10530: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:49:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:49:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:49:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:49:35.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:35.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:35.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:35.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:36 smithi022 ceph-mon[31360]: pgmap v10531: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/531620561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:36 smithi022 ceph-mon[40250]: pgmap v10531: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/531620561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:36 smithi122 ceph-mon[32255]: pgmap v10531: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/531620561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:49:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:38 smithi122 ceph-mon[32255]: pgmap v10532: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:38 smithi022 ceph-mon[31360]: pgmap v10532: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:38 smithi022 ceph-mon[40250]: pgmap v10532: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:40] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:49:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:40 smithi122 ceph-mon[32255]: pgmap v10533: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:40 smithi022 ceph-mon[31360]: pgmap v10533: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:40 smithi022 ceph-mon[40250]: pgmap v10533: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:40.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:49:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:49:41.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:41.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:41.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:42 smithi122 ceph-mon[32255]: pgmap v10534: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1177860631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:42 smithi022 ceph-mon[31360]: pgmap v10534: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1177860631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:42 smithi022 ceph-mon[40250]: pgmap v10534: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1177860631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:49:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:49:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:44 smithi122 ceph-mon[32255]: pgmap v10535: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:44 smithi022 ceph-mon[31360]: pgmap v10535: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:44 smithi022 ceph-mon[40250]: pgmap v10535: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:46 smithi122 ceph-mon[32255]: pgmap v10536: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:49:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:49:46.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:49:46.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[31360]: pgmap v10536: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:46.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:49:46.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:49:46.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:49:46.262 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[40250]: pgmap v10536: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:46.262 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:49:46.263 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:49:46.263 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:49:46.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:46.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:46.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:46.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:49:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:47 smithi122 ceph-mon[32255]: pgmap v10537: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2075405491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:49:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:49:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:47 smithi022 ceph-mon[31360]: pgmap v10537: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2075405491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:49:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:47 smithi022 ceph-mon[40250]: pgmap v10537: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2075405491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:49 smithi022 ceph-mon[31360]: pgmap v10538: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:49 smithi022 ceph-mon[40250]: pgmap v10538: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:49 smithi122 ceph-mon[32255]: pgmap v10538: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:50] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:49:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:49:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:49:51.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:52 smithi022 ceph-mon[31360]: pgmap v10539: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3869540024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:52 smithi022 ceph-mon[40250]: pgmap v10539: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3869540024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:52 smithi122 ceph-mon[32255]: pgmap v10539: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:49:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3869540024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:49:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:49:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:54 smithi022 ceph-mon[31360]: pgmap v10540: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:54 smithi022 ceph-mon[40250]: pgmap v10540: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:54 smithi122 ceph-mon[32255]: pgmap v10540: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:56 smithi022 ceph-mon[31360]: pgmap v10541: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:56 smithi022 ceph-mon[40250]: pgmap v10541: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:56 smithi122 ceph-mon[32255]: pgmap v10541: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:49:56.998 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:49:57.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:49:57.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:49:57.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:49:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:49:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:58 smithi022 ceph-mon[31360]: pgmap v10542: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:49:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4100994998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:58 smithi022 ceph-mon[40250]: pgmap v10542: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:49:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4100994998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:49:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:58 smithi122 ceph-mon[32255]: pgmap v10542: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:49:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:49:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4100994998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:00 smithi022 ceph-mon[31360]: pgmap v10543: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-15T23:50:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:00 smithi022 ceph-mon[40250]: pgmap v10543: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-15T23:50:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:00] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:00 smithi122 ceph-mon[32255]: pgmap v10543: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-15T23:50:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:50:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:50:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:02 smithi122 ceph-mon[32255]: pgmap v10544: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:02 smithi022 ceph-mon[31360]: pgmap v10544: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:02 smithi022 ceph-mon[40250]: pgmap v10544: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:02.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:50:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:50:02.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:02.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:02.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3266443379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3266443379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3266443379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:04 smithi022 ceph-mon[31360]: pgmap v10545: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:04 smithi022 ceph-mon[40250]: pgmap v10545: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:04 smithi122 ceph-mon[32255]: pgmap v10545: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:06 smithi122 ceph-mon[32255]: pgmap v10546: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:06 smithi022 ceph-mon[31360]: pgmap v10546: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:06 smithi022 ceph-mon[40250]: pgmap v10546: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:07.746 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:07.746 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:50:08.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:08.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:08.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:08 smithi122 ceph-mon[32255]: pgmap v10547: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:08 smithi022 ceph-mon[31360]: pgmap v10547: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:08 smithi022 ceph-mon[40250]: pgmap v10547: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2972477666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2972477666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2972477666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:10 smithi122 ceph-mon[32255]: pgmap v10548: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:10] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:50:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:10 smithi022 ceph-mon[31360]: pgmap v10548: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:10 smithi022 ceph-mon[40250]: pgmap v10548: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:50:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:50:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:12 smithi122 ceph-mon[32255]: pgmap v10549: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:12 smithi022 ceph-mon[31360]: pgmap v10549: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:12 smithi022 ceph-mon[40250]: pgmap v10549: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:50:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:50:13.115 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:13.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:13.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:13.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:14 smithi122 ceph-mon[32255]: pgmap v10550: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3843714495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:14 smithi022 ceph-mon[31360]: pgmap v10550: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3843714495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:14 smithi022 ceph-mon[40250]: pgmap v10550: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3843714495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:16 smithi122 ceph-mon[32255]: pgmap v10551: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:16 smithi022 ceph-mon[31360]: pgmap v10551: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:16 smithi022 ceph-mon[40250]: pgmap v10551: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:50:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:18 smithi122 ceph-mon[32255]: pgmap v10552: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:18.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:18 smithi022 ceph-mon[31360]: pgmap v10552: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:18.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:18 smithi022 ceph-mon[40250]: pgmap v10552: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:18.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:18.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:18.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:18.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2814954818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2814954818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2814954818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:20 smithi122 ceph-mon[32255]: pgmap v10553: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:20] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:50:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:20 smithi022 ceph-mon[31360]: pgmap v10553: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:20 smithi022 ceph-mon[40250]: pgmap v10553: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:21.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:50:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:50:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:22 smithi122 ceph-mon[32255]: pgmap v10554: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:22 smithi022 ceph-mon[31360]: pgmap v10554: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:22 smithi022 ceph-mon[40250]: pgmap v10554: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:50:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:50:23.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:24.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:24.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:24.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:24.229 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:24 smithi022 ceph-mon[31360]: pgmap v10555: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:24.230 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:24 smithi022 ceph-mon[40250]: pgmap v10555: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:24 smithi122 ceph-mon[32255]: pgmap v10555: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1723547437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1723547437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1723547437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:26 smithi122 ceph-mon[32255]: pgmap v10556: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:26 smithi022 ceph-mon[31360]: pgmap v10556: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:26 smithi022 ceph-mon[40250]: pgmap v10556: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:50:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:28 smithi122 ceph-mon[32255]: pgmap v10557: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:28 smithi022 ceph-mon[31360]: pgmap v10557: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:28 smithi022 ceph-mon[40250]: pgmap v10557: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:29.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:29.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:29.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:29.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:30] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:50:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:30 smithi122 ceph-mon[32255]: pgmap v10558: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/820529401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:30 smithi022 ceph-mon[31360]: pgmap v10558: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/820529401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:30 smithi022 ceph-mon[40250]: pgmap v10558: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/820529401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:50:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:50:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:50:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:50:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:32 smithi122 ceph-mon[32255]: pgmap v10559: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:32 smithi022 ceph-mon[31360]: pgmap v10559: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:32 smithi022 ceph-mon[40250]: pgmap v10559: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:34 smithi122 ceph-mon[32255]: pgmap v10560: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:50:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:50:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:50:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:50:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[31360]: pgmap v10560: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:50:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:50:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:50:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:50:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[40250]: pgmap v10560: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:50:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:50:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:50:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:50:34.605 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:34.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:34.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:34.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2180406654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2180406654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2180406654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:36 smithi122 ceph-mon[32255]: pgmap v10561: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:36 smithi022 ceph-mon[31360]: pgmap v10561: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:36.572 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:36 smithi022 ceph-mon[40250]: pgmap v10561: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:50:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:38 smithi122 ceph-mon[32255]: pgmap v10562: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:38 smithi022 ceph-mon[31360]: pgmap v10562: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:38 smithi022 ceph-mon[40250]: pgmap v10562: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:39.973 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:40 smithi022 ceph-mon[31360]: pgmap v10563: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:40 smithi022 ceph-mon[40250]: pgmap v10563: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:40.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:40.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:40.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:40 smithi122 ceph-mon[32255]: pgmap v10563: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:40] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-15T23:50:41.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:50:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:50:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/686849465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/686849465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/686849465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:50:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:50:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:42 smithi122 ceph-mon[32255]: pgmap v10564: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:42 smithi022 ceph-mon[31360]: pgmap v10564: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:42 smithi022 ceph-mon[40250]: pgmap v10564: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:44 smithi122 ceph-mon[32255]: pgmap v10565: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:44 smithi022 ceph-mon[31360]: pgmap v10565: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:44 smithi022 ceph-mon[40250]: pgmap v10565: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:45.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:45.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:45.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:45.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:46.362 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:46 smithi122 ceph-mon[32255]: pgmap v10566: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:46.362 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3297451066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:46.372 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:46 smithi022 ceph-mon[31360]: pgmap v10566: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:46.372 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3297451066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:46.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:46 smithi022 ceph-mon[40250]: pgmap v10566: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:46.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3297451066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:50:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:50:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:50:47.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:47.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:50:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:50:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:50:47.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:47.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:50:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:50:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:50:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:50:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:48 smithi122 ceph-mon[32255]: pgmap v10567: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:48 smithi022 ceph-mon[31360]: pgmap v10567: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:48 smithi022 ceph-mon[40250]: pgmap v10567: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-15T23:50:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:50:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:50 smithi122 ceph-mon[32255]: pgmap v10568: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:50] "GET /metrics HTTP/1.1" 200 202404 "" "Prometheus/2.18.1" 2021-12-15T23:50:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:50 smithi022 ceph-mon[31360]: pgmap v10568: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:50 smithi022 ceph-mon[40250]: pgmap v10568: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:50.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:51.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:51.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:51.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:51.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:50:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:50:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/816205091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/816205091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/816205091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:52 smithi122 ceph-mon[32255]: pgmap v10569: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:50:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:50:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:52 smithi022 ceph-mon[31360]: pgmap v10569: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:52 smithi022 ceph-mon[40250]: pgmap v10569: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:50:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:54 smithi122 ceph-mon[32255]: pgmap v10570: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:54 smithi022 ceph-mon[31360]: pgmap v10570: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:54 smithi022 ceph-mon[40250]: pgmap v10570: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:56.083 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:50:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:56 smithi122 ceph-mon[32255]: pgmap v10571: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:56.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:50:56.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:50:56.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:50:56.453 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:56 smithi022 ceph-mon[31360]: pgmap v10571: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:56.454 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:56 smithi022 ceph-mon[40250]: pgmap v10571: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:50:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2295499962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2295499962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2295499962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:50:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:50:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:50:58 smithi122 ceph-mon[32255]: pgmap v10572: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:50:58 smithi022 ceph-mon[31360]: pgmap v10572: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:50:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:50:58 smithi022 ceph-mon[40250]: pgmap v10572: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:00] "GET /metrics HTTP/1.1" 200 202404 "" "Prometheus/2.18.1" 2021-12-15T23:51:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:00 smithi122 ceph-mon[32255]: pgmap v10573: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:00 smithi022 ceph-mon[31360]: pgmap v10573: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:00 smithi022 ceph-mon[40250]: pgmap v10573: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:01.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:51:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:51:01.455 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:01.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:01.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:01.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:02 smithi122 ceph-mon[32255]: pgmap v10574: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/853806517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:51:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:51:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:02 smithi022 ceph-mon[31360]: pgmap v10574: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/853806517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:02 smithi022 ceph-mon[40250]: pgmap v10574: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/853806517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:04 smithi122 ceph-mon[32255]: pgmap v10575: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:04 smithi022 ceph-mon[31360]: pgmap v10575: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:04 smithi022 ceph-mon[40250]: pgmap v10575: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:06 smithi122 ceph-mon[32255]: pgmap v10576: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:06 smithi022 ceph-mon[31360]: pgmap v10576: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:06 smithi022 ceph-mon[40250]: pgmap v10576: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:06.826 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:07.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:07.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:07.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:51:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:08 smithi122 ceph-mon[32255]: pgmap v10577: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/610774301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:08 smithi022 ceph-mon[31360]: pgmap v10577: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/610774301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:08 smithi022 ceph-mon[40250]: pgmap v10577: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/610774301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:10 smithi122 ceph-mon[32255]: pgmap v10578: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:10] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:51:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:10 smithi022 ceph-mon[31360]: pgmap v10578: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:10 smithi022 ceph-mon[40250]: pgmap v10578: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:51:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:51:12.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:12 smithi122 ceph-mon[32255]: pgmap v10579: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:51:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:51:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:12 smithi022 ceph-mon[31360]: pgmap v10579: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:12 smithi022 ceph-mon[40250]: pgmap v10579: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:12.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:12.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:12.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1708782060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1708782060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1708782060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:14 smithi122 ceph-mon[32255]: pgmap v10580: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:14 smithi022 ceph-mon[31360]: pgmap v10580: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:14 smithi022 ceph-mon[40250]: pgmap v10580: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:16 smithi122 ceph-mon[32255]: pgmap v10581: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:16 smithi022 ceph-mon[31360]: pgmap v10581: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:16 smithi022 ceph-mon[40250]: pgmap v10581: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:17.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:51:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:17.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:18 smithi022 ceph-mon[31360]: pgmap v10582: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1722594665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:18 smithi022 ceph-mon[40250]: pgmap v10582: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1722594665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:18 smithi122 ceph-mon[32255]: pgmap v10582: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1722594665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:20 smithi122 ceph-mon[32255]: pgmap v10583: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:20] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:51:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:20 smithi022 ceph-mon[31360]: pgmap v10583: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:20 smithi022 ceph-mon[40250]: pgmap v10583: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:21.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:51:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:51:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:22 smithi122 ceph-mon[32255]: pgmap v10584: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:22 smithi022 ceph-mon[31360]: pgmap v10584: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:22 smithi022 ceph-mon[40250]: pgmap v10584: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:51:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:51:22.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:23.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:23.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:23.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:24 smithi122 ceph-mon[32255]: pgmap v10585: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1888274409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:24 smithi022 ceph-mon[31360]: pgmap v10585: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1888274409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:24 smithi022 ceph-mon[40250]: pgmap v10585: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1888274409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:26 smithi122 ceph-mon[32255]: pgmap v10586: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:26 smithi022 ceph-mon[31360]: pgmap v10586: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:26 smithi022 ceph-mon[40250]: pgmap v10586: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:51:28.310 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:28.311 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:28 smithi022 ceph-mon[31360]: pgmap v10587: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:28.311 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:28 smithi022 ceph-mon[40250]: pgmap v10587: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:28 smithi122 ceph-mon[32255]: pgmap v10587: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:28.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:28.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:28.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1913980748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1913980748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1913980748' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:30 smithi122 ceph-mon[32255]: pgmap v10588: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:30] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:51:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:30 smithi022 ceph-mon[31360]: pgmap v10588: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:30 smithi022 ceph-mon[40250]: pgmap v10588: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:51:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:51:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:32 smithi122 ceph-mon[32255]: pgmap v10589: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:32 smithi022 ceph-mon[31360]: pgmap v10589: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:32 smithi022 ceph-mon[40250]: pgmap v10589: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:51:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:51:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:34.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:34.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:34.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:34 smithi122 ceph-mon[32255]: pgmap v10590: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:51:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:51:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:51:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:51:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/962226477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[31360]: pgmap v10590: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:51:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:51:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:51:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:51:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/962226477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[40250]: pgmap v10590: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:51:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:51:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:51:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:51:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/962226477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:36 smithi122 ceph-mon[32255]: pgmap v10591: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:36 smithi022 ceph-mon[31360]: pgmap v10591: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:36 smithi022 ceph-mon[40250]: pgmap v10591: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:51:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:38 smithi122 ceph-mon[32255]: pgmap v10592: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:38 smithi022 ceph-mon[31360]: pgmap v10592: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:38 smithi022 ceph-mon[40250]: pgmap v10592: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:39.050 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:39.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:39.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:39.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:40] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:51:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:40 smithi122 ceph-mon[32255]: pgmap v10593: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2355461535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:40 smithi022 ceph-mon[31360]: pgmap v10593: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2355461535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:40 smithi022 ceph-mon[40250]: pgmap v10593: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2355461535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:51:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:51:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:42 smithi122 ceph-mon[32255]: pgmap v10594: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:51:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:51:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:42 smithi022 ceph-mon[31360]: pgmap v10594: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:42 smithi022 ceph-mon[40250]: pgmap v10594: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:44.421 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:44.421 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:44 smithi022 ceph-mon[40250]: pgmap v10595: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:44.422 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:44 smithi022 ceph-mon[31360]: pgmap v10595: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:44 smithi122 ceph-mon[32255]: pgmap v10595: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:44.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:44.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:44.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/722498632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/722498632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/722498632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:46 smithi122 ceph-mon[32255]: pgmap v10596: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:46 smithi022 ceph-mon[31360]: pgmap v10596: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:46 smithi022 ceph-mon[40250]: pgmap v10596: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:51:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:48 smithi122 ceph-mon[32255]: pgmap v10597: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:51:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:51:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:51:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:51:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[31360]: pgmap v10597: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:51:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:51:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:51:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:51:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[40250]: pgmap v10597: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:51:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:51:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:51:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:51:49.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:50.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:50.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:50.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:50] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:51:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:50 smithi122 ceph-mon[32255]: pgmap v10598: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:50 smithi022 ceph-mon[31360]: pgmap v10598: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:50 smithi022 ceph-mon[40250]: pgmap v10598: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2102209964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:51.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:51:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:51:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2102209964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2102209964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:52 smithi122 ceph-mon[32255]: pgmap v10599: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:51:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:51:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:52 smithi022 ceph-mon[31360]: pgmap v10599: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:52 smithi022 ceph-mon[40250]: pgmap v10599: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:51:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:54 smithi122 ceph-mon[32255]: pgmap v10600: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:54 smithi022 ceph-mon[31360]: pgmap v10600: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:54 smithi022 ceph-mon[40250]: pgmap v10600: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:55.160 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:51:55.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:51:55.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:51:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:51:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:56 smithi122 ceph-mon[32255]: pgmap v10601: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/574836443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:56 smithi022 ceph-mon[31360]: pgmap v10601: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/574836443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:56 smithi022 ceph-mon[40250]: pgmap v10601: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:51:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/574836443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:51:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:51:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:51:58 smithi122 ceph-mon[32255]: pgmap v10602: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:51:58 smithi022 ceph-mon[31360]: pgmap v10602: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:51:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:51:58 smithi022 ceph-mon[40250]: pgmap v10602: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:00] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:52:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:00 smithi122 ceph-mon[32255]: pgmap v10603: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:00 smithi022 ceph-mon[31360]: pgmap v10603: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:00 smithi022 ceph-mon[40250]: pgmap v10603: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:00.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:00.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:00.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:00.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/143121074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:01.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:52:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:52:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/143121074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/143121074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:02 smithi122 ceph-mon[32255]: pgmap v10604: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:52:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:52:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:02 smithi022 ceph-mon[31360]: pgmap v10604: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:02 smithi022 ceph-mon[40250]: pgmap v10604: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:04 smithi122 ceph-mon[32255]: pgmap v10605: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:04 smithi022 ceph-mon[31360]: pgmap v10605: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:04 smithi022 ceph-mon[40250]: pgmap v10605: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:05.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:06.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:06.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:06.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:06.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:06 smithi022 ceph-mon[31360]: pgmap v10606: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:06.274 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:06 smithi022 ceph-mon[40250]: pgmap v10606: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:06 smithi122 ceph-mon[32255]: pgmap v10606: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3131917547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3131917547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3131917547' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:52:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:08 smithi122 ceph-mon[32255]: pgmap v10607: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:08 smithi022 ceph-mon[31360]: pgmap v10607: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:08 smithi022 ceph-mon[40250]: pgmap v10607: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:10 smithi122 ceph-mon[32255]: pgmap v10608: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:10] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:52:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:10 smithi022 ceph-mon[31360]: pgmap v10608: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:10 smithi022 ceph-mon[40250]: pgmap v10608: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:11.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:52:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:52:11.274 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:11.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:11.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:11.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:52:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:52:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:12 smithi122 ceph-mon[32255]: pgmap v10609: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3963320954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:12 smithi022 ceph-mon[31360]: pgmap v10609: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3963320954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:12 smithi022 ceph-mon[40250]: pgmap v10609: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3963320954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:14 smithi122 ceph-mon[32255]: pgmap v10610: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:14 smithi022 ceph-mon[31360]: pgmap v10610: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:14 smithi022 ceph-mon[40250]: pgmap v10610: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:16 smithi122 ceph-mon[32255]: pgmap v10611: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:16 smithi022 ceph-mon[31360]: pgmap v10611: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:16 smithi022 ceph-mon[40250]: pgmap v10611: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:16.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:17.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:17.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1186540964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1186540964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1186540964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:52:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:18 smithi122 ceph-mon[32255]: pgmap v10612: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:18 smithi022 ceph-mon[31360]: pgmap v10612: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:18 smithi022 ceph-mon[40250]: pgmap v10612: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:20 smithi122 ceph-mon[32255]: pgmap v10613: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:20] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:52:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:20 smithi022 ceph-mon[31360]: pgmap v10613: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:20 smithi022 ceph-mon[40250]: pgmap v10613: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:21.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:52:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:52:22.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:22 smithi022 ceph-mon[31360]: pgmap v10614: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:22 smithi022 ceph-mon[40250]: pgmap v10614: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:22.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:22.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:22.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:52:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:52:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:22 smithi122 ceph-mon[32255]: pgmap v10614: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1715274881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1715274881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1715274881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:24 smithi122 ceph-mon[32255]: pgmap v10615: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:24 smithi022 ceph-mon[31360]: pgmap v10615: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:24 smithi022 ceph-mon[40250]: pgmap v10615: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:26 smithi122 ceph-mon[32255]: pgmap v10616: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:26 smithi022 ceph-mon[31360]: pgmap v10616: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:26 smithi022 ceph-mon[40250]: pgmap v10616: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:27.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:27.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:27.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:27.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:27.757 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:52:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:28 smithi122 ceph-mon[32255]: pgmap v10617: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4083873838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:28 smithi022 ceph-mon[31360]: pgmap v10617: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4083873838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:28 smithi022 ceph-mon[40250]: pgmap v10617: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4083873838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:30] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:52:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:30 smithi122 ceph-mon[32255]: pgmap v10618: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:30 smithi022 ceph-mon[31360]: pgmap v10618: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:30 smithi022 ceph-mon[40250]: pgmap v10618: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:31.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:52:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:52:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:52:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:52:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:32 smithi122 ceph-mon[32255]: pgmap v10619: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:32 smithi022 ceph-mon[31360]: pgmap v10619: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:32 smithi022 ceph-mon[40250]: pgmap v10619: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:32.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:33.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:33.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:33.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2489597883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2489597883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2489597883' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:34 smithi122 ceph-mon[32255]: pgmap v10620: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:52:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:52:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:52:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:52:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[31360]: pgmap v10620: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:52:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:52:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:52:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:52:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[40250]: pgmap v10620: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:52:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:52:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:52:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:52:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:36 smithi122 ceph-mon[32255]: pgmap v10621: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:36 smithi022 ceph-mon[31360]: pgmap v10621: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:36 smithi022 ceph-mon[40250]: pgmap v10621: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:52:38.130 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:38 smithi122 ceph-mon[32255]: pgmap v10622: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:38.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:38.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:38.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:38.501 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:38 smithi022 ceph-mon[40250]: pgmap v10622: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:38.501 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:38 smithi022 ceph-mon[31360]: pgmap v10622: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4097549287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4097549287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4097549287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:40 smithi122 ceph-mon[32255]: pgmap v10623: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:40] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:52:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:40 smithi022 ceph-mon[31360]: pgmap v10623: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:40 smithi022 ceph-mon[40250]: pgmap v10623: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:41.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:52:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:52:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:42 smithi122 ceph-mon[32255]: pgmap v10624: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:42 smithi022 ceph-mon[31360]: pgmap v10624: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:42 smithi022 ceph-mon[40250]: pgmap v10624: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:52:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:52:43.502 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:43.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:43.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:43.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:44 smithi022 ceph-mon[40250]: pgmap v10625: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/798873598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:44 smithi022 ceph-mon[31360]: pgmap v10625: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:44.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/798873598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:44 smithi122 ceph-mon[32255]: pgmap v10625: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/798873598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:46 smithi122 ceph-mon[32255]: pgmap v10626: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:46 smithi022 ceph-mon[40250]: pgmap v10626: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:46 smithi022 ceph-mon[31360]: pgmap v10626: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:47.685 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:52:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:48 smithi122 ceph-mon[32255]: pgmap v10627: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:52:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:52:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:52:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[31360]: pgmap v10627: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:52:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:52:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:52:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[40250]: pgmap v10627: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:52:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:52:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:52:48.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:49.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:49.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:49.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:49.239 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:52:49.240 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:52:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:52:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:50] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:52:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:50 smithi122 ceph-mon[32255]: pgmap v10628: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3649982679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:50 smithi022 ceph-mon[31360]: pgmap v10628: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3649982679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:50 smithi022 ceph-mon[40250]: pgmap v10628: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3649982679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:52:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:52:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:52 smithi122 ceph-mon[32255]: pgmap v10629: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:52:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:52:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:52 smithi022 ceph-mon[31360]: pgmap v10629: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:52 smithi022 ceph-mon[40250]: pgmap v10629: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:52:54.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:54 smithi122 ceph-mon[32255]: pgmap v10630: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:54 smithi022 ceph-mon[31360]: pgmap v10630: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:54 smithi022 ceph-mon[40250]: pgmap v10630: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:54.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:54.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:54.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:52:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2909703686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2909703686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2909703686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:52:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:56 smithi122 ceph-mon[32255]: pgmap v10631: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:56 smithi022 ceph-mon[31360]: pgmap v10631: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:56 smithi022 ceph-mon[40250]: pgmap v10631: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:52:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:52:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:52:58 smithi122 ceph-mon[32255]: pgmap v10632: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:52:58 smithi022 ceph-mon[31360]: pgmap v10632: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:52:58 smithi022 ceph-mon[40250]: pgmap v10632: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:52:59.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:52:59.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:52:59.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:52:59.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:00 smithi022 ceph-mon[31360]: pgmap v10633: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3157026678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:00 smithi022 ceph-mon[40250]: pgmap v10633: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3157026678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:00] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:53:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:00 smithi122 ceph-mon[32255]: pgmap v10633: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3157026678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:53:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:53:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:02 smithi122 ceph-mon[32255]: pgmap v10634: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:53:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:53:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:02 smithi022 ceph-mon[31360]: pgmap v10634: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:02 smithi022 ceph-mon[40250]: pgmap v10634: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:04 smithi122 ceph-mon[32255]: pgmap v10635: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:04 smithi022 ceph-mon[31360]: pgmap v10635: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:04 smithi022 ceph-mon[40250]: pgmap v10635: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:04.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:05.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:05.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:06 smithi122 ceph-mon[32255]: pgmap v10636: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2419332483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:06 smithi022 ceph-mon[31360]: pgmap v10636: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2419332483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:06 smithi022 ceph-mon[40250]: pgmap v10636: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2419332483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:53:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:08 smithi122 ceph-mon[32255]: pgmap v10637: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:08 smithi022 ceph-mon[31360]: pgmap v10637: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:08 smithi022 ceph-mon[40250]: pgmap v10637: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:10.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:10] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:53:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:10 smithi122 ceph-mon[32255]: pgmap v10638: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:10 smithi022 ceph-mon[31360]: pgmap v10638: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:10 smithi022 ceph-mon[40250]: pgmap v10638: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:10.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:10.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:10.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2897311101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:53:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:53:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2897311101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2897311101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:12 smithi122 ceph-mon[32255]: pgmap v10639: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:53:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:53:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:12 smithi022 ceph-mon[31360]: pgmap v10639: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:12 smithi022 ceph-mon[40250]: pgmap v10639: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:14 smithi122 ceph-mon[32255]: pgmap v10640: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:14 smithi022 ceph-mon[31360]: pgmap v10640: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:14 smithi022 ceph-mon[40250]: pgmap v10640: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:16.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:16.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:16.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:16 smithi122 ceph-mon[32255]: pgmap v10641: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3165299869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:16 smithi022 ceph-mon[31360]: pgmap v10641: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3165299869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:16 smithi022 ceph-mon[40250]: pgmap v10641: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3165299869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:53:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:18 smithi122 ceph-mon[32255]: pgmap v10642: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:18 smithi022 ceph-mon[31360]: pgmap v10642: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:18 smithi022 ceph-mon[40250]: pgmap v10642: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:20] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:53:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:20 smithi122 ceph-mon[32255]: pgmap v10643: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:20 smithi022 ceph-mon[31360]: pgmap v10643: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:20 smithi022 ceph-mon[40250]: pgmap v10643: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:21.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:53:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:53:21.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:21.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:21.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:22 smithi122 ceph-mon[32255]: pgmap v10644: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/696604683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:53:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:53:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:22 smithi022 ceph-mon[40250]: pgmap v10644: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/696604683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:22 smithi022 ceph-mon[31360]: pgmap v10644: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/696604683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:24 smithi122 ceph-mon[32255]: pgmap v10645: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:24 smithi022 ceph-mon[40250]: pgmap v10645: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:24 smithi022 ceph-mon[31360]: pgmap v10645: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:26 smithi122 ceph-mon[32255]: pgmap v10646: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:26.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:26.466 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:26 smithi022 ceph-mon[40250]: pgmap v10646: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:26.467 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:26 smithi022 ceph-mon[31360]: pgmap v10646: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:27 smithi122 ceph-mon[32255]: pgmap v10647: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3132106087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:27 smithi022 ceph-mon[31360]: pgmap v10647: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3132106087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:27.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:53:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:27 smithi022 ceph-mon[40250]: pgmap v10647: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:27.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3132106087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:29 smithi122 ceph-mon[32255]: pgmap v10648: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:29 smithi022 ceph-mon[31360]: pgmap v10648: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:29 smithi022 ceph-mon[40250]: pgmap v10648: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:30] "GET /metrics HTTP/1.1" 200 202391 "" "Prometheus/2.18.1" 2021-12-15T23:53:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:53:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:53:31.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:32.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:32.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:32.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:32.205 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:32 smithi022 ceph-mon[31360]: pgmap v10649: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:32.205 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:32.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:32 smithi022 ceph-mon[40250]: pgmap v10649: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:32.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:32 smithi122 ceph-mon[32255]: pgmap v10649: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:53:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:53:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3553592393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3553592393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3553592393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:34 smithi122 ceph-mon[32255]: pgmap v10650: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:53:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:53:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:53:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:53:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[31360]: pgmap v10650: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:53:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:53:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:53:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:53:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[40250]: pgmap v10650: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:53:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:53:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:53:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:53:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:36 smithi122 ceph-mon[32255]: pgmap v10651: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:36 smithi022 ceph-mon[31360]: pgmap v10651: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:36 smithi022 ceph-mon[40250]: pgmap v10651: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:37.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:53:37.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:37.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:37.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:38 smithi122 ceph-mon[32255]: pgmap v10652: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2475306520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:38 smithi022 ceph-mon[31360]: pgmap v10652: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2475306520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:38 smithi022 ceph-mon[40250]: pgmap v10652: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2475306520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:40] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:53:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:40 smithi122 ceph-mon[32255]: pgmap v10653: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:40 smithi022 ceph-mon[31360]: pgmap v10653: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:40 smithi022 ceph-mon[40250]: pgmap v10653: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:53:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:53:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:53:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:53:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:42 smithi122 ceph-mon[32255]: pgmap v10654: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:42 smithi022 ceph-mon[31360]: pgmap v10654: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:42 smithi022 ceph-mon[40250]: pgmap v10654: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:42.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:42.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:42.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:42.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3073359308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3073359308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3073359308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:44 smithi122 ceph-mon[32255]: pgmap v10655: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:44 smithi022 ceph-mon[31360]: pgmap v10655: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:44 smithi022 ceph-mon[40250]: pgmap v10655: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:46 smithi122 ceph-mon[32255]: pgmap v10656: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:46 smithi022 ceph-mon[31360]: pgmap v10656: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:46 smithi022 ceph-mon[40250]: pgmap v10656: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:53:47.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:48 smithi022 ceph-mon[31360]: pgmap v10657: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:48 smithi022 ceph-mon[40250]: pgmap v10657: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:48.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:48.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:48.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:48 smithi122 ceph-mon[32255]: pgmap v10657: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:53:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:53:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:53:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/592571946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:53:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:53:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:53:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:53:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/592571946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:53:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:53:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:53:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:53:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/592571946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:53:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:50 smithi122 ceph-mon[32255]: pgmap v10658: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:50] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-15T23:53:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:50 smithi022 ceph-mon[31360]: pgmap v10658: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:50 smithi022 ceph-mon[40250]: pgmap v10658: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:53:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:53:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:52 smithi122 ceph-mon[32255]: pgmap v10659: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:53:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:53:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:52 smithi022 ceph-mon[31360]: pgmap v10659: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:52.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:52 smithi022 ceph-mon[40250]: pgmap v10659: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:53:53.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:53.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:53.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:53.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:54 smithi122 ceph-mon[32255]: pgmap v10660: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/888824204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:54 smithi022 ceph-mon[31360]: pgmap v10660: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/888824204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:54 smithi022 ceph-mon[40250]: pgmap v10660: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/888824204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:56 smithi122 ceph-mon[32255]: pgmap v10661: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:56 smithi022 ceph-mon[31360]: pgmap v10661: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:56 smithi022 ceph-mon[40250]: pgmap v10661: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:53:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:53:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:58 smithi122 ceph-mon[32255]: pgmap v10662: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:58 smithi022 ceph-mon[31360]: pgmap v10662: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:58 smithi022 ceph-mon[40250]: pgmap v10662: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:53:58.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:53:59.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:53:59.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:53:59.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:53:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:53:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/158538221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:53:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/158538221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:53:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:53:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/158538221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:00] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-15T23:54:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:00 smithi122 ceph-mon[32255]: pgmap v10663: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:00 smithi022 ceph-mon[31360]: pgmap v10663: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:00 smithi022 ceph-mon[40250]: pgmap v10663: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:54:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:54:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:54:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:54:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:02 smithi122 ceph-mon[32255]: pgmap v10664: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:02 smithi022 ceph-mon[31360]: pgmap v10664: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:02 smithi022 ceph-mon[40250]: pgmap v10664: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:04.056 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:04.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:04.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:04.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:04.423 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:04 smithi022 ceph-mon[40250]: pgmap v10665: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:04.424 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:04 smithi022 ceph-mon[31360]: pgmap v10665: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:04 smithi122 ceph-mon[32255]: pgmap v10665: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/113227263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/113227263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/113227263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:06 smithi122 ceph-mon[32255]: pgmap v10666: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:06 smithi022 ceph-mon[31360]: pgmap v10666: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:06 smithi022 ceph-mon[40250]: pgmap v10666: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:54:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:08 smithi122 ceph-mon[32255]: pgmap v10667: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:08 smithi022 ceph-mon[31360]: pgmap v10667: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:08 smithi022 ceph-mon[40250]: pgmap v10667: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:09.424 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:09.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:09.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:09.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:10 smithi122 ceph-mon[32255]: pgmap v10668: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2320852345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:10] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:54:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:10 smithi022 ceph-mon[31360]: pgmap v10668: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2320852345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:10 smithi022 ceph-mon[40250]: pgmap v10668: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2320852345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:54:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:54:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:12 smithi122 ceph-mon[32255]: pgmap v10669: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:54:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:54:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:12 smithi022 ceph-mon[31360]: pgmap v10669: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:12 smithi022 ceph-mon[40250]: pgmap v10669: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:14 smithi122 ceph-mon[32255]: pgmap v10670: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:14 smithi022 ceph-mon[31360]: pgmap v10670: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:14 smithi022 ceph-mon[40250]: pgmap v10670: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:14.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:15.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:15.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:15.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:16 smithi122 ceph-mon[32255]: pgmap v10671: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2304177421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:16 smithi022 ceph-mon[31360]: pgmap v10671: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2304177421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:16 smithi022 ceph-mon[40250]: pgmap v10671: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2304177421' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:54:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:18 smithi122 ceph-mon[32255]: pgmap v10672: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:18 smithi022 ceph-mon[31360]: pgmap v10672: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:18 smithi022 ceph-mon[40250]: pgmap v10672: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:20.163 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:20] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:54:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:20 smithi122 ceph-mon[32255]: pgmap v10673: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:20 smithi022 ceph-mon[31360]: pgmap v10673: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:20 smithi022 ceph-mon[40250]: pgmap v10673: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:20.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:20.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:20.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:54:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:54:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/96331344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/96331344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/96331344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:22 smithi122 ceph-mon[32255]: pgmap v10674: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:54:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:54:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:22 smithi022 ceph-mon[31360]: pgmap v10674: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:22 smithi022 ceph-mon[40250]: pgmap v10674: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:24 smithi122 ceph-mon[32255]: pgmap v10675: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:24 smithi022 ceph-mon[31360]: pgmap v10675: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:24 smithi022 ceph-mon[40250]: pgmap v10675: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:25.538 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:25.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:25.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:25.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:26 smithi122 ceph-mon[32255]: pgmap v10676: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3259454217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:26 smithi022 ceph-mon[31360]: pgmap v10676: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3259454217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:26 smithi022 ceph-mon[40250]: pgmap v10676: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3259454217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:54:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:28 smithi122 ceph-mon[32255]: pgmap v10677: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:28 smithi022 ceph-mon[31360]: pgmap v10677: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:28 smithi022 ceph-mon[40250]: pgmap v10677: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:30] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:54:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:30 smithi122 ceph-mon[32255]: pgmap v10678: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:30 smithi022 ceph-mon[31360]: pgmap v10678: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:30 smithi022 ceph-mon[40250]: pgmap v10678: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:30.906 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:31.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:31.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:31.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:54:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:54:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:54:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:54:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:32 smithi122 ceph-mon[32255]: pgmap v10679: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1210131722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:32 smithi022 ceph-mon[31360]: pgmap v10679: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1210131722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:32 smithi022 ceph-mon[40250]: pgmap v10679: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1210131722' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:34 smithi122 ceph-mon[32255]: pgmap v10680: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:54:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:54:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:54:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:54:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[31360]: pgmap v10680: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:54:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:54:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:54:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:54:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[40250]: pgmap v10680: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:54:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:54:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:54:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:54:36.277 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:36 smithi122 ceph-mon[32255]: pgmap v10681: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:36 smithi022 ceph-mon[31360]: pgmap v10681: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:36 smithi022 ceph-mon[40250]: pgmap v10681: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:36.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:36.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:36.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1915422516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1915422516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:54:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1915422516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:38 smithi122 ceph-mon[32255]: pgmap v10682: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:38 smithi022 ceph-mon[31360]: pgmap v10682: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:38 smithi022 ceph-mon[40250]: pgmap v10682: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:40] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:54:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:40 smithi122 ceph-mon[32255]: pgmap v10683: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:40 smithi022 ceph-mon[31360]: pgmap v10683: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:40 smithi022 ceph-mon[40250]: pgmap v10683: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:54:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:54:41.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:42.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:42.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:42.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:42 smithi022 ceph-mon[31360]: pgmap v10684: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1545977407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:42 smithi022 ceph-mon[40250]: pgmap v10684: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1545977407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:54:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:54:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:42 smithi122 ceph-mon[32255]: pgmap v10684: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1545977407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:44 smithi122 ceph-mon[32255]: pgmap v10685: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:44 smithi022 ceph-mon[31360]: pgmap v10685: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:44 smithi022 ceph-mon[40250]: pgmap v10685: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:46 smithi122 ceph-mon[32255]: pgmap v10686: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:46 smithi022 ceph-mon[31360]: pgmap v10686: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:46 smithi022 ceph-mon[40250]: pgmap v10686: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:47.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:47.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:47.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:47.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:54:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:48 smithi122 ceph-mon[32255]: pgmap v10687: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2351674430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:48 smithi022 ceph-mon[31360]: pgmap v10687: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2351674430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:48 smithi022 ceph-mon[40250]: pgmap v10687: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2351674430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:54:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:54:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:54:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:54:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:54:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:54:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:54:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:54:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:54:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:50] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:54:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:50 smithi122 ceph-mon[32255]: pgmap v10688: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:50 smithi022 ceph-mon[31360]: pgmap v10688: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:50 smithi022 ceph-mon[40250]: pgmap v10688: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:51.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:54:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:54:52.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:52 smithi022 ceph-mon[40250]: pgmap v10689: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:52.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:52.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:52 smithi022 ceph-mon[31360]: pgmap v10689: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:52.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:52.393 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:54:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:54:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:52 smithi122 ceph-mon[32255]: pgmap v10689: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:54:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:53.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/323665587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/323665587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:53.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:53.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-15T23:54:53.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:53.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/323665587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:54 smithi122 ceph-mon[32255]: pgmap v10690: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:54 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3110M 2021-12-15T23:54:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[31360]: pgmap v10690: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3110M 2021-12-15T23:54:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[40250]: pgmap v10690: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3110M 2021-12-15T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:54:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:56 smithi122 ceph-mon[32255]: pgmap v10691: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:56 smithi022 ceph-mon[31360]: pgmap v10691: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:56 smithi022 ceph-mon[40250]: pgmap v10691: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:54:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:54:57.754 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:54:58.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:54:58.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:54:58.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:54:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:58 smithi122 ceph-mon[32255]: pgmap v10692: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:54:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/842877783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:58 smithi022 ceph-mon[31360]: pgmap v10692: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:54:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/842877783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:54:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:58 smithi022 ceph-mon[40250]: pgmap v10692: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:54:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:54:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/842877783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:00 smithi122 ceph-mon[32255]: pgmap v10693: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:00] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:55:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:00 smithi022 ceph-mon[31360]: pgmap v10693: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:00 smithi022 ceph-mon[40250]: pgmap v10693: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:01.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:55:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:55:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:02 smithi122 ceph-mon[32255]: pgmap v10694: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:55:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:55:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:02 smithi022 ceph-mon[31360]: pgmap v10694: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:02 smithi022 ceph-mon[40250]: pgmap v10694: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:03.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:03.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:03.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:04 smithi122 ceph-mon[32255]: pgmap v10695: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2504912448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:04 smithi022 ceph-mon[31360]: pgmap v10695: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2504912448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:04 smithi022 ceph-mon[40250]: pgmap v10695: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2504912448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:06 smithi122 ceph-mon[32255]: pgmap v10696: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:06 smithi022 ceph-mon[31360]: pgmap v10696: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:06 smithi022 ceph-mon[40250]: pgmap v10696: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:55:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:08 smithi122 ceph-mon[32255]: pgmap v10697: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:08.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:08.500 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:08 smithi022 ceph-mon[31360]: pgmap v10697: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:08.501 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:08 smithi022 ceph-mon[40250]: pgmap v10697: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:08.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:08.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:08.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/832025069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/832025069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/832025069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:10] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-15T23:55:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:10 smithi122 ceph-mon[32255]: pgmap v10698: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:10 smithi022 ceph-mon[31360]: pgmap v10698: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:10 smithi022 ceph-mon[40250]: pgmap v10698: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:55:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:55:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:12 smithi122 ceph-mon[32255]: pgmap v10699: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:55:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:55:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:12 smithi022 ceph-mon[31360]: pgmap v10699: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:12 smithi022 ceph-mon[40250]: pgmap v10699: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:13.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:14.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:14.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:14.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:14 smithi122 ceph-mon[32255]: pgmap v10700: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:14 smithi022 ceph-mon[31360]: pgmap v10700: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:14 smithi022 ceph-mon[40250]: pgmap v10700: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-15T23:55:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/476918435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/476918435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/476918435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:16 smithi122 ceph-mon[32255]: pgmap v10701: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:16 smithi022 ceph-mon[31360]: pgmap v10701: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:16 smithi022 ceph-mon[40250]: pgmap v10701: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:55:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:18 smithi122 ceph-mon[32255]: pgmap v10702: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:18 smithi022 ceph-mon[31360]: pgmap v10702: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:18 smithi022 ceph-mon[40250]: pgmap v10702: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:19.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:19.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:19.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:19.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:20] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:55:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:20 smithi122 ceph-mon[32255]: pgmap v10703: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1121335802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:20 smithi022 ceph-mon[31360]: pgmap v10703: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1121335802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:20 smithi022 ceph-mon[40250]: pgmap v10703: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1121335802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:55:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:55:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:22 smithi122 ceph-mon[32255]: pgmap v10704: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:55:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:55:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:22 smithi022 ceph-mon[31360]: pgmap v10704: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:22 smithi022 ceph-mon[40250]: pgmap v10704: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:24 smithi122 ceph-mon[32255]: pgmap v10705: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:24 smithi022 ceph-mon[31360]: pgmap v10705: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:24 smithi022 ceph-mon[40250]: pgmap v10705: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:24.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:24.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:24.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:24.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2161118925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2161118925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2161118925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:26 smithi122 ceph-mon[32255]: pgmap v10706: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:26 smithi022 ceph-mon[31360]: pgmap v10706: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:26 smithi022 ceph-mon[40250]: pgmap v10706: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:55:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:28 smithi022 ceph-mon[31360]: pgmap v10707: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:28 smithi022 ceph-mon[40250]: pgmap v10707: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:28 smithi122 ceph-mon[32255]: pgmap v10707: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:29.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:30 smithi022 ceph-mon[31360]: pgmap v10708: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:30 smithi022 ceph-mon[40250]: pgmap v10708: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:30.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:30.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:30.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:30] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:55:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:30 smithi122 ceph-mon[32255]: pgmap v10708: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2978610038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:55:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:55:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2978610038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2978610038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:32 smithi022 ceph-mon[31360]: pgmap v10709: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:32 smithi022 ceph-mon[40250]: pgmap v10709: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:32 smithi122 ceph-mon[32255]: pgmap v10709: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:55:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:55:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:33 smithi022 ceph-mon[31360]: pgmap v10710: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:33 smithi022 ceph-mon[40250]: pgmap v10710: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:33 smithi122 ceph-mon[32255]: pgmap v10710: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:55:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:55:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:55:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:55:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:55:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:55:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:55:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:55:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:55:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:55:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:55:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:55:35.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:35.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:35.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:35.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:36 smithi122 ceph-mon[32255]: pgmap v10711: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1428192528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:36 smithi022 ceph-mon[31360]: pgmap v10711: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1428192528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:36 smithi022 ceph-mon[40250]: pgmap v10711: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1428192528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:55:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:38 smithi122 ceph-mon[32255]: pgmap v10712: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:38 smithi022 ceph-mon[31360]: pgmap v10712: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:38 smithi022 ceph-mon[40250]: pgmap v10712: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:40 smithi122 ceph-mon[32255]: pgmap v10713: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:40] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-15T23:55:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:40 smithi022 ceph-mon[31360]: pgmap v10713: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:40 smithi022 ceph-mon[40250]: pgmap v10713: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:40.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:41.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:41.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:41.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1058149367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:55:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:55:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1058149367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1058149367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:42 smithi122 ceph-mon[32255]: pgmap v10714: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:55:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:55:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:42 smithi022 ceph-mon[31360]: pgmap v10714: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:42 smithi022 ceph-mon[40250]: pgmap v10714: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:44 smithi122 ceph-mon[32255]: pgmap v10715: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:44 smithi022 ceph-mon[31360]: pgmap v10715: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:44 smithi022 ceph-mon[40250]: pgmap v10715: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:46.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:46 smithi122 ceph-mon[32255]: pgmap v10716: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:46.464 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:46 smithi022 ceph-mon[31360]: pgmap v10716: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:46.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:46 smithi022 ceph-mon[40250]: pgmap v10716: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3623478044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3623478044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:47.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3623478044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:55:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:48 smithi122 ceph-mon[32255]: pgmap v10717: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:48 smithi022 ceph-mon[31360]: pgmap v10717: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:48 smithi022 ceph-mon[40250]: pgmap v10717: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:50] "GET /metrics HTTP/1.1" 200 202400 "" "Prometheus/2.18.1" 2021-12-15T23:55:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:50 smithi122 ceph-mon[32255]: pgmap v10718: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:50 smithi022 ceph-mon[31360]: pgmap v10718: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:50 smithi022 ceph-mon[40250]: pgmap v10718: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:55:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:55:51.465 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:51.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:51.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:51.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:55:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:55:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:52 smithi122 ceph-mon[32255]: pgmap v10719: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/60260182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:52 smithi022 ceph-mon[31360]: pgmap v10719: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/60260182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:52 smithi022 ceph-mon[40250]: pgmap v10719: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:55:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/60260182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:54 smithi122 ceph-mon[32255]: pgmap v10720: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:55:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:55:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:55:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:55:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[31360]: pgmap v10720: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:55:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:55:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:55:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:55:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[40250]: pgmap v10720: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:55:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:55:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:55:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:55:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:56 smithi122 ceph-mon[32255]: pgmap v10721: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:56 smithi022 ceph-mon[31360]: pgmap v10721: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:56 smithi022 ceph-mon[40250]: pgmap v10721: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:55:56.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:55:57.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:55:57.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:55:57.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:55:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:55:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:58 smithi122 ceph-mon[32255]: pgmap v10722: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:55:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/957844370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:58 smithi022 ceph-mon[31360]: pgmap v10722: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:55:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/957844370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:55:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:58 smithi022 ceph-mon[40250]: pgmap v10722: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:55:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:55:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/957844370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:00] "GET /metrics HTTP/1.1" 200 202400 "" "Prometheus/2.18.1" 2021-12-15T23:56:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:00 smithi122 ceph-mon[32255]: pgmap v10723: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:00 smithi022 ceph-mon[31360]: pgmap v10723: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:00 smithi022 ceph-mon[40250]: pgmap v10723: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:01.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:56:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:56:02.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:56:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:56:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:02 smithi122 ceph-mon[32255]: pgmap v10724: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:02 smithi022 ceph-mon[31360]: pgmap v10724: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:02 smithi022 ceph-mon[40250]: pgmap v10724: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:02.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:02.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:02.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/190874178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/190874178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/190874178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:04 smithi122 ceph-mon[32255]: pgmap v10725: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:04 smithi022 ceph-mon[31360]: pgmap v10725: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:04 smithi022 ceph-mon[40250]: pgmap v10725: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:06 smithi122 ceph-mon[32255]: pgmap v10726: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:06 smithi022 ceph-mon[31360]: pgmap v10726: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:06 smithi022 ceph-mon[40250]: pgmap v10726: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:07.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:56:07.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:07.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:07.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:08 smithi022 ceph-mon[31360]: pgmap v10727: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/219855642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:08 smithi022 ceph-mon[40250]: pgmap v10727: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/219855642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:08 smithi122 ceph-mon[32255]: pgmap v10727: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/219855642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:10] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:56:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:10 smithi122 ceph-mon[32255]: pgmap v10728: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:10 smithi022 ceph-mon[31360]: pgmap v10728: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:10 smithi022 ceph-mon[40250]: pgmap v10728: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:56:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:56:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:56:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:56:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:12 smithi122 ceph-mon[32255]: pgmap v10729: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:12 smithi022 ceph-mon[31360]: pgmap v10729: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:12 smithi022 ceph-mon[40250]: pgmap v10729: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:12.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:14 smithi122 ceph-mon[32255]: pgmap v10730: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:14.650 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1851770320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:14.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:14 smithi022 ceph-mon[31360]: pgmap v10730: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:14.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1851770320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:14.651 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:14 smithi022 ceph-mon[40250]: pgmap v10730: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:14.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1851770320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:16 smithi122 ceph-mon[32255]: pgmap v10731: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:16 smithi022 ceph-mon[40250]: pgmap v10731: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:16 smithi022 ceph-mon[31360]: pgmap v10731: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:56:18.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:18 smithi122 ceph-mon[32255]: pgmap v10732: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:18 smithi022 ceph-mon[31360]: pgmap v10732: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:18 smithi022 ceph-mon[40250]: pgmap v10732: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:18.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:18.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:18.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/475652388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/475652388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/475652388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:20] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:56:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:20 smithi122 ceph-mon[32255]: pgmap v10733: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:20 smithi022 ceph-mon[31360]: pgmap v10733: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:20 smithi022 ceph-mon[40250]: pgmap v10733: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:56:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:56:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:22 smithi122 ceph-mon[32255]: pgmap v10734: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:56:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:56:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:22 smithi022 ceph-mon[31360]: pgmap v10734: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:22 smithi022 ceph-mon[40250]: pgmap v10734: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:23.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:24.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:24 smithi122 ceph-mon[32255]: pgmap v10735: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2333586196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:24 smithi022 ceph-mon[31360]: pgmap v10735: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2333586196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:24 smithi022 ceph-mon[40250]: pgmap v10735: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2333586196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:26 smithi122 ceph-mon[32255]: pgmap v10736: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:26 smithi022 ceph-mon[31360]: pgmap v10736: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:26 smithi022 ceph-mon[40250]: pgmap v10736: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:56:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:28 smithi122 ceph-mon[32255]: pgmap v10737: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:28 smithi022 ceph-mon[31360]: pgmap v10737: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:28 smithi022 ceph-mon[40250]: pgmap v10737: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:29.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:29.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:29.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:29.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:30] "GET /metrics HTTP/1.1" 200 202390 "" "Prometheus/2.18.1" 2021-12-15T23:56:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:30 smithi122 ceph-mon[32255]: pgmap v10738: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2145779038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:30 smithi022 ceph-mon[31360]: pgmap v10738: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2145779038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:30 smithi022 ceph-mon[40250]: pgmap v10738: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2145779038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:31.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:56:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:56:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:32 smithi122 ceph-mon[32255]: pgmap v10739: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:56:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:56:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:32 smithi022 ceph-mon[31360]: pgmap v10739: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:32 smithi022 ceph-mon[40250]: pgmap v10739: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:34.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:34.430 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[31360]: pgmap v10740: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:34.430 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:56:34.430 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:56:34.430 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:56:34.431 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:56:34.431 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[40250]: pgmap v10740: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:34.431 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:56:34.431 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:56:34.432 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:56:34.432 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:56:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:34 smithi122 ceph-mon[32255]: pgmap v10740: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:56:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:56:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:56:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:56:34.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:34.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:34.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2004654499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2004654499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2004654499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:36 smithi122 ceph-mon[32255]: pgmap v10741: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:36 smithi022 ceph-mon[31360]: pgmap v10741: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:36 smithi022 ceph-mon[40250]: pgmap v10741: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:56:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:38 smithi122 ceph-mon[32255]: pgmap v10742: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:38 smithi022 ceph-mon[31360]: pgmap v10742: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:38 smithi022 ceph-mon[40250]: pgmap v10742: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:39.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:40.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:40.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:40.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:40] "GET /metrics HTTP/1.1" 200 202392 "" "Prometheus/2.18.1" 2021-12-15T23:56:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:40 smithi122 ceph-mon[32255]: pgmap v10743: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1953264334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:40 smithi022 ceph-mon[31360]: pgmap v10743: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1953264334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:40 smithi022 ceph-mon[40250]: pgmap v10743: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1953264334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:56:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:56:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:42 smithi022 ceph-mon[31360]: pgmap v10744: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:42 smithi022 ceph-mon[40250]: pgmap v10744: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:56:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:56:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:42 smithi122 ceph-mon[32255]: pgmap v10744: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:44 smithi022 ceph-mon[31360]: pgmap v10745: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:44 smithi022 ceph-mon[40250]: pgmap v10745: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:44 smithi122 ceph-mon[32255]: pgmap v10745: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:45.171 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:45.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:45.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:45.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:46 smithi022 ceph-mon[31360]: pgmap v10746: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/941410075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:46 smithi022 ceph-mon[40250]: pgmap v10746: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/941410075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:46 smithi122 ceph-mon[32255]: pgmap v10746: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/941410075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:56:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:48 smithi022 ceph-mon[31360]: pgmap v10747: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:48 smithi022 ceph-mon[40250]: pgmap v10747: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:48 smithi122 ceph-mon[32255]: pgmap v10747: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:50 smithi122 ceph-mon[32255]: pgmap v10748: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:50] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-15T23:56:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:50 smithi022 ceph-mon[31360]: pgmap v10748: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:50 smithi022 ceph-mon[40250]: pgmap v10748: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:50.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/598713734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:51.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:56:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:56:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/598713734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/598713734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:52 smithi022 ceph-mon[31360]: pgmap v10749: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:52 smithi022 ceph-mon[40250]: pgmap v10749: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:52 smithi122 ceph-mon[32255]: pgmap v10749: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:56:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:56:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:56:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:53 smithi022 ceph-mon[31360]: pgmap v10750: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:53 smithi022 ceph-mon[40250]: pgmap v10750: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:53 smithi122 ceph-mon[32255]: pgmap v10750: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:56:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:56:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:56:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:56:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:56:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:56:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:56:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:56:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:56:55.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:56:55.915 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:55 smithi022 ceph-mon[40250]: pgmap v10751: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:55.915 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:56:55.915 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:55 smithi022 ceph-mon[31360]: pgmap v10751: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:55.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:56:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:56:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:55 smithi122 ceph-mon[32255]: pgmap v10751: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:56:56.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:56:56.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:56:56.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:56:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2603989052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2603989052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2603989052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:56:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:57 smithi022 ceph-mon[31360]: pgmap v10752: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:56:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:57 smithi022 ceph-mon[40250]: pgmap v10752: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:57 smithi122 ceph-mon[32255]: pgmap v10752: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:56:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:56:59 smithi122 ceph-mon[32255]: pgmap v10753: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:56:59 smithi022 ceph-mon[31360]: pgmap v10753: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:56:59 smithi022 ceph-mon[40250]: pgmap v10753: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:00] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-15T23:57:01.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:01.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:57:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:57:01.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:01.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:01.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:02 smithi122 ceph-mon[32255]: pgmap v10754: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/655794199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:57:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:57:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:02 smithi022 ceph-mon[31360]: pgmap v10754: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/655794199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:02 smithi022 ceph-mon[40250]: pgmap v10754: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/655794199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:04 smithi122 ceph-mon[32255]: pgmap v10755: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:04 smithi022 ceph-mon[31360]: pgmap v10755: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:04 smithi022 ceph-mon[40250]: pgmap v10755: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:06 smithi122 ceph-mon[32255]: pgmap v10756: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:06 smithi022 ceph-mon[31360]: pgmap v10756: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:06 smithi022 ceph-mon[40250]: pgmap v10756: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:06.653 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:07.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:07.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:07.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/79180814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/79180814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/79180814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:57:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:08 smithi122 ceph-mon[32255]: pgmap v10757: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:08 smithi022 ceph-mon[31360]: pgmap v10757: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:08 smithi022 ceph-mon[40250]: pgmap v10757: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:10] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:57:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:10 smithi122 ceph-mon[32255]: pgmap v10758: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:10 smithi022 ceph-mon[31360]: pgmap v10758: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:10 smithi022 ceph-mon[40250]: pgmap v10758: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:57:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:57:12.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:12 smithi022 ceph-mon[31360]: pgmap v10759: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:12 smithi022 ceph-mon[40250]: pgmap v10759: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:12.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:12.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:12.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:12 smithi122 ceph-mon[32255]: pgmap v10759: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:57:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:57:13.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1416557914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1416557914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1416557914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:14 smithi122 ceph-mon[32255]: pgmap v10760: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:14 smithi022 ceph-mon[31360]: pgmap v10760: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:14 smithi022 ceph-mon[40250]: pgmap v10760: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:16 smithi122 ceph-mon[32255]: pgmap v10761: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:16 smithi022 ceph-mon[31360]: pgmap v10761: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:16 smithi022 ceph-mon[40250]: pgmap v10761: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:17.393 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:17.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:17.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:17.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:17.761 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:57:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:18 smithi122 ceph-mon[32255]: pgmap v10762: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744998413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:18 smithi022 ceph-mon[31360]: pgmap v10762: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744998413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:18 smithi022 ceph-mon[40250]: pgmap v10762: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744998413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:20 smithi122 ceph-mon[32255]: pgmap v10763: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:20] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:57:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:20 smithi022 ceph-mon[31360]: pgmap v10763: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:20 smithi022 ceph-mon[40250]: pgmap v10763: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:21.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:57:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:57:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:22 smithi122 ceph-mon[32255]: pgmap v10764: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:57:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:57:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:22 smithi022 ceph-mon[31360]: pgmap v10764: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:22 smithi022 ceph-mon[40250]: pgmap v10764: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:22.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:23.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:23.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:23.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2095860469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2095860469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2095860469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:24 smithi122 ceph-mon[32255]: pgmap v10765: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:24 smithi022 ceph-mon[31360]: pgmap v10765: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:24 smithi022 ceph-mon[40250]: pgmap v10765: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:26 smithi122 ceph-mon[32255]: pgmap v10766: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:26 smithi022 ceph-mon[31360]: pgmap v10766: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:26 smithi022 ceph-mon[40250]: pgmap v10766: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:57:28.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:28 smithi122 ceph-mon[32255]: pgmap v10767: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:28.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:28.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:28.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:28.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:28 smithi022 ceph-mon[31360]: pgmap v10767: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:28.505 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:28 smithi022 ceph-mon[40250]: pgmap v10767: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1674880598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1674880598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1674880598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:30 smithi122 ceph-mon[32255]: pgmap v10768: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:30] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:57:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:30 smithi022 ceph-mon[31360]: pgmap v10768: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:30 smithi022 ceph-mon[40250]: pgmap v10768: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:57:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:57:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:57:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:57:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:32 smithi122 ceph-mon[32255]: pgmap v10769: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:32 smithi022 ceph-mon[31360]: pgmap v10769: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:32 smithi022 ceph-mon[40250]: pgmap v10769: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:33.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:33.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:33.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:33.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:34 smithi122 ceph-mon[32255]: pgmap v10770: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:57:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:57:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:57:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:57:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1638562821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[31360]: pgmap v10770: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:57:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:57:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:57:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:57:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1638562821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[40250]: pgmap v10770: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:57:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:57:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:57:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:57:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1638562821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:36 smithi122 ceph-mon[32255]: pgmap v10771: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:36 smithi022 ceph-mon[31360]: pgmap v10771: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:36 smithi022 ceph-mon[40250]: pgmap v10771: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:57:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:38 smithi122 ceph-mon[32255]: pgmap v10772: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:38 smithi022 ceph-mon[31360]: pgmap v10772: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:38 smithi022 ceph-mon[40250]: pgmap v10772: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:38.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:39.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:39.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:39.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:40] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-15T23:57:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:40 smithi122 ceph-mon[32255]: pgmap v10773: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3259644896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:40 smithi022 ceph-mon[31360]: pgmap v10773: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3259644896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:40 smithi022 ceph-mon[40250]: pgmap v10773: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3259644896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:41.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:57:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:57:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:57:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:57:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:42 smithi122 ceph-mon[32255]: pgmap v10774: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:42 smithi022 ceph-mon[31360]: pgmap v10774: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:42 smithi022 ceph-mon[40250]: pgmap v10774: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:44.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:44 smithi122 ceph-mon[32255]: pgmap v10775: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:44 smithi022 ceph-mon[31360]: pgmap v10775: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:44 smithi022 ceph-mon[40250]: pgmap v10775: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3780981407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3780981407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3780981407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:46 smithi122 ceph-mon[32255]: pgmap v10776: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:46 smithi022 ceph-mon[31360]: pgmap v10776: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:46 smithi022 ceph-mon[40250]: pgmap v10776: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:57:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:48 smithi022 ceph-mon[31360]: pgmap v10777: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:48 smithi022 ceph-mon[40250]: pgmap v10777: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:48 smithi122 ceph-mon[32255]: pgmap v10777: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:49.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:49.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:49.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:49.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:50 smithi022 ceph-mon[31360]: pgmap v10778: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3077257285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:50 smithi022 ceph-mon[40250]: pgmap v10778: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:50.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3077257285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:50] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:57:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:50 smithi122 ceph-mon[32255]: pgmap v10778: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3077257285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:57:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:57:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:52 smithi022 ceph-mon[31360]: pgmap v10779: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:52 smithi022 ceph-mon[40250]: pgmap v10779: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:52 smithi122 ceph-mon[32255]: pgmap v10779: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:57:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:57:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:57:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:54 smithi022 ceph-mon[31360]: pgmap v10780: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:54 smithi022 ceph-mon[40250]: pgmap v10780: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:54.588 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:54 smithi122 ceph-mon[32255]: pgmap v10780: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:54.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:57:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:57:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:57:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:57:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:57:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:57:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:57:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:57:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:57:55.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:57:55.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:57:55.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:57:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:57:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:57:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:57:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:57:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:56 smithi022 ceph-mon[31360]: pgmap v10781: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1103395233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:56 smithi022 ceph-mon[40250]: pgmap v10781: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1103395233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:56 smithi122 ceph-mon[32255]: pgmap v10781: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:57:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1103395233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:57:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:57:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:57:58 smithi022 ceph-mon[31360]: pgmap v10782: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:57:58 smithi022 ceph-mon[40250]: pgmap v10782: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:57:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:57:58 smithi122 ceph-mon[32255]: pgmap v10782: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:00.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:00] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:58:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:00 smithi122 ceph-mon[32255]: pgmap v10783: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:00 smithi022 ceph-mon[31360]: pgmap v10783: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:00 smithi022 ceph-mon[40250]: pgmap v10783: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:00.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:00.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:00.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2772813123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2772813123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:01.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:58:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:58:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2772813123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:02 smithi022 ceph-mon[31360]: pgmap v10784: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:02 smithi022 ceph-mon[40250]: pgmap v10784: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:02 smithi122 ceph-mon[32255]: pgmap v10784: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:58:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:58:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:04 smithi022 ceph-mon[31360]: pgmap v10785: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:04 smithi022 ceph-mon[40250]: pgmap v10785: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:04 smithi122 ceph-mon[32255]: pgmap v10785: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:05.719 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:06.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:06.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:06.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:06 smithi022 ceph-mon[31360]: pgmap v10786: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/692548058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:06 smithi022 ceph-mon[40250]: pgmap v10786: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/692548058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:06 smithi122 ceph-mon[32255]: pgmap v10786: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/692548058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:58:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:08 smithi022 ceph-mon[31360]: pgmap v10787: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:08 smithi022 ceph-mon[40250]: pgmap v10787: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:08 smithi122 ceph-mon[32255]: pgmap v10787: 132 pgs: 132 active+clean; 582 KiB data, 143 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-15T23:58:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:09 smithi022 ceph-mon[31360]: pgmap v10788: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:58:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:09 smithi022 ceph-mon[40250]: pgmap v10788: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:58:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:09 smithi122 ceph-mon[32255]: pgmap v10788: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-15T23:58:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:10] "GET /metrics HTTP/1.1" 200 202387 "" "Prometheus/2.18.1" 2021-12-15T23:58:11.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:11.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:11.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:11.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:58:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:58:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:58:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:58:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:12 smithi122 ceph-mon[32255]: pgmap v10789: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3122083251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:12 smithi022 ceph-mon[31360]: pgmap v10789: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3122083251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:12 smithi022 ceph-mon[40250]: pgmap v10789: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3122083251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:14 smithi122 ceph-mon[32255]: pgmap v10790: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:14 smithi022 ceph-mon[31360]: pgmap v10790: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:14 smithi022 ceph-mon[40250]: pgmap v10790: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:16 smithi122 ceph-mon[32255]: pgmap v10791: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:58:16.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:16.461 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:16 smithi022 ceph-mon[31360]: pgmap v10791: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:58:16.461 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:16 smithi022 ceph-mon[40250]: pgmap v10791: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:58:16.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:16.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:16.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:17.133 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/934775507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/934775507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:17.513 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/934775507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:58:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:18 smithi122 ceph-mon[32255]: pgmap v10792: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:18 smithi022 ceph-mon[31360]: pgmap v10792: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:18 smithi022 ceph-mon[40250]: pgmap v10792: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-15T23:58:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:20] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:58:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:20 smithi122 ceph-mon[32255]: pgmap v10793: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:58:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:20 smithi022 ceph-mon[31360]: pgmap v10793: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:58:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:20 smithi022 ceph-mon[40250]: pgmap v10793: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-15T23:58:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:58:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:58:21.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:22.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:22.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:22.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:22 smithi122 ceph-mon[32255]: pgmap v10794: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:58:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:58:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:22 smithi022 ceph-mon[31360]: pgmap v10794: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:22 smithi022 ceph-mon[40250]: pgmap v10794: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2287231894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2287231894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2287231894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:24 smithi122 ceph-mon[32255]: pgmap v10795: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:24 smithi022 ceph-mon[31360]: pgmap v10795: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:24 smithi022 ceph-mon[40250]: pgmap v10795: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:26 smithi122 ceph-mon[32255]: pgmap v10796: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:26 smithi022 ceph-mon[31360]: pgmap v10796: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:26 smithi022 ceph-mon[40250]: pgmap v10796: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:27.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:58:27.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:27.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:27.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:28 smithi122 ceph-mon[32255]: pgmap v10797: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1803698693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:28 smithi022 ceph-mon[31360]: pgmap v10797: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1803698693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:28 smithi022 ceph-mon[40250]: pgmap v10797: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1803698693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:30] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:58:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:30 smithi122 ceph-mon[32255]: pgmap v10798: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:30 smithi022 ceph-mon[31360]: pgmap v10798: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:30 smithi022 ceph-mon[40250]: pgmap v10798: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:58:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:58:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:32 smithi122 ceph-mon[32255]: pgmap v10799: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:58:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:58:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:32 smithi022 ceph-mon[31360]: pgmap v10799: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:32 smithi022 ceph-mon[40250]: pgmap v10799: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:32.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:32.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:32.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2894090657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2894090657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2894090657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:34 smithi122 ceph-mon[32255]: pgmap v10800: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:58:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:58:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:58:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:58:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[31360]: pgmap v10800: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:58:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:58:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:58:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:58:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[40250]: pgmap v10800: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:58:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:58:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:58:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:58:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:36 smithi122 ceph-mon[32255]: pgmap v10801: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:36 smithi022 ceph-mon[31360]: pgmap v10801: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:36 smithi022 ceph-mon[40250]: pgmap v10801: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:58:37.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:38 smithi022 ceph-mon[31360]: pgmap v10802: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:38 smithi022 ceph-mon[40250]: pgmap v10802: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:38.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:38.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:38.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:38 smithi122 ceph-mon[32255]: pgmap v10802: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/838832013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/838832013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/838832013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:40] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-15T23:58:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:40 smithi122 ceph-mon[32255]: pgmap v10803: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:40 smithi022 ceph-mon[31360]: pgmap v10803: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:40 smithi022 ceph-mon[40250]: pgmap v10803: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:58:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:58:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:42 smithi122 ceph-mon[32255]: pgmap v10804: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:58:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:58:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:42 smithi022 ceph-mon[31360]: pgmap v10804: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:42 smithi022 ceph-mon[40250]: pgmap v10804: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:43.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:43.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:43.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:44 smithi122 ceph-mon[32255]: pgmap v10805: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/302903596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:44 smithi022 ceph-mon[31360]: pgmap v10805: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/302903596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:44 smithi022 ceph-mon[40250]: pgmap v10805: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/302903596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:46 smithi022 ceph-mon[31360]: pgmap v10806: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:46 smithi022 ceph-mon[40250]: pgmap v10806: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:46 smithi122 ceph-mon[32255]: pgmap v10806: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:58:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:48 smithi022 ceph-mon[31360]: pgmap v10807: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:48 smithi022 ceph-mon[40250]: pgmap v10807: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:48.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:48 smithi122 ceph-mon[32255]: pgmap v10807: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:49.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:49.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:49.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4077407627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4077407627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4077407627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:50] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:58:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:50 smithi122 ceph-mon[32255]: pgmap v10808: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:50 smithi022 ceph-mon[31360]: pgmap v10808: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:50 smithi022 ceph-mon[40250]: pgmap v10808: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:58:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:58:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:52 smithi022 ceph-mon[31360]: pgmap v10809: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:52 smithi022 ceph-mon[40250]: pgmap v10809: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:52 smithi122 ceph-mon[32255]: pgmap v10809: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:58:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:58:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:58:54.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:58:54.421 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:54 smithi022 ceph-mon[31360]: pgmap v10810: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:54.422 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:54 smithi022 ceph-mon[40250]: pgmap v10810: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:54 smithi122 ceph-mon[32255]: pgmap v10810: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3721113284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:58:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:58:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:58:55.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3721113284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:55.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:58:55.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:58:55.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:58:55.478 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3721113284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:58:55.478 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:58:55.478 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:58:55.478 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:58:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:56 smithi022 ceph-mon[31360]: pgmap v10811: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:58:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:56 smithi022 ceph-mon[40250]: pgmap v10811: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:58:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:56 smithi122 ceph-mon[32255]: pgmap v10811: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:58:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:58:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:58:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:58:58 smithi022 ceph-mon[31360]: pgmap v10812: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:58:58 smithi022 ceph-mon[40250]: pgmap v10812: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:58:58 smithi122 ceph-mon[32255]: pgmap v10812: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:58:59.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:58:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:58:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:58:59.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:00] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-15T23:59:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:00 smithi122 ceph-mon[32255]: pgmap v10813: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2355861061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:00 smithi022 ceph-mon[31360]: pgmap v10813: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2355861061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:00 smithi022 ceph-mon[40250]: pgmap v10813: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2355861061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:59:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:59:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:02 smithi022 ceph-mon[31360]: pgmap v10814: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:02 smithi022 ceph-mon[40250]: pgmap v10814: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:59:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:59:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:02 smithi122 ceph-mon[32255]: pgmap v10814: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:04 smithi022 ceph-mon[31360]: pgmap v10815: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:04 smithi022 ceph-mon[40250]: pgmap v10815: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:04 smithi122 ceph-mon[32255]: pgmap v10815: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:04.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:05.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:05.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:05.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:06 smithi122 ceph-mon[32255]: pgmap v10816: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1935455734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:06 smithi022 ceph-mon[31360]: pgmap v10816: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1935455734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:06 smithi022 ceph-mon[40250]: pgmap v10816: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1935455734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:59:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:08 smithi122 ceph-mon[32255]: pgmap v10817: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:08 smithi022 ceph-mon[31360]: pgmap v10817: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:08 smithi022 ceph-mon[40250]: pgmap v10817: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:10.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:10] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-15T23:59:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:10 smithi122 ceph-mon[32255]: pgmap v10818: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:10.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:10.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:10.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:10 smithi022 ceph-mon[40250]: pgmap v10818: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:10 smithi022 ceph-mon[31360]: pgmap v10818: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2317378877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:59:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:59:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2317378877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2317378877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:12 smithi122 ceph-mon[32255]: pgmap v10819: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:59:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:59:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:12 smithi022 ceph-mon[31360]: pgmap v10819: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:12 smithi022 ceph-mon[40250]: pgmap v10819: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:14 smithi122 ceph-mon[32255]: pgmap v10820: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:14 smithi022 ceph-mon[31360]: pgmap v10820: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:14 smithi022 ceph-mon[40250]: pgmap v10820: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:15.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:15.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:15.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:15.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:16 smithi122 ceph-mon[32255]: pgmap v10821: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1572017059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:16 smithi022 ceph-mon[31360]: pgmap v10821: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1572017059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:16 smithi022 ceph-mon[40250]: pgmap v10821: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1572017059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:59:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:18 smithi122 ceph-mon[32255]: pgmap v10822: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:18 smithi022 ceph-mon[31360]: pgmap v10822: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:18 smithi022 ceph-mon[40250]: pgmap v10822: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:20] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:59:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:20 smithi122 ceph-mon[32255]: pgmap v10823: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:20 smithi022 ceph-mon[31360]: pgmap v10823: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:20 smithi022 ceph-mon[40250]: pgmap v10823: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:20.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:21.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:21.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:21.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:59:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:59:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:22 smithi122 ceph-mon[32255]: pgmap v10824: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3459627430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:59:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:59:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:22 smithi022 ceph-mon[31360]: pgmap v10824: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3459627430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:22 smithi022 ceph-mon[40250]: pgmap v10824: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3459627430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:24 smithi122 ceph-mon[32255]: pgmap v10825: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:24 smithi022 ceph-mon[40250]: pgmap v10825: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:24 smithi022 ceph-mon[31360]: pgmap v10825: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:26.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:26 smithi122 ceph-mon[32255]: pgmap v10826: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:26 smithi022 ceph-mon[31360]: pgmap v10826: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:26 smithi022 ceph-mon[40250]: pgmap v10826: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:26.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:26.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2646663977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2646663977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:59:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2646663977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:28 smithi122 ceph-mon[32255]: pgmap v10827: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:28 smithi022 ceph-mon[31360]: pgmap v10827: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:28 smithi022 ceph-mon[40250]: pgmap v10827: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:30 smithi122 ceph-mon[32255]: pgmap v10828: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:30] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:59:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:30 smithi022 ceph-mon[31360]: pgmap v10828: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:30 smithi022 ceph-mon[40250]: pgmap v10828: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:59:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:59:32.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:32.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:32.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:32 smithi022 ceph-mon[31360]: pgmap v10829: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3686097018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:32 smithi022 ceph-mon[40250]: pgmap v10829: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3686097018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:32 smithi122 ceph-mon[32255]: pgmap v10829: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3686097018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:59:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:59:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[31360]: pgmap v10830: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:59:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:59:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:59:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:59:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[40250]: pgmap v10830: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:59:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:59:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:59:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:59:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:34 smithi122 ceph-mon[32255]: pgmap v10830: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:59:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:59:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-15T23:59:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-15T23:59:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:36 smithi022 ceph-mon[31360]: pgmap v10831: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:36 smithi022 ceph-mon[40250]: pgmap v10831: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:36 smithi122 ceph-mon[32255]: pgmap v10831: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:37.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:37.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:37.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:37.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:59:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:38 smithi022 ceph-mon[31360]: pgmap v10832: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1749835372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:38 smithi022 ceph-mon[40250]: pgmap v10832: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1749835372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:38 smithi122 ceph-mon[32255]: pgmap v10832: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1749835372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:40] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-15T23:59:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:40 smithi122 ceph-mon[32255]: pgmap v10833: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:40 smithi022 ceph-mon[31360]: pgmap v10833: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:40 smithi022 ceph-mon[40250]: pgmap v10833: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:41.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:59:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:59:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:42 smithi022 ceph-mon[31360]: pgmap v10834: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:42 smithi022 ceph-mon[40250]: pgmap v10834: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:42 smithi122 ceph-mon[32255]: pgmap v10834: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:59:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:59:42.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:42.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:42.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3343211057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3343211057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3343211057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:44 smithi022 ceph-mon[31360]: pgmap v10835: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:44 smithi022 ceph-mon[40250]: pgmap v10835: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:44 smithi122 ceph-mon[32255]: pgmap v10835: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:46 smithi022 ceph-mon[31360]: pgmap v10836: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:46 smithi022 ceph-mon[40250]: pgmap v10836: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:46 smithi122 ceph-mon[32255]: pgmap v10836: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:47.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:47.750 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:59:48.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:48.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:48.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:48 smithi022 ceph-mon[31360]: pgmap v10837: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/920400153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:48 smithi022 ceph-mon[40250]: pgmap v10837: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/920400153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:48 smithi122 ceph-mon[32255]: pgmap v10837: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/920400153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 15 23:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:50] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-15T23:59:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:50 smithi122 ceph-mon[32255]: pgmap v10838: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:50 smithi022 ceph-mon[31360]: pgmap v10838: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:50 smithi022 ceph-mon[40250]: pgmap v10838: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 15 23:59:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-15T23:59:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:52 smithi022 ceph-mon[31360]: pgmap v10839: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:52 smithi022 ceph-mon[40250]: pgmap v10839: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:52 smithi122 ceph-mon[32255]: pgmap v10839: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 15 23:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-15T23:59:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-15T23:59:53.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:53.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:53.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:53.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:53.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:53.491 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-15T23:59:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:54 smithi022 ceph-mon[31360]: pgmap v10840: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/968087587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:54 smithi022 ceph-mon[40250]: pgmap v10840: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/968087587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:54 smithi122 ceph-mon[32255]: pgmap v10840: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:54.722 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/968087587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:56.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[31360]: pgmap v10841: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:56.229 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:59:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:59:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:59:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:59:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[40250]: pgmap v10841: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:59:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:59:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:59:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:59:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:56 smithi122 ceph-mon[32255]: pgmap v10841: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-15T23:59:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-15T23:59:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-15T23:59:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-15T23:59:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-15T23:59:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 15 23:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [15/Dec/2021:23:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-15T23:59:58.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-15T23:59:58.492 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:58 smithi022 ceph-mon[40250]: pgmap v10842: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:58.493 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:58 smithi022 ceph-mon[31360]: pgmap v10842: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:58 smithi122 ceph-mon[32255]: pgmap v10842: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-15T23:59:58.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-15T23:59:58.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-15T23:59:58.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-15T23:59:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 15 23:59:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3780212499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 15 23:59:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3780212499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-15T23:59:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 15 23:59:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3780212499' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:00] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:00:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:00 smithi122 ceph-mon[32255]: pgmap v10843: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T00:00:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:00 smithi022 ceph-mon[31360]: pgmap v10843: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T00:00:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:00 smithi022 ceph-mon[40250]: pgmap v10843: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T00:00:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:01 smithi022 ceph-mon[31360]: pgmap v10844: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:01 smithi022 ceph-mon[40250]: pgmap v10844: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:01.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:00:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:00:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:01 smithi122 ceph-mon[32255]: pgmap v10844: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:00:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:00:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:03 smithi022 ceph-mon[40250]: pgmap v10845: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:03 smithi022 ceph-mon[31360]: pgmap v10845: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:03 smithi122 ceph-mon[32255]: pgmap v10845: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:03.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:04.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:04.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:04.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1328498787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1328498787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1328498787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:06 smithi122 ceph-mon[32255]: pgmap v10846: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T00:00:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:06 smithi022 ceph-mon[31360]: pgmap v10846: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T00:00:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:06 smithi022 ceph-mon[40250]: pgmap v10846: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T00:00:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:00:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:08 smithi122 ceph-mon[32255]: pgmap v10847: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T00:00:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:08 smithi022 ceph-mon[31360]: pgmap v10847: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T00:00:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:08 smithi022 ceph-mon[40250]: pgmap v10847: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T00:00:09.229 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:09.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:09.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:09.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:10] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-16T00:00:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:10 smithi122 ceph-mon[32255]: pgmap v10848: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-16T00:00:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1264024768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:10 smithi022 ceph-mon[31360]: pgmap v10848: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-16T00:00:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1264024768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:10 smithi022 ceph-mon[40250]: pgmap v10848: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-16T00:00:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1264024768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:00:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:00:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:12 smithi122 ceph-mon[32255]: pgmap v10849: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2021-12-16T00:00:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:00:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:00:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:12 smithi022 ceph-mon[31360]: pgmap v10849: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2021-12-16T00:00:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:12 smithi022 ceph-mon[40250]: pgmap v10849: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2021-12-16T00:00:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:14 smithi122 ceph-mon[32255]: pgmap v10850: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2021-12-16T00:00:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:14 smithi022 ceph-mon[31360]: pgmap v10850: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2021-12-16T00:00:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:14 smithi022 ceph-mon[40250]: pgmap v10850: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2021-12-16T00:00:14.596 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:14.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:14.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:14.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/207862798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/207862798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/207862798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:16 smithi122 ceph-mon[32255]: pgmap v10851: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-16T00:00:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:16 smithi022 ceph-mon[31360]: pgmap v10851: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-16T00:00:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:16 smithi022 ceph-mon[40250]: pgmap v10851: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2021-12-16T00:00:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:00:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:18 smithi122 ceph-mon[32255]: pgmap v10852: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:00:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:18 smithi022 ceph-mon[31360]: pgmap v10852: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:00:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:18 smithi022 ceph-mon[40250]: pgmap v10852: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:00:19.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:20 smithi022 ceph-mon[31360]: pgmap v10853: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:00:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:20 smithi022 ceph-mon[40250]: pgmap v10853: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:00:20.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:20.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:20.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:20] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-16T00:00:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:20 smithi122 ceph-mon[32255]: pgmap v10853: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:00:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4127334952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:00:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:00:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4127334952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4127334952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:22 smithi122 ceph-mon[32255]: pgmap v10854: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:00:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:00:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:22 smithi022 ceph-mon[31360]: pgmap v10854: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:22 smithi022 ceph-mon[40250]: pgmap v10854: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:24 smithi122 ceph-mon[32255]: pgmap v10855: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:24 smithi022 ceph-mon[31360]: pgmap v10855: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:24 smithi022 ceph-mon[40250]: pgmap v10855: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:25.341 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:25.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:25.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:25.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:26 smithi122 ceph-mon[32255]: pgmap v10856: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3398937824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:26 smithi022 ceph-mon[31360]: pgmap v10856: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3398937824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:26 smithi022 ceph-mon[40250]: pgmap v10856: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3398937824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:00:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:28 smithi022 ceph-mon[31360]: pgmap v10857: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:28 smithi022 ceph-mon[40250]: pgmap v10857: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:28 smithi122 ceph-mon[32255]: pgmap v10857: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:30] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-16T00:00:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:30 smithi122 ceph-mon[32255]: pgmap v10858: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:30 smithi022 ceph-mon[31360]: pgmap v10858: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:30 smithi022 ceph-mon[40250]: pgmap v10858: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:30.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:31.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:31.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:31.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3302458264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3302458264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3302458264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:31.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:00:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:00:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:32 smithi022 ceph-mon[31360]: pgmap v10859: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:32 smithi022 ceph-mon[40250]: pgmap v10859: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:32 smithi122 ceph-mon[32255]: pgmap v10859: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:00:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:00:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[31360]: pgmap v10860: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:00:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:00:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:00:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:00:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[40250]: pgmap v10860: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:00:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:00:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:00:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:00:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:34 smithi122 ceph-mon[32255]: pgmap v10860: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:00:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:00:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:00:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:00:36.082 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:36.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:36.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:36.456 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:36 smithi022 ceph-mon[31360]: pgmap v10861: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:36.457 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:36 smithi022 ceph-mon[40250]: pgmap v10861: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:36 smithi122 ceph-mon[32255]: pgmap v10861: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4112042029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:00:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4112042029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4112042029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:38 smithi022 ceph-mon[31360]: pgmap v10862: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:38 smithi022 ceph-mon[40250]: pgmap v10862: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:38 smithi122 ceph-mon[32255]: pgmap v10862: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:40] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:00:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:40 smithi122 ceph-mon[32255]: pgmap v10863: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:40 smithi022 ceph-mon[31360]: pgmap v10863: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:40 smithi022 ceph-mon[40250]: pgmap v10863: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:41.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:00:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:00:41.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:41.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:41.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:42 smithi022 ceph-mon[31360]: pgmap v10864: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/516646585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:42 smithi022 ceph-mon[40250]: pgmap v10864: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/516646585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:00:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:00:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:42 smithi122 ceph-mon[32255]: pgmap v10864: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/516646585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:44 smithi022 ceph-mon[31360]: pgmap v10865: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:44 smithi022 ceph-mon[40250]: pgmap v10865: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:44 smithi122 ceph-mon[32255]: pgmap v10865: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:46 smithi022 ceph-mon[31360]: pgmap v10866: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:46 smithi022 ceph-mon[40250]: pgmap v10866: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:46 smithi122 ceph-mon[32255]: pgmap v10866: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:46.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:47.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/109306434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:00:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/109306434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/109306434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:48 smithi022 ceph-mon[31360]: pgmap v10867: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:48 smithi022 ceph-mon[40250]: pgmap v10867: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:48 smithi122 ceph-mon[32255]: pgmap v10867: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:00:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:50] "GET /metrics HTTP/1.1" 200 202403 "" "Prometheus/2.18.1" 2021-12-16T00:00:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:50 smithi122 ceph-mon[32255]: pgmap v10868: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:50 smithi022 ceph-mon[31360]: pgmap v10868: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:50 smithi022 ceph-mon[40250]: pgmap v10868: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:00:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:00:52.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:52 smithi022 ceph-mon[31360]: pgmap v10869: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:52 smithi022 ceph-mon[40250]: pgmap v10869: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:52 smithi122 ceph-mon[32255]: pgmap v10869: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:00:52.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:00:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2148367664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2148367664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:00:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2148367664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:54 smithi022 ceph-mon[31360]: pgmap v10870: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:54 smithi022 ceph-mon[40250]: pgmap v10870: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:54 smithi122 ceph-mon[32255]: pgmap v10870: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[31360]: pgmap v10871: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:00:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:00:56.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:00:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[40250]: pgmap v10871: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:56.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:00:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:00:56.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:00:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:56 smithi122 ceph-mon[32255]: pgmap v10871: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:00:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:00:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:00:57.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:00:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:57 smithi122 ceph-mon[32255]: pgmap v10872: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:00:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[31360]: pgmap v10872: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[40250]: pgmap v10872: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:00:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:00:57.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:00:57.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:00:57.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:00:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2046308255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2046308255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2046308255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:00:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:00:59 smithi122 ceph-mon[32255]: pgmap v10873: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:00:59 smithi022 ceph-mon[31360]: pgmap v10873: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:00:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:00:59 smithi022 ceph-mon[40250]: pgmap v10873: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:00] "GET /metrics HTTP/1.1" 200 202403 "" "Prometheus/2.18.1" 2021-12-16T00:01:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:01:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:01:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:02 smithi122 ceph-mon[32255]: pgmap v10874: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:01:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:01:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:02 smithi022 ceph-mon[31360]: pgmap v10874: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:02 smithi022 ceph-mon[40250]: pgmap v10874: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:02.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:03.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:03.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:03.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:04 smithi122 ceph-mon[32255]: pgmap v10875: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3988725608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:04 smithi022 ceph-mon[31360]: pgmap v10875: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3988725608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:04 smithi022 ceph-mon[40250]: pgmap v10875: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:04.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3988725608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:06 smithi122 ceph-mon[32255]: pgmap v10876: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:06 smithi022 ceph-mon[31360]: pgmap v10876: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:06 smithi022 ceph-mon[40250]: pgmap v10876: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:01:08.321 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:08 smithi122 ceph-mon[32255]: pgmap v10877: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:08 smithi022 ceph-mon[31360]: pgmap v10877: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:08 smithi022 ceph-mon[40250]: pgmap v10877: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:08.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:08.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:08.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/780460912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/780460912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/780460912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:10 smithi122 ceph-mon[32255]: pgmap v10878: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:10] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-16T00:01:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:10 smithi022 ceph-mon[31360]: pgmap v10878: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:10 smithi022 ceph-mon[40250]: pgmap v10878: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:01:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:01:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:12 smithi122 ceph-mon[32255]: pgmap v10879: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:01:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:01:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:12 smithi022 ceph-mon[31360]: pgmap v10879: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:12 smithi022 ceph-mon[40250]: pgmap v10879: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:13.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:14.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:14.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:14.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:14 smithi122 ceph-mon[32255]: pgmap v10880: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/296155762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:14 smithi022 ceph-mon[31360]: pgmap v10880: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/296155762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:14 smithi022 ceph-mon[40250]: pgmap v10880: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/296155762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:16 smithi022 ceph-mon[31360]: pgmap v10881: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:16 smithi022 ceph-mon[40250]: pgmap v10881: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:16 smithi122 ceph-mon[32255]: pgmap v10881: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:01:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:18 smithi022 ceph-mon[31360]: pgmap v10882: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:18 smithi022 ceph-mon[40250]: pgmap v10882: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:18 smithi122 ceph-mon[32255]: pgmap v10882: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:19.063 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:19.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:19.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:19.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:20] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-16T00:01:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:20 smithi122 ceph-mon[32255]: pgmap v10883: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/51157957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:20 smithi022 ceph-mon[31360]: pgmap v10883: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/51157957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:20 smithi022 ceph-mon[40250]: pgmap v10883: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/51157957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:01:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:01:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:22 smithi022 ceph-mon[31360]: pgmap v10884: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:22 smithi022 ceph-mon[40250]: pgmap v10884: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:22 smithi122 ceph-mon[32255]: pgmap v10884: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:01:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:01:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:24.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:24 smithi022 ceph-mon[31360]: pgmap v10885: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:24 smithi022 ceph-mon[40250]: pgmap v10885: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:24 smithi122 ceph-mon[32255]: pgmap v10885: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:24.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:24.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:24.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2844239724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2844239724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2844239724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:26 smithi022 ceph-mon[31360]: pgmap v10886: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:26 smithi022 ceph-mon[40250]: pgmap v10886: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:26 smithi122 ceph-mon[32255]: pgmap v10886: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:01:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:28 smithi022 ceph-mon[31360]: pgmap v10887: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:28 smithi022 ceph-mon[40250]: pgmap v10887: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:28 smithi122 ceph-mon[32255]: pgmap v10887: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:30.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:30.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:30.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:30] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-16T00:01:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:30 smithi122 ceph-mon[32255]: pgmap v10888: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1344160220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:30 smithi022 ceph-mon[31360]: pgmap v10888: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1344160220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:30 smithi022 ceph-mon[40250]: pgmap v10888: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1344160220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:01:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:01:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:32 smithi022 ceph-mon[31360]: pgmap v10889: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:32 smithi022 ceph-mon[40250]: pgmap v10889: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:32 smithi122 ceph-mon[32255]: pgmap v10889: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:01:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:01:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[31360]: pgmap v10890: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:01:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:01:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:01:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:01:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[40250]: pgmap v10890: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:01:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:01:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:01:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:01:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:34 smithi122 ceph-mon[32255]: pgmap v10890: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:01:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:01:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:01:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:01:35.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:35.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:35.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:35.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:36 smithi022 ceph-mon[31360]: pgmap v10891: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/160597572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:36 smithi022 ceph-mon[40250]: pgmap v10891: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/160597572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:36 smithi122 ceph-mon[32255]: pgmap v10891: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/160597572' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:01:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:38 smithi022 ceph-mon[31360]: pgmap v10892: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:38 smithi022 ceph-mon[40250]: pgmap v10892: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:38 smithi122 ceph-mon[32255]: pgmap v10892: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:40] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-16T00:01:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:40 smithi122 ceph-mon[32255]: pgmap v10893: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:40 smithi022 ceph-mon[31360]: pgmap v10893: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:40 smithi022 ceph-mon[40250]: pgmap v10893: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:40.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:40.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:40.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:40.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1179293134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1179293134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1179293134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:41.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:01:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:01:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:42 smithi022 ceph-mon[31360]: pgmap v10894: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:42.563 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:42 smithi022 ceph-mon[40250]: pgmap v10894: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:42 smithi122 ceph-mon[32255]: pgmap v10894: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:01:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:01:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:44 smithi022 ceph-mon[31360]: pgmap v10895: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:44 smithi022 ceph-mon[40250]: pgmap v10895: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:44 smithi122 ceph-mon[32255]: pgmap v10895: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:45.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:46.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:46.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:46.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:46 smithi022 ceph-mon[31360]: pgmap v10896: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:46 smithi022 ceph-mon[40250]: pgmap v10896: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:46 smithi122 ceph-mon[32255]: pgmap v10896: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3844462607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:01:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3844462607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3844462607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:48 smithi022 ceph-mon[31360]: pgmap v10897: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:48 smithi022 ceph-mon[40250]: pgmap v10897: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:48 smithi122 ceph-mon[32255]: pgmap v10897: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:50 smithi122 ceph-mon[32255]: pgmap v10898: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:50] "GET /metrics HTTP/1.1" 200 202400 "" "Prometheus/2.18.1" 2021-12-16T00:01:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:50 smithi022 ceph-mon[31360]: pgmap v10898: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:50 smithi022 ceph-mon[40250]: pgmap v10898: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:51.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:51.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:51.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:51.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:01:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:01:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:52 smithi022 ceph-mon[31360]: pgmap v10899: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/942033409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:52 smithi022 ceph-mon[40250]: pgmap v10899: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/942033409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:01:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:01:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:52 smithi122 ceph-mon[32255]: pgmap v10899: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/942033409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:53 smithi022 ceph-mon[31360]: pgmap v10900: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:53 smithi022 ceph-mon[40250]: pgmap v10900: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:01:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:53 smithi122 ceph-mon[32255]: pgmap v10900: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:56 smithi122 ceph-mon[32255]: pgmap v10901: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:56 smithi022 ceph-mon[31360]: pgmap v10901: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:56 smithi022 ceph-mon[40250]: pgmap v10901: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:01:56.661 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:01:57.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:01:57.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:01:57.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:01:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:01:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:01:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:01:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/208837531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:01:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:01:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:01:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/208837531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:01:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:01:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:01:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/208837531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:01:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:01:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:58 smithi122 ceph-mon[32255]: pgmap v10902: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:01:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:01:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:58 smithi022 ceph-mon[31360]: pgmap v10902: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:01:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:01:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:58 smithi022 ceph-mon[40250]: pgmap v10902: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:01:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:01:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:02:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:00] "GET /metrics HTTP/1.1" 200 202400 "" "Prometheus/2.18.1" 2021-12-16T00:02:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:00 smithi122 ceph-mon[32255]: pgmap v10903: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:00 smithi022 ceph-mon[31360]: pgmap v10903: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:00 smithi022 ceph-mon[40250]: pgmap v10903: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:02:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:02:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:02.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:02.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:02.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:02.408 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:02 smithi022 ceph-mon[31360]: pgmap v10904: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:02.409 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:02 smithi022 ceph-mon[40250]: pgmap v10904: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:02 smithi122 ceph-mon[32255]: pgmap v10904: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:02:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:02:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1919799549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1919799549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1919799549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:04 smithi022 ceph-mon[31360]: pgmap v10905: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:04 smithi022 ceph-mon[40250]: pgmap v10905: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:04 smithi122 ceph-mon[32255]: pgmap v10905: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:06 smithi022 ceph-mon[31360]: pgmap v10906: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:06 smithi022 ceph-mon[40250]: pgmap v10906: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:06 smithi122 ceph-mon[32255]: pgmap v10906: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:07.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:07.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:07.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:07.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:07.887 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:02:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:08 smithi022 ceph-mon[31360]: pgmap v10907: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2370094689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:08 smithi022 ceph-mon[40250]: pgmap v10907: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2370094689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:08 smithi122 ceph-mon[32255]: pgmap v10907: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2370094689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:10] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-16T00:02:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:10 smithi122 ceph-mon[32255]: pgmap v10908: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:10 smithi022 ceph-mon[31360]: pgmap v10908: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:10 smithi022 ceph-mon[40250]: pgmap v10908: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:02:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:02:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:12 smithi022 ceph-mon[31360]: pgmap v10909: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:12 smithi022 ceph-mon[40250]: pgmap v10909: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:02:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:02:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:12 smithi122 ceph-mon[32255]: pgmap v10909: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:12.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:13.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:13.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:13.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3245908553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3245908553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3245908553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:14 smithi022 ceph-mon[31360]: pgmap v10910: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:14 smithi022 ceph-mon[40250]: pgmap v10910: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:14 smithi122 ceph-mon[32255]: pgmap v10910: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:16 smithi022 ceph-mon[31360]: pgmap v10911: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:16 smithi022 ceph-mon[40250]: pgmap v10911: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:16 smithi122 ceph-mon[32255]: pgmap v10911: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:02:18.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:18.521 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:18 smithi022 ceph-mon[31360]: pgmap v10912: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:18.521 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:18 smithi022 ceph-mon[40250]: pgmap v10912: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:18 smithi122 ceph-mon[32255]: pgmap v10912: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2501023119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2501023119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2501023119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:20] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:02:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:20 smithi122 ceph-mon[32255]: pgmap v10913: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:20 smithi022 ceph-mon[31360]: pgmap v10913: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:20 smithi022 ceph-mon[40250]: pgmap v10913: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:02:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:02:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:22 smithi022 ceph-mon[31360]: pgmap v10914: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:22 smithi022 ceph-mon[40250]: pgmap v10914: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:22 smithi122 ceph-mon[32255]: pgmap v10914: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:02:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:02:23.522 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:23.522 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:23.523 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:23.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:23.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:23.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:24 smithi022 ceph-mon[31360]: pgmap v10915: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/662441145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:24 smithi022 ceph-mon[40250]: pgmap v10915: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/662441145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:24 smithi122 ceph-mon[32255]: pgmap v10915: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/662441145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:26 smithi022 ceph-mon[31360]: pgmap v10916: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:26 smithi022 ceph-mon[40250]: pgmap v10916: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:26 smithi122 ceph-mon[32255]: pgmap v10916: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:02:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:28 smithi022 ceph-mon[31360]: pgmap v10917: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:28 smithi022 ceph-mon[40250]: pgmap v10917: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:28 smithi122 ceph-mon[32255]: pgmap v10917: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:28.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:29.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:29.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:29.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:30] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:02:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:30 smithi122 ceph-mon[32255]: pgmap v10918: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/188170931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:30 smithi022 ceph-mon[40250]: pgmap v10918: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/188170931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:30 smithi022 ceph-mon[31360]: pgmap v10918: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/188170931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:02:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:02:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:32 smithi022 ceph-mon[31360]: pgmap v10919: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:32 smithi022 ceph-mon[40250]: pgmap v10919: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:32 smithi122 ceph-mon[32255]: pgmap v10919: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:02:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:02:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:34.257 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[31360]: pgmap v10920: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:02:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:02:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:02:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:02:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[40250]: pgmap v10920: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:02:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:02:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:02:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:02:34.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:34.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:34.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:34 smithi122 ceph-mon[32255]: pgmap v10920: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:02:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3785216000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3785216000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3785216000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:36 smithi022 ceph-mon[31360]: pgmap v10921: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:36 smithi022 ceph-mon[40250]: pgmap v10921: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:36.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:36 smithi122 ceph-mon[32255]: pgmap v10921: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:02:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:38 smithi022 ceph-mon[31360]: pgmap v10922: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:38 smithi022 ceph-mon[40250]: pgmap v10922: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:38 smithi122 ceph-mon[32255]: pgmap v10922: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:39.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:40.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:40.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:40.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:40 smithi122 ceph-mon[32255]: pgmap v10923: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/517073258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:40] "GET /metrics HTTP/1.1" 200 202400 "" "Prometheus/2.18.1" 2021-12-16T00:02:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:40 smithi022 ceph-mon[31360]: pgmap v10923: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/517073258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:40 smithi022 ceph-mon[40250]: pgmap v10923: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/517073258' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:02:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:02:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:42 smithi022 ceph-mon[31360]: pgmap v10924: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:42 smithi022 ceph-mon[40250]: pgmap v10924: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:42 smithi122 ceph-mon[32255]: pgmap v10924: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:02:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:02:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:44 smithi022 ceph-mon[31360]: pgmap v10925: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:44 smithi022 ceph-mon[40250]: pgmap v10925: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:44 smithi122 ceph-mon[32255]: pgmap v10925: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:45.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:45.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:45.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:45.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:46 smithi022 ceph-mon[31360]: pgmap v10926: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4007599511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:46 smithi022 ceph-mon[40250]: pgmap v10926: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4007599511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:46 smithi122 ceph-mon[32255]: pgmap v10926: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4007599511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:02:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:48 smithi022 ceph-mon[31360]: pgmap v10927: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:48 smithi022 ceph-mon[40250]: pgmap v10927: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:48 smithi122 ceph-mon[32255]: pgmap v10927: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:49 smithi022 ceph-mon[31360]: pgmap v10928: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:49 smithi022 ceph-mon[40250]: pgmap v10928: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:49 smithi122 ceph-mon[32255]: pgmap v10928: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:50.376 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:50] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-16T00:02:50.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:50.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:50.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1235477460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1235477460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1235477460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:51.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:02:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:02:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:52 smithi022 ceph-mon[31360]: pgmap v10929: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:52 smithi022 ceph-mon[40250]: pgmap v10929: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:52 smithi122 ceph-mon[32255]: pgmap v10929: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:02:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:02:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:02:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:54 smithi022 ceph-mon[31360]: pgmap v10930: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:54 smithi022 ceph-mon[40250]: pgmap v10930: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:54 smithi122 ceph-mon[32255]: pgmap v10930: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:55.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:02:56.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:02:56.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:02:56.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:02:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:56 smithi022 ceph-mon[31360]: pgmap v10931: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3920207975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:56 smithi022 ceph-mon[40250]: pgmap v10931: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3920207975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:56 smithi122 ceph-mon[32255]: pgmap v10931: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:02:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3920207975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:02:57.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:02:58.201 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[31360]: pgmap v10932: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:58.202 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:02:58.202 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:02:58.202 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:02:58.202 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:02:58.203 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[40250]: pgmap v10932: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:58.203 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:02:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:02:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:02:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:02:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:02:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:58 smithi122 ceph-mon[32255]: pgmap v10932: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:02:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:02:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:02:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:02:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:02:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:03:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:00] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-16T00:03:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:00 smithi122 ceph-mon[32255]: pgmap v10933: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:00 smithi022 ceph-mon[31360]: pgmap v10933: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:00 smithi022 ceph-mon[40250]: pgmap v10933: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:01.124 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:01.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:01.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:01.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:03:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:03:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:02 smithi022 ceph-mon[31360]: pgmap v10934: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/953663160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:02 smithi022 ceph-mon[40250]: pgmap v10934: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/953663160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:03:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:03:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:02 smithi122 ceph-mon[32255]: pgmap v10934: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/953663160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:04 smithi022 ceph-mon[31360]: pgmap v10935: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:04 smithi022 ceph-mon[40250]: pgmap v10935: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:04 smithi122 ceph-mon[32255]: pgmap v10935: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:06.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:06.496 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:06 smithi022 ceph-mon[31360]: pgmap v10936: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:06.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:06 smithi022 ceph-mon[40250]: pgmap v10936: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:06 smithi122 ceph-mon[32255]: pgmap v10936: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:06.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:06.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:06.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1671760281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:03:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1671760281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1671760281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:08 smithi022 ceph-mon[31360]: pgmap v10937: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:08 smithi022 ceph-mon[40250]: pgmap v10937: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:08 smithi122 ceph-mon[32255]: pgmap v10937: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:10 smithi122 ceph-mon[32255]: pgmap v10938: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:10] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-16T00:03:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:10 smithi022 ceph-mon[31360]: pgmap v10938: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:10 smithi022 ceph-mon[40250]: pgmap v10938: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:11.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:03:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:03:12.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:12.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:12.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:12 smithi022 ceph-mon[31360]: pgmap v10939: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:12 smithi022 ceph-mon[40250]: pgmap v10939: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:12 smithi122 ceph-mon[32255]: pgmap v10939: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:03:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:03:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3431116363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3431116363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3431116363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:14 smithi022 ceph-mon[31360]: pgmap v10940: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:14 smithi022 ceph-mon[40250]: pgmap v10940: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:14 smithi122 ceph-mon[32255]: pgmap v10940: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:16 smithi022 ceph-mon[31360]: pgmap v10941: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:16 smithi022 ceph-mon[40250]: pgmap v10941: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:16 smithi122 ceph-mon[32255]: pgmap v10941: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:17.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:03:17.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:17.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:17.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:18 smithi022 ceph-mon[31360]: pgmap v10942: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3595135670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:18 smithi022 ceph-mon[40250]: pgmap v10942: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3595135670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:18 smithi122 ceph-mon[32255]: pgmap v10942: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3595135670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:20] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-16T00:03:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:20 smithi122 ceph-mon[32255]: pgmap v10943: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:20 smithi022 ceph-mon[31360]: pgmap v10943: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:20 smithi022 ceph-mon[40250]: pgmap v10943: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:03:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:03:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:22 smithi022 ceph-mon[31360]: pgmap v10944: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:22 smithi022 ceph-mon[40250]: pgmap v10944: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:22.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:22 smithi122 ceph-mon[32255]: pgmap v10944: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:03:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:03:22.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:22.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:22.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:23.246 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:23.247 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/844472870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/844472870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/844472870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:24 smithi022 ceph-mon[31360]: pgmap v10945: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:24 smithi022 ceph-mon[40250]: pgmap v10945: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:24 smithi122 ceph-mon[32255]: pgmap v10945: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:26 smithi022 ceph-mon[31360]: pgmap v10946: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:26 smithi022 ceph-mon[40250]: pgmap v10946: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:26 smithi122 ceph-mon[32255]: pgmap v10946: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:03:27.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:28.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:28.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:28.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:28 smithi022 ceph-mon[31360]: pgmap v10947: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:28 smithi022 ceph-mon[40250]: pgmap v10947: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:28 smithi122 ceph-mon[32255]: pgmap v10947: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4087254494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4087254494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4087254494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:30 smithi122 ceph-mon[32255]: pgmap v10948: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:30] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-16T00:03:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:30 smithi022 ceph-mon[31360]: pgmap v10948: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:30 smithi022 ceph-mon[40250]: pgmap v10948: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:03:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:03:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:32 smithi022 ceph-mon[31360]: pgmap v10949: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:32 smithi022 ceph-mon[40250]: pgmap v10949: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:32 smithi122 ceph-mon[32255]: pgmap v10949: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:03:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:03:33.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:33.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:33.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:33.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[40250]: pgmap v10950: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:03:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:03:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:03:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:03:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3083463322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[31360]: pgmap v10950: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:03:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:03:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:03:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:03:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3083463322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:34 smithi122 ceph-mon[32255]: pgmap v10950: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:03:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:03:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:03:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:03:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3083463322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:36 smithi022 ceph-mon[31360]: pgmap v10951: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:36 smithi022 ceph-mon[40250]: pgmap v10951: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:36 smithi122 ceph-mon[32255]: pgmap v10951: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:03:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:38 smithi022 ceph-mon[31360]: pgmap v10952: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:38 smithi022 ceph-mon[40250]: pgmap v10952: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:38 smithi122 ceph-mon[32255]: pgmap v10952: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:38.722 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:39.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:39.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:39.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2790218472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2790218472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2790218472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:40 smithi122 ceph-mon[32255]: pgmap v10953: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:40] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-16T00:03:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:40 smithi022 ceph-mon[31360]: pgmap v10953: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:40 smithi022 ceph-mon[40250]: pgmap v10953: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:03:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:03:41.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:41 smithi122 ceph-mon[32255]: pgmap v10954: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:41 smithi022 ceph-mon[31360]: pgmap v10954: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:41 smithi022 ceph-mon[40250]: pgmap v10954: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:03:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:03:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:43 smithi122 ceph-mon[32255]: pgmap v10955: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:43 smithi022 ceph-mon[31360]: pgmap v10955: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:43 smithi022 ceph-mon[40250]: pgmap v10955: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:44.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:44.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:44.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:44.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/937208846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/937208846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/937208846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:46 smithi022 ceph-mon[31360]: pgmap v10956: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:46 smithi022 ceph-mon[40250]: pgmap v10956: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:46 smithi122 ceph-mon[32255]: pgmap v10956: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:03:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:48 smithi022 ceph-mon[31360]: pgmap v10957: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:48 smithi022 ceph-mon[40250]: pgmap v10957: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:48 smithi122 ceph-mon[32255]: pgmap v10957: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:49.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:49.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:49.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:50] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-16T00:03:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:50 smithi122 ceph-mon[32255]: pgmap v10958: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4225151437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:50 smithi022 ceph-mon[31360]: pgmap v10958: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4225151437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:50 smithi022 ceph-mon[40250]: pgmap v10958: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4225151437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:03:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:03:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:52 smithi022 ceph-mon[31360]: pgmap v10959: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:52 smithi022 ceph-mon[40250]: pgmap v10959: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:52 smithi122 ceph-mon[32255]: pgmap v10959: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:03:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:03:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:03:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:54 smithi022 ceph-mon[31360]: pgmap v10960: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:54 smithi022 ceph-mon[40250]: pgmap v10960: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:54 smithi122 ceph-mon[32255]: pgmap v10960: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:54.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:03:55.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:03:55.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:03:55.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:03:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2790222810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2790222810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2790222810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:03:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:56 smithi022 ceph-mon[31360]: pgmap v10961: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:56 smithi022 ceph-mon[40250]: pgmap v10961: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:56 smithi122 ceph-mon[32255]: pgmap v10961: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:03:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:03:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[31360]: pgmap v10962: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:03:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:03:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:03:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:03:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[40250]: pgmap v10962: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:03:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:03:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:03:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:03:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:03:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:58 smithi122 ceph-mon[32255]: pgmap v10962: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:03:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:03:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:03:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:03:58.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:03:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:04:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:00 smithi122 ceph-mon[32255]: pgmap v10963: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:00] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-16T00:04:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:00 smithi022 ceph-mon[31360]: pgmap v10963: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:00 smithi022 ceph-mon[40250]: pgmap v10963: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:00.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:00.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:00.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2685914075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2685914075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2685914075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:01.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:04:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:04:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:02 smithi022 ceph-mon[31360]: pgmap v10964: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:02 smithi022 ceph-mon[40250]: pgmap v10964: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:02 smithi122 ceph-mon[32255]: pgmap v10964: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:04:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:04:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:04 smithi022 ceph-mon[31360]: pgmap v10965: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:04 smithi022 ceph-mon[40250]: pgmap v10965: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:04 smithi122 ceph-mon[32255]: pgmap v10965: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:05.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:05.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:05.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:05.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:06 smithi022 ceph-mon[31360]: pgmap v10966: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/171336129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:06 smithi022 ceph-mon[40250]: pgmap v10966: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/171336129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:06 smithi122 ceph-mon[32255]: pgmap v10966: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/171336129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:04:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:08 smithi022 ceph-mon[31360]: pgmap v10967: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:08 smithi022 ceph-mon[40250]: pgmap v10967: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:08 smithi122 ceph-mon[32255]: pgmap v10967: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:10] "GET /metrics HTTP/1.1" 200 202403 "" "Prometheus/2.18.1" 2021-12-16T00:04:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:10 smithi122 ceph-mon[32255]: pgmap v10968: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:10 smithi022 ceph-mon[31360]: pgmap v10968: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:10 smithi022 ceph-mon[40250]: pgmap v10968: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:10.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:11.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:11.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:11.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:04:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:04:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:12 smithi022 ceph-mon[40250]: pgmap v10969: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1221654191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:12 smithi022 ceph-mon[31360]: pgmap v10969: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1221654191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:12 smithi122 ceph-mon[32255]: pgmap v10969: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1221654191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:04:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:04:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:14 smithi022 ceph-mon[31360]: pgmap v10970: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:14 smithi022 ceph-mon[40250]: pgmap v10970: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:14 smithi122 ceph-mon[32255]: pgmap v10970: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:16.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:16 smithi022 ceph-mon[31360]: pgmap v10971: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:16 smithi022 ceph-mon[40250]: pgmap v10971: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:16.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:16.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:16.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:16 smithi122 ceph-mon[32255]: pgmap v10971: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3979507303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:04:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3979507303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3979507303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:18 smithi022 ceph-mon[31360]: pgmap v10972: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:18 smithi022 ceph-mon[40250]: pgmap v10972: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:18 smithi122 ceph-mon[32255]: pgmap v10972: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:20 smithi122 ceph-mon[32255]: pgmap v10973: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:20] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-16T00:04:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:20 smithi022 ceph-mon[31360]: pgmap v10973: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:20 smithi022 ceph-mon[40250]: pgmap v10973: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:21.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:04:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:04:22.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:22.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:22.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:22 smithi022 ceph-mon[40250]: pgmap v10974: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2497521768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:22 smithi022 ceph-mon[31360]: pgmap v10974: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2497521768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:22 smithi122 ceph-mon[32255]: pgmap v10974: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2497521768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:04:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:04:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:24 smithi022 ceph-mon[40250]: pgmap v10975: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:24 smithi022 ceph-mon[31360]: pgmap v10975: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:24 smithi122 ceph-mon[32255]: pgmap v10975: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:26 smithi022 ceph-mon[31360]: pgmap v10976: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:26 smithi022 ceph-mon[40250]: pgmap v10976: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:26 smithi122 ceph-mon[32255]: pgmap v10976: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:27.053 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:27.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:27.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:27.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:04:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:28 smithi122 ceph-mon[32255]: pgmap v10977: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2289099465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:28 smithi022 ceph-mon[31360]: pgmap v10977: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2289099465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:28 smithi022 ceph-mon[40250]: pgmap v10977: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2289099465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:30] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-16T00:04:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:30 smithi122 ceph-mon[32255]: pgmap v10978: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:30 smithi022 ceph-mon[31360]: pgmap v10978: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:30 smithi022 ceph-mon[40250]: pgmap v10978: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:04:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:04:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:31 smithi122 ceph-mon[32255]: pgmap v10979: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:31 smithi022 ceph-mon[31360]: pgmap v10979: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:31 smithi022 ceph-mon[40250]: pgmap v10979: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:04:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:04:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:32.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:32.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:32.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:33 smithi122 ceph-mon[32255]: pgmap v10980: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1417096900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:33 smithi022 ceph-mon[31360]: pgmap v10980: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1417096900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:33 smithi022 ceph-mon[40250]: pgmap v10980: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1417096900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:04:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:04:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:04:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:04:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:04:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:04:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:04:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:04:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:04:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:04:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:04:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:04:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:36 smithi022 ceph-mon[31360]: pgmap v10981: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:36 smithi022 ceph-mon[40250]: pgmap v10981: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:36 smithi122 ceph-mon[32255]: pgmap v10981: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:04:37.793 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:38.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:38.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:38.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:38 smithi022 ceph-mon[31360]: pgmap v10982: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2375097869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:38 smithi022 ceph-mon[40250]: pgmap v10982: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2375097869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:38 smithi122 ceph-mon[32255]: pgmap v10982: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2375097869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:40] "GET /metrics HTTP/1.1" 200 202400 "" "Prometheus/2.18.1" 2021-12-16T00:04:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:40 smithi122 ceph-mon[32255]: pgmap v10983: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:40 smithi022 ceph-mon[31360]: pgmap v10983: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:40 smithi022 ceph-mon[40250]: pgmap v10983: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:41.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:04:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:04:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:42 smithi022 ceph-mon[31360]: pgmap v10984: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:42 smithi022 ceph-mon[40250]: pgmap v10984: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:42 smithi122 ceph-mon[32255]: pgmap v10984: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:04:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:04:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:43.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:43.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:43.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:44 smithi022 ceph-mon[31360]: pgmap v10985: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2116586096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:44 smithi022 ceph-mon[40250]: pgmap v10985: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2116586096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:44 smithi122 ceph-mon[32255]: pgmap v10985: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2116586096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:46 smithi022 ceph-mon[31360]: pgmap v10986: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:46 smithi022 ceph-mon[40250]: pgmap v10986: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:46 smithi122 ceph-mon[32255]: pgmap v10986: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:04:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:48 smithi022 ceph-mon[31360]: pgmap v10987: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:48 smithi022 ceph-mon[40250]: pgmap v10987: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:48 smithi122 ceph-mon[32255]: pgmap v10987: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:48.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:48.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:48.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:49.231 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2610437025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2610437025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2610437025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:50] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-16T00:04:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:50 smithi122 ceph-mon[32255]: pgmap v10988: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:50 smithi022 ceph-mon[31360]: pgmap v10988: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:50 smithi022 ceph-mon[40250]: pgmap v10988: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:04:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:04:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:52 smithi022 ceph-mon[31360]: pgmap v10989: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:52 smithi022 ceph-mon[40250]: pgmap v10989: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:52 smithi122 ceph-mon[32255]: pgmap v10989: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:04:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:04:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:04:53.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:54.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:54.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:54.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:04:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:54 smithi022 ceph-mon[31360]: pgmap v10990: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:54 smithi022 ceph-mon[40250]: pgmap v10990: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:54 smithi122 ceph-mon[32255]: pgmap v10990: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3819754274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3819754274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3819754274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:04:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:56 smithi022 ceph-mon[31360]: pgmap v10991: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:56 smithi022 ceph-mon[40250]: pgmap v10991: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:56 smithi122 ceph-mon[32255]: pgmap v10991: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:04:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:04:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[31360]: pgmap v10992: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:04:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:04:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:04:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[40250]: pgmap v10992: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:04:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:04:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:04:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:04:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:58 smithi122 ceph-mon[32255]: pgmap v10992: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:04:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:04:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:04:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:04:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:04:59.278 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:04:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:04:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:04:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:00] "GET /metrics HTTP/1.1" 200 202393 "" "Prometheus/2.18.1" 2021-12-16T00:05:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:00 smithi122 ceph-mon[32255]: pgmap v10993: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3449562345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:00 smithi022 ceph-mon[31360]: pgmap v10993: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3449562345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:00 smithi022 ceph-mon[40250]: pgmap v10993: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3449562345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:01.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:05:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:05:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:02 smithi022 ceph-mon[31360]: pgmap v10994: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:02 smithi022 ceph-mon[40250]: pgmap v10994: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:05:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:05:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:02 smithi122 ceph-mon[32255]: pgmap v10994: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:04.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:04 smithi122 ceph-mon[32255]: pgmap v10995: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:04.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:04 smithi022 ceph-mon[31360]: pgmap v10995: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:04 smithi022 ceph-mon[40250]: pgmap v10995: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:04.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:05.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/855074445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/855074445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:05.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/855074445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:05:05.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:06 smithi122 ceph-mon[32255]: pgmap v10996: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:06 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3100M 2021-12-16T00:05:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[31360]: pgmap v10996: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3100M 2021-12-16T00:05:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[40250]: pgmap v10996: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3100M 2021-12-16T00:05:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:05:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:05:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:08 smithi022 ceph-mon[31360]: pgmap v10997: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:08 smithi022 ceph-mon[40250]: pgmap v10997: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:08 smithi122 ceph-mon[32255]: pgmap v10997: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:10.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:10.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:10.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:10.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:10 smithi122 ceph-mon[32255]: pgmap v10998: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:10] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:05:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:10 smithi022 ceph-mon[31360]: pgmap v10998: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:10 smithi022 ceph-mon[40250]: pgmap v10998: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1642084776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1642084776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:11.541 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1642084776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:05:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:05:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:12 smithi022 ceph-mon[31360]: pgmap v10999: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:12 smithi022 ceph-mon[40250]: pgmap v10999: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:12 smithi122 ceph-mon[32255]: pgmap v10999: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:05:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:05:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:14 smithi122 ceph-mon[32255]: pgmap v11000: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:14 smithi022 ceph-mon[31360]: pgmap v11000: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:14 smithi022 ceph-mon[40250]: pgmap v11000: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:15.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:15.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:15.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:16 smithi122 ceph-mon[32255]: pgmap v11001: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3022283729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:16 smithi022 ceph-mon[31360]: pgmap v11001: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3022283729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:16 smithi022 ceph-mon[40250]: pgmap v11001: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3022283729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:05:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:18 smithi122 ceph-mon[32255]: pgmap v11002: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:18 smithi022 ceph-mon[31360]: pgmap v11002: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:18 smithi022 ceph-mon[40250]: pgmap v11002: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:20 smithi122 ceph-mon[32255]: pgmap v11003: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:20] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-16T00:05:20.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:20.756 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:20 smithi022 ceph-mon[31360]: pgmap v11003: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:20.756 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:20 smithi022 ceph-mon[40250]: pgmap v11003: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:21.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:21.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:21.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1394415460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1394415460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:21.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1394415460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:05:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:05:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:05:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:05:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:22 smithi122 ceph-mon[32255]: pgmap v11004: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:22 smithi022 ceph-mon[31360]: pgmap v11004: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:22 smithi022 ceph-mon[40250]: pgmap v11004: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:23 smithi122 ceph-mon[32255]: pgmap v11005: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:23 smithi022 ceph-mon[31360]: pgmap v11005: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:23 smithi022 ceph-mon[40250]: pgmap v11005: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:26.124 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:26.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:26.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:26.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:26.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:26 smithi022 ceph-mon[31360]: pgmap v11006: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:26.489 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:26 smithi022 ceph-mon[40250]: pgmap v11006: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:26 smithi122 ceph-mon[32255]: pgmap v11006: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:05:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1774142379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1774142379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1774142379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:28 smithi022 ceph-mon[31360]: pgmap v11007: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:28 smithi022 ceph-mon[40250]: pgmap v11007: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:28 smithi122 ceph-mon[32255]: pgmap v11007: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:30] "GET /metrics HTTP/1.1" 200 202399 "" "Prometheus/2.18.1" 2021-12-16T00:05:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:30 smithi122 ceph-mon[32255]: pgmap v11008: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:30 smithi022 ceph-mon[31360]: pgmap v11008: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:30 smithi022 ceph-mon[40250]: pgmap v11008: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:31.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:31.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:31.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:05:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:05:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:32 smithi022 ceph-mon[31360]: pgmap v11009: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1509254544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:32 smithi022 ceph-mon[40250]: pgmap v11009: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1509254544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:32 smithi122 ceph-mon[32255]: pgmap v11009: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1509254544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:05:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:05:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[31360]: pgmap v11010: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:05:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:05:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:05:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:05:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[40250]: pgmap v11010: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:05:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:05:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:05:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:05:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:34 smithi122 ceph-mon[32255]: pgmap v11010: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:05:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:05:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:05:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:05:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:36 smithi022 ceph-mon[31360]: pgmap v11011: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:36 smithi022 ceph-mon[40250]: pgmap v11011: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:36 smithi122 ceph-mon[32255]: pgmap v11011: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:36.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:37.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:37.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:37.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1807537522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:05:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1807537522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1807537522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:38 smithi022 ceph-mon[31360]: pgmap v11012: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:38 smithi022 ceph-mon[40250]: pgmap v11012: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:38 smithi122 ceph-mon[32255]: pgmap v11012: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:40] "GET /metrics HTTP/1.1" 200 202401 "" "Prometheus/2.18.1" 2021-12-16T00:05:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:40 smithi122 ceph-mon[32255]: pgmap v11013: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:40 smithi022 ceph-mon[31360]: pgmap v11013: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:40 smithi022 ceph-mon[40250]: pgmap v11013: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:41.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:05:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:05:42.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:42 smithi022 ceph-mon[31360]: pgmap v11014: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:42 smithi022 ceph-mon[40250]: pgmap v11014: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:42.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:42.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:42.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:42 smithi122 ceph-mon[32255]: pgmap v11014: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:05:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:05:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/179567328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/179567328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/179567328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:44 smithi022 ceph-mon[31360]: pgmap v11015: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:44 smithi022 ceph-mon[40250]: pgmap v11015: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:44 smithi122 ceph-mon[32255]: pgmap v11015: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:46 smithi022 ceph-mon[31360]: pgmap v11016: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:46 smithi022 ceph-mon[40250]: pgmap v11016: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:46 smithi122 ceph-mon[32255]: pgmap v11016: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:47.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:05:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:48 smithi022 ceph-mon[31360]: pgmap v11017: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1642947807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:48 smithi022 ceph-mon[40250]: pgmap v11017: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1642947807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:48 smithi122 ceph-mon[32255]: pgmap v11017: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1642947807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:50] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:05:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:50 smithi122 ceph-mon[32255]: pgmap v11018: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:50 smithi022 ceph-mon[31360]: pgmap v11018: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:50 smithi022 ceph-mon[40250]: pgmap v11018: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:05:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:05:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:52 smithi022 ceph-mon[31360]: pgmap v11019: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:52 smithi022 ceph-mon[40250]: pgmap v11019: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:52 smithi122 ceph-mon[32255]: pgmap v11019: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:05:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:05:52.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:53.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:53.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:53.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:05:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:54 smithi022 ceph-mon[31360]: pgmap v11020: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2036648727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:54 smithi022 ceph-mon[40250]: pgmap v11020: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2036648727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:54 smithi122 ceph-mon[32255]: pgmap v11020: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2036648727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:56 smithi122 ceph-mon[32255]: pgmap v11021: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:56 smithi022 ceph-mon[31360]: pgmap v11021: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:56 smithi022 ceph-mon[40250]: pgmap v11021: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:05:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:05:58.342 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:05:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:58 smithi122 ceph-mon[32255]: pgmap v11022: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:58.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:05:58.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:05:58.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:05:58.713 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:58 smithi022 ceph-mon[31360]: pgmap v11022: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:58.714 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:58 smithi022 ceph-mon[40250]: pgmap v11022: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:05:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:05:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4132271427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:05:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4132271427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:05:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:05:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4132271427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:00] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:06:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:00 smithi122 ceph-mon[32255]: pgmap v11023: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:00 smithi022 ceph-mon[31360]: pgmap v11023: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:00 smithi022 ceph-mon[40250]: pgmap v11023: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:01.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:06:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:06:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:02 smithi122 ceph-mon[32255]: pgmap v11024: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:06:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:06:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:02 smithi022 ceph-mon[31360]: pgmap v11024: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:02 smithi022 ceph-mon[40250]: pgmap v11024: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:03.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:03.715 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:03.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:04.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:04.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:04.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:04 smithi022 ceph-mon[31360]: pgmap v11025: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3326183997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:04 smithi022 ceph-mon[40250]: pgmap v11025: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3326183997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:04 smithi122 ceph-mon[32255]: pgmap v11025: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3326183997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:06 smithi122 ceph-mon[32255]: pgmap v11026: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:06:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:06:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:06:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[31360]: pgmap v11026: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:06:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:06:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:06:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[40250]: pgmap v11026: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:06:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:06:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:06:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:06:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:06:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:07 smithi022 ceph-mon[31360]: pgmap v11027: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:06:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:07 smithi022 ceph-mon[40250]: pgmap v11027: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:06:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:07 smithi122 ceph-mon[32255]: pgmap v11027: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:09.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:09 smithi122 ceph-mon[32255]: pgmap v11028: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1199024204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:09 smithi022 ceph-mon[31360]: pgmap v11028: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1199024204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:09 smithi022 ceph-mon[40250]: pgmap v11028: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1199024204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:10] "GET /metrics HTTP/1.1" 200 202397 "" "Prometheus/2.18.1" 2021-12-16T00:06:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:06:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:06:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:12 smithi022 ceph-mon[31360]: pgmap v11029: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:12 smithi022 ceph-mon[40250]: pgmap v11029: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:12 smithi122 ceph-mon[32255]: pgmap v11029: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:06:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:06:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:14.452 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:14 smithi022 ceph-mon[31360]: pgmap v11030: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:14 smithi022 ceph-mon[40250]: pgmap v11030: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:14 smithi122 ceph-mon[32255]: pgmap v11030: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:14.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:14.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:14.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3293510416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3293510416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3293510416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:16 smithi022 ceph-mon[31360]: pgmap v11031: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:16 smithi022 ceph-mon[40250]: pgmap v11031: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:16 smithi122 ceph-mon[32255]: pgmap v11031: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:06:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:18 smithi022 ceph-mon[31360]: pgmap v11032: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:18 smithi022 ceph-mon[40250]: pgmap v11032: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:18 smithi122 ceph-mon[32255]: pgmap v11032: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:20.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:20.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:20.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:20] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-16T00:06:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:20 smithi122 ceph-mon[32255]: pgmap v11033: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/905245594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:20 smithi022 ceph-mon[31360]: pgmap v11033: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/905245594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:20 smithi022 ceph-mon[40250]: pgmap v11033: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/905245594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:06:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:06:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:22 smithi022 ceph-mon[31360]: pgmap v11034: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:22 smithi022 ceph-mon[40250]: pgmap v11034: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:22 smithi122 ceph-mon[32255]: pgmap v11034: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:06:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:06:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:24 smithi022 ceph-mon[31360]: pgmap v11035: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:24 smithi022 ceph-mon[40250]: pgmap v11035: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:24 smithi122 ceph-mon[32255]: pgmap v11035: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:25.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:25.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:25.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:25.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:26 smithi022 ceph-mon[31360]: pgmap v11036: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1501691851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:26 smithi022 ceph-mon[40250]: pgmap v11036: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1501691851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:26 smithi122 ceph-mon[32255]: pgmap v11036: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1501691851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:06:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:28 smithi022 ceph-mon[31360]: pgmap v11037: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:28 smithi022 ceph-mon[40250]: pgmap v11037: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:28 smithi122 ceph-mon[32255]: pgmap v11037: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:30 smithi122 ceph-mon[32255]: pgmap v11038: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:30] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-16T00:06:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:30 smithi022 ceph-mon[31360]: pgmap v11038: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:30 smithi022 ceph-mon[40250]: pgmap v11038: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:30.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:30.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:30.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:30.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2852940810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2852940810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:31.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2852940810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:06:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:06:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:32 smithi022 ceph-mon[31360]: pgmap v11039: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:32 smithi022 ceph-mon[40250]: pgmap v11039: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:32 smithi122 ceph-mon[32255]: pgmap v11039: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:06:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:06:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:34 smithi122 ceph-mon[32255]: pgmap v11040: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:06:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:06:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:06:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:06:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[31360]: pgmap v11040: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:06:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:06:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:06:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:06:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[40250]: pgmap v11040: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:06:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:06:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:06:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:06:35.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:36.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:36.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:36.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:36 smithi122 ceph-mon[32255]: pgmap v11041: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:36 smithi022 ceph-mon[31360]: pgmap v11041: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:36 smithi022 ceph-mon[40250]: pgmap v11041: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1157888298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1157888298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:06:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1157888298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:38 smithi122 ceph-mon[32255]: pgmap v11042: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:38 smithi022 ceph-mon[31360]: pgmap v11042: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:38 smithi022 ceph-mon[40250]: pgmap v11042: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:40 smithi122 ceph-mon[32255]: pgmap v11043: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:40] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:06:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:40 smithi022 ceph-mon[31360]: pgmap v11043: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:40 smithi022 ceph-mon[40250]: pgmap v11043: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:41.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:41.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:41.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:06:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:06:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:42 smithi122 ceph-mon[32255]: pgmap v11044: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/220744822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:06:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:06:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:42 smithi022 ceph-mon[31360]: pgmap v11044: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/220744822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:42 smithi022 ceph-mon[40250]: pgmap v11044: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/220744822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:44 smithi122 ceph-mon[32255]: pgmap v11045: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:44 smithi022 ceph-mon[31360]: pgmap v11045: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:44 smithi022 ceph-mon[40250]: pgmap v11045: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:46.677 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:46.678 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:46 smithi022 ceph-mon[31360]: pgmap v11046: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:46.679 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:46 smithi022 ceph-mon[40250]: pgmap v11046: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:46 smithi122 ceph-mon[32255]: pgmap v11046: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:47.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:47.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:47.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:47.309 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2244771765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:47.310 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2244771765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2244771765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:06:48.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:48 smithi122 ceph-mon[32255]: pgmap v11047: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:48 smithi022 ceph-mon[31360]: pgmap v11047: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:48 smithi022 ceph-mon[40250]: pgmap v11047: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:49 smithi122 ceph-mon[32255]: pgmap v11048: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:49 smithi022 ceph-mon[31360]: pgmap v11048: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:49 smithi022 ceph-mon[40250]: pgmap v11048: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:50] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-16T00:06:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:06:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:06:52.050 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:52.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:52.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:52.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:52.420 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:52 smithi022 ceph-mon[31360]: pgmap v11049: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:52.420 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:52 smithi022 ceph-mon[40250]: pgmap v11049: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:52 smithi122 ceph-mon[32255]: pgmap v11049: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:06:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:06:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1054370984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1054370984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:06:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1054370984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:54 smithi022 ceph-mon[31360]: pgmap v11050: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:54 smithi022 ceph-mon[40250]: pgmap v11050: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:54 smithi122 ceph-mon[32255]: pgmap v11050: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:56 smithi022 ceph-mon[31360]: pgmap v11051: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:56 smithi022 ceph-mon[40250]: pgmap v11051: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:56 smithi122 ceph-mon[32255]: pgmap v11051: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:06:57.421 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:06:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:06:57.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:06:57.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:06:57.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:06:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:58 smithi022 ceph-mon[31360]: pgmap v11052: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:06:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/710316018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:58 smithi022 ceph-mon[40250]: pgmap v11052: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:06:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/710316018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:06:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:58 smithi122 ceph-mon[32255]: pgmap v11052: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:06:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:06:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/710316018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:00] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-16T00:07:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:00 smithi122 ceph-mon[32255]: pgmap v11053: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:00 smithi022 ceph-mon[31360]: pgmap v11053: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:00 smithi022 ceph-mon[40250]: pgmap v11053: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:01.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:07:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:07:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:02 smithi022 ceph-mon[31360]: pgmap v11054: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:02 smithi022 ceph-mon[40250]: pgmap v11054: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:02 smithi122 ceph-mon[32255]: pgmap v11054: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:07:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:07:02.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:03.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:03.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:03.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4136178336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4136178336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4136178336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:04 smithi022 ceph-mon[31360]: pgmap v11055: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:04 smithi022 ceph-mon[40250]: pgmap v11055: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:04 smithi122 ceph-mon[32255]: pgmap v11055: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:06 smithi022 ceph-mon[31360]: pgmap v11056: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:06 smithi022 ceph-mon[40250]: pgmap v11056: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:06.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:06 smithi122 ceph-mon[32255]: pgmap v11056: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:07:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:07:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:07:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:07:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:07:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:07:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:07:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:07:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:07:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:07:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:07:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:07:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:07:08.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:08 smithi022 ceph-mon[31360]: pgmap v11057: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:08 smithi022 ceph-mon[40250]: pgmap v11057: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:08.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:08.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:08.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:08 smithi122 ceph-mon[32255]: pgmap v11057: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4197201861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4197201861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4197201861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:10] "GET /metrics HTTP/1.1" 200 202402 "" "Prometheus/2.18.1" 2021-12-16T00:07:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:10 smithi122 ceph-mon[32255]: pgmap v11058: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:10 smithi022 ceph-mon[31360]: pgmap v11058: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:10 smithi022 ceph-mon[40250]: pgmap v11058: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:07:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:07:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:12 smithi022 ceph-mon[31360]: pgmap v11059: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:12 smithi022 ceph-mon[40250]: pgmap v11059: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:12 smithi122 ceph-mon[32255]: pgmap v11059: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:07:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:07:13.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:13.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:13.541 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:13.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:13.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:13.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:14 smithi122 ceph-mon[32255]: pgmap v11060: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/457472944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:14 smithi022 ceph-mon[31360]: pgmap v11060: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/457472944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:14 smithi022 ceph-mon[40250]: pgmap v11060: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/457472944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:16 smithi122 ceph-mon[32255]: pgmap v11061: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:16 smithi022 ceph-mon[31360]: pgmap v11061: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:16 smithi022 ceph-mon[40250]: pgmap v11061: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:07:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:18 smithi122 ceph-mon[32255]: pgmap v11062: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:18 smithi022 ceph-mon[31360]: pgmap v11062: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:18 smithi022 ceph-mon[40250]: pgmap v11062: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:18.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:19.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:19.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:19.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4107245375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4107245375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4107245375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:20] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:07:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:20 smithi122 ceph-mon[32255]: pgmap v11063: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:20 smithi022 ceph-mon[31360]: pgmap v11063: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:20 smithi022 ceph-mon[40250]: pgmap v11063: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:07:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:07:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:22 smithi122 ceph-mon[32255]: pgmap v11064: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:07:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:07:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:22 smithi022 ceph-mon[31360]: pgmap v11064: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:22 smithi022 ceph-mon[40250]: pgmap v11064: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:24.282 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:24.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:24.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:24.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:24.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:24 smithi022 ceph-mon[31360]: pgmap v11065: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:24.654 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:24 smithi022 ceph-mon[40250]: pgmap v11065: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:24 smithi122 ceph-mon[32255]: pgmap v11065: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2912191184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2912191184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2912191184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:26 smithi122 ceph-mon[32255]: pgmap v11066: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:26 smithi022 ceph-mon[31360]: pgmap v11066: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:26 smithi022 ceph-mon[40250]: pgmap v11066: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:07:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:28 smithi122 ceph-mon[32255]: pgmap v11067: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:28 smithi022 ceph-mon[31360]: pgmap v11067: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:28 smithi022 ceph-mon[40250]: pgmap v11067: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:29.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:30.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:30.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:30.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:30] "GET /metrics HTTP/1.1" 200 202398 "" "Prometheus/2.18.1" 2021-12-16T00:07:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:30 smithi122 ceph-mon[32255]: pgmap v11068: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2312413567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:30 smithi022 ceph-mon[31360]: pgmap v11068: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2312413567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:30 smithi022 ceph-mon[40250]: pgmap v11068: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2312413567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:07:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:07:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:32 smithi122 ceph-mon[32255]: pgmap v11069: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:07:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:07:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:32 smithi022 ceph-mon[31360]: pgmap v11069: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:32 smithi022 ceph-mon[40250]: pgmap v11069: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:33 smithi122 ceph-mon[32255]: pgmap v11070: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:33 smithi022 ceph-mon[31360]: pgmap v11070: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:33 smithi022 ceph-mon[40250]: pgmap v11070: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:07:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:07:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:07:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:07:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:07:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:07:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:07:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:07:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:07:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:07:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:07:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:07:35.026 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:35.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:35.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:35.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:36 smithi022 ceph-mon[31360]: pgmap v11071: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1351585847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:36 smithi022 ceph-mon[40250]: pgmap v11071: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1351585847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:36 smithi122 ceph-mon[32255]: pgmap v11071: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1351585847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:07:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:38 smithi022 ceph-mon[31360]: pgmap v11072: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:38 smithi022 ceph-mon[40250]: pgmap v11072: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:38 smithi122 ceph-mon[32255]: pgmap v11072: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:40.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:40] "GET /metrics HTTP/1.1" 200 202396 "" "Prometheus/2.18.1" 2021-12-16T00:07:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:40 smithi122 ceph-mon[32255]: pgmap v11073: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:40 smithi022 ceph-mon[31360]: pgmap v11073: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:40 smithi022 ceph-mon[40250]: pgmap v11073: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:40.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:40.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:40.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3332602647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3332602647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:41.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3332602647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:41.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:07:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:07:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:42 smithi022 ceph-mon[31360]: pgmap v11074: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:42 smithi022 ceph-mon[40250]: pgmap v11074: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:42 smithi122 ceph-mon[32255]: pgmap v11074: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:07:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:07:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:44 smithi022 ceph-mon[31360]: pgmap v11075: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:44 smithi022 ceph-mon[40250]: pgmap v11075: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:44 smithi122 ceph-mon[32255]: pgmap v11075: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:45.776 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:46.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:46.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:46.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:46 smithi022 ceph-mon[31360]: pgmap v11076: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1660194889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:46 smithi022 ceph-mon[40250]: pgmap v11076: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1660194889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:46 smithi122 ceph-mon[32255]: pgmap v11076: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1660194889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:07:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:48 smithi022 ceph-mon[31360]: pgmap v11077: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:48 smithi022 ceph-mon[40250]: pgmap v11077: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:48 smithi122 ceph-mon[32255]: pgmap v11077: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:50] "GET /metrics HTTP/1.1" 200 202405 "" "Prometheus/2.18.1" 2021-12-16T00:07:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:50 smithi122 ceph-mon[32255]: pgmap v11078: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:50 smithi022 ceph-mon[31360]: pgmap v11078: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:50 smithi022 ceph-mon[40250]: pgmap v11078: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:51.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:51.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:51.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:51.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:07:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:07:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:52 smithi022 ceph-mon[31360]: pgmap v11079: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1376656598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:52 smithi022 ceph-mon[40250]: pgmap v11079: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1376656598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:07:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:07:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:52 smithi122 ceph-mon[32255]: pgmap v11079: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1376656598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:07:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:54 smithi122 ceph-mon[32255]: pgmap v11080: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:54 smithi022 ceph-mon[31360]: pgmap v11080: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:54 smithi022 ceph-mon[40250]: pgmap v11080: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:56.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:07:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:56 smithi122 ceph-mon[32255]: pgmap v11081: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:56 smithi022 ceph-mon[31360]: pgmap v11081: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:56 smithi022 ceph-mon[40250]: pgmap v11081: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:07:56.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:07:56.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:07:56.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:07:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3632077584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3632077584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:07:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3632077584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:07:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:07:58 smithi122 ceph-mon[32255]: pgmap v11082: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:07:58 smithi022 ceph-mon[31360]: pgmap v11082: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:07:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:07:58 smithi022 ceph-mon[40250]: pgmap v11082: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:00] "GET /metrics HTTP/1.1" 200 202405 "" "Prometheus/2.18.1" 2021-12-16T00:08:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:00 smithi122 ceph-mon[32255]: pgmap v11083: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:00 smithi022 ceph-mon[31360]: pgmap v11083: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:00 smithi022 ceph-mon[40250]: pgmap v11083: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:08:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:08:02.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:02.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:02.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:02 smithi022 ceph-mon[31360]: pgmap v11084: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3588344784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:02 smithi022 ceph-mon[40250]: pgmap v11084: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3588344784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:02 smithi122 ceph-mon[32255]: pgmap v11084: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3588344784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:08:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:08:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:04 smithi122 ceph-mon[32255]: pgmap v11085: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:04 smithi022 ceph-mon[31360]: pgmap v11085: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:04 smithi022 ceph-mon[40250]: pgmap v11085: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:06 smithi122 ceph-mon[32255]: pgmap v11086: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:06 smithi022 ceph-mon[31360]: pgmap v11086: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:06 smithi022 ceph-mon[40250]: pgmap v11086: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:07.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:08:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:08:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:08:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:08:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:08:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:08:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:08:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:08:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:08:07.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:07.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:07.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:08:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:08 smithi122 ceph-mon[32255]: pgmap v11087: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/989756236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:08 smithi022 ceph-mon[31360]: pgmap v11087: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/989756236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:08 smithi022 ceph-mon[40250]: pgmap v11087: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:08:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/989756236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:10 smithi122 ceph-mon[32255]: pgmap v11088: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:08:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:10] "GET /metrics HTTP/1.1" 200 202395 "" "Prometheus/2.18.1" 2021-12-16T00:08:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:10 smithi022 ceph-mon[31360]: pgmap v11088: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:08:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:10 smithi022 ceph-mon[40250]: pgmap v11088: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:08:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:08:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:08:12.622 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:12 smithi022 ceph-mon[31360]: pgmap v11089: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:12.622 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:12 smithi022 ceph-mon[40250]: pgmap v11089: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:12.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:12 smithi122 ceph-mon[32255]: pgmap v11089: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:08:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:08:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:12.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:12.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/5471778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/5471778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/5471778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:14 smithi122 ceph-mon[32255]: pgmap v11090: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:14 smithi022 ceph-mon[31360]: pgmap v11090: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:14 smithi022 ceph-mon[40250]: pgmap v11090: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:16 smithi122 ceph-mon[32255]: pgmap v11091: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:08:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:16 smithi022 ceph-mon[31360]: pgmap v11091: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:08:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:16 smithi022 ceph-mon[40250]: pgmap v11091: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:08:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:08:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:18.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:18.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:18.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:18 smithi122 ceph-mon[32255]: pgmap v11092: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:18 smithi022 ceph-mon[31360]: pgmap v11092: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:18 smithi022 ceph-mon[40250]: pgmap v11092: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:08:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2472606471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:19 smithi122 ceph-mon[32255]: pgmap v11093: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:08:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2472606471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:19 smithi022 ceph-mon[31360]: pgmap v11093: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:08:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2472606471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:19 smithi022 ceph-mon[40250]: pgmap v11093: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:08:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:20] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-16T00:08:22.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:08:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:08:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:22 smithi022 ceph-mon[31360]: pgmap v11094: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:22 smithi022 ceph-mon[40250]: pgmap v11094: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:22 smithi122 ceph-mon[32255]: pgmap v11094: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:08:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:08:23.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:23.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:23.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:23.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:24 smithi022 ceph-mon[31360]: pgmap v11095: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/492003666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:24 smithi022 ceph-mon[40250]: pgmap v11095: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/492003666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:24 smithi122 ceph-mon[32255]: pgmap v11095: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/492003666' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:26 smithi022 ceph-mon[40250]: pgmap v11096: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:26 smithi022 ceph-mon[31360]: pgmap v11096: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:26 smithi122 ceph-mon[32255]: pgmap v11096: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:08:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:28 smithi022 ceph-mon[31360]: pgmap v11097: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:28 smithi022 ceph-mon[40250]: pgmap v11097: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:28 smithi122 ceph-mon[32255]: pgmap v11097: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:28.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:29.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:29.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:29.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2740719046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2740719046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2740719046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:30 smithi122 ceph-mon[32255]: pgmap v11098: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:30] "GET /metrics HTTP/1.1" 200 202386 "" "Prometheus/2.18.1" 2021-12-16T00:08:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:30 smithi022 ceph-mon[31360]: pgmap v11098: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:30 smithi022 ceph-mon[40250]: pgmap v11098: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:08:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:08:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:32 smithi022 ceph-mon[31360]: pgmap v11099: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:32 smithi022 ceph-mon[40250]: pgmap v11099: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:32 smithi122 ceph-mon[32255]: pgmap v11099: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:08:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:08:33.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:32 smithi022 sudo[298498]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:33.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:32 smithi022 sudo[298498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:33.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:32 smithi022 sudo[298498]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:33.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:32 smithi022 sudo[298498]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:33.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298501]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:33.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298501]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:33.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298501]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:33.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298501]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:34.028 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298504]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:34.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298504]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:34.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298504]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:34.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:33 smithi022 sudo[298504]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:34.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:34.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:34.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:34.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:34.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[31360]: pgmap v11100: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:34.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:08:34.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:08:34.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:08:34.466 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:08:34.466 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[40250]: pgmap v11100: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:34.467 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:08:34.467 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:08:34.467 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:08:34.468 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:08:34.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298507]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:34.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298507]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:34.469 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298507]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:34.469 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298507]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:34 smithi122 ceph-mon[32255]: pgmap v11100: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:08:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:08:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:08:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:08:34.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298535]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:34.779 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:34.779 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298535]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:34.779 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298535]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:35.278 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298538]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:35.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:35.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298538]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:35.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 00:08:34 smithi022 sudo[298538]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4268101915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4268101915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4268101915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:35.779 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298541]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:35.780 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298541]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:35.780 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298541]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:35.780 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298541]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:36.278 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298544]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:36.279 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:36.279 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298544]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:36.279 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 00:08:35 smithi022 sudo[298544]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:36.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:36 smithi122 ceph-mon[32255]: pgmap v11101: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:36.369 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177851]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:36.369 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177851]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:36.369 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177851]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:36.370 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177851]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:36.630 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177854]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:36.630 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177854]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:36.630 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177854]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:36.630 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177854]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:36 smithi022 ceph-mon[31360]: pgmap v11101: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:36 smithi022 ceph-mon[40250]: pgmap v11101: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:08:36.881 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177857]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:36.881 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177857]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:36.881 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177857]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:36.881 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177857]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:37.136 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177860]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:37.136 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177860]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:37.137 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177860]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:37.137 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 00:08:36 smithi122 sudo[177860]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:37.137 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177863]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:37.138 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177863]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:37.138 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177863]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:37.138 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177863]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:37.394 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177866]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:37.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:37.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177866]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:37.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177866]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:37 smithi122 ceph-mon[32255]: pgmap v11102: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:37.686 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177869]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:37.686 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:37.687 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177869]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:37.687 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177869]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:08:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:37 smithi022 ceph-mon[31360]: pgmap v11102: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:08:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:37 smithi022 ceph-mon[40250]: pgmap v11102: 132 pgs: 132 active+clean; 582 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:38.185 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177872]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:38.186 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177872]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:38.186 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177872]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:38.186 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 00:08:37 smithi122 sudo[177872]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298547]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298547]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298547]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298547]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:39.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298550]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:39.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298550]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:39.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298550]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:39.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:38 smithi022 sudo[298550]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:39.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298553]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298553]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298553]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298553]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:39.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:39.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:39.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:39.995 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:39.995 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:39.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:39.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:39.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:39.997 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:39.997 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:39.997 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:40.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:40.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:40.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298578]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:40.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298578]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:40.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298578]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:40.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 sudo[298578]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:40.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:40.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:40.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:40.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:40.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:40.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:40.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:40.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:40.254 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:40] "GET /metrics HTTP/1.1" 200 202394 "" "Prometheus/2.18.1" 2021-12-16T00:08:40.255 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 sudo[177875]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2021-12-16T00:08:40.255 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 sudo[177875]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:40.255 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:39 smithi122 sudo[177875]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:40.255 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 sudo[177875]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:40.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 sudo[177878]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2021-12-16T00:08:40.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 sudo[177878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2021-12-16T00:08:40.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 sudo[177878]: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-12-16T00:08:40.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 sudo[177878]: pam_unix(sudo:session): session closed for user root 2021-12-16T00:08:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: pgmap v11103: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 7 op/s 2021-12-16T00:08:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:41.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:41.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/207037529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:41.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[31360]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:41.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: pgmap v11103: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 7 op/s 2021-12-16T00:08:41.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:41.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:41.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:41.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:41.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:41.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:41.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/207037529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:41.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:40 smithi022 ceph-mon[40250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: pgmap v11103: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 7 op/s 2021-12-16T00:08:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-12-16T00:08:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-12-16T00:08:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-12-16T00:08:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2021-12-16T00:08:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/207037529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:41.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:40 smithi122 ceph-mon[32255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2021-12-16T00:08:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:41 smithi022 ceph-mon[31360]: pgmap v11104: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 6 op/s 2021-12-16T00:08:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:41 smithi022 ceph-mon[40250]: pgmap v11104: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 6 op/s 2021-12-16T00:08:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:08:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:08:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:41 smithi122 ceph-mon[32255]: pgmap v11104: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 6 op/s 2021-12-16T00:08:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:08:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:08:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:43 smithi022 ceph-mon[31360]: pgmap v11105: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 6 op/s 2021-12-16T00:08:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:43 smithi022 ceph-mon[40250]: pgmap v11105: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 6 op/s 2021-12-16T00:08:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:43 smithi122 ceph-mon[32255]: pgmap v11105: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 65 KiB/s wr, 6 op/s 2021-12-16T00:08:44.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:45.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:45.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:45.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:45 smithi122 ceph-mon[32255]: pgmap v11106: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2021-12-16T00:08:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4097970840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:45 smithi022 ceph-mon[31360]: pgmap v11106: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2021-12-16T00:08:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4097970840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:45 smithi022 ceph-mon[40250]: pgmap v11106: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2021-12-16T00:08:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4097970840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:08:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:47 smithi122 ceph-mon[32255]: pgmap v11107: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2021-12-16T00:08:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:47 smithi022 ceph-mon[31360]: pgmap v11107: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2021-12-16T00:08:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:47 smithi022 ceph-mon[40250]: pgmap v11107: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2021-12-16T00:08:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:49 smithi122 ceph-mon[32255]: pgmap v11108: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2021-12-16T00:08:50.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:50.201 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:49 smithi022 ceph-mon[40250]: pgmap v11108: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2021-12-16T00:08:50.201 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:49 smithi022 ceph-mon[31360]: pgmap v11108: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2021-12-16T00:08:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:50] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:08:50.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:50.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:50.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/92291220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/92291220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/92291220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:08:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:08:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:52 smithi022 ceph-mon[31360]: pgmap v11109: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2021-12-16T00:08:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:52 smithi022 ceph-mon[40250]: pgmap v11109: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2021-12-16T00:08:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:52 smithi122 ceph-mon[32255]: pgmap v11109: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2021-12-16T00:08:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:08:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:08:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:08:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:54 smithi022 ceph-mon[31360]: pgmap v11110: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2021-12-16T00:08:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:54 smithi022 ceph-mon[40250]: pgmap v11110: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2021-12-16T00:08:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:54 smithi122 ceph-mon[32255]: pgmap v11110: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 22 KiB/s wr, 2 op/s 2021-12-16T00:08:55.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:08:55.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:08:55.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:08:55.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:08:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:56 smithi022 ceph-mon[31360]: pgmap v11111: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 22 KiB/s wr, 3 op/s 2021-12-16T00:08:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3244537977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:56 smithi022 ceph-mon[40250]: pgmap v11111: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 22 KiB/s wr, 3 op/s 2021-12-16T00:08:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3244537977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:56 smithi122 ceph-mon[32255]: pgmap v11111: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 22 KiB/s wr, 3 op/s 2021-12-16T00:08:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3244537977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:08:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:08:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:08:58 smithi122 ceph-mon[32255]: pgmap v11112: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:08:58 smithi022 ceph-mon[31360]: pgmap v11112: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:08:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:08:58 smithi022 ceph-mon[40250]: pgmap v11112: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:00] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:09:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:00 smithi122 ceph-mon[32255]: pgmap v11113: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:00 smithi022 ceph-mon[31360]: pgmap v11113: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:00 smithi022 ceph-mon[40250]: pgmap v11113: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:01.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:01.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:01.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:09:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:09:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:09:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:09:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:02 smithi122 ceph-mon[32255]: pgmap v11114: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1074261930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:02 smithi022 ceph-mon[31360]: pgmap v11114: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1074261930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:02 smithi022 ceph-mon[40250]: pgmap v11114: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1074261930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:04 smithi122 ceph-mon[32255]: pgmap v11115: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:04 smithi022 ceph-mon[31360]: pgmap v11115: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:04 smithi022 ceph-mon[40250]: pgmap v11115: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:06.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:06.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:06.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:06.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:06.683 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:06 smithi022 ceph-mon[40250]: pgmap v11116: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:06.684 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:06 smithi022 ceph-mon[31360]: pgmap v11116: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:06 smithi122 ceph-mon[32255]: pgmap v11116: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:07.562 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:09:07.621 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/115095849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:07.621 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/115095849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:07.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/115095849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:08 smithi122 ceph-mon[32255]: pgmap v11117: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:09:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:09:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:09:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:09:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[31360]: pgmap v11117: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:09:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:09:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:09:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:09:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[40250]: pgmap v11117: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:09:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:09:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:09:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:09:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:10] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:09:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:10 smithi122 ceph-mon[32255]: pgmap v11118: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:10 smithi022 ceph-mon[31360]: pgmap v11118: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:10 smithi022 ceph-mon[40250]: pgmap v11118: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:11.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:09:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:09:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:12 smithi122 ceph-mon[32255]: pgmap v11119: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1703026960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:09:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:09:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:12 smithi022 ceph-mon[31360]: pgmap v11119: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1703026960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:12 smithi022 ceph-mon[40250]: pgmap v11119: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1703026960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:14 smithi122 ceph-mon[32255]: pgmap v11120: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:14 smithi022 ceph-mon[31360]: pgmap v11120: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:14 smithi022 ceph-mon[40250]: pgmap v11120: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:16 smithi122 ceph-mon[32255]: pgmap v11121: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:16 smithi022 ceph-mon[31360]: pgmap v11121: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:16 smithi022 ceph-mon[40250]: pgmap v11121: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:17.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:17.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:17.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:17.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:09:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:18 smithi122 ceph-mon[32255]: pgmap v11122: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2499273324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:18 smithi022 ceph-mon[31360]: pgmap v11122: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2499273324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:18 smithi022 ceph-mon[40250]: pgmap v11122: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2499273324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:20] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:09:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:20 smithi122 ceph-mon[32255]: pgmap v11123: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:20 smithi022 ceph-mon[31360]: pgmap v11123: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:20 smithi022 ceph-mon[40250]: pgmap v11123: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:22.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:09:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:09:22.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:22 smithi122 ceph-mon[32255]: pgmap v11124: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:09:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:09:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:22 smithi022 ceph-mon[31360]: pgmap v11124: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:22 smithi022 ceph-mon[40250]: pgmap v11124: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:22.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:22.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:22.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/347600866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/347600866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/347600866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:24 smithi122 ceph-mon[32255]: pgmap v11125: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:24 smithi022 ceph-mon[31360]: pgmap v11125: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:24 smithi022 ceph-mon[40250]: pgmap v11125: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:26 smithi022 ceph-mon[31360]: pgmap v11126: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:26 smithi022 ceph-mon[40250]: pgmap v11126: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:26 smithi122 ceph-mon[32255]: pgmap v11126: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:09:27.801 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:28.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:28.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:28.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:28 smithi022 ceph-mon[31360]: pgmap v11127: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2239217836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:28 smithi022 ceph-mon[40250]: pgmap v11127: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2239217836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:28 smithi122 ceph-mon[32255]: pgmap v11127: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2239217836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:30] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:09:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:30 smithi122 ceph-mon[32255]: pgmap v11128: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:30 smithi022 ceph-mon[31360]: pgmap v11128: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:30 smithi022 ceph-mon[40250]: pgmap v11128: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:09:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:09:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:32 smithi022 ceph-mon[31360]: pgmap v11129: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:32 smithi022 ceph-mon[40250]: pgmap v11129: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:32 smithi122 ceph-mon[32255]: pgmap v11129: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:09:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:09:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:33.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:33.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:33.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:34 smithi122 ceph-mon[32255]: pgmap v11130: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1772474159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:09:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:09:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:09:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:09:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[31360]: pgmap v11130: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1772474159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:09:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:09:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:09:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:09:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[40250]: pgmap v11130: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1772474159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:09:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:09:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:09:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:09:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:36 smithi122 ceph-mon[32255]: pgmap v11131: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:36 smithi022 ceph-mon[31360]: pgmap v11131: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:36 smithi022 ceph-mon[40250]: pgmap v11131: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:09:38.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:38.549 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:38 smithi022 ceph-mon[40250]: pgmap v11132: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:38.549 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:38 smithi022 ceph-mon[31360]: pgmap v11132: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:38 smithi122 ceph-mon[32255]: pgmap v11132: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:38.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:38.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:38.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1298090041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1298090041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1298090041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:40] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:09:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:40 smithi122 ceph-mon[32255]: pgmap v11133: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:40 smithi022 ceph-mon[31360]: pgmap v11133: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:40 smithi022 ceph-mon[40250]: pgmap v11133: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:09:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:09:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:42 smithi122 ceph-mon[32255]: pgmap v11134: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:09:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:09:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:42 smithi022 ceph-mon[31360]: pgmap v11134: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:42 smithi022 ceph-mon[40250]: pgmap v11134: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:43.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:44.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:44.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:44.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:44 smithi122 ceph-mon[32255]: pgmap v11135: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3556040540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:44 smithi022 ceph-mon[31360]: pgmap v11135: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3556040540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:44 smithi022 ceph-mon[40250]: pgmap v11135: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3556040540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:46 smithi122 ceph-mon[32255]: pgmap v11136: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:46 smithi022 ceph-mon[31360]: pgmap v11136: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:46 smithi022 ceph-mon[40250]: pgmap v11136: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:09:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:48 smithi122 ceph-mon[32255]: pgmap v11137: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:48 smithi022 ceph-mon[31360]: pgmap v11137: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:48 smithi022 ceph-mon[40250]: pgmap v11137: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:49.292 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:49.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:49.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:49.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:50] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:09:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:50 smithi122 ceph-mon[32255]: pgmap v11138: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1344212921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:50 smithi022 ceph-mon[31360]: pgmap v11138: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1344212921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:50 smithi022 ceph-mon[40250]: pgmap v11138: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1344212921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:09:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:09:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:52 smithi122 ceph-mon[32255]: pgmap v11139: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:09:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:09:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:52 smithi022 ceph-mon[31360]: pgmap v11139: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:52 smithi022 ceph-mon[40250]: pgmap v11139: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:09:54.662 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:09:54.663 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:54 smithi022 ceph-mon[31360]: pgmap v11140: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:54.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:54 smithi022 ceph-mon[40250]: pgmap v11140: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:54 smithi122 ceph-mon[32255]: pgmap v11140: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:55.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:09:55.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:09:55.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:09:55.322 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/100954140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/100954140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/100954140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:09:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:56 smithi122 ceph-mon[32255]: pgmap v11141: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:56 smithi022 ceph-mon[31360]: pgmap v11141: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:56 smithi022 ceph-mon[40250]: pgmap v11141: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:09:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:09:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:09:58 smithi122 ceph-mon[32255]: pgmap v11142: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:09:58 smithi022 ceph-mon[31360]: pgmap v11142: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:09:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:09:58 smithi022 ceph-mon[40250]: pgmap v11142: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:00.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:00.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:00.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:00.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:00] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:10:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:00 smithi122 ceph-mon[32255]: pgmap v11143: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T00:10:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:00 smithi022 ceph-mon[31360]: pgmap v11143: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T00:10:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:00 smithi022 ceph-mon[40250]: pgmap v11143: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T00:10:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221627843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221627843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221627843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:10:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:10:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:02 smithi122 ceph-mon[32255]: pgmap v11144: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:10:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:10:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:02 smithi022 ceph-mon[31360]: pgmap v11144: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:02 smithi022 ceph-mon[40250]: pgmap v11144: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:04 smithi122 ceph-mon[32255]: pgmap v11145: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:04 smithi022 ceph-mon[31360]: pgmap v11145: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:04 smithi022 ceph-mon[40250]: pgmap v11145: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:05.398 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:05.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:05.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:05.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:06 smithi022 ceph-mon[31360]: pgmap v11146: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4289501459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:06 smithi022 ceph-mon[40250]: pgmap v11146: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4289501459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:06 smithi122 ceph-mon[32255]: pgmap v11146: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4289501459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:10:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:08 smithi122 ceph-mon[32255]: pgmap v11147: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:10:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:10:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:10:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[40250]: pgmap v11147: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:10:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:10:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:10:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[31360]: pgmap v11147: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:10:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:10:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:10:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:10:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:09 smithi122 ceph-mon[32255]: pgmap v11148: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:10:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:09 smithi022 ceph-mon[31360]: pgmap v11148: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:10:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:09 smithi022 ceph-mon[40250]: pgmap v11148: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:10] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:10:10.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:11.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:11.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:11.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1842750177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1842750177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1842750177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:10:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:10:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:12 smithi122 ceph-mon[32255]: pgmap v11149: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:10:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:10:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:12 smithi022 ceph-mon[31360]: pgmap v11149: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:12 smithi022 ceph-mon[40250]: pgmap v11149: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:14 smithi122 ceph-mon[32255]: pgmap v11150: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:14 smithi022 ceph-mon[31360]: pgmap v11150: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:14 smithi022 ceph-mon[40250]: pgmap v11150: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:16.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:16.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:16.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:16.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:16.515 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:16 smithi022 ceph-mon[31360]: pgmap v11151: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:16.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:16 smithi022 ceph-mon[40250]: pgmap v11151: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:16 smithi122 ceph-mon[32255]: pgmap v11151: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1878289545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1878289545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:10:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1878289545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:18 smithi122 ceph-mon[32255]: pgmap v11152: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:18 smithi022 ceph-mon[31360]: pgmap v11152: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:18 smithi022 ceph-mon[40250]: pgmap v11152: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:20 smithi122 ceph-mon[32255]: pgmap v11153: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:20] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:10:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:20 smithi022 ceph-mon[31360]: pgmap v11153: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:20 smithi022 ceph-mon[40250]: pgmap v11153: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:21.516 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:21.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:21.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:21.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:22.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:10:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:10:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:10:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:10:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:22 smithi122 ceph-mon[32255]: pgmap v11154: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3651735013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:22 smithi022 ceph-mon[31360]: pgmap v11154: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3651735013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:22 smithi022 ceph-mon[40250]: pgmap v11154: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3651735013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:24 smithi122 ceph-mon[32255]: pgmap v11155: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:24 smithi022 ceph-mon[31360]: pgmap v11155: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:24 smithi022 ceph-mon[40250]: pgmap v11155: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:26 smithi122 ceph-mon[32255]: pgmap v11156: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:26 smithi022 ceph-mon[31360]: pgmap v11156: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:26 smithi022 ceph-mon[40250]: pgmap v11156: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:26.886 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:27.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:27.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:27.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3765411093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:10:27.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3765411093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3765411093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:28 smithi122 ceph-mon[32255]: pgmap v11157: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:28 smithi022 ceph-mon[31360]: pgmap v11157: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:28 smithi022 ceph-mon[40250]: pgmap v11157: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:30] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:10:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:30 smithi122 ceph-mon[32255]: pgmap v11158: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:30 smithi022 ceph-mon[31360]: pgmap v11158: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:30 smithi022 ceph-mon[40250]: pgmap v11158: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:10:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:10:32.255 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:32 smithi022 ceph-mon[31360]: pgmap v11159: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:32 smithi022 ceph-mon[40250]: pgmap v11159: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:32.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:32.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:32.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:32 smithi122 ceph-mon[32255]: pgmap v11159: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:10:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:10:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3003631668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3003631668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3003631668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:34 smithi122 ceph-mon[32255]: pgmap v11160: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:10:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:10:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:10:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:10:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[31360]: pgmap v11160: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:10:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:10:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:10:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:10:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[40250]: pgmap v11160: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:10:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:10:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:10:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:10:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:36 smithi122 ceph-mon[32255]: pgmap v11161: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:36 smithi022 ceph-mon[31360]: pgmap v11161: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:36 smithi022 ceph-mon[40250]: pgmap v11161: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:37.628 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:10:37.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:37.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:37.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:38 smithi122 ceph-mon[32255]: pgmap v11162: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3666786121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:38 smithi022 ceph-mon[31360]: pgmap v11162: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3666786121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:38 smithi022 ceph-mon[40250]: pgmap v11162: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3666786121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:40 smithi122 ceph-mon[32255]: pgmap v11163: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:40] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:10:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:40 smithi022 ceph-mon[31360]: pgmap v11163: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:40 smithi022 ceph-mon[40250]: pgmap v11163: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:10:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:10:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:42 smithi122 ceph-mon[32255]: pgmap v11164: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:10:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:10:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:42 smithi022 ceph-mon[31360]: pgmap v11164: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:42 smithi022 ceph-mon[40250]: pgmap v11164: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:42.998 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:43.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:43.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:43.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:44 smithi122 ceph-mon[32255]: pgmap v11165: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/899108591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:44 smithi022 ceph-mon[31360]: pgmap v11165: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/899108591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:44 smithi022 ceph-mon[40250]: pgmap v11165: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/899108591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:46 smithi122 ceph-mon[32255]: pgmap v11166: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:46 smithi022 ceph-mon[31360]: pgmap v11166: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:46 smithi022 ceph-mon[40250]: pgmap v11166: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:10:48.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:48 smithi122 ceph-mon[32255]: pgmap v11167: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:48.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:48.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:48.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:48.742 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:48 smithi022 ceph-mon[31360]: pgmap v11167: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:48.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:48 smithi022 ceph-mon[40250]: pgmap v11167: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/273264289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/273264289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/273264289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:50 smithi122 ceph-mon[32255]: pgmap v11168: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:50] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:10:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:50 smithi022 ceph-mon[31360]: pgmap v11168: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:50 smithi022 ceph-mon[40250]: pgmap v11168: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:10:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:10:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:52 smithi122 ceph-mon[32255]: pgmap v11169: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:10:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:10:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:52 smithi022 ceph-mon[31360]: pgmap v11169: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:52 smithi022 ceph-mon[40250]: pgmap v11169: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:53.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:53.743 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:53.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:10:54.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:54.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:54.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:10:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:54 smithi022 ceph-mon[31360]: pgmap v11170: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/40714246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:54 smithi022 ceph-mon[40250]: pgmap v11170: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/40714246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:54 smithi122 ceph-mon[32255]: pgmap v11170: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/40714246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:10:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:56 smithi122 ceph-mon[32255]: pgmap v11171: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:56 smithi022 ceph-mon[31360]: pgmap v11171: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:56 smithi022 ceph-mon[40250]: pgmap v11171: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:10:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:10:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:10:58 smithi122 ceph-mon[32255]: pgmap v11172: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:10:58 smithi022 ceph-mon[31360]: pgmap v11172: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:10:58 smithi022 ceph-mon[40250]: pgmap v11172: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:10:59.110 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:10:59.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:10:59.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:10:59.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:00] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:11:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:00 smithi122 ceph-mon[32255]: pgmap v11173: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2888561946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:00 smithi022 ceph-mon[40250]: pgmap v11173: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2888561946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:00 smithi022 ceph-mon[31360]: pgmap v11173: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2888561946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:11:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:11:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:02 smithi122 ceph-mon[32255]: pgmap v11174: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:11:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:11:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:02 smithi022 ceph-mon[31360]: pgmap v11174: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:02 smithi022 ceph-mon[40250]: pgmap v11174: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:04.482 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:04 smithi122 ceph-mon[32255]: pgmap v11175: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:04 smithi022 ceph-mon[31360]: pgmap v11175: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:04 smithi022 ceph-mon[40250]: pgmap v11175: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:04.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:04.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3989959078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3989959078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3989959078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:06 smithi122 ceph-mon[32255]: pgmap v11176: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:06 smithi022 ceph-mon[31360]: pgmap v11176: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:06 smithi022 ceph-mon[40250]: pgmap v11176: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:11:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:08 smithi122 ceph-mon[32255]: pgmap v11177: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:08 smithi022 ceph-mon[31360]: pgmap v11177: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:08 smithi022 ceph-mon[40250]: pgmap v11177: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:11:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:11:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:11:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:11:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:11:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:11:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:11:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:11:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:11:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:11:09.853 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:10.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:10.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:10.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:10] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:11:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:10 smithi122 ceph-mon[32255]: pgmap v11178: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2670384043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:10 smithi022 ceph-mon[31360]: pgmap v11178: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2670384043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:10 smithi022 ceph-mon[40250]: pgmap v11178: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2670384043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:11:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:11:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:12 smithi122 ceph-mon[32255]: pgmap v11179: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:11:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:11:12.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:12 smithi022 ceph-mon[31360]: pgmap v11179: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:12.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:12 smithi022 ceph-mon[40250]: pgmap v11179: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:14 smithi122 ceph-mon[32255]: pgmap v11180: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:14 smithi022 ceph-mon[31360]: pgmap v11180: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:14 smithi022 ceph-mon[40250]: pgmap v11180: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:15.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:15.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:15.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:15.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:16 smithi122 ceph-mon[32255]: pgmap v11181: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2570857450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:16 smithi022 ceph-mon[31360]: pgmap v11181: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2570857450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:16 smithi022 ceph-mon[40250]: pgmap v11181: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2570857450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:11:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:18 smithi122 ceph-mon[32255]: pgmap v11182: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:18 smithi022 ceph-mon[31360]: pgmap v11182: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:18 smithi022 ceph-mon[40250]: pgmap v11182: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:20] "GET /metrics HTTP/1.1" 200 202414 "" "Prometheus/2.18.1" 2021-12-16T00:11:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:20 smithi122 ceph-mon[32255]: pgmap v11183: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:20 smithi022 ceph-mon[31360]: pgmap v11183: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:20 smithi022 ceph-mon[40250]: pgmap v11183: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:20.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:20.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:20.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:21 smithi122 ceph-mon[32255]: pgmap v11184: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4163772531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:21 smithi022 ceph-mon[31360]: pgmap v11184: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4163772531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:21 smithi022 ceph-mon[40250]: pgmap v11184: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4163772531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:22.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:11:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:11:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:11:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:11:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:23 smithi122 ceph-mon[32255]: pgmap v11185: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:23 smithi022 ceph-mon[31360]: pgmap v11185: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:23 smithi022 ceph-mon[40250]: pgmap v11185: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:25.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:26.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:26.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:26.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:26 smithi122 ceph-mon[32255]: pgmap v11186: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:26 smithi022 ceph-mon[31360]: pgmap v11186: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:26 smithi022 ceph-mon[40250]: pgmap v11186: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4025986933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4025986933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:11:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4025986933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:28 smithi122 ceph-mon[32255]: pgmap v11187: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:28 smithi022 ceph-mon[31360]: pgmap v11187: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:28 smithi022 ceph-mon[40250]: pgmap v11187: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:30] "GET /metrics HTTP/1.1" 200 202414 "" "Prometheus/2.18.1" 2021-12-16T00:11:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:30 smithi122 ceph-mon[32255]: pgmap v11188: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:30 smithi022 ceph-mon[31360]: pgmap v11188: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:30 smithi022 ceph-mon[40250]: pgmap v11188: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:31.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:31.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:31.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:31.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:11:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:11:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:32 smithi122 ceph-mon[32255]: pgmap v11189: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1899492470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:11:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:11:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:32 smithi022 ceph-mon[31360]: pgmap v11189: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1899492470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:32 smithi022 ceph-mon[40250]: pgmap v11189: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1899492470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:34 smithi122 ceph-mon[32255]: pgmap v11190: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:11:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:11:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:11:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:11:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[31360]: pgmap v11190: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:11:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:11:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:11:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:11:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[40250]: pgmap v11190: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:11:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:11:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:11:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:11:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:36 smithi122 ceph-mon[32255]: pgmap v11191: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:36.704 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:36.705 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:36 smithi022 ceph-mon[40250]: pgmap v11191: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:36.705 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:36 smithi022 ceph-mon[31360]: pgmap v11191: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:37.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:37.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:37.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:37.321 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3061244792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3061244792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:11:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3061244792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:38 smithi122 ceph-mon[32255]: pgmap v11192: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:38 smithi022 ceph-mon[31360]: pgmap v11192: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:38 smithi022 ceph-mon[40250]: pgmap v11192: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:40 smithi122 ceph-mon[32255]: pgmap v11193: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:40] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:11:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:40 smithi022 ceph-mon[31360]: pgmap v11193: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:40 smithi022 ceph-mon[40250]: pgmap v11193: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:42.073 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:11:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:11:42.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:42.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:42.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:42 smithi122 ceph-mon[32255]: pgmap v11194: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:11:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:11:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:42 smithi022 ceph-mon[31360]: pgmap v11194: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:42 smithi022 ceph-mon[40250]: pgmap v11194: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1497215789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1497215789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1497215789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:44 smithi122 ceph-mon[32255]: pgmap v11195: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:44 smithi022 ceph-mon[31360]: pgmap v11195: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:44 smithi022 ceph-mon[40250]: pgmap v11195: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:46 smithi122 ceph-mon[32255]: pgmap v11196: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:46 smithi022 ceph-mon[31360]: pgmap v11196: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:46 smithi022 ceph-mon[40250]: pgmap v11196: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:47.443 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:11:47.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:47.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:47.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:48 smithi122 ceph-mon[32255]: pgmap v11197: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2914390415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:48 smithi022 ceph-mon[31360]: pgmap v11197: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2914390415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:48 smithi022 ceph-mon[40250]: pgmap v11197: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2914390415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:50 smithi122 ceph-mon[32255]: pgmap v11198: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:50] "GET /metrics HTTP/1.1" 200 202414 "" "Prometheus/2.18.1" 2021-12-16T00:11:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:50 smithi022 ceph-mon[31360]: pgmap v11198: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:50 smithi022 ceph-mon[40250]: pgmap v11198: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:11:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:11:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:52 smithi122 ceph-mon[32255]: pgmap v11199: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:11:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:11:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:52 smithi022 ceph-mon[31360]: pgmap v11199: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:52 smithi022 ceph-mon[40250]: pgmap v11199: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:52.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:53.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:53.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:53.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2631626137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2631626137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:11:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2631626137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:54 smithi122 ceph-mon[32255]: pgmap v11200: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:54 smithi022 ceph-mon[31360]: pgmap v11200: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:54 smithi022 ceph-mon[40250]: pgmap v11200: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:56 smithi122 ceph-mon[32255]: pgmap v11201: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:56 smithi022 ceph-mon[31360]: pgmap v11201: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:56 smithi022 ceph-mon[40250]: pgmap v11201: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:11:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:11:58.184 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:11:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:58 smithi022 ceph-mon[31360]: pgmap v11202: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:58 smithi022 ceph-mon[40250]: pgmap v11202: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:58.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:11:58.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:11:58.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:11:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:58 smithi122 ceph-mon[32255]: pgmap v11202: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:11:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:11:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2447866296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:11:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2447866296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:11:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:11:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2447866296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:00 smithi122 ceph-mon[32255]: pgmap v11203: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:00] "GET /metrics HTTP/1.1" 200 202414 "" "Prometheus/2.18.1" 2021-12-16T00:12:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:00 smithi022 ceph-mon[31360]: pgmap v11203: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:00 smithi022 ceph-mon[40250]: pgmap v11203: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:01 smithi122 ceph-mon[32255]: pgmap v11204: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:01 smithi022 ceph-mon[31360]: pgmap v11204: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:01 smithi022 ceph-mon[40250]: pgmap v11204: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:12:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:12:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:12:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:12:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:03.557 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:03 smithi122 ceph-mon[32255]: pgmap v11205: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:03 smithi022 ceph-mon[31360]: pgmap v11205: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:03 smithi022 ceph-mon[40250]: pgmap v11205: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:03.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:03.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:03.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2315493747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2315493747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2315493747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:06 smithi122 ceph-mon[32255]: pgmap v11206: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:06 smithi022 ceph-mon[31360]: pgmap v11206: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:06 smithi022 ceph-mon[40250]: pgmap v11206: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:12:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:08 smithi122 ceph-mon[32255]: pgmap v11207: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:08 smithi022 ceph-mon[31360]: pgmap v11207: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:08 smithi022 ceph-mon[40250]: pgmap v11207: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:12:08.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:09.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:09.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:09.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/226020700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:12:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:12:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:12:09.715 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/226020700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:09.716 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:12:09.716 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:12:09.716 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:12:09.716 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/226020700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:09.717 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:12:09.717 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:12:09.717 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:12:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:10 smithi122 ceph-mon[32255]: pgmap v11208: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:12:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:10] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:12:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:10 smithi022 ceph-mon[31360]: pgmap v11208: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:12:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:10 smithi022 ceph-mon[40250]: pgmap v11208: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:12:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:12:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:12:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:12 smithi122 ceph-mon[32255]: pgmap v11209: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:12:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:12:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:12 smithi022 ceph-mon[31360]: pgmap v11209: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:12 smithi022 ceph-mon[40250]: pgmap v11209: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:14.296 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:14.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:14.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:14.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:14.667 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:14 smithi022 ceph-mon[31360]: pgmap v11210: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:14.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:14 smithi022 ceph-mon[40250]: pgmap v11210: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:14 smithi122 ceph-mon[32255]: pgmap v11210: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3953520525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3953520525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3953520525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:16 smithi122 ceph-mon[32255]: pgmap v11211: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:16 smithi022 ceph-mon[31360]: pgmap v11211: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:16 smithi022 ceph-mon[40250]: pgmap v11211: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:12:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:18 smithi122 ceph-mon[32255]: pgmap v11212: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:18 smithi022 ceph-mon[31360]: pgmap v11212: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:18 smithi022 ceph-mon[40250]: pgmap v11212: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:19.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:20] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:12:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:20 smithi122 ceph-mon[32255]: pgmap v11213: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/667494080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:20 smithi022 ceph-mon[31360]: pgmap v11213: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/667494080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:20 smithi022 ceph-mon[40250]: pgmap v11213: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/667494080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:22.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:12:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:12:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:22 smithi122 ceph-mon[32255]: pgmap v11214: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:12:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:12:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:22 smithi022 ceph-mon[31360]: pgmap v11214: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:22 smithi022 ceph-mon[40250]: pgmap v11214: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:24 smithi122 ceph-mon[32255]: pgmap v11215: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:24 smithi022 ceph-mon[31360]: pgmap v11215: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:24 smithi022 ceph-mon[40250]: pgmap v11215: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:25.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:25.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:26 smithi122 ceph-mon[32255]: pgmap v11216: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/707809797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:26 smithi022 ceph-mon[31360]: pgmap v11216: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/707809797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:26 smithi022 ceph-mon[40250]: pgmap v11216: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/707809797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:12:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:28 smithi122 ceph-mon[32255]: pgmap v11217: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:28 smithi022 ceph-mon[31360]: pgmap v11217: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:28 smithi022 ceph-mon[40250]: pgmap v11217: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:30.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:30 smithi122 ceph-mon[32255]: pgmap v11218: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:30] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:12:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:30 smithi022 ceph-mon[31360]: pgmap v11218: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:30 smithi022 ceph-mon[40250]: pgmap v11218: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:30.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4066122336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4066122336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4066122336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:32.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:12:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:12:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:32 smithi122 ceph-mon[32255]: pgmap v11219: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:12:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:12:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:32 smithi022 ceph-mon[31360]: pgmap v11219: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:32 smithi022 ceph-mon[40250]: pgmap v11219: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:34 smithi122 ceph-mon[32255]: pgmap v11220: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:12:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:12:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:12:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:12:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[40250]: pgmap v11220: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:12:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:12:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:12:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:12:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[31360]: pgmap v11220: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:12:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:12:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:12:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:12:35.782 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:36.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:36.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:36 smithi022 ceph-mon[31360]: pgmap v11221: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1108028585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:36 smithi022 ceph-mon[40250]: pgmap v11221: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1108028585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:36 smithi122 ceph-mon[32255]: pgmap v11221: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:36.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1108028585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:12:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:38 smithi122 ceph-mon[32255]: pgmap v11222: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:38 smithi022 ceph-mon[31360]: pgmap v11222: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:38 smithi022 ceph-mon[40250]: pgmap v11222: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:40 smithi122 ceph-mon[32255]: pgmap v11223: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:40] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:12:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:40 smithi022 ceph-mon[31360]: pgmap v11223: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:40 smithi022 ceph-mon[40250]: pgmap v11223: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:41.150 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:41.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:41.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:41.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:42.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:12:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:12:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:42 smithi122 ceph-mon[32255]: pgmap v11224: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1205302528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:12:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:12:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:42 smithi022 ceph-mon[31360]: pgmap v11224: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1205302528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:42 smithi022 ceph-mon[40250]: pgmap v11224: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1205302528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:44 smithi122 ceph-mon[32255]: pgmap v11225: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:44 smithi022 ceph-mon[31360]: pgmap v11225: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:44 smithi022 ceph-mon[40250]: pgmap v11225: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:46.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:46 smithi122 ceph-mon[32255]: pgmap v11226: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:46 smithi022 ceph-mon[31360]: pgmap v11226: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:46 smithi022 ceph-mon[40250]: pgmap v11226: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:46.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:46.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:46.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2128680915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2128680915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:12:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2128680915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:48 smithi122 ceph-mon[32255]: pgmap v11227: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:48 smithi022 ceph-mon[31360]: pgmap v11227: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:48 smithi022 ceph-mon[40250]: pgmap v11227: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:50] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:12:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:50 smithi122 ceph-mon[32255]: pgmap v11228: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:50 smithi022 ceph-mon[31360]: pgmap v11228: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:50 smithi022 ceph-mon[40250]: pgmap v11228: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:51.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:52.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:12:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:12:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:52 smithi022 ceph-mon[31360]: pgmap v11229: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4208286828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:52 smithi022 ceph-mon[40250]: pgmap v11229: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4208286828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:52 smithi122 ceph-mon[32255]: pgmap v11229: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4208286828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:12:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:12:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:12:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:54 smithi122 ceph-mon[32255]: pgmap v11230: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:54 smithi022 ceph-mon[31360]: pgmap v11230: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:54 smithi022 ceph-mon[40250]: pgmap v11230: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:56 smithi122 ceph-mon[32255]: pgmap v11231: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:56 smithi022 ceph-mon[31360]: pgmap v11231: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:56 smithi022 ceph-mon[40250]: pgmap v11231: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:12:57.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:12:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:12:57.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:12:57.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:12:57.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:12:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:58 smithi122 ceph-mon[32255]: pgmap v11232: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:12:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1740462440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:58 smithi022 ceph-mon[31360]: pgmap v11232: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:12:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1740462440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:12:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:58 smithi022 ceph-mon[40250]: pgmap v11232: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:12:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:12:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1740462440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:00] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:13:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:00 smithi122 ceph-mon[32255]: pgmap v11233: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:00 smithi022 ceph-mon[31360]: pgmap v11233: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:00 smithi022 ceph-mon[40250]: pgmap v11233: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:13:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:13:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:02.637 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:02 smithi022 ceph-mon[40250]: pgmap v11234: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:02.637 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:02 smithi022 ceph-mon[31360]: pgmap v11234: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:02 smithi122 ceph-mon[32255]: pgmap v11234: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:13:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:13:03.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:03.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:03.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/85284583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/85284583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/85284583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:04 smithi122 ceph-mon[32255]: pgmap v11235: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:04 smithi022 ceph-mon[31360]: pgmap v11235: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:04 smithi022 ceph-mon[40250]: pgmap v11235: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:06 smithi122 ceph-mon[32255]: pgmap v11236: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:06 smithi022 ceph-mon[31360]: pgmap v11236: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:06 smithi022 ceph-mon[40250]: pgmap v11236: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:13:08.008 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:08.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:08.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:08.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:08 smithi122 ceph-mon[32255]: pgmap v11237: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3401529563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:08 smithi022 ceph-mon[31360]: pgmap v11237: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3401529563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:08 smithi022 ceph-mon[40250]: pgmap v11237: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3401529563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:09 smithi122 ceph-mon[32255]: pgmap v11238: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:09.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:09 smithi022 ceph-mon[40250]: pgmap v11238: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:09.768 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:09 smithi022 ceph-mon[31360]: pgmap v11238: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:10.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:10] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:13:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:13:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:13:10.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:13:10.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:13:10.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:13:10.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:13:10.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:13:10.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:13:10.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:13:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:13:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:13:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:13:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:11 smithi122 ceph-mon[32255]: pgmap v11239: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:11 smithi022 ceph-mon[31360]: pgmap v11239: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:11 smithi022 ceph-mon[40250]: pgmap v11239: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:12.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:13:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:13:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:13:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:13:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:13.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:13.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:13.752 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:13 smithi022 ceph-mon[31360]: pgmap v11240: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:13.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:13 smithi022 ceph-mon[40250]: pgmap v11240: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:13 smithi122 ceph-mon[32255]: pgmap v11240: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/831098000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/831098000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/831098000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:16 smithi122 ceph-mon[32255]: pgmap v11241: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:16 smithi022 ceph-mon[31360]: pgmap v11241: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:16 smithi022 ceph-mon[40250]: pgmap v11241: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:13:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:18 smithi122 ceph-mon[32255]: pgmap v11242: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:18.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:18.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:18 smithi022 ceph-mon[40250]: pgmap v11242: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:18.755 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:18 smithi022 ceph-mon[31360]: pgmap v11242: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:19.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:19.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:19.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3741107194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3741107194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3741107194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:20 smithi122 ceph-mon[32255]: pgmap v11243: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:20] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:13:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:20 smithi022 ceph-mon[31360]: pgmap v11243: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:20 smithi022 ceph-mon[40250]: pgmap v11243: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:22.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:13:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:13:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:13:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:13:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:22 smithi122 ceph-mon[32255]: pgmap v11244: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:22 smithi022 ceph-mon[31360]: pgmap v11244: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:22 smithi022 ceph-mon[40250]: pgmap v11244: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:24.123 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:24.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:24.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:24.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:24.495 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:24 smithi022 ceph-mon[31360]: pgmap v11245: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:24.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:24 smithi022 ceph-mon[40250]: pgmap v11245: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:24 smithi122 ceph-mon[32255]: pgmap v11245: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2609158136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2609158136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2609158136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:26 smithi122 ceph-mon[32255]: pgmap v11246: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:26 smithi022 ceph-mon[31360]: pgmap v11246: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:26 smithi022 ceph-mon[40250]: pgmap v11246: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:13:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:28 smithi122 ceph-mon[32255]: pgmap v11247: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:28.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:28 smithi022 ceph-mon[31360]: pgmap v11247: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:28.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:28 smithi022 ceph-mon[40250]: pgmap v11247: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:29.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:29.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:29.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:29.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:30] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:13:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:30 smithi122 ceph-mon[32255]: pgmap v11248: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/221482634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:30 smithi022 ceph-mon[31360]: pgmap v11248: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/221482634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:30 smithi022 ceph-mon[40250]: pgmap v11248: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/221482634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:32.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:13:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:13:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:32 smithi122 ceph-mon[32255]: pgmap v11249: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:13:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:13:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:32 smithi022 ceph-mon[31360]: pgmap v11249: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:32 smithi022 ceph-mon[40250]: pgmap v11249: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:34 smithi122 ceph-mon[32255]: pgmap v11250: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:13:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:13:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:13:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[31360]: pgmap v11250: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:13:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:13:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[40250]: pgmap v11250: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:13:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:13:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:13:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:35.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:35.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:35.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/471319381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/471319381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/471319381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:36 smithi122 ceph-mon[32255]: pgmap v11251: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:36 smithi022 ceph-mon[31360]: pgmap v11251: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:36 smithi022 ceph-mon[40250]: pgmap v11251: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:13:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:38 smithi122 ceph-mon[32255]: pgmap v11252: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:38 smithi022 ceph-mon[40250]: pgmap v11252: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:38 smithi022 ceph-mon[31360]: pgmap v11252: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:40.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:40] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:13:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:40 smithi122 ceph-mon[32255]: pgmap v11253: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:40 smithi022 ceph-mon[31360]: pgmap v11253: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:40 smithi022 ceph-mon[40250]: pgmap v11253: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:40.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:40.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:40.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/761811245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/761811245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/761811245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:42.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:13:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:13:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:13:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:13:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:42 smithi122 ceph-mon[32255]: pgmap v11254: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:42.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:42 smithi022 ceph-mon[31360]: pgmap v11254: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:42 smithi022 ceph-mon[40250]: pgmap v11254: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:43.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:43 smithi122 ceph-mon[32255]: pgmap v11255: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:43 smithi022 ceph-mon[31360]: pgmap v11255: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:43 smithi022 ceph-mon[40250]: pgmap v11255: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:45.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:45.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:45.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:45.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:46 smithi122 ceph-mon[32255]: pgmap v11256: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2963204696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:46 smithi022 ceph-mon[31360]: pgmap v11256: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2963204696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:46 smithi022 ceph-mon[40250]: pgmap v11256: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2963204696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:13:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:48 smithi122 ceph-mon[32255]: pgmap v11257: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:48 smithi022 ceph-mon[31360]: pgmap v11257: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:48 smithi022 ceph-mon[40250]: pgmap v11257: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:50] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:13:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:50 smithi122 ceph-mon[32255]: pgmap v11258: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:50 smithi022 ceph-mon[31360]: pgmap v11258: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:50 smithi022 ceph-mon[40250]: pgmap v11258: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:50.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:51.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:51.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:51.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:51 smithi122 ceph-mon[32255]: pgmap v11259: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2140102385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:51 smithi022 ceph-mon[31360]: pgmap v11259: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2140102385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:51 smithi022 ceph-mon[40250]: pgmap v11259: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2140102385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:52.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:13:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:13:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:13:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:13:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:13:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:53 smithi122 ceph-mon[32255]: pgmap v11260: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:53 smithi022 ceph-mon[31360]: pgmap v11260: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:53 smithi022 ceph-mon[40250]: pgmap v11260: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:56.347 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:13:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:56 smithi122 ceph-mon[32255]: pgmap v11261: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:56.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:13:56.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:13:56.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:13:56.717 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:56 smithi022 ceph-mon[31360]: pgmap v11261: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:56.717 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:56 smithi022 ceph-mon[40250]: pgmap v11261: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:13:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3074332623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3074332623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:13:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3074332623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:13:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:13:58 smithi122 ceph-mon[32255]: pgmap v11262: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:13:58 smithi022 ceph-mon[31360]: pgmap v11262: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:13:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:13:58 smithi022 ceph-mon[40250]: pgmap v11262: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:00 smithi122 ceph-mon[32255]: pgmap v11263: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:00] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:14:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:00 smithi022 ceph-mon[31360]: pgmap v11263: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:00 smithi022 ceph-mon[40250]: pgmap v11263: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:01.718 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:02.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:02.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:02.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:14:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:14:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:02 smithi122 ceph-mon[32255]: pgmap v11264: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2376989259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:14:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:14:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:02 smithi022 ceph-mon[31360]: pgmap v11264: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2376989259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:02 smithi022 ceph-mon[40250]: pgmap v11264: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2376989259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:04 smithi122 ceph-mon[32255]: pgmap v11265: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:04 smithi022 ceph-mon[31360]: pgmap v11265: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:04 smithi022 ceph-mon[40250]: pgmap v11265: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:06 smithi122 ceph-mon[32255]: pgmap v11266: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:06 smithi022 ceph-mon[31360]: pgmap v11266: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:06 smithi022 ceph-mon[40250]: pgmap v11266: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:07.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:07.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:07.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:07.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:14:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:08 smithi122 ceph-mon[32255]: pgmap v11267: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2174499042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:08 smithi022 ceph-mon[31360]: pgmap v11267: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2174499042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:08 smithi022 ceph-mon[40250]: pgmap v11267: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2174499042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:10] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:14:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:10 smithi122 ceph-mon[32255]: pgmap v11268: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:14:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:14:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:14:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[31360]: pgmap v11268: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:14:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:14:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:14:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[40250]: pgmap v11268: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:14:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:14:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:14:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:14:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:11 smithi122 ceph-mon[32255]: pgmap v11269: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:14:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:11 smithi022 ceph-mon[31360]: pgmap v11269: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:14:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:11 smithi022 ceph-mon[40250]: pgmap v11269: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:12.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:14:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:14:12.459 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:12.658 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:14:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:14:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:12.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:12.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:12.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1972967289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:13 smithi122 ceph-mon[32255]: pgmap v11270: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1972967289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:13 smithi022 ceph-mon[31360]: pgmap v11270: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1972967289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:13 smithi022 ceph-mon[40250]: pgmap v11270: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:16 smithi122 ceph-mon[32255]: pgmap v11271: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:16 smithi022 ceph-mon[31360]: pgmap v11271: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:16 smithi022 ceph-mon[40250]: pgmap v11271: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:14:17.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:18 smithi022 ceph-mon[31360]: pgmap v11272: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1694448363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:18 smithi022 ceph-mon[40250]: pgmap v11272: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1694448363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:18 smithi122 ceph-mon[32255]: pgmap v11272: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1694448363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:20] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:14:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:20 smithi122 ceph-mon[32255]: pgmap v11273: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:20 smithi022 ceph-mon[31360]: pgmap v11273: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:20 smithi022 ceph-mon[40250]: pgmap v11273: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:22.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:14:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:14:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:22 smithi122 ceph-mon[32255]: pgmap v11274: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:14:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:14:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:22 smithi022 ceph-mon[31360]: pgmap v11274: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:22 smithi022 ceph-mon[40250]: pgmap v11274: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:23.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:23.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:23.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:24 smithi122 ceph-mon[32255]: pgmap v11275: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2092214224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:24 smithi022 ceph-mon[31360]: pgmap v11275: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2092214224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:24 smithi022 ceph-mon[40250]: pgmap v11275: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2092214224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:26 smithi122 ceph-mon[32255]: pgmap v11276: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:26 smithi022 ceph-mon[31360]: pgmap v11276: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:26 smithi022 ceph-mon[40250]: pgmap v11276: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:14:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:28 smithi122 ceph-mon[32255]: pgmap v11277: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:28 smithi022 ceph-mon[31360]: pgmap v11277: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:28 smithi022 ceph-mon[40250]: pgmap v11277: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:28.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:28.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:28.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1306021608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1306021608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1306021608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:30] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:14:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:30 smithi122 ceph-mon[32255]: pgmap v11278: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:30 smithi022 ceph-mon[31360]: pgmap v11278: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:30 smithi022 ceph-mon[40250]: pgmap v11278: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:32.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:14:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:14:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:32 smithi122 ceph-mon[32255]: pgmap v11279: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:14:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:14:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:32 smithi022 ceph-mon[31360]: pgmap v11279: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:32 smithi022 ceph-mon[40250]: pgmap v11279: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:34.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:34.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:34.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:34 smithi122 ceph-mon[32255]: pgmap v11280: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:14:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:14:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:14:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:14:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1610488345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[31360]: pgmap v11280: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:14:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:14:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:14:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:14:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1610488345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[40250]: pgmap v11280: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:14:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:14:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:14:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:14:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1610488345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:36 smithi122 ceph-mon[32255]: pgmap v11281: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:36 smithi022 ceph-mon[31360]: pgmap v11281: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:36 smithi022 ceph-mon[40250]: pgmap v11281: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:14:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:38 smithi122 ceph-mon[32255]: pgmap v11282: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:38 smithi022 ceph-mon[31360]: pgmap v11282: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:38 smithi022 ceph-mon[40250]: pgmap v11282: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:39.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:40] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:14:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:40 smithi122 ceph-mon[32255]: pgmap v11283: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396074509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:40 smithi022 ceph-mon[31360]: pgmap v11283: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396074509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:40 smithi022 ceph-mon[40250]: pgmap v11283: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396074509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:42.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:14:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:14:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:42 smithi122 ceph-mon[32255]: pgmap v11284: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:14:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:14:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:42 smithi022 ceph-mon[31360]: pgmap v11284: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:42 smithi022 ceph-mon[40250]: pgmap v11284: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:44.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:44.684 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:44 smithi022 ceph-mon[40250]: pgmap v11285: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:44.684 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:44 smithi022 ceph-mon[31360]: pgmap v11285: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:44 smithi122 ceph-mon[32255]: pgmap v11285: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:45.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:45.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:45.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3459233508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3459233508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3459233508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:46 smithi122 ceph-mon[32255]: pgmap v11286: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:46 smithi022 ceph-mon[31360]: pgmap v11286: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:46 smithi022 ceph-mon[40250]: pgmap v11286: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:47 smithi122 ceph-mon[32255]: pgmap v11287: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:47 smithi022 ceph-mon[31360]: pgmap v11287: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:14:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:47 smithi022 ceph-mon[40250]: pgmap v11287: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:50.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:49 smithi022 ceph-mon[31360]: pgmap v11288: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:49 smithi022 ceph-mon[40250]: pgmap v11288: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:50.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:50.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:50.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:50] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:14:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:49 smithi122 ceph-mon[32255]: pgmap v11288: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2524184477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2524184477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2524184477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:52.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:14:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:14:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:52 smithi122 ceph-mon[32255]: pgmap v11289: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:14:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:14:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:52 smithi022 ceph-mon[31360]: pgmap v11289: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:52 smithi022 ceph-mon[40250]: pgmap v11289: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:14:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:54 smithi122 ceph-mon[32255]: pgmap v11290: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:54 smithi022 ceph-mon[31360]: pgmap v11290: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:54.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:54 smithi022 ceph-mon[40250]: pgmap v11290: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:55.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:14:55.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:14:55.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:14:55.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:14:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:56 smithi122 ceph-mon[32255]: pgmap v11291: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1995554859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:56 smithi022 ceph-mon[31360]: pgmap v11291: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:56.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1995554859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:56 smithi022 ceph-mon[40250]: pgmap v11291: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:14:56.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1995554859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:14:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:14:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:14:58 smithi122 ceph-mon[32255]: pgmap v11292: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:14:58 smithi022 ceph-mon[31360]: pgmap v11292: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:14:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:14:58 smithi022 ceph-mon[40250]: pgmap v11292: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:00 smithi122 ceph-mon[32255]: pgmap v11293: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:00] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:15:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:00 smithi022 ceph-mon[31360]: pgmap v11293: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:00 smithi022 ceph-mon[40250]: pgmap v11293: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:00.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:01.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:01.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:01.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2848216230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2848216230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2848216230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:02.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:15:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:15:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:02 smithi122 ceph-mon[32255]: pgmap v11294: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:15:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:15:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:02 smithi022 ceph-mon[31360]: pgmap v11294: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:02 smithi022 ceph-mon[40250]: pgmap v11294: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:04 smithi122 ceph-mon[32255]: pgmap v11295: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:04 smithi022 ceph-mon[31360]: pgmap v11295: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:04 smithi022 ceph-mon[40250]: pgmap v11295: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:06.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:06 smithi022 ceph-mon[31360]: pgmap v11296: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:06 smithi022 ceph-mon[40250]: pgmap v11296: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:06.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:06.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:06.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:06 smithi122 ceph-mon[32255]: pgmap v11296: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2577559760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2577559760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:15:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2577559760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:08 smithi122 ceph-mon[32255]: pgmap v11297: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:08 smithi022 ceph-mon[31360]: pgmap v11297: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:08 smithi022 ceph-mon[40250]: pgmap v11297: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:10 smithi122 ceph-mon[32255]: pgmap v11298: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:10] "GET /metrics HTTP/1.1" 200 202413 "" "Prometheus/2.18.1" 2021-12-16T00:15:10.690 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:10 smithi022 ceph-mon[40250]: pgmap v11298: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:10.691 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:10 smithi022 ceph-mon[31360]: pgmap v11298: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:15:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:15:11.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:15:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:15:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:15:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:15:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:15:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:15:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:15:11.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:11.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:11.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:12.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:15:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:15:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:15:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:15:12.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:12 smithi122 ceph-mon[32255]: pgmap v11299: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:12.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3064671761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:12 smithi022 ceph-mon[31360]: pgmap v11299: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3064671761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:12 smithi022 ceph-mon[40250]: pgmap v11299: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3064671761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:14 smithi122 ceph-mon[32255]: pgmap v11300: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:14 smithi022 ceph-mon[31360]: pgmap v11300: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:14 smithi022 ceph-mon[40250]: pgmap v11300: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3090M 2021-12-16T00:15:15.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:15.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: pgmap v11301: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:15.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3090M 2021-12-16T00:15:15.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:15.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: pgmap v11301: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:15.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:15.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:15:15.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3090M 2021-12-16T00:15:15.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:15.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: pgmap v11301: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:15.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:16.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:16.920 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:16.920 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:15:17.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:17.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:17.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:17.610 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:15:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:17 smithi122 ceph-mon[32255]: pgmap v11302: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3808367350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:17 smithi022 ceph-mon[31360]: pgmap v11302: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3808367350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:17 smithi022 ceph-mon[40250]: pgmap v11302: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3808367350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:19 smithi022 ceph-mon[31360]: pgmap v11303: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:19 smithi022 ceph-mon[40250]: pgmap v11303: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:20] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:15:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:19 smithi122 ceph-mon[32255]: pgmap v11303: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:22.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:22.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:15:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:15:22.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:22.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:22.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:22.658 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:22 smithi022 ceph-mon[31360]: pgmap v11304: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:22.659 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:22 smithi022 ceph-mon[40250]: pgmap v11304: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:22 smithi122 ceph-mon[32255]: pgmap v11304: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:15:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:15:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2393519075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2393519075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2393519075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:24 smithi122 ceph-mon[32255]: pgmap v11305: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:24 smithi022 ceph-mon[31360]: pgmap v11305: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:24 smithi022 ceph-mon[40250]: pgmap v11305: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:26 smithi122 ceph-mon[32255]: pgmap v11306: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:26 smithi022 ceph-mon[31360]: pgmap v11306: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:26 smithi022 ceph-mon[40250]: pgmap v11306: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:27.659 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:15:27.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:28.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:28.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:28.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:28 smithi122 ceph-mon[32255]: pgmap v11307: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3375789818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:28 smithi022 ceph-mon[31360]: pgmap v11307: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3375789818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:28 smithi022 ceph-mon[40250]: pgmap v11307: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3375789818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:30 smithi122 ceph-mon[32255]: pgmap v11308: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:30] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:15:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:30 smithi022 ceph-mon[31360]: pgmap v11308: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:30 smithi022 ceph-mon[40250]: pgmap v11308: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:32.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:15:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:15:32.365 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:15:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:15:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:32 smithi122 ceph-mon[32255]: pgmap v11309: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:32 smithi022 ceph-mon[31360]: pgmap v11309: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:32 smithi022 ceph-mon[40250]: pgmap v11309: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:33.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:33.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:33.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:33.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:33.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:34 smithi122 ceph-mon[32255]: pgmap v11310: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3583832209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:15:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:15:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:15:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:15:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[31360]: pgmap v11310: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3583832209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:15:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:15:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:15:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:15:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[40250]: pgmap v11310: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3583832209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:15:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:15:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:15:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:15:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:36 smithi122 ceph-mon[32255]: pgmap v11311: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:36 smithi022 ceph-mon[31360]: pgmap v11311: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:36 smithi022 ceph-mon[40250]: pgmap v11311: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:15:38.397 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:38 smithi122 ceph-mon[32255]: pgmap v11312: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:38.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:38.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:38.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:38.769 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:38 smithi022 ceph-mon[40250]: pgmap v11312: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:38.769 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:38 smithi022 ceph-mon[31360]: pgmap v11312: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1142638367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:39.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1142638367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:39.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1142638367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:40 smithi122 ceph-mon[32255]: pgmap v11313: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:40] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:15:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:40 smithi022 ceph-mon[31360]: pgmap v11313: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:40 smithi022 ceph-mon[40250]: pgmap v11313: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:42.385 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:15:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:15:42.385 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:15:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:15:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:42 smithi122 ceph-mon[32255]: pgmap v11314: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:42 smithi022 ceph-mon[31360]: pgmap v11314: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:42 smithi022 ceph-mon[40250]: pgmap v11314: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:43.770 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:43.770 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:43.771 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:44.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:44.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:44.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:44 smithi122 ceph-mon[32255]: pgmap v11315: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/776961575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:44 smithi022 ceph-mon[31360]: pgmap v11315: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/776961575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:44 smithi022 ceph-mon[40250]: pgmap v11315: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/776961575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:46 smithi122 ceph-mon[32255]: pgmap v11316: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:46 smithi022 ceph-mon[31360]: pgmap v11316: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:46 smithi022 ceph-mon[40250]: pgmap v11316: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:15:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:48 smithi122 ceph-mon[32255]: pgmap v11317: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:48 smithi022 ceph-mon[31360]: pgmap v11317: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:48 smithi022 ceph-mon[40250]: pgmap v11317: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:49.139 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:49.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:49.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:49.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:49 smithi022 ceph-mon[31360]: pgmap v11318: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:49 smithi022 ceph-mon[40250]: pgmap v11318: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:49 smithi122 ceph-mon[32255]: pgmap v11318: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:50] "GET /metrics HTTP/1.1" 200 202417 "" "Prometheus/2.18.1" 2021-12-16T00:15:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3365143687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3365143687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3365143687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:51 smithi022 ceph-mon[31360]: pgmap v11319: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:51 smithi022 ceph-mon[40250]: pgmap v11319: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:51 smithi122 ceph-mon[32255]: pgmap v11319: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:52.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:15:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:15:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:15:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:15:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:15:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:53 smithi022 ceph-mon[31360]: pgmap v11320: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:53 smithi022 ceph-mon[40250]: pgmap v11320: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:53 smithi122 ceph-mon[32255]: pgmap v11320: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:54.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:15:54.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:15:54.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:15:54.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:15:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3735225015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3735225015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3735225015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:15:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:56 smithi122 ceph-mon[32255]: pgmap v11321: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:56 smithi022 ceph-mon[31360]: pgmap v11321: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:56 smithi022 ceph-mon[40250]: pgmap v11321: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:15:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:15:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:15:58 smithi122 ceph-mon[32255]: pgmap v11322: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:15:58 smithi022 ceph-mon[31360]: pgmap v11322: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:15:58 smithi022 ceph-mon[40250]: pgmap v11322: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:15:59.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:00.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:00.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:00.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:00] "GET /metrics HTTP/1.1" 200 202417 "" "Prometheus/2.18.1" 2021-12-16T00:16:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:00 smithi122 ceph-mon[32255]: pgmap v11323: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2779933092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:00 smithi022 ceph-mon[31360]: pgmap v11323: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2779933092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:00 smithi022 ceph-mon[40250]: pgmap v11323: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2779933092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:02.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:16:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:16:02.367 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:16:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:16:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:02 smithi122 ceph-mon[32255]: pgmap v11324: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:02 smithi022 ceph-mon[31360]: pgmap v11324: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:02 smithi022 ceph-mon[40250]: pgmap v11324: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:04 smithi122 ceph-mon[32255]: pgmap v11325: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:04 smithi022 ceph-mon[31360]: pgmap v11325: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:04 smithi022 ceph-mon[40250]: pgmap v11325: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:05.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:05.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:05.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:05.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:06 smithi122 ceph-mon[32255]: pgmap v11326: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2590307108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:06 smithi022 ceph-mon[31360]: pgmap v11326: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2590307108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:06 smithi022 ceph-mon[40250]: pgmap v11326: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2590307108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:16:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:08 smithi122 ceph-mon[32255]: pgmap v11327: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:08 smithi022 ceph-mon[31360]: pgmap v11327: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:08 smithi022 ceph-mon[40250]: pgmap v11327: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:10 smithi122 ceph-mon[32255]: pgmap v11328: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:10] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:16:10.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:10.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:10 smithi022 ceph-mon[31360]: pgmap v11328: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:10.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:10 smithi022 ceph-mon[40250]: pgmap v11328: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:10.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:10.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2673212468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2673212468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2673212468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:12.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:16:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:16:12.387 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:16:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:16:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:12 smithi122 ceph-mon[32255]: pgmap v11329: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:12 smithi022 ceph-mon[31360]: pgmap v11329: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:12 smithi022 ceph-mon[40250]: pgmap v11329: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:13.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:14 smithi122 ceph-mon[32255]: pgmap v11330: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:14.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:14 smithi022 ceph-mon[31360]: pgmap v11330: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:14 smithi022 ceph-mon[40250]: pgmap v11330: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:15.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:16.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:16.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:16.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:16 smithi122 ceph-mon[32255]: pgmap v11331: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:16:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:16:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:16:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:16:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2704123618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[31360]: pgmap v11331: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:16:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:16:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:16:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:16:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2704123618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[40250]: pgmap v11331: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:16:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:16:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:16:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:16:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2704123618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:16:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:18 smithi122 ceph-mon[32255]: pgmap v11332: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:18 smithi022 ceph-mon[31360]: pgmap v11332: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:18 smithi022 ceph-mon[40250]: pgmap v11332: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:20] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:16:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:20 smithi122 ceph-mon[32255]: pgmap v11333: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:20 smithi022 ceph-mon[31360]: pgmap v11333: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:20 smithi022 ceph-mon[40250]: pgmap v11333: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:21.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:21.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:21.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:21.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:21.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:21 smithi022 ceph-mon[31360]: pgmap v11334: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:21.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:21 smithi022 ceph-mon[40250]: pgmap v11334: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:21 smithi122 ceph-mon[32255]: pgmap v11334: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:22.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:16:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:16:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:16:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:16:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3906587018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3906587018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3906587018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:23 smithi022 ceph-mon[31360]: pgmap v11335: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:23 smithi022 ceph-mon[40250]: pgmap v11335: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:23 smithi122 ceph-mon[32255]: pgmap v11335: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:26 smithi122 ceph-mon[32255]: pgmap v11336: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:26.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:26.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:26 smithi022 ceph-mon[40250]: pgmap v11336: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:26.739 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:26 smithi022 ceph-mon[31360]: pgmap v11336: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:27.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:27.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:27.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1715642465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1715642465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:16:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1715642465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:28 smithi122 ceph-mon[32255]: pgmap v11337: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:28 smithi022 ceph-mon[31360]: pgmap v11337: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:28 smithi022 ceph-mon[40250]: pgmap v11337: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:30] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:16:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:30 smithi122 ceph-mon[32255]: pgmap v11338: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:30 smithi022 ceph-mon[31360]: pgmap v11338: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:30 smithi022 ceph-mon[40250]: pgmap v11338: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:32.115 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:32.378 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:16:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:16:32.378 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:16:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:16:32.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:32.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:32.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:32 smithi122 ceph-mon[32255]: pgmap v11339: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:32 smithi022 ceph-mon[31360]: pgmap v11339: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:32 smithi022 ceph-mon[40250]: pgmap v11339: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1703772553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1703772553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1703772553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:34 smithi122 ceph-mon[32255]: pgmap v11340: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:16:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:16:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:16:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:16:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[31360]: pgmap v11340: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:16:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:16:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:16:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:16:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[40250]: pgmap v11340: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:16:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:16:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:16:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:16:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:36 smithi122 ceph-mon[32255]: pgmap v11341: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:36 smithi022 ceph-mon[31360]: pgmap v11341: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:36 smithi022 ceph-mon[40250]: pgmap v11341: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:16:37.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:37.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:37.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:38 smithi122 ceph-mon[32255]: pgmap v11342: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2248166974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:38 smithi022 ceph-mon[40250]: pgmap v11342: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2248166974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:38 smithi022 ceph-mon[31360]: pgmap v11342: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2248166974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:40 smithi122 ceph-mon[32255]: pgmap v11343: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:40] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:16:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:40 smithi022 ceph-mon[31360]: pgmap v11343: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:40 smithi022 ceph-mon[40250]: pgmap v11343: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:42.395 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:16:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:16:42.396 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:16:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:16:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:42 smithi122 ceph-mon[32255]: pgmap v11344: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:42 smithi022 ceph-mon[31360]: pgmap v11344: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:42 smithi022 ceph-mon[40250]: pgmap v11344: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:42.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:43.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:43.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:43.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1040559813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1040559813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1040559813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:44 smithi122 ceph-mon[32255]: pgmap v11345: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:45.707 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:44 smithi022 ceph-mon[31360]: pgmap v11345: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:45.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:44 smithi022 ceph-mon[40250]: pgmap v11345: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:46 smithi122 ceph-mon[32255]: pgmap v11346: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:46 smithi022 ceph-mon[31360]: pgmap v11346: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:46 smithi022 ceph-mon[40250]: pgmap v11346: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:16:48.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:48 smithi022 ceph-mon[31360]: pgmap v11347: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:48 smithi022 ceph-mon[40250]: pgmap v11347: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:48.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:48.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:48.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:48 smithi122 ceph-mon[32255]: pgmap v11347: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3380170002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:49 smithi122 ceph-mon[32255]: pgmap v11348: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3380170002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:49 smithi022 ceph-mon[31360]: pgmap v11348: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3380170002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:49 smithi022 ceph-mon[40250]: pgmap v11348: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:50] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:16:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:52 smithi122 ceph-mon[32255]: pgmap v11349: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:16:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:16:52.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:16:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:16:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:52 smithi022 ceph-mon[31360]: pgmap v11349: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:52 smithi022 ceph-mon[40250]: pgmap v11349: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:53.601 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:53.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:16:53.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:53.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:53.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:54 smithi122 ceph-mon[32255]: pgmap v11350: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2970010160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:54 smithi022 ceph-mon[31360]: pgmap v11350: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2970010160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:54 smithi022 ceph-mon[40250]: pgmap v11350: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2970010160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:56 smithi122 ceph-mon[32255]: pgmap v11351: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:56 smithi022 ceph-mon[31360]: pgmap v11351: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:56 smithi022 ceph-mon[40250]: pgmap v11351: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:16:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:16:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:58 smithi122 ceph-mon[32255]: pgmap v11352: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:58 smithi022 ceph-mon[31360]: pgmap v11352: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:58 smithi022 ceph-mon[40250]: pgmap v11352: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:16:58.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:16:59.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:16:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:16:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:16:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:16:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4107391519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:16:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4107391519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:16:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:16:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4107391519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:00] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:17:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:00 smithi122 ceph-mon[32255]: pgmap v11353: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:00 smithi022 ceph-mon[31360]: pgmap v11353: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:00 smithi022 ceph-mon[40250]: pgmap v11353: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:02.391 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:17:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:17:02.392 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:17:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:17:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:02 smithi122 ceph-mon[32255]: pgmap v11354: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:02 smithi022 ceph-mon[31360]: pgmap v11354: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:02 smithi022 ceph-mon[40250]: pgmap v11354: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:04.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:04 smithi122 ceph-mon[32255]: pgmap v11355: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:04.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:04.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:04.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:04.707 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:04 smithi022 ceph-mon[31360]: pgmap v11355: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:04.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:04 smithi022 ceph-mon[40250]: pgmap v11355: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3211178356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3211178356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3211178356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:06 smithi122 ceph-mon[32255]: pgmap v11356: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:06 smithi022 ceph-mon[31360]: pgmap v11356: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:06 smithi022 ceph-mon[40250]: pgmap v11356: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:17:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:08 smithi122 ceph-mon[32255]: pgmap v11357: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:08 smithi022 ceph-mon[31360]: pgmap v11357: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:08 smithi022 ceph-mon[40250]: pgmap v11357: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:09.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:10.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:10.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:10.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:10] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:17:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:10 smithi122 ceph-mon[32255]: pgmap v11358: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2923060128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:10 smithi022 ceph-mon[31360]: pgmap v11358: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2923060128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:10 smithi022 ceph-mon[40250]: pgmap v11358: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2923060128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:12.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:17:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:17:12.411 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:17:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:17:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:12 smithi122 ceph-mon[32255]: pgmap v11359: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:12 smithi022 ceph-mon[31360]: pgmap v11359: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:12 smithi022 ceph-mon[40250]: pgmap v11359: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:14 smithi122 ceph-mon[32255]: pgmap v11360: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:14 smithi022 ceph-mon[31360]: pgmap v11360: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:15.378 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:14 smithi022 ceph-mon[40250]: pgmap v11360: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:15.378 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:15.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:15.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:15.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:16 smithi122 ceph-mon[32255]: pgmap v11361: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2269627574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:17:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:17:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:17:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[31360]: pgmap v11361: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2269627574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:17:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:17:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:17:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[40250]: pgmap v11361: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2269627574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:17:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:17:16.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:17:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:17:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:17:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:17 smithi022 ceph-mon[31360]: pgmap v11362: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:17:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:17 smithi022 ceph-mon[40250]: pgmap v11362: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:17:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:17 smithi122 ceph-mon[32255]: pgmap v11362: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:19 smithi022 ceph-mon[31360]: pgmap v11363: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:19 smithi022 ceph-mon[40250]: pgmap v11363: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:20] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:17:20.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:19 smithi122 ceph-mon[32255]: pgmap v11363: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:20.452 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:20.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:20.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:20.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/371903242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/371903242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/371903242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:22 smithi122 ceph-mon[32255]: pgmap v11364: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:22.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:17:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:17:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:17:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:17:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:22 smithi022 ceph-mon[31360]: pgmap v11364: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:22 smithi022 ceph-mon[40250]: pgmap v11364: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:24 smithi122 ceph-mon[32255]: pgmap v11365: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:24 smithi022 ceph-mon[31360]: pgmap v11365: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:24 smithi022 ceph-mon[40250]: pgmap v11365: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:25.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:26.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:26.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:26.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:26 smithi022 ceph-mon[31360]: pgmap v11366: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3840598983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:26 smithi022 ceph-mon[40250]: pgmap v11366: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3840598983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:26 smithi122 ceph-mon[32255]: pgmap v11366: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3840598983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:17:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:28 smithi122 ceph-mon[32255]: pgmap v11367: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:28 smithi022 ceph-mon[31360]: pgmap v11367: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:28 smithi022 ceph-mon[40250]: pgmap v11367: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:30] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:17:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:30 smithi122 ceph-mon[32255]: pgmap v11368: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:30 smithi022 ceph-mon[31360]: pgmap v11368: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:30 smithi022 ceph-mon[40250]: pgmap v11368: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:31.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:31.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:31.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:31.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:32.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:17:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:17:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:32 smithi122 ceph-mon[32255]: pgmap v11369: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1671297904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:17:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:17:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:32 smithi022 ceph-mon[31360]: pgmap v11369: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1671297904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:32 smithi022 ceph-mon[40250]: pgmap v11369: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1671297904' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:34 smithi122 ceph-mon[32255]: pgmap v11370: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:17:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:17:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:17:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:17:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[31360]: pgmap v11370: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:17:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:17:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:17:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:17:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[40250]: pgmap v11370: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:17:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:17:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:17:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:17:36.572 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:36 smithi122 ceph-mon[32255]: pgmap v11371: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:36 smithi022 ceph-mon[31360]: pgmap v11371: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:36 smithi022 ceph-mon[40250]: pgmap v11371: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:36.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:36.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:36.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2032453972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2032453972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:17:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2032453972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:38 smithi122 ceph-mon[32255]: pgmap v11372: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:38 smithi022 ceph-mon[31360]: pgmap v11372: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:38 smithi022 ceph-mon[40250]: pgmap v11372: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:40 smithi122 ceph-mon[32255]: pgmap v11373: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:40] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:17:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:40 smithi022 ceph-mon[31360]: pgmap v11373: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:40 smithi022 ceph-mon[40250]: pgmap v11373: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:42.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:42.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:42.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:42.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:17:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:17:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:42 smithi122 ceph-mon[32255]: pgmap v11374: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/47773390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:17:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:17:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:42 smithi022 ceph-mon[40250]: pgmap v11374: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/47773390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:42 smithi022 ceph-mon[31360]: pgmap v11374: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:42.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/47773390' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:43 smithi122 ceph-mon[32255]: pgmap v11375: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:43 smithi022 ceph-mon[31360]: pgmap v11375: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:43 smithi022 ceph-mon[40250]: pgmap v11375: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:46 smithi122 ceph-mon[32255]: pgmap v11376: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:46 smithi022 ceph-mon[31360]: pgmap v11376: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:46 smithi022 ceph-mon[40250]: pgmap v11376: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:47.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:47.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:47.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:47.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:47.683 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:17:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:48 smithi122 ceph-mon[32255]: pgmap v11377: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3387662498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:48 smithi022 ceph-mon[31360]: pgmap v11377: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3387662498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:48 smithi022 ceph-mon[40250]: pgmap v11377: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3387662498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:50] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:17:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:50 smithi122 ceph-mon[32255]: pgmap v11378: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:50 smithi022 ceph-mon[31360]: pgmap v11378: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:50 smithi022 ceph-mon[40250]: pgmap v11378: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:17:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:52 smithi122 ceph-mon[32255]: pgmap v11379: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:52.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:17:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:17:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:17:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:17:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:52.684 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:52 smithi022 ceph-mon[31360]: pgmap v11379: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:52.685 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:52 smithi022 ceph-mon[40250]: pgmap v11379: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:53.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:53.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:53.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:53.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:53.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/221677776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/221677776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:17:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/221677776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:54 smithi122 ceph-mon[32255]: pgmap v11380: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:54 smithi022 ceph-mon[31360]: pgmap v11380: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:54 smithi022 ceph-mon[40250]: pgmap v11380: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:17:56.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:56 smithi122 ceph-mon[32255]: pgmap v11381: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:17:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:56 smithi022 ceph-mon[31360]: pgmap v11381: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:17:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:56 smithi022 ceph-mon[40250]: pgmap v11381: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:17:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:17:58.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:17:58.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:17:58.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:17:58.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:17:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:58 smithi122 ceph-mon[32255]: pgmap v11382: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:17:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:58 smithi022 ceph-mon[31360]: pgmap v11382: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:17:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:58 smithi022 ceph-mon[40250]: pgmap v11382: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T00:17:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:17:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1399958581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:17:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1399958581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:17:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:17:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1399958581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:00 smithi122 ceph-mon[32255]: pgmap v11383: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T00:18:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:00] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:18:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:00 smithi022 ceph-mon[31360]: pgmap v11383: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T00:18:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:00 smithi022 ceph-mon[40250]: pgmap v11383: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T00:18:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:02 smithi122 ceph-mon[32255]: pgmap v11384: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T00:18:02.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:18:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:18:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:18:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:18:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:02 smithi022 ceph-mon[31360]: pgmap v11384: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T00:18:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:02 smithi022 ceph-mon[40250]: pgmap v11384: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T00:18:03.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:03.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:03.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:03.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:04 smithi122 ceph-mon[32255]: pgmap v11385: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T00:18:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744258551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:04 smithi022 ceph-mon[31360]: pgmap v11385: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T00:18:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744258551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:04 smithi022 ceph-mon[40250]: pgmap v11385: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T00:18:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744258551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:06 smithi122 ceph-mon[32255]: pgmap v11386: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T00:18:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:06 smithi022 ceph-mon[31360]: pgmap v11386: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T00:18:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:06 smithi022 ceph-mon[40250]: pgmap v11386: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T00:18:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:18:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:08 smithi122 ceph-mon[32255]: pgmap v11387: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:18:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:08 smithi022 ceph-mon[31360]: pgmap v11387: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:18:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:08 smithi022 ceph-mon[40250]: pgmap v11387: 132 pgs: 132 active+clean; 582 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:18:08.788 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:09.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:09.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:09.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:09.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:09 smithi122 ceph-mon[32255]: pgmap v11388: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:18:09.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1966280381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:09 smithi022 ceph-mon[40250]: pgmap v11388: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:18:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1966280381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:09 smithi022 ceph-mon[31360]: pgmap v11388: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:18:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1966280381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:10] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:18:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:12 smithi122 ceph-mon[32255]: pgmap v11389: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:12.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:18:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:18:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:18:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:18:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:12 smithi022 ceph-mon[31360]: pgmap v11389: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:12 smithi022 ceph-mon[40250]: pgmap v11389: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:14.156 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:14.525 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:14 smithi022 ceph-mon[40250]: pgmap v11390: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:14.526 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:14 smithi022 ceph-mon[31360]: pgmap v11390: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:14.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:14.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:14.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:14 smithi122 ceph-mon[32255]: pgmap v11390: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2117951787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2117951787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2117951787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:16 smithi122 ceph-mon[32255]: pgmap v11391: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:18:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:16 smithi022 ceph-mon[31360]: pgmap v11391: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:18:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:16 smithi022 ceph-mon[40250]: pgmap v11391: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:18:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:18:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:18:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:18:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:18:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:18:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:18:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:18:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:18:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:18:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:18:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:18:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:18:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:18:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:18 smithi122 ceph-mon[32255]: pgmap v11392: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:18 smithi022 ceph-mon[31360]: pgmap v11392: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:18 smithi022 ceph-mon[40250]: pgmap v11392: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:18:19.529 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:19.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:19.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:19.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:20 smithi122 ceph-mon[32255]: pgmap v11393: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:18:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3819824765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:20] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:18:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:20 smithi022 ceph-mon[31360]: pgmap v11393: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:18:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3819824765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:20 smithi022 ceph-mon[40250]: pgmap v11393: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:18:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3819824765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:22 smithi122 ceph-mon[32255]: pgmap v11394: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:18:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:18:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:18:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:18:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:22 smithi022 ceph-mon[31360]: pgmap v11394: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:22 smithi022 ceph-mon[40250]: pgmap v11394: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:24 smithi122 ceph-mon[32255]: pgmap v11395: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:24 smithi022 ceph-mon[31360]: pgmap v11395: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:24 smithi022 ceph-mon[40250]: pgmap v11395: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:24.897 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:25.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:25.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:25.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1451057814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1451057814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1451057814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:26 smithi122 ceph-mon[32255]: pgmap v11396: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:26 smithi022 ceph-mon[31360]: pgmap v11396: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:26 smithi022 ceph-mon[40250]: pgmap v11396: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:27 smithi122 ceph-mon[32255]: pgmap v11397: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:27 smithi022 ceph-mon[31360]: pgmap v11397: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:18:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:27 smithi022 ceph-mon[40250]: pgmap v11397: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:30.269 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:30.270 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:30 smithi022 ceph-mon[31360]: pgmap v11398: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:30.270 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:30 smithi022 ceph-mon[40250]: pgmap v11398: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:30] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:18:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:30 smithi122 ceph-mon[32255]: pgmap v11398: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:30.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:30.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:30.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2742908945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2742908945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2742908945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:32 smithi122 ceph-mon[32255]: pgmap v11399: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:18:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:18:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:18:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:18:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:32 smithi022 ceph-mon[31360]: pgmap v11399: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:32 smithi022 ceph-mon[40250]: pgmap v11399: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:34 smithi122 ceph-mon[32255]: pgmap v11400: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:18:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:18:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:18:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:18:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[31360]: pgmap v11400: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:18:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:18:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:18:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:18:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[40250]: pgmap v11400: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:18:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:18:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:18:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:18:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:36.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:36.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:36.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:36 smithi122 ceph-mon[32255]: pgmap v11401: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/294704831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:36 smithi022 ceph-mon[40250]: pgmap v11401: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/294704831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:36 smithi022 ceph-mon[31360]: pgmap v11401: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/294704831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:18:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:38 smithi122 ceph-mon[32255]: pgmap v11402: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:38 smithi022 ceph-mon[31360]: pgmap v11402: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:38 smithi022 ceph-mon[40250]: pgmap v11402: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:40] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:18:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:40 smithi122 ceph-mon[32255]: pgmap v11403: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:40 smithi022 ceph-mon[31360]: pgmap v11403: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:40 smithi022 ceph-mon[40250]: pgmap v11403: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:41.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:41.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:41.384 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:41.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4066603226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4066603226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4066603226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:42.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:18:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:18:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:42 smithi122 ceph-mon[32255]: pgmap v11404: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:18:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:18:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:42 smithi022 ceph-mon[31360]: pgmap v11404: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:42 smithi022 ceph-mon[40250]: pgmap v11404: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:43.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:43.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:44 smithi122 ceph-mon[32255]: pgmap v11405: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:44 smithi022 ceph-mon[31360]: pgmap v11405: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:44 smithi022 ceph-mon[40250]: pgmap v11405: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:46.386 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:46 smithi122 ceph-mon[32255]: pgmap v11406: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:46.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:46.757 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:46 smithi022 ceph-mon[31360]: pgmap v11406: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:46.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:46 smithi022 ceph-mon[40250]: pgmap v11406: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3327126128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3327126128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:18:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3327126128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:48 smithi122 ceph-mon[32255]: pgmap v11407: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:48 smithi022 ceph-mon[31360]: pgmap v11407: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:48 smithi022 ceph-mon[40250]: pgmap v11407: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:49 smithi122 ceph-mon[32255]: pgmap v11408: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:49 smithi022 ceph-mon[31360]: pgmap v11408: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:49 smithi022 ceph-mon[40250]: pgmap v11408: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:50] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:18:51.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:52.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:52.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:52.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:18:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:18:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:18:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:18:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:52 smithi122 ceph-mon[32255]: pgmap v11409: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2038520466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:52 smithi022 ceph-mon[31360]: pgmap v11409: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2038520466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:52 smithi022 ceph-mon[40250]: pgmap v11409: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2038520466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:18:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:54 smithi122 ceph-mon[32255]: pgmap v11410: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:54 smithi022 ceph-mon[31360]: pgmap v11410: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:54 smithi022 ceph-mon[40250]: pgmap v11410: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:56.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:56 smithi122 ceph-mon[32255]: pgmap v11411: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:56 smithi022 ceph-mon[31360]: pgmap v11411: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:56 smithi022 ceph-mon[40250]: pgmap v11411: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:18:57.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:18:57.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:18:57.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:18:57.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:18:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:18:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:58 smithi122 ceph-mon[32255]: pgmap v11412: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:58.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:18:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1552252433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:58 smithi022 ceph-mon[31360]: pgmap v11412: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:18:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1552252433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:18:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:58 smithi022 ceph-mon[40250]: pgmap v11412: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:18:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:18:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1552252433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:00 smithi122 ceph-mon[32255]: pgmap v11413: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:00] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:19:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:00 smithi022 ceph-mon[31360]: pgmap v11413: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:00 smithi022 ceph-mon[40250]: pgmap v11413: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:02.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:02 smithi122 ceph-mon[32255]: pgmap v11414: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:19:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:19:02.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:19:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:19:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:02 smithi022 ceph-mon[31360]: pgmap v11414: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:02 smithi022 ceph-mon[40250]: pgmap v11414: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:02.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:02.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:02.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1750338118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1750338118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1750338118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:04 smithi122 ceph-mon[32255]: pgmap v11415: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:04 smithi022 ceph-mon[31360]: pgmap v11415: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:04 smithi022 ceph-mon[40250]: pgmap v11415: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:06 smithi122 ceph-mon[32255]: pgmap v11416: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:06 smithi022 ceph-mon[31360]: pgmap v11416: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:06 smithi022 ceph-mon[40250]: pgmap v11416: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:19:07.866 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:08.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:08.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:08.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:08 smithi022 ceph-mon[31360]: pgmap v11417: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1831874509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:08 smithi022 ceph-mon[40250]: pgmap v11417: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1831874509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:08 smithi122 ceph-mon[32255]: pgmap v11417: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1831874509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:10.429 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:10] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:19:10.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:10 smithi122 ceph-mon[32255]: pgmap v11418: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:10 smithi022 ceph-mon[31360]: pgmap v11418: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:10 smithi022 ceph-mon[40250]: pgmap v11418: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:11 smithi122 ceph-mon[32255]: pgmap v11419: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:11 smithi022 ceph-mon[31360]: pgmap v11419: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:11 smithi022 ceph-mon[40250]: pgmap v11419: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:19:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:19:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:19:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:19:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:13.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:13 smithi022 ceph-mon[31360]: pgmap v11420: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:13 smithi022 ceph-mon[40250]: pgmap v11420: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:13.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:13.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:13.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:13 smithi122 ceph-mon[32255]: pgmap v11420: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3474792781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3474792781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3474792781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:16 smithi122 ceph-mon[32255]: pgmap v11421: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:16 smithi022 ceph-mon[31360]: pgmap v11421: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:16 smithi022 ceph-mon[40250]: pgmap v11421: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:19:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:19:17.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:19:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:19:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:19:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:19:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:19:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:19:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:19:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:19:18.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:18 smithi122 ceph-mon[32255]: pgmap v11422: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:19:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:18 smithi022 ceph-mon[31360]: pgmap v11422: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:19:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:18 smithi022 ceph-mon[40250]: pgmap v11422: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:19:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:18.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:18.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3216960324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3216960324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3216960324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:20 smithi122 ceph-mon[32255]: pgmap v11423: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:20] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:19:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:20 smithi022 ceph-mon[31360]: pgmap v11423: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:20 smithi022 ceph-mon[40250]: pgmap v11423: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:22 smithi122 ceph-mon[32255]: pgmap v11424: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:19:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:19:22.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:19:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:19:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:22 smithi022 ceph-mon[31360]: pgmap v11424: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:22 smithi022 ceph-mon[40250]: pgmap v11424: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:23.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:24.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:24.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:24.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:24 smithi122 ceph-mon[32255]: pgmap v11425: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4187921635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:24.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:24 smithi022 ceph-mon[31360]: pgmap v11425: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:24.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4187921635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:24.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:24 smithi022 ceph-mon[40250]: pgmap v11425: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:24.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4187921635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:26 smithi122 ceph-mon[32255]: pgmap v11426: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:26 smithi022 ceph-mon[31360]: pgmap v11426: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:26 smithi022 ceph-mon[40250]: pgmap v11426: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:19:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:28 smithi122 ceph-mon[32255]: pgmap v11427: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:28 smithi022 ceph-mon[31360]: pgmap v11427: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:28 smithi022 ceph-mon[40250]: pgmap v11427: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:29.349 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:29.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:29.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:29.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:30.428 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:30] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:19:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:30 smithi122 ceph-mon[32255]: pgmap v11428: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:30.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3161741925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:30 smithi022 ceph-mon[31360]: pgmap v11428: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3161741925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:30 smithi022 ceph-mon[40250]: pgmap v11428: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3161741925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:32 smithi122 ceph-mon[32255]: pgmap v11429: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:19:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:19:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:19:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:19:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:32 smithi022 ceph-mon[31360]: pgmap v11429: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:32 smithi022 ceph-mon[40250]: pgmap v11429: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:34.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:34.721 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[40250]: pgmap v11430: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:34.721 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:19:34.722 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:19:34.722 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:19:34.722 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:19:34.723 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[31360]: pgmap v11430: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:34.723 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:19:34.723 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:19:34.723 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:19:34.724 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:19:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:34 smithi122 ceph-mon[32255]: pgmap v11430: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:19:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:19:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:19:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:19:35.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:35.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:35.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:35 smithi022 ceph-mon[31360]: pgmap v11431: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/717782199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:35 smithi022 ceph-mon[40250]: pgmap v11431: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/717782199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:35 smithi122 ceph-mon[32255]: pgmap v11431: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/717782199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:19:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:38 smithi122 ceph-mon[32255]: pgmap v11432: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:38 smithi022 ceph-mon[31360]: pgmap v11432: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:38 smithi022 ceph-mon[40250]: pgmap v11432: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:40.085 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:40] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:19:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:40 smithi122 ceph-mon[32255]: pgmap v11433: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:40.453 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:40 smithi022 ceph-mon[31360]: pgmap v11433: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:40.453 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:40 smithi022 ceph-mon[40250]: pgmap v11433: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2465094953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2465094953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2465094953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:42 smithi122 ceph-mon[32255]: pgmap v11434: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:19:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:19:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:19:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:19:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:42 smithi022 ceph-mon[31360]: pgmap v11434: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:42 smithi022 ceph-mon[40250]: pgmap v11434: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:44 smithi122 ceph-mon[32255]: pgmap v11435: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:44 smithi022 ceph-mon[31360]: pgmap v11435: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:44 smithi022 ceph-mon[40250]: pgmap v11435: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:45.454 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:45.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:45.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:45.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:46 smithi122 ceph-mon[32255]: pgmap v11436: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1741049150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:46 smithi022 ceph-mon[31360]: pgmap v11436: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1741049150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:46 smithi022 ceph-mon[40250]: pgmap v11436: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1741049150' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:19:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:48 smithi122 ceph-mon[32255]: pgmap v11437: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:48 smithi022 ceph-mon[31360]: pgmap v11437: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:48 smithi022 ceph-mon[40250]: pgmap v11437: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:50.425 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:50] "GET /metrics HTTP/1.1" 200 202415 "" "Prometheus/2.18.1" 2021-12-16T00:19:50.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:50 smithi122 ceph-mon[32255]: pgmap v11438: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:50 smithi022 ceph-mon[31360]: pgmap v11438: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:50 smithi022 ceph-mon[40250]: pgmap v11438: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:50.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:51.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:51.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:51.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4065208025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4065208025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4065208025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:19:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:19:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:52 smithi122 ceph-mon[32255]: pgmap v11439: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:19:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:19:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:52 smithi022 ceph-mon[31360]: pgmap v11439: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:52 smithi022 ceph-mon[40250]: pgmap v11439: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:19:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:54 smithi022 ceph-mon[31360]: pgmap v11440: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:54 smithi022 ceph-mon[40250]: pgmap v11440: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:54 smithi122 ceph-mon[32255]: pgmap v11440: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:56.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:19:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:56 smithi022 ceph-mon[31360]: pgmap v11441: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:56 smithi022 ceph-mon[40250]: pgmap v11441: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:56.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:19:56.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:19:56.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:19:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:56 smithi122 ceph-mon[32255]: pgmap v11441: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:19:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2014084616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2014084616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:19:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2014084616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:19:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:19:58 smithi122 ceph-mon[32255]: pgmap v11442: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:19:58 smithi022 ceph-mon[31360]: pgmap v11442: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:19:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:19:58 smithi022 ceph-mon[40250]: pgmap v11442: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:00] "GET /metrics HTTP/1.1" 200 202415 "" "Prometheus/2.18.1" 2021-12-16T00:20:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:00 smithi122 ceph-mon[32255]: pgmap v11443: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T00:20:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:00 smithi022 ceph-mon[31360]: pgmap v11443: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T00:20:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:00 smithi022 ceph-mon[40250]: pgmap v11443: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T00:20:01.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:01.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:01.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:01.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:20:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:20:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:20:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:20:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:02 smithi122 ceph-mon[32255]: pgmap v11444: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2052520597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:02 smithi022 ceph-mon[31360]: pgmap v11444: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2052520597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:02 smithi022 ceph-mon[40250]: pgmap v11444: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2052520597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:04 smithi122 ceph-mon[32255]: pgmap v11445: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:04 smithi022 ceph-mon[31360]: pgmap v11445: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:04 smithi022 ceph-mon[40250]: pgmap v11445: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:06 smithi122 ceph-mon[32255]: pgmap v11446: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:06 smithi022 ceph-mon[31360]: pgmap v11446: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:06 smithi022 ceph-mon[40250]: pgmap v11446: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:06.943 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:07.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:07.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:07.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/495076802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:20:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/495076802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/495076802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:08 smithi022 ceph-mon[31360]: pgmap v11447: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:08 smithi022 ceph-mon[40250]: pgmap v11447: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:08 smithi122 ceph-mon[32255]: pgmap v11447: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:10] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:20:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:10 smithi022 ceph-mon[31360]: pgmap v11448: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:10 smithi022 ceph-mon[40250]: pgmap v11448: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:10 smithi122 ceph-mon[32255]: pgmap v11448: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:11 smithi022 ceph-mon[31360]: pgmap v11449: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:11 smithi022 ceph-mon[40250]: pgmap v11449: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:11 smithi122 ceph-mon[32255]: pgmap v11449: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:12.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:12.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:12.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:12.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:12.686 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:12.687 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:12.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:12.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:20:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:20:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:20:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:20:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1044620886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:13 smithi022 ceph-mon[31360]: pgmap v11450: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1044620886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:13 smithi022 ceph-mon[40250]: pgmap v11450: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1044620886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:13 smithi122 ceph-mon[32255]: pgmap v11450: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:16 smithi122 ceph-mon[32255]: pgmap v11451: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:16 smithi022 ceph-mon[31360]: pgmap v11451: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:16 smithi022 ceph-mon[40250]: pgmap v11451: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:17.687 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:20:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:18.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:18.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:18.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:18 smithi122 ceph-mon[32255]: pgmap v11452: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:20:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:20:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:20:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:20:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3627063996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[40250]: pgmap v11452: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:20:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:20:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:20:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:20:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3627063996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[31360]: pgmap v11452: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:18.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:20:18.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:20:18.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:20:18.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:20:18.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3627063996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:20] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:20:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:20 smithi122 ceph-mon[32255]: pgmap v11453: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:20 smithi022 ceph-mon[31360]: pgmap v11453: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:20 smithi022 ceph-mon[40250]: pgmap v11453: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:22 smithi122 ceph-mon[32255]: pgmap v11454: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:20:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:20:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:20:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:20:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:22 smithi022 ceph-mon[31360]: pgmap v11454: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:22 smithi022 ceph-mon[40250]: pgmap v11454: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:23.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:23.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:23.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:23.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/508821665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/508821665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/508821665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:24 smithi122 ceph-mon[32255]: pgmap v11455: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:24 smithi022 ceph-mon[31360]: pgmap v11455: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:24 smithi022 ceph-mon[40250]: pgmap v11455: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:26 smithi122 ceph-mon[32255]: pgmap v11456: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:26 smithi022 ceph-mon[31360]: pgmap v11456: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:26 smithi022 ceph-mon[40250]: pgmap v11456: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:20:28.417 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:28 smithi122 ceph-mon[32255]: pgmap v11457: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:28 smithi022 ceph-mon[31360]: pgmap v11457: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:28 smithi022 ceph-mon[40250]: pgmap v11457: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:28.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:28.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:28.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/414013393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/414013393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/414013393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:30] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:20:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:30 smithi022 ceph-mon[31360]: pgmap v11458: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:30 smithi022 ceph-mon[40250]: pgmap v11458: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:30 smithi122 ceph-mon[32255]: pgmap v11458: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:32 smithi122 ceph-mon[32255]: pgmap v11459: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:20:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:20:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:20:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:20:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:32 smithi022 ceph-mon[31360]: pgmap v11459: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:32 smithi022 ceph-mon[40250]: pgmap v11459: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:33.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:34.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:34.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:34.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[31360]: pgmap v11460: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:20:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:20:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:20:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:20:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2114407093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[40250]: pgmap v11460: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:20:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:20:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:20:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:20:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2114407093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:34 smithi122 ceph-mon[32255]: pgmap v11460: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:20:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:20:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:20:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:20:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2114407093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:36 smithi122 ceph-mon[32255]: pgmap v11461: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:36 smithi022 ceph-mon[31360]: pgmap v11461: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:36 smithi022 ceph-mon[40250]: pgmap v11461: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:20:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:38 smithi122 ceph-mon[32255]: pgmap v11462: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:38 smithi022 ceph-mon[31360]: pgmap v11462: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:38 smithi022 ceph-mon[40250]: pgmap v11462: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:39.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:39.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:39.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:40.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:40] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:20:40.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:40 smithi122 ceph-mon[32255]: pgmap v11463: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:40.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2550588788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:40 smithi022 ceph-mon[31360]: pgmap v11463: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2550588788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:40 smithi022 ceph-mon[40250]: pgmap v11463: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2550588788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:42 smithi122 ceph-mon[32255]: pgmap v11464: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:20:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:20:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:20:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:20:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:42 smithi022 ceph-mon[31360]: pgmap v11464: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:42 smithi022 ceph-mon[40250]: pgmap v11464: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:44.527 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:44 smithi122 ceph-mon[32255]: pgmap v11465: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:44 smithi022 ceph-mon[31360]: pgmap v11465: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:44 smithi022 ceph-mon[40250]: pgmap v11465: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:44.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:44.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:44.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1538028789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1538028789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1538028789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:46 smithi022 ceph-mon[31360]: pgmap v11466: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:46 smithi022 ceph-mon[40250]: pgmap v11466: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:46 smithi122 ceph-mon[32255]: pgmap v11466: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:20:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:48 smithi022 ceph-mon[31360]: pgmap v11467: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:48 smithi022 ceph-mon[40250]: pgmap v11467: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:48 smithi122 ceph-mon[32255]: pgmap v11467: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:49.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:50.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:50.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:50.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:50] "GET /metrics HTTP/1.1" 200 202417 "" "Prometheus/2.18.1" 2021-12-16T00:20:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:50 smithi022 ceph-mon[31360]: pgmap v11468: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/941359975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:50 smithi022 ceph-mon[40250]: pgmap v11468: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/941359975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:50 smithi122 ceph-mon[32255]: pgmap v11468: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/941359975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:20:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:20:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:52 smithi122 ceph-mon[32255]: pgmap v11469: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:20:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:20:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:52 smithi022 ceph-mon[31360]: pgmap v11469: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:52 smithi022 ceph-mon[40250]: pgmap v11469: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:20:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:54 smithi022 ceph-mon[31360]: pgmap v11470: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:54 smithi022 ceph-mon[40250]: pgmap v11470: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:54 smithi122 ceph-mon[32255]: pgmap v11470: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:55.265 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:20:55.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:20:55.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:20:55.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:20:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:56 smithi122 ceph-mon[32255]: pgmap v11471: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1682380644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:56 smithi022 ceph-mon[31360]: pgmap v11471: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1682380644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:56 smithi022 ceph-mon[40250]: pgmap v11471: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:20:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1682380644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:20:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:20:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:20:58 smithi122 ceph-mon[32255]: pgmap v11472: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:20:58 smithi022 ceph-mon[31360]: pgmap v11472: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:20:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:20:58 smithi022 ceph-mon[40250]: pgmap v11472: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:00.428 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:00] "GET /metrics HTTP/1.1" 200 202417 "" "Prometheus/2.18.1" 2021-12-16T00:21:00.637 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:00.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:00 smithi122 ceph-mon[32255]: pgmap v11473: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:00 smithi022 ceph-mon[31360]: pgmap v11473: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:00 smithi022 ceph-mon[40250]: pgmap v11473: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:01.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1477859473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1477859473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1477859473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:21:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:21:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:02 smithi122 ceph-mon[32255]: pgmap v11474: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:21:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:21:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:02 smithi022 ceph-mon[31360]: pgmap v11474: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:02 smithi022 ceph-mon[40250]: pgmap v11474: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:04 smithi022 ceph-mon[31360]: pgmap v11475: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:04 smithi022 ceph-mon[40250]: pgmap v11475: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:04 smithi122 ceph-mon[32255]: pgmap v11475: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:06.002 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:06.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:06.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:06.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:06 smithi022 ceph-mon[31360]: pgmap v11476: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3181360685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:06 smithi022 ceph-mon[40250]: pgmap v11476: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3181360685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:06 smithi122 ceph-mon[32255]: pgmap v11476: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3181360685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:21:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:08 smithi022 ceph-mon[31360]: pgmap v11477: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:08 smithi022 ceph-mon[40250]: pgmap v11477: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:08 smithi122 ceph-mon[32255]: pgmap v11477: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:09 smithi022 ceph-mon[31360]: pgmap v11478: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:09 smithi022 ceph-mon[40250]: pgmap v11478: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:09 smithi122 ceph-mon[32255]: pgmap v11478: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:10] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:21:11.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:11.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:21:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:21:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:12 smithi122 ceph-mon[32255]: pgmap v11479: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3270123789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:21:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:21:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:12 smithi022 ceph-mon[31360]: pgmap v11479: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3270123789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:12 smithi022 ceph-mon[40250]: pgmap v11479: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3270123789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:14 smithi122 ceph-mon[32255]: pgmap v11480: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:14 smithi022 ceph-mon[31360]: pgmap v11480: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:14 smithi022 ceph-mon[40250]: pgmap v11480: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:16.744 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:16.745 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:16 smithi022 ceph-mon[40250]: pgmap v11481: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:16.745 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:16 smithi022 ceph-mon[31360]: pgmap v11481: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:16 smithi122 ceph-mon[32255]: pgmap v11481: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:17.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:17.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:17.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1094315858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:21:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1094315858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1094315858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:18 smithi122 ceph-mon[32255]: pgmap v11482: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:21:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:21:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:21:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:18.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[40250]: pgmap v11482: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:18.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:21:18.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:21:18.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:21:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[31360]: pgmap v11482: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:21:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:21:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:21:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:19 smithi122 ceph-mon[32255]: pgmap v11483: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:19 smithi022 ceph-mon[31360]: pgmap v11483: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:21:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:19 smithi022 ceph-mon[40250]: pgmap v11483: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:20] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:21:22.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:22.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:22.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:22.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:22 smithi122 ceph-mon[32255]: pgmap v11484: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:21:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:21:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:21:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:21:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:22 smithi022 ceph-mon[31360]: pgmap v11484: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:22 smithi022 ceph-mon[40250]: pgmap v11484: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3057390755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3057390755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3057390755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:24 smithi022 ceph-mon[31360]: pgmap v11485: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:24 smithi022 ceph-mon[40250]: pgmap v11485: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:24 smithi122 ceph-mon[32255]: pgmap v11485: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:26 smithi022 ceph-mon[31360]: pgmap v11486: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:26 smithi022 ceph-mon[40250]: pgmap v11486: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:26 smithi122 ceph-mon[32255]: pgmap v11486: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:27.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:21:27.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:27.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:27.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:28 smithi022 ceph-mon[31360]: pgmap v11487: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4094057333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:28 smithi022 ceph-mon[40250]: pgmap v11487: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4094057333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:28 smithi122 ceph-mon[32255]: pgmap v11487: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4094057333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:30] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:21:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:30 smithi022 ceph-mon[31360]: pgmap v11488: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:30 smithi022 ceph-mon[40250]: pgmap v11488: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:30 smithi122 ceph-mon[32255]: pgmap v11488: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:21:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:21:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:32 smithi122 ceph-mon[32255]: pgmap v11489: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:21:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:21:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:32 smithi022 ceph-mon[31360]: pgmap v11489: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:32 smithi022 ceph-mon[40250]: pgmap v11489: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:32.856 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:33.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:33.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:33.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1343736788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1343736788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1343736788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[31360]: pgmap v11490: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:21:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:21:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:21:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:21:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[40250]: pgmap v11490: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:21:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:21:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:21:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:21:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:34 smithi122 ceph-mon[32255]: pgmap v11490: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:21:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:21:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:21:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:21:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:36 smithi122 ceph-mon[32255]: pgmap v11491: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:36 smithi022 ceph-mon[31360]: pgmap v11491: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:36 smithi022 ceph-mon[40250]: pgmap v11491: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:21:38.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:38 smithi022 ceph-mon[31360]: pgmap v11492: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:38 smithi022 ceph-mon[40250]: pgmap v11492: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:38.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:38.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:38.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:38 smithi122 ceph-mon[32255]: pgmap v11492: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2462716889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2462716889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2462716889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:40] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:21:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:40 smithi022 ceph-mon[31360]: pgmap v11493: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:40 smithi022 ceph-mon[40250]: pgmap v11493: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:40 smithi122 ceph-mon[32255]: pgmap v11493: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:42 smithi122 ceph-mon[32255]: pgmap v11494: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:21:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:21:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:21:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:21:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:42 smithi022 ceph-mon[31360]: pgmap v11494: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:42 smithi022 ceph-mon[40250]: pgmap v11494: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:43.605 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:43.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:43.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:43.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:44 smithi022 ceph-mon[31360]: pgmap v11495: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3718768462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:44 smithi022 ceph-mon[40250]: pgmap v11495: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3718768462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:44 smithi122 ceph-mon[32255]: pgmap v11495: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3718768462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:46 smithi022 ceph-mon[31360]: pgmap v11496: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:46 smithi022 ceph-mon[40250]: pgmap v11496: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:46 smithi122 ceph-mon[32255]: pgmap v11496: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:21:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:48 smithi022 ceph-mon[31360]: pgmap v11497: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:48 smithi022 ceph-mon[40250]: pgmap v11497: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:48 smithi122 ceph-mon[32255]: pgmap v11497: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:48.979 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:49.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:49.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:49.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/952053855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/952053855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/952053855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:50] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:21:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:50 smithi022 ceph-mon[31360]: pgmap v11498: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:50 smithi022 ceph-mon[40250]: pgmap v11498: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:50 smithi122 ceph-mon[32255]: pgmap v11498: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:51 smithi022 ceph-mon[31360]: pgmap v11499: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:51 smithi022 ceph-mon[40250]: pgmap v11499: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:51 smithi122 ceph-mon[32255]: pgmap v11499: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:21:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:21:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:21:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:21:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:21:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:53 smithi022 ceph-mon[31360]: pgmap v11500: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:53 smithi022 ceph-mon[40250]: pgmap v11500: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:53 smithi122 ceph-mon[32255]: pgmap v11500: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:54.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:21:54.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:21:54.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:21:54.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:21:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1030415370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1030415370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1030415370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:21:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:56 smithi022 ceph-mon[31360]: pgmap v11501: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:56 smithi022 ceph-mon[40250]: pgmap v11501: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:56 smithi122 ceph-mon[32255]: pgmap v11501: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:21:57.792 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:21:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:21:58 smithi022 ceph-mon[31360]: pgmap v11502: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:21:58 smithi022 ceph-mon[40250]: pgmap v11502: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:21:58 smithi122 ceph-mon[32255]: pgmap v11502: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:21:59.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:00.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:00.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:00.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:00] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:22:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:00 smithi022 ceph-mon[31360]: pgmap v11503: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1628640323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:00 smithi022 ceph-mon[40250]: pgmap v11503: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1628640323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:00 smithi122 ceph-mon[32255]: pgmap v11503: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1628640323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:02 smithi122 ceph-mon[32255]: pgmap v11504: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:02.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:22:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:22:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:22:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:22:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:02 smithi022 ceph-mon[31360]: pgmap v11504: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:02 smithi022 ceph-mon[40250]: pgmap v11504: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:04 smithi022 ceph-mon[31360]: pgmap v11505: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:04 smithi022 ceph-mon[40250]: pgmap v11505: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:04.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:04 smithi122 ceph-mon[32255]: pgmap v11505: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:05.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:05.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:05.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:05.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:06 smithi022 ceph-mon[31360]: pgmap v11506: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3388018828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:06 smithi022 ceph-mon[40250]: pgmap v11506: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3388018828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:06 smithi122 ceph-mon[32255]: pgmap v11506: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3388018828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:22:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:08 smithi022 ceph-mon[31360]: pgmap v11507: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:08 smithi022 ceph-mon[40250]: pgmap v11507: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:08 smithi122 ceph-mon[32255]: pgmap v11507: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:10] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:22:10.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:10 smithi022 ceph-mon[31360]: pgmap v11508: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:10 smithi022 ceph-mon[40250]: pgmap v11508: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:10.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:10.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:10.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:10 smithi122 ceph-mon[32255]: pgmap v11508: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4088014596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4088014596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4088014596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:12 smithi122 ceph-mon[32255]: pgmap v11509: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:12.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:22:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:22:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:22:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:22:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:12 smithi022 ceph-mon[31360]: pgmap v11509: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:12 smithi022 ceph-mon[40250]: pgmap v11509: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:14 smithi022 ceph-mon[40250]: pgmap v11510: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:14 smithi022 ceph-mon[31360]: pgmap v11510: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:14 smithi122 ceph-mon[32255]: pgmap v11510: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:16.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:16.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:16.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:16 smithi022 ceph-mon[31360]: pgmap v11511: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:16.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/214967996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:16.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:16 smithi022 ceph-mon[40250]: pgmap v11511: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:16.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/214967996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:16 smithi122 ceph-mon[32255]: pgmap v11511: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/214967996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:17.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:17 smithi022 ceph-mon[31360]: pgmap v11512: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:22:17.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:17 smithi022 ceph-mon[40250]: pgmap v11512: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:17 smithi122 ceph-mon[32255]: pgmap v11512: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:22:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:22:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:22:19.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:22:19.245 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:22:19.245 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:22:19.246 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:22:19.246 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:22:19.246 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:22:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:20] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:22:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:20 smithi122 ceph-mon[32255]: pgmap v11513: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:22:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:20 smithi022 ceph-mon[31360]: pgmap v11513: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:22:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:20 smithi022 ceph-mon[40250]: pgmap v11513: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:22:21.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:21.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:21.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:21.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:22.696 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:22:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:22:22.697 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:22:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:22:22.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:22 smithi122 ceph-mon[32255]: pgmap v11514: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:22.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1849319714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:22 smithi022 ceph-mon[31360]: pgmap v11514: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1849319714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:22 smithi022 ceph-mon[40250]: pgmap v11514: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1849319714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:24 smithi022 ceph-mon[31360]: pgmap v11515: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:24 smithi022 ceph-mon[40250]: pgmap v11515: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:24.969 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:24 smithi122 ceph-mon[32255]: pgmap v11515: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:22:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:26 smithi022 ceph-mon[31360]: pgmap v11516: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:26 smithi022 ceph-mon[40250]: pgmap v11516: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:26 smithi122 ceph-mon[32255]: pgmap v11516: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:26.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:26.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:26.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:22:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3018276141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3018276141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3018276141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:28 smithi022 ceph-mon[31360]: pgmap v11517: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:28 smithi022 ceph-mon[40250]: pgmap v11517: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:28 smithi122 ceph-mon[32255]: pgmap v11517: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:30] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:22:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:30 smithi022 ceph-mon[31360]: pgmap v11518: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:30 smithi022 ceph-mon[40250]: pgmap v11518: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:30 smithi122 ceph-mon[32255]: pgmap v11518: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:31 smithi022 ceph-mon[31360]: pgmap v11519: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:31 smithi022 ceph-mon[40250]: pgmap v11519: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:31 smithi122 ceph-mon[32255]: pgmap v11519: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:31.951 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:32.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:32.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:32.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:22:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:22:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:22:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:22:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4252830068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4252830068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4252830068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:33 smithi022 ceph-mon[31360]: pgmap v11520: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:33 smithi022 ceph-mon[40250]: pgmap v11520: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:33 smithi122 ceph-mon[32255]: pgmap v11520: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:22:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:22:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:22:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:22:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:22:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:22:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:22:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:22:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:36 smithi022 ceph-mon[31360]: pgmap v11521: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:36 smithi022 ceph-mon[40250]: pgmap v11521: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:36 smithi122 ceph-mon[32255]: pgmap v11521: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:37.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:37.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:37.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:37.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:37.694 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:22:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:38 smithi022 ceph-mon[31360]: pgmap v11522: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1341057300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:38 smithi022 ceph-mon[40250]: pgmap v11522: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1341057300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:38 smithi122 ceph-mon[32255]: pgmap v11522: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1341057300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:40] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:22:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:40 smithi022 ceph-mon[31360]: pgmap v11523: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:40 smithi022 ceph-mon[40250]: pgmap v11523: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:40 smithi122 ceph-mon[32255]: pgmap v11523: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:42 smithi122 ceph-mon[32255]: pgmap v11524: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:22:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:22:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:22:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:22:42.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:42 smithi022 ceph-mon[31360]: pgmap v11524: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:42 smithi022 ceph-mon[40250]: pgmap v11524: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:43.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:43.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2010267622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:43.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:43.460 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2010267622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2010267622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:44 smithi022 ceph-mon[31360]: pgmap v11525: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:44 smithi022 ceph-mon[40250]: pgmap v11525: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:44 smithi122 ceph-mon[32255]: pgmap v11525: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:46 smithi022 ceph-mon[31360]: pgmap v11526: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:46 smithi022 ceph-mon[40250]: pgmap v11526: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:46 smithi122 ceph-mon[32255]: pgmap v11526: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:22:48.062 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:48.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:48.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:48.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:48 smithi022 ceph-mon[31360]: pgmap v11527: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3776084241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:48 smithi022 ceph-mon[40250]: pgmap v11527: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3776084241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:48 smithi122 ceph-mon[32255]: pgmap v11527: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3776084241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:49 smithi022 ceph-mon[31360]: pgmap v11528: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:49 smithi022 ceph-mon[40250]: pgmap v11528: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:49 smithi122 ceph-mon[32255]: pgmap v11528: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:50] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:22:52.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:22:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:22:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:52 smithi122 ceph-mon[32255]: pgmap v11529: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:22:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:22:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:52 smithi022 ceph-mon[31360]: pgmap v11529: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:52 smithi022 ceph-mon[40250]: pgmap v11529: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:53.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:53.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:53.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:53.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:22:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:54 smithi022 ceph-mon[31360]: pgmap v11530: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4124191093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:54 smithi022 ceph-mon[40250]: pgmap v11530: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4124191093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:54 smithi122 ceph-mon[32255]: pgmap v11530: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4124191093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:56 smithi022 ceph-mon[31360]: pgmap v11531: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:56 smithi022 ceph-mon[40250]: pgmap v11531: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:56 smithi122 ceph-mon[32255]: pgmap v11531: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:22:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:22:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:58 smithi022 ceph-mon[31360]: pgmap v11532: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:58 smithi022 ceph-mon[40250]: pgmap v11532: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:58.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:22:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:58 smithi122 ceph-mon[32255]: pgmap v11532: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:22:59.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:22:59.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:22:59.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:22:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:22:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/830611649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:22:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/830611649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:22:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:22:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/830611649' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:00] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:23:00.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:00 smithi022 ceph-mon[31360]: pgmap v11533: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:00.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:00 smithi022 ceph-mon[40250]: pgmap v11533: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:00 smithi122 ceph-mon[32255]: pgmap v11533: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:01 smithi022 ceph-mon[31360]: pgmap v11534: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:01 smithi022 ceph-mon[40250]: pgmap v11534: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:01 smithi122 ceph-mon[32255]: pgmap v11534: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:23:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:23:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:23:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:23:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:03 smithi022 ceph-mon[31360]: pgmap v11535: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:03 smithi022 ceph-mon[40250]: pgmap v11535: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:03.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:03 smithi122 ceph-mon[32255]: pgmap v11535: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:04.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:04.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:04.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:04.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/13094430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/13094430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/13094430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:06 smithi022 ceph-mon[31360]: pgmap v11536: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:06 smithi022 ceph-mon[40250]: pgmap v11536: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:06 smithi122 ceph-mon[32255]: pgmap v11536: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:23:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:08 smithi022 ceph-mon[31360]: pgmap v11537: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:08 smithi022 ceph-mon[40250]: pgmap v11537: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:08 smithi122 ceph-mon[32255]: pgmap v11537: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:09.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:09.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:09.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:10] "GET /metrics HTTP/1.1" 200 202417 "" "Prometheus/2.18.1" 2021-12-16T00:23:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:10 smithi022 ceph-mon[31360]: pgmap v11538: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2385252929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:10 smithi022 ceph-mon[40250]: pgmap v11538: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2385252929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:10 smithi122 ceph-mon[32255]: pgmap v11538: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2385252929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:12 smithi122 ceph-mon[32255]: pgmap v11539: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:23:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:23:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:23:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:23:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:12 smithi022 ceph-mon[31360]: pgmap v11539: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:12 smithi022 ceph-mon[40250]: pgmap v11539: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:14 smithi022 ceph-mon[31360]: pgmap v11540: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:14 smithi022 ceph-mon[40250]: pgmap v11540: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:14.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:14 smithi122 ceph-mon[32255]: pgmap v11540: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:15.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:15.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3886749090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3886749090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3886749090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:16 smithi022 ceph-mon[31360]: pgmap v11541: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:16 smithi022 ceph-mon[40250]: pgmap v11541: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:16 smithi122 ceph-mon[32255]: pgmap v11541: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:17 smithi022 ceph-mon[31360]: pgmap v11542: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:23:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:17 smithi022 ceph-mon[40250]: pgmap v11542: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:17 smithi122 ceph-mon[32255]: pgmap v11542: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:20.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:20] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:23:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:20 smithi122 ceph-mon[32255]: pgmap v11543: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:23:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:23:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:23:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:23:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[40250]: pgmap v11543: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:23:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:23:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:23:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[31360]: pgmap v11543: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:23:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:23:20.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:23:20.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:20.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:20.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/127263083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/127263083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/127263083' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:22.705 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:23:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:23:22.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:22 smithi122 ceph-mon[32255]: pgmap v11544: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:22.706 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:23:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:23:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:22 smithi022 ceph-mon[31360]: pgmap v11544: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:22 smithi022 ceph-mon[40250]: pgmap v11544: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:24 smithi022 ceph-mon[31360]: pgmap v11545: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:24 smithi022 ceph-mon[40250]: pgmap v11545: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:24 smithi122 ceph-mon[32255]: pgmap v11545: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:25.661 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:26.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:26.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:26.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:26.788 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:26 smithi022 ceph-mon[31360]: pgmap v11546: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:26.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1703223159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:26.790 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:26 smithi022 ceph-mon[40250]: pgmap v11546: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:26.791 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1703223159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:26 smithi122 ceph-mon[32255]: pgmap v11546: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1703223159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:27 smithi022 ceph-mon[31360]: pgmap v11547: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:23:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:27 smithi022 ceph-mon[40250]: pgmap v11547: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:27 smithi122 ceph-mon[32255]: pgmap v11547: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:30 smithi122 ceph-mon[32255]: pgmap v11548: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:30] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:23:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:30 smithi022 ceph-mon[31360]: pgmap v11548: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:30 smithi022 ceph-mon[40250]: pgmap v11548: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:31.031 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:31.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:31.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:31.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3151964136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3151964136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3151964136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:23:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:23:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:32 smithi122 ceph-mon[32255]: pgmap v11549: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:23:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:23:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:32 smithi022 ceph-mon[31360]: pgmap v11549: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:32 smithi022 ceph-mon[40250]: pgmap v11549: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[31360]: pgmap v11550: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[40250]: pgmap v11550: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:23:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:34 smithi122 ceph-mon[32255]: pgmap v11550: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:23:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:23:36.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:36.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:36.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:36.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:36.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:36 smithi022 ceph-mon[31360]: pgmap v11551: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:36.776 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:36 smithi022 ceph-mon[40250]: pgmap v11551: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:36 smithi122 ceph-mon[32255]: pgmap v11551: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3878146441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:23:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3878146441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3878146441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:38.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:38 smithi022 ceph-mon[31360]: pgmap v11552: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:38.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:38 smithi022 ceph-mon[40250]: pgmap v11552: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:38 smithi122 ceph-mon[32255]: pgmap v11552: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:39 smithi022 ceph-mon[31360]: pgmap v11553: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:39 smithi022 ceph-mon[40250]: pgmap v11553: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:39.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:39 smithi122 ceph-mon[32255]: pgmap v11553: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:40] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:23:41.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:42.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:42.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:42.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:42.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:23:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:23:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:23:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:23:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:42 smithi122 ceph-mon[32255]: pgmap v11554: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2522695764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:42 smithi022 ceph-mon[31360]: pgmap v11554: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2522695764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:42 smithi022 ceph-mon[40250]: pgmap v11554: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:42.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2522695764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:44 smithi022 ceph-mon[31360]: pgmap v11555: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:44 smithi022 ceph-mon[40250]: pgmap v11555: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:44 smithi122 ceph-mon[32255]: pgmap v11555: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:46 smithi022 ceph-mon[31360]: pgmap v11556: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:46 smithi022 ceph-mon[40250]: pgmap v11556: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:46 smithi122 ceph-mon[32255]: pgmap v11556: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:47.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:47.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:47.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:47.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:47.518 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:23:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:48 smithi022 ceph-mon[31360]: pgmap v11557: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2005198777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:48 smithi022 ceph-mon[40250]: pgmap v11557: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2005198777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:48 smithi122 ceph-mon[32255]: pgmap v11557: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2005198777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:50] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:23:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:50 smithi022 ceph-mon[31360]: pgmap v11558: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:50 smithi022 ceph-mon[40250]: pgmap v11558: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:50 smithi122 ceph-mon[32255]: pgmap v11558: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:51 smithi022 ceph-mon[31360]: pgmap v11559: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:51 smithi022 ceph-mon[40250]: pgmap v11559: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:51 smithi122 ceph-mon[32255]: pgmap v11559: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:52.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:23:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:23:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:23:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:23:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:53.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3498000153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3498000153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3498000153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:54.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:53 smithi122 ceph-mon[32255]: pgmap v11560: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:53 smithi022 ceph-mon[31360]: pgmap v11560: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:23:54.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:53 smithi022 ceph-mon[40250]: pgmap v11560: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:56 smithi022 ceph-mon[31360]: pgmap v11561: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:56 smithi022 ceph-mon[40250]: pgmap v11561: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:56 smithi122 ceph-mon[32255]: pgmap v11561: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:23:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:23:57.889 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:23:58.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:23:58.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:23:58.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:23:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:58 smithi022 ceph-mon[31360]: pgmap v11562: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:23:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3110977552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:58 smithi022 ceph-mon[40250]: pgmap v11562: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:23:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3110977552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:23:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:58 smithi122 ceph-mon[32255]: pgmap v11562: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:23:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:23:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3110977552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:00] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:24:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:00 smithi022 ceph-mon[31360]: pgmap v11563: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:00 smithi022 ceph-mon[40250]: pgmap v11563: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:00 smithi122 ceph-mon[32255]: pgmap v11563: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:24:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:24:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:02 smithi122 ceph-mon[32255]: pgmap v11564: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:24:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:24:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:02 smithi022 ceph-mon[31360]: pgmap v11564: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:02.959 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:02 smithi022 ceph-mon[40250]: pgmap v11564: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:03.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:03.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:03.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:03.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:04 smithi022 ceph-mon[31360]: pgmap v11565: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2662093141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:04 smithi022 ceph-mon[40250]: pgmap v11565: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2662093141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:04 smithi122 ceph-mon[32255]: pgmap v11565: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2662093141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:06 smithi022 ceph-mon[31360]: pgmap v11566: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:06 smithi022 ceph-mon[40250]: pgmap v11566: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:06 smithi122 ceph-mon[32255]: pgmap v11566: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:24:08.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:08 smithi022 ceph-mon[31360]: pgmap v11567: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:08 smithi022 ceph-mon[40250]: pgmap v11567: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:08 smithi122 ceph-mon[32255]: pgmap v11567: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:09.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:09.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:09.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1316084262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1316084262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1316084262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:10] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:24:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:10 smithi022 ceph-mon[31360]: pgmap v11568: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:10 smithi022 ceph-mon[40250]: pgmap v11568: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:10.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:10 smithi122 ceph-mon[32255]: pgmap v11568: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:12 smithi122 ceph-mon[32255]: pgmap v11569: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:12.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:24:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:24:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:24:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:24:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:12 smithi022 ceph-mon[31360]: pgmap v11569: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:12 smithi022 ceph-mon[40250]: pgmap v11569: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:14.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:14.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:14 smithi022 ceph-mon[40250]: pgmap v11570: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3457240356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:14 smithi022 ceph-mon[31360]: pgmap v11570: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3457240356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:14 smithi122 ceph-mon[32255]: pgmap v11570: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3457240356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:16 smithi022 ceph-mon[31360]: pgmap v11571: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:16 smithi022 ceph-mon[40250]: pgmap v11571: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:16 smithi122 ceph-mon[32255]: pgmap v11571: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:24:18.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:18 smithi022 ceph-mon[31360]: pgmap v11572: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:18.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:18 smithi022 ceph-mon[40250]: pgmap v11572: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:18 smithi122 ceph-mon[32255]: pgmap v11572: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:19.382 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:19.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:19.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:19.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:20] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:24:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[31360]: pgmap v11573: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:24:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:24:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:24:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1463990676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:20.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:24:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[40250]: pgmap v11573: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:24:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:24:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:24:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1463990676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:20.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:24:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:20 smithi122 ceph-mon[32255]: pgmap v11573: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:24:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:24:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:24:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1463990676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:24:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:22 smithi122 ceph-mon[32255]: pgmap v11574: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:22.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:24:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:24:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:24:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:24:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:22 smithi022 ceph-mon[31360]: pgmap v11574: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:22 smithi022 ceph-mon[40250]: pgmap v11574: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:24.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:24.755 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:24 smithi022 ceph-mon[40250]: pgmap v11575: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:24.756 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:24.756 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:24 smithi022 ceph-mon[31360]: pgmap v11575: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:24.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:24 smithi122 ceph-mon[32255]: pgmap v11575: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:25.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:25.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:25.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2985326729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2985326729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2985326729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:26 smithi022 ceph-mon[31360]: pgmap v11576: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:26 smithi022 ceph-mon[40250]: pgmap v11576: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:26 smithi122 ceph-mon[32255]: pgmap v11576: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:27 smithi022 ceph-mon[31360]: pgmap v11577: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:24:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:27 smithi022 ceph-mon[40250]: pgmap v11577: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:27 smithi122 ceph-mon[32255]: pgmap v11577: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:30.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:30 smithi122 ceph-mon[32255]: pgmap v11578: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:30] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:24:30.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:30.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:30.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:30.497 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:30 smithi022 ceph-mon[31360]: pgmap v11578: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:30.498 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:30 smithi022 ceph-mon[40250]: pgmap v11578: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/790657170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/790657170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/790657170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:32 smithi122 ceph-mon[32255]: pgmap v11579: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:24:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:24:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:24:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:24:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:32 smithi022 ceph-mon[31360]: pgmap v11579: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:32 smithi022 ceph-mon[40250]: pgmap v11579: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[31360]: pgmap v11580: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:24:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:24:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:24:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:24:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[40250]: pgmap v11580: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:24:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:24:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:24:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:24:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:34 smithi122 ceph-mon[32255]: pgmap v11580: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:24:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:24:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:24:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:24:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:35.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:35.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:35.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:36 smithi022 ceph-mon[31360]: pgmap v11581: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:36.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2078835081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:36 smithi022 ceph-mon[40250]: pgmap v11581: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:36.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2078835081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:36 smithi122 ceph-mon[32255]: pgmap v11581: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2078835081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:37 smithi022 ceph-mon[31360]: pgmap v11582: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:24:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:37 smithi022 ceph-mon[40250]: pgmap v11582: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:37 smithi122 ceph-mon[32255]: pgmap v11582: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:40 smithi122 ceph-mon[32255]: pgmap v11583: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:40] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:24:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:40 smithi022 ceph-mon[31360]: pgmap v11583: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:40 smithi022 ceph-mon[40250]: pgmap v11583: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:40.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:41.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:41.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:41.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1141777835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1141777835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1141777835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:42 smithi122 ceph-mon[32255]: pgmap v11584: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:42.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:24:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:24:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:24:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:24:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:42 smithi022 ceph-mon[31360]: pgmap v11584: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:42 smithi022 ceph-mon[40250]: pgmap v11584: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:44 smithi022 ceph-mon[31360]: pgmap v11585: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:44 smithi022 ceph-mon[40250]: pgmap v11585: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:44 smithi122 ceph-mon[32255]: pgmap v11585: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:46.234 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:46 smithi022 ceph-mon[40250]: pgmap v11586: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:46 smithi022 ceph-mon[31360]: pgmap v11586: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:46.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:46.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:46.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:46.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:46 smithi122 ceph-mon[32255]: pgmap v11586: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:24:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/672684045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:47 smithi022 ceph-mon[31360]: pgmap v11587: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/672684045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:47 smithi022 ceph-mon[40250]: pgmap v11587: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/672684045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:47 smithi122 ceph-mon[32255]: pgmap v11587: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:50] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:24:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:50 smithi122 ceph-mon[32255]: pgmap v11588: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:50 smithi022 ceph-mon[31360]: pgmap v11588: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:50 smithi022 ceph-mon[40250]: pgmap v11588: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:51.604 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:51.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:51.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:51.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:52.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:24:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:24:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:52 smithi122 ceph-mon[32255]: pgmap v11589: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2877529743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:24:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:24:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:52 smithi022 ceph-mon[31360]: pgmap v11589: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2877529743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:52 smithi022 ceph-mon[40250]: pgmap v11589: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2877529743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:54 smithi022 ceph-mon[31360]: pgmap v11590: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:54 smithi022 ceph-mon[40250]: pgmap v11590: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:54.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:24:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:54 smithi122 ceph-mon[32255]: pgmap v11590: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:56 smithi022 ceph-mon[31360]: pgmap v11591: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:56 smithi022 ceph-mon[40250]: pgmap v11591: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:56.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:56 smithi122 ceph-mon[32255]: pgmap v11591: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:56.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:24:57.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:24:57.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:24:57.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:24:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/102970270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:24:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/102970270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/102970270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:24:58.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:58 smithi022 ceph-mon[31360]: pgmap v11592: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:58.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:58 smithi022 ceph-mon[40250]: pgmap v11592: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:58 smithi122 ceph-mon[32255]: pgmap v11592: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:24:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:24:59 smithi022 ceph-mon[31360]: pgmap v11593: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:24:59 smithi022 ceph-mon[40250]: pgmap v11593: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:24:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:24:59 smithi122 ceph-mon[32255]: pgmap v11593: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:00] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:25:02.345 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:02.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:25:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:25:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:25:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:25:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:02 smithi122 ceph-mon[32255]: pgmap v11594: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:02.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:02.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:02.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:02.719 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:02 smithi022 ceph-mon[40250]: pgmap v11594: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:02.719 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:02 smithi022 ceph-mon[31360]: pgmap v11594: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3252677867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3252677867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3252677867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:04 smithi022 ceph-mon[31360]: pgmap v11595: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:04 smithi022 ceph-mon[40250]: pgmap v11595: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:04.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:04 smithi122 ceph-mon[32255]: pgmap v11595: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:06 smithi022 ceph-mon[31360]: pgmap v11596: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:06 smithi022 ceph-mon[40250]: pgmap v11596: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:06 smithi122 ceph-mon[32255]: pgmap v11596: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:07.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:07.720 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:25:08.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:08.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:08.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:08 smithi022 ceph-mon[31360]: pgmap v11597: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/867702984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:08 smithi022 ceph-mon[40250]: pgmap v11597: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/867702984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:08 smithi122 ceph-mon[32255]: pgmap v11597: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/867702984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:10] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:25:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:10 smithi022 ceph-mon[31360]: pgmap v11598: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:10 smithi022 ceph-mon[40250]: pgmap v11598: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:10 smithi122 ceph-mon[32255]: pgmap v11598: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:12.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:25:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:25:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:25:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:25:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:12 smithi122 ceph-mon[32255]: pgmap v11599: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:12 smithi022 ceph-mon[31360]: pgmap v11599: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:12 smithi022 ceph-mon[40250]: pgmap v11599: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:13.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:13.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:13.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:13.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3532060813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3532060813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3532060813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:14 smithi022 ceph-mon[31360]: pgmap v11600: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:14 smithi022 ceph-mon[40250]: pgmap v11600: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:14 smithi122 ceph-mon[32255]: pgmap v11600: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:16 smithi022 ceph-mon[31360]: pgmap v11601: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:16 smithi022 ceph-mon[40250]: pgmap v11601: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:16 smithi122 ceph-mon[32255]: pgmap v11601: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:25:18.464 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:18 smithi022 ceph-mon[31360]: pgmap v11602: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:18 smithi022 ceph-mon[40250]: pgmap v11602: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:18.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:18.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:18.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:18 smithi122 ceph-mon[32255]: pgmap v11602: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:19.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2617114601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:19.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2617114601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:20.023 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2617114601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:20] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:25:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[31360]: pgmap v11603: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:25:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:25:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:25:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[40250]: pgmap v11603: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:25:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:25:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:25:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:20 smithi122 ceph-mon[32255]: pgmap v11603: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:25:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:25:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:25:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:21 smithi022 ceph-mon[31360]: pgmap v11604: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:21 smithi022 ceph-mon[40250]: pgmap v11604: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:21 smithi122 ceph-mon[32255]: pgmap v11604: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:22.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:25:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:25:22.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:25:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:25:23.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:24.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:24.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:24.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:24 smithi122 ceph-mon[32255]: pgmap v11605: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:24 smithi022 ceph-mon[31360]: pgmap v11605: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:24 smithi022 ceph-mon[40250]: pgmap v11605: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:25.146 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.456 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3080M 2021-12-16T00:25:25.457 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:25.458 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3580519548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:25.459 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3080M 2021-12-16T00:25:25.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:25.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3580519548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:25.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:25.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:25:25.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3080M 2021-12-16T00:25:25.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:25.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3580519548' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:25.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:26 smithi022 ceph-mon[31360]: pgmap v11606: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:26 smithi022 ceph-mon[40250]: pgmap v11606: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:26 smithi122 ceph-mon[32255]: pgmap v11606: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:25:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:25:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:28 smithi022 ceph-mon[31360]: pgmap v11607: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:28 smithi022 ceph-mon[40250]: pgmap v11607: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:28 smithi122 ceph-mon[32255]: pgmap v11607: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:29.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:29.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:29.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:29.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:30] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:25:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:30 smithi122 ceph-mon[32255]: pgmap v11608: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/869918461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:30 smithi022 ceph-mon[31360]: pgmap v11608: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/869918461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:30 smithi022 ceph-mon[40250]: pgmap v11608: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/869918461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:32.578 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:32 smithi122 ceph-mon[32255]: pgmap v11609: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:32.578 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:25:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:25:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:32 smithi022 ceph-mon[31360]: pgmap v11609: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:32 smithi022 ceph-mon[40250]: pgmap v11609: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:32.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:25:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:25:34.577 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[31360]: pgmap v11610: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:25:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:25:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:25:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:25:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[40250]: pgmap v11610: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:25:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:25:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:25:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:25:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:34 smithi122 ceph-mon[32255]: pgmap v11610: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:25:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:25:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:25:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:25:34.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:34.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:34.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/511773134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/511773134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/511773134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:36 smithi022 ceph-mon[31360]: pgmap v11611: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:36 smithi022 ceph-mon[40250]: pgmap v11611: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:36 smithi122 ceph-mon[32255]: pgmap v11611: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:37 smithi022 ceph-mon[31360]: pgmap v11612: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:25:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:37 smithi022 ceph-mon[40250]: pgmap v11612: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:37 smithi122 ceph-mon[32255]: pgmap v11612: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:39.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:40 smithi022 ceph-mon[31360]: pgmap v11613: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:40 smithi022 ceph-mon[40250]: pgmap v11613: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:40.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:40.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:40.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:40] "GET /metrics HTTP/1.1" 200 202417 "" "Prometheus/2.18.1" 2021-12-16T00:25:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:40 smithi122 ceph-mon[32255]: pgmap v11613: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4209241442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4209241442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4209241442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:42.579 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:42 smithi122 ceph-mon[32255]: pgmap v11614: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:42.580 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:25:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:25:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:42 smithi022 ceph-mon[31360]: pgmap v11614: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:42 smithi022 ceph-mon[40250]: pgmap v11614: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:42.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:25:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:25:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:44 smithi022 ceph-mon[31360]: pgmap v11615: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:44 smithi022 ceph-mon[40250]: pgmap v11615: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:44 smithi122 ceph-mon[32255]: pgmap v11615: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:45.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:45.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:46 smithi022 ceph-mon[31360]: pgmap v11616: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4176896241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:46 smithi022 ceph-mon[40250]: pgmap v11616: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4176896241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:46 smithi122 ceph-mon[32255]: pgmap v11616: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4176896241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:47 smithi022 ceph-mon[31360]: pgmap v11617: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:25:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:47 smithi022 ceph-mon[40250]: pgmap v11617: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:47 smithi122 ceph-mon[32255]: pgmap v11617: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:50 smithi122 ceph-mon[32255]: pgmap v11618: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:50] "GET /metrics HTTP/1.1" 200 202415 "" "Prometheus/2.18.1" 2021-12-16T00:25:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:50 smithi022 ceph-mon[31360]: pgmap v11618: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:50 smithi022 ceph-mon[40250]: pgmap v11618: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:50.692 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:51.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:51.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:51.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1500877414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1500877414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1500877414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:52.590 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:52 smithi122 ceph-mon[32255]: pgmap v11619: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:52.591 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:25:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:25:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:52 smithi022 ceph-mon[31360]: pgmap v11619: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:52 smithi022 ceph-mon[40250]: pgmap v11619: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:52.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:25:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:25:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:53 smithi122 ceph-mon[32255]: pgmap v11620: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:53 smithi022 ceph-mon[31360]: pgmap v11620: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:25:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:53 smithi022 ceph-mon[40250]: pgmap v11620: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:56.063 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:25:56.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:25:56.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:25:56.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:25:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:56 smithi022 ceph-mon[31360]: pgmap v11621: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/351178020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:56 smithi022 ceph-mon[40250]: pgmap v11621: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/351178020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:56.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:56 smithi122 ceph-mon[32255]: pgmap v11621: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:25:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/351178020' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:25:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:25:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:25:58 smithi022 ceph-mon[31360]: pgmap v11622: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:25:58 smithi022 ceph-mon[40250]: pgmap v11622: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:25:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:25:58 smithi122 ceph-mon[32255]: pgmap v11622: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:00] "GET /metrics HTTP/1.1" 200 202415 "" "Prometheus/2.18.1" 2021-12-16T00:26:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:00 smithi022 ceph-mon[31360]: pgmap v11623: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:00 smithi022 ceph-mon[40250]: pgmap v11623: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:00.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:00 smithi122 ceph-mon[32255]: pgmap v11623: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:01.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:01 smithi022 ceph-mon[31360]: pgmap v11624: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:01 smithi022 ceph-mon[40250]: pgmap v11624: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:01 smithi122 ceph-mon[32255]: pgmap v11624: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:02.601 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3205014466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:02.601 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:26:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:26:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3205014466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3205014466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:02.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:26:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:26:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:03 smithi022 ceph-mon[31360]: pgmap v11625: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:03 smithi022 ceph-mon[40250]: pgmap v11625: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:03.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:03 smithi122 ceph-mon[32255]: pgmap v11625: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:06 smithi022 ceph-mon[31360]: pgmap v11626: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:06 smithi022 ceph-mon[40250]: pgmap v11626: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:06.802 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:06 smithi122 ceph-mon[32255]: pgmap v11626: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:07.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:07.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:07.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:07.494 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:26:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1306309858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1306309858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:07.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1306309858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:08 smithi022 ceph-mon[31360]: pgmap v11627: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:08 smithi022 ceph-mon[40250]: pgmap v11627: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:08 smithi122 ceph-mon[32255]: pgmap v11627: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:09 smithi022 ceph-mon[31360]: pgmap v11628: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:09 smithi022 ceph-mon[40250]: pgmap v11628: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:09 smithi122 ceph-mon[32255]: pgmap v11628: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:10] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:26:12.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:12.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:12.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:12.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:12.612 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:12 smithi122 ceph-mon[32255]: pgmap v11629: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:12.612 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:26:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:26:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:12 smithi022 ceph-mon[31360]: pgmap v11629: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:12 smithi022 ceph-mon[40250]: pgmap v11629: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:12.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:26:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:26:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/24872862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/24872862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/24872862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:14 smithi022 ceph-mon[31360]: pgmap v11630: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:14 smithi022 ceph-mon[40250]: pgmap v11630: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:14 smithi122 ceph-mon[32255]: pgmap v11630: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:16 smithi022 ceph-mon[31360]: pgmap v11631: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:16 smithi022 ceph-mon[40250]: pgmap v11631: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:16 smithi122 ceph-mon[32255]: pgmap v11631: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:26:17.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:17.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:17.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:18 smithi022 ceph-mon[31360]: pgmap v11632: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2098597379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:18 smithi022 ceph-mon[40250]: pgmap v11632: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2098597379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:18 smithi122 ceph-mon[32255]: pgmap v11632: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2098597379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:20] "GET /metrics HTTP/1.1" 200 202415 "" "Prometheus/2.18.1" 2021-12-16T00:26:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:20 smithi022 ceph-mon[31360]: pgmap v11633: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:20 smithi022 ceph-mon[40250]: pgmap v11633: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:20 smithi122 ceph-mon[32255]: pgmap v11633: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:21 smithi022 ceph-mon[31360]: pgmap v11634: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:21 smithi022 ceph-mon[40250]: pgmap v11634: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:21 smithi122 ceph-mon[32255]: pgmap v11634: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:22.622 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:26:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:26:22.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:22.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:26:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:26:23.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:23.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:23.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:24 smithi122 ceph-mon[32255]: pgmap v11635: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1075110972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:24 smithi022 ceph-mon[31360]: pgmap v11635: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1075110972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:24 smithi022 ceph-mon[40250]: pgmap v11635: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1075110972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:26 smithi122 ceph-mon[32255]: pgmap v11636: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:26:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:26:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:26:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:26:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[31360]: pgmap v11636: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:26:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:26:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:26:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:26:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[40250]: pgmap v11636: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:26:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:26:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:26:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:26:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:26:28.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:28 smithi122 ceph-mon[32255]: pgmap v11637: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:28 smithi022 ceph-mon[31360]: pgmap v11637: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:28 smithi022 ceph-mon[40250]: pgmap v11637: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:28.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:28.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/85076836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/85076836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/85076836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:30 smithi122 ceph-mon[32255]: pgmap v11638: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:30] "GET /metrics HTTP/1.1" 200 202415 "" "Prometheus/2.18.1" 2021-12-16T00:26:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:30 smithi022 ceph-mon[31360]: pgmap v11638: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:30 smithi022 ceph-mon[40250]: pgmap v11638: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:32.633 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:32 smithi122 ceph-mon[32255]: pgmap v11639: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:32.634 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:26:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:26:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:32 smithi022 ceph-mon[31360]: pgmap v11639: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:32 smithi022 ceph-mon[40250]: pgmap v11639: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:32.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:26:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:26:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:34.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:34.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:34.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[31360]: pgmap v11640: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:26:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:26:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:26:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:26:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2949484625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[40250]: pgmap v11640: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:26:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:26:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:26:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:26:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2949484625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:34 smithi122 ceph-mon[32255]: pgmap v11640: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:26:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:26:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2949484625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:36 smithi022 ceph-mon[31360]: pgmap v11641: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:36 smithi022 ceph-mon[40250]: pgmap v11641: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:36 smithi122 ceph-mon[32255]: pgmap v11641: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:26:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:38 smithi022 ceph-mon[31360]: pgmap v11642: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:38 smithi022 ceph-mon[40250]: pgmap v11642: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:38.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:38 smithi122 ceph-mon[32255]: pgmap v11642: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:39.026 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:39.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:39.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:39.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1467815246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1467815246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:39.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1467815246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:40] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:26:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:40 smithi022 ceph-mon[31360]: pgmap v11643: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:40 smithi022 ceph-mon[40250]: pgmap v11643: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:40.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:40 smithi122 ceph-mon[32255]: pgmap v11643: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:41 smithi022 ceph-mon[31360]: pgmap v11644: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:41 smithi022 ceph-mon[40250]: pgmap v11644: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:41 smithi122 ceph-mon[32255]: pgmap v11644: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:42.643 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:26:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:26:42.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:26:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:26:44.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:44 smithi122 ceph-mon[32255]: pgmap v11645: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:44.539 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:44.539 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:44 smithi022 ceph-mon[31360]: pgmap v11645: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:44.540 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:44.540 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:44 smithi022 ceph-mon[40250]: pgmap v11645: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:44.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:44.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:44.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1723800398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1723800398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1723800398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:46 smithi022 ceph-mon[31360]: pgmap v11646: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:46 smithi022 ceph-mon[40250]: pgmap v11646: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:46.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:46 smithi122 ceph-mon[32255]: pgmap v11646: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:26:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:48 smithi022 ceph-mon[31360]: pgmap v11647: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:48 smithi022 ceph-mon[40250]: pgmap v11647: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:48.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:48 smithi122 ceph-mon[32255]: pgmap v11647: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:49.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:49.767 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:49 smithi022 ceph-mon[31360]: pgmap v11648: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:49.768 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:49 smithi022 ceph-mon[40250]: pgmap v11648: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:49.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:49 smithi122 ceph-mon[32255]: pgmap v11648: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:50.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:50.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:50.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:50] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:26:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2250927956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2250927956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2250927956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:51 smithi022 ceph-mon[31360]: pgmap v11649: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:51 smithi022 ceph-mon[40250]: pgmap v11649: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:51 smithi122 ceph-mon[32255]: pgmap v11649: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:52.647 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:26:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:26:52.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:26:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:26:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:54 smithi122 ceph-mon[32255]: pgmap v11650: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:54.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:54 smithi022 ceph-mon[31360]: pgmap v11650: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:54.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:26:54.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:54 smithi022 ceph-mon[40250]: pgmap v11650: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:55.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:26:55.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:26:55.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:26:55.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:26:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:56 smithi022 ceph-mon[40250]: pgmap v11651: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2726582688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:56 smithi022 ceph-mon[31360]: pgmap v11651: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2726582688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:56 smithi122 ceph-mon[32255]: pgmap v11651: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2726582688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:26:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:26:58.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:58 smithi022 ceph-mon[31360]: pgmap v11652: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:58.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:58 smithi022 ceph-mon[40250]: pgmap v11652: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:58 smithi122 ceph-mon[32255]: pgmap v11652: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:26:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:26:59 smithi022 ceph-mon[31360]: pgmap v11653: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:26:59 smithi022 ceph-mon[40250]: pgmap v11653: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:26:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:26:59 smithi122 ceph-mon[32255]: pgmap v11653: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:00] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:27:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:00.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:00.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:00.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1669525004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1669525004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:01.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1669525004' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:02.657 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:02 smithi122 ceph-mon[32255]: pgmap v11654: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:02.657 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:27:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:27:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:02 smithi022 ceph-mon[31360]: pgmap v11654: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:02 smithi022 ceph-mon[40250]: pgmap v11654: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:02.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:27:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:27:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:03 smithi022 ceph-mon[31360]: pgmap v11655: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:03 smithi022 ceph-mon[40250]: pgmap v11655: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:03 smithi122 ceph-mon[32255]: pgmap v11655: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:05.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:06.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:06.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:06.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:06 smithi022 ceph-mon[31360]: pgmap v11656: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/242553480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:06 smithi022 ceph-mon[40250]: pgmap v11656: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/242553480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:06.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:06 smithi122 ceph-mon[32255]: pgmap v11656: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/242553480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:27:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:08 smithi022 ceph-mon[31360]: pgmap v11657: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:08 smithi022 ceph-mon[40250]: pgmap v11657: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:08 smithi122 ceph-mon[32255]: pgmap v11657: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:09 smithi022 ceph-mon[40250]: pgmap v11658: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:09 smithi022 ceph-mon[31360]: pgmap v11658: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:09 smithi122 ceph-mon[32255]: pgmap v11658: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:10] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:27:11.257 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:11.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:11.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:11.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:12.667 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:27:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:27:12.668 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:12 smithi122 ceph-mon[32255]: pgmap v11659: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:12.668 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4058891683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:12 smithi022 ceph-mon[31360]: pgmap v11659: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4058891683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:12 smithi022 ceph-mon[40250]: pgmap v11659: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4058891683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:12.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:27:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:27:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:13.798 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:13 smithi022 ceph-mon[40250]: pgmap v11660: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:13.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:13.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:13 smithi022 ceph-mon[31360]: pgmap v11660: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:13 smithi122 ceph-mon[32255]: pgmap v11660: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:16.628 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:16 smithi022 ceph-mon[31360]: pgmap v11661: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:16 smithi022 ceph-mon[40250]: pgmap v11661: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:16 smithi122 ceph-mon[32255]: pgmap v11661: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:16.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:16.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:17.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3712407541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:27:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3712407541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3712407541' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:18 smithi022 ceph-mon[31360]: pgmap v11662: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:18 smithi022 ceph-mon[40250]: pgmap v11662: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:18 smithi122 ceph-mon[32255]: pgmap v11662: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:19 smithi022 ceph-mon[31360]: pgmap v11663: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:19 smithi022 ceph-mon[40250]: pgmap v11663: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:19 smithi122 ceph-mon[32255]: pgmap v11663: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:20] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:27:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:21 smithi122 ceph-mon[32255]: pgmap v11664: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:22.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:22.002 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:21 smithi022 ceph-mon[31360]: pgmap v11664: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:22.002 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:21 smithi022 ceph-mon[40250]: pgmap v11664: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:22.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:22.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:22.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:22.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2373913162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:22.677 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:27:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:27:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2373913162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2373913162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:22.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:27:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:27:23.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:23 smithi122 ceph-mon[32255]: pgmap v11665: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:24.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:23 smithi022 ceph-mon[31360]: pgmap v11665: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:24.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:24.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:23 smithi022 ceph-mon[40250]: pgmap v11665: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[31360]: pgmap v11666: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:27:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:27:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:27:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:27:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[40250]: pgmap v11666: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:27:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:27:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:27:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:27:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:26 smithi122 ceph-mon[32255]: pgmap v11666: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:27:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:27:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:27:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:27:27.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:27.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:27.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:27.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:27.740 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:27:27.741 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:27 smithi022 ceph-mon[31360]: pgmap v11667: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:27.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:27 smithi022 ceph-mon[40250]: pgmap v11667: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:27 smithi122 ceph-mon[32255]: pgmap v11667: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3215165739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3215165739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:28.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3215165739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:29 smithi022 ceph-mon[31360]: pgmap v11668: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:29 smithi022 ceph-mon[40250]: pgmap v11668: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:29 smithi122 ceph-mon[32255]: pgmap v11668: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:30] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:27:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:31 smithi022 ceph-mon[31360]: pgmap v11669: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:31 smithi022 ceph-mon[40250]: pgmap v11669: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:31 smithi122 ceph-mon[32255]: pgmap v11669: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:32.683 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:27:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:27:32.741 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:32.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:27:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:27:33.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:33.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:33.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1885384231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1885384231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1885384231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[31360]: pgmap v11670: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:27:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:27:34.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:27:34.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:27:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[40250]: pgmap v11670: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:27:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:27:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:27:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:27:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:34 smithi122 ceph-mon[32255]: pgmap v11670: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:27:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:27:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:27:34.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:27:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:36 smithi022 ceph-mon[31360]: pgmap v11671: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:36 smithi022 ceph-mon[40250]: pgmap v11671: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:36 smithi122 ceph-mon[32255]: pgmap v11671: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:27:38.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:38.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:38.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:38.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:38.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:38 smithi022 ceph-mon[31360]: pgmap v11672: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:38.487 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:38 smithi022 ceph-mon[40250]: pgmap v11672: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:38 smithi122 ceph-mon[32255]: pgmap v11672: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1172390098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1172390098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1172390098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:40 smithi122 ceph-mon[32255]: pgmap v11673: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:40] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:27:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:40 smithi022 ceph-mon[31360]: pgmap v11673: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:40 smithi022 ceph-mon[40250]: pgmap v11673: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:41 smithi022 ceph-mon[31360]: pgmap v11674: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:41 smithi022 ceph-mon[40250]: pgmap v11674: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:41 smithi122 ceph-mon[32255]: pgmap v11674: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:27:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:27:43.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:27:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:27:43.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:43.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:43.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:43.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:44 smithi022 ceph-mon[31360]: pgmap v11675: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1935123463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:44 smithi022 ceph-mon[40250]: pgmap v11675: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1935123463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:44 smithi122 ceph-mon[32255]: pgmap v11675: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1935123463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:46 smithi022 ceph-mon[31360]: pgmap v11676: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:46 smithi022 ceph-mon[40250]: pgmap v11676: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:46 smithi122 ceph-mon[32255]: pgmap v11676: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:27:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:48 smithi022 ceph-mon[31360]: pgmap v11677: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:48 smithi022 ceph-mon[40250]: pgmap v11677: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:48 smithi122 ceph-mon[32255]: pgmap v11677: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:49.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:49.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:49.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:50] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:27:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:50 smithi122 ceph-mon[32255]: pgmap v11678: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/166142445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:50 smithi022 ceph-mon[31360]: pgmap v11678: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/166142445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:50 smithi022 ceph-mon[40250]: pgmap v11678: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/166142445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:51 smithi022 ceph-mon[31360]: pgmap v11679: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:51 smithi022 ceph-mon[40250]: pgmap v11679: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:51 smithi122 ceph-mon[32255]: pgmap v11679: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:27:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:27:53.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:27:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:27:54.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:54 smithi022 ceph-mon[31360]: pgmap v11680: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:54 smithi022 ceph-mon[40250]: pgmap v11680: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:54.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:54.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:54.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:27:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:27:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:54 smithi122 ceph-mon[32255]: pgmap v11680: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3728006082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3728006082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3728006082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:27:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:56 smithi022 ceph-mon[31360]: pgmap v11681: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:56 smithi022 ceph-mon[40250]: pgmap v11681: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:56 smithi122 ceph-mon[32255]: pgmap v11681: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:27:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:27:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:27:58 smithi022 ceph-mon[31360]: pgmap v11682: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:27:58 smithi022 ceph-mon[40250]: pgmap v11682: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:27:58 smithi122 ceph-mon[32255]: pgmap v11682: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:27:59.596 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:27:59.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:27:59.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:27:59.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:00 smithi022 ceph-mon[31360]: pgmap v11683: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1238179247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:00 smithi022 ceph-mon[40250]: pgmap v11683: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1238179247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:00] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:28:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:00 smithi122 ceph-mon[32255]: pgmap v11683: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1238179247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:01 smithi022 ceph-mon[31360]: pgmap v11684: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:01 smithi022 ceph-mon[40250]: pgmap v11684: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:01.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:01 smithi122 ceph-mon[32255]: pgmap v11684: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:28:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:28:03.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:28:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:28:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:04 smithi022 ceph-mon[31360]: pgmap v11685: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:04 smithi022 ceph-mon[40250]: pgmap v11685: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:04 smithi122 ceph-mon[32255]: pgmap v11685: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:04.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:05.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:05.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:05.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:06 smithi022 ceph-mon[31360]: pgmap v11686: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3457935410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:06 smithi022 ceph-mon[40250]: pgmap v11686: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3457935410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:06 smithi122 ceph-mon[32255]: pgmap v11686: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3457935410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:28:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:08 smithi022 ceph-mon[31360]: pgmap v11687: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:08 smithi022 ceph-mon[40250]: pgmap v11687: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:08 smithi122 ceph-mon[32255]: pgmap v11687: 132 pgs: 132 active+clean; 582 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:28:10.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:10] "GET /metrics HTTP/1.1" 200 202415 "" "Prometheus/2.18.1" 2021-12-16T00:28:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:10 smithi122 ceph-mon[32255]: pgmap v11688: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:28:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:10 smithi022 ceph-mon[31360]: pgmap v11688: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:28:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:10 smithi022 ceph-mon[40250]: pgmap v11688: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:28:10.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:10.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:10.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/204975307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:11 smithi022 ceph-mon[31360]: pgmap v11689: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/204975307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:11 smithi022 ceph-mon[40250]: pgmap v11689: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/204975307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:11 smithi122 ceph-mon[32255]: pgmap v11689: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:28:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:28:13.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:28:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:28:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:14 smithi022 ceph-mon[31360]: pgmap v11690: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:14 smithi022 ceph-mon[40250]: pgmap v11690: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:14 smithi122 ceph-mon[32255]: pgmap v11690: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:15.726 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:16.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:16.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:16.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:16 smithi022 ceph-mon[31360]: pgmap v11691: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:28:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1847322879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:16 smithi022 ceph-mon[40250]: pgmap v11691: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:28:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1847322879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:16 smithi122 ceph-mon[32255]: pgmap v11691: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:28:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1847322879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:28:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:18 smithi022 ceph-mon[31360]: pgmap v11692: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:18 smithi022 ceph-mon[40250]: pgmap v11692: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:18 smithi122 ceph-mon[32255]: pgmap v11692: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:28:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:20] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:28:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:20 smithi122 ceph-mon[32255]: pgmap v11693: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:28:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:20 smithi022 ceph-mon[31360]: pgmap v11693: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:28:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:20 smithi022 ceph-mon[40250]: pgmap v11693: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:28:21.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:21.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:21.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:21.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:21 smithi022 ceph-mon[31360]: pgmap v11694: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2320118554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:21 smithi022 ceph-mon[40250]: pgmap v11694: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2320118554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:21 smithi122 ceph-mon[32255]: pgmap v11694: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2320118554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:28:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:28:23.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:28:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:28:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:24.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:24 smithi022 ceph-mon[31360]: pgmap v11695: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:24.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:24 smithi022 ceph-mon[40250]: pgmap v11695: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:24 smithi122 ceph-mon[32255]: pgmap v11695: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:26.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:26.468 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[40250]: pgmap v11696: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:26.468 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:28:26.469 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:28:26.469 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:28:26.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[31360]: pgmap v11696: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:26.470 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:28:26.470 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:28:26.470 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:28:26.601 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:26 smithi122 ceph-mon[32255]: pgmap v11696: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:26.602 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:28:26.602 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:28:26.602 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:28:26.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:26.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:26.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:28:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:28:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/708043353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:27 smithi022 ceph-mon[31360]: pgmap v11697: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:28:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/708043353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:27.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:27 smithi022 ceph-mon[40250]: pgmap v11697: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:28:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/708043353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:27 smithi122 ceph-mon[32255]: pgmap v11697: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:30 smithi122 ceph-mon[32255]: pgmap v11698: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:30] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:28:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:30 smithi022 ceph-mon[31360]: pgmap v11698: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:30 smithi022 ceph-mon[40250]: pgmap v11698: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:31 smithi022 ceph-mon[31360]: pgmap v11699: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:31 smithi022 ceph-mon[40250]: pgmap v11699: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:31.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:31 smithi122 ceph-mon[32255]: pgmap v11699: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:32.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:32.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:32.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2692889802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:28:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:28:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2692889802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2692889802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:33.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:28:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:28:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:33 smithi122 ceph-mon[32255]: pgmap v11700: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:33 smithi022 ceph-mon[31360]: pgmap v11700: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:33 smithi022 ceph-mon[40250]: pgmap v11700: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:28:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:28:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:28:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:28:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:28:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:28:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:28:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:28:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:28:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:28:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:28:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:28:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:35 smithi122 ceph-mon[32255]: pgmap v11701: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:35 smithi022 ceph-mon[31360]: pgmap v11701: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:35 smithi022 ceph-mon[40250]: pgmap v11701: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:37.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:28:37.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:37.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:37.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:38 smithi022 ceph-mon[31360]: pgmap v11702: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1971081778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:38 smithi022 ceph-mon[40250]: pgmap v11702: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1971081778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:38 smithi122 ceph-mon[32255]: pgmap v11702: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1971081778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:40] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:28:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:40 smithi122 ceph-mon[32255]: pgmap v11703: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:40 smithi022 ceph-mon[31360]: pgmap v11703: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:40 smithi022 ceph-mon[40250]: pgmap v11703: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:41 smithi022 ceph-mon[31360]: pgmap v11704: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:41 smithi022 ceph-mon[40250]: pgmap v11704: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:41 smithi122 ceph-mon[32255]: pgmap v11704: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:42.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:28:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:28:42.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:42.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:42.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:43.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/131746508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:28:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:28:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/131746508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/131746508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:43 smithi022 ceph-mon[31360]: pgmap v11705: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:43 smithi022 ceph-mon[40250]: pgmap v11705: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:43 smithi122 ceph-mon[32255]: pgmap v11705: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:46 smithi022 ceph-mon[31360]: pgmap v11706: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:46 smithi022 ceph-mon[40250]: pgmap v11706: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:46 smithi122 ceph-mon[32255]: pgmap v11706: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:28:47.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:48.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:48.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:48.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:48 smithi022 ceph-mon[31360]: pgmap v11707: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:48 smithi022 ceph-mon[40250]: pgmap v11707: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:48 smithi122 ceph-mon[32255]: pgmap v11707: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1650222780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1650222780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1650222780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:50] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:28:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:50 smithi122 ceph-mon[32255]: pgmap v11708: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:50 smithi022 ceph-mon[31360]: pgmap v11708: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:50 smithi022 ceph-mon[40250]: pgmap v11708: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:51 smithi022 ceph-mon[31360]: pgmap v11709: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:51 smithi022 ceph-mon[40250]: pgmap v11709: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:51 smithi122 ceph-mon[32255]: pgmap v11709: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:28:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:28:53.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:28:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:28:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:53.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:53.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:53.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:54 smithi022 ceph-mon[31360]: pgmap v11710: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/631169728' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:54 smithi022 ceph-mon[40250]: pgmap v11710: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/631169728' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:28:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:54 smithi122 ceph-mon[32255]: pgmap v11710: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:28:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/631169728' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:56 smithi022 ceph-mon[31360]: pgmap v11711: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:56 smithi022 ceph-mon[40250]: pgmap v11711: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:56 smithi122 ceph-mon[32255]: pgmap v11711: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:28:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:58 smithi022 ceph-mon[31360]: pgmap v11712: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:58 smithi022 ceph-mon[40250]: pgmap v11712: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:58.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:28:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:58 smithi122 ceph-mon[32255]: pgmap v11712: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:28:59.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:28:59.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:28:59.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:28:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:28:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2325172190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:28:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2325172190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:28:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:28:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2325172190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:00 smithi122 ceph-mon[32255]: pgmap v11713: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:00] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:29:00.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:00 smithi022 ceph-mon[31360]: pgmap v11713: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:00.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:00 smithi022 ceph-mon[40250]: pgmap v11713: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:01.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:01 smithi122 ceph-mon[32255]: pgmap v11714: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:01 smithi022 ceph-mon[31360]: pgmap v11714: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:01 smithi022 ceph-mon[40250]: pgmap v11714: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:29:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:29:03.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:29:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:29:04.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:04.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:04.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:04.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:04.428 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:04.429 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:04 smithi022 ceph-mon[31360]: pgmap v11715: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:04.429 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:04.429 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:04 smithi022 ceph-mon[40250]: pgmap v11715: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:04 smithi122 ceph-mon[32255]: pgmap v11715: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3060120342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3060120342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3060120342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:06 smithi022 ceph-mon[31360]: pgmap v11716: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:06 smithi022 ceph-mon[40250]: pgmap v11716: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:06 smithi122 ceph-mon[32255]: pgmap v11716: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:29:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:08 smithi022 ceph-mon[31360]: pgmap v11717: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:08 smithi022 ceph-mon[40250]: pgmap v11717: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:08 smithi122 ceph-mon[32255]: pgmap v11717: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:09.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:09.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:09.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:10] "GET /metrics HTTP/1.1" 200 202419 "" "Prometheus/2.18.1" 2021-12-16T00:29:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:10 smithi122 ceph-mon[32255]: pgmap v11718: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/605575917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:10.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:10 smithi022 ceph-mon[31360]: pgmap v11718: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:10.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/605575917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:10.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:10 smithi022 ceph-mon[40250]: pgmap v11718: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:10.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/605575917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:11 smithi122 ceph-mon[32255]: pgmap v11719: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:11 smithi022 ceph-mon[31360]: pgmap v11719: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:11 smithi022 ceph-mon[40250]: pgmap v11719: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:29:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:29:13.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:29:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:29:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:14 smithi022 ceph-mon[31360]: pgmap v11720: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:14 smithi022 ceph-mon[40250]: pgmap v11720: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:14 smithi122 ceph-mon[32255]: pgmap v11720: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:14.802 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:15.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:15.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:15.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4072944428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4072944428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4072944428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:16 smithi022 ceph-mon[31360]: pgmap v11721: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:16 smithi022 ceph-mon[40250]: pgmap v11721: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:16 smithi122 ceph-mon[32255]: pgmap v11721: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:29:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:18 smithi022 ceph-mon[31360]: pgmap v11722: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:18 smithi022 ceph-mon[40250]: pgmap v11722: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:18 smithi122 ceph-mon[32255]: pgmap v11722: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:20.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:20] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:29:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:20 smithi122 ceph-mon[32255]: pgmap v11723: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:20 smithi022 ceph-mon[31360]: pgmap v11723: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:20 smithi022 ceph-mon[40250]: pgmap v11723: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:20.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:20.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:20.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3122435428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3122435428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3122435428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:22 smithi022 ceph-mon[31360]: pgmap v11724: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:22 smithi022 ceph-mon[40250]: pgmap v11724: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:22 smithi122 ceph-mon[32255]: pgmap v11724: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:29:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:29:23.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:29:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:29:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:24 smithi022 ceph-mon[31360]: pgmap v11725: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:24 smithi022 ceph-mon[40250]: pgmap v11725: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:24 smithi122 ceph-mon[32255]: pgmap v11725: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:25.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:25.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:25.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:25.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:26 smithi022 ceph-mon[31360]: pgmap v11726: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2161009149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:26 smithi022 ceph-mon[40250]: pgmap v11726: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2161009149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:26.678 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:26 smithi122 ceph-mon[32255]: pgmap v11726: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:26.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2161009149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:29:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:29:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:29:27.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:29:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:29:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:29:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:29:27.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:29:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:29:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:29:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:29:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:29:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:29:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:28 smithi022 ceph-mon[31360]: pgmap v11727: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:28 smithi022 ceph-mon[40250]: pgmap v11727: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:28 smithi122 ceph-mon[32255]: pgmap v11727: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:30 smithi122 ceph-mon[32255]: pgmap v11728: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:30] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:29:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:30 smithi022 ceph-mon[31360]: pgmap v11728: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:30 smithi022 ceph-mon[40250]: pgmap v11728: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:30.914 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:31.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:31.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:31.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:31 smithi122 ceph-mon[32255]: pgmap v11729: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/74960550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:31 smithi022 ceph-mon[31360]: pgmap v11729: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/74960550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:31 smithi022 ceph-mon[40250]: pgmap v11729: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/74960550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:29:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:29:33.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:29:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:29:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[31360]: pgmap v11730: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:29:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:29:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:29:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:29:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[40250]: pgmap v11730: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:29:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:29:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:29:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:29:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:34 smithi122 ceph-mon[32255]: pgmap v11730: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:29:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:29:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:29:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:29:36.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:36 smithi022 ceph-mon[31360]: pgmap v11731: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:36 smithi022 ceph-mon[40250]: pgmap v11731: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:36.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:36.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:36.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:36 smithi122 ceph-mon[32255]: pgmap v11731: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/523805480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:29:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/523805480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/523805480' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:38 smithi022 ceph-mon[31360]: pgmap v11732: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:38 smithi022 ceph-mon[40250]: pgmap v11732: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:38 smithi122 ceph-mon[32255]: pgmap v11732: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:40 smithi122 ceph-mon[32255]: pgmap v11733: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:40] "GET /metrics HTTP/1.1" 200 202410 "" "Prometheus/2.18.1" 2021-12-16T00:29:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:40 smithi022 ceph-mon[31360]: pgmap v11733: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:40 smithi022 ceph-mon[40250]: pgmap v11733: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:41.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:41 smithi122 ceph-mon[32255]: pgmap v11734: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:42.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:42.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:42.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:42.021 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:41 smithi022 ceph-mon[31360]: pgmap v11734: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:42.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:41 smithi022 ceph-mon[40250]: pgmap v11734: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2752437118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:29:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:29:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2752437118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2752437118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:43.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:29:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:29:43.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:43 smithi122 ceph-mon[32255]: pgmap v11735: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:43 smithi022 ceph-mon[31360]: pgmap v11735: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:43 smithi022 ceph-mon[40250]: pgmap v11735: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:46 smithi022 ceph-mon[31360]: pgmap v11736: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:46 smithi022 ceph-mon[40250]: pgmap v11736: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:46 smithi122 ceph-mon[32255]: pgmap v11736: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:47.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:47.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:47.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:47.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:29:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:48 smithi022 ceph-mon[31360]: pgmap v11737: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2095792475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:48 smithi022 ceph-mon[40250]: pgmap v11737: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2095792475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:48 smithi122 ceph-mon[32255]: pgmap v11737: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2095792475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:50 smithi122 ceph-mon[32255]: pgmap v11738: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:50] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:29:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:50 smithi022 ceph-mon[31360]: pgmap v11738: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:50 smithi022 ceph-mon[40250]: pgmap v11738: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:51 smithi122 ceph-mon[32255]: pgmap v11739: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:51 smithi022 ceph-mon[31360]: pgmap v11739: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:51 smithi022 ceph-mon[40250]: pgmap v11739: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:52.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:29:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:29:52.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/177602228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/177602228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:53.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:29:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:29:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/177602228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:54.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:53 smithi122 ceph-mon[32255]: pgmap v11740: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:53 smithi022 ceph-mon[31360]: pgmap v11740: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:29:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:53 smithi022 ceph-mon[40250]: pgmap v11740: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:29:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:56 smithi022 ceph-mon[31360]: pgmap v11741: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:56 smithi022 ceph-mon[40250]: pgmap v11741: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:56 smithi122 ceph-mon[32255]: pgmap v11741: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:57.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:29:57.764 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:29:58.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:29:58.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:29:58.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:29:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:58 smithi022 ceph-mon[31360]: pgmap v11742: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:29:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3614098979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:58 smithi022 ceph-mon[40250]: pgmap v11742: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:29:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3614098979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:29:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:58 smithi122 ceph-mon[32255]: pgmap v11742: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:29:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:29:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3614098979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:00] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:30:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:00 smithi122 ceph-mon[32255]: pgmap v11743: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T00:30:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:00 smithi022 ceph-mon[31360]: pgmap v11743: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T00:30:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:00 smithi022 ceph-mon[40250]: pgmap v11743: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T00:30:01.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:01 smithi122 ceph-mon[32255]: pgmap v11744: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:01 smithi022 ceph-mon[31360]: pgmap v11744: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:01 smithi022 ceph-mon[40250]: pgmap v11744: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:30:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:30:03.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:03.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:30:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:30:03.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:03.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:03.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:04 smithi022 ceph-mon[31360]: pgmap v11745: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3940634437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:04 smithi022 ceph-mon[40250]: pgmap v11745: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3940634437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:04 smithi122 ceph-mon[32255]: pgmap v11745: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3940634437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:06 smithi022 ceph-mon[31360]: pgmap v11746: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:06 smithi022 ceph-mon[40250]: pgmap v11746: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:06 smithi122 ceph-mon[32255]: pgmap v11746: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:30:08.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:08.514 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:08 smithi022 ceph-mon[40250]: pgmap v11747: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:08.514 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:08 smithi022 ceph-mon[31360]: pgmap v11747: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:08 smithi122 ceph-mon[32255]: pgmap v11747: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:08.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:08.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:08.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1361944173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1361944173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1361944173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:10] "GET /metrics HTTP/1.1" 200 202420 "" "Prometheus/2.18.1" 2021-12-16T00:30:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:10 smithi122 ceph-mon[32255]: pgmap v11748: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:10 smithi022 ceph-mon[31360]: pgmap v11748: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:10 smithi022 ceph-mon[40250]: pgmap v11748: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:11 smithi122 ceph-mon[32255]: pgmap v11749: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:11 smithi022 ceph-mon[31360]: pgmap v11749: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:11 smithi022 ceph-mon[40250]: pgmap v11749: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:30:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:30:13.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:30:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:30:13.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:14.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:14.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:14.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:14 smithi022 ceph-mon[31360]: pgmap v11750: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:14 smithi022 ceph-mon[40250]: pgmap v11750: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:14 smithi122 ceph-mon[32255]: pgmap v11750: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3465949696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3465949696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3465949696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:16 smithi022 ceph-mon[31360]: pgmap v11751: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:16 smithi022 ceph-mon[40250]: pgmap v11751: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:16 smithi122 ceph-mon[32255]: pgmap v11751: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:30:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:18 smithi022 ceph-mon[31360]: pgmap v11752: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:18 smithi022 ceph-mon[40250]: pgmap v11752: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:18 smithi122 ceph-mon[32255]: pgmap v11752: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:19.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:19.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:19.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:19.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:20] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:30:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:20 smithi122 ceph-mon[32255]: pgmap v11753: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3552908553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:20 smithi022 ceph-mon[31360]: pgmap v11753: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3552908553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:20 smithi022 ceph-mon[40250]: pgmap v11753: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3552908553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:21 smithi122 ceph-mon[32255]: pgmap v11754: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:21 smithi022 ceph-mon[31360]: pgmap v11754: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:21 smithi022 ceph-mon[40250]: pgmap v11754: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:30:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:30:23.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:30:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:30:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:24 smithi022 ceph-mon[31360]: pgmap v11755: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:24 smithi022 ceph-mon[40250]: pgmap v11755: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:24.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:24 smithi122 ceph-mon[32255]: pgmap v11755: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:24.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:24.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:24.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2777210147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2777210147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2777210147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:26 smithi022 ceph-mon[31360]: pgmap v11756: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:26 smithi022 ceph-mon[40250]: pgmap v11756: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:26 smithi122 ceph-mon[32255]: pgmap v11756: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:30:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:30:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:30:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:30:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:30:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:30:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:30:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:30:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:30:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:30:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:28 smithi022 ceph-mon[31360]: pgmap v11757: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:30:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:28 smithi022 ceph-mon[40250]: pgmap v11757: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:30:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:28 smithi122 ceph-mon[32255]: pgmap v11757: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:30:29.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:30.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:30.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:30.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:30 smithi122 ceph-mon[32255]: pgmap v11758: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:30] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:30:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:30 smithi022 ceph-mon[31360]: pgmap v11758: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:30 smithi022 ceph-mon[40250]: pgmap v11758: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1365614897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1365614897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1365614897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:32 smithi022 ceph-mon[31360]: pgmap v11759: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:32 smithi022 ceph-mon[40250]: pgmap v11759: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:32 smithi122 ceph-mon[32255]: pgmap v11759: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:30:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:30:33.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:30:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:30:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[31360]: pgmap v11760: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:30:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:30:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:30:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:30:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[40250]: pgmap v11760: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:30:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:30:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:30:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:30:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:34 smithi122 ceph-mon[32255]: pgmap v11760: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:30:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:30:35.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:35.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:36 smithi022 ceph-mon[31360]: pgmap v11761: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2409293885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:36 smithi022 ceph-mon[40250]: pgmap v11761: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2409293885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:36 smithi122 ceph-mon[32255]: pgmap v11761: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2409293885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:30:38.262 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:38 smithi122 ceph-mon[32255]: pgmap v11762: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:38 smithi022 ceph-mon[31360]: pgmap v11762: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:38 smithi022 ceph-mon[40250]: pgmap v11762: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:40 smithi122 ceph-mon[32255]: pgmap v11763: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:40] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:30:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:40 smithi022 ceph-mon[31360]: pgmap v11763: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:40 smithi022 ceph-mon[40250]: pgmap v11763: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:40.734 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:41.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:41.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:41.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3795931458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:41 smithi122 ceph-mon[32255]: pgmap v11764: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3795931458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:41 smithi022 ceph-mon[31360]: pgmap v11764: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3795931458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:41 smithi022 ceph-mon[40250]: pgmap v11764: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:30:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:30:43.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:30:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:30:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:44 smithi022 ceph-mon[31360]: pgmap v11765: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:44 smithi022 ceph-mon[40250]: pgmap v11765: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:44 smithi122 ceph-mon[32255]: pgmap v11765: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:46.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:46.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:46.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:46.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:46.475 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:46 smithi022 ceph-mon[31360]: pgmap v11766: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:46.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:46 smithi022 ceph-mon[40250]: pgmap v11766: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:46 smithi122 ceph-mon[32255]: pgmap v11766: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4293542705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:30:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4293542705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4293542705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:48 smithi022 ceph-mon[31360]: pgmap v11767: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:48 smithi022 ceph-mon[40250]: pgmap v11767: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:48 smithi122 ceph-mon[32255]: pgmap v11767: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:50 smithi122 ceph-mon[32255]: pgmap v11768: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:50] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:30:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:50 smithi022 ceph-mon[31360]: pgmap v11768: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:50 smithi022 ceph-mon[40250]: pgmap v11768: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:51.476 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:51 smithi022 ceph-mon[31360]: pgmap v11769: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:51 smithi022 ceph-mon[40250]: pgmap v11769: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:51.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:51.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:51.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:51 smithi122 ceph-mon[32255]: pgmap v11769: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2583141932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:30:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:30:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2583141932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2583141932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:53.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:30:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:30:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:53 smithi122 ceph-mon[32255]: pgmap v11770: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:53 smithi022 ceph-mon[31360]: pgmap v11770: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:30:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:53 smithi022 ceph-mon[40250]: pgmap v11770: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:30:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:56 smithi022 ceph-mon[31360]: pgmap v11771: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:56 smithi022 ceph-mon[40250]: pgmap v11771: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:56 smithi122 ceph-mon[32255]: pgmap v11771: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:56.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:30:57.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:30:57.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:30:57.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:30:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3804718923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:30:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3804718923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3804718923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:30:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:30:58 smithi022 ceph-mon[31360]: pgmap v11772: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:30:58 smithi022 ceph-mon[40250]: pgmap v11772: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:30:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:30:58 smithi122 ceph-mon[32255]: pgmap v11772: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:00] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:31:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:00 smithi122 ceph-mon[32255]: pgmap v11773: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:00 smithi022 ceph-mon[31360]: pgmap v11773: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:00 smithi022 ceph-mon[40250]: pgmap v11773: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:01 smithi122 ceph-mon[32255]: pgmap v11774: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:01 smithi022 ceph-mon[31360]: pgmap v11774: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:01 smithi022 ceph-mon[40250]: pgmap v11774: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:02.217 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:02.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:02.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:02.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:02.629 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:31:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:31:02.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3853108944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:02.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:31:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:31:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3853108944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3853108944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:03 smithi122 ceph-mon[32255]: pgmap v11775: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:03 smithi022 ceph-mon[31360]: pgmap v11775: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:03 smithi022 ceph-mon[40250]: pgmap v11775: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:06 smithi022 ceph-mon[31360]: pgmap v11776: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:06 smithi022 ceph-mon[40250]: pgmap v11776: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:06 smithi122 ceph-mon[32255]: pgmap v11776: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:07.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:31:07.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:07.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:07.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:08 smithi022 ceph-mon[31360]: pgmap v11777: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1918557832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:08 smithi022 ceph-mon[40250]: pgmap v11777: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1918557832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:08.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:08 smithi122 ceph-mon[32255]: pgmap v11777: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:08.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1918557832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:10 smithi122 ceph-mon[32255]: pgmap v11778: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:10] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:31:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:10 smithi022 ceph-mon[31360]: pgmap v11778: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:10 smithi022 ceph-mon[40250]: pgmap v11778: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:11 smithi122 ceph-mon[32255]: pgmap v11779: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:11 smithi022 ceph-mon[31360]: pgmap v11779: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:11 smithi022 ceph-mon[40250]: pgmap v11779: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:31:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:31:12.965 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:13.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:31:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:31:13.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:13.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:13.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:14 smithi022 ceph-mon[31360]: pgmap v11780: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1115730042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:14 smithi022 ceph-mon[40250]: pgmap v11780: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1115730042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:14 smithi122 ceph-mon[32255]: pgmap v11780: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1115730042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:16 smithi022 ceph-mon[31360]: pgmap v11781: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:16 smithi022 ceph-mon[40250]: pgmap v11781: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:16 smithi122 ceph-mon[32255]: pgmap v11781: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:31:18.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:18 smithi022 ceph-mon[31360]: pgmap v11782: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:18 smithi022 ceph-mon[40250]: pgmap v11782: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:18 smithi122 ceph-mon[32255]: pgmap v11782: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:18.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:18.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:18.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3324169454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3324169454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3324169454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:20 smithi122 ceph-mon[32255]: pgmap v11783: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:20] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:31:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:20 smithi022 ceph-mon[31360]: pgmap v11783: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:20 smithi022 ceph-mon[40250]: pgmap v11783: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:21 smithi122 ceph-mon[32255]: pgmap v11784: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:22.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:21 smithi022 ceph-mon[31360]: pgmap v11784: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:22.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:21 smithi022 ceph-mon[40250]: pgmap v11784: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:31:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:31:23.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:31:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:31:23.713 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:24.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:24.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:24.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:24 smithi022 ceph-mon[31360]: pgmap v11785: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1438518425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:24 smithi022 ceph-mon[40250]: pgmap v11785: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1438518425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:24 smithi122 ceph-mon[32255]: pgmap v11785: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1438518425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:26 smithi022 ceph-mon[31360]: pgmap v11786: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:26 smithi022 ceph-mon[40250]: pgmap v11786: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:26 smithi122 ceph-mon[32255]: pgmap v11786: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:27.645 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:31:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[31360]: pgmap v11787: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:31:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:31:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:31:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[40250]: pgmap v11787: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:31:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:31:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:31:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:28 smithi122 ceph-mon[32255]: pgmap v11787: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:31:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:31:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:31:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:29.424 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:31:30.425 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:30] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:31:30.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:30 smithi122 ceph-mon[32255]: pgmap v11788: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3471705431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:30 smithi022 ceph-mon[31360]: pgmap v11788: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3471705431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:30 smithi022 ceph-mon[40250]: pgmap v11788: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3471705431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:31 smithi122 ceph-mon[32255]: pgmap v11789: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:31 smithi022 ceph-mon[31360]: pgmap v11789: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:31 smithi022 ceph-mon[40250]: pgmap v11789: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:31:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:31:33.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:31:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:31:34.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[40250]: pgmap v11790: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:31:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:31:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:31:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:31:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[31360]: pgmap v11790: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:31:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:31:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:31:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:31:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:34 smithi122 ceph-mon[32255]: pgmap v11790: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:31:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:31:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:31:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:31:34.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:34.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:34.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:35.264 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3042030938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3042030938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3042030938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:36 smithi022 ceph-mon[31360]: pgmap v11791: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:36 smithi022 ceph-mon[40250]: pgmap v11791: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:36 smithi122 ceph-mon[32255]: pgmap v11791: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:31:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:38 smithi022 ceph-mon[31360]: pgmap v11792: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:38 smithi022 ceph-mon[40250]: pgmap v11792: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:38 smithi122 ceph-mon[32255]: pgmap v11792: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:39.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:40.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:40.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:40.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:40.443 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:40] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:40 smithi122 ceph-mon[32255]: pgmap v11793: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:40.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/842320011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:40.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:40 smithi022 ceph-mon[31360]: pgmap v11793: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:40.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/842320011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:40.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:40 smithi022 ceph-mon[40250]: pgmap v11793: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:40.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/842320011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:41 smithi122 ceph-mon[32255]: pgmap v11794: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:41 smithi022 ceph-mon[31360]: pgmap v11794: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:41 smithi022 ceph-mon[40250]: pgmap v11794: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:31:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:31:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:31:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:31:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:44 smithi022 ceph-mon[31360]: pgmap v11795: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:44 smithi022 ceph-mon[40250]: pgmap v11795: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:44 smithi122 ceph-mon[32255]: pgmap v11795: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:45.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:46 smithi022 ceph-mon[31360]: pgmap v11796: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/598625279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:46 smithi022 ceph-mon[40250]: pgmap v11796: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/598625279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:46.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:46 smithi122 ceph-mon[32255]: pgmap v11796: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:46.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/598625279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:31:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:48 smithi022 ceph-mon[31360]: pgmap v11797: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:48 smithi022 ceph-mon[40250]: pgmap v11797: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:48 smithi122 ceph-mon[32255]: pgmap v11797: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:50] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:31:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:50 smithi122 ceph-mon[32255]: pgmap v11798: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:50 smithi022 ceph-mon[31360]: pgmap v11798: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:50 smithi022 ceph-mon[40250]: pgmap v11798: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:50.582 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:50.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:50.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:50.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3601070428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:51 smithi122 ceph-mon[32255]: pgmap v11799: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3601070428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:51 smithi022 ceph-mon[31360]: pgmap v11799: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3601070428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:51 smithi022 ceph-mon[40250]: pgmap v11799: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:31:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:31:53.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:31:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:31:54.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:54.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:54 smithi022 ceph-mon[31360]: pgmap v11800: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:54.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:54.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:54 smithi022 ceph-mon[40250]: pgmap v11800: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:54.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:31:54.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:54 smithi122 ceph-mon[32255]: pgmap v11800: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:31:55.951 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:31:56.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:31:56.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:31:56.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:31:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:56 smithi022 ceph-mon[31360]: pgmap v11801: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:56 smithi022 ceph-mon[40250]: pgmap v11801: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:56 smithi122 ceph-mon[32255]: pgmap v11801: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/120909040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:31:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/120909040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/120909040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:31:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:31:58 smithi022 ceph-mon[31360]: pgmap v11802: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:31:58 smithi022 ceph-mon[40250]: pgmap v11802: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:31:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:31:58 smithi122 ceph-mon[32255]: pgmap v11802: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:00] "GET /metrics HTTP/1.1" 200 202418 "" "Prometheus/2.18.1" 2021-12-16T00:32:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:00 smithi122 ceph-mon[32255]: pgmap v11803: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:00 smithi022 ceph-mon[31360]: pgmap v11803: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:00 smithi022 ceph-mon[40250]: pgmap v11803: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:01.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:01.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:01.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:01.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:01.691 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:01 smithi022 ceph-mon[31360]: pgmap v11804: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:01.692 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:01 smithi022 ceph-mon[40250]: pgmap v11804: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:01 smithi122 ceph-mon[32255]: pgmap v11804: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:02.570 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:32:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:32:02.929 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/403520494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/403520494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/403520494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:03.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:32:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:32:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:03 smithi122 ceph-mon[32255]: pgmap v11805: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:03 smithi022 ceph-mon[31360]: pgmap v11805: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:03 smithi022 ceph-mon[40250]: pgmap v11805: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:06 smithi022 ceph-mon[31360]: pgmap v11806: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:06 smithi022 ceph-mon[40250]: pgmap v11806: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:06 smithi122 ceph-mon[32255]: pgmap v11806: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:06.692 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:07.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:07.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:07.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2694480443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:07.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2694480443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2694480443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:32:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:08 smithi022 ceph-mon[31360]: pgmap v11807: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:08 smithi022 ceph-mon[40250]: pgmap v11807: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:08 smithi122 ceph-mon[32255]: pgmap v11807: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:10] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:32:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:10 smithi122 ceph-mon[32255]: pgmap v11808: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:10 smithi022 ceph-mon[31360]: pgmap v11808: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:10 smithi022 ceph-mon[40250]: pgmap v11808: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:11 smithi122 ceph-mon[32255]: pgmap v11809: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:11 smithi022 ceph-mon[31360]: pgmap v11809: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:11 smithi022 ceph-mon[40250]: pgmap v11809: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:12.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:12.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:12.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:12.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:12.573 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:32:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:32:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/840335153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/840335153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/840335153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:32:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:32:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:13 smithi122 ceph-mon[32255]: pgmap v11810: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:13 smithi022 ceph-mon[31360]: pgmap v11810: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:13 smithi022 ceph-mon[40250]: pgmap v11810: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:16.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:16 smithi022 ceph-mon[31360]: pgmap v11811: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:16.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:16 smithi022 ceph-mon[40250]: pgmap v11811: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:16 smithi122 ceph-mon[32255]: pgmap v11811: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:17.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:32:17.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:17.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:17.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:18 smithi022 ceph-mon[31360]: pgmap v11812: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2806886034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:18 smithi022 ceph-mon[40250]: pgmap v11812: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2806886034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:18 smithi122 ceph-mon[32255]: pgmap v11812: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2806886034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:20] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:32:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:20 smithi122 ceph-mon[32255]: pgmap v11813: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:20 smithi022 ceph-mon[31360]: pgmap v11813: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:20 smithi022 ceph-mon[40250]: pgmap v11813: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:21 smithi122 ceph-mon[32255]: pgmap v11814: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:21 smithi022 ceph-mon[31360]: pgmap v11814: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:21 smithi022 ceph-mon[40250]: pgmap v11814: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:32:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:32:22.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:23.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:23.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:23.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:23.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:32:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:32:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3321232692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3321232692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3321232692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:24 smithi022 ceph-mon[31360]: pgmap v11815: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:24 smithi022 ceph-mon[40250]: pgmap v11815: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:24 smithi122 ceph-mon[32255]: pgmap v11815: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:26 smithi022 ceph-mon[31360]: pgmap v11816: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:26 smithi022 ceph-mon[40250]: pgmap v11816: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:26 smithi122 ceph-mon[32255]: pgmap v11816: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:32:28.168 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:28.453 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:28 smithi022 ceph-mon[31360]: pgmap v11817: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:28.453 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:28 smithi022 ceph-mon[40250]: pgmap v11817: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:28.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:28.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:28.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:28 smithi122 ceph-mon[32255]: pgmap v11817: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:32:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:32:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:32:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3581970137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:32:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:32:29.539 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:32:29.539 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:32:29.539 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3581970137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:29.539 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:32:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:32:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:32:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:32:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3581970137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:32:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:30 smithi122 ceph-mon[32255]: pgmap v11818: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:30] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:32:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:30 smithi022 ceph-mon[31360]: pgmap v11818: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:30 smithi022 ceph-mon[40250]: pgmap v11818: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:31 smithi122 ceph-mon[32255]: pgmap v11819: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:31 smithi022 ceph-mon[31360]: pgmap v11819: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:31 smithi022 ceph-mon[40250]: pgmap v11819: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:32:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:32:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:32:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:32:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:32:33.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:33.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:33.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:33.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[40250]: pgmap v11820: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:32:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:32:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:32:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:32:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/993003778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[31360]: pgmap v11820: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:32:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/993003778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:34 smithi122 ceph-mon[32255]: pgmap v11820: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:32:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:32:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:32:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:32:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/993003778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:36 smithi122 ceph-mon[32255]: pgmap v11821: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:36 smithi022 ceph-mon[31360]: pgmap v11821: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:36 smithi022 ceph-mon[40250]: pgmap v11821: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:32:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:38 smithi122 ceph-mon[32255]: pgmap v11822: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:38 smithi022 ceph-mon[31360]: pgmap v11822: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:38 smithi022 ceph-mon[40250]: pgmap v11822: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:38.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:39.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:39.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:39.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3261856396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3261856396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3261856396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:40 smithi122 ceph-mon[32255]: pgmap v11823: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:40] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:32:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:40 smithi022 ceph-mon[31360]: pgmap v11823: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:40 smithi022 ceph-mon[40250]: pgmap v11823: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:41 smithi122 ceph-mon[32255]: pgmap v11824: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:41 smithi022 ceph-mon[31360]: pgmap v11824: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:41 smithi022 ceph-mon[40250]: pgmap v11824: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:32:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:32:43.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:32:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:32:44.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:44.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:44.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:44.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:44.664 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:44.665 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:44 smithi022 ceph-mon[31360]: pgmap v11825: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:44.665 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:44.665 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:44 smithi022 ceph-mon[40250]: pgmap v11825: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:44 smithi122 ceph-mon[32255]: pgmap v11825: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2297194798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2297194798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2297194798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:46 smithi122 ceph-mon[32255]: pgmap v11826: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:46 smithi022 ceph-mon[31360]: pgmap v11826: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:46 smithi022 ceph-mon[40250]: pgmap v11826: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:32:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:48 smithi122 ceph-mon[32255]: pgmap v11827: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:48 smithi022 ceph-mon[31360]: pgmap v11827: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:48 smithi022 ceph-mon[40250]: pgmap v11827: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:49.665 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:50.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:50.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:50.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:50] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:32:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:50 smithi122 ceph-mon[32255]: pgmap v11828: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3401115329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:50 smithi022 ceph-mon[31360]: pgmap v11828: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3401115329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:50 smithi022 ceph-mon[40250]: pgmap v11828: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3401115329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:51 smithi122 ceph-mon[32255]: pgmap v11829: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:51 smithi022 ceph-mon[31360]: pgmap v11829: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:51 smithi022 ceph-mon[40250]: pgmap v11829: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:32:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:32:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:32:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:32:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:54 smithi122 ceph-mon[32255]: pgmap v11830: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:54 smithi022 ceph-mon[31360]: pgmap v11830: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:32:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:54 smithi022 ceph-mon[40250]: pgmap v11830: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:32:55.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:32:55.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:32:55.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:32:55.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:32:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:56 smithi122 ceph-mon[32255]: pgmap v11831: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3171662376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:56 smithi022 ceph-mon[31360]: pgmap v11831: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3171662376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:56 smithi022 ceph-mon[40250]: pgmap v11831: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3171662376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:32:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:32:58.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:32:58 smithi122 ceph-mon[32255]: pgmap v11832: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:32:58 smithi022 ceph-mon[31360]: pgmap v11832: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:32:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:32:58 smithi022 ceph-mon[40250]: pgmap v11832: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:00.406 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:00] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:33:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:00 smithi122 ceph-mon[32255]: pgmap v11833: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:00.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:00.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:00.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:00 smithi022 ceph-mon[31360]: pgmap v11833: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:00.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:00 smithi022 ceph-mon[40250]: pgmap v11833: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2527334664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:01 smithi122 ceph-mon[32255]: pgmap v11834: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2527334664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:01 smithi022 ceph-mon[31360]: pgmap v11834: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2527334664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:01 smithi022 ceph-mon[40250]: pgmap v11834: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:33:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:33:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:33:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:33:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:04 smithi122 ceph-mon[32255]: pgmap v11835: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:04 smithi022 ceph-mon[31360]: pgmap v11835: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:04 smithi022 ceph-mon[40250]: pgmap v11835: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:06.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:06.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:06.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:06 smithi022 ceph-mon[31360]: pgmap v11836: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2890866873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:06 smithi022 ceph-mon[40250]: pgmap v11836: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2890866873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:06 smithi122 ceph-mon[32255]: pgmap v11836: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2890866873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:33:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:08 smithi122 ceph-mon[32255]: pgmap v11837: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:08 smithi022 ceph-mon[31360]: pgmap v11837: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:08 smithi022 ceph-mon[40250]: pgmap v11837: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:10] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:33:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:10 smithi122 ceph-mon[32255]: pgmap v11838: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:10 smithi022 ceph-mon[31360]: pgmap v11838: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:10 smithi022 ceph-mon[40250]: pgmap v11838: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:11.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:11.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:11.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:11.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:11 smithi022 ceph-mon[31360]: pgmap v11839: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1261119376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:11 smithi022 ceph-mon[40250]: pgmap v11839: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1261119376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:11 smithi122 ceph-mon[32255]: pgmap v11839: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1261119376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:33:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:33:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:33:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:33:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:14 smithi122 ceph-mon[32255]: pgmap v11840: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:14 smithi022 ceph-mon[40250]: pgmap v11840: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:14 smithi022 ceph-mon[31360]: pgmap v11840: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:16 smithi122 ceph-mon[32255]: pgmap v11841: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:16 smithi022 ceph-mon[31360]: pgmap v11841: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:16 smithi022 ceph-mon[40250]: pgmap v11841: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:16.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:16.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:16.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3391210497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3391210497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:33:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3391210497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:18 smithi122 ceph-mon[32255]: pgmap v11842: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:18 smithi022 ceph-mon[31360]: pgmap v11842: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:18 smithi022 ceph-mon[40250]: pgmap v11842: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:20] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:33:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:20 smithi122 ceph-mon[32255]: pgmap v11843: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:20 smithi022 ceph-mon[31360]: pgmap v11843: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:20 smithi022 ceph-mon[40250]: pgmap v11843: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:21.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:21.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:21 smithi022 ceph-mon[31360]: pgmap v11844: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:21.889 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:21 smithi022 ceph-mon[40250]: pgmap v11844: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:21 smithi122 ceph-mon[32255]: pgmap v11844: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:22.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:22.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:22.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:22.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:33:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:33:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4021466422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4021466422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4021466422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:33:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:33:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:23 smithi122 ceph-mon[32255]: pgmap v11845: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:23 smithi022 ceph-mon[31360]: pgmap v11845: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:23 smithi022 ceph-mon[40250]: pgmap v11845: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:26 smithi122 ceph-mon[32255]: pgmap v11846: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:26 smithi022 ceph-mon[31360]: pgmap v11846: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:26 smithi022 ceph-mon[40250]: pgmap v11846: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:27.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:33:27.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:27.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:27.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:28 smithi122 ceph-mon[32255]: pgmap v11847: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3877605993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:28 smithi022 ceph-mon[31360]: pgmap v11847: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3877605993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:28 smithi022 ceph-mon[40250]: pgmap v11847: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3877605993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:33:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:33:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:33:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:33:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:33:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:33:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:33:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:33:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:33:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:30] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:33:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:30 smithi122 ceph-mon[32255]: pgmap v11848: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:33:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:30 smithi022 ceph-mon[31360]: pgmap v11848: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:33:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:30 smithi022 ceph-mon[40250]: pgmap v11848: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:33:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:31 smithi122 ceph-mon[32255]: pgmap v11849: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:31 smithi022 ceph-mon[31360]: pgmap v11849: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:31 smithi022 ceph-mon[40250]: pgmap v11849: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:32.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:33:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:33:32.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:32.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:32.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2515183019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2515183019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2515183019' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:33.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:33:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:33:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:34 smithi122 ceph-mon[32255]: pgmap v11850: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:33:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:33:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[31360]: pgmap v11850: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:33:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:33:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[40250]: pgmap v11850: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:33:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:36 smithi122 ceph-mon[32255]: pgmap v11851: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:36 smithi022 ceph-mon[31360]: pgmap v11851: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:36 smithi022 ceph-mon[40250]: pgmap v11851: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:33:37.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:38.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:38.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:38.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:38 smithi122 ceph-mon[32255]: pgmap v11852: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:38 smithi022 ceph-mon[31360]: pgmap v11852: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:38 smithi022 ceph-mon[40250]: pgmap v11852: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/673541830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/673541830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/673541830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:40 smithi122 ceph-mon[32255]: pgmap v11853: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:40] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:33:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:40 smithi022 ceph-mon[31360]: pgmap v11853: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:40 smithi022 ceph-mon[40250]: pgmap v11853: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:41.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:41 smithi122 ceph-mon[32255]: pgmap v11854: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:41 smithi022 ceph-mon[31360]: pgmap v11854: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:41 smithi022 ceph-mon[40250]: pgmap v11854: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:33:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:33:43.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:33:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:33:43.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:43.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:43.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:44 smithi122 ceph-mon[32255]: pgmap v11855: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:44.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1439025552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:44 smithi022 ceph-mon[31360]: pgmap v11855: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1439025552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:44 smithi022 ceph-mon[40250]: pgmap v11855: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1439025552' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:46 smithi122 ceph-mon[32255]: pgmap v11856: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:46 smithi022 ceph-mon[31360]: pgmap v11856: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:46 smithi022 ceph-mon[40250]: pgmap v11856: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:47.786 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:33:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:48 smithi122 ceph-mon[32255]: pgmap v11857: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:48.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:48.733 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:48 smithi022 ceph-mon[31360]: pgmap v11857: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:48.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:48 smithi022 ceph-mon[40250]: pgmap v11857: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:49.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:49.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:49.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/496967964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/496967964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/496967964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:50] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:33:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:50 smithi122 ceph-mon[32255]: pgmap v11858: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:50 smithi022 ceph-mon[31360]: pgmap v11858: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:50 smithi022 ceph-mon[40250]: pgmap v11858: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:51 smithi122 ceph-mon[32255]: pgmap v11859: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:51 smithi022 ceph-mon[31360]: pgmap v11859: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:51 smithi022 ceph-mon[40250]: pgmap v11859: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:33:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:33:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:33:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:33:54.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:54.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:54.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:54.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:33:54.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:54.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:54 smithi022 ceph-mon[31360]: pgmap v11860: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:54.468 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:54.469 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:54 smithi022 ceph-mon[40250]: pgmap v11860: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:33:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:54 smithi122 ceph-mon[32255]: pgmap v11860: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:33:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3440739630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3440739630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3440739630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:33:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:56 smithi122 ceph-mon[32255]: pgmap v11861: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:56 smithi022 ceph-mon[31360]: pgmap v11861: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:56 smithi022 ceph-mon[40250]: pgmap v11861: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:33:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:33:58 smithi122 ceph-mon[32255]: pgmap v11862: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:33:58 smithi022 ceph-mon[31360]: pgmap v11862: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:33:58 smithi022 ceph-mon[40250]: pgmap v11862: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:33:59.469 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:33:59.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:33:59.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:33:59.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:00] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T00:34:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:00 smithi122 ceph-mon[32255]: pgmap v11863: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3455716377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:00 smithi022 ceph-mon[31360]: pgmap v11863: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3455716377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:00 smithi022 ceph-mon[40250]: pgmap v11863: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3455716377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:01 smithi122 ceph-mon[32255]: pgmap v11864: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:01 smithi022 ceph-mon[31360]: pgmap v11864: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:01 smithi022 ceph-mon[40250]: pgmap v11864: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:34:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:34:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:34:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:34:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:04 smithi122 ceph-mon[32255]: pgmap v11865: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:04 smithi022 ceph-mon[31360]: pgmap v11865: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:04 smithi022 ceph-mon[40250]: pgmap v11865: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:04.839 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:05.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:05.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:05.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1899180336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1899180336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1899180336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:06 smithi122 ceph-mon[32255]: pgmap v11866: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:06 smithi022 ceph-mon[31360]: pgmap v11866: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:06 smithi022 ceph-mon[40250]: pgmap v11866: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:34:08.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:08 smithi122 ceph-mon[32255]: pgmap v11867: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:08 smithi022 ceph-mon[31360]: pgmap v11867: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:08.919 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:08 smithi022 ceph-mon[40250]: pgmap v11867: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:10.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:10 smithi122 ceph-mon[32255]: pgmap v11868: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:10] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:34:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:10 smithi022 ceph-mon[31360]: pgmap v11868: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:10 smithi022 ceph-mon[40250]: pgmap v11868: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:10.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:10.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1247583702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1247583702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1247583702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:12 smithi122 ceph-mon[32255]: pgmap v11869: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:34:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:34:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:12 smithi022 ceph-mon[31360]: pgmap v11869: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:12 smithi022 ceph-mon[40250]: pgmap v11869: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:34:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:34:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:14 smithi122 ceph-mon[32255]: pgmap v11870: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:14 smithi022 ceph-mon[31360]: pgmap v11870: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:14 smithi022 ceph-mon[40250]: pgmap v11870: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:15.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:15.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:15.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:15.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:16 smithi122 ceph-mon[32255]: pgmap v11871: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/995121918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:16 smithi022 ceph-mon[31360]: pgmap v11871: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/995121918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:16 smithi022 ceph-mon[40250]: pgmap v11871: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/995121918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:34:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:18 smithi122 ceph-mon[32255]: pgmap v11872: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:18 smithi022 ceph-mon[31360]: pgmap v11872: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:18 smithi022 ceph-mon[40250]: pgmap v11872: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:20 smithi122 ceph-mon[32255]: pgmap v11873: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:20] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:34:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:20 smithi022 ceph-mon[31360]: pgmap v11873: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:20 smithi022 ceph-mon[40250]: pgmap v11873: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:20.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:21.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:21.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:21.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:21 smithi122 ceph-mon[32255]: pgmap v11874: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/334820096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:22.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:21 smithi022 ceph-mon[31360]: pgmap v11874: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:22.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/334820096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:21 smithi022 ceph-mon[40250]: pgmap v11874: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/334820096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:34:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:34:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:34:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:34:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:24 smithi122 ceph-mon[32255]: pgmap v11875: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:24 smithi022 ceph-mon[31360]: pgmap v11875: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:24 smithi022 ceph-mon[40250]: pgmap v11875: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:26.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:26 smithi122 ceph-mon[32255]: pgmap v11876: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:26.689 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:26 smithi022 ceph-mon[40250]: pgmap v11876: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:26.689 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:26 smithi022 ceph-mon[31360]: pgmap v11876: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1714669850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1714669850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:34:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1714669850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:28 smithi122 ceph-mon[32255]: pgmap v11877: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:28 smithi022 ceph-mon[31360]: pgmap v11877: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:28 smithi022 ceph-mon[40250]: pgmap v11877: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:30] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:34:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:30 smithi122 ceph-mon[32255]: pgmap v11878: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:34:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:34:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:34:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:34:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[31360]: pgmap v11878: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:34:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:34:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:34:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:34:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[40250]: pgmap v11878: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:34:30.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:34:30.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:34:30.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:34:31.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:31 smithi122 ceph-mon[32255]: pgmap v11879: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:31 smithi022 ceph-mon[31360]: pgmap v11879: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:31 smithi022 ceph-mon[40250]: pgmap v11879: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:32.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:32.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:32.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:32.604 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:34:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:34:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4052122817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4052122817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4052122817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:34:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:34:33.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:33.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:33 smithi122 ceph-mon[32255]: pgmap v11880: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:33 smithi022 ceph-mon[31360]: pgmap v11880: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:33 smithi022 ceph-mon[40250]: pgmap v11880: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:34:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:34:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:34:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:34:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:34:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:34:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:34:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:34:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:34:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:34:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:34:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:34:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:35 smithi122 ceph-mon[32255]: pgmap v11881: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:35 smithi022 ceph-mon[31360]: pgmap v11881: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:35 smithi022 ceph-mon[40250]: pgmap v11881: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:37.103 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:37.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:37.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:37.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:34:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:38 smithi122 ceph-mon[32255]: pgmap v11882: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3009717178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:38 smithi022 ceph-mon[31360]: pgmap v11882: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3009717178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:38 smithi022 ceph-mon[40250]: pgmap v11882: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3009717178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:40 smithi122 ceph-mon[32255]: pgmap v11883: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:40] "GET /metrics HTTP/1.1" 200 202416 "" "Prometheus/2.18.1" 2021-12-16T00:34:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:40 smithi022 ceph-mon[31360]: pgmap v11883: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:40 smithi022 ceph-mon[40250]: pgmap v11883: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:41 smithi122 ceph-mon[32255]: pgmap v11884: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:41 smithi022 ceph-mon[31360]: pgmap v11884: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:41 smithi022 ceph-mon[40250]: pgmap v11884: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:42.435 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:34:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:34:42.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:42.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:42.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:34:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:34:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3890466632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3890466632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3890466632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:43 smithi122 ceph-mon[32255]: pgmap v11885: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:43 smithi022 ceph-mon[31360]: pgmap v11885: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:43 smithi022 ceph-mon[40250]: pgmap v11885: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:46 smithi122 ceph-mon[32255]: pgmap v11886: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:46 smithi022 ceph-mon[31360]: pgmap v11886: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:46 smithi022 ceph-mon[40250]: pgmap v11886: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:34:47.808 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:48.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:48.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:48.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:48 smithi022 ceph-mon[31360]: pgmap v11887: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1375189490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:48 smithi022 ceph-mon[40250]: pgmap v11887: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1375189490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:48 smithi122 ceph-mon[32255]: pgmap v11887: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1375189490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:50 smithi122 ceph-mon[32255]: pgmap v11888: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:50] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:34:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:50 smithi022 ceph-mon[31360]: pgmap v11888: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:50 smithi022 ceph-mon[40250]: pgmap v11888: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:51 smithi122 ceph-mon[32255]: pgmap v11889: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:51 smithi022 ceph-mon[31360]: pgmap v11889: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:51 smithi022 ceph-mon[40250]: pgmap v11889: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:34:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:34:53.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:34:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:34:53.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:53.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:53.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:54 smithi122 ceph-mon[32255]: pgmap v11890: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1505988677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:54 smithi022 ceph-mon[31360]: pgmap v11890: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1505988677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:34:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:54 smithi022 ceph-mon[40250]: pgmap v11890: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:34:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1505988677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:56 smithi122 ceph-mon[32255]: pgmap v11891: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:56 smithi022 ceph-mon[31360]: pgmap v11891: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:56 smithi022 ceph-mon[40250]: pgmap v11891: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:34:58.553 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:34:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:58 smithi122 ceph-mon[32255]: pgmap v11892: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:58 smithi022 ceph-mon[31360]: pgmap v11892: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:58 smithi022 ceph-mon[40250]: pgmap v11892: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:34:58.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:34:58.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:34:58.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:34:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:34:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2511054785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:34:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2511054785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:34:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:34:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2511054785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:00 smithi122 ceph-mon[32255]: pgmap v11893: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:00] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:35:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:00 smithi022 ceph-mon[31360]: pgmap v11893: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:00 smithi022 ceph-mon[40250]: pgmap v11893: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:01 smithi122 ceph-mon[32255]: pgmap v11894: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:01 smithi022 ceph-mon[31360]: pgmap v11894: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:01 smithi022 ceph-mon[40250]: pgmap v11894: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:35:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:35:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:35:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:35:03.924 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:04.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:04.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:04.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:04 smithi122 ceph-mon[32255]: pgmap v11895: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/81995808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:04 smithi022 ceph-mon[31360]: pgmap v11895: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/81995808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:04 smithi022 ceph-mon[40250]: pgmap v11895: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/81995808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:06 smithi122 ceph-mon[32255]: pgmap v11896: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:06 smithi022 ceph-mon[31360]: pgmap v11896: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:06 smithi022 ceph-mon[40250]: pgmap v11896: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:35:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:08 smithi122 ceph-mon[32255]: pgmap v11897: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:08 smithi022 ceph-mon[31360]: pgmap v11897: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:08 smithi022 ceph-mon[40250]: pgmap v11897: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:09.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:09.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:09.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:09.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:10] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:35:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:10 smithi122 ceph-mon[32255]: pgmap v11898: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3771690158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:10 smithi022 ceph-mon[31360]: pgmap v11898: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3771690158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:10 smithi022 ceph-mon[40250]: pgmap v11898: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3771690158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:11 smithi122 ceph-mon[32255]: pgmap v11899: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:11 smithi022 ceph-mon[31360]: pgmap v11899: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:11 smithi022 ceph-mon[40250]: pgmap v11899: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:35:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:35:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:35:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:35:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:14.662 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:14.662 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:14 smithi022 ceph-mon[31360]: pgmap v11900: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:14.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:14.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:14 smithi022 ceph-mon[40250]: pgmap v11900: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:14 smithi122 ceph-mon[32255]: pgmap v11900: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:15.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:15.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:15.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:15.330 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3722095485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:15.331 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3722095485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3722095485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:16 smithi122 ceph-mon[32255]: pgmap v11901: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:16 smithi022 ceph-mon[31360]: pgmap v11901: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:16 smithi022 ceph-mon[40250]: pgmap v11901: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:35:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:18 smithi122 ceph-mon[32255]: pgmap v11902: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:18 smithi022 ceph-mon[31360]: pgmap v11902: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:18 smithi022 ceph-mon[40250]: pgmap v11902: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:20.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:20.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:20.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:20.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:20 smithi122 ceph-mon[32255]: pgmap v11903: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:20] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:35:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:20 smithi022 ceph-mon[31360]: pgmap v11903: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:20 smithi022 ceph-mon[40250]: pgmap v11903: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/314292190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/314292190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/314292190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:22 smithi122 ceph-mon[32255]: pgmap v11904: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:35:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:35:22.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:22 smithi022 ceph-mon[31360]: pgmap v11904: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:22.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:22 smithi022 ceph-mon[40250]: pgmap v11904: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:23.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:35:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:35:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:24 smithi122 ceph-mon[32255]: pgmap v11905: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:24.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:24 smithi022 ceph-mon[31360]: pgmap v11905: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:24.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:24.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:24 smithi022 ceph-mon[40250]: pgmap v11905: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:25.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:25.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:25.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:26 smithi122 ceph-mon[32255]: pgmap v11906: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1410025680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:26 smithi022 ceph-mon[31360]: pgmap v11906: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1410025680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:26 smithi022 ceph-mon[40250]: pgmap v11906: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1410025680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:35:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:28 smithi122 ceph-mon[32255]: pgmap v11907: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:28 smithi022 ceph-mon[31360]: pgmap v11907: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:28 smithi022 ceph-mon[40250]: pgmap v11907: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:30] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:35:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:30 smithi122 ceph-mon[32255]: pgmap v11908: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:35:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:35:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:35:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:30.776 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[31360]: pgmap v11908: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:30.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:35:30.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:35:30.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:35:30.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[40250]: pgmap v11908: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:35:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:35:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:35:31.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:31.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:31.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/440528876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:31 smithi122 ceph-mon[32255]: pgmap v11909: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/440528876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:31 smithi022 ceph-mon[31360]: pgmap v11909: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/440528876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:31 smithi022 ceph-mon[40250]: pgmap v11909: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:35:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:35:33.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:35:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:35:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: pgmap v11910: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:35:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:35:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:35:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:35:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: pgmap v11910: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:35:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:35:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.660 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:34.660 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: pgmap v11910: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:34.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:35:34.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:35:34.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:35:34.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:35:34.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:34.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:36.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:36 smithi022 ceph-mon[31360]: pgmap v11911: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:36 smithi022 ceph-mon[40250]: pgmap v11911: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:36.282 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:36.282 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:36 smithi122 ceph-mon[32255]: pgmap v11911: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:36.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:36.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:36.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1277274398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1277274398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:37.287 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1277274398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:37.287 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:37.288 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.288 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.288 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.289 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.289 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.289 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.289 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.289 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:35:37.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:35:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3070M 2021-12-16T00:35:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[31360]: pgmap v11912: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3070M 2021-12-16T00:35:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[40250]: pgmap v11912: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:38 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3070M 2021-12-16T00:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:38 smithi122 ceph-mon[32255]: pgmap v11912: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:35:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:40] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:35:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:40 smithi122 ceph-mon[32255]: pgmap v11913: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:40 smithi022 ceph-mon[31360]: pgmap v11913: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:40 smithi022 ceph-mon[40250]: pgmap v11913: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:41.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:41 smithi022 ceph-mon[31360]: pgmap v11914: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:41 smithi022 ceph-mon[40250]: pgmap v11914: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:41.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:41.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:41.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:41 smithi122 ceph-mon[32255]: pgmap v11914: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:42.617 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:35:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:35:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4028047508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4028047508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4028047508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:35:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:35:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:43 smithi122 ceph-mon[32255]: pgmap v11915: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:43 smithi022 ceph-mon[31360]: pgmap v11915: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:43 smithi022 ceph-mon[40250]: pgmap v11915: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:46 smithi122 ceph-mon[32255]: pgmap v11916: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:46 smithi022 ceph-mon[31360]: pgmap v11916: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:46 smithi022 ceph-mon[40250]: pgmap v11916: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:46.893 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:47.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:47.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:47.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3798070988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:35:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3798070988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3798070988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:48 smithi122 ceph-mon[32255]: pgmap v11917: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:48 smithi022 ceph-mon[31360]: pgmap v11917: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:48 smithi022 ceph-mon[40250]: pgmap v11917: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:50 smithi122 ceph-mon[32255]: pgmap v11918: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:50] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:35:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:50 smithi022 ceph-mon[31360]: pgmap v11918: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:50 smithi022 ceph-mon[40250]: pgmap v11918: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:51 smithi122 ceph-mon[32255]: pgmap v11919: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:51 smithi022 ceph-mon[31360]: pgmap v11919: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:51 smithi022 ceph-mon[40250]: pgmap v11919: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:52.268 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:52.630 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:35:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:35:52.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:52.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:52.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:52.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3336254796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3336254796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3336254796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:35:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:35:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:53 smithi122 ceph-mon[32255]: pgmap v11920: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:53 smithi022 ceph-mon[31360]: pgmap v11920: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:35:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:53 smithi022 ceph-mon[40250]: pgmap v11920: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:35:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:56 smithi122 ceph-mon[32255]: pgmap v11921: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:56 smithi022 ceph-mon[31360]: pgmap v11921: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:56 smithi022 ceph-mon[40250]: pgmap v11921: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:57.643 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:35:57.644 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:35:58.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:35:58.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:35:58.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:35:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:58 smithi122 ceph-mon[32255]: pgmap v11922: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:35:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4224480945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:58 smithi022 ceph-mon[31360]: pgmap v11922: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:35:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4224480945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:35:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:58 smithi022 ceph-mon[40250]: pgmap v11922: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:35:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:35:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4224480945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:00] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:36:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:00 smithi122 ceph-mon[32255]: pgmap v11923: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:00 smithi022 ceph-mon[31360]: pgmap v11923: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:00 smithi022 ceph-mon[40250]: pgmap v11923: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:01 smithi122 ceph-mon[32255]: pgmap v11924: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:01 smithi022 ceph-mon[31360]: pgmap v11924: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:01 smithi022 ceph-mon[40250]: pgmap v11924: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:36:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:36:03.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:03.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:03.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:03.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:36:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:36:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:04 smithi122 ceph-mon[32255]: pgmap v11925: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2202087253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:04 smithi022 ceph-mon[31360]: pgmap v11925: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2202087253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:04 smithi022 ceph-mon[40250]: pgmap v11925: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2202087253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:06 smithi122 ceph-mon[32255]: pgmap v11926: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:06 smithi022 ceph-mon[31360]: pgmap v11926: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:06 smithi022 ceph-mon[40250]: pgmap v11926: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:36:08.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:08 smithi122 ceph-mon[32255]: pgmap v11927: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:08.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:08.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:08.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:08.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:08 smithi022 ceph-mon[40250]: pgmap v11927: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:08.758 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:08 smithi022 ceph-mon[31360]: pgmap v11927: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1010074591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1010074591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1010074591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:10] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T00:36:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:10 smithi122 ceph-mon[32255]: pgmap v11928: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:10 smithi022 ceph-mon[31360]: pgmap v11928: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:10 smithi022 ceph-mon[40250]: pgmap v11928: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:11 smithi122 ceph-mon[32255]: pgmap v11929: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:11 smithi022 ceph-mon[31360]: pgmap v11929: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:11 smithi022 ceph-mon[40250]: pgmap v11929: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:36:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:36:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:36:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:36:13.759 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:14.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:14.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:14.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:14 smithi022 ceph-mon[31360]: pgmap v11930: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/86436181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:14 smithi022 ceph-mon[40250]: pgmap v11930: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/86436181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:14 smithi122 ceph-mon[32255]: pgmap v11930: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/86436181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:16 smithi122 ceph-mon[32255]: pgmap v11931: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:16 smithi022 ceph-mon[31360]: pgmap v11931: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:16.863 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:16 smithi022 ceph-mon[40250]: pgmap v11931: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:36:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:18 smithi122 ceph-mon[32255]: pgmap v11932: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:18 smithi022 ceph-mon[31360]: pgmap v11932: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:18 smithi022 ceph-mon[40250]: pgmap v11932: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:19.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:19.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:19.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:19.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:20] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:36:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:20 smithi122 ceph-mon[32255]: pgmap v11933: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1576932184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:20 smithi022 ceph-mon[31360]: pgmap v11933: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1576932184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:20 smithi022 ceph-mon[40250]: pgmap v11933: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1576932184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:21 smithi122 ceph-mon[32255]: pgmap v11934: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:21 smithi022 ceph-mon[31360]: pgmap v11934: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:21 smithi022 ceph-mon[40250]: pgmap v11934: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:36:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:36:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:36:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:36:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:24 smithi122 ceph-mon[32255]: pgmap v11935: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:24 smithi022 ceph-mon[31360]: pgmap v11935: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:24.792 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:24 smithi022 ceph-mon[40250]: pgmap v11935: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:24.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3524132778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3524132778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3524132778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:26 smithi122 ceph-mon[32255]: pgmap v11936: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:26 smithi022 ceph-mon[31360]: pgmap v11936: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:26 smithi022 ceph-mon[40250]: pgmap v11936: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:36:28.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:28 smithi122 ceph-mon[32255]: pgmap v11937: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:28 smithi022 ceph-mon[40250]: pgmap v11937: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:28 smithi022 ceph-mon[31360]: pgmap v11937: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:30.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:30.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:30.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:30] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:36:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:30 smithi122 ceph-mon[32255]: pgmap v11938: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3376159771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:30 smithi022 ceph-mon[31360]: pgmap v11938: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3376159771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:30 smithi022 ceph-mon[40250]: pgmap v11938: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3376159771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:31 smithi122 ceph-mon[32255]: pgmap v11939: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:31 smithi022 ceph-mon[31360]: pgmap v11939: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:31 smithi022 ceph-mon[40250]: pgmap v11939: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:36:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:36:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:36:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:36:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:34 smithi122 ceph-mon[32255]: pgmap v11940: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:36:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:36:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:36:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:36:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[31360]: pgmap v11940: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:36:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:36:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:36:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:36:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[40250]: pgmap v11940: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:36:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:36:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:36:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:36:35.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:35.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:36 smithi122 ceph-mon[32255]: pgmap v11941: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2300123375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:36 smithi022 ceph-mon[31360]: pgmap v11941: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2300123375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:36 smithi022 ceph-mon[40250]: pgmap v11941: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2300123375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:37.751 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:36:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:38 smithi122 ceph-mon[32255]: pgmap v11942: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:36:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:36:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:36:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:36:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[31360]: pgmap v11942: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:36:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:36:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:36:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:36:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[40250]: pgmap v11942: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:36:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:36:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:36:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:36:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:40 smithi122 ceph-mon[32255]: pgmap v11943: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:40] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:36:40.610 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:40.611 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:40 smithi022 ceph-mon[31360]: pgmap v11943: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:40.612 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:40 smithi022 ceph-mon[40250]: pgmap v11943: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:40.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:40.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:40.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4061048093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:41 smithi122 ceph-mon[32255]: pgmap v11944: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4061048093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:41 smithi022 ceph-mon[31360]: pgmap v11944: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4061048093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:41 smithi022 ceph-mon[40250]: pgmap v11944: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:36:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:36:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:36:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:36:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:44 smithi122 ceph-mon[32255]: pgmap v11945: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:44 smithi022 ceph-mon[31360]: pgmap v11945: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:44 smithi022 ceph-mon[40250]: pgmap v11945: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:45.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:46.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:46.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:46.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:46 smithi122 ceph-mon[32255]: pgmap v11946: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3522018215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:46 smithi022 ceph-mon[31360]: pgmap v11946: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3522018215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:46 smithi022 ceph-mon[40250]: pgmap v11946: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3522018215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:36:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:48 smithi122 ceph-mon[32255]: pgmap v11947: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:48 smithi022 ceph-mon[31360]: pgmap v11947: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:48 smithi022 ceph-mon[40250]: pgmap v11947: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:50] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:36:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:50 smithi122 ceph-mon[32255]: pgmap v11948: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:50 smithi022 ceph-mon[31360]: pgmap v11948: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:50 smithi022 ceph-mon[40250]: pgmap v11948: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:51.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:51.639 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:51 smithi022 ceph-mon[40250]: pgmap v11949: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:51.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:51.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:51.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:51 smithi122 ceph-mon[32255]: pgmap v11949: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:51 smithi022 ceph-mon[31360]: pgmap v11949: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:52.641 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:36:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:36:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4292299101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4292299101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4292299101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:53.647 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:36:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:36:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:53 smithi122 ceph-mon[32255]: pgmap v11950: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:53 smithi022 ceph-mon[31360]: pgmap v11950: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:36:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:53 smithi022 ceph-mon[40250]: pgmap v11950: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:36:56.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:56 smithi122 ceph-mon[32255]: pgmap v11951: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:56.743 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:36:56.743 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:56 smithi022 ceph-mon[31360]: pgmap v11951: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:56.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:56 smithi022 ceph-mon[40250]: pgmap v11951: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:57.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:36:57.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:36:57.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:36:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3440055622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3440055622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:36:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3440055622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:36:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:36:58 smithi122 ceph-mon[32255]: pgmap v11952: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:36:58 smithi022 ceph-mon[31360]: pgmap v11952: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:36:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:36:58 smithi022 ceph-mon[40250]: pgmap v11952: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:00 smithi122 ceph-mon[32255]: pgmap v11953: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:00] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:37:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:00 smithi022 ceph-mon[31360]: pgmap v11953: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:00 smithi022 ceph-mon[40250]: pgmap v11953: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:01 smithi122 ceph-mon[32255]: pgmap v11954: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:01 smithi022 ceph-mon[31360]: pgmap v11954: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:01 smithi022 ceph-mon[40250]: pgmap v11954: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:02.097 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:02.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:02.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:02.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:02.643 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:37:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:37:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1209217685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1209217685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1209217685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:03.650 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:37:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:37:03.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:03 smithi122 ceph-mon[32255]: pgmap v11955: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:03 smithi022 ceph-mon[31360]: pgmap v11955: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:03 smithi022 ceph-mon[40250]: pgmap v11955: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:06 smithi122 ceph-mon[32255]: pgmap v11956: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:06 smithi022 ceph-mon[31360]: pgmap v11956: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:06 smithi022 ceph-mon[40250]: pgmap v11956: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:07.468 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:37:07.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:07.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:07.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:08 smithi122 ceph-mon[32255]: pgmap v11957: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3790080693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:08 smithi022 ceph-mon[31360]: pgmap v11957: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3790080693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:08 smithi022 ceph-mon[40250]: pgmap v11957: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3790080693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:10 smithi122 ceph-mon[32255]: pgmap v11958: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:10] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:37:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:10 smithi022 ceph-mon[31360]: pgmap v11958: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:10 smithi022 ceph-mon[40250]: pgmap v11958: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:11 smithi122 ceph-mon[32255]: pgmap v11959: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:11 smithi022 ceph-mon[31360]: pgmap v11959: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:11 smithi022 ceph-mon[40250]: pgmap v11959: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:37:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:37:12.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:13.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:13.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:13.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2372203900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2372203900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2372203900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:37:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:37:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:14 smithi022 ceph-mon[31360]: pgmap v11960: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:14 smithi022 ceph-mon[40250]: pgmap v11960: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:14 smithi122 ceph-mon[32255]: pgmap v11960: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:16 smithi122 ceph-mon[32255]: pgmap v11961: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:16 smithi022 ceph-mon[31360]: pgmap v11961: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:16 smithi022 ceph-mon[40250]: pgmap v11961: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:37:18.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:18.566 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:18 smithi022 ceph-mon[31360]: pgmap v11962: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:18.567 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:18 smithi022 ceph-mon[40250]: pgmap v11962: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:18.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:18.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:18.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:18 smithi122 ceph-mon[32255]: pgmap v11962: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3428976768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3428976768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3428976768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:20] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:37:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:20 smithi122 ceph-mon[32255]: pgmap v11963: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:20 smithi022 ceph-mon[31360]: pgmap v11963: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:20 smithi022 ceph-mon[40250]: pgmap v11963: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:21 smithi122 ceph-mon[32255]: pgmap v11964: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:21 smithi022 ceph-mon[31360]: pgmap v11964: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:21 smithi022 ceph-mon[40250]: pgmap v11964: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:37:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:37:23.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:37:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:37:23.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:23.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:23.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:24 smithi122 ceph-mon[32255]: pgmap v11965: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3379218602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:24 smithi022 ceph-mon[31360]: pgmap v11965: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3379218602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:24 smithi022 ceph-mon[40250]: pgmap v11965: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3379218602' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:26 smithi122 ceph-mon[32255]: pgmap v11966: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:26 smithi022 ceph-mon[31360]: pgmap v11966: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:26 smithi022 ceph-mon[40250]: pgmap v11966: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:37:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:28 smithi122 ceph-mon[32255]: pgmap v11967: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:28 smithi022 ceph-mon[31360]: pgmap v11967: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:28 smithi022 ceph-mon[40250]: pgmap v11967: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:28.949 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:29.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:29.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:29.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2019054457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2019054457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2019054457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:30 smithi122 ceph-mon[32255]: pgmap v11968: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:30] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:37:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:30 smithi022 ceph-mon[31360]: pgmap v11968: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:30 smithi022 ceph-mon[40250]: pgmap v11968: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:31 smithi122 ceph-mon[32255]: pgmap v11969: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:31 smithi022 ceph-mon[31360]: pgmap v11969: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:31 smithi022 ceph-mon[40250]: pgmap v11969: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:37:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:37:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:37:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:37:34.321 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:34 smithi122 ceph-mon[32255]: pgmap v11970: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:37:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:37:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:37:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:37:34.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:34.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:34.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:34.694 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:34.694 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[31360]: pgmap v11970: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:34.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:37:34.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:37:34.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:37:34.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:37:34.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:34.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[40250]: pgmap v11970: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:34.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:37:34.696 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:37:34.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:37:34.697 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:37:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/88024516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/88024516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/88024516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:36 smithi122 ceph-mon[32255]: pgmap v11971: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:36 smithi022 ceph-mon[31360]: pgmap v11971: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:36 smithi022 ceph-mon[40250]: pgmap v11971: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:37:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[31360]: pgmap v11972: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:37:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:37:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:37:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[40250]: pgmap v11972: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:37:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:37:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:37:38.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:38 smithi122 ceph-mon[32255]: pgmap v11972: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:38.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:37:38.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:37:38.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:37:39.696 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:37:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:39 smithi122 ceph-mon[32255]: pgmap v11973: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:37:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:39 smithi022 ceph-mon[31360]: pgmap v11973: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:37:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:39 smithi022 ceph-mon[40250]: pgmap v11973: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:40.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:40.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:40.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:40] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:37:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2578124320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2578124320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2578124320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:41 smithi122 ceph-mon[32255]: pgmap v11974: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:41 smithi022 ceph-mon[31360]: pgmap v11974: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:41 smithi022 ceph-mon[40250]: pgmap v11974: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:37:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:37:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:37:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:37:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:44 smithi122 ceph-mon[32255]: pgmap v11975: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:44 smithi022 ceph-mon[31360]: pgmap v11975: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:44 smithi022 ceph-mon[40250]: pgmap v11975: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:45.064 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:45.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:45.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:45.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:46 smithi122 ceph-mon[32255]: pgmap v11976: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3496439846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:46 smithi022 ceph-mon[31360]: pgmap v11976: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3496439846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:46 smithi022 ceph-mon[40250]: pgmap v11976: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3496439846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:37:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:48 smithi122 ceph-mon[32255]: pgmap v11977: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:48 smithi022 ceph-mon[31360]: pgmap v11977: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:48 smithi022 ceph-mon[40250]: pgmap v11977: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:50] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:37:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:50 smithi122 ceph-mon[32255]: pgmap v11978: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:50.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:50 smithi022 ceph-mon[31360]: pgmap v11978: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:50 smithi022 ceph-mon[40250]: pgmap v11978: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:50.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:50.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3782242675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:51 smithi122 ceph-mon[32255]: pgmap v11979: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3782242675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:51 smithi022 ceph-mon[31360]: pgmap v11979: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3782242675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:51 smithi022 ceph-mon[40250]: pgmap v11979: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:37:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:37:53.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:37:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:37:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:54.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:54 smithi122 ceph-mon[32255]: pgmap v11980: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:54 smithi022 ceph-mon[31360]: pgmap v11980: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:37:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:54 smithi022 ceph-mon[40250]: pgmap v11980: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:37:55.812 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:37:56.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:37:56.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:37:56.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:37:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:56 smithi022 ceph-mon[31360]: pgmap v11981: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4107409966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:56 smithi022 ceph-mon[40250]: pgmap v11981: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4107409966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:56 smithi122 ceph-mon[32255]: pgmap v11981: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4107409966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:37:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:37:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:37:58 smithi122 ceph-mon[32255]: pgmap v11982: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:37:58 smithi022 ceph-mon[31360]: pgmap v11982: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:37:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:37:58 smithi022 ceph-mon[40250]: pgmap v11982: 132 pgs: 132 active+clean; 582 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:00] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:38:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:00 smithi122 ceph-mon[32255]: pgmap v11983: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:00 smithi022 ceph-mon[31360]: pgmap v11983: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:00 smithi022 ceph-mon[40250]: pgmap v11983: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:01.179 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:01.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:01.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:01.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:01 smithi122 ceph-mon[32255]: pgmap v11984: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3606599706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:02.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:01 smithi022 ceph-mon[31360]: pgmap v11984: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:02.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3606599706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:02.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:01 smithi022 ceph-mon[40250]: pgmap v11984: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:02.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3606599706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:38:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:38:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:38:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:38:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:04 smithi122 ceph-mon[32255]: pgmap v11985: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:04 smithi022 ceph-mon[31360]: pgmap v11985: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:04 smithi022 ceph-mon[40250]: pgmap v11985: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:06.551 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:06 smithi122 ceph-mon[32255]: pgmap v11986: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:06 smithi022 ceph-mon[31360]: pgmap v11986: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:06 smithi022 ceph-mon[40250]: pgmap v11986: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:06.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:06.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:06.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2184993109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2184993109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:38:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2184993109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:08 smithi122 ceph-mon[32255]: pgmap v11987: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:08 smithi022 ceph-mon[31360]: pgmap v11987: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:08 smithi022 ceph-mon[40250]: pgmap v11987: 132 pgs: 132 active+clean; 582 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:38:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:10 smithi122 ceph-mon[32255]: pgmap v11988: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:38:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:10] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:38:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:10 smithi022 ceph-mon[31360]: pgmap v11988: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:38:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:10 smithi022 ceph-mon[40250]: pgmap v11988: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:38:11.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:11.924 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:11 smithi022 ceph-mon[31360]: pgmap v11989: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:11.924 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:11 smithi022 ceph-mon[40250]: pgmap v11989: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:11 smithi122 ceph-mon[32255]: pgmap v11989: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:12.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:12.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:12.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:12.656 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:38:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:38:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/868064577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/868064577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/868064577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:13.662 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:38:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:38:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:13 smithi122 ceph-mon[32255]: pgmap v11990: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:38:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:13 smithi022 ceph-mon[31360]: pgmap v11990: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:38:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:13 smithi022 ceph-mon[40250]: pgmap v11990: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:38:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:16 smithi122 ceph-mon[32255]: pgmap v11991: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:16 smithi022 ceph-mon[31360]: pgmap v11991: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:16 smithi022 ceph-mon[40250]: pgmap v11991: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:17.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:17.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:17.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:17.664 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:38:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:18 smithi122 ceph-mon[32255]: pgmap v11992: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1529722680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:18 smithi022 ceph-mon[31360]: pgmap v11992: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1529722680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:18 smithi022 ceph-mon[40250]: pgmap v11992: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:38:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1529722680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:20] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:38:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:20 smithi122 ceph-mon[32255]: pgmap v11993: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:38:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:20 smithi022 ceph-mon[31360]: pgmap v11993: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:38:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:20 smithi022 ceph-mon[40250]: pgmap v11993: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:38:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:21 smithi122 ceph-mon[32255]: pgmap v11994: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:21 smithi022 ceph-mon[31360]: pgmap v11994: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:21 smithi022 ceph-mon[40250]: pgmap v11994: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:22.665 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:38:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:38:23.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:23.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:23.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3366151018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:38:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:38:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3366151018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3366151018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:24 smithi122 ceph-mon[32255]: pgmap v11995: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:24 smithi022 ceph-mon[31360]: pgmap v11995: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:24 smithi022 ceph-mon[40250]: pgmap v11995: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:26 smithi122 ceph-mon[32255]: pgmap v11996: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:26 smithi022 ceph-mon[31360]: pgmap v11996: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:26 smithi022 ceph-mon[40250]: pgmap v11996: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:38:28.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:28.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:28.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:28.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:28 smithi122 ceph-mon[32255]: pgmap v11997: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:28 smithi022 ceph-mon[31360]: pgmap v11997: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:28 smithi022 ceph-mon[40250]: pgmap v11997: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1306308522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1306308522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1306308522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:30 smithi122 ceph-mon[32255]: pgmap v11998: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:30] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:38:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:30 smithi022 ceph-mon[31360]: pgmap v11998: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:30 smithi022 ceph-mon[40250]: pgmap v11998: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:31 smithi122 ceph-mon[32255]: pgmap v11999: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:31 smithi022 ceph-mon[31360]: pgmap v11999: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:31 smithi022 ceph-mon[40250]: pgmap v11999: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:38:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:38:33.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:38:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:38:33.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:33.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:33.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:34 smithi122 ceph-mon[32255]: pgmap v12000: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:38:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:38:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:38:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:38:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3570357355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[31360]: pgmap v12000: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:38:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:38:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:38:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:38:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3570357355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[40250]: pgmap v12000: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:38:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:38:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:38:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:38:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3570357355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:36 smithi122 ceph-mon[32255]: pgmap v12001: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:36 smithi022 ceph-mon[31360]: pgmap v12001: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:36 smithi022 ceph-mon[40250]: pgmap v12001: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:38:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:38 smithi122 ceph-mon[32255]: pgmap v12002: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:38.682 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:38 smithi022 ceph-mon[31360]: pgmap v12002: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:38.683 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:38 smithi022 ceph-mon[40250]: pgmap v12002: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:38.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:39.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:38:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:38:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:38:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:38:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/791815490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:39.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:38:39.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:38:39.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:38:39.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:38:39.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/791815490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:39.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:38:39.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:38:39.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:38:39.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:38:39.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/791815490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:40 smithi122 ceph-mon[32255]: pgmap v12003: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:40] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:38:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:40 smithi022 ceph-mon[31360]: pgmap v12003: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:40 smithi022 ceph-mon[40250]: pgmap v12003: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:41 smithi122 ceph-mon[32255]: pgmap v12004: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:41 smithi022 ceph-mon[31360]: pgmap v12004: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:41 smithi022 ceph-mon[40250]: pgmap v12004: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:38:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:38:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:38:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:38:44.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:44.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:44.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:44.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:44.520 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:44.520 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:44 smithi022 ceph-mon[31360]: pgmap v12005: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:44.520 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:44.521 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:44 smithi022 ceph-mon[40250]: pgmap v12005: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:44 smithi122 ceph-mon[32255]: pgmap v12005: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/635603943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/635603943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/635603943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:46 smithi122 ceph-mon[32255]: pgmap v12006: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:46 smithi022 ceph-mon[31360]: pgmap v12006: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:46 smithi022 ceph-mon[40250]: pgmap v12006: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:38:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:48 smithi122 ceph-mon[32255]: pgmap v12007: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:48 smithi022 ceph-mon[31360]: pgmap v12007: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:48 smithi022 ceph-mon[40250]: pgmap v12007: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:49.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:49.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:49.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:50] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:38:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:50 smithi122 ceph-mon[32255]: pgmap v12008: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3731216537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:50 smithi022 ceph-mon[31360]: pgmap v12008: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3731216537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:50 smithi022 ceph-mon[40250]: pgmap v12008: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3731216537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:51 smithi122 ceph-mon[32255]: pgmap v12009: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:51 smithi022 ceph-mon[31360]: pgmap v12009: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:51 smithi022 ceph-mon[40250]: pgmap v12009: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:38:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:38:53.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:38:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:38:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:54 smithi122 ceph-mon[32255]: pgmap v12010: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:54.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:54.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:54 smithi022 ceph-mon[31360]: pgmap v12010: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:54.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:38:54.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:54 smithi022 ceph-mon[40250]: pgmap v12010: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:38:54.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:38:55.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:38:55.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:38:55.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:38:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/832413975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/832413975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:55.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/832413975' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:38:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:56 smithi122 ceph-mon[32255]: pgmap v12011: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:56 smithi022 ceph-mon[31360]: pgmap v12011: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:56 smithi022 ceph-mon[40250]: pgmap v12011: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:38:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:38:58 smithi122 ceph-mon[32255]: pgmap v12012: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:38:58 smithi022 ceph-mon[31360]: pgmap v12012: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:38:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:38:58 smithi022 ceph-mon[40250]: pgmap v12012: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:00.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:00] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:00 smithi122 ceph-mon[32255]: pgmap v12013: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:00 smithi022 ceph-mon[31360]: pgmap v12013: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:00 smithi022 ceph-mon[40250]: pgmap v12013: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:00.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:00.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1501043891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1501043891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1501043891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:02 smithi122 ceph-mon[32255]: pgmap v12014: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:39:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:39:02.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:02 smithi022 ceph-mon[31360]: pgmap v12014: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:02.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:02 smithi022 ceph-mon[40250]: pgmap v12014: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:39:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:39:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:04 smithi122 ceph-mon[32255]: pgmap v12015: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:04 smithi022 ceph-mon[31360]: pgmap v12015: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:04 smithi022 ceph-mon[40250]: pgmap v12015: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:05.633 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:05.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:06.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:06.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:06 smithi122 ceph-mon[32255]: pgmap v12016: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/392156839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:06 smithi022 ceph-mon[40250]: pgmap v12016: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/392156839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:06 smithi022 ceph-mon[31360]: pgmap v12016: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/392156839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:39:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:08 smithi122 ceph-mon[32255]: pgmap v12017: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:08 smithi022 ceph-mon[40250]: pgmap v12017: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:08 smithi022 ceph-mon[31360]: pgmap v12017: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:10 smithi122 ceph-mon[32255]: pgmap v12018: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:10] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:39:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:10 smithi022 ceph-mon[40250]: pgmap v12018: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:10 smithi022 ceph-mon[31360]: pgmap v12018: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:11.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:11.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:11.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:11.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:11.668 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:11 smithi022 ceph-mon[40250]: pgmap v12019: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:11.668 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3212057358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:11 smithi122 ceph-mon[32255]: pgmap v12019: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3212057358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:11 smithi022 ceph-mon[31360]: pgmap v12019: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3212057358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:39:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:39:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:39:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:39:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:14 smithi122 ceph-mon[32255]: pgmap v12020: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:14 smithi022 ceph-mon[31360]: pgmap v12020: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:14 smithi022 ceph-mon[40250]: pgmap v12020: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:16 smithi122 ceph-mon[32255]: pgmap v12021: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:16.743 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:16 smithi022 ceph-mon[31360]: pgmap v12021: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:16.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:16 smithi022 ceph-mon[40250]: pgmap v12021: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1656755082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1656755082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:39:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1656755082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:18.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:18 smithi122 ceph-mon[32255]: pgmap v12022: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:18 smithi022 ceph-mon[31360]: pgmap v12022: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:18 smithi022 ceph-mon[40250]: pgmap v12022: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:20] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:39:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:20 smithi122 ceph-mon[32255]: pgmap v12023: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:20 smithi022 ceph-mon[31360]: pgmap v12023: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:20 smithi022 ceph-mon[40250]: pgmap v12023: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:21.744 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:21 smithi122 ceph-mon[32255]: pgmap v12024: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:21 smithi022 ceph-mon[31360]: pgmap v12024: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:21 smithi022 ceph-mon[40250]: pgmap v12024: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:22.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:22.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:22.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:22.671 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:39:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:39:22.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3971482206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3971482206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3971482206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:23.676 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:39:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:39:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:23.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:23 smithi122 ceph-mon[32255]: pgmap v12025: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:23 smithi022 ceph-mon[31360]: pgmap v12025: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:23 smithi022 ceph-mon[40250]: pgmap v12025: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:26 smithi122 ceph-mon[32255]: pgmap v12026: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:26 smithi022 ceph-mon[31360]: pgmap v12026: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:26 smithi022 ceph-mon[40250]: pgmap v12026: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:27.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:27.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:27.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:27.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:39:28.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:28 smithi122 ceph-mon[32255]: pgmap v12027: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:28.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2022748109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:28 smithi022 ceph-mon[31360]: pgmap v12027: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2022748109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:28 smithi022 ceph-mon[40250]: pgmap v12027: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2022748109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:30] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:39:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:30 smithi122 ceph-mon[32255]: pgmap v12028: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:30 smithi022 ceph-mon[31360]: pgmap v12028: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:30 smithi022 ceph-mon[40250]: pgmap v12028: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:31 smithi122 ceph-mon[32255]: pgmap v12029: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:31 smithi022 ceph-mon[31360]: pgmap v12029: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:31 smithi022 ceph-mon[40250]: pgmap v12029: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:32.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:39:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:39:32.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:32.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:32.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3067707385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3067707385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3067707385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:33.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:39:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:39:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:33 smithi122 ceph-mon[32255]: pgmap v12030: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:39:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:39:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:39:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:39:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[31360]: pgmap v12030: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:34.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:39:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:39:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:39:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:39:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[40250]: pgmap v12030: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:39:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:39:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:39:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:39:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:36 smithi122 ceph-mon[32255]: pgmap v12031: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:36 smithi022 ceph-mon[31360]: pgmap v12031: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:36 smithi022 ceph-mon[40250]: pgmap v12031: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:39:37.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:38.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:38.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:38.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:38 smithi022 ceph-mon[31360]: pgmap v12032: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/761905514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:38 smithi022 ceph-mon[40250]: pgmap v12032: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/761905514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:38 smithi122 ceph-mon[32255]: pgmap v12032: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/761905514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:39:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:39:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:39:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:39:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:39:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:39:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:39:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:39:39.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:39:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:40] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:39:40.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:40 smithi122 ceph-mon[32255]: pgmap v12033: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:40.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:39:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:40 smithi022 ceph-mon[31360]: pgmap v12033: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:39:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:40 smithi022 ceph-mon[40250]: pgmap v12033: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:39:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:41 smithi122 ceph-mon[32255]: pgmap v12034: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:41 smithi022 ceph-mon[31360]: pgmap v12034: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:41 smithi022 ceph-mon[40250]: pgmap v12034: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:39:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:39:43.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:43.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:43.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:43.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:43.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:39:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:39:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:44 smithi122 ceph-mon[32255]: pgmap v12035: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3074683832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:44 smithi022 ceph-mon[31360]: pgmap v12035: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3074683832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:44 smithi022 ceph-mon[40250]: pgmap v12035: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3074683832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:46 smithi122 ceph-mon[32255]: pgmap v12036: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:46 smithi022 ceph-mon[31360]: pgmap v12036: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:46 smithi022 ceph-mon[40250]: pgmap v12036: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:39:48.599 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:48 smithi122 ceph-mon[32255]: pgmap v12037: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:48 smithi022 ceph-mon[31360]: pgmap v12037: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:48 smithi022 ceph-mon[40250]: pgmap v12037: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:48.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:48.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:48.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3498568884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3498568884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3498568884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:50] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:39:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:50 smithi122 ceph-mon[32255]: pgmap v12038: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:50 smithi022 ceph-mon[31360]: pgmap v12038: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:50 smithi022 ceph-mon[40250]: pgmap v12038: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:51 smithi122 ceph-mon[32255]: pgmap v12039: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:51 smithi022 ceph-mon[31360]: pgmap v12039: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:51 smithi022 ceph-mon[40250]: pgmap v12039: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:39:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:39:53.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:39:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:39:53.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:54.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:54.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:54.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:39:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:54 smithi122 ceph-mon[32255]: pgmap v12040: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/358415294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:54 smithi022 ceph-mon[31360]: pgmap v12040: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/358415294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:39:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:54 smithi022 ceph-mon[40250]: pgmap v12040: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:39:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/358415294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:39:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:56 smithi122 ceph-mon[32255]: pgmap v12041: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:56 smithi022 ceph-mon[31360]: pgmap v12041: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:56 smithi022 ceph-mon[40250]: pgmap v12041: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:39:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:39:58 smithi122 ceph-mon[32255]: pgmap v12042: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:58.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:39:58 smithi022 ceph-mon[31360]: pgmap v12042: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:58.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:39:58 smithi022 ceph-mon[40250]: pgmap v12042: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:39:59.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:39:59.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:39:59.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:39:59.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:00] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:40:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:00 smithi122 ceph-mon[32255]: pgmap v12043: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2733016064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T00:40:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:00 smithi022 ceph-mon[31360]: pgmap v12043: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2733016064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T00:40:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:00 smithi022 ceph-mon[40250]: pgmap v12043: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2733016064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T00:40:01.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:01 smithi022 ceph-mon[31360]: pgmap v12044: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:01.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:01 smithi022 ceph-mon[40250]: pgmap v12044: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:01 smithi122 ceph-mon[32255]: pgmap v12044: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:40:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:40:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:40:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:40:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:04 smithi122 ceph-mon[32255]: pgmap v12045: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:04.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:04.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:04.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:04 smithi022 ceph-mon[31360]: pgmap v12045: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:04.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:04.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:04 smithi022 ceph-mon[40250]: pgmap v12045: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2680178331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:05.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2680178331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2680178331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:06 smithi122 ceph-mon[32255]: pgmap v12046: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:06 smithi022 ceph-mon[31360]: pgmap v12046: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:06 smithi022 ceph-mon[40250]: pgmap v12046: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:08 smithi122 ceph-mon[32255]: pgmap v12047: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:08 smithi022 ceph-mon[31360]: pgmap v12047: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:08 smithi022 ceph-mon[40250]: pgmap v12047: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:10.106 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:10.425 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:10] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:40:10.426 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:10 smithi022 ceph-mon[40250]: pgmap v12048: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:10.427 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:10 smithi022 ceph-mon[31360]: pgmap v12048: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:10.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:10.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:10.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:10.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:10 smithi122 ceph-mon[32255]: pgmap v12048: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2098184436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2098184436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2098184436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:12 smithi122 ceph-mon[32255]: pgmap v12049: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:40:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:40:12.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:12 smithi022 ceph-mon[31360]: pgmap v12049: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:12.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:12 smithi022 ceph-mon[40250]: pgmap v12049: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:13.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:40:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:40:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:14 smithi122 ceph-mon[32255]: pgmap v12050: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:14 smithi022 ceph-mon[31360]: pgmap v12050: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:14 smithi022 ceph-mon[40250]: pgmap v12050: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:15.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:15.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:15.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:15.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:16 smithi122 ceph-mon[32255]: pgmap v12051: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3422914041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:16 smithi022 ceph-mon[31360]: pgmap v12051: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3422914041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:16 smithi022 ceph-mon[40250]: pgmap v12051: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3422914041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:40:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:18 smithi022 ceph-mon[31360]: pgmap v12052: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:18 smithi022 ceph-mon[40250]: pgmap v12052: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:18 smithi122 ceph-mon[32255]: pgmap v12052: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:40:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:20 smithi022 ceph-mon[31360]: pgmap v12053: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:20 smithi022 ceph-mon[40250]: pgmap v12053: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:20 smithi122 ceph-mon[32255]: pgmap v12053: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:21.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:21.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:21.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/165582576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:21 smithi122 ceph-mon[32255]: pgmap v12054: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/165582576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:21 smithi022 ceph-mon[31360]: pgmap v12054: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/165582576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:21 smithi022 ceph-mon[40250]: pgmap v12054: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:40:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:40:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:40:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:40:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:24 smithi122 ceph-mon[32255]: pgmap v12055: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:24 smithi022 ceph-mon[31360]: pgmap v12055: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:24 smithi022 ceph-mon[40250]: pgmap v12055: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:26.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:26 smithi022 ceph-mon[31360]: pgmap v12056: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:26 smithi022 ceph-mon[40250]: pgmap v12056: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:26.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:26 smithi122 ceph-mon[32255]: pgmap v12056: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/137219597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/137219597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:40:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/137219597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:28 smithi122 ceph-mon[32255]: pgmap v12057: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:28 smithi022 ceph-mon[31360]: pgmap v12057: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:28 smithi022 ceph-mon[40250]: pgmap v12057: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:30.431 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:40:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:30 smithi122 ceph-mon[32255]: pgmap v12058: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:30 smithi022 ceph-mon[31360]: pgmap v12058: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:30 smithi022 ceph-mon[40250]: pgmap v12058: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:31.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:31 smithi122 ceph-mon[32255]: pgmap v12059: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:31.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:31.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:31.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:31.960 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:31 smithi022 ceph-mon[31360]: pgmap v12059: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:31.960 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:31 smithi022 ceph-mon[40250]: pgmap v12059: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T00:40:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:40:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:40:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3494820137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3494820137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3494820137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:40:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:40:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[31360]: pgmap v12060: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:40:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:40:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:40:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:40:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[40250]: pgmap v12060: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:40:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:40:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:40:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:40:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:33 smithi122 ceph-mon[32255]: pgmap v12060: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T00:40:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:40:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:40:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:40:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:40:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:36 smithi122 ceph-mon[32255]: pgmap v12061: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:36 smithi022 ceph-mon[31360]: pgmap v12061: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:36 smithi022 ceph-mon[40250]: pgmap v12061: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:37.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:37.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:37.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3256858653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3256858653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:40:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3256858653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:38 smithi122 ceph-mon[32255]: pgmap v12062: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:38 smithi022 ceph-mon[31360]: pgmap v12062: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:38 smithi022 ceph-mon[40250]: pgmap v12062: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:40.432 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:40] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:40:40.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:40 smithi122 ceph-mon[32255]: pgmap v12063: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:40.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:40:40.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:40:40.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:40:40.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:40:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[31360]: pgmap v12063: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:40:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:40:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:40:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:40:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[40250]: pgmap v12063: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:40:40.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:40:40.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:40:40.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:40:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:41 smithi022 ceph-mon[31360]: pgmap v12064: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:41 smithi022 ceph-mon[40250]: pgmap v12064: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:41 smithi122 ceph-mon[32255]: pgmap v12064: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:42.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:40:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:40:42.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:42.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:42.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1990486605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1990486605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1990486605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:40:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:40:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:43 smithi022 ceph-mon[31360]: pgmap v12065: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:43 smithi022 ceph-mon[40250]: pgmap v12065: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:43 smithi122 ceph-mon[32255]: pgmap v12065: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:46 smithi122 ceph-mon[32255]: pgmap v12066: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:46 smithi022 ceph-mon[31360]: pgmap v12066: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:46 smithi022 ceph-mon[40250]: pgmap v12066: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:47.704 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:47.705 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:40:48.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:48.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:48.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:48 smithi122 ceph-mon[32255]: pgmap v12067: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/983547175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:48 smithi022 ceph-mon[31360]: pgmap v12067: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/983547175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:48 smithi022 ceph-mon[40250]: pgmap v12067: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/983547175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:50.431 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:40:50.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:50 smithi122 ceph-mon[32255]: pgmap v12068: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:50 smithi022 ceph-mon[31360]: pgmap v12068: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:50 smithi022 ceph-mon[40250]: pgmap v12068: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:51 smithi022 ceph-mon[31360]: pgmap v12069: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:51 smithi022 ceph-mon[40250]: pgmap v12069: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:51 smithi122 ceph-mon[32255]: pgmap v12069: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:40:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:40:53.076 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:53.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:53.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:53.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:53.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:40:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:40:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:54 smithi122 ceph-mon[32255]: pgmap v12070: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2751179343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:54 smithi022 ceph-mon[40250]: pgmap v12070: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2751179343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:54 smithi022 ceph-mon[31360]: pgmap v12070: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:40:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:40:54.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2751179343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:56 smithi122 ceph-mon[32255]: pgmap v12071: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:56 smithi022 ceph-mon[31360]: pgmap v12071: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:56 smithi022 ceph-mon[40250]: pgmap v12071: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:40:58.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:40:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:58 smithi122 ceph-mon[32255]: pgmap v12072: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:58 smithi022 ceph-mon[31360]: pgmap v12072: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:58 smithi022 ceph-mon[40250]: pgmap v12072: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:40:58.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:40:58.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:40:58.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:40:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:40:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/571676334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:40:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/571676334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:40:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:40:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/571676334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:41:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:00 smithi022 ceph-mon[31360]: pgmap v12073: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:00 smithi022 ceph-mon[40250]: pgmap v12073: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:00 smithi122 ceph-mon[32255]: pgmap v12073: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:01 smithi022 ceph-mon[31360]: pgmap v12074: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:01 smithi022 ceph-mon[40250]: pgmap v12074: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:01 smithi122 ceph-mon[32255]: pgmap v12074: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:41:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:41:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:41:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:41:03.820 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:04.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:04.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:04.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:04 smithi022 ceph-mon[31360]: pgmap v12075: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3659568668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:04 smithi022 ceph-mon[40250]: pgmap v12075: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3659568668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:04 smithi122 ceph-mon[32255]: pgmap v12075: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3659568668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:06 smithi122 ceph-mon[32255]: pgmap v12076: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:06 smithi022 ceph-mon[31360]: pgmap v12076: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:06 smithi022 ceph-mon[40250]: pgmap v12076: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:41:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:08 smithi022 ceph-mon[31360]: pgmap v12077: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:08 smithi022 ceph-mon[40250]: pgmap v12077: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:08 smithi122 ceph-mon[32255]: pgmap v12077: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:09.189 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:09.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:09.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:09.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:10] "GET /metrics HTTP/1.1" 200 202417 "" "Prometheus/2.18.1" 2021-12-16T00:41:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:10 smithi022 ceph-mon[31360]: pgmap v12078: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/818740677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:10 smithi022 ceph-mon[40250]: pgmap v12078: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/818740677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:10.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:10 smithi122 ceph-mon[32255]: pgmap v12078: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:10.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/818740677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:11 smithi022 ceph-mon[31360]: pgmap v12079: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:11 smithi022 ceph-mon[40250]: pgmap v12079: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:11 smithi122 ceph-mon[32255]: pgmap v12079: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:41:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:41:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:41:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:41:14.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:14 smithi022 ceph-mon[31360]: pgmap v12080: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:14 smithi022 ceph-mon[40250]: pgmap v12080: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:14.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:14.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:14.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:14 smithi122 ceph-mon[32255]: pgmap v12080: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3406801438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3406801438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3406801438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:16 smithi022 ceph-mon[31360]: pgmap v12081: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:16 smithi022 ceph-mon[40250]: pgmap v12081: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:16 smithi122 ceph-mon[32255]: pgmap v12081: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:41:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:18 smithi022 ceph-mon[31360]: pgmap v12082: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:18 smithi022 ceph-mon[40250]: pgmap v12082: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:18 smithi122 ceph-mon[32255]: pgmap v12082: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:19.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:20.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:20.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:20.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:20] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:41:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:20 smithi022 ceph-mon[31360]: pgmap v12083: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/581561136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:20 smithi022 ceph-mon[40250]: pgmap v12083: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/581561136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:20 smithi122 ceph-mon[32255]: pgmap v12083: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/581561136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:21 smithi022 ceph-mon[31360]: pgmap v12084: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:21 smithi022 ceph-mon[40250]: pgmap v12084: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:21 smithi122 ceph-mon[32255]: pgmap v12084: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:41:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:41:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:41:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:41:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:24 smithi022 ceph-mon[31360]: pgmap v12085: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:24 smithi022 ceph-mon[40250]: pgmap v12085: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:24 smithi122 ceph-mon[32255]: pgmap v12085: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:25.304 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:25.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:25.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:25.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:26 smithi022 ceph-mon[31360]: pgmap v12086: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3132444370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:26 smithi022 ceph-mon[40250]: pgmap v12086: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3132444370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:26 smithi122 ceph-mon[32255]: pgmap v12086: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3132444370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:41:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:28 smithi022 ceph-mon[31360]: pgmap v12087: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:28 smithi022 ceph-mon[40250]: pgmap v12087: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:28 smithi122 ceph-mon[32255]: pgmap v12087: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:30] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:41:30.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:30 smithi022 ceph-mon[31360]: pgmap v12088: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:30 smithi022 ceph-mon[40250]: pgmap v12088: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:30 smithi122 ceph-mon[32255]: pgmap v12088: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:31.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:31.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:31.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4109554132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:31 smithi022 ceph-mon[31360]: pgmap v12089: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4109554132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:31 smithi022 ceph-mon[40250]: pgmap v12089: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4109554132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:31 smithi122 ceph-mon[32255]: pgmap v12089: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:41:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:41:33.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:41:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:41:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[31360]: pgmap v12090: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:34.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:34.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:41:34.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:41:34.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:41:34.796 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:41:34.796 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[40250]: pgmap v12090: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:34.796 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:34.797 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:41:34.797 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:41:34.797 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:41:34.797 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:34 smithi122 ceph-mon[32255]: pgmap v12090: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:41:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:41:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:41:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:41:36.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:36 smithi022 ceph-mon[31360]: pgmap v12091: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3148535626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:36 smithi022 ceph-mon[40250]: pgmap v12091: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3148535626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:36 smithi122 ceph-mon[32255]: pgmap v12091: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3148535626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:41:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:38 smithi022 ceph-mon[31360]: pgmap v12092: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:38 smithi022 ceph-mon[40250]: pgmap v12092: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:38 smithi122 ceph-mon[32255]: pgmap v12092: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:40.443 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:40] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:41:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[31360]: pgmap v12093: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:41:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:41:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:41:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[40250]: pgmap v12093: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:41:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:41:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:41:40.865 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:40 smithi122 ceph-mon[32255]: pgmap v12093: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:40.865 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:41:40.865 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:41:40.866 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:41:41.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:41.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:41.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:41.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:41.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.799 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.800 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.800 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.800 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:41:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:42 smithi122 ceph-mon[32255]: pgmap v12094: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2084008882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:41:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:41:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:42 smithi022 ceph-mon[31360]: pgmap v12094: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2084008882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:42 smithi022 ceph-mon[40250]: pgmap v12094: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2084008882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:43.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:41:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:41:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:44 smithi022 ceph-mon[31360]: pgmap v12095: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:44 smithi022 ceph-mon[40250]: pgmap v12095: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:44 smithi122 ceph-mon[32255]: pgmap v12095: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:46 smithi022 ceph-mon[31360]: pgmap v12096: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:46 smithi022 ceph-mon[40250]: pgmap v12096: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:46.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:46 smithi122 ceph-mon[32255]: pgmap v12096: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:47.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:47.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:47.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:47.469 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:41:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/690787599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/690787599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/690787599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:48 smithi022 ceph-mon[31360]: pgmap v12097: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:48 smithi022 ceph-mon[40250]: pgmap v12097: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:48.952 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:48 smithi122 ceph-mon[32255]: pgmap v12097: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:50] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:41:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:50 smithi022 ceph-mon[31360]: pgmap v12098: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:50 smithi022 ceph-mon[40250]: pgmap v12098: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:50.953 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:50 smithi122 ceph-mon[32255]: pgmap v12098: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:52.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:51 smithi022 ceph-mon[31360]: pgmap v12099: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:52.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:51 smithi022 ceph-mon[40250]: pgmap v12099: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:52.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:51 smithi122 ceph-mon[32255]: pgmap v12099: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:52.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:52.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:52.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:52.723 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:41:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:41:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3347696108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3347696108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3347696108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:53.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:41:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:41:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:53 smithi022 ceph-mon[31360]: pgmap v12100: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:53 smithi022 ceph-mon[40250]: pgmap v12100: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:54.198 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:53 smithi122 ceph-mon[32255]: pgmap v12100: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:41:54.198 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:41:56.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:56 smithi022 ceph-mon[31360]: pgmap v12101: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:56.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:56 smithi022 ceph-mon[40250]: pgmap v12101: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:56 smithi122 ceph-mon[32255]: pgmap v12101: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:57.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:41:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:41:57.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:41:57.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:41:57.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:41:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:58 smithi022 ceph-mon[31360]: pgmap v12102: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:41:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/291867011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:58 smithi022 ceph-mon[40250]: pgmap v12102: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:41:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/291867011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:41:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:58 smithi122 ceph-mon[32255]: pgmap v12102: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:41:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:41:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/291867011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:00] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:42:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:00 smithi022 ceph-mon[31360]: pgmap v12103: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:00 smithi022 ceph-mon[40250]: pgmap v12103: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:00 smithi122 ceph-mon[32255]: pgmap v12103: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:01 smithi022 ceph-mon[31360]: pgmap v12104: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:01 smithi022 ceph-mon[40250]: pgmap v12104: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:01 smithi122 ceph-mon[32255]: pgmap v12104: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:42:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:42:02.888 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:03.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:03.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:03.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3633963245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3633963245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:42:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:42:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3633963245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:04 smithi122 ceph-mon[32255]: pgmap v12105: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:04 smithi022 ceph-mon[31360]: pgmap v12105: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:04 smithi022 ceph-mon[40250]: pgmap v12105: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:06 smithi022 ceph-mon[31360]: pgmap v12106: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:06 smithi022 ceph-mon[40250]: pgmap v12106: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:06 smithi122 ceph-mon[32255]: pgmap v12106: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:42:08.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:08 smithi022 ceph-mon[31360]: pgmap v12107: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:08 smithi022 ceph-mon[40250]: pgmap v12107: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:08.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:08.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:08.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:08 smithi122 ceph-mon[32255]: pgmap v12107: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1948960269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1948960269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:09.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1948960269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:10] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:42:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:10 smithi022 ceph-mon[31360]: pgmap v12108: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:10 smithi022 ceph-mon[40250]: pgmap v12108: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:10 smithi122 ceph-mon[32255]: pgmap v12108: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:11 smithi022 ceph-mon[31360]: pgmap v12109: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:11 smithi022 ceph-mon[40250]: pgmap v12109: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:11 smithi122 ceph-mon[32255]: pgmap v12109: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:42:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:42:13.633 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:42:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:42:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:14 smithi022 ceph-mon[31360]: pgmap v12110: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4171795516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:14 smithi022 ceph-mon[40250]: pgmap v12110: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4171795516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:14 smithi122 ceph-mon[32255]: pgmap v12110: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4171795516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:16 smithi022 ceph-mon[31360]: pgmap v12111: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:16 smithi022 ceph-mon[40250]: pgmap v12111: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:16 smithi122 ceph-mon[32255]: pgmap v12111: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:42:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:18 smithi022 ceph-mon[31360]: pgmap v12112: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:18 smithi022 ceph-mon[40250]: pgmap v12112: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:18 smithi122 ceph-mon[32255]: pgmap v12112: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:19.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:19.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:19.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:19.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/260531746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/260531746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/260531746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:20] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:42:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:20 smithi022 ceph-mon[31360]: pgmap v12113: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:20 smithi022 ceph-mon[40250]: pgmap v12113: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:20 smithi122 ceph-mon[32255]: pgmap v12113: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:21 smithi022 ceph-mon[31360]: pgmap v12114: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:21 smithi022 ceph-mon[40250]: pgmap v12114: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:21 smithi122 ceph-mon[32255]: pgmap v12114: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:42:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:42:23.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:42:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:42:24.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:24.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:24.746 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:24 smithi022 ceph-mon[40250]: pgmap v12115: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:24.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:24.747 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:24 smithi022 ceph-mon[31360]: pgmap v12115: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:24.747 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:24 smithi122 ceph-mon[32255]: pgmap v12115: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1897732011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1897732011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1897732011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:26 smithi022 ceph-mon[31360]: pgmap v12116: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:26 smithi022 ceph-mon[40250]: pgmap v12116: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:26 smithi122 ceph-mon[32255]: pgmap v12116: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:42:28.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:28 smithi022 ceph-mon[31360]: pgmap v12117: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:28.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:28 smithi022 ceph-mon[40250]: pgmap v12117: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:28 smithi122 ceph-mon[32255]: pgmap v12117: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:29.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:30.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:30.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:30.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:30] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:42:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:30 smithi022 ceph-mon[31360]: pgmap v12118: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2826498614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:30 smithi022 ceph-mon[40250]: pgmap v12118: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2826498614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:30 smithi122 ceph-mon[32255]: pgmap v12118: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2826498614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:31 smithi022 ceph-mon[31360]: pgmap v12119: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:31 smithi022 ceph-mon[40250]: pgmap v12119: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:31 smithi122 ceph-mon[32255]: pgmap v12119: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:42:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:42:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:42:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:42:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[31360]: pgmap v12120: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:42:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:42:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:42:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:42:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[40250]: pgmap v12120: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:42:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:42:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:42:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:34 smithi122 ceph-mon[32255]: pgmap v12120: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:42:35.119 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:35.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:35.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:35.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:36 smithi022 ceph-mon[31360]: pgmap v12121: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1147994045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:36 smithi022 ceph-mon[40250]: pgmap v12121: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1147994045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:36 smithi122 ceph-mon[32255]: pgmap v12121: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1147994045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:42:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:38 smithi022 ceph-mon[31360]: pgmap v12122: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:38 smithi022 ceph-mon[40250]: pgmap v12122: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:38 smithi122 ceph-mon[32255]: pgmap v12122: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:40] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:42:40.490 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:40 smithi022 ceph-mon[31360]: pgmap v12123: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:40.884 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:40 smithi022 ceph-mon[40250]: pgmap v12123: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:40.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:40.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:40.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:40.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:40 smithi122 ceph-mon[32255]: pgmap v12123: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1977020935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:42:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:42:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:42:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:42:42.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[31360]: pgmap v12124: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1977020935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:42:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:42:42.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:42:42.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:42:42.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:41 smithi022 ceph-mon[40250]: pgmap v12124: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1977020935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:42:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:42:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:42:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:42:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:41 smithi122 ceph-mon[32255]: pgmap v12124: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:42:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:42:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:42:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:42:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:44 smithi022 ceph-mon[31360]: pgmap v12125: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:44 smithi022 ceph-mon[40250]: pgmap v12125: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:44 smithi122 ceph-mon[32255]: pgmap v12125: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:45.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:46.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:46.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:46.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:46 smithi022 ceph-mon[40250]: pgmap v12126: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1475535949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:46 smithi022 ceph-mon[31360]: pgmap v12126: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1475535949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:46.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:46 smithi122 ceph-mon[32255]: pgmap v12126: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1475535949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:42:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:48 smithi022 ceph-mon[31360]: pgmap v12127: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:48 smithi022 ceph-mon[40250]: pgmap v12127: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:48 smithi122 ceph-mon[32255]: pgmap v12127: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:42:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:50 smithi022 ceph-mon[31360]: pgmap v12128: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:50 smithi022 ceph-mon[40250]: pgmap v12128: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:50 smithi122 ceph-mon[32255]: pgmap v12128: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:51.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:51.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:51.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:51.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:51 smithi022 ceph-mon[31360]: pgmap v12129: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1537158185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:51 smithi022 ceph-mon[40250]: pgmap v12129: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1537158185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:51 smithi122 ceph-mon[32255]: pgmap v12129: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1537158185' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:42:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:42:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:42:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:42:54.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:54 smithi022 ceph-mon[31360]: pgmap v12130: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:54.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:54.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:54 smithi022 ceph-mon[40250]: pgmap v12130: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:54.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:54 smithi122 ceph-mon[32255]: pgmap v12130: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:42:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:42:56.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:42:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:56 smithi022 ceph-mon[31360]: pgmap v12131: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:56 smithi022 ceph-mon[40250]: pgmap v12131: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:56 smithi122 ceph-mon[32255]: pgmap v12131: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:56.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:42:56.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:42:56.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:42:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2664961639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:42:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2664961639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2664961639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:42:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:42:58 smithi022 ceph-mon[31360]: pgmap v12132: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:42:58 smithi022 ceph-mon[40250]: pgmap v12132: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:42:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:42:58 smithi122 ceph-mon[32255]: pgmap v12132: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:43:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:00 smithi022 ceph-mon[31360]: pgmap v12133: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:00 smithi022 ceph-mon[40250]: pgmap v12133: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:00 smithi122 ceph-mon[32255]: pgmap v12133: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:01.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:01.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:01 smithi022 ceph-mon[31360]: pgmap v12134: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:01.981 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:01 smithi022 ceph-mon[40250]: pgmap v12134: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:01 smithi122 ceph-mon[32255]: pgmap v12134: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:02.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:02.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:02.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:43:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:43:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/296279544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/296279544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/296279544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:43:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:43:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:03 smithi122 ceph-mon[32255]: pgmap v12135: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:03.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:03 smithi022 ceph-mon[31360]: pgmap v12135: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:03 smithi022 ceph-mon[40250]: pgmap v12135: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:04.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:06 smithi022 ceph-mon[31360]: pgmap v12136: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:06 smithi022 ceph-mon[40250]: pgmap v12136: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:06 smithi122 ceph-mon[32255]: pgmap v12136: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:07.351 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:07.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:07.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:07.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:07.722 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:43:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:08 smithi022 ceph-mon[31360]: pgmap v12137: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1152049403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:08 smithi022 ceph-mon[40250]: pgmap v12137: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1152049403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:08 smithi122 ceph-mon[32255]: pgmap v12137: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1152049403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:10] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T00:43:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:10 smithi022 ceph-mon[31360]: pgmap v12138: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:10 smithi022 ceph-mon[40250]: pgmap v12138: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:10 smithi122 ceph-mon[32255]: pgmap v12138: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:11 smithi022 ceph-mon[31360]: pgmap v12139: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:11 smithi022 ceph-mon[40250]: pgmap v12139: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:11 smithi122 ceph-mon[32255]: pgmap v12139: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:43:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:43:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:13.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:13.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:13.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/563156596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/563156596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/563156596' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:43:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:43:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:14 smithi122 ceph-mon[32255]: pgmap v12140: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:14 smithi022 ceph-mon[31360]: pgmap v12140: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:14 smithi022 ceph-mon[40250]: pgmap v12140: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:16 smithi022 ceph-mon[31360]: pgmap v12141: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:16 smithi022 ceph-mon[40250]: pgmap v12141: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:16 smithi122 ceph-mon[32255]: pgmap v12141: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:43:18.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:18 smithi022 ceph-mon[31360]: pgmap v12142: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1317325179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:18 smithi022 ceph-mon[40250]: pgmap v12142: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1317325179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:18 smithi122 ceph-mon[32255]: pgmap v12142: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1317325179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:20] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:43:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:20 smithi022 ceph-mon[31360]: pgmap v12143: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:20 smithi022 ceph-mon[40250]: pgmap v12143: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:20 smithi122 ceph-mon[32255]: pgmap v12143: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:21 smithi022 ceph-mon[31360]: pgmap v12144: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:21 smithi022 ceph-mon[40250]: pgmap v12144: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:21 smithi122 ceph-mon[32255]: pgmap v12144: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:43:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:43:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:43:23.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:23.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:23.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:23.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:23.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:43:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:43:24.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:24 smithi022 ceph-mon[31360]: pgmap v12145: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:24.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:24.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2118040241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:24.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:24 smithi022 ceph-mon[40250]: pgmap v12145: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:24.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:24.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2118040241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:24 smithi122 ceph-mon[32255]: pgmap v12145: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2118040241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:26 smithi022 ceph-mon[31360]: pgmap v12146: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:26 smithi022 ceph-mon[40250]: pgmap v12146: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:26 smithi122 ceph-mon[32255]: pgmap v12146: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:43:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:28 smithi022 ceph-mon[31360]: pgmap v12147: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:28 smithi022 ceph-mon[40250]: pgmap v12147: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:28.835 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:28 smithi122 ceph-mon[32255]: pgmap v12147: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:29.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:29.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:29.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:29.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1949647938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:29.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1949647938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1949647938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:30] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:43:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:30 smithi022 ceph-mon[31360]: pgmap v12148: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:30 smithi022 ceph-mon[40250]: pgmap v12148: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:30 smithi122 ceph-mon[32255]: pgmap v12148: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:32.039 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:31 smithi022 ceph-mon[31360]: pgmap v12149: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:32.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:31 smithi022 ceph-mon[40250]: pgmap v12149: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:32.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:31 smithi122 ceph-mon[32255]: pgmap v12149: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:43:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:43:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:43:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:43:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:34.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:34.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:34.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[31360]: pgmap v12150: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:43:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:43:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:43:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:43:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[40250]: pgmap v12150: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:43:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:43:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:43:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:34 smithi122 ceph-mon[32255]: pgmap v12150: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:43:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:43:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:43:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:43:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/257215997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/257215997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:35.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/257215997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:36 smithi022 ceph-mon[31360]: pgmap v12151: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:36.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:36 smithi022 ceph-mon[40250]: pgmap v12151: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:36 smithi122 ceph-mon[32255]: pgmap v12151: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:43:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:38 smithi022 ceph-mon[31360]: pgmap v12152: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:38 smithi022 ceph-mon[40250]: pgmap v12152: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:38.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:38 smithi122 ceph-mon[32255]: pgmap v12152: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:39.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:39.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:39.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:40.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:40] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:43:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:40 smithi022 ceph-mon[31360]: pgmap v12153: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3425612021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:40 smithi022 ceph-mon[40250]: pgmap v12153: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3425612021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:40 smithi122 ceph-mon[32255]: pgmap v12153: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3425612021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:43:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:43:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:43:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[31360]: pgmap v12154: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:43:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:43:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:43:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:41 smithi022 ceph-mon[40250]: pgmap v12154: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:43:42.302 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:43:42.302 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:43:42.302 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:41 smithi122 ceph-mon[32255]: pgmap v12154: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:43:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:43:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:43:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:43:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:43:43.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:43:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:43:44.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:43 smithi122 ceph-mon[32255]: pgmap v12155: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:43 smithi022 ceph-mon[31360]: pgmap v12155: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:43 smithi022 ceph-mon[40250]: pgmap v12155: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:44.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:45.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:45.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:45.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1039830215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:45.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1039830215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:45.795 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1039830215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:46 smithi122 ceph-mon[32255]: pgmap v12156: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:46 smithi022 ceph-mon[31360]: pgmap v12156: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:46 smithi022 ceph-mon[40250]: pgmap v12156: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:43:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:48 smithi022 ceph-mon[31360]: pgmap v12157: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:48 smithi022 ceph-mon[40250]: pgmap v12157: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:48 smithi122 ceph-mon[32255]: pgmap v12157: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:50.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:50] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:43:50.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:50.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:50.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:50.694 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:50 smithi022 ceph-mon[31360]: pgmap v12158: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:50.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:50 smithi022 ceph-mon[40250]: pgmap v12158: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:50 smithi122 ceph-mon[32255]: pgmap v12158: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1402829742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1402829742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1402829742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:52 smithi122 ceph-mon[32255]: pgmap v12159: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:43:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:43:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:52 smithi022 ceph-mon[31360]: pgmap v12159: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:52 smithi022 ceph-mon[40250]: pgmap v12159: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:43:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:43:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:54 smithi022 ceph-mon[31360]: pgmap v12160: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:54 smithi022 ceph-mon[40250]: pgmap v12160: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:54 smithi122 ceph-mon[32255]: pgmap v12160: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:43:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:43:55.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:43:56.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:43:56.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:43:56.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:43:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:56 smithi022 ceph-mon[31360]: pgmap v12161: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/868008708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:56 smithi022 ceph-mon[40250]: pgmap v12161: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/868008708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:56 smithi122 ceph-mon[32255]: pgmap v12161: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/868008708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:43:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:43:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:43:58 smithi022 ceph-mon[31360]: pgmap v12162: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:43:58 smithi022 ceph-mon[40250]: pgmap v12162: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:43:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:43:58 smithi122 ceph-mon[32255]: pgmap v12162: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:00] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:44:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:00 smithi022 ceph-mon[31360]: pgmap v12163: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:00 smithi022 ceph-mon[40250]: pgmap v12163: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:00 smithi122 ceph-mon[32255]: pgmap v12163: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:01.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:01.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:01.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2206642206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:01 smithi022 ceph-mon[31360]: pgmap v12164: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2206642206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:01 smithi022 ceph-mon[40250]: pgmap v12164: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2206642206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:01 smithi122 ceph-mon[32255]: pgmap v12164: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:44:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:44:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:44:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:44:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:04 smithi022 ceph-mon[31360]: pgmap v12165: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:04 smithi022 ceph-mon[40250]: pgmap v12165: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:04 smithi122 ceph-mon[32255]: pgmap v12165: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:06.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:06 smithi022 ceph-mon[31360]: pgmap v12166: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:06 smithi022 ceph-mon[40250]: pgmap v12166: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:06.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:06.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:06.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:06 smithi122 ceph-mon[32255]: pgmap v12166: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2112394371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:44:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2112394371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2112394371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:08 smithi022 ceph-mon[31360]: pgmap v12167: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:08 smithi022 ceph-mon[40250]: pgmap v12167: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:08 smithi122 ceph-mon[32255]: pgmap v12167: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:10] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T00:44:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:10 smithi022 ceph-mon[31360]: pgmap v12168: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:10 smithi022 ceph-mon[40250]: pgmap v12168: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:10 smithi122 ceph-mon[32255]: pgmap v12168: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:11 smithi022 ceph-mon[31360]: pgmap v12169: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:11 smithi022 ceph-mon[40250]: pgmap v12169: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:12.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:12.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:12.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:11 smithi122 ceph-mon[32255]: pgmap v12169: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:44:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:44:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2422619887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:13.098 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2422619887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2422619887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:44:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:44:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:13 smithi122 ceph-mon[32255]: pgmap v12170: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:13 smithi022 ceph-mon[31360]: pgmap v12170: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:13 smithi022 ceph-mon[40250]: pgmap v12170: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:16 smithi022 ceph-mon[40250]: pgmap v12171: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:16 smithi022 ceph-mon[31360]: pgmap v12171: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:16 smithi122 ceph-mon[32255]: pgmap v12171: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:17.171 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:44:17.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:17.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:17.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:18 smithi022 ceph-mon[40250]: pgmap v12172: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3206153788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:18 smithi022 ceph-mon[31360]: pgmap v12172: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3206153788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:18 smithi122 ceph-mon[32255]: pgmap v12172: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3206153788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T00:44:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:20 smithi022 ceph-mon[31360]: pgmap v12173: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:20 smithi022 ceph-mon[40250]: pgmap v12173: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:20 smithi122 ceph-mon[32255]: pgmap v12173: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:21 smithi022 ceph-mon[31360]: pgmap v12174: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:21 smithi022 ceph-mon[40250]: pgmap v12174: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:21 smithi122 ceph-mon[32255]: pgmap v12174: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:44:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:44:22.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:22.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:22.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1860671982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1860671982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1860671982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:23.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:44:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:44:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:23 smithi022 ceph-mon[31360]: pgmap v12175: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:23 smithi022 ceph-mon[40250]: pgmap v12175: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:23 smithi122 ceph-mon[32255]: pgmap v12175: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:26 smithi022 ceph-mon[31360]: pgmap v12176: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:26 smithi022 ceph-mon[40250]: pgmap v12176: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:26 smithi122 ceph-mon[32255]: pgmap v12176: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:44:27.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:28.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:28.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:28.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:28 smithi022 ceph-mon[31360]: pgmap v12177: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:28.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1198130371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:28 smithi022 ceph-mon[40250]: pgmap v12177: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:28.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1198130371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:28 smithi122 ceph-mon[32255]: pgmap v12177: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1198130371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T00:44:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:30 smithi022 ceph-mon[31360]: pgmap v12178: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:30 smithi022 ceph-mon[40250]: pgmap v12178: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:30 smithi122 ceph-mon[32255]: pgmap v12178: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:32.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:31 smithi022 ceph-mon[31360]: pgmap v12179: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:31 smithi022 ceph-mon[40250]: pgmap v12179: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:31 smithi122 ceph-mon[32255]: pgmap v12179: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:44:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:44:33.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:33.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:33.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:33.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:33.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:44:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:44:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[40250]: pgmap v12180: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3544851280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:44:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:44:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:44:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:44:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[31360]: pgmap v12180: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3544851280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:44:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:44:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:44:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:44:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:34 smithi122 ceph-mon[32255]: pgmap v12180: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3544851280' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:44:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:44:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:44:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:44:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:36 smithi022 ceph-mon[31360]: pgmap v12181: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:36 smithi022 ceph-mon[40250]: pgmap v12181: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:36 smithi122 ceph-mon[32255]: pgmap v12181: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:44:38.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:38 smithi022 ceph-mon[31360]: pgmap v12182: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:38 smithi022 ceph-mon[40250]: pgmap v12182: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:38 smithi122 ceph-mon[32255]: pgmap v12182: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:39.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:39.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:39.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3204315113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3204315113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:39.946 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3204315113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:40] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:44:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:40 smithi022 ceph-mon[31360]: pgmap v12183: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:40 smithi022 ceph-mon[40250]: pgmap v12183: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:40 smithi122 ceph-mon[32255]: pgmap v12183: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:41 smithi022 ceph-mon[31360]: pgmap v12184: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:41 smithi022 ceph-mon[40250]: pgmap v12184: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:41 smithi122 ceph-mon[32255]: pgmap v12184: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:44:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:44:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:44:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:44:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:44:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:44:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:44:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:44:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:44:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:44:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:44:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:44:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:44:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:44:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:44:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:44:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:43 smithi122 ceph-mon[32255]: pgmap v12185: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:44.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:43 smithi022 ceph-mon[31360]: pgmap v12185: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:43 smithi022 ceph-mon[40250]: pgmap v12185: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:44.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:44.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:44.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2139303688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2139303688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:45.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2139303688' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:45 smithi022 ceph-mon[31360]: pgmap v12186: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:45 smithi022 ceph-mon[40250]: pgmap v12186: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:46.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:45 smithi122 ceph-mon[32255]: pgmap v12186: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:44:48.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:48 smithi022 ceph-mon[31360]: pgmap v12187: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:48.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:48 smithi022 ceph-mon[40250]: pgmap v12187: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:48 smithi122 ceph-mon[32255]: pgmap v12187: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:49.397 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:49.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:49.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:49.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:50.459 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:44:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:50 smithi022 ceph-mon[31360]: pgmap v12188: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3258927375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:50 smithi022 ceph-mon[40250]: pgmap v12188: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3258927375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:50 smithi122 ceph-mon[32255]: pgmap v12188: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3258927375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:51 smithi022 ceph-mon[31360]: pgmap v12189: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:51 smithi022 ceph-mon[40250]: pgmap v12189: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:51 smithi122 ceph-mon[32255]: pgmap v12189: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:44:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:44:53.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:44:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:44:54.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:44:54.767 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:54 smithi022 ceph-mon[31360]: pgmap v12190: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:54.768 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:54.768 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:54 smithi022 ceph-mon[40250]: pgmap v12190: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:54.768 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:54 smithi122 ceph-mon[32255]: pgmap v12190: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:44:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:44:55.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:44:55.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:44:55.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:44:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3390644982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3390644982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3390644982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:44:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:56 smithi022 ceph-mon[31360]: pgmap v12191: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:56 smithi022 ceph-mon[40250]: pgmap v12191: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:56 smithi122 ceph-mon[32255]: pgmap v12191: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:44:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:44:58 smithi022 ceph-mon[31360]: pgmap v12192: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:44:58 smithi022 ceph-mon[40250]: pgmap v12192: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:44:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:44:58 smithi122 ceph-mon[32255]: pgmap v12192: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:00.139 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:00.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:45:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:00 smithi022 ceph-mon[31360]: pgmap v12193: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4066854726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:00 smithi022 ceph-mon[40250]: pgmap v12193: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4066854726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:00 smithi122 ceph-mon[32255]: pgmap v12193: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4066854726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:01 smithi022 ceph-mon[31360]: pgmap v12194: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:01 smithi022 ceph-mon[40250]: pgmap v12194: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:01 smithi122 ceph-mon[32255]: pgmap v12194: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:45:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:45:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:45:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:45:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:04 smithi022 ceph-mon[31360]: pgmap v12195: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:04 smithi022 ceph-mon[40250]: pgmap v12195: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:04 smithi122 ceph-mon[32255]: pgmap v12195: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:05.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:05.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:05.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:05.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:06 smithi022 ceph-mon[31360]: pgmap v12196: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/339326898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:06 smithi022 ceph-mon[40250]: pgmap v12196: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/339326898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:06 smithi122 ceph-mon[32255]: pgmap v12196: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/339326898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:07.803 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:45:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:08 smithi022 ceph-mon[31360]: pgmap v12197: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:08 smithi022 ceph-mon[40250]: pgmap v12197: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:08 smithi122 ceph-mon[32255]: pgmap v12197: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:10] "GET /metrics HTTP/1.1" 200 202421 "" "Prometheus/2.18.1" 2021-12-16T00:45:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:10 smithi022 ceph-mon[31360]: pgmap v12198: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:10 smithi022 ceph-mon[40250]: pgmap v12198: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:10.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:10 smithi122 ceph-mon[32255]: pgmap v12198: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/277844921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:11 smithi022 ceph-mon[31360]: pgmap v12199: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/277844921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:11 smithi022 ceph-mon[40250]: pgmap v12199: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/277844921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:11 smithi122 ceph-mon[32255]: pgmap v12199: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:45:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:45:13.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:45:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:45:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:14 smithi022 ceph-mon[31360]: pgmap v12200: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:14 smithi022 ceph-mon[40250]: pgmap v12200: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:14 smithi122 ceph-mon[32255]: pgmap v12200: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:16.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:16.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:16.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:16.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:16 smithi022 ceph-mon[31360]: pgmap v12201: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:16 smithi022 ceph-mon[40250]: pgmap v12201: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:16 smithi122 ceph-mon[32255]: pgmap v12201: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3451893033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:45:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3451893033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:17.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3451893033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:18 smithi022 ceph-mon[31360]: pgmap v12202: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:18 smithi022 ceph-mon[40250]: pgmap v12202: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:18 smithi122 ceph-mon[32255]: pgmap v12202: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:20] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:45:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:20 smithi022 ceph-mon[31360]: pgmap v12203: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:20.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:20 smithi022 ceph-mon[40250]: pgmap v12203: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:20 smithi122 ceph-mon[32255]: pgmap v12203: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:21.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:21.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:21.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:21.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:21.987 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:21 smithi022 ceph-mon[31360]: pgmap v12204: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:21.988 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:21 smithi022 ceph-mon[40250]: pgmap v12204: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:21 smithi122 ceph-mon[32255]: pgmap v12204: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:45:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:45:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3694534333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3694534333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:23.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3694534333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:23 smithi122 ceph-mon[32255]: pgmap v12205: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:23.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:45:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:45:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:23 smithi022 ceph-mon[31360]: pgmap v12205: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:23 smithi022 ceph-mon[40250]: pgmap v12205: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:26 smithi022 ceph-mon[31360]: pgmap v12206: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:26 smithi022 ceph-mon[40250]: pgmap v12206: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:26 smithi122 ceph-mon[32255]: pgmap v12206: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:26.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:27.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:27.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:27.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/106541833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:45:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/106541833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/106541833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:28 smithi022 ceph-mon[31360]: pgmap v12207: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:28 smithi022 ceph-mon[40250]: pgmap v12207: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:28 smithi122 ceph-mon[32255]: pgmap v12207: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:30] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:45:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:30 smithi022 ceph-mon[31360]: pgmap v12208: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:30 smithi022 ceph-mon[40250]: pgmap v12208: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:30 smithi122 ceph-mon[32255]: pgmap v12208: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:31 smithi022 ceph-mon[31360]: pgmap v12209: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:31 smithi022 ceph-mon[40250]: pgmap v12209: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:31 smithi122 ceph-mon[32255]: pgmap v12209: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:32.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:45:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:45:32.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:32.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:32.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/939512438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:33.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/939512438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/939512438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:33.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:45:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:45:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:33 smithi122 ceph-mon[32255]: pgmap v12210: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:45:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:45:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:45:33.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:45:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[31360]: pgmap v12210: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:45:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:45:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:45:34.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:45:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[40250]: pgmap v12210: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:45:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:45:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:45:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:45:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:36 smithi022 ceph-mon[31360]: pgmap v12211: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:36 smithi022 ceph-mon[40250]: pgmap v12211: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:36 smithi122 ceph-mon[32255]: pgmap v12211: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:37.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:37.729 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:45:38.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:38.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:38.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:38 smithi022 ceph-mon[31360]: pgmap v12212: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1282386912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:38 smithi022 ceph-mon[40250]: pgmap v12212: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1282386912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:38.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:38 smithi122 ceph-mon[32255]: pgmap v12212: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:38.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1282386912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:40] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:45:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:40 smithi022 ceph-mon[31360]: pgmap v12213: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:40 smithi022 ceph-mon[40250]: pgmap v12213: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:40 smithi122 ceph-mon[32255]: pgmap v12213: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:41 smithi022 ceph-mon[31360]: pgmap v12214: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:41 smithi022 ceph-mon[40250]: pgmap v12214: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:41 smithi122 ceph-mon[32255]: pgmap v12214: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:45:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:45:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:45:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:45:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:45:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:45:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:45:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:45:43.097 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:45:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:45:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:45:43.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:43.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:43.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:43 smithi122 ceph-mon[32255]: pgmap v12215: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3994960012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:43.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:45:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:45:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:43 smithi022 ceph-mon[31360]: pgmap v12215: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3994960012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:43 smithi022 ceph-mon[40250]: pgmap v12215: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3994960012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: pgmap v12216: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:46.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: pgmap v12216: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:46.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:46.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: pgmap v12216: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:45:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3061M 2021-12-16T00:45:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3061M 2021-12-16T00:45:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3061M 2021-12-16T00:45:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:45:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:45:48.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:48 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3266M 2021-12-16T00:45:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:48 smithi022 ceph-mon[31360]: pgmap v12217: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:48 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3266M 2021-12-16T00:45:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:48 smithi022 ceph-mon[40250]: pgmap v12217: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:48.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:48.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:48.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:48 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3266M 2021-12-16T00:45:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:48 smithi122 ceph-mon[32255]: pgmap v12217: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3415779204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3415779204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3415779204' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:50] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:45:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:50 smithi022 ceph-mon[31360]: pgmap v12218: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:50 smithi022 ceph-mon[40250]: pgmap v12218: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:50 smithi122 ceph-mon[32255]: pgmap v12218: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:51 smithi022 ceph-mon[31360]: pgmap v12219: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:51 smithi022 ceph-mon[40250]: pgmap v12219: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:51 smithi122 ceph-mon[32255]: pgmap v12219: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:45:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:45:53.865 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:45:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:45:54.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:54.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:54.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:45:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:54 smithi022 ceph-mon[31360]: pgmap v12220: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/558780515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:54 smithi022 ceph-mon[40250]: pgmap v12220: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/558780515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:54 smithi122 ceph-mon[32255]: pgmap v12220: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:45:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:45:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/558780515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:45:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:56 smithi022 ceph-mon[31360]: pgmap v12221: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:56 smithi022 ceph-mon[40250]: pgmap v12221: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:56.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:56 smithi122 ceph-mon[32255]: pgmap v12221: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:45:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:45:58 smithi022 ceph-mon[31360]: pgmap v12222: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:45:58 smithi022 ceph-mon[40250]: pgmap v12222: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:45:58 smithi122 ceph-mon[32255]: pgmap v12222: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:45:59.232 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:45:59.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:45:59.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:45:59.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:00] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:46:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:00 smithi022 ceph-mon[31360]: pgmap v12223: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2464747932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:00 smithi022 ceph-mon[40250]: pgmap v12223: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2464747932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:00 smithi122 ceph-mon[32255]: pgmap v12223: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2464747932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:01 smithi022 ceph-mon[31360]: pgmap v12224: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:01 smithi022 ceph-mon[40250]: pgmap v12224: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:01 smithi122 ceph-mon[32255]: pgmap v12224: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:46:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:46:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:46:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:46:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:04 smithi022 ceph-mon[31360]: pgmap v12225: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:04 smithi022 ceph-mon[40250]: pgmap v12225: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:04 smithi122 ceph-mon[32255]: pgmap v12225: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:04.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:04.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:04.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2156972302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2156972302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2156972302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:06 smithi022 ceph-mon[31360]: pgmap v12226: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:06 smithi022 ceph-mon[40250]: pgmap v12226: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:06 smithi122 ceph-mon[32255]: pgmap v12226: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:46:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:08 smithi122 ceph-mon[32255]: pgmap v12227: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:08 smithi022 ceph-mon[40250]: pgmap v12227: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:08 smithi022 ceph-mon[31360]: pgmap v12227: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:09.973 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:10.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:10.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:10.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:10] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:46:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:10 smithi022 ceph-mon[31360]: pgmap v12228: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3597789562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:10 smithi022 ceph-mon[40250]: pgmap v12228: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3597789562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:10.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:10 smithi122 ceph-mon[32255]: pgmap v12228: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:10.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3597789562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:11 smithi022 ceph-mon[31360]: pgmap v12229: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:11 smithi022 ceph-mon[40250]: pgmap v12229: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:11 smithi122 ceph-mon[32255]: pgmap v12229: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:46:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:46:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:46:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:46:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:14 smithi022 ceph-mon[31360]: pgmap v12230: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:14 smithi022 ceph-mon[40250]: pgmap v12230: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:14 smithi122 ceph-mon[32255]: pgmap v12230: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:15.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:15.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:15.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:15.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:16 smithi022 ceph-mon[31360]: pgmap v12231: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1518979442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:16 smithi022 ceph-mon[40250]: pgmap v12231: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1518979442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:16 smithi122 ceph-mon[32255]: pgmap v12231: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1518979442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:46:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:18 smithi122 ceph-mon[32255]: pgmap v12232: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:18 smithi022 ceph-mon[40250]: pgmap v12232: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:18 smithi022 ceph-mon[31360]: pgmap v12232: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T00:46:20.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:20 smithi122 ceph-mon[32255]: pgmap v12233: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:20 smithi022 ceph-mon[31360]: pgmap v12233: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:20 smithi022 ceph-mon[40250]: pgmap v12233: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:21.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:21.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:21.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3557556734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:21 smithi022 ceph-mon[31360]: pgmap v12234: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3557556734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:21 smithi022 ceph-mon[40250]: pgmap v12234: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3557556734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:21 smithi122 ceph-mon[32255]: pgmap v12234: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:46:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:46:23.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:46:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:46:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:24 smithi022 ceph-mon[31360]: pgmap v12235: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:24 smithi022 ceph-mon[40250]: pgmap v12235: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:24 smithi122 ceph-mon[32255]: pgmap v12235: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:26.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:26.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:26.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:26 smithi022 ceph-mon[31360]: pgmap v12236: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/184678001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:26 smithi022 ceph-mon[40250]: pgmap v12236: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/184678001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:26 smithi122 ceph-mon[32255]: pgmap v12236: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/184678001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:46:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:28 smithi122 ceph-mon[32255]: pgmap v12237: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:28 smithi022 ceph-mon[31360]: pgmap v12237: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:28 smithi022 ceph-mon[40250]: pgmap v12237: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T00:46:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:30 smithi122 ceph-mon[32255]: pgmap v12238: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:30 smithi022 ceph-mon[31360]: pgmap v12238: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:30 smithi022 ceph-mon[40250]: pgmap v12238: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:31.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:31 smithi022 ceph-mon[40250]: pgmap v12239: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:31.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:31.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:31.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:31 smithi022 ceph-mon[31360]: pgmap v12239: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:31 smithi122 ceph-mon[32255]: pgmap v12239: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:46:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:46:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4151937780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4151937780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4151937780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:46:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:46:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:33 smithi122 ceph-mon[32255]: pgmap v12240: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:46:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:46:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:46:33.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:46:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[31360]: pgmap v12240: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:46:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:46:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:46:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:46:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[40250]: pgmap v12240: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:46:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:46:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:46:34.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:46:36.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:36.830 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:36 smithi022 ceph-mon[31360]: pgmap v12241: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:36.830 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:36 smithi022 ceph-mon[40250]: pgmap v12241: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:36 smithi122 ceph-mon[32255]: pgmap v12241: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:37.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:37.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:37.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:46:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3318561558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3318561558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3318561558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:38 smithi122 ceph-mon[32255]: pgmap v12242: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:38 smithi022 ceph-mon[31360]: pgmap v12242: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:38 smithi022 ceph-mon[40250]: pgmap v12242: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:40] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:46:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:40 smithi122 ceph-mon[32255]: pgmap v12243: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:40 smithi022 ceph-mon[31360]: pgmap v12243: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:40 smithi022 ceph-mon[40250]: pgmap v12243: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:41 smithi022 ceph-mon[31360]: pgmap v12244: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:41 smithi022 ceph-mon[40250]: pgmap v12244: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:41 smithi122 ceph-mon[32255]: pgmap v12244: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:42.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:42.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:42.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:42.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:46:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:46:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1939791745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1939791745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1939791745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:46:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:46:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:43 smithi122 ceph-mon[32255]: pgmap v12245: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:43 smithi022 ceph-mon[31360]: pgmap v12245: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:43 smithi022 ceph-mon[40250]: pgmap v12245: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:46 smithi122 ceph-mon[32255]: pgmap v12246: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:46 smithi022 ceph-mon[31360]: pgmap v12246: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:46 smithi022 ceph-mon[40250]: pgmap v12246: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:46:47.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:47.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:47.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:47.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:47.922 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:46:47.923 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:46:47.923 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:46:47.923 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:46:47.924 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:46:47.924 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:46:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:46:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:46:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:46:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:48 smithi122 ceph-mon[32255]: pgmap v12247: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:46:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/964925581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:48 smithi022 ceph-mon[31360]: pgmap v12247: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:46:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/964925581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:48 smithi022 ceph-mon[40250]: pgmap v12247: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:46:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/964925581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:50] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:46:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:50 smithi122 ceph-mon[32255]: pgmap v12248: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:50 smithi022 ceph-mon[31360]: pgmap v12248: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:50 smithi022 ceph-mon[40250]: pgmap v12248: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:51 smithi022 ceph-mon[31360]: pgmap v12249: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:51 smithi022 ceph-mon[40250]: pgmap v12249: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:51 smithi122 ceph-mon[32255]: pgmap v12249: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:46:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:46:52.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:53.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:53.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:53.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:53.658 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/118989935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/118989935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/118989935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:46:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:46:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:46:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:54 smithi122 ceph-mon[32255]: pgmap v12250: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:54 smithi022 ceph-mon[31360]: pgmap v12250: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:54 smithi022 ceph-mon[40250]: pgmap v12250: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:46:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:46:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:56 smithi122 ceph-mon[32255]: pgmap v12251: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:56 smithi022 ceph-mon[31360]: pgmap v12251: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:56 smithi022 ceph-mon[40250]: pgmap v12251: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:46:58.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:46:58.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:46:58.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:46:58.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:46:58.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:58 smithi022 ceph-mon[40250]: pgmap v12252: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:58.664 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:58 smithi022 ceph-mon[31360]: pgmap v12252: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:58 smithi122 ceph-mon[32255]: pgmap v12252: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:46:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:46:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2846325367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:46:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2846325367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:46:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2846325367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:00] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:47:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:00 smithi122 ceph-mon[32255]: pgmap v12253: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:00 smithi022 ceph-mon[31360]: pgmap v12253: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:00 smithi022 ceph-mon[40250]: pgmap v12253: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:01 smithi022 ceph-mon[31360]: pgmap v12254: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:01 smithi022 ceph-mon[40250]: pgmap v12254: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:01 smithi122 ceph-mon[32255]: pgmap v12254: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:47:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:47:03.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:47:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:47:04.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:04 smithi122 ceph-mon[32255]: pgmap v12255: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/40338264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:04 smithi022 ceph-mon[31360]: pgmap v12255: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/40338264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:04 smithi022 ceph-mon[40250]: pgmap v12255: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/40338264' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:06 smithi122 ceph-mon[32255]: pgmap v12256: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:06 smithi022 ceph-mon[31360]: pgmap v12256: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:06 smithi022 ceph-mon[40250]: pgmap v12256: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:47:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:08 smithi122 ceph-mon[32255]: pgmap v12257: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:08 smithi022 ceph-mon[31360]: pgmap v12257: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:08 smithi022 ceph-mon[40250]: pgmap v12257: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:09.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:09.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:09.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:09.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3036732681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3036732681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3036732681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:10] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:47:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:10 smithi122 ceph-mon[32255]: pgmap v12258: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:10 smithi022 ceph-mon[40250]: pgmap v12258: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:10 smithi022 ceph-mon[31360]: pgmap v12258: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:11 smithi022 ceph-mon[31360]: pgmap v12259: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:11 smithi022 ceph-mon[40250]: pgmap v12259: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:11 smithi122 ceph-mon[32255]: pgmap v12259: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:47:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:47:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:47:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:47:14.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:14.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:14.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:14.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:14.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:14 smithi022 ceph-mon[31360]: pgmap v12260: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:14.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:14.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:14 smithi022 ceph-mon[40250]: pgmap v12260: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:14.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:14 smithi122 ceph-mon[32255]: pgmap v12260: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1166875798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1166875798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1166875798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:16 smithi122 ceph-mon[32255]: pgmap v12261: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:16 smithi022 ceph-mon[31360]: pgmap v12261: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:16 smithi022 ceph-mon[40250]: pgmap v12261: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:47:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:18 smithi122 ceph-mon[32255]: pgmap v12262: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:18 smithi022 ceph-mon[31360]: pgmap v12262: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:18 smithi022 ceph-mon[40250]: pgmap v12262: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:19.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:20.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:20.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:20.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:20] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:47:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:20 smithi122 ceph-mon[32255]: pgmap v12263: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1601493825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:20 smithi022 ceph-mon[31360]: pgmap v12263: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1601493825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:20 smithi022 ceph-mon[40250]: pgmap v12263: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1601493825' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:21 smithi022 ceph-mon[31360]: pgmap v12264: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:21 smithi022 ceph-mon[40250]: pgmap v12264: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:21 smithi122 ceph-mon[32255]: pgmap v12264: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:47:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:47:24.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:47:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:47:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:24 smithi122 ceph-mon[32255]: pgmap v12265: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:24 smithi022 ceph-mon[31360]: pgmap v12265: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:24 smithi022 ceph-mon[40250]: pgmap v12265: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:25.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/983417704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/983417704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/983417704' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:26 smithi122 ceph-mon[32255]: pgmap v12266: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:26 smithi022 ceph-mon[31360]: pgmap v12266: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:26 smithi022 ceph-mon[40250]: pgmap v12266: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:47:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:28 smithi122 ceph-mon[32255]: pgmap v12267: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:28 smithi022 ceph-mon[31360]: pgmap v12267: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:28 smithi022 ceph-mon[40250]: pgmap v12267: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:30] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:47:30.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:30 smithi022 ceph-mon[31360]: pgmap v12268: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:30 smithi022 ceph-mon[40250]: pgmap v12268: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:30.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:30.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:30.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:30 smithi122 ceph-mon[32255]: pgmap v12268: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/92038153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:31 smithi022 ceph-mon[31360]: pgmap v12269: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/92038153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:31 smithi022 ceph-mon[40250]: pgmap v12269: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/92038153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:31 smithi122 ceph-mon[32255]: pgmap v12269: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:47:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:47:34.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:47:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:47:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:34 smithi122 ceph-mon[32255]: pgmap v12270: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:47:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:47:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:47:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:47:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[31360]: pgmap v12270: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[40250]: pgmap v12270: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:47:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:47:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:35.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:36.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:36.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:36.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:36 smithi122 ceph-mon[32255]: pgmap v12271: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/210310039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:36 smithi022 ceph-mon[31360]: pgmap v12271: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/210310039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:36 smithi022 ceph-mon[40250]: pgmap v12271: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/210310039' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:47:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:38 smithi122 ceph-mon[32255]: pgmap v12272: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:38 smithi022 ceph-mon[31360]: pgmap v12272: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:38 smithi022 ceph-mon[40250]: pgmap v12272: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:40] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:47:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:40 smithi122 ceph-mon[32255]: pgmap v12273: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:40 smithi022 ceph-mon[31360]: pgmap v12273: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:40 smithi022 ceph-mon[40250]: pgmap v12273: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:41.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:41.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:41.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:41.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:41 smithi022 ceph-mon[31360]: pgmap v12274: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1740837405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:41 smithi022 ceph-mon[40250]: pgmap v12274: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1740837405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:41 smithi122 ceph-mon[32255]: pgmap v12274: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1740837405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:47:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:47:44.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:47:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:47:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:44 smithi122 ceph-mon[32255]: pgmap v12275: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:44 smithi022 ceph-mon[31360]: pgmap v12275: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:44 smithi022 ceph-mon[40250]: pgmap v12275: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:46.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:46 smithi122 ceph-mon[32255]: pgmap v12276: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:46.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:46.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:46.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:46.992 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:46 smithi022 ceph-mon[40250]: pgmap v12276: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:46.992 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:46 smithi022 ceph-mon[31360]: pgmap v12276: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:47.649 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:47:47.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1765826412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:47.650 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1765826412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1765826412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:48 smithi122 ceph-mon[32255]: pgmap v12277: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:47:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:47:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:47:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:47:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[31360]: pgmap v12277: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:47:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:47:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:47:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:47:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[40250]: pgmap v12277: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:47:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:47:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:47:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:47:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:50] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:47:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:50 smithi122 ceph-mon[32255]: pgmap v12278: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:50 smithi022 ceph-mon[31360]: pgmap v12278: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:50 smithi022 ceph-mon[40250]: pgmap v12278: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:51.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:51 smithi122 ceph-mon[32255]: pgmap v12279: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:51 smithi022 ceph-mon[31360]: pgmap v12279: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:51 smithi022 ceph-mon[40250]: pgmap v12279: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:52.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:52.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:52.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:47:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:47:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1398258161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1398258161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1398258161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:53 smithi122 ceph-mon[32255]: pgmap v12280: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:54.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:47:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:47:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:53 smithi022 ceph-mon[31360]: pgmap v12280: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:53 smithi022 ceph-mon[40250]: pgmap v12280: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:47:55.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:47:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:55 smithi122 ceph-mon[32255]: pgmap v12281: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:55 smithi022 ceph-mon[31360]: pgmap v12281: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:55 smithi022 ceph-mon[40250]: pgmap v12281: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:57.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:47:57.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:47:57.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:47:57.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:47:57.757 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:47:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:58 smithi122 ceph-mon[32255]: pgmap v12282: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:47:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1804974915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:58 smithi022 ceph-mon[31360]: pgmap v12282: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:47:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1804974915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:47:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:58 smithi022 ceph-mon[40250]: pgmap v12282: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:47:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:47:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1804974915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:00] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:48:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:00 smithi122 ceph-mon[32255]: pgmap v12283: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:00 smithi022 ceph-mon[31360]: pgmap v12283: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:00 smithi022 ceph-mon[40250]: pgmap v12283: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:01 smithi122 ceph-mon[32255]: pgmap v12284: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:01 smithi022 ceph-mon[31360]: pgmap v12284: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:01 smithi022 ceph-mon[40250]: pgmap v12284: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:48:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:48:02.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:03.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:03.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:03.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3078982161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3078982161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3078982161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:04.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:48:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:48:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:04 smithi122 ceph-mon[32255]: pgmap v12285: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:04 smithi022 ceph-mon[31360]: pgmap v12285: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:04 smithi022 ceph-mon[40250]: pgmap v12285: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:06.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:06 smithi122 ceph-mon[32255]: pgmap v12286: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:06 smithi022 ceph-mon[31360]: pgmap v12286: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:06 smithi022 ceph-mon[40250]: pgmap v12286: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:48:08.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:08.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:08.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:08.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:08 smithi022 ceph-mon[31360]: pgmap v12287: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3998622697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:08 smithi022 ceph-mon[40250]: pgmap v12287: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3998622697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:08 smithi122 ceph-mon[32255]: pgmap v12287: 132 pgs: 132 active+clean; 582 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:48:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3998622697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:10] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T00:48:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:10 smithi122 ceph-mon[32255]: pgmap v12288: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:48:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:10 smithi022 ceph-mon[31360]: pgmap v12288: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:48:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:10 smithi022 ceph-mon[40250]: pgmap v12288: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:48:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:11 smithi122 ceph-mon[32255]: pgmap v12289: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:11 smithi022 ceph-mon[31360]: pgmap v12289: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:11 smithi022 ceph-mon[40250]: pgmap v12289: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:48:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:48:13.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:13.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:13.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:13.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:14.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:48:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:48:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:14 smithi122 ceph-mon[32255]: pgmap v12290: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:48:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2668716158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:15.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:14 smithi022 ceph-mon[40250]: pgmap v12290: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:48:15.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2668716158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:15.044 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:14 smithi022 ceph-mon[31360]: pgmap v12290: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:48:15.044 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2668716158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:16.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:16 smithi122 ceph-mon[32255]: pgmap v12291: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:16 smithi022 ceph-mon[31360]: pgmap v12291: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:16 smithi022 ceph-mon[40250]: pgmap v12291: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:48:18.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:18.845 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:18 smithi022 ceph-mon[31360]: pgmap v12292: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:18.846 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:18 smithi022 ceph-mon[40250]: pgmap v12292: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:18 smithi122 ceph-mon[32255]: pgmap v12292: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:48:19.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:19.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:19.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1195885086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1195885086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1195885086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:20] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:48:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:20 smithi122 ceph-mon[32255]: pgmap v12293: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:48:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:20 smithi022 ceph-mon[31360]: pgmap v12293: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:48:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:20 smithi022 ceph-mon[40250]: pgmap v12293: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:48:22.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:21 smithi122 ceph-mon[32255]: pgmap v12294: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:21 smithi022 ceph-mon[31360]: pgmap v12294: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:21 smithi022 ceph-mon[40250]: pgmap v12294: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:48:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:48:24.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:48:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:48:24.213 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:24.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:24.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:24.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:24 smithi122 ceph-mon[32255]: pgmap v12295: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:24 smithi022 ceph-mon[31360]: pgmap v12295: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:24 smithi022 ceph-mon[40250]: pgmap v12295: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/760154554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/760154554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/760154554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:26 smithi122 ceph-mon[32255]: pgmap v12296: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:26 smithi022 ceph-mon[31360]: pgmap v12296: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:26 smithi022 ceph-mon[40250]: pgmap v12296: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:48:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:28 smithi122 ceph-mon[32255]: pgmap v12297: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:28 smithi022 ceph-mon[31360]: pgmap v12297: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:28 smithi022 ceph-mon[40250]: pgmap v12297: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:29.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:29.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:29.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:30] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:48:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:30 smithi122 ceph-mon[32255]: pgmap v12298: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/482107763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:30 smithi022 ceph-mon[31360]: pgmap v12298: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/482107763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:30 smithi022 ceph-mon[40250]: pgmap v12298: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/482107763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:31 smithi122 ceph-mon[32255]: pgmap v12299: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:31 smithi022 ceph-mon[31360]: pgmap v12299: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:31 smithi022 ceph-mon[40250]: pgmap v12299: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:48:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:48:34.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:48:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:48:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:34 smithi122 ceph-mon[32255]: pgmap v12300: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:48:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:48:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:48:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:48:34.954 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:34.955 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[31360]: pgmap v12300: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:34.955 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:48:34.956 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:48:34.956 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:48:34.956 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:48:34.957 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[40250]: pgmap v12300: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:34.957 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:48:34.958 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:48:34.958 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:48:34.958 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:48:35.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:35.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:35.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/842153239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/842153239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:35.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:35.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/842153239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:36 smithi122 ceph-mon[32255]: pgmap v12301: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:36 smithi022 ceph-mon[31360]: pgmap v12301: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:36 smithi022 ceph-mon[40250]: pgmap v12301: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:48:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:38 smithi122 ceph-mon[32255]: pgmap v12302: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:38 smithi022 ceph-mon[31360]: pgmap v12302: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:38 smithi022 ceph-mon[40250]: pgmap v12302: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:40] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:48:40.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:40.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:40.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:40 smithi122 ceph-mon[32255]: pgmap v12303: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:40 smithi022 ceph-mon[31360]: pgmap v12303: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:40 smithi022 ceph-mon[40250]: pgmap v12303: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1428443714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1428443714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1428443714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:42.581 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:48:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:48:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:42 smithi122 ceph-mon[32255]: pgmap v12304: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:42 smithi022 ceph-mon[31360]: pgmap v12304: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:42 smithi022 ceph-mon[40250]: pgmap v12304: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:44.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:48:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:48:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:44 smithi122 ceph-mon[32255]: pgmap v12305: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:44 smithi022 ceph-mon[31360]: pgmap v12305: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:44 smithi022 ceph-mon[40250]: pgmap v12305: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:45.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:46.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:46.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:46.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:46.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:46.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:46 smithi122 ceph-mon[32255]: pgmap v12306: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3143698899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:46 smithi022 ceph-mon[31360]: pgmap v12306: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3143698899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:46 smithi022 ceph-mon[40250]: pgmap v12306: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3143698899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:48:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:48 smithi122 ceph-mon[32255]: pgmap v12307: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:48:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:48:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:48:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:48:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[31360]: pgmap v12307: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:48:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:48:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:48:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:48:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[40250]: pgmap v12307: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:48:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:48:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:48:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:48:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:50] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:48:50.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:50 smithi122 ceph-mon[32255]: pgmap v12308: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:50 smithi022 ceph-mon[31360]: pgmap v12308: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:50 smithi022 ceph-mon[40250]: pgmap v12308: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:51.071 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:51.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:51.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:51.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1200095007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:51 smithi122 ceph-mon[32255]: pgmap v12309: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1200095007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:51 smithi022 ceph-mon[31360]: pgmap v12309: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1200095007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:51 smithi022 ceph-mon[40250]: pgmap v12309: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:48:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:48:54.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:48:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:48:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:54 smithi122 ceph-mon[32255]: pgmap v12310: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:54 smithi022 ceph-mon[31360]: pgmap v12310: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:55.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:54 smithi022 ceph-mon[40250]: pgmap v12310: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:48:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:48:56.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:48:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:56 smithi022 ceph-mon[31360]: pgmap v12311: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:56 smithi022 ceph-mon[40250]: pgmap v12311: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:56.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:48:56.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:48:56.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:48:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:56 smithi122 ceph-mon[32255]: pgmap v12311: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3906607668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:48:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3906607668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3906607668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:48:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:48:58 smithi122 ceph-mon[32255]: pgmap v12312: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:48:58 smithi022 ceph-mon[31360]: pgmap v12312: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:48:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:48:58 smithi022 ceph-mon[40250]: pgmap v12312: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:00] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:49:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:00 smithi122 ceph-mon[32255]: pgmap v12313: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:00 smithi022 ceph-mon[31360]: pgmap v12313: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:00 smithi022 ceph-mon[40250]: pgmap v12313: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:01.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:02.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:02.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:02.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:02.184 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:01 smithi022 ceph-mon[31360]: pgmap v12314: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:02.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:01 smithi022 ceph-mon[40250]: pgmap v12314: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:01 smithi122 ceph-mon[32255]: pgmap v12314: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:49:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:49:03.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1531680549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1531680549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1531680549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:03 smithi122 ceph-mon[32255]: pgmap v12315: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:04.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:49:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:49:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:03 smithi022 ceph-mon[31360]: pgmap v12315: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:03 smithi022 ceph-mon[40250]: pgmap v12315: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:05 smithi122 ceph-mon[32255]: pgmap v12316: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:05 smithi022 ceph-mon[31360]: pgmap v12316: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:05 smithi022 ceph-mon[40250]: pgmap v12316: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:07.191 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:49:07.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:07.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:07.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2380049238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2380049238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2380049238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:08 smithi122 ceph-mon[32255]: pgmap v12317: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:08 smithi022 ceph-mon[31360]: pgmap v12317: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:08 smithi022 ceph-mon[40250]: pgmap v12317: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:10] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T00:49:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:10 smithi122 ceph-mon[32255]: pgmap v12318: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:10 smithi022 ceph-mon[31360]: pgmap v12318: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:10 smithi022 ceph-mon[40250]: pgmap v12318: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:11 smithi122 ceph-mon[32255]: pgmap v12319: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:12.297 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:11 smithi022 ceph-mon[31360]: pgmap v12319: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:12.298 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:11 smithi022 ceph-mon[40250]: pgmap v12319: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:12.557 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:49:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:49:12.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:12.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:12.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1071774024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1071774024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1071774024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:14.103 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:13 smithi122 ceph-mon[32255]: pgmap v12320: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:14.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:49:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:49:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:13 smithi022 ceph-mon[31360]: pgmap v12320: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:13 smithi022 ceph-mon[40250]: pgmap v12320: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:16.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:15 smithi022 ceph-mon[31360]: pgmap v12321: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:16.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:15 smithi022 ceph-mon[40250]: pgmap v12321: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:16.289 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:15 smithi122 ceph-mon[32255]: pgmap v12321: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:49:17.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:18.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:18.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:18.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:18 smithi122 ceph-mon[32255]: pgmap v12322: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/269836643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:18 smithi022 ceph-mon[31360]: pgmap v12322: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/269836643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:18 smithi022 ceph-mon[40250]: pgmap v12322: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/269836643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:20] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:49:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:20 smithi122 ceph-mon[32255]: pgmap v12323: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:21.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:20 smithi022 ceph-mon[31360]: pgmap v12323: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:21.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:20 smithi022 ceph-mon[40250]: pgmap v12323: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:21 smithi122 ceph-mon[32255]: pgmap v12324: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:21 smithi022 ceph-mon[31360]: pgmap v12324: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:21 smithi022 ceph-mon[40250]: pgmap v12324: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:49:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:49:23.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:23.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:23.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:23.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:24.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:49:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:49:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:24 smithi122 ceph-mon[32255]: pgmap v12325: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2540125296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:24 smithi022 ceph-mon[31360]: pgmap v12325: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2540125296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:24 smithi022 ceph-mon[40250]: pgmap v12325: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2540125296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:26.905 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:26 smithi122 ceph-mon[32255]: pgmap v12326: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:26 smithi022 ceph-mon[31360]: pgmap v12326: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:26 smithi022 ceph-mon[40250]: pgmap v12326: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:49:28.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:28 smithi122 ceph-mon[32255]: pgmap v12327: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:28 smithi022 ceph-mon[31360]: pgmap v12327: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:28 smithi022 ceph-mon[40250]: pgmap v12327: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1051805587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1051805587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1051805587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:30] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:49:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:30 smithi122 ceph-mon[32255]: pgmap v12328: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:30 smithi022 ceph-mon[31360]: pgmap v12328: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:30 smithi022 ceph-mon[40250]: pgmap v12328: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:31 smithi122 ceph-mon[32255]: pgmap v12329: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:31 smithi022 ceph-mon[31360]: pgmap v12329: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:31 smithi022 ceph-mon[40250]: pgmap v12329: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:49:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:49:34.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:34.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:49:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:49:34.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:34.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:34.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:34.790 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[40250]: pgmap v12330: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:34.791 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:49:34.791 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:49:34.791 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:49:34.791 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:49:34.792 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/494157263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:34.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[31360]: pgmap v12330: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:34.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:49:34.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:49:34.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:49:34.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:49:34.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/494157263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:34 smithi122 ceph-mon[32255]: pgmap v12330: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:49:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:49:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:49:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:49:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/494157263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:36 smithi122 ceph-mon[32255]: pgmap v12331: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:36 smithi022 ceph-mon[31360]: pgmap v12331: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:36 smithi022 ceph-mon[40250]: pgmap v12331: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:49:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:38 smithi122 ceph-mon[32255]: pgmap v12332: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:38 smithi022 ceph-mon[31360]: pgmap v12332: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:38 smithi022 ceph-mon[40250]: pgmap v12332: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:39.407 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:39.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:39.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:39.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:40] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T00:49:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:40 smithi122 ceph-mon[32255]: pgmap v12333: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/43987725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:41.039 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:40 smithi022 ceph-mon[31360]: pgmap v12333: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:41.039 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/43987725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:41.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:40 smithi022 ceph-mon[40250]: pgmap v12333: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:41.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/43987725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:41 smithi122 ceph-mon[32255]: pgmap v12334: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:41 smithi022 ceph-mon[31360]: pgmap v12334: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:41 smithi022 ceph-mon[40250]: pgmap v12334: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:49:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:49:44.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:49:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:49:44.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:44 smithi122 ceph-mon[32255]: pgmap v12335: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:44 smithi022 ceph-mon[31360]: pgmap v12335: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:44 smithi022 ceph-mon[40250]: pgmap v12335: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:45.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:45.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:45.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/817448926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:46.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:46.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/817448926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/817448926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:46.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:46 smithi122 ceph-mon[32255]: pgmap v12336: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:46 smithi022 ceph-mon[31360]: pgmap v12336: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:46 smithi022 ceph-mon[40250]: pgmap v12336: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:49:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:48 smithi122 ceph-mon[32255]: pgmap v12337: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:48.947 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:49:48.947 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:49:48.947 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:49:49.008 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[40250]: pgmap v12337: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:49.008 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:49:49.008 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:49:49.009 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:49:49.009 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[31360]: pgmap v12337: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:49.009 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:49:49.009 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:49:49.010 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:49:50.151 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:50.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:49:50.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:50 smithi022 ceph-mon[31360]: pgmap v12338: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:50.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:49:50.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:50 smithi022 ceph-mon[40250]: pgmap v12338: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:50.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:49:50.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:49:50.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:50 smithi122 ceph-mon[32255]: pgmap v12338: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:50.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:50.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:50.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1894289156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1894289156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1894289156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:52.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:52 smithi022 ceph-mon[31360]: pgmap v12339: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:52.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:52 smithi022 ceph-mon[40250]: pgmap v12339: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:52 smithi122 ceph-mon[32255]: pgmap v12339: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:49:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:49:54.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:49:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:49:54.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:54 smithi122 ceph-mon[32255]: pgmap v12340: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:54 smithi022 ceph-mon[31360]: pgmap v12340: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:55.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:54 smithi022 ceph-mon[40250]: pgmap v12340: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:49:55.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:49:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:55.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:49:55.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:49:55.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:49:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:49:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:56 smithi122 ceph-mon[32255]: pgmap v12341: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2084828435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:56 smithi022 ceph-mon[31360]: pgmap v12341: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2084828435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:56 smithi022 ceph-mon[40250]: pgmap v12341: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2084828435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:49:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:49:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:49:58 smithi122 ceph-mon[32255]: pgmap v12342: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:49:58 smithi022 ceph-mon[31360]: pgmap v12342: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:49:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:49:58 smithi022 ceph-mon[40250]: pgmap v12342: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:50:00.894 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:00.894 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:00 smithi022 ceph-mon[40250]: pgmap v12343: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:00.895 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T00:50:00.895 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:00 smithi022 ceph-mon[31360]: pgmap v12343: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:00.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T00:50:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:00 smithi122 ceph-mon[32255]: pgmap v12343: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T00:50:01.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:01.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:01.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1359132885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:01 smithi122 ceph-mon[32255]: pgmap v12344: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1359132885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:01 smithi022 ceph-mon[31360]: pgmap v12344: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1359132885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:01 smithi022 ceph-mon[40250]: pgmap v12344: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:50:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:50:04.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:50:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:50:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:04 smithi122 ceph-mon[32255]: pgmap v12345: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:04 smithi022 ceph-mon[31360]: pgmap v12345: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:04 smithi022 ceph-mon[40250]: pgmap v12345: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:06.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:06.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:06.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:06.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:06 smithi122 ceph-mon[32255]: pgmap v12346: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:06 smithi022 ceph-mon[31360]: pgmap v12346: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:06 smithi022 ceph-mon[40250]: pgmap v12346: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1791647881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:50:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1791647881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:07.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1791647881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:08 smithi122 ceph-mon[32255]: pgmap v12347: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:08 smithi022 ceph-mon[31360]: pgmap v12347: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:08 smithi022 ceph-mon[40250]: pgmap v12347: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:10] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T00:50:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:10 smithi122 ceph-mon[32255]: pgmap v12348: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:10 smithi022 ceph-mon[31360]: pgmap v12348: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:10 smithi022 ceph-mon[40250]: pgmap v12348: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:11.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:12.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:12.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:12.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:12.009 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:11 smithi022 ceph-mon[40250]: pgmap v12349: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:12.009 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:11 smithi022 ceph-mon[31360]: pgmap v12349: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:12.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:11 smithi122 ceph-mon[32255]: pgmap v12349: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:50:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:50:13.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/666324607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/666324607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/666324607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:13 smithi122 ceph-mon[32255]: pgmap v12350: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:14.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:50:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:50:14.307 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:13 smithi022 ceph-mon[31360]: pgmap v12350: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:14.308 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:13 smithi022 ceph-mon[40250]: pgmap v12350: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:15 smithi122 ceph-mon[32255]: pgmap v12351: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:15 smithi022 ceph-mon[31360]: pgmap v12351: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:15 smithi022 ceph-mon[40250]: pgmap v12351: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:17.008 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:17.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:17.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:17.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3176837357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3176837357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:50:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3176837357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:18.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:18 smithi122 ceph-mon[32255]: pgmap v12352: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:18 smithi022 ceph-mon[31360]: pgmap v12352: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:18 smithi022 ceph-mon[40250]: pgmap v12352: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:20] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T00:50:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:20 smithi122 ceph-mon[32255]: pgmap v12353: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:20 smithi022 ceph-mon[31360]: pgmap v12353: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:20 smithi022 ceph-mon[40250]: pgmap v12353: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:21 smithi122 ceph-mon[32255]: pgmap v12354: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:21 smithi022 ceph-mon[31360]: pgmap v12354: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:21 smithi022 ceph-mon[40250]: pgmap v12354: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:22.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:50:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:50:22.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:22.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:22.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1745759935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1745759935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1745759935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:24.209 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:50:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:50:24.209 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:23 smithi122 ceph-mon[32255]: pgmap v12355: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:23 smithi022 ceph-mon[31360]: pgmap v12355: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:23 smithi022 ceph-mon[40250]: pgmap v12355: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:25 smithi122 ceph-mon[32255]: pgmap v12356: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:25 smithi022 ceph-mon[31360]: pgmap v12356: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:25 smithi022 ceph-mon[40250]: pgmap v12356: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:27.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:27.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:50:28.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:28.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:28.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:28 smithi122 ceph-mon[32255]: pgmap v12357: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1343878173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:28 smithi022 ceph-mon[31360]: pgmap v12357: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1343878173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:28 smithi022 ceph-mon[40250]: pgmap v12357: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1343878173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:30] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T00:50:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:30 smithi122 ceph-mon[32255]: pgmap v12358: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:30 smithi022 ceph-mon[31360]: pgmap v12358: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:30 smithi022 ceph-mon[40250]: pgmap v12358: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:31 smithi122 ceph-mon[32255]: pgmap v12359: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:31 smithi022 ceph-mon[31360]: pgmap v12359: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:31 smithi022 ceph-mon[40250]: pgmap v12359: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:50:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:50:33.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:33.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:33.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:33.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4014139089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4014139089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:33.856 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4014139089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:34.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:50:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:50:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:34 smithi122 ceph-mon[32255]: pgmap v12360: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:50:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:50:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:50:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:50:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[31360]: pgmap v12360: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:50:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:50:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:50:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:50:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[40250]: pgmap v12360: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:50:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:50:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:50:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:50:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:36 smithi122 ceph-mon[32255]: pgmap v12361: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:36 smithi022 ceph-mon[31360]: pgmap v12361: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:36 smithi022 ceph-mon[40250]: pgmap v12361: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:50:38.490 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:38 smithi022 ceph-mon[31360]: pgmap v12362: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:38 smithi022 ceph-mon[40250]: pgmap v12362: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:38.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:38.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:38.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:38 smithi122 ceph-mon[32255]: pgmap v12362: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3657785933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3657785933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3657785933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:40] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:50:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:40 smithi122 ceph-mon[32255]: pgmap v12363: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:40 smithi022 ceph-mon[31360]: pgmap v12363: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:40 smithi022 ceph-mon[40250]: pgmap v12363: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:41 smithi122 ceph-mon[32255]: pgmap v12364: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:41 smithi022 ceph-mon[31360]: pgmap v12364: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:41 smithi022 ceph-mon[40250]: pgmap v12364: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:50:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:50:43.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:44.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:50:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:50:44.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:44.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:44.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:44 smithi122 ceph-mon[32255]: pgmap v12365: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4153144829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:44 smithi022 ceph-mon[31360]: pgmap v12365: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4153144829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:44 smithi022 ceph-mon[40250]: pgmap v12365: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4153144829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:46 smithi122 ceph-mon[32255]: pgmap v12366: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:46 smithi022 ceph-mon[31360]: pgmap v12366: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:46 smithi022 ceph-mon[40250]: pgmap v12366: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:50:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:48 smithi122 ceph-mon[32255]: pgmap v12367: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:48 smithi022 ceph-mon[31360]: pgmap v12367: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:48 smithi022 ceph-mon[40250]: pgmap v12367: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:49.232 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:49.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:49.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:49.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:50:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:50:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:50:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:50:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508718208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:50:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:50:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:50:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:50:49.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508718208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:50:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:50:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:50:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:50:49.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508718208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:50] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:50:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:50 smithi122 ceph-mon[32255]: pgmap v12368: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:50 smithi022 ceph-mon[31360]: pgmap v12368: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:50 smithi022 ceph-mon[40250]: pgmap v12368: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:51 smithi122 ceph-mon[32255]: pgmap v12369: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:51 smithi022 ceph-mon[31360]: pgmap v12369: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:51 smithi022 ceph-mon[40250]: pgmap v12369: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:50:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:50:54.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:50:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:50:54.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:50:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:54 smithi122 ceph-mon[32255]: pgmap v12370: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:54.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:50:54.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:50:54.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:50:54.975 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:54 smithi022 ceph-mon[31360]: pgmap v12370: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:54.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:54 smithi022 ceph-mon[40250]: pgmap v12370: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:50:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/803547171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/803547171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:50:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/803547171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:50:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:56 smithi122 ceph-mon[32255]: pgmap v12371: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:56 smithi022 ceph-mon[31360]: pgmap v12371: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:56 smithi022 ceph-mon[40250]: pgmap v12371: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:50:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:50:58 smithi122 ceph-mon[32255]: pgmap v12372: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:50:58 smithi022 ceph-mon[31360]: pgmap v12372: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:50:58 smithi022 ceph-mon[40250]: pgmap v12372: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:50:59.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:00.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:00.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:00.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:00] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:51:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:00 smithi122 ceph-mon[32255]: pgmap v12373: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1693760841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:00 smithi022 ceph-mon[31360]: pgmap v12373: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1693760841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:00 smithi022 ceph-mon[40250]: pgmap v12373: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1693760841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:01 smithi122 ceph-mon[32255]: pgmap v12374: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:01 smithi022 ceph-mon[31360]: pgmap v12374: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:01 smithi022 ceph-mon[40250]: pgmap v12374: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:51:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:51:04.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:51:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:51:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:04 smithi122 ceph-mon[32255]: pgmap v12375: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:05.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:04 smithi022 ceph-mon[31360]: pgmap v12375: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:05.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:04 smithi022 ceph-mon[40250]: pgmap v12375: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:05.616 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:05.616 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:05.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:05.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:05.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:06 smithi122 ceph-mon[32255]: pgmap v12376: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1162654872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:06 smithi022 ceph-mon[31360]: pgmap v12376: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1162654872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:06 smithi022 ceph-mon[40250]: pgmap v12376: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1162654872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:51:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:08 smithi122 ceph-mon[32255]: pgmap v12377: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:08 smithi022 ceph-mon[31360]: pgmap v12377: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:08 smithi022 ceph-mon[40250]: pgmap v12377: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:10] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:51:10.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:10.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:10 smithi122 ceph-mon[32255]: pgmap v12378: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:10 smithi022 ceph-mon[31360]: pgmap v12378: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:10 smithi022 ceph-mon[40250]: pgmap v12378: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:11.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:11.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/693574388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:11 smithi122 ceph-mon[32255]: pgmap v12379: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/693574388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:11 smithi022 ceph-mon[31360]: pgmap v12379: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/693574388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:11 smithi022 ceph-mon[40250]: pgmap v12379: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:51:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:51:14.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:51:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:51:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:14 smithi122 ceph-mon[32255]: pgmap v12380: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:14 smithi022 ceph-mon[31360]: pgmap v12380: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:14 smithi022 ceph-mon[40250]: pgmap v12380: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:16.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:16.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:16.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:16.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:16 smithi022 ceph-mon[31360]: pgmap v12381: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2949940322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:16 smithi022 ceph-mon[40250]: pgmap v12381: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2949940322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:16 smithi122 ceph-mon[32255]: pgmap v12381: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2949940322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:51:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:18 smithi122 ceph-mon[32255]: pgmap v12382: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:18 smithi022 ceph-mon[31360]: pgmap v12382: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:18 smithi022 ceph-mon[40250]: pgmap v12382: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:51:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:20 smithi122 ceph-mon[32255]: pgmap v12383: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:20 smithi022 ceph-mon[31360]: pgmap v12383: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:20 smithi022 ceph-mon[40250]: pgmap v12383: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:21.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:21.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:21.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:21.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:21 smithi122 ceph-mon[32255]: pgmap v12384: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2498599954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:21 smithi022 ceph-mon[31360]: pgmap v12384: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2498599954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:21 smithi022 ceph-mon[40250]: pgmap v12384: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2498599954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:51:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:51:24.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:51:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:51:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:24 smithi122 ceph-mon[32255]: pgmap v12385: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:24 smithi022 ceph-mon[31360]: pgmap v12385: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:24 smithi022 ceph-mon[40250]: pgmap v12385: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:26.832 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:26 smithi122 ceph-mon[32255]: pgmap v12386: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:26 smithi022 ceph-mon[31360]: pgmap v12386: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:26 smithi022 ceph-mon[40250]: pgmap v12386: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:27.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:27.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:27.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:51:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3544849394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3544849394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3544849394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:28 smithi122 ceph-mon[32255]: pgmap v12387: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:28 smithi022 ceph-mon[40250]: pgmap v12387: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:28 smithi022 ceph-mon[31360]: pgmap v12387: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:51:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:30 smithi122 ceph-mon[32255]: pgmap v12388: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:30 smithi022 ceph-mon[31360]: pgmap v12388: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:30 smithi022 ceph-mon[40250]: pgmap v12388: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:31 smithi122 ceph-mon[32255]: pgmap v12389: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:32.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:32.206 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:31 smithi022 ceph-mon[31360]: pgmap v12389: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:32.208 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:31 smithi022 ceph-mon[40250]: pgmap v12389: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:32.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:32.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:32.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:51:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:51:33.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1755223056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1755223056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:33.307 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1755223056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:34.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:51:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:51:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:33 smithi122 ceph-mon[32255]: pgmap v12390: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:51:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:51:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:51:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:51:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[31360]: pgmap v12390: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:51:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:51:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:51:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:51:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[40250]: pgmap v12390: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:51:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:51:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:51:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:51:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:35 smithi122 ceph-mon[32255]: pgmap v12391: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:35 smithi022 ceph-mon[31360]: pgmap v12391: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:35 smithi022 ceph-mon[40250]: pgmap v12391: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:37.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:51:37.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:37.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:37.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:38 smithi122 ceph-mon[32255]: pgmap v12392: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/658393952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:38 smithi022 ceph-mon[40250]: pgmap v12392: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/658393952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:38 smithi022 ceph-mon[31360]: pgmap v12392: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/658393952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:40] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T00:51:40.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:40 smithi122 ceph-mon[32255]: pgmap v12393: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:40 smithi022 ceph-mon[31360]: pgmap v12393: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:40 smithi022 ceph-mon[40250]: pgmap v12393: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:41 smithi122 ceph-mon[32255]: pgmap v12394: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:41 smithi022 ceph-mon[31360]: pgmap v12394: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:41 smithi022 ceph-mon[40250]: pgmap v12394: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:51:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:51:42.949 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:43.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:43.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:43.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1626361844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1626361844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1626361844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:44.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:51:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:51:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:44 smithi122 ceph-mon[32255]: pgmap v12395: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:44 smithi022 ceph-mon[31360]: pgmap v12395: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:44.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:44 smithi022 ceph-mon[40250]: pgmap v12395: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:46 smithi122 ceph-mon[32255]: pgmap v12396: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:46 smithi022 ceph-mon[31360]: pgmap v12396: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:46 smithi022 ceph-mon[40250]: pgmap v12396: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:51:48.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:48.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:48.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:48 smithi122 ceph-mon[32255]: pgmap v12397: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:48 smithi022 ceph-mon[31360]: pgmap v12397: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:48 smithi022 ceph-mon[40250]: pgmap v12397: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:49.892 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3671022022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:49.892 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:51:49.892 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:51:49.893 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:51:49.901 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3671022022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:49.901 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:51:49.901 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:51:49.902 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:51:49.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3671022022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:49.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:51:49.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:51:49.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:51:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:50] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:51:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:50 smithi122 ceph-mon[32255]: pgmap v12398: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[31360]: pgmap v12398: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[40250]: pgmap v12398: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:51:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:51 smithi122 ceph-mon[32255]: pgmap v12399: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:51 smithi022 ceph-mon[31360]: pgmap v12399: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:51 smithi022 ceph-mon[40250]: pgmap v12399: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:51:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:51:53.696 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:54.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:54.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:54.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:54.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:51:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:51:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:54 smithi122 ceph-mon[32255]: pgmap v12400: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1436422013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:54 smithi022 ceph-mon[31360]: pgmap v12400: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1436422013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:54 smithi022 ceph-mon[40250]: pgmap v12400: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:51:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1436422013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:55.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:51:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:56 smithi122 ceph-mon[32255]: pgmap v12401: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:56 smithi022 ceph-mon[31360]: pgmap v12401: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:56 smithi022 ceph-mon[40250]: pgmap v12401: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:51:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:58 smithi122 ceph-mon[32255]: pgmap v12402: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:59.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:58 smithi022 ceph-mon[31360]: pgmap v12402: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:59.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:58 smithi022 ceph-mon[40250]: pgmap v12402: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:51:59.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:51:59.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:51:59.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:51:59.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:51:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:51:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1738546209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:51:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1738546209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:51:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:51:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1738546209' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:00] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:52:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:00 smithi122 ceph-mon[32255]: pgmap v12403: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:00 smithi022 ceph-mon[31360]: pgmap v12403: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:00 smithi022 ceph-mon[40250]: pgmap v12403: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:01 smithi122 ceph-mon[32255]: pgmap v12404: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:01 smithi022 ceph-mon[31360]: pgmap v12404: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:01 smithi022 ceph-mon[40250]: pgmap v12404: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:52:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:52:04.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:52:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:52:04.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:04 smithi022 ceph-mon[31360]: pgmap v12405: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:04 smithi022 ceph-mon[40250]: pgmap v12405: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:04.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:04.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:04.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:04 smithi122 ceph-mon[32255]: pgmap v12405: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4111698780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4111698780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4111698780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:06 smithi122 ceph-mon[32255]: pgmap v12406: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:06 smithi022 ceph-mon[31360]: pgmap v12406: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:06 smithi022 ceph-mon[40250]: pgmap v12406: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:52:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:08 smithi122 ceph-mon[32255]: pgmap v12407: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:08 smithi022 ceph-mon[31360]: pgmap v12407: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:08 smithi022 ceph-mon[40250]: pgmap v12407: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:09.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:10.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:10.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:10.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:10] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:52:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:10 smithi122 ceph-mon[32255]: pgmap v12408: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3740428073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:10 smithi022 ceph-mon[31360]: pgmap v12408: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3740428073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:10 smithi022 ceph-mon[40250]: pgmap v12408: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3740428073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:11 smithi122 ceph-mon[32255]: pgmap v12409: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:11 smithi022 ceph-mon[31360]: pgmap v12409: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:11 smithi022 ceph-mon[40250]: pgmap v12409: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:52:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:52:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:52:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:52:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:14 smithi122 ceph-mon[32255]: pgmap v12410: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:14 smithi022 ceph-mon[31360]: pgmap v12410: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:14 smithi022 ceph-mon[40250]: pgmap v12410: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:15.182 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:15.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3948182425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3948182425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3948182425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:16 smithi122 ceph-mon[32255]: pgmap v12411: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:16 smithi022 ceph-mon[31360]: pgmap v12411: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:16 smithi022 ceph-mon[40250]: pgmap v12411: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:52:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:18 smithi122 ceph-mon[32255]: pgmap v12412: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:18 smithi022 ceph-mon[31360]: pgmap v12412: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:18 smithi022 ceph-mon[40250]: pgmap v12412: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:20] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T00:52:20.550 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:20.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:20.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:20.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:20.925 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:20 smithi022 ceph-mon[40250]: pgmap v12413: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:20.926 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:20 smithi022 ceph-mon[31360]: pgmap v12413: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:20 smithi122 ceph-mon[32255]: pgmap v12413: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1516503750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:21 smithi122 ceph-mon[32255]: pgmap v12414: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1516503750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:21 smithi022 ceph-mon[31360]: pgmap v12414: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1516503750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:21 smithi022 ceph-mon[40250]: pgmap v12414: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:22.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:52:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:52:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:52:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:52:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:24 smithi122 ceph-mon[32255]: pgmap v12415: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:24 smithi022 ceph-mon[31360]: pgmap v12415: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:25.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:24 smithi022 ceph-mon[40250]: pgmap v12415: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:25.924 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:25.925 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:25.926 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:26.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:26.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:26.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:26 smithi122 ceph-mon[32255]: pgmap v12416: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1873775636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:26 smithi022 ceph-mon[31360]: pgmap v12416: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1873775636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:26 smithi022 ceph-mon[40250]: pgmap v12416: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1873775636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:52:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:28 smithi122 ceph-mon[32255]: pgmap v12417: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:28 smithi022 ceph-mon[31360]: pgmap v12417: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:28 smithi022 ceph-mon[40250]: pgmap v12417: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:30] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T00:52:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:30 smithi122 ceph-mon[32255]: pgmap v12418: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:30 smithi022 ceph-mon[31360]: pgmap v12418: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:30 smithi022 ceph-mon[40250]: pgmap v12418: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:31.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:31.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:31.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:31 smithi022 ceph-mon[31360]: pgmap v12419: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2757306656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:31 smithi022 ceph-mon[40250]: pgmap v12419: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2757306656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:31 smithi122 ceph-mon[32255]: pgmap v12419: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2757306656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:52:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:52:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:52:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:52:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:34 smithi122 ceph-mon[32255]: pgmap v12420: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:52:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:52:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:52:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:52:35.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[31360]: pgmap v12420: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:35.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:52:35.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:52:35.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:52:35.037 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:52:35.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[40250]: pgmap v12420: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:35.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:52:35.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:52:35.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:52:35.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:52:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:36 smithi122 ceph-mon[32255]: pgmap v12421: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:36 smithi022 ceph-mon[31360]: pgmap v12421: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:36 smithi022 ceph-mon[40250]: pgmap v12421: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:37.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:37.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:37.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:37.649 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:52:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2945438554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2945438554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2945438554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:38 smithi122 ceph-mon[32255]: pgmap v12422: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:38 smithi022 ceph-mon[31360]: pgmap v12422: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:38 smithi022 ceph-mon[40250]: pgmap v12422: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:40] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T00:52:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:40 smithi122 ceph-mon[32255]: pgmap v12423: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:40 smithi022 ceph-mon[31360]: pgmap v12423: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:40 smithi022 ceph-mon[40250]: pgmap v12423: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:42.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:41 smithi122 ceph-mon[32255]: pgmap v12424: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:42.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:41 smithi022 ceph-mon[31360]: pgmap v12424: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:42.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:41 smithi022 ceph-mon[40250]: pgmap v12424: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:42.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:42.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:42.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:52:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:52:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4152950193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4152950193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4152950193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:43 smithi122 ceph-mon[32255]: pgmap v12425: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:52:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:52:44.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:43 smithi022 ceph-mon[31360]: pgmap v12425: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:44.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:43 smithi022 ceph-mon[40250]: pgmap v12425: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:45.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:45 smithi122 ceph-mon[32255]: pgmap v12426: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:45 smithi022 ceph-mon[31360]: pgmap v12426: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:45 smithi022 ceph-mon[40250]: pgmap v12426: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:47.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:47.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:47.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:47.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:47.774 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:52:48.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:48 smithi122 ceph-mon[32255]: pgmap v12427: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2549748437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:48 smithi022 ceph-mon[31360]: pgmap v12427: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2549748437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:48 smithi022 ceph-mon[40250]: pgmap v12427: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2549748437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:50] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:52:50.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[31360]: pgmap v12428: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:50.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:52:50.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:52:50.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:52:50.650 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[40250]: pgmap v12428: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:50.651 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:52:50.651 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:52:50.651 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:52:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:50 smithi122 ceph-mon[32255]: pgmap v12428: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:52:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:52:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:52:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:52:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:51 smithi122 ceph-mon[32255]: pgmap v12429: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:52:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:51 smithi022 ceph-mon[31360]: pgmap v12429: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:52:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:51 smithi022 ceph-mon[40250]: pgmap v12429: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:52:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:52:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:53.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:53.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:53.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/620343856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/620343856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/620343856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:54 smithi122 ceph-mon[32255]: pgmap v12430: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:52:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:52:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:54 smithi022 ceph-mon[31360]: pgmap v12430: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:54 smithi022 ceph-mon[40250]: pgmap v12430: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:52:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:52:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:56 smithi022 ceph-mon[31360]: pgmap v12431: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:56 smithi022 ceph-mon[40250]: pgmap v12431: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:56 smithi122 ceph-mon[32255]: pgmap v12431: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:52:58.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:52:58.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:52:58.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:52:58.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:52:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:58 smithi022 ceph-mon[31360]: pgmap v12432: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:52:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1362476982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:58 smithi022 ceph-mon[40250]: pgmap v12432: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:52:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1362476982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:52:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:58 smithi122 ceph-mon[32255]: pgmap v12432: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:52:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:52:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1362476982' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:00] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T00:53:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:00 smithi122 ceph-mon[32255]: pgmap v12433: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:00 smithi022 ceph-mon[31360]: pgmap v12433: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:00 smithi022 ceph-mon[40250]: pgmap v12433: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:02.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:01 smithi122 ceph-mon[32255]: pgmap v12434: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:01 smithi022 ceph-mon[31360]: pgmap v12434: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:01 smithi022 ceph-mon[40250]: pgmap v12434: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:53:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:53:03.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:03.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:03.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:03.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:04.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:53:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:53:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:04 smithi122 ceph-mon[32255]: pgmap v12435: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/740508869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:04 smithi022 ceph-mon[31360]: pgmap v12435: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/740508869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:04 smithi022 ceph-mon[40250]: pgmap v12435: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/740508869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:06 smithi122 ceph-mon[32255]: pgmap v12436: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:06 smithi022 ceph-mon[31360]: pgmap v12436: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:06 smithi022 ceph-mon[40250]: pgmap v12436: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:53:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:08 smithi122 ceph-mon[32255]: pgmap v12437: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:08 smithi022 ceph-mon[31360]: pgmap v12437: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:08 smithi022 ceph-mon[40250]: pgmap v12437: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:09.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:09.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:09.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1605632407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1605632407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1605632407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:10] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:53:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:10 smithi122 ceph-mon[32255]: pgmap v12438: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:10 smithi022 ceph-mon[31360]: pgmap v12438: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:10 smithi022 ceph-mon[40250]: pgmap v12438: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:11 smithi122 ceph-mon[32255]: pgmap v12439: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:11 smithi022 ceph-mon[31360]: pgmap v12439: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:11 smithi022 ceph-mon[40250]: pgmap v12439: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:53:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:53:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:53:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:53:14.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:14.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:14.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:14 smithi122 ceph-mon[32255]: pgmap v12440: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1058269513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:14 smithi022 ceph-mon[31360]: pgmap v12440: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1058269513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:14 smithi022 ceph-mon[40250]: pgmap v12440: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1058269513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:16 smithi122 ceph-mon[32255]: pgmap v12441: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:16 smithi022 ceph-mon[31360]: pgmap v12441: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:16 smithi022 ceph-mon[40250]: pgmap v12441: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:53:18.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:18 smithi122 ceph-mon[32255]: pgmap v12442: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:18 smithi022 ceph-mon[31360]: pgmap v12442: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:18 smithi022 ceph-mon[40250]: pgmap v12442: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:19.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:19.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:19.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:19.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:53:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:20 smithi122 ceph-mon[32255]: pgmap v12443: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/89163397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:20 smithi022 ceph-mon[31360]: pgmap v12443: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/89163397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:20 smithi022 ceph-mon[40250]: pgmap v12443: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/89163397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:21 smithi122 ceph-mon[32255]: pgmap v12444: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:21 smithi022 ceph-mon[31360]: pgmap v12444: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:21 smithi022 ceph-mon[40250]: pgmap v12444: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:53:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:53:24.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:53:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:53:24.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:24 smithi122 ceph-mon[32255]: pgmap v12445: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:25.000 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:25.000 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:24 smithi022 ceph-mon[31360]: pgmap v12445: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:25.001 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:24 smithi022 ceph-mon[40250]: pgmap v12445: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:25.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:25.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:25.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:25.658 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:25.658 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2652092684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2652092684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2652092684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:26 smithi122 ceph-mon[32255]: pgmap v12446: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:26 smithi022 ceph-mon[31360]: pgmap v12446: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:26 smithi022 ceph-mon[40250]: pgmap v12446: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:27.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:53:28.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:28 smithi122 ceph-mon[32255]: pgmap v12447: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:28 smithi022 ceph-mon[31360]: pgmap v12447: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:28 smithi022 ceph-mon[40250]: pgmap v12447: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:30.371 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:53:30.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:30.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:30.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:30 smithi122 ceph-mon[32255]: pgmap v12448: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:30 smithi022 ceph-mon[31360]: pgmap v12448: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:30 smithi022 ceph-mon[40250]: pgmap v12448: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1839190965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1839190965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1839190965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:32.668 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:53:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:53:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:32 smithi122 ceph-mon[32255]: pgmap v12449: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:32 smithi022 ceph-mon[31360]: pgmap v12449: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:32 smithi022 ceph-mon[40250]: pgmap v12449: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:53:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:53:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:34 smithi122 ceph-mon[32255]: pgmap v12450: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:53:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:53:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:53:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:53:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[31360]: pgmap v12450: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:53:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:53:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:53:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:53:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[40250]: pgmap v12450: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:53:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:53:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:53:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:53:35.740 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:36.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:36.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:36.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:36 smithi122 ceph-mon[32255]: pgmap v12451: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/719161324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:36 smithi022 ceph-mon[31360]: pgmap v12451: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/719161324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:36 smithi022 ceph-mon[40250]: pgmap v12451: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/719161324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:53:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:38 smithi122 ceph-mon[32255]: pgmap v12452: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:38 smithi022 ceph-mon[31360]: pgmap v12452: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:38 smithi022 ceph-mon[40250]: pgmap v12452: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:40] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T00:53:40.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:40 smithi122 ceph-mon[32255]: pgmap v12453: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:40 smithi022 ceph-mon[31360]: pgmap v12453: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:40 smithi022 ceph-mon[40250]: pgmap v12453: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:41.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:41.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/339997174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:41 smithi122 ceph-mon[32255]: pgmap v12454: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/339997174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:41 smithi022 ceph-mon[31360]: pgmap v12454: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/339997174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:41 smithi022 ceph-mon[40250]: pgmap v12454: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:53:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:53:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:53:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:53:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:44 smithi122 ceph-mon[32255]: pgmap v12455: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:44 smithi022 ceph-mon[31360]: pgmap v12455: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:44 smithi022 ceph-mon[40250]: pgmap v12455: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:46.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:46 smithi022 ceph-mon[31360]: pgmap v12456: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:46 smithi022 ceph-mon[40250]: pgmap v12456: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:46.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:46.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:46.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:46 smithi122 ceph-mon[32255]: pgmap v12456: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:47.670 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:53:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2516757712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:48.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2516757712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:48.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2516757712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:48 smithi122 ceph-mon[32255]: pgmap v12457: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:48 smithi022 ceph-mon[31360]: pgmap v12457: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:48 smithi022 ceph-mon[40250]: pgmap v12457: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T00:53:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:50] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:53:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:50 smithi122 ceph-mon[32255]: pgmap v12458: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:50 smithi022 ceph-mon[31360]: pgmap v12458: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:50 smithi022 ceph-mon[40250]: pgmap v12458: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:51.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:53:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:53:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:53:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:53:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:53:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:53:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:53:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:53:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:53:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:53:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:53:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:53:52.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:52.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:52.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:52.678 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:53:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:53:52.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:52 smithi122 ceph-mon[32255]: pgmap v12459: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:52.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1149162458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:52 smithi022 ceph-mon[31360]: pgmap v12459: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1149162458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:52 smithi022 ceph-mon[40250]: pgmap v12459: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1149162458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:53:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:53:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:54 smithi122 ceph-mon[32255]: pgmap v12460: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:54 smithi022 ceph-mon[31360]: pgmap v12460: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:55.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:54 smithi022 ceph-mon[40250]: pgmap v12460: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:53:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:53:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:56 smithi022 ceph-mon[31360]: pgmap v12461: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:56 smithi022 ceph-mon[40250]: pgmap v12461: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:57.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:56 smithi122 ceph-mon[32255]: pgmap v12461: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:57.224 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:53:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:53:57.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:53:57.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:53:57.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:53:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1568671604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:58.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1568671604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1568671604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:53:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:53:58 smithi022 ceph-mon[31360]: pgmap v12462: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:53:58 smithi022 ceph-mon[40250]: pgmap v12462: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:53:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:53:58 smithi122 ceph-mon[32255]: pgmap v12462: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:00] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T00:54:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:00 smithi022 ceph-mon[31360]: pgmap v12463: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:00 smithi022 ceph-mon[40250]: pgmap v12463: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:00 smithi122 ceph-mon[32255]: pgmap v12463: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:01 smithi122 ceph-mon[32255]: pgmap v12464: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:01 smithi022 ceph-mon[31360]: pgmap v12464: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:01 smithi022 ceph-mon[40250]: pgmap v12464: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:54:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:54:02.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:02.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:02.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1023432664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1023432664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1023432664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:04 smithi022 ceph-mon[31360]: pgmap v12465: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:04 smithi022 ceph-mon[40250]: pgmap v12465: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:04.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:54:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:54:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:04 smithi122 ceph-mon[32255]: pgmap v12465: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:06 smithi022 ceph-mon[31360]: pgmap v12466: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:06 smithi022 ceph-mon[40250]: pgmap v12466: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:06 smithi122 ceph-mon[32255]: pgmap v12466: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:54:07.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:08.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:08.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:08.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:08.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:08 smithi022 ceph-mon[40250]: pgmap v12467: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:08.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/39151636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:08 smithi122 ceph-mon[32255]: pgmap v12467: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/39151636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:08 smithi022 ceph-mon[31360]: pgmap v12467: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/39151636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:10] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:54:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:10 smithi122 ceph-mon[32255]: pgmap v12468: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:10 smithi022 ceph-mon[31360]: pgmap v12468: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:10 smithi022 ceph-mon[40250]: pgmap v12468: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:11 smithi122 ceph-mon[32255]: pgmap v12469: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:11 smithi022 ceph-mon[31360]: pgmap v12469: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:11 smithi022 ceph-mon[40250]: pgmap v12469: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:54:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:54:13.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:13.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:13.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:13.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:14.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:54:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:54:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:14 smithi122 ceph-mon[32255]: pgmap v12470: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/259454515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:14 smithi022 ceph-mon[31360]: pgmap v12470: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/259454515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:14 smithi022 ceph-mon[40250]: pgmap v12470: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/259454515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:15.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:16 smithi122 ceph-mon[32255]: pgmap v12471: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:16 smithi022 ceph-mon[31360]: pgmap v12471: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:16 smithi022 ceph-mon[40250]: pgmap v12471: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:54:18.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:18 smithi122 ceph-mon[32255]: pgmap v12472: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:18 smithi022 ceph-mon[31360]: pgmap v12472: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:18 smithi022 ceph-mon[40250]: pgmap v12472: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:19.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:19.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:19.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1361965943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1361965943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1361965943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T00:54:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:20 smithi122 ceph-mon[32255]: pgmap v12473: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:20 smithi022 ceph-mon[31360]: pgmap v12473: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:20 smithi022 ceph-mon[40250]: pgmap v12473: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:21 smithi122 ceph-mon[32255]: pgmap v12474: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:21 smithi022 ceph-mon[31360]: pgmap v12474: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:21 smithi022 ceph-mon[40250]: pgmap v12474: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:54:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:54:24.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:24.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:54:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:54:24.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:24.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:24.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:24 smithi022 ceph-mon[31360]: pgmap v12475: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1231006855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:24 smithi022 ceph-mon[40250]: pgmap v12475: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1231006855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:24 smithi122 ceph-mon[32255]: pgmap v12475: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1231006855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:26 smithi122 ceph-mon[32255]: pgmap v12476: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:26 smithi022 ceph-mon[31360]: pgmap v12476: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:26 smithi022 ceph-mon[40250]: pgmap v12476: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:54:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:28 smithi122 ceph-mon[32255]: pgmap v12477: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:28 smithi022 ceph-mon[31360]: pgmap v12477: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:28 smithi022 ceph-mon[40250]: pgmap v12477: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:29.438 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:29.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:29.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:29.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T00:54:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:30 smithi122 ceph-mon[32255]: pgmap v12478: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3018291628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:30 smithi022 ceph-mon[31360]: pgmap v12478: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3018291628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:30 smithi022 ceph-mon[40250]: pgmap v12478: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3018291628' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:31 smithi122 ceph-mon[32255]: pgmap v12479: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:31 smithi022 ceph-mon[31360]: pgmap v12479: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:31 smithi022 ceph-mon[40250]: pgmap v12479: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:54:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:54:34.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:54:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:54:34.809 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:34 smithi122 ceph-mon[32255]: pgmap v12480: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:54:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:54:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:54:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:54:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[31360]: pgmap v12480: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:54:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:54:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:54:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:54:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[40250]: pgmap v12480: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:54:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:54:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:54:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:54:35.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:35.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:35.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3807116217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3807116217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3807116217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:36 smithi122 ceph-mon[32255]: pgmap v12481: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:36 smithi022 ceph-mon[31360]: pgmap v12481: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:36 smithi022 ceph-mon[40250]: pgmap v12481: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:54:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:38 smithi122 ceph-mon[32255]: pgmap v12482: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:38 smithi022 ceph-mon[31360]: pgmap v12482: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:38 smithi022 ceph-mon[40250]: pgmap v12482: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:40.176 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:40] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T00:54:40.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:40.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:40.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:40 smithi022 ceph-mon[31360]: pgmap v12483: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/867363012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:40 smithi022 ceph-mon[40250]: pgmap v12483: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/867363012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:40 smithi122 ceph-mon[32255]: pgmap v12483: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/867363012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:41 smithi122 ceph-mon[32255]: pgmap v12484: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:41 smithi022 ceph-mon[31360]: pgmap v12484: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:41 smithi022 ceph-mon[40250]: pgmap v12484: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:54:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:54:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:54:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:54:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:44 smithi122 ceph-mon[32255]: pgmap v12485: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:44 smithi022 ceph-mon[31360]: pgmap v12485: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:44 smithi022 ceph-mon[40250]: pgmap v12485: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:45.552 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:45.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:45.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:45.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:45.938 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:45.938 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:46 smithi122 ceph-mon[32255]: pgmap v12486: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1833182611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:46 smithi022 ceph-mon[31360]: pgmap v12486: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1833182611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:46 smithi022 ceph-mon[40250]: pgmap v12486: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1833182611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:54:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:48 smithi022 ceph-mon[31360]: pgmap v12487: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:48 smithi022 ceph-mon[40250]: pgmap v12487: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:48 smithi122 ceph-mon[32255]: pgmap v12487: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:54:50.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:50 smithi022 ceph-mon[31360]: pgmap v12488: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:50 smithi022 ceph-mon[40250]: pgmap v12488: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:50 smithi122 ceph-mon[32255]: pgmap v12488: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:51.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:51.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:51.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:54:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:54:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:54:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/523907853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[31360]: pgmap v12489: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:54:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:54:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:54:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:54:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/523907853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[40250]: pgmap v12489: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:52.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:54:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:54:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:54:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:54:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/523907853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:51 smithi122 ceph-mon[32255]: pgmap v12489: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:54:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:54:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:54:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:54:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:54:54.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:54 smithi122 ceph-mon[32255]: pgmap v12490: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:54 smithi022 ceph-mon[31360]: pgmap v12490: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:55.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:54 smithi022 ceph-mon[40250]: pgmap v12490: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:54:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:54:56.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:54:56.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:54:56.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:54:56.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:54:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:56 smithi022 ceph-mon[40250]: pgmap v12491: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3172812593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:56 smithi022 ceph-mon[31360]: pgmap v12491: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3172812593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:56 smithi122 ceph-mon[32255]: pgmap v12491: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3172812593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:54:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:54:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:54:58 smithi022 ceph-mon[31360]: pgmap v12492: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:54:58 smithi022 ceph-mon[40250]: pgmap v12492: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:54:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:54:58 smithi122 ceph-mon[32255]: pgmap v12492: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T00:55:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:00 smithi022 ceph-mon[31360]: pgmap v12493: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:00 smithi022 ceph-mon[40250]: pgmap v12493: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:00 smithi122 ceph-mon[32255]: pgmap v12493: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:01.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:01 smithi022 ceph-mon[31360]: pgmap v12494: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:01 smithi022 ceph-mon[40250]: pgmap v12494: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:02.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:01 smithi122 ceph-mon[32255]: pgmap v12494: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:55:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:55:03.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2765607221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2765607221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2765607221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:04.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:55:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:55:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:03 smithi122 ceph-mon[32255]: pgmap v12495: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:03 smithi022 ceph-mon[31360]: pgmap v12495: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:03 smithi022 ceph-mon[40250]: pgmap v12495: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:05 smithi122 ceph-mon[32255]: pgmap v12496: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:05 smithi022 ceph-mon[31360]: pgmap v12496: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:05 smithi022 ceph-mon[40250]: pgmap v12496: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:07.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:07.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:07.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:07.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4253104582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:55:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4253104582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4253104582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:08 smithi022 ceph-mon[31360]: pgmap v12497: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:08 smithi022 ceph-mon[40250]: pgmap v12497: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:08 smithi122 ceph-mon[32255]: pgmap v12497: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:10] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:55:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:10 smithi022 ceph-mon[31360]: pgmap v12498: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:10 smithi022 ceph-mon[40250]: pgmap v12498: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:10 smithi122 ceph-mon[32255]: pgmap v12498: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:11 smithi122 ceph-mon[32255]: pgmap v12499: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:11 smithi022 ceph-mon[31360]: pgmap v12499: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:11 smithi022 ceph-mon[40250]: pgmap v12499: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:12.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:55:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:55:12.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:12.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:12.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:13.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1163586447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1163586447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1163586447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:14.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:13 smithi122 ceph-mon[32255]: pgmap v12500: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:14.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:55:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:55:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:13 smithi022 ceph-mon[31360]: pgmap v12500: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:13 smithi022 ceph-mon[40250]: pgmap v12500: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:15 smithi122 ceph-mon[32255]: pgmap v12501: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:15 smithi022 ceph-mon[31360]: pgmap v12501: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:15 smithi022 ceph-mon[40250]: pgmap v12501: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:55:17.801 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:18.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:18.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:18.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:18 smithi022 ceph-mon[31360]: pgmap v12502: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/232893947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:18 smithi022 ceph-mon[40250]: pgmap v12502: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/232893947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:18 smithi122 ceph-mon[32255]: pgmap v12502: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/232893947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:20] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T00:55:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:20 smithi022 ceph-mon[31360]: pgmap v12503: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:20 smithi022 ceph-mon[40250]: pgmap v12503: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:20 smithi122 ceph-mon[32255]: pgmap v12503: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:21 smithi122 ceph-mon[32255]: pgmap v12504: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:21 smithi022 ceph-mon[31360]: pgmap v12504: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:21 smithi022 ceph-mon[40250]: pgmap v12504: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:55:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:55:23.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:23.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:23.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:23.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3878394934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3878394934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3878394934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:24.440 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:55:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:55:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:24 smithi122 ceph-mon[32255]: pgmap v12505: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:24 smithi022 ceph-mon[31360]: pgmap v12505: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:24 smithi022 ceph-mon[40250]: pgmap v12505: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:26 smithi122 ceph-mon[32255]: pgmap v12506: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:26 smithi022 ceph-mon[31360]: pgmap v12506: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:26 smithi022 ceph-mon[40250]: pgmap v12506: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:55:28.544 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:28.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:28.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:28.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:28.917 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:28 smithi022 ceph-mon[31360]: pgmap v12507: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:28.917 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:28 smithi022 ceph-mon[40250]: pgmap v12507: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:28 smithi122 ceph-mon[32255]: pgmap v12507: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3388558683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3388558683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3388558683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:30] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T00:55:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:30 smithi022 ceph-mon[31360]: pgmap v12508: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:30 smithi022 ceph-mon[40250]: pgmap v12508: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:30 smithi122 ceph-mon[32255]: pgmap v12508: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:32.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:31 smithi122 ceph-mon[32255]: pgmap v12509: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:31 smithi022 ceph-mon[31360]: pgmap v12509: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:31 smithi022 ceph-mon[40250]: pgmap v12509: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:55:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:55:33.918 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:34.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:34.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:34.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:34.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:55:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:55:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[31360]: pgmap v12510: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:55:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:55:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:55:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:55:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1839559651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[40250]: pgmap v12510: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:55:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:55:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:55:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:55:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1839559651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:35.195 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:34 smithi122 ceph-mon[32255]: pgmap v12510: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:35.195 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:55:35.196 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:55:35.197 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:55:35.198 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:55:35.198 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1839559651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:36 smithi022 ceph-mon[31360]: pgmap v12511: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:36 smithi022 ceph-mon[40250]: pgmap v12511: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:36 smithi122 ceph-mon[32255]: pgmap v12511: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:55:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:38 smithi022 ceph-mon[31360]: pgmap v12512: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:38 smithi022 ceph-mon[40250]: pgmap v12512: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:38 smithi122 ceph-mon[32255]: pgmap v12512: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:39.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:39.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:39.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:39.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/688897307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/688897307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/688897307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:55:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:40 smithi022 ceph-mon[31360]: pgmap v12513: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:40 smithi022 ceph-mon[40250]: pgmap v12513: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:40 smithi122 ceph-mon[32255]: pgmap v12513: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:41 smithi122 ceph-mon[32255]: pgmap v12514: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:41 smithi022 ceph-mon[31360]: pgmap v12514: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:41 smithi022 ceph-mon[40250]: pgmap v12514: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:55:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:55:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:55:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:55:44.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:45.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:45.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:45.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:45.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:44 smithi022 ceph-mon[31360]: pgmap v12515: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:45.025 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:44 smithi022 ceph-mon[40250]: pgmap v12515: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:44 smithi122 ceph-mon[32255]: pgmap v12515: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4277803991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4277803991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4277803991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:46 smithi022 ceph-mon[31360]: pgmap v12516: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:46 smithi022 ceph-mon[40250]: pgmap v12516: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:46 smithi122 ceph-mon[32255]: pgmap v12516: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:55:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:48 smithi022 ceph-mon[31360]: pgmap v12517: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:48 smithi022 ceph-mon[40250]: pgmap v12517: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:49.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:48 smithi122 ceph-mon[32255]: pgmap v12517: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:50.025 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:50.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:50.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:50.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:50] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:55:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:50 smithi022 ceph-mon[31360]: pgmap v12518: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/108281434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:50 smithi022 ceph-mon[40250]: pgmap v12518: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/108281434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:51.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:50 smithi122 ceph-mon[32255]: pgmap v12518: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/108281434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[31360]: pgmap v12519: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:55:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:55:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:55:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[40250]: pgmap v12519: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:55:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:55:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:55:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:51 smithi122 ceph-mon[32255]: pgmap v12519: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:55:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:55:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:55:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:55:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:55:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:55:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:55:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:54 smithi022 ceph-mon[31360]: pgmap v12520: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:55.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:54 smithi022 ceph-mon[40250]: pgmap v12520: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:55.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:54 smithi122 ceph-mon[32255]: pgmap v12520: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:55:55.394 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:55:55.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:55.699 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:55.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:55:55.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:55:55.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:55:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:55:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: pgmap v12521: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1982280585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:56.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:56.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:56.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:56.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: pgmap v12521: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1982280585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.040 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: pgmap v12521: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:57.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1982280585' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:55:57.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.043 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.045 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.045 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T00:55:57.045 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.045 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:55:57.704 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:55:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:57 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3052M 2021-12-16T00:55:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:57 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3260M 2021-12-16T00:55:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:57 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3052M 2021-12-16T00:55:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:57 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3260M 2021-12-16T00:55:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:57 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3052M 2021-12-16T00:55:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:57 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3260M 2021-12-16T00:55:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:55:58 smithi022 ceph-mon[31360]: pgmap v12522: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:55:58 smithi022 ceph-mon[40250]: pgmap v12522: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:55:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:55:58 smithi122 ceph-mon[32255]: pgmap v12522: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:00] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:56:00.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:00 smithi022 ceph-mon[31360]: pgmap v12523: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:00 smithi022 ceph-mon[40250]: pgmap v12523: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:01.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:01.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:01.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:00 smithi122 ceph-mon[32255]: pgmap v12523: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1178121397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:01 smithi122 ceph-mon[32255]: pgmap v12524: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1178121397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:01 smithi022 ceph-mon[40250]: pgmap v12524: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1178121397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:01 smithi022 ceph-mon[31360]: pgmap v12524: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:56:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:56:04.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:56:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:56:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:04 smithi022 ceph-mon[31360]: pgmap v12525: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:04 smithi022 ceph-mon[40250]: pgmap v12525: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:04 smithi122 ceph-mon[32255]: pgmap v12525: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:06.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:06.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:06.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:06.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:06 smithi022 ceph-mon[31360]: pgmap v12526: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3585980875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:06 smithi022 ceph-mon[40250]: pgmap v12526: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3585980875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:06 smithi122 ceph-mon[32255]: pgmap v12526: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3585980875' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:56:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:08 smithi022 ceph-mon[31360]: pgmap v12527: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:08 smithi022 ceph-mon[40250]: pgmap v12527: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:08 smithi122 ceph-mon[32255]: pgmap v12527: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:10] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:56:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:10 smithi022 ceph-mon[31360]: pgmap v12528: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:10 smithi022 ceph-mon[40250]: pgmap v12528: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:10 smithi122 ceph-mon[32255]: pgmap v12528: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:11.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:11.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:11.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:11.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:11 smithi122 ceph-mon[32255]: pgmap v12529: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3382900728' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:12.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:11 smithi022 ceph-mon[31360]: pgmap v12529: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:12.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3382900728' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:12.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:11 smithi022 ceph-mon[40250]: pgmap v12529: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:12.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3382900728' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:56:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:56:14.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:56:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:56:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:14 smithi022 ceph-mon[31360]: pgmap v12530: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:14 smithi022 ceph-mon[40250]: pgmap v12530: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:14 smithi122 ceph-mon[32255]: pgmap v12530: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:16.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:17.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:16 smithi022 ceph-mon[31360]: pgmap v12531: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:17.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:16 smithi022 ceph-mon[40250]: pgmap v12531: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:16 smithi122 ceph-mon[32255]: pgmap v12531: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:17.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:17.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:17.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:56:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2521553477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:18.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2521553477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:18.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2521553477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:18 smithi022 ceph-mon[31360]: pgmap v12532: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:18 smithi022 ceph-mon[40250]: pgmap v12532: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:18 smithi122 ceph-mon[32255]: pgmap v12532: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:20] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:56:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:20 smithi022 ceph-mon[31360]: pgmap v12533: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:20 smithi022 ceph-mon[40250]: pgmap v12533: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:20 smithi122 ceph-mon[32255]: pgmap v12533: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:21 smithi122 ceph-mon[32255]: pgmap v12534: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:22.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:22.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:21 smithi022 ceph-mon[31360]: pgmap v12534: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:22.264 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:21 smithi022 ceph-mon[40250]: pgmap v12534: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:22.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:22.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:22.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:56:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:56:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2203674383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2203674383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:23.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2203674383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:24.146 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:23 smithi122 ceph-mon[32255]: pgmap v12535: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:23 smithi022 ceph-mon[31360]: pgmap v12535: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:23 smithi022 ceph-mon[40250]: pgmap v12535: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:24.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:56:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:56:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:25 smithi122 ceph-mon[32255]: pgmap v12536: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:25 smithi022 ceph-mon[31360]: pgmap v12536: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:25 smithi022 ceph-mon[40250]: pgmap v12536: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:27.636 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:56:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:28 smithi022 ceph-mon[31360]: pgmap v12537: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2675517497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:28 smithi022 ceph-mon[40250]: pgmap v12537: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2675517497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:28 smithi122 ceph-mon[32255]: pgmap v12537: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2675517497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:30] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:56:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:30 smithi022 ceph-mon[31360]: pgmap v12538: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:30 smithi022 ceph-mon[40250]: pgmap v12538: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:30 smithi122 ceph-mon[32255]: pgmap v12538: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:31 smithi122 ceph-mon[32255]: pgmap v12539: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:31 smithi022 ceph-mon[31360]: pgmap v12539: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:31 smithi022 ceph-mon[40250]: pgmap v12539: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:56:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:56:33.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:33.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:33.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:33.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3673208812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3673208812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3673208812' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:34.424 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:56:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:56:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:34 smithi122 ceph-mon[32255]: pgmap v12540: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:56:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:56:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:56:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:56:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[31360]: pgmap v12540: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:56:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:56:34.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:56:34.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:56:34.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[40250]: pgmap v12540: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:56:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:56:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:56:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:56:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:36 smithi122 ceph-mon[32255]: pgmap v12541: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:36 smithi022 ceph-mon[31360]: pgmap v12541: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:36 smithi022 ceph-mon[40250]: pgmap v12541: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:56:38.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:38.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:38.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:38.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:38 smithi022 ceph-mon[31360]: pgmap v12542: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:38 smithi022 ceph-mon[40250]: pgmap v12542: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:38 smithi122 ceph-mon[32255]: pgmap v12542: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4233533854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4233533854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4233533854' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:40] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T00:56:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:40 smithi022 ceph-mon[31360]: pgmap v12543: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:40 smithi022 ceph-mon[40250]: pgmap v12543: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:40 smithi122 ceph-mon[32255]: pgmap v12543: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:41 smithi122 ceph-mon[32255]: pgmap v12544: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:41 smithi022 ceph-mon[31360]: pgmap v12544: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:41 smithi022 ceph-mon[40250]: pgmap v12544: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:56:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:56:43.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:44.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:44.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:44.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:44.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:56:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:56:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:44 smithi022 ceph-mon[31360]: pgmap v12545: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/463687782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:44 smithi022 ceph-mon[40250]: pgmap v12545: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/463687782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:44 smithi122 ceph-mon[32255]: pgmap v12545: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/463687782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:46 smithi022 ceph-mon[31360]: pgmap v12546: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:46 smithi022 ceph-mon[40250]: pgmap v12546: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:46 smithi122 ceph-mon[32255]: pgmap v12546: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:56:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:48 smithi022 ceph-mon[31360]: pgmap v12547: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:48 smithi022 ceph-mon[40250]: pgmap v12547: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:49.122 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:48 smithi122 ceph-mon[32255]: pgmap v12547: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:49.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:49.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:49.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3625745651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3625745651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:50.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3625745651' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:50] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:56:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:50 smithi022 ceph-mon[31360]: pgmap v12548: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:50 smithi022 ceph-mon[40250]: pgmap v12548: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:50 smithi122 ceph-mon[32255]: pgmap v12548: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:51 smithi122 ceph-mon[32255]: pgmap v12549: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:51 smithi022 ceph-mon[31360]: pgmap v12549: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:51 smithi022 ceph-mon[40250]: pgmap v12549: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:56:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:56:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:56:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:56:54.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:56:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:54 smithi022 ceph-mon[40250]: pgmap v12550: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:54 smithi022 ceph-mon[31360]: pgmap v12550: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:54.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:56:54.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:56:54.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:56:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:54 smithi122 ceph-mon[32255]: pgmap v12550: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:56:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2106482865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2106482865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:56:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2106482865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:56:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[31360]: pgmap v12551: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:56:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:56:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:56:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[40250]: pgmap v12551: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:56:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:56:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:56:57.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:56 smithi122 ceph-mon[32255]: pgmap v12551: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:57.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:56:57.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:56:57.128 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:56:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:56:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:56:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:56:58 smithi122 ceph-mon[32255]: pgmap v12552: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:56:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:56:58 smithi022 ceph-mon[31360]: pgmap v12552: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:56:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:56:58 smithi022 ceph-mon[40250]: pgmap v12552: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:56:59.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:00.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:00.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:00.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:00] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:57:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:00 smithi022 ceph-mon[31360]: pgmap v12553: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/249592417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:00 smithi022 ceph-mon[40250]: pgmap v12553: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/249592417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:00 smithi122 ceph-mon[32255]: pgmap v12553: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/249592417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:01 smithi122 ceph-mon[32255]: pgmap v12554: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:01 smithi022 ceph-mon[40250]: pgmap v12554: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:01 smithi022 ceph-mon[31360]: pgmap v12554: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:57:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:57:04.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:57:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:57:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:04 smithi022 ceph-mon[31360]: pgmap v12555: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:04 smithi022 ceph-mon[40250]: pgmap v12555: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:04 smithi122 ceph-mon[32255]: pgmap v12555: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:05.240 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:05.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:05.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:05.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3264263142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3264263142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3264263142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:06 smithi022 ceph-mon[31360]: pgmap v12556: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:06 smithi022 ceph-mon[40250]: pgmap v12556: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:06 smithi122 ceph-mon[32255]: pgmap v12556: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:57:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:08 smithi022 ceph-mon[31360]: pgmap v12557: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:08 smithi022 ceph-mon[40250]: pgmap v12557: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:08 smithi122 ceph-mon[32255]: pgmap v12557: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:10] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T00:57:10.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:10.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:10.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:10.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:10.985 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:10 smithi022 ceph-mon[31360]: pgmap v12558: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:10.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:10 smithi022 ceph-mon[40250]: pgmap v12558: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:10 smithi122 ceph-mon[32255]: pgmap v12558: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4141618763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:11 smithi122 ceph-mon[32255]: pgmap v12559: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4141618763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:11 smithi022 ceph-mon[31360]: pgmap v12559: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4141618763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:11 smithi022 ceph-mon[40250]: pgmap v12559: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:57:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:57:14.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:57:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:57:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:14 smithi022 ceph-mon[31360]: pgmap v12560: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:14 smithi022 ceph-mon[40250]: pgmap v12560: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:14 smithi122 ceph-mon[32255]: pgmap v12560: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:15.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:15.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:15.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:16 smithi022 ceph-mon[31360]: pgmap v12561: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3899422077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:16 smithi022 ceph-mon[40250]: pgmap v12561: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3899422077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:16 smithi122 ceph-mon[32255]: pgmap v12561: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3899422077' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:57:19.052 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:18 smithi022 ceph-mon[31360]: pgmap v12562: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:19.052 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:18 smithi022 ceph-mon[40250]: pgmap v12562: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:18 smithi122 ceph-mon[32255]: pgmap v12562: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:57:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:20 smithi022 ceph-mon[31360]: pgmap v12563: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:20 smithi022 ceph-mon[40250]: pgmap v12563: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:21.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:20 smithi122 ceph-mon[32255]: pgmap v12563: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:21.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:21.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:21.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:21.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:21 smithi022 ceph-mon[31360]: pgmap v12564: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1502481046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:21 smithi022 ceph-mon[40250]: pgmap v12564: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1502481046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:21 smithi122 ceph-mon[32255]: pgmap v12564: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1502481046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:57:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:57:24.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:57:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:57:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:24 smithi022 ceph-mon[31360]: pgmap v12565: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:24 smithi022 ceph-mon[40250]: pgmap v12565: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:24 smithi122 ceph-mon[32255]: pgmap v12565: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:27.040 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:26 smithi022 ceph-mon[31360]: pgmap v12566: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:27.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:26 smithi022 ceph-mon[40250]: pgmap v12566: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:27.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:27.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:27.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:26 smithi122 ceph-mon[32255]: pgmap v12566: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:27.737 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:57:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1245325025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1245325025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1245325025' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:28 smithi022 ceph-mon[31360]: pgmap v12567: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:28 smithi022 ceph-mon[40250]: pgmap v12567: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:28 smithi122 ceph-mon[32255]: pgmap v12567: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T00:57:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:30 smithi022 ceph-mon[31360]: pgmap v12568: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:30 smithi022 ceph-mon[40250]: pgmap v12568: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:30 smithi122 ceph-mon[32255]: pgmap v12568: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:32.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:31 smithi122 ceph-mon[32255]: pgmap v12569: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:31 smithi022 ceph-mon[31360]: pgmap v12569: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:31 smithi022 ceph-mon[40250]: pgmap v12569: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:32.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:32.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:32.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:57:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:57:33.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3620264923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3620264923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3620264923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:33 smithi122 ceph-mon[32255]: pgmap v12570: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:57:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:57:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:57:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:57:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[31360]: pgmap v12570: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:57:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:57:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:57:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:57:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[40250]: pgmap v12570: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:57:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:57:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:57:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:57:34.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:57:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:57:35.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:36.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:35 smithi122 ceph-mon[32255]: pgmap v12571: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:35 smithi022 ceph-mon[31360]: pgmap v12571: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:35 smithi022 ceph-mon[40250]: pgmap v12571: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:37.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:57:37.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:37.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:37.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:38 smithi022 ceph-mon[31360]: pgmap v12572: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4271869660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:38 smithi022 ceph-mon[40250]: pgmap v12572: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4271869660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:38 smithi122 ceph-mon[32255]: pgmap v12572: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4271869660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:57:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:40 smithi022 ceph-mon[31360]: pgmap v12573: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:40 smithi022 ceph-mon[40250]: pgmap v12573: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:40 smithi122 ceph-mon[32255]: pgmap v12573: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:41 smithi122 ceph-mon[32255]: pgmap v12574: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:41 smithi022 ceph-mon[31360]: pgmap v12574: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:41 smithi022 ceph-mon[40250]: pgmap v12574: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:57:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:57:42.842 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:43.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:43.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:43.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/516438479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/516438479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/516438479' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:44 smithi022 ceph-mon[31360]: pgmap v12575: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:44 smithi022 ceph-mon[40250]: pgmap v12575: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:44 smithi122 ceph-mon[32255]: pgmap v12575: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:44.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:57:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:57:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:45.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:46 smithi022 ceph-mon[31360]: pgmap v12576: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:46 smithi022 ceph-mon[40250]: pgmap v12576: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:46 smithi122 ceph-mon[32255]: pgmap v12576: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:57:48.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:48.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:48.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:48.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:48 smithi022 ceph-mon[31360]: pgmap v12577: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2732787520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:48 smithi022 ceph-mon[40250]: pgmap v12577: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2732787520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:48 smithi122 ceph-mon[32255]: pgmap v12577: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2732787520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:50] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:57:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:50 smithi022 ceph-mon[31360]: pgmap v12578: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:50 smithi022 ceph-mon[40250]: pgmap v12578: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:50 smithi122 ceph-mon[32255]: pgmap v12578: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:51 smithi122 ceph-mon[32255]: pgmap v12579: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:51 smithi022 ceph-mon[31360]: pgmap v12579: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:52.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:51 smithi022 ceph-mon[40250]: pgmap v12579: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:57:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:57:53.583 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:53.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:53.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:53.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:54.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:57:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:57:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:54 smithi022 ceph-mon[31360]: pgmap v12580: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3989059362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:54 smithi022 ceph-mon[40250]: pgmap v12580: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3989059362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:54 smithi122 ceph-mon[32255]: pgmap v12580: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:57:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3989059362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:57:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:57:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:56 smithi022 ceph-mon[31360]: pgmap v12581: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:56 smithi022 ceph-mon[40250]: pgmap v12581: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:57.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:56 smithi122 ceph-mon[32255]: pgmap v12581: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:57:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:57:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:57:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:57:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:57:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:57:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:57:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:57:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:57:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:57:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:57:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:57:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:57:58.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:57:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:58 smithi022 ceph-mon[31360]: pgmap v12582: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:58 smithi022 ceph-mon[40250]: pgmap v12582: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:58 smithi122 ceph-mon[32255]: pgmap v12582: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:57:59.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:57:59.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:57:59.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:57:59.749 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:57:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1095305545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:57:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1095305545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:57:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1095305545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:00] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:58:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:00 smithi022 ceph-mon[31360]: pgmap v12583: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:00 smithi022 ceph-mon[40250]: pgmap v12583: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:00 smithi122 ceph-mon[32255]: pgmap v12583: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:01 smithi122 ceph-mon[32255]: pgmap v12584: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:01 smithi022 ceph-mon[31360]: pgmap v12584: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:01 smithi022 ceph-mon[40250]: pgmap v12584: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:58:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:58:04.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:04.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:58:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:58:04.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:04.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:04.702 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:04 smithi022 ceph-mon[31360]: pgmap v12585: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/609647921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:04 smithi022 ceph-mon[40250]: pgmap v12585: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/609647921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:04 smithi122 ceph-mon[32255]: pgmap v12585: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/609647921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:06 smithi022 ceph-mon[31360]: pgmap v12586: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:06 smithi022 ceph-mon[40250]: pgmap v12586: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:06 smithi122 ceph-mon[32255]: pgmap v12586: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:58:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:08 smithi022 ceph-mon[31360]: pgmap v12587: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:08 smithi022 ceph-mon[40250]: pgmap v12587: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:08 smithi122 ceph-mon[32255]: pgmap v12587: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T00:58:09.703 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:10.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:10.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:10.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:10] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:58:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:10 smithi022 ceph-mon[31360]: pgmap v12588: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:58:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1170484468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:10 smithi022 ceph-mon[40250]: pgmap v12588: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:58:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1170484468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:10 smithi122 ceph-mon[32255]: pgmap v12588: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T00:58:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1170484468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:11 smithi122 ceph-mon[32255]: pgmap v12589: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:11 smithi022 ceph-mon[31360]: pgmap v12589: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:11 smithi022 ceph-mon[40250]: pgmap v12589: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:58:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:58:14.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:58:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:58:15.042 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:14 smithi022 ceph-mon[31360]: pgmap v12590: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:58:15.042 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:14 smithi022 ceph-mon[40250]: pgmap v12590: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:58:15.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:14 smithi122 ceph-mon[32255]: pgmap v12590: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:58:15.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:15.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:15.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4283671848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4283671848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4283671848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:16 smithi022 ceph-mon[31360]: pgmap v12591: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:16 smithi022 ceph-mon[40250]: pgmap v12591: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:17.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:16 smithi122 ceph-mon[32255]: pgmap v12591: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:58:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:18 smithi022 ceph-mon[31360]: pgmap v12592: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:18 smithi022 ceph-mon[40250]: pgmap v12592: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:18 smithi122 ceph-mon[32255]: pgmap v12592: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T00:58:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:20] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:58:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:20.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:20.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:20.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:20 smithi022 ceph-mon[31360]: pgmap v12593: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:58:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:20 smithi022 ceph-mon[40250]: pgmap v12593: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:58:21.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:20 smithi122 ceph-mon[32255]: pgmap v12593: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T00:58:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/818273962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/818273962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/818273962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:58:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:58:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:22 smithi022 ceph-mon[31360]: pgmap v12594: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:22 smithi022 ceph-mon[40250]: pgmap v12594: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:23.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:22 smithi122 ceph-mon[32255]: pgmap v12594: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:24.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:58:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:58:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:24 smithi022 ceph-mon[31360]: pgmap v12595: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:24 smithi022 ceph-mon[40250]: pgmap v12595: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:25.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:24 smithi122 ceph-mon[32255]: pgmap v12595: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:25.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:26.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:26.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:26.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:26 smithi022 ceph-mon[31360]: pgmap v12596: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2232950425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:26 smithi022 ceph-mon[40250]: pgmap v12596: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2232950425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:26 smithi122 ceph-mon[32255]: pgmap v12596: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2232950425' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:58:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:28 smithi022 ceph-mon[31360]: pgmap v12597: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:28 smithi022 ceph-mon[40250]: pgmap v12597: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:28 smithi122 ceph-mon[32255]: pgmap v12597: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:30] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T00:58:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:30 smithi022 ceph-mon[31360]: pgmap v12598: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:30 smithi022 ceph-mon[40250]: pgmap v12598: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:30 smithi122 ceph-mon[32255]: pgmap v12598: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:31.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:31.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:31.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:31.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/338524412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:31 smithi122 ceph-mon[32255]: pgmap v12599: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/338524412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:31 smithi022 ceph-mon[31360]: pgmap v12599: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/338524412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:31 smithi022 ceph-mon[40250]: pgmap v12599: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:58:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:58:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:58:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:58:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:58:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:58:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:58:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:58:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:58:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:58:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:58:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:58:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:58:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:58:34.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:58:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:58:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:34 smithi122 ceph-mon[32255]: pgmap v12600: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:34 smithi022 ceph-mon[31360]: pgmap v12600: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:34 smithi022 ceph-mon[40250]: pgmap v12600: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:36.556 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:36.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:36.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:36.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:36.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:36 smithi022 ceph-mon[31360]: pgmap v12601: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:36.930 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:36 smithi022 ceph-mon[40250]: pgmap v12601: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:36 smithi122 ceph-mon[32255]: pgmap v12601: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:58:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3644123238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3644123238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3644123238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:38 smithi122 ceph-mon[32255]: pgmap v12602: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:38 smithi022 ceph-mon[31360]: pgmap v12602: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:38 smithi022 ceph-mon[40250]: pgmap v12602: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:40] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:58:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:40 smithi122 ceph-mon[32255]: pgmap v12603: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:40 smithi022 ceph-mon[31360]: pgmap v12603: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:40 smithi022 ceph-mon[40250]: pgmap v12603: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:41.930 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:41 smithi122 ceph-mon[32255]: pgmap v12604: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:41 smithi022 ceph-mon[31360]: pgmap v12604: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:41 smithi022 ceph-mon[40250]: pgmap v12604: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:42.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:42.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:42.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:58:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:58:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/913213698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/913213698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/913213698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:43 smithi122 ceph-mon[32255]: pgmap v12605: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:43 smithi022 ceph-mon[31360]: pgmap v12605: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:43 smithi022 ceph-mon[40250]: pgmap v12605: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:44.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:58:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:58:45.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:45 smithi122 ceph-mon[32255]: pgmap v12606: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:45 smithi022 ceph-mon[31360]: pgmap v12606: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:45 smithi022 ceph-mon[40250]: pgmap v12606: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:47.297 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:47.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:47.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:47.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:47.665 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:58:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4216986563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4216986563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4216986563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:48 smithi022 ceph-mon[31360]: pgmap v12607: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:48 smithi022 ceph-mon[40250]: pgmap v12607: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:48 smithi122 ceph-mon[32255]: pgmap v12607: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:50] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T00:58:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:50 smithi022 ceph-mon[31360]: pgmap v12608: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:50 smithi022 ceph-mon[40250]: pgmap v12608: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:50 smithi122 ceph-mon[32255]: pgmap v12608: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:51 smithi122 ceph-mon[32255]: pgmap v12609: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:51 smithi022 ceph-mon[31360]: pgmap v12609: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:51 smithi022 ceph-mon[40250]: pgmap v12609: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:52.666 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:58:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:58:53.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:53.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:53.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/970745098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/970745098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/970745098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:54 smithi122 ceph-mon[32255]: pgmap v12610: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:54.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:58:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:58:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:54 smithi022 ceph-mon[31360]: pgmap v12610: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:54 smithi022 ceph-mon[40250]: pgmap v12610: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:58:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:58:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:56 smithi122 ceph-mon[32255]: pgmap v12611: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:56 smithi022 ceph-mon[31360]: pgmap v12611: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:56 smithi022 ceph-mon[40250]: pgmap v12611: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:57.667 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:58:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:58:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:58:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:58:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:58:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:58:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:58:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:58:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:58:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:58:58.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:58:58.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:58:58.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:58:58.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:58:58.678 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:58 smithi022 ceph-mon[31360]: pgmap v12612: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:58.679 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:58:58.679 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:58:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1738939515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:58.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:58 smithi122 ceph-mon[32255]: pgmap v12612: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:58:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:58:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1738939515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:58:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:58 smithi022 ceph-mon[40250]: pgmap v12612: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:58:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T00:58:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:58:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1738939515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:00] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T00:59:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:00 smithi022 ceph-mon[31360]: pgmap v12613: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:00 smithi022 ceph-mon[40250]: pgmap v12613: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:00 smithi122 ceph-mon[32255]: pgmap v12613: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:01 smithi122 ceph-mon[32255]: pgmap v12614: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:01 smithi022 ceph-mon[31360]: pgmap v12614: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:01 smithi022 ceph-mon[40250]: pgmap v12614: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:59:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:59:03.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:03.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:03.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:03.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:04.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:59:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:59:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:04 smithi022 ceph-mon[31360]: pgmap v12615: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2458767101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:04 smithi022 ceph-mon[40250]: pgmap v12615: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2458767101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:04 smithi122 ceph-mon[32255]: pgmap v12615: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2458767101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:06 smithi022 ceph-mon[31360]: pgmap v12616: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:06 smithi022 ceph-mon[40250]: pgmap v12616: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:06 smithi122 ceph-mon[32255]: pgmap v12616: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:59:08.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:08 smithi022 ceph-mon[31360]: pgmap v12617: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:08 smithi022 ceph-mon[40250]: pgmap v12617: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:09.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:09.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:09.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:08 smithi122 ceph-mon[32255]: pgmap v12617: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/908787055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/908787055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/908787055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:10] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T00:59:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:10 smithi022 ceph-mon[31360]: pgmap v12618: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:10 smithi022 ceph-mon[40250]: pgmap v12618: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:10 smithi122 ceph-mon[32255]: pgmap v12618: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:11 smithi122 ceph-mon[32255]: pgmap v12619: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:11 smithi022 ceph-mon[31360]: pgmap v12619: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:11 smithi022 ceph-mon[40250]: pgmap v12619: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:59:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:59:14.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:14.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:59:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:59:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:14 smithi022 ceph-mon[31360]: pgmap v12620: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3984424908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:14 smithi022 ceph-mon[40250]: pgmap v12620: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3984424908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:15.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:14 smithi122 ceph-mon[32255]: pgmap v12620: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3984424908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:16 smithi022 ceph-mon[31360]: pgmap v12621: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:16 smithi022 ceph-mon[40250]: pgmap v12621: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:17.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:16 smithi122 ceph-mon[32255]: pgmap v12621: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:59:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:18 smithi022 ceph-mon[31360]: pgmap v12622: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:18 smithi022 ceph-mon[40250]: pgmap v12622: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:18 smithi122 ceph-mon[32255]: pgmap v12622: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:19.508 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:19.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:19.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:19.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:20] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T00:59:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:20 smithi022 ceph-mon[31360]: pgmap v12623: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2221682375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:20 smithi022 ceph-mon[40250]: pgmap v12623: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2221682375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:21.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:20 smithi122 ceph-mon[32255]: pgmap v12623: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2221682375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:21 smithi122 ceph-mon[32255]: pgmap v12624: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:21 smithi022 ceph-mon[31360]: pgmap v12624: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:21 smithi022 ceph-mon[40250]: pgmap v12624: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:59:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:59:24.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:59:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:59:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:24 smithi022 ceph-mon[31360]: pgmap v12625: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:24 smithi022 ceph-mon[40250]: pgmap v12625: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:25.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:24 smithi122 ceph-mon[32255]: pgmap v12625: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:25.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:25.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:25.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2349420765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2349420765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2349420765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:26 smithi022 ceph-mon[31360]: pgmap v12626: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:26 smithi022 ceph-mon[40250]: pgmap v12626: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:26 smithi122 ceph-mon[32255]: pgmap v12626: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:59:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:28 smithi022 ceph-mon[31360]: pgmap v12627: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:28 smithi022 ceph-mon[40250]: pgmap v12627: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:28 smithi122 ceph-mon[32255]: pgmap v12627: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:30.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:30] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T00:59:30.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:30.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:30.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:30 smithi022 ceph-mon[31360]: pgmap v12628: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2542192070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:30 smithi022 ceph-mon[40250]: pgmap v12628: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2542192070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:30 smithi122 ceph-mon[32255]: pgmap v12628: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2542192070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:31 smithi122 ceph-mon[32255]: pgmap v12629: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:31 smithi022 ceph-mon[31360]: pgmap v12629: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:31 smithi022 ceph-mon[40250]: pgmap v12629: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:59:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:59:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:59:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:59:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:59:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:59:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:59:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:59:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:59:34.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:59:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:59:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:59:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T00:59:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T00:59:34.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:59:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:59:35.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:34 smithi122 ceph-mon[32255]: pgmap v12630: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:34 smithi022 ceph-mon[31360]: pgmap v12630: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:34 smithi022 ceph-mon[40250]: pgmap v12630: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:35.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:35.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:35.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:35.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:35.999 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:36.000 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:36 smithi122 ceph-mon[32255]: pgmap v12631: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:37.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2006136445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:36 smithi022 ceph-mon[31360]: pgmap v12631: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:37.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2006136445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:37.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:36 smithi022 ceph-mon[40250]: pgmap v12631: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:37.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2006136445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:59:39.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:38 smithi122 ceph-mon[32255]: pgmap v12632: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:38 smithi022 ceph-mon[31360]: pgmap v12632: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:38 smithi022 ceph-mon[40250]: pgmap v12632: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:40] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T00:59:41.000 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:40 smithi122 ceph-mon[32255]: pgmap v12633: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:40 smithi022 ceph-mon[40250]: pgmap v12633: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:40 smithi022 ceph-mon[31360]: pgmap v12633: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:41.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:41.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:41.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1904008325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:41 smithi122 ceph-mon[32255]: pgmap v12634: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1904008325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:41 smithi022 ceph-mon[31360]: pgmap v12634: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1904008325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:41 smithi022 ceph-mon[40250]: pgmap v12634: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:59:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:59:44.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:59:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:59:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:44 smithi022 ceph-mon[31360]: pgmap v12635: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:44 smithi022 ceph-mon[40250]: pgmap v12635: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:44 smithi122 ceph-mon[32255]: pgmap v12635: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:46.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:47.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:46 smithi022 ceph-mon[40250]: pgmap v12636: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:47.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/758062712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:47.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:46 smithi022 ceph-mon[31360]: pgmap v12636: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:47.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/758062712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:46 smithi122 ceph-mon[32255]: pgmap v12636: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/758062712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:59:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:48 smithi022 ceph-mon[31360]: pgmap v12637: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:48 smithi022 ceph-mon[40250]: pgmap v12637: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:48 smithi122 ceph-mon[32255]: pgmap v12637: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 00:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:50] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T00:59:51.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:50 smithi122 ceph-mon[32255]: pgmap v12638: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:50 smithi022 ceph-mon[31360]: pgmap v12638: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:50 smithi022 ceph-mon[40250]: pgmap v12638: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:51.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:51 smithi022 ceph-mon[31360]: pgmap v12639: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:51 smithi022 ceph-mon[40250]: pgmap v12639: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:52.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:52.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:52.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:51 smithi122 ceph-mon[32255]: pgmap v12639: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 00:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T00:59:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T00:59:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2130184732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:53.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2130184732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2130184732' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:53 smithi022 ceph-mon[31360]: pgmap v12640: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:53 smithi022 ceph-mon[40250]: pgmap v12640: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:54.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:53 smithi122 ceph-mon[32255]: pgmap v12640: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T00:59:54.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 00:59:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T00:59:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T00:59:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:55 smithi022 ceph-mon[31360]: pgmap v12641: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:55 smithi022 ceph-mon[40250]: pgmap v12641: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:55 smithi122 ceph-mon[32255]: pgmap v12641: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:57.115 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T00:59:57.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T00:59:57.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T00:59:57.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T00:59:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1529208741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:57.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1529208741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 00:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:00:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T00:59:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1529208741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T00:59:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:58 smithi122 ceph-mon[32255]: pgmap v12642: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:59:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:59:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:59:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[31360]: pgmap v12642: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:59:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:59:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:59:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[40250]: pgmap v12642: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T00:59:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T00:59:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T00:59:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T00:59:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 00:59:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:00:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 00:59:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:00:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 00:59:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:00:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:00] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:00:00.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T01:00:00.231Z caller=compact.go:495 component=tsdb msg="write block" mint=1639605600000 maxt=1639612800000 ulid=01FQ0C1JSJ5PB1RZDJVDS55TAY duration=53.03465ms 2021-12-16T01:00:00.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T01:00:00.241Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=2.591038ms 2021-12-16T01:00:00.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T01:00:00.408Z caller=head.go:734 component=tsdb msg="WAL checkpoint complete" first=0 last=1 duration=166.551804ms 2021-12-16T01:00:00.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:00 smithi122 ceph-mon[32255]: pgmap v12643: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T01:00:01.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:00 smithi022 ceph-mon[31360]: pgmap v12643: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:01.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T01:00:01.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:00 smithi022 ceph-mon[40250]: pgmap v12643: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:01.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T01:00:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:01 smithi122 ceph-mon[32255]: pgmap v12644: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:01 smithi022 ceph-mon[31360]: pgmap v12644: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:01 smithi022 ceph-mon[40250]: pgmap v12644: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:02.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:00:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:00:02.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:02.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:02.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/849822305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/849822305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/849822305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:04.291 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:03 smithi022 ceph-mon[31360]: pgmap v12645: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:04.292 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:03 smithi022 ceph-mon[40250]: pgmap v12645: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:04.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:03 smithi122 ceph-mon[32255]: pgmap v12645: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:04.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:00:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:00:05.295 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:05.295 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:05.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:05 smithi022 ceph-mon[31360]: pgmap v12646: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:05 smithi022 ceph-mon[40250]: pgmap v12646: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:05 smithi122 ceph-mon[32255]: pgmap v12646: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:07.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:00:07.858 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:08.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:08.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:08.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:08 smithi022 ceph-mon[31360]: pgmap v12647: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1184680884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:08 smithi022 ceph-mon[40250]: pgmap v12647: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1184680884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:08 smithi122 ceph-mon[32255]: pgmap v12647: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1184680884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:10] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:00:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:10 smithi122 ceph-mon[32255]: pgmap v12648: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:10 smithi022 ceph-mon[31360]: pgmap v12648: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:10 smithi022 ceph-mon[40250]: pgmap v12648: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:11 smithi022 ceph-mon[31360]: pgmap v12649: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:11 smithi022 ceph-mon[40250]: pgmap v12649: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:11 smithi122 ceph-mon[32255]: pgmap v12649: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:00:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:00:13.235 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:13.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:13.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:13.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3691305535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3691305535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3691305535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:14.647 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:00:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:00:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:14 smithi122 ceph-mon[32255]: pgmap v12650: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:14 smithi022 ceph-mon[31360]: pgmap v12650: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:14 smithi022 ceph-mon[40250]: pgmap v12650: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:15.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:16 smithi122 ceph-mon[32255]: pgmap v12651: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:16 smithi022 ceph-mon[31360]: pgmap v12651: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:16 smithi022 ceph-mon[40250]: pgmap v12651: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:00:18.605 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:18.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:18.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:18.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:18.978 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:18 smithi022 ceph-mon[31360]: pgmap v12652: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:18.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:18 smithi022 ceph-mon[40250]: pgmap v12652: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:18 smithi122 ceph-mon[32255]: pgmap v12652: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3025930525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3025930525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3025930525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:20] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:00:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:20 smithi122 ceph-mon[32255]: pgmap v12653: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:20 smithi022 ceph-mon[40250]: pgmap v12653: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:20 smithi022 ceph-mon[31360]: pgmap v12653: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:21 smithi022 ceph-mon[31360]: pgmap v12654: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:21 smithi022 ceph-mon[40250]: pgmap v12654: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:21 smithi122 ceph-mon[32255]: pgmap v12654: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:00:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:00:23.979 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:24.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:24.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:24.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:24.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:00:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:00:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:24 smithi122 ceph-mon[32255]: pgmap v12655: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:25.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3042219414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:24 smithi022 ceph-mon[31360]: pgmap v12655: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3042219414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:24 smithi022 ceph-mon[40250]: pgmap v12655: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3042219414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:26 smithi122 ceph-mon[32255]: pgmap v12656: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:27.299 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:26 smithi022 ceph-mon[31360]: pgmap v12656: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:27.299 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:26 smithi022 ceph-mon[40250]: pgmap v12656: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:00:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:28 smithi122 ceph-mon[32255]: pgmap v12657: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:28 smithi022 ceph-mon[31360]: pgmap v12657: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:28 smithi022 ceph-mon[40250]: pgmap v12657: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:29.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:29.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:29.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:29.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3103263866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3103263866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3103263866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:30] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:00:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:30 smithi122 ceph-mon[32255]: pgmap v12658: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:30 smithi022 ceph-mon[31360]: pgmap v12658: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:30 smithi022 ceph-mon[40250]: pgmap v12658: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:31 smithi022 ceph-mon[31360]: pgmap v12659: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:31 smithi022 ceph-mon[40250]: pgmap v12659: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:31 smithi122 ceph-mon[32255]: pgmap v12659: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:00:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:00:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:00:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:00:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:00:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:00:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:00:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:00:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:00:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:00:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:00:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:00:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:00:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:00:34.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:00:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:00:34.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:34 smithi022 ceph-mon[31360]: pgmap v12660: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:34 smithi022 ceph-mon[40250]: pgmap v12660: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:35.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:35.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:35.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:35.198 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:34 smithi122 ceph-mon[32255]: pgmap v12660: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/447551737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/447551737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/447551737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:36 smithi122 ceph-mon[32255]: pgmap v12661: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:36 smithi022 ceph-mon[31360]: pgmap v12661: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:36 smithi022 ceph-mon[40250]: pgmap v12661: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:00:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:38 smithi122 ceph-mon[32255]: pgmap v12662: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:38 smithi022 ceph-mon[31360]: pgmap v12662: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:38 smithi022 ceph-mon[40250]: pgmap v12662: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:40.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:40] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:00:40.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:40.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:40.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:40 smithi122 ceph-mon[32255]: pgmap v12663: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/188536542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:40 smithi022 ceph-mon[31360]: pgmap v12663: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:41.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/188536542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:40 smithi022 ceph-mon[40250]: pgmap v12663: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:41.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/188536542' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:41 smithi022 ceph-mon[31360]: pgmap v12664: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:41 smithi022 ceph-mon[40250]: pgmap v12664: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:41 smithi122 ceph-mon[32255]: pgmap v12664: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:00:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:00:44.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:00:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:00:45.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:44 smithi122 ceph-mon[32255]: pgmap v12665: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:44 smithi022 ceph-mon[31360]: pgmap v12665: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:44 smithi022 ceph-mon[40250]: pgmap v12665: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:45.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:45.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:45.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:45.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:46 smithi122 ceph-mon[32255]: pgmap v12666: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/912258724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:46 smithi022 ceph-mon[31360]: pgmap v12666: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/912258724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:46 smithi022 ceph-mon[40250]: pgmap v12666: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/912258724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:00:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:48 smithi122 ceph-mon[32255]: pgmap v12667: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:48 smithi022 ceph-mon[31360]: pgmap v12667: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:48 smithi022 ceph-mon[40250]: pgmap v12667: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:50.443 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:50] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:00:50.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:51.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:50 smithi122 ceph-mon[32255]: pgmap v12668: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:51.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:51.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:51.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:51.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:50 smithi022 ceph-mon[31360]: pgmap v12668: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:51.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:50 smithi022 ceph-mon[40250]: pgmap v12668: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3851998307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:51 smithi022 ceph-mon[31360]: pgmap v12669: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3851998307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:51 smithi022 ceph-mon[40250]: pgmap v12669: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3851998307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:51 smithi122 ceph-mon[32255]: pgmap v12669: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:00:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:00:54.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:00:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:00:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:54 smithi122 ceph-mon[32255]: pgmap v12670: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:54 smithi022 ceph-mon[31360]: pgmap v12670: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:54 smithi022 ceph-mon[40250]: pgmap v12670: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:00:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:56.205 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:00:56.206 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:56.206 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:00:56.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:00:56.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:00:56.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:00:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:56 smithi022 ceph-mon[31360]: pgmap v12671: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/22471044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:56 smithi022 ceph-mon[40250]: pgmap v12671: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/22471044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:56 smithi122 ceph-mon[32255]: pgmap v12671: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/22471044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:00:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:00:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:58 smithi122 ceph-mon[32255]: pgmap v12672: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:00:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:00:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:00:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:00:59.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[31360]: pgmap v12672: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:59.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:00:59.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:00:59.021 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:00:59.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[40250]: pgmap v12672: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:00:59.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:00:59.022 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:00:59.022 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:00:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:01:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:00] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:01:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:01:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:00 smithi122 ceph-mon[32255]: pgmap v12673: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:00.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:01:00.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:00 smithi022 ceph-mon[31360]: pgmap v12673: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:00.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:01:00.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:00 smithi022 ceph-mon[40250]: pgmap v12673: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:01.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:01.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:01.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:01.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:01 smithi022 ceph-mon[31360]: pgmap v12674: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2826551627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:01 smithi022 ceph-mon[40250]: pgmap v12674: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2826551627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:01 smithi122 ceph-mon[32255]: pgmap v12674: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2826551627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:01:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:01:04.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:01:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:01:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:04 smithi122 ceph-mon[32255]: pgmap v12675: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:05.290 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:04 smithi022 ceph-mon[31360]: pgmap v12675: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:05.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:04 smithi022 ceph-mon[40250]: pgmap v12675: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:06.953 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:06 smithi122 ceph-mon[32255]: pgmap v12676: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:06 smithi022 ceph-mon[31360]: pgmap v12676: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:06 smithi022 ceph-mon[40250]: pgmap v12676: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:07.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:07.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:07.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:01:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3773245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3773245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3773245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:08 smithi122 ceph-mon[32255]: pgmap v12677: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:08 smithi022 ceph-mon[31360]: pgmap v12677: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:08 smithi022 ceph-mon[40250]: pgmap v12677: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:10] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:01:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:10 smithi122 ceph-mon[32255]: pgmap v12678: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:10 smithi022 ceph-mon[31360]: pgmap v12678: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:10 smithi022 ceph-mon[40250]: pgmap v12678: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:11 smithi022 ceph-mon[31360]: pgmap v12679: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:11 smithi022 ceph-mon[40250]: pgmap v12679: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:11 smithi122 ceph-mon[32255]: pgmap v12679: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:12.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:01:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:01:12.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:12.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:12.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/257016518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/257016518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/257016518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:13 smithi022 ceph-mon[31360]: pgmap v12680: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:13 smithi022 ceph-mon[40250]: pgmap v12680: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:14.389 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:13 smithi122 ceph-mon[32255]: pgmap v12680: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:14.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:01:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:01:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:15 smithi022 ceph-mon[31360]: pgmap v12681: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:15 smithi022 ceph-mon[40250]: pgmap v12681: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:15 smithi122 ceph-mon[32255]: pgmap v12681: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:17.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:17.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:01:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:18 smithi122 ceph-mon[32255]: pgmap v12682: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1189416802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:18 smithi022 ceph-mon[31360]: pgmap v12682: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1189416802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:18 smithi022 ceph-mon[40250]: pgmap v12682: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1189416802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:20] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:01:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:20 smithi122 ceph-mon[32255]: pgmap v12683: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:20 smithi022 ceph-mon[31360]: pgmap v12683: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:20 smithi022 ceph-mon[40250]: pgmap v12683: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:21 smithi022 ceph-mon[31360]: pgmap v12684: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:21 smithi022 ceph-mon[40250]: pgmap v12684: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:21 smithi122 ceph-mon[32255]: pgmap v12684: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:01:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:01:23.074 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:23.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:23.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:23.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2860883250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2860883250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2860883250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:24.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:01:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:01:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:24 smithi122 ceph-mon[32255]: pgmap v12685: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:24 smithi022 ceph-mon[31360]: pgmap v12685: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:24 smithi022 ceph-mon[40250]: pgmap v12685: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:26 smithi022 ceph-mon[31360]: pgmap v12686: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:26 smithi022 ceph-mon[40250]: pgmap v12686: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:26 smithi122 ceph-mon[32255]: pgmap v12686: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:01:28.447 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:28.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:28.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:28.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:28 smithi122 ceph-mon[32255]: pgmap v12687: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:28 smithi022 ceph-mon[31360]: pgmap v12687: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:28 smithi022 ceph-mon[40250]: pgmap v12687: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1399305776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1399305776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1399305776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:30] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:01:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:30 smithi122 ceph-mon[32255]: pgmap v12688: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:30 smithi022 ceph-mon[31360]: pgmap v12688: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:30 smithi022 ceph-mon[40250]: pgmap v12688: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:31 smithi022 ceph-mon[31360]: pgmap v12689: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:31 smithi022 ceph-mon[40250]: pgmap v12689: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:32.316 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:01:32.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:01:32.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:31 smithi122 ceph-mon[32255]: pgmap v12689: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:33.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:01:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:01:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:01:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:01:34.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:34.193 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:01:34.193 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:01:34.193 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:01:34.194 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:01:34.194 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:01:34.195 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:01:34.195 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:01:34.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:01:34.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:01:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:01:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:34 smithi122 ceph-mon[32255]: pgmap v12690: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3575228670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:34 smithi022 ceph-mon[31360]: pgmap v12690: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3575228670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:34 smithi022 ceph-mon[40250]: pgmap v12690: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3575228670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:36 smithi122 ceph-mon[32255]: pgmap v12691: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:36 smithi022 ceph-mon[31360]: pgmap v12691: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:36 smithi022 ceph-mon[40250]: pgmap v12691: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:01:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:38 smithi122 ceph-mon[32255]: pgmap v12692: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:39.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:39.194 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:38 smithi022 ceph-mon[31360]: pgmap v12692: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:39.194 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:38 smithi022 ceph-mon[40250]: pgmap v12692: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:39.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:39.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:39.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2391848298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2391848298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2391848298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:40] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:01:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:40 smithi122 ceph-mon[32255]: pgmap v12693: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:40 smithi022 ceph-mon[31360]: pgmap v12693: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:40 smithi022 ceph-mon[40250]: pgmap v12693: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:41 smithi022 ceph-mon[31360]: pgmap v12694: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:41 smithi022 ceph-mon[40250]: pgmap v12694: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:41 smithi122 ceph-mon[32255]: pgmap v12694: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:01:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:01:44.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:44.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:01:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:01:44.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:44.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:44.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:44.940 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:44 smithi022 ceph-mon[31360]: pgmap v12695: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:44.941 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:44 smithi022 ceph-mon[40250]: pgmap v12695: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:44 smithi122 ceph-mon[32255]: pgmap v12695: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3570264355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3570264355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3570264355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:46 smithi122 ceph-mon[32255]: pgmap v12696: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:46 smithi022 ceph-mon[31360]: pgmap v12696: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:46 smithi022 ceph-mon[40250]: pgmap v12696: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:01:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:48 smithi122 ceph-mon[32255]: pgmap v12697: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:48 smithi022 ceph-mon[31360]: pgmap v12697: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:48 smithi022 ceph-mon[40250]: pgmap v12697: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:49.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:50.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:50.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:50.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:50] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:01:51.200 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:50 smithi122 ceph-mon[32255]: pgmap v12698: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:51.200 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3498013861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:50 smithi022 ceph-mon[40250]: pgmap v12698: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3498013861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:50 smithi022 ceph-mon[31360]: pgmap v12698: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3498013861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:51 smithi022 ceph-mon[31360]: pgmap v12699: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:51 smithi022 ceph-mon[40250]: pgmap v12699: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:51 smithi122 ceph-mon[32255]: pgmap v12699: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:01:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:01:54.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:01:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:01:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:54 smithi122 ceph-mon[32255]: pgmap v12700: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:55.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:54 smithi022 ceph-mon[31360]: pgmap v12700: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:54 smithi022 ceph-mon[40250]: pgmap v12700: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:01:55.307 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:01:55.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:01:55.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:01:55.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:01:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1626644926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1626644926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:01:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1626644926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:01:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:56 smithi122 ceph-mon[32255]: pgmap v12701: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:56 smithi022 ceph-mon[31360]: pgmap v12701: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:56 smithi022 ceph-mon[40250]: pgmap v12701: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:01:59.069 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:58 smithi122 ceph-mon[32255]: pgmap v12702: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:59.075 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:58 smithi022 ceph-mon[31360]: pgmap v12702: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:01:59.075 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:58 smithi022 ceph-mon[40250]: pgmap v12702: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:02:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:02:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:02:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:01:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:02:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:02:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:02:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:02:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:02:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:02:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:00.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:01:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:00] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:02:00.678 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:00 smithi022 ceph-mon[31360]: pgmap v12703: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:00 smithi022 ceph-mon[40250]: pgmap v12703: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:01.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:01.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:01.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:00 smithi122 ceph-mon[32255]: pgmap v12703: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:02:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1478192382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:01 smithi022 ceph-mon[31360]: pgmap v12704: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1478192382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:01 smithi022 ceph-mon[40250]: pgmap v12704: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1478192382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:01 smithi122 ceph-mon[32255]: pgmap v12704: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:02:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:02:04.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:02:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:02:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:04 smithi122 ceph-mon[32255]: pgmap v12705: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:04 smithi022 ceph-mon[31360]: pgmap v12705: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:04 smithi022 ceph-mon[40250]: pgmap v12705: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:06.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:06.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:06.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:06.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:06 smithi122 ceph-mon[32255]: pgmap v12706: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2035012200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:07.295 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:06 smithi022 ceph-mon[31360]: pgmap v12706: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:07.296 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2035012200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:07.296 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:06 smithi022 ceph-mon[40250]: pgmap v12706: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:07.297 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2035012200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:02:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:08 smithi122 ceph-mon[32255]: pgmap v12707: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:08 smithi022 ceph-mon[31360]: pgmap v12707: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:08 smithi022 ceph-mon[40250]: pgmap v12707: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:10] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:02:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:10 smithi122 ceph-mon[32255]: pgmap v12708: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:10 smithi022 ceph-mon[31360]: pgmap v12708: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:10 smithi022 ceph-mon[40250]: pgmap v12708: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:11.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:11.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:11.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:11 smithi022 ceph-mon[31360]: pgmap v12709: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1268675387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:11 smithi022 ceph-mon[40250]: pgmap v12709: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1268675387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:11 smithi122 ceph-mon[32255]: pgmap v12709: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1268675387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:02:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:02:14.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:02:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:02:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:14 smithi122 ceph-mon[32255]: pgmap v12710: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:14 smithi022 ceph-mon[31360]: pgmap v12710: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:14 smithi022 ceph-mon[40250]: pgmap v12710: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:16.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:17.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:17.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:17.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:17.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:16 smithi022 ceph-mon[31360]: pgmap v12711: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:17.167 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:16 smithi022 ceph-mon[40250]: pgmap v12711: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:16 smithi122 ceph-mon[32255]: pgmap v12711: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:02:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/47730592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/47730592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/47730592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:18 smithi122 ceph-mon[32255]: pgmap v12712: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:18 smithi022 ceph-mon[31360]: pgmap v12712: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:18 smithi022 ceph-mon[40250]: pgmap v12712: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:02:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:20 smithi122 ceph-mon[32255]: pgmap v12713: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:21.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:20 smithi022 ceph-mon[31360]: pgmap v12713: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:21.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:20 smithi022 ceph-mon[40250]: pgmap v12713: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:22.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:21 smithi022 ceph-mon[31360]: pgmap v12714: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:21 smithi022 ceph-mon[40250]: pgmap v12714: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:22.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:21 smithi122 ceph-mon[32255]: pgmap v12714: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:22.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:22.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:22.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:02:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:02:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4206759147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:23.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4206759147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4206759147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:23 smithi022 ceph-mon[31360]: pgmap v12715: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:23 smithi022 ceph-mon[40250]: pgmap v12715: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:23 smithi122 ceph-mon[32255]: pgmap v12715: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:24.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:02:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:02:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:25.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:25 smithi022 ceph-mon[31360]: pgmap v12716: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:25 smithi022 ceph-mon[40250]: pgmap v12716: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:25 smithi122 ceph-mon[32255]: pgmap v12716: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:27.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:02:27.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:27.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:27.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:28 smithi122 ceph-mon[32255]: pgmap v12717: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1950506486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:28 smithi022 ceph-mon[31360]: pgmap v12717: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1950506486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:28 smithi022 ceph-mon[40250]: pgmap v12717: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1950506486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:02:31.207 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:30 smithi122 ceph-mon[32255]: pgmap v12718: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:30 smithi022 ceph-mon[31360]: pgmap v12718: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:30 smithi022 ceph-mon[40250]: pgmap v12718: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:31 smithi022 ceph-mon[31360]: pgmap v12719: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:31 smithi022 ceph-mon[40250]: pgmap v12719: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:32.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:31 smithi122 ceph-mon[32255]: pgmap v12719: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:02:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:02:32.907 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:33.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:33.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:33.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4085464094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4085464094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4085464094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:34.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:02:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:02:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:02:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:02:34.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:34 smithi122 ceph-mon[32255]: pgmap v12720: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:02:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:02:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:02:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:02:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[31360]: pgmap v12720: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:02:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:02:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:02:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:02:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:34 smithi022 ceph-mon[40250]: pgmap v12720: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:36 smithi122 ceph-mon[32255]: pgmap v12721: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:36 smithi022 ceph-mon[31360]: pgmap v12721: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:36 smithi022 ceph-mon[40250]: pgmap v12721: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:02:38.282 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:38.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:38.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:38.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:39.043 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:38 smithi022 ceph-mon[31360]: pgmap v12722: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:39.043 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3941990377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:39.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:38 smithi022 ceph-mon[40250]: pgmap v12722: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:39.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3941990377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:38 smithi122 ceph-mon[32255]: pgmap v12722: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3941990377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:40] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:02:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:40 smithi122 ceph-mon[32255]: pgmap v12723: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:40 smithi022 ceph-mon[31360]: pgmap v12723: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:40 smithi022 ceph-mon[40250]: pgmap v12723: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:41 smithi022 ceph-mon[31360]: pgmap v12724: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:41 smithi022 ceph-mon[40250]: pgmap v12724: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:41 smithi122 ceph-mon[32255]: pgmap v12724: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:02:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:02:43.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:44.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:44.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:44.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:44.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:02:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:02:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:44 smithi122 ceph-mon[32255]: pgmap v12725: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2788374831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:44 smithi022 ceph-mon[31360]: pgmap v12725: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2788374831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:44 smithi022 ceph-mon[40250]: pgmap v12725: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:45.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2788374831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:46 smithi122 ceph-mon[32255]: pgmap v12726: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:47.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:46 smithi022 ceph-mon[31360]: pgmap v12726: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:47.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:46 smithi022 ceph-mon[40250]: pgmap v12726: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:02:49.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:48 smithi122 ceph-mon[32255]: pgmap v12727: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:49.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:48 smithi022 ceph-mon[31360]: pgmap v12727: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:49.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:48 smithi022 ceph-mon[40250]: pgmap v12727: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:49.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:49.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:49.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/476509492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:50.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/476509492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:50.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/476509492' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:50] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:02:51.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:50 smithi122 ceph-mon[32255]: pgmap v12728: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:50 smithi022 ceph-mon[31360]: pgmap v12728: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:50 smithi022 ceph-mon[40250]: pgmap v12728: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:51 smithi022 ceph-mon[31360]: pgmap v12729: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:51 smithi022 ceph-mon[40250]: pgmap v12729: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:51 smithi122 ceph-mon[32255]: pgmap v12729: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:02:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:02:54.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:02:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:02:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:02:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:02:54.827 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:02:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:02:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:54 smithi022 ceph-mon[31360]: pgmap v12730: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2512357261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:54 smithi022 ceph-mon[40250]: pgmap v12730: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2512357261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:55.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:54 smithi122 ceph-mon[32255]: pgmap v12730: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:02:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2512357261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:02:56.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:02:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:56 smithi122 ceph-mon[32255]: pgmap v12731: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:56 smithi022 ceph-mon[31360]: pgmap v12731: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:56 smithi022 ceph-mon[40250]: pgmap v12731: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:02:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:02:58 smithi122 ceph-mon[32255]: pgmap v12732: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:02:58 smithi022 ceph-mon[31360]: pgmap v12732: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:02:58 smithi022 ceph-mon[40250]: pgmap v12732: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:02:59.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:00.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:00.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:00.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:00.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:00] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:03:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:00 smithi122 ceph-mon[32255]: pgmap v12733: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:03:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:03:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:03:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/676285293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:03:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[31360]: pgmap v12733: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:03:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:03:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:03:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/676285293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:03:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[40250]: pgmap v12733: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:03:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:03:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:03:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/676285293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:03:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:01 smithi022 ceph-mon[31360]: pgmap v12734: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:01 smithi022 ceph-mon[40250]: pgmap v12734: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:01 smithi122 ceph-mon[32255]: pgmap v12734: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:03:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:03:04.831 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:03:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:03:05.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:05.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:04 smithi022 ceph-mon[31360]: pgmap v12735: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:05.153 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:04 smithi022 ceph-mon[40250]: pgmap v12735: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:04 smithi122 ceph-mon[32255]: pgmap v12735: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:05.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:05.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:05.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/348013132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/348013132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/348013132' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:06 smithi122 ceph-mon[32255]: pgmap v12736: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:07.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:06 smithi022 ceph-mon[31360]: pgmap v12736: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:07.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:06 smithi022 ceph-mon[40250]: pgmap v12736: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:03:09.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:08 smithi122 ceph-mon[32255]: pgmap v12737: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:08 smithi022 ceph-mon[31360]: pgmap v12737: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:08 smithi022 ceph-mon[40250]: pgmap v12737: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:10] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:03:10.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:10.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:10.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:10.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:10 smithi122 ceph-mon[32255]: pgmap v12738: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:10 smithi022 ceph-mon[31360]: pgmap v12738: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:10 smithi022 ceph-mon[40250]: pgmap v12738: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/688651117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/688651117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/688651117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:03:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:03:13.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:12 smithi122 ceph-mon[32255]: pgmap v12739: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:12 smithi022 ceph-mon[31360]: pgmap v12739: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:12 smithi022 ceph-mon[40250]: pgmap v12739: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:14.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:03:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:03:15.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:14 smithi122 ceph-mon[32255]: pgmap v12740: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:14 smithi022 ceph-mon[31360]: pgmap v12740: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:14 smithi022 ceph-mon[40250]: pgmap v12740: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:15.895 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:16.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:16.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:16.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:16.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:16.263 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:16 smithi122 ceph-mon[32255]: pgmap v12741: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2969557065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:16 smithi022 ceph-mon[31360]: pgmap v12741: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2969557065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:16 smithi022 ceph-mon[40250]: pgmap v12741: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2969557065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:03:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:18 smithi122 ceph-mon[32255]: pgmap v12742: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:18 smithi022 ceph-mon[31360]: pgmap v12742: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:18 smithi022 ceph-mon[40250]: pgmap v12742: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:20] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:03:21.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:20 smithi122 ceph-mon[32255]: pgmap v12743: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:21.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:21.264 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:20 smithi022 ceph-mon[40250]: pgmap v12743: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:21.264 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:20 smithi022 ceph-mon[31360]: pgmap v12743: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:21.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:21.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:21.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2345763943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:22.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:21 smithi022 ceph-mon[31360]: pgmap v12744: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:22.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2345763943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:22.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:21 smithi022 ceph-mon[40250]: pgmap v12744: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2345763943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:21 smithi122 ceph-mon[32255]: pgmap v12744: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:03:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:03:24.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:03:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:03:25.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:24 smithi122 ceph-mon[32255]: pgmap v12745: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:24 smithi022 ceph-mon[31360]: pgmap v12745: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:24 smithi022 ceph-mon[40250]: pgmap v12745: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:27.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:27.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:27.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:27.003 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:26 smithi022 ceph-mon[31360]: pgmap v12746: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:27.004 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:26 smithi022 ceph-mon[40250]: pgmap v12746: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:26 smithi122 ceph-mon[32255]: pgmap v12746: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:03:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/88557979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/88557979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/88557979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:28 smithi122 ceph-mon[32255]: pgmap v12747: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:28 smithi022 ceph-mon[31360]: pgmap v12747: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:28 smithi022 ceph-mon[40250]: pgmap v12747: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:30] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:03:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:30 smithi122 ceph-mon[32255]: pgmap v12748: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:30 smithi022 ceph-mon[31360]: pgmap v12748: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:30 smithi022 ceph-mon[40250]: pgmap v12748: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:32.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:31 smithi022 ceph-mon[31360]: pgmap v12749: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:31 smithi022 ceph-mon[40250]: pgmap v12749: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:31 smithi122 ceph-mon[32255]: pgmap v12749: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:32.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:32.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:32.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:03:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:03:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/820479643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/820479643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/820479643' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:03:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:03:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:03:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:03:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[31360]: pgmap v12750: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:03:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:03:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:03:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:03:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:33 smithi022 ceph-mon[40250]: pgmap v12750: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:03:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:03:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:03:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:03:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:33 smithi122 ceph-mon[32255]: pgmap v12750: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:03:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:03:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:35 smithi022 ceph-mon[31360]: pgmap v12751: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:35 smithi022 ceph-mon[40250]: pgmap v12751: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:35 smithi122 ceph-mon[32255]: pgmap v12751: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:37.371 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:37.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:37.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:37.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:37.741 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:03:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1696372527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1696372527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1696372527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:38 smithi122 ceph-mon[32255]: pgmap v12752: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:38 smithi022 ceph-mon[40250]: pgmap v12752: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:38 smithi022 ceph-mon[31360]: pgmap v12752: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:40] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:03:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:40 smithi122 ceph-mon[32255]: pgmap v12753: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:40 smithi022 ceph-mon[31360]: pgmap v12753: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:40 smithi022 ceph-mon[40250]: pgmap v12753: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:41 smithi022 ceph-mon[31360]: pgmap v12754: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:41 smithi022 ceph-mon[40250]: pgmap v12754: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:41 smithi122 ceph-mon[32255]: pgmap v12754: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:03:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:03:42.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:43.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:43.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:43.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/828363437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/828363437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/828363437' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:44 smithi122 ceph-mon[32255]: pgmap v12755: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:44 smithi022 ceph-mon[31360]: pgmap v12755: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:44 smithi022 ceph-mon[40250]: pgmap v12755: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:44.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:03:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:03:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:46 smithi122 ceph-mon[32255]: pgmap v12756: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:46 smithi022 ceph-mon[31360]: pgmap v12756: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:46 smithi022 ceph-mon[40250]: pgmap v12756: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:03:48.110 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:48.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:48.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:48.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:48 smithi122 ceph-mon[32255]: pgmap v12757: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3532214405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:48 smithi022 ceph-mon[31360]: pgmap v12757: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3532214405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:48 smithi022 ceph-mon[40250]: pgmap v12757: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3532214405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:50] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:03:51.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:50 smithi122 ceph-mon[32255]: pgmap v12758: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:50 smithi022 ceph-mon[31360]: pgmap v12758: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:50 smithi022 ceph-mon[40250]: pgmap v12758: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:51 smithi022 ceph-mon[31360]: pgmap v12759: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:51 smithi022 ceph-mon[40250]: pgmap v12759: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:51 smithi122 ceph-mon[32255]: pgmap v12759: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:03:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:03:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:03:53.480 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:53.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:53.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:53.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:53 smithi122 ceph-mon[32255]: pgmap v12760: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3562307549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:53 smithi022 ceph-mon[31360]: pgmap v12760: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3562307549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:53 smithi022 ceph-mon[40250]: pgmap v12760: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:03:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3562307549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:03:54.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:03:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:03:55.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:03:56.212 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:55 smithi122 ceph-mon[32255]: pgmap v12761: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:55 smithi022 ceph-mon[31360]: pgmap v12761: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:55 smithi022 ceph-mon[40250]: pgmap v12761: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:03:58.853 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:03:59.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:58 smithi122 ceph-mon[32255]: pgmap v12762: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:59.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:03:59.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:03:59.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:03:59.224 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:58 smithi022 ceph-mon[31360]: pgmap v12762: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:03:59.224 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:58 smithi022 ceph-mon[40250]: pgmap v12762: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:03:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1035770647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:03:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1035770647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:03:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1035770647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:00] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:04:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:00 smithi122 ceph-mon[32255]: pgmap v12763: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:04:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:04:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:04:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:04:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[31360]: pgmap v12763: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:04:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:04:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:04:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:04:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[40250]: pgmap v12763: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:04:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:04:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:04:01.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:04:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:01 smithi022 ceph-mon[31360]: pgmap v12764: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:01 smithi022 ceph-mon[40250]: pgmap v12764: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:01 smithi122 ceph-mon[32255]: pgmap v12764: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:04:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:04:04.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:04.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:04.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:04.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:04.849 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:04:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:04:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:04 smithi122 ceph-mon[32255]: pgmap v12765: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4177850012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:04 smithi022 ceph-mon[31360]: pgmap v12765: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4177850012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:04 smithi022 ceph-mon[40250]: pgmap v12765: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4177850012' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:06 smithi122 ceph-mon[32255]: pgmap v12766: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:06 smithi022 ceph-mon[31360]: pgmap v12766: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:06 smithi022 ceph-mon[40250]: pgmap v12766: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:04:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:08 smithi122 ceph-mon[32255]: pgmap v12767: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:08 smithi022 ceph-mon[31360]: pgmap v12767: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:08 smithi022 ceph-mon[40250]: pgmap v12767: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:09.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:09.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:09.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:09.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:10] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:04:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:10 smithi122 ceph-mon[32255]: pgmap v12768: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/171887876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:10 smithi022 ceph-mon[31360]: pgmap v12768: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/171887876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:10 smithi022 ceph-mon[40250]: pgmap v12768: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/171887876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:11 smithi022 ceph-mon[31360]: pgmap v12769: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:11 smithi022 ceph-mon[40250]: pgmap v12769: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:12.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:11 smithi122 ceph-mon[32255]: pgmap v12769: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:04:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:04:14.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:04:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:04:14.966 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:14 smithi122 ceph-mon[32255]: pgmap v12770: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:15.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:14 smithi022 ceph-mon[31360]: pgmap v12770: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:15.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:14 smithi022 ceph-mon[40250]: pgmap v12770: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:15.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:15.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:15.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:16.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/899803772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/899803772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/899803772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:16 smithi122 ceph-mon[32255]: pgmap v12771: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:16 smithi022 ceph-mon[31360]: pgmap v12771: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:16 smithi022 ceph-mon[40250]: pgmap v12771: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:04:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:18 smithi122 ceph-mon[32255]: pgmap v12772: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:18 smithi022 ceph-mon[31360]: pgmap v12772: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:18 smithi022 ceph-mon[40250]: pgmap v12772: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:20.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:20] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:04:20.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:20.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:20.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:20 smithi022 ceph-mon[31360]: pgmap v12773: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2731625536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:20 smithi022 ceph-mon[40250]: pgmap v12773: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2731625536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:20 smithi122 ceph-mon[32255]: pgmap v12773: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:21.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2731625536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:21 smithi022 ceph-mon[31360]: pgmap v12774: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:21 smithi022 ceph-mon[40250]: pgmap v12774: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:21 smithi122 ceph-mon[32255]: pgmap v12774: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:04:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:04:24.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:04:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:04:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:24 smithi122 ceph-mon[32255]: pgmap v12775: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:24 smithi022 ceph-mon[31360]: pgmap v12775: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:24 smithi022 ceph-mon[40250]: pgmap v12775: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:25.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:26.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:26.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:26.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:27.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:26 smithi122 ceph-mon[32255]: pgmap v12776: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/394776881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:26 smithi022 ceph-mon[31360]: pgmap v12776: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/394776881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:26 smithi022 ceph-mon[40250]: pgmap v12776: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/394776881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:04:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:28 smithi122 ceph-mon[32255]: pgmap v12777: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:28 smithi022 ceph-mon[31360]: pgmap v12777: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:28 smithi022 ceph-mon[40250]: pgmap v12777: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:30] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:04:31.084 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:30 smithi122 ceph-mon[32255]: pgmap v12778: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:30 smithi022 ceph-mon[31360]: pgmap v12778: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:30 smithi022 ceph-mon[40250]: pgmap v12778: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:31.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:31.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:31.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3669308783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:31 smithi022 ceph-mon[31360]: pgmap v12779: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3669308783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:31 smithi022 ceph-mon[40250]: pgmap v12779: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3669308783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:31 smithi122 ceph-mon[32255]: pgmap v12779: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:04:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:04:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:04:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:04:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:04:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:04:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:04:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:04:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:04:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:04:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:04:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:04:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:04:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:04:34.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:04:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:04:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:34 smithi122 ceph-mon[32255]: pgmap v12780: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:34 smithi022 ceph-mon[31360]: pgmap v12780: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:34 smithi022 ceph-mon[40250]: pgmap v12780: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:36.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:36.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:36.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:36.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:36 smithi122 ceph-mon[32255]: pgmap v12781: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1191349279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:36 smithi022 ceph-mon[31360]: pgmap v12781: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1191349279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:36 smithi022 ceph-mon[40250]: pgmap v12781: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1191349279' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:04:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:38 smithi122 ceph-mon[32255]: pgmap v12782: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:39.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:38 smithi022 ceph-mon[31360]: pgmap v12782: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:39.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:38 smithi022 ceph-mon[40250]: pgmap v12782: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:40] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:04:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:40 smithi122 ceph-mon[32255]: pgmap v12783: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:40 smithi022 ceph-mon[31360]: pgmap v12783: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:40 smithi022 ceph-mon[40250]: pgmap v12783: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:41.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:42.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:42.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:42.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:42.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:41 smithi022 ceph-mon[31360]: pgmap v12784: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:42.203 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:41 smithi022 ceph-mon[40250]: pgmap v12784: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:41 smithi122 ceph-mon[32255]: pgmap v12784: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:04:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:04:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2549357268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2549357268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2549357268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:44 smithi022 ceph-mon[31360]: pgmap v12785: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:44 smithi022 ceph-mon[40250]: pgmap v12785: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:44 smithi122 ceph-mon[32255]: pgmap v12785: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:44.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:04:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:04:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:46 smithi022 ceph-mon[31360]: pgmap v12786: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:46 smithi022 ceph-mon[40250]: pgmap v12786: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:46 smithi122 ceph-mon[32255]: pgmap v12786: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:47.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:04:47.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:47.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:47.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:47.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3374731754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3374731754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3374731754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:48.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:48 smithi122 ceph-mon[32255]: pgmap v12787: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:48 smithi022 ceph-mon[31360]: pgmap v12787: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:48 smithi022 ceph-mon[40250]: pgmap v12787: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:50.452 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:50] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:04:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:50 smithi122 ceph-mon[32255]: pgmap v12788: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:50 smithi022 ceph-mon[31360]: pgmap v12788: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:51.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:50 smithi022 ceph-mon[40250]: pgmap v12788: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:51 smithi022 ceph-mon[31360]: pgmap v12789: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:51 smithi022 ceph-mon[40250]: pgmap v12789: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:52.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:51 smithi122 ceph-mon[32255]: pgmap v12789: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:04:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:04:52.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:52.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:52.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1959238840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1959238840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1959238840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:54 smithi022 ceph-mon[31360]: pgmap v12790: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:54 smithi022 ceph-mon[40250]: pgmap v12790: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:54 smithi122 ceph-mon[32255]: pgmap v12790: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:04:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:04:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:04:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:04:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:56 smithi022 ceph-mon[31360]: pgmap v12791: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:56 smithi022 ceph-mon[40250]: pgmap v12791: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:56 smithi122 ceph-mon[32255]: pgmap v12791: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:04:57.943 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:04:58.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:04:58.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:04:58.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:04:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:58 smithi122 ceph-mon[32255]: pgmap v12792: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:04:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/541003673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:58 smithi022 ceph-mon[31360]: pgmap v12792: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:04:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/541003673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:04:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:58 smithi022 ceph-mon[40250]: pgmap v12792: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:04:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:04:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/541003673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:00] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:05:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:00 smithi122 ceph-mon[32255]: pgmap v12793: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:05:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:05:01.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:05:01.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[31360]: pgmap v12793: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:01.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:05:01.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:05:01.269 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:05:01.269 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[40250]: pgmap v12793: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:01.270 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:05:01.270 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:05:01.270 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:05:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:05:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:02 smithi122 ceph-mon[32255]: pgmap v12794: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:05:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:05:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:05:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:02 smithi022 ceph-mon[31360]: pgmap v12794: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:05:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:02 smithi022 ceph-mon[40250]: pgmap v12794: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:03.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:03.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:04.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1136029041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:04.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1136029041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:04.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1136029041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:04.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:04 smithi122 ceph-mon[32255]: pgmap v12795: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:04.940 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:05:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:05:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:04 smithi022 ceph-mon[31360]: pgmap v12795: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:04 smithi022 ceph-mon[40250]: pgmap v12795: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:06.043 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:06.044 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:06 smithi022 ceph-mon[31360]: pgmap v12796: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:06 smithi022 ceph-mon[40250]: pgmap v12796: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:06 smithi122 ceph-mon[32255]: pgmap v12796: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:05:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:08 smithi022 ceph-mon[31360]: pgmap v12797: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:08 smithi022 ceph-mon[40250]: pgmap v12797: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:09.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:09.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:09.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:08 smithi122 ceph-mon[32255]: pgmap v12797: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:10.182 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/652696966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/652696966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/652696966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:05:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:10 smithi122 ceph-mon[32255]: pgmap v12798: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:10 smithi022 ceph-mon[31360]: pgmap v12798: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:10 smithi022 ceph-mon[40250]: pgmap v12798: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:12 smithi022 ceph-mon[31360]: pgmap v12799: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:12 smithi022 ceph-mon[40250]: pgmap v12799: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:12 smithi122 ceph-mon[32255]: pgmap v12799: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:12.697 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:05:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:05:14.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:14.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:14.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:14.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:14.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:05:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:05:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:14 smithi122 ceph-mon[32255]: pgmap v12800: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:15.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2336443476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:14 smithi022 ceph-mon[40250]: pgmap v12800: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2336443476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:14 smithi022 ceph-mon[31360]: pgmap v12800: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2336443476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:16 smithi122 ceph-mon[32255]: pgmap v12801: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:16 smithi022 ceph-mon[31360]: pgmap v12801: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:16 smithi022 ceph-mon[40250]: pgmap v12801: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:05:19.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:18 smithi122 ceph-mon[32255]: pgmap v12802: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:18 smithi022 ceph-mon[31360]: pgmap v12802: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:18 smithi022 ceph-mon[40250]: pgmap v12802: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:19.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:19.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:19.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:19.791 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:20.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1776935304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1776935304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1776935304' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:20] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:05:21.197 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:20 smithi122 ceph-mon[32255]: pgmap v12803: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:20 smithi022 ceph-mon[31360]: pgmap v12803: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:20 smithi022 ceph-mon[40250]: pgmap v12803: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:22.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:22 smithi022 ceph-mon[31360]: pgmap v12804: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:22.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:22 smithi022 ceph-mon[40250]: pgmap v12804: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:22 smithi122 ceph-mon[32255]: pgmap v12804: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:05:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:05:24.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:24.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:05:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:05:25.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:25.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:25.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:25.161 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:24 smithi022 ceph-mon[40250]: pgmap v12805: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:25.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:24 smithi022 ceph-mon[31360]: pgmap v12805: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:25.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:24 smithi122 ceph-mon[32255]: pgmap v12805: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/905990765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/905990765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/905990765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:26 smithi122 ceph-mon[32255]: pgmap v12806: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:26 smithi022 ceph-mon[31360]: pgmap v12806: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:26 smithi022 ceph-mon[40250]: pgmap v12806: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:05:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:28 smithi122 ceph-mon[32255]: pgmap v12807: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:28 smithi022 ceph-mon[31360]: pgmap v12807: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:28 smithi022 ceph-mon[40250]: pgmap v12807: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:30.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:30] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:05:30.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:30.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:30.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:30 smithi122 ceph-mon[32255]: pgmap v12808: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1627118777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:30 smithi022 ceph-mon[31360]: pgmap v12808: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1627118777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:30 smithi022 ceph-mon[40250]: pgmap v12808: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1627118777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:32 smithi022 ceph-mon[31360]: pgmap v12809: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:32 smithi022 ceph-mon[40250]: pgmap v12809: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:32 smithi122 ceph-mon[32255]: pgmap v12809: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:05:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:05:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:05:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:05:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:05:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:05:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:05:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:05:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:05:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:05:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:05:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:05:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:05:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:05:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:05:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:05:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:34 smithi122 ceph-mon[32255]: pgmap v12810: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:34 smithi022 ceph-mon[31360]: pgmap v12810: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:34 smithi022 ceph-mon[40250]: pgmap v12810: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:35.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:35.829 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:35.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:35.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:35.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:36 smithi122 ceph-mon[32255]: pgmap v12811: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2040709269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:36 smithi022 ceph-mon[31360]: pgmap v12811: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2040709269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:36 smithi022 ceph-mon[40250]: pgmap v12811: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2040709269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:05:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:38 smithi122 ceph-mon[32255]: pgmap v12812: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:38 smithi022 ceph-mon[31360]: pgmap v12812: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:38 smithi022 ceph-mon[40250]: pgmap v12812: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:40] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:05:40.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:40 smithi122 ceph-mon[32255]: pgmap v12813: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:41.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:41.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:41.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:41.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:40 smithi022 ceph-mon[40250]: pgmap v12813: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:40 smithi022 ceph-mon[31360]: pgmap v12813: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1257356988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:42 smithi022 ceph-mon[31360]: pgmap v12814: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1257356988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:42 smithi022 ceph-mon[40250]: pgmap v12814: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:42.320 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1257356988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:42.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:42 smithi122 ceph-mon[32255]: pgmap v12814: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:42.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:05:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:05:44.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:05:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:05:45.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:44 smithi122 ceph-mon[32255]: pgmap v12815: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:44 smithi022 ceph-mon[31360]: pgmap v12815: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:44 smithi022 ceph-mon[40250]: pgmap v12815: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:46.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:46.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:46.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:46.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:46.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:46.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:46 smithi022 ceph-mon[31360]: pgmap v12816: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2933690438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:46 smithi022 ceph-mon[40250]: pgmap v12816: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2933690438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:46 smithi122 ceph-mon[32255]: pgmap v12816: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2933690438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:05:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:48 smithi122 ceph-mon[32255]: pgmap v12817: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:48 smithi022 ceph-mon[31360]: pgmap v12817: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:48 smithi022 ceph-mon[40250]: pgmap v12817: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:50] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:05:51.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:50 smithi122 ceph-mon[32255]: pgmap v12818: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:50 smithi022 ceph-mon[40250]: pgmap v12818: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:50 smithi022 ceph-mon[31360]: pgmap v12818: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:51.640 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:52.008 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:52 smithi022 ceph-mon[31360]: pgmap v12819: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:52.008 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1994416676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:52.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:52.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:52 smithi022 ceph-mon[40250]: pgmap v12819: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1994416676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:52 smithi122 ceph-mon[32255]: pgmap v12819: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1994416676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:05:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:05:54.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:05:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:05:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:54 smithi122 ceph-mon[32255]: pgmap v12820: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:54 smithi022 ceph-mon[31360]: pgmap v12820: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:54 smithi022 ceph-mon[40250]: pgmap v12820: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:05:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:05:57.013 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:05:57.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:56 smithi122 ceph-mon[32255]: pgmap v12821: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:56 smithi022 ceph-mon[31360]: pgmap v12821: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:56 smithi022 ceph-mon[40250]: pgmap v12821: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:57.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:05:57.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:05:57.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:05:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:05:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1741411698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1741411698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1741411698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:05:59.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:05:58 smithi122 ceph-mon[32255]: pgmap v12822: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:05:58 smithi022 ceph-mon[31360]: pgmap v12822: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:05:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:05:58 smithi022 ceph-mon[40250]: pgmap v12822: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:00.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:00] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:06:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:00 smithi122 ceph-mon[32255]: pgmap v12823: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:01.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:00 smithi022 ceph-mon[31360]: pgmap v12823: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:01.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:00 smithi022 ceph-mon[40250]: pgmap v12823: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:06:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:06:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:06:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[31360]: pgmap v12824: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:06:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:06:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:06:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:02 smithi022 ceph-mon[40250]: pgmap v12824: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:06:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:06:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:06:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:02 smithi122 ceph-mon[32255]: pgmap v12824: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:02.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:06:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:06:02.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:02.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:02.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:03.011 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3688337587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3688337587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:03.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3688337587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:04 smithi022 ceph-mon[31360]: pgmap v12825: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:04 smithi022 ceph-mon[40250]: pgmap v12825: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:04 smithi122 ceph-mon[32255]: pgmap v12825: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:04.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:06:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:06:05.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:05.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:05.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:06 smithi122 ceph-mon[32255]: pgmap v12826: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.998 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.998 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[31360]: pgmap v12826: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:06.998 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.999 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.999 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:06.999 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[40250]: pgmap v12826: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:07.000 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:07.000 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:07.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:07.759 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:06:08.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:08.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:08.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:09.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: pgmap v12827: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:09.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:09.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.181 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.182 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2977732513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:09.183 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:09.331 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: pgmap v12827: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:09.331 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:09.332 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.332 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.332 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.333 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.333 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2977732513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:09.334 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.334 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.334 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.335 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.335 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:09.336 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: pgmap v12827: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:09.336 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:09.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.337 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.338 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2977732513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:09.338 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.338 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.338 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.338 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:09.339 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3042M 2021-12-16T01:06:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3042M 2021-12-16T01:06:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3042M 2021-12-16T01:06:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:06:10.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:06:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:10] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:06:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:10 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3254M 2021-12-16T01:06:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:10 smithi122 ceph-mon[32255]: pgmap v12828: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:10 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3254M 2021-12-16T01:06:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:10 smithi022 ceph-mon[31360]: pgmap v12828: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:10 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3254M 2021-12-16T01:06:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:10 smithi022 ceph-mon[40250]: pgmap v12828: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:12 smithi022 ceph-mon[31360]: pgmap v12829: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:12 smithi022 ceph-mon[40250]: pgmap v12829: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:12 smithi122 ceph-mon[32255]: pgmap v12829: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:06:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:06:13.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:13.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:13.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:13.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/855236945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/855236945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/855236945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:14 smithi122 ceph-mon[32255]: pgmap v12830: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:14.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:06:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:06:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:14 smithi022 ceph-mon[31360]: pgmap v12830: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:14 smithi022 ceph-mon[40250]: pgmap v12830: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:15.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:16 smithi122 ceph-mon[32255]: pgmap v12831: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:16 smithi022 ceph-mon[31360]: pgmap v12831: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:16 smithi022 ceph-mon[40250]: pgmap v12831: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:06:18.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:18.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:18.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:18.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:18 smithi122 ceph-mon[32255]: pgmap v12832: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1220418714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:18 smithi022 ceph-mon[31360]: pgmap v12832: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1220418714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:18 smithi022 ceph-mon[40250]: pgmap v12832: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1220418714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:20] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:06:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:20 smithi122 ceph-mon[32255]: pgmap v12833: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:20 smithi022 ceph-mon[31360]: pgmap v12833: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:20 smithi022 ceph-mon[40250]: pgmap v12833: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:22 smithi022 ceph-mon[31360]: pgmap v12834: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:22 smithi022 ceph-mon[40250]: pgmap v12834: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:22 smithi122 ceph-mon[32255]: pgmap v12834: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:06:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:06:23.888 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:24.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:24.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:24.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:24.891 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:06:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:06:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:24 smithi122 ceph-mon[32255]: pgmap v12835: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2114526708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:24 smithi022 ceph-mon[31360]: pgmap v12835: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2114526708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:24 smithi022 ceph-mon[40250]: pgmap v12835: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2114526708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:26 smithi122 ceph-mon[32255]: pgmap v12836: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:26 smithi022 ceph-mon[31360]: pgmap v12836: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:26 smithi022 ceph-mon[40250]: pgmap v12836: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:06:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:28 smithi122 ceph-mon[32255]: pgmap v12837: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:29.253 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:29.254 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:28 smithi022 ceph-mon[40250]: pgmap v12837: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:29.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:28 smithi022 ceph-mon[31360]: pgmap v12837: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:29.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:29.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:29.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1715986331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1715986331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1715986331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:30] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:06:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:30 smithi122 ceph-mon[32255]: pgmap v12838: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:30 smithi022 ceph-mon[31360]: pgmap v12838: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:30 smithi022 ceph-mon[40250]: pgmap v12838: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:32 smithi022 ceph-mon[31360]: pgmap v12839: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:32 smithi022 ceph-mon[40250]: pgmap v12839: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:32 smithi122 ceph-mon[32255]: pgmap v12839: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:06:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:06:34.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:06:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:06:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:06:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:06:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:06:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:06:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:06:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:06:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:06:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:06:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:06:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:06:34.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:34 smithi122 ceph-mon[32255]: pgmap v12840: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:34.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:06:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:06:34.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:34.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:34.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:34.997 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:34 smithi022 ceph-mon[40250]: pgmap v12840: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:34.997 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:34 smithi022 ceph-mon[31360]: pgmap v12840: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:36.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:36.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/451600725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/451600725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/451600725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:36 smithi122 ceph-mon[32255]: pgmap v12841: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:36 smithi022 ceph-mon[31360]: pgmap v12841: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:36 smithi022 ceph-mon[40250]: pgmap v12841: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:06:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:38 smithi122 ceph-mon[32255]: pgmap v12842: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:38 smithi022 ceph-mon[31360]: pgmap v12842: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:38 smithi022 ceph-mon[40250]: pgmap v12842: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:39.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:40.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:40.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:40.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:40] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:06:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:40 smithi122 ceph-mon[32255]: pgmap v12843: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3165712536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:40 smithi022 ceph-mon[31360]: pgmap v12843: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3165712536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:40 smithi022 ceph-mon[40250]: pgmap v12843: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3165712536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:42 smithi022 ceph-mon[31360]: pgmap v12844: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:42 smithi022 ceph-mon[40250]: pgmap v12844: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:42 smithi122 ceph-mon[32255]: pgmap v12844: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:42.698 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:06:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:06:44.897 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:06:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:06:45.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:44 smithi122 ceph-mon[32255]: pgmap v12845: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:44 smithi022 ceph-mon[31360]: pgmap v12845: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:44 smithi022 ceph-mon[40250]: pgmap v12845: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:45.366 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:45.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:45.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:45.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2627139116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2627139116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2627139116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:47.203 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:46 smithi122 ceph-mon[32255]: pgmap v12846: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:46 smithi022 ceph-mon[31360]: pgmap v12846: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:46 smithi022 ceph-mon[40250]: pgmap v12846: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:06:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:48 smithi122 ceph-mon[32255]: pgmap v12847: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:48 smithi022 ceph-mon[31360]: pgmap v12847: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:48 smithi022 ceph-mon[40250]: pgmap v12847: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:49 smithi022 ceph-mon[31360]: pgmap v12848: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:49 smithi022 ceph-mon[40250]: pgmap v12848: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:50] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:06:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:49 smithi122 ceph-mon[32255]: pgmap v12848: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:50.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:51.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:51.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:51.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3156185586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:52 smithi022 ceph-mon[31360]: pgmap v12849: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3156185586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:52 smithi022 ceph-mon[40250]: pgmap v12849: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3156185586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:52 smithi122 ceph-mon[32255]: pgmap v12849: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:06:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:06:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:06:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:06:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:54 smithi122 ceph-mon[32255]: pgmap v12850: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:54 smithi022 ceph-mon[31360]: pgmap v12850: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:54 smithi022 ceph-mon[40250]: pgmap v12850: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:06:56.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:06:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:06:56.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:06:56.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:06:56.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:06:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:56 smithi122 ceph-mon[32255]: pgmap v12851: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4107548581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:56 smithi022 ceph-mon[31360]: pgmap v12851: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4107548581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:56 smithi022 ceph-mon[40250]: pgmap v12851: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4107548581' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:06:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:06:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:58 smithi122 ceph-mon[32255]: pgmap v12852: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:59.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:58 smithi022 ceph-mon[31360]: pgmap v12852: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:06:59.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:58 smithi022 ceph-mon[40250]: pgmap v12852: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:00.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:06:59 smithi122 ceph-mon[32255]: pgmap v12853: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:06:59 smithi022 ceph-mon[31360]: pgmap v12853: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:06:59 smithi022 ceph-mon[40250]: pgmap v12853: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:00] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:07:01.480 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:01.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:01.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:01.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1790571049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:02 smithi022 ceph-mon[31360]: pgmap v12854: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1790571049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:02 smithi022 ceph-mon[40250]: pgmap v12854: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1790571049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:02 smithi122 ceph-mon[32255]: pgmap v12854: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:07:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:07:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:04 smithi122 ceph-mon[32255]: pgmap v12855: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:04.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:07:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:07:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:04 smithi022 ceph-mon[31360]: pgmap v12855: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:04 smithi022 ceph-mon[40250]: pgmap v12855: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:06.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:06 smithi122 ceph-mon[32255]: pgmap v12856: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:07.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:07.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:07.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:07.227 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:06 smithi022 ceph-mon[31360]: pgmap v12856: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:07.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:06 smithi022 ceph-mon[40250]: pgmap v12856: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:07:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3689755382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:07 smithi122 ceph-mon[32255]: pgmap v12857: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3689755382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:07 smithi022 ceph-mon[31360]: pgmap v12857: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3689755382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:07 smithi022 ceph-mon[40250]: pgmap v12857: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:07:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:07:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:07:09.713 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:07:09.714 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:07:09.714 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:07:09.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:07:09.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:07:09.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:07:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:10] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:07:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:07:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:10 smithi022 ceph-mon[31360]: pgmap v12858: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:07:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:10 smithi022 ceph-mon[40250]: pgmap v12858: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:07:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:10 smithi122 ceph-mon[32255]: pgmap v12858: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:12.226 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:12.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:12 smithi022 ceph-mon[31360]: pgmap v12859: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:12.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:12 smithi022 ceph-mon[40250]: pgmap v12859: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:12 smithi122 ceph-mon[32255]: pgmap v12859: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:12.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:12.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:12.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:07:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:07:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3414403251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3414403251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3414403251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:14.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:14 smithi122 ceph-mon[32255]: pgmap v12860: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:14 smithi022 ceph-mon[31360]: pgmap v12860: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:14 smithi022 ceph-mon[40250]: pgmap v12860: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:14.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:07:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:07:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:16 smithi122 ceph-mon[32255]: pgmap v12861: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:16 smithi022 ceph-mon[31360]: pgmap v12861: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:16 smithi022 ceph-mon[40250]: pgmap v12861: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:17.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:07:17.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:17.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:17.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:18 smithi122 ceph-mon[32255]: pgmap v12862: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2353017571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:18 smithi022 ceph-mon[31360]: pgmap v12862: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2353017571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:18 smithi022 ceph-mon[40250]: pgmap v12862: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2353017571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:07:21.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:20 smithi122 ceph-mon[32255]: pgmap v12863: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:20 smithi022 ceph-mon[31360]: pgmap v12863: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:20 smithi022 ceph-mon[40250]: pgmap v12863: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:22 smithi022 ceph-mon[31360]: pgmap v12864: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:22 smithi022 ceph-mon[40250]: pgmap v12864: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:22 smithi122 ceph-mon[32255]: pgmap v12864: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:07:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:07:22.976 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:23.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:23.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:23.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/536019172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/536019172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/536019172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:24.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:24 smithi122 ceph-mon[32255]: pgmap v12865: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:24 smithi022 ceph-mon[31360]: pgmap v12865: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:24 smithi022 ceph-mon[40250]: pgmap v12865: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:24.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:07:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:07:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:25.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:25.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:26 smithi122 ceph-mon[32255]: pgmap v12866: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:26 smithi022 ceph-mon[31360]: pgmap v12866: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:26 smithi022 ceph-mon[40250]: pgmap v12866: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:07:28.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:28.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:28.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:28.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:28 smithi022 ceph-mon[31360]: pgmap v12867: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/136916645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:28 smithi022 ceph-mon[40250]: pgmap v12867: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/136916645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:28 smithi122 ceph-mon[32255]: pgmap v12867: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/136916645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:07:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:30 smithi122 ceph-mon[32255]: pgmap v12868: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:30 smithi022 ceph-mon[31360]: pgmap v12868: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:30 smithi022 ceph-mon[40250]: pgmap v12868: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:32 smithi022 ceph-mon[31360]: pgmap v12869: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:32 smithi022 ceph-mon[40250]: pgmap v12869: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:32 smithi122 ceph-mon[32255]: pgmap v12869: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:07:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:07:33.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:07:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:07:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:07:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:07:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:07:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:07:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:07:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:07:34.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:34.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:34.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:07:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:07:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:07:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:07:34.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:07:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:07:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:34 smithi122 ceph-mon[32255]: pgmap v12870: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4274708282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:34 smithi022 ceph-mon[31360]: pgmap v12870: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4274708282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:34 smithi022 ceph-mon[40250]: pgmap v12870: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4274708282' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:36.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:36 smithi122 ceph-mon[32255]: pgmap v12871: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:36 smithi022 ceph-mon[31360]: pgmap v12871: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:36 smithi022 ceph-mon[40250]: pgmap v12871: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:07:39.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:38 smithi122 ceph-mon[32255]: pgmap v12872: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:38 smithi022 ceph-mon[31360]: pgmap v12872: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:38 smithi022 ceph-mon[40250]: pgmap v12872: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:39.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:39.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4247707675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4247707675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4247707675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:40.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:40] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:07:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:40 smithi122 ceph-mon[32255]: pgmap v12873: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:40 smithi022 ceph-mon[31360]: pgmap v12873: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:40 smithi022 ceph-mon[40250]: pgmap v12873: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:42.322 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:42 smithi122 ceph-mon[32255]: pgmap v12874: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:42 smithi022 ceph-mon[31360]: pgmap v12874: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:42 smithi022 ceph-mon[40250]: pgmap v12874: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:07:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:07:44.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:44.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:44.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:44.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:44.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:07:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:07:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:44 smithi122 ceph-mon[32255]: pgmap v12875: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3650306932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:44 smithi022 ceph-mon[31360]: pgmap v12875: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3650306932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:44 smithi022 ceph-mon[40250]: pgmap v12875: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3650306932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:46 smithi122 ceph-mon[32255]: pgmap v12876: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:46 smithi022 ceph-mon[31360]: pgmap v12876: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:46 smithi022 ceph-mon[40250]: pgmap v12876: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:07:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:47 smithi122 ceph-mon[32255]: pgmap v12877: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:47 smithi022 ceph-mon[31360]: pgmap v12877: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:47 smithi022 ceph-mon[40250]: pgmap v12877: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:49.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:50.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:50.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:50.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:50] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:07:51.195 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:50 smithi122 ceph-mon[32255]: pgmap v12878: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:51.196 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2785065806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:50 smithi022 ceph-mon[31360]: pgmap v12878: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2785065806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:50 smithi022 ceph-mon[40250]: pgmap v12878: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2785065806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:52.326 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:52 smithi122 ceph-mon[32255]: pgmap v12879: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:52 smithi022 ceph-mon[31360]: pgmap v12879: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:52 smithi022 ceph-mon[40250]: pgmap v12879: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:07:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:07:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:54 smithi122 ceph-mon[32255]: pgmap v12880: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:54.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:07:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:07:55.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:07:55.202 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:54 smithi022 ceph-mon[31360]: pgmap v12880: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:55.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:54 smithi022 ceph-mon[40250]: pgmap v12880: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:07:55.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:07:55.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:07:55.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:07:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3944871118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:55 smithi122 ceph-mon[32255]: pgmap v12881: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3944871118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:55 smithi022 ceph-mon[31360]: pgmap v12881: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:07:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3944871118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:07:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:55 smithi022 ceph-mon[40250]: pgmap v12881: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:07:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:57 smithi022 ceph-mon[31360]: pgmap v12882: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:57 smithi022 ceph-mon[40250]: pgmap v12882: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:07:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:57 smithi122 ceph-mon[32255]: pgmap v12882: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:07:59 smithi022 ceph-mon[31360]: pgmap v12883: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:07:59 smithi022 ceph-mon[40250]: pgmap v12883: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:07:59 smithi122 ceph-mon[32255]: pgmap v12883: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:00] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:08:00.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:00.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:00.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:00.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/630392251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/630392251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/630392251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:02 smithi122 ceph-mon[32255]: pgmap v12884: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:02.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:02 smithi022 ceph-mon[31360]: pgmap v12884: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:02.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:02 smithi022 ceph-mon[40250]: pgmap v12884: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:08:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:08:04.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:03 smithi022 ceph-mon[31360]: pgmap v12885: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:04.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:03 smithi022 ceph-mon[40250]: pgmap v12885: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:03 smithi122 ceph-mon[32255]: pgmap v12885: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:04.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:08:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:08:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:05.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:05 smithi022 ceph-mon[31360]: pgmap v12886: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:05 smithi022 ceph-mon[40250]: pgmap v12886: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:06.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:06.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:06.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:05 smithi122 ceph-mon[32255]: pgmap v12886: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1444345668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:07.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1444345668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1444345668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:07.796 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:08:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:08 smithi122 ceph-mon[32255]: pgmap v12887: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:08.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:08 smithi022 ceph-mon[40250]: pgmap v12887: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:08.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:08 smithi022 ceph-mon[31360]: pgmap v12887: 132 pgs: 132 active+clean; 582 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:08:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:08:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:08:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:08:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:08:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:08:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:08:10.156 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:08:10.157 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:08:10.157 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:08:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:10] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:08:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:10 smithi122 ceph-mon[32255]: pgmap v12888: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:08:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:08:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:10 smithi022 ceph-mon[31360]: pgmap v12888: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:08:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:08:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:10 smithi022 ceph-mon[40250]: pgmap v12888: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:08:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:08:11.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:11.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:11.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:11.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:12.323 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2058426327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:12.324 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:12 smithi122 ceph-mon[32255]: pgmap v12889: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2058426327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:12 smithi022 ceph-mon[31360]: pgmap v12889: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2058426327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:12 smithi022 ceph-mon[40250]: pgmap v12889: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:08:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:08:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:13 smithi022 ceph-mon[31360]: pgmap v12890: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:08:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:13 smithi022 ceph-mon[40250]: pgmap v12890: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:08:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:13 smithi122 ceph-mon[32255]: pgmap v12890: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:08:14.976 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:08:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:08:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:15.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:15 smithi022 ceph-mon[31360]: pgmap v12891: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:15 smithi022 ceph-mon[40250]: pgmap v12891: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:15 smithi122 ceph-mon[32255]: pgmap v12891: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:16.695 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187717106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187717106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187717106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:08:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:18 smithi122 ceph-mon[32255]: pgmap v12892: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:18 smithi022 ceph-mon[31360]: pgmap v12892: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:18 smithi022 ceph-mon[40250]: pgmap v12892: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:08:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:19 smithi022 ceph-mon[31360]: pgmap v12893: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:08:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:19 smithi022 ceph-mon[40250]: pgmap v12893: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:08:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:08:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:19 smithi122 ceph-mon[32255]: pgmap v12893: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:08:22.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:22 smithi122 ceph-mon[32255]: pgmap v12894: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:22.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:22.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:22.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:22.423 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:22 smithi022 ceph-mon[31360]: pgmap v12894: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:22.423 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:22 smithi022 ceph-mon[40250]: pgmap v12894: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:22.695 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:08:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:08:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1383855223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:23.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1383855223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:23.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1383855223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:24 smithi122 ceph-mon[32255]: pgmap v12895: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:24.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:24 smithi022 ceph-mon[31360]: pgmap v12895: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:24.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:24 smithi022 ceph-mon[40250]: pgmap v12895: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:25.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:08:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:08:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:26 smithi122 ceph-mon[32255]: pgmap v12896: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:26.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:26 smithi022 ceph-mon[31360]: pgmap v12896: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:26.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:26 smithi022 ceph-mon[40250]: pgmap v12896: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:27.424 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:08:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2179841709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2179841709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2179841709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:28 smithi122 ceph-mon[32255]: pgmap v12897: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:28 smithi022 ceph-mon[31360]: pgmap v12897: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:28 smithi022 ceph-mon[40250]: pgmap v12897: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:29 smithi022 ceph-mon[31360]: pgmap v12898: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:29 smithi022 ceph-mon[40250]: pgmap v12898: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:08:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:29 smithi122 ceph-mon[32255]: pgmap v12898: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:32 smithi122 ceph-mon[32255]: pgmap v12899: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:32 smithi022 ceph-mon[31360]: pgmap v12899: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:32 smithi022 ceph-mon[40250]: pgmap v12899: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:08:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:08:32.796 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:33.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:33.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:33.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/316853545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/316853545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/316853545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:08:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:08:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:08:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:08:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[31360]: pgmap v12900: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:08:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:34 smithi022 ceph-mon[40250]: pgmap v12900: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:08:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:08:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:08:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:08:34.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:34 smithi122 ceph-mon[32255]: pgmap v12900: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:35.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:08:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:08:35.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:35.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:36 smithi022 ceph-mon[31360]: pgmap v12901: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:36 smithi022 ceph-mon[40250]: pgmap v12901: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:36 smithi122 ceph-mon[32255]: pgmap v12901: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:08:38.166 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:37 smithi022 ceph-mon[31360]: pgmap v12902: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:37 smithi022 ceph-mon[40250]: pgmap v12902: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:37 smithi122 ceph-mon[32255]: pgmap v12902: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:38.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:38.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:38.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3508581228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3508581228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3508581228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:39 smithi022 ceph-mon[31360]: pgmap v12903: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:39 smithi022 ceph-mon[40250]: pgmap v12903: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:39 smithi122 ceph-mon[32255]: pgmap v12903: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:40] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:08:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:42 smithi122 ceph-mon[32255]: pgmap v12904: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:42 smithi022 ceph-mon[31360]: pgmap v12904: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:42 smithi022 ceph-mon[40250]: pgmap v12904: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:08:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:08:43.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:43.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:43.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:43.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:43 smithi022 ceph-mon[31360]: pgmap v12905: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3868754066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:43 smithi022 ceph-mon[40250]: pgmap v12905: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3868754066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:43 smithi122 ceph-mon[32255]: pgmap v12905: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3868754066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:08:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:08:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:45 smithi022 ceph-mon[31360]: pgmap v12906: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:45 smithi022 ceph-mon[40250]: pgmap v12906: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:45 smithi122 ceph-mon[32255]: pgmap v12906: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:08:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:47 smithi022 ceph-mon[31360]: pgmap v12907: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:47 smithi022 ceph-mon[40250]: pgmap v12907: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:47 smithi122 ceph-mon[32255]: pgmap v12907: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:48.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:49.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:49.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:49.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/433448389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/433448389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/433448389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:50 smithi122 ceph-mon[32255]: pgmap v12908: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:50] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T01:08:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:50 smithi022 ceph-mon[31360]: pgmap v12908: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:50 smithi022 ceph-mon[40250]: pgmap v12908: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:52 smithi122 ceph-mon[32255]: pgmap v12909: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:52 smithi022 ceph-mon[31360]: pgmap v12909: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:52 smithi022 ceph-mon[40250]: pgmap v12909: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:08:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:08:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:53 smithi022 ceph-mon[31360]: pgmap v12910: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:53 smithi022 ceph-mon[40250]: pgmap v12910: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:54.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:08:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:53 smithi122 ceph-mon[32255]: pgmap v12910: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:08:54.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:08:54.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:08:54.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:08:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:08:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:08:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1915334275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1915334275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1915334275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:08:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:08:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:55 smithi022 ceph-mon[31360]: pgmap v12911: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:55 smithi022 ceph-mon[40250]: pgmap v12911: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:55 smithi122 ceph-mon[32255]: pgmap v12911: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:08:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:57 smithi022 ceph-mon[31360]: pgmap v12912: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:57 smithi022 ceph-mon[40250]: pgmap v12912: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:57 smithi122 ceph-mon[32255]: pgmap v12912: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:08:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:00.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:00.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:00.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:08:59 smithi022 ceph-mon[31360]: pgmap v12913: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:08:59 smithi022 ceph-mon[40250]: pgmap v12913: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:08:59 smithi122 ceph-mon[32255]: pgmap v12913: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:00] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T01:09:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3931337644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3931337644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3931337644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:02.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:02 smithi122 ceph-mon[32255]: pgmap v12914: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:02 smithi022 ceph-mon[31360]: pgmap v12914: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:02 smithi022 ceph-mon[40250]: pgmap v12914: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:09:02.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:09:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:03 smithi022 ceph-mon[31360]: pgmap v12915: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:03 smithi022 ceph-mon[40250]: pgmap v12915: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:03 smithi122 ceph-mon[32255]: pgmap v12915: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:09:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:09:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:06.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2472397385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:06.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:05 smithi022 ceph-mon[31360]: pgmap v12916: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:06.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2472397385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:06.290 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:05 smithi022 ceph-mon[40250]: pgmap v12916: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2472397385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:05 smithi122 ceph-mon[32255]: pgmap v12916: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:09:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:07 smithi022 ceph-mon[31360]: pgmap v12917: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:07 smithi022 ceph-mon[40250]: pgmap v12917: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:07 smithi122 ceph-mon[32255]: pgmap v12917: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:10.254 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:09 smithi122 ceph-mon[32255]: pgmap v12918: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:10.255 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:10] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:09:10.257 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:09 smithi022 ceph-mon[31360]: pgmap v12918: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:10.257 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:09 smithi022 ceph-mon[40250]: pgmap v12918: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:10.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:10.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:10.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:10.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:09:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:09:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:09:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:09:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2260994546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:09:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:09:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:09:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:09:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2260994546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:09:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:09:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:09:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:09:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2260994546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:12 smithi122 ceph-mon[32255]: pgmap v12919: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:12 smithi022 ceph-mon[31360]: pgmap v12919: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:12 smithi022 ceph-mon[40250]: pgmap v12919: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:09:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:09:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:13 smithi022 ceph-mon[31360]: pgmap v12920: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:13 smithi022 ceph-mon[40250]: pgmap v12920: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:13 smithi122 ceph-mon[32255]: pgmap v12920: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:15.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:15.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:09:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:09:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:15.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:15 smithi022 ceph-mon[31360]: pgmap v12921: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:15 smithi022 ceph-mon[40250]: pgmap v12921: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:15 smithi122 ceph-mon[32255]: pgmap v12921: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2017784917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2017784917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2017784917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:09:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:18 smithi122 ceph-mon[32255]: pgmap v12922: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:18 smithi022 ceph-mon[31360]: pgmap v12922: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:18 smithi022 ceph-mon[40250]: pgmap v12922: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:19 smithi022 ceph-mon[31360]: pgmap v12923: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:19 smithi022 ceph-mon[40250]: pgmap v12923: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:20] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T01:09:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:19 smithi122 ceph-mon[32255]: pgmap v12923: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:21.130 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:21.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:21.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:21.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3954391740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:22 smithi122 ceph-mon[32255]: pgmap v12924: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3954391740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:22 smithi022 ceph-mon[31360]: pgmap v12924: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3954391740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:22 smithi022 ceph-mon[40250]: pgmap v12924: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:09:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:09:24.290 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:23 smithi022 ceph-mon[31360]: pgmap v12925: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:24.291 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:23 smithi022 ceph-mon[40250]: pgmap v12925: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:23 smithi122 ceph-mon[32255]: pgmap v12925: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:25.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:09:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:09:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:25 smithi022 ceph-mon[31360]: pgmap v12926: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:25 smithi022 ceph-mon[40250]: pgmap v12926: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:25 smithi122 ceph-mon[32255]: pgmap v12926: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:26.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:26.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:26.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:26.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2387450639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2387450639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2387450639' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:09:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:28 smithi122 ceph-mon[32255]: pgmap v12927: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:28 smithi022 ceph-mon[31360]: pgmap v12927: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:28 smithi022 ceph-mon[40250]: pgmap v12927: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:29 smithi022 ceph-mon[31360]: pgmap v12928: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:29 smithi022 ceph-mon[40250]: pgmap v12928: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:30] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T01:09:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:29 smithi122 ceph-mon[32255]: pgmap v12928: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:31.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:32.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:32.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:32.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:32.238 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:32 smithi022 ceph-mon[31360]: pgmap v12929: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:32.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:32 smithi022 ceph-mon[40250]: pgmap v12929: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:32 smithi122 ceph-mon[32255]: pgmap v12929: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:09:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:09:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4100804619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4100804619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4100804619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:09:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:09:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:09:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:09:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:34 smithi122 ceph-mon[32255]: pgmap v12930: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:09:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:09:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:09:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:09:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[31360]: pgmap v12930: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:09:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:09:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:09:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:09:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:34 smithi022 ceph-mon[40250]: pgmap v12930: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:35.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:09:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:09:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:36 smithi122 ceph-mon[32255]: pgmap v12931: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:36 smithi022 ceph-mon[31360]: pgmap v12931: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:36 smithi022 ceph-mon[40250]: pgmap v12931: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:09:37.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:37.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:37.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/503230080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/503230080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/503230080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:38.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:38 smithi122 ceph-mon[32255]: pgmap v12932: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:38 smithi022 ceph-mon[31360]: pgmap v12932: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:38 smithi022 ceph-mon[40250]: pgmap v12932: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:39 smithi022 ceph-mon[31360]: pgmap v12933: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:39 smithi022 ceph-mon[40250]: pgmap v12933: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:40] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:09:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:39 smithi122 ceph-mon[32255]: pgmap v12933: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:42 smithi122 ceph-mon[32255]: pgmap v12934: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:42 smithi022 ceph-mon[31360]: pgmap v12934: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:42 smithi022 ceph-mon[40250]: pgmap v12934: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:42.609 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:09:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:09:42.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:42.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:42.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4165193357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4165193357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:43.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4165193357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:44 smithi122 ceph-mon[32255]: pgmap v12935: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:44 smithi022 ceph-mon[31360]: pgmap v12935: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:44 smithi022 ceph-mon[40250]: pgmap v12935: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:45.068 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:09:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:09:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:46 smithi122 ceph-mon[32255]: pgmap v12936: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:46 smithi022 ceph-mon[31360]: pgmap v12936: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:46 smithi022 ceph-mon[40250]: pgmap v12936: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:09:47.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:47 smithi022 ceph-mon[31360]: pgmap v12937: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:47 smithi022 ceph-mon[40250]: pgmap v12937: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:48.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:48.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:48.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:47 smithi122 ceph-mon[32255]: pgmap v12937: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3182648903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3182648903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3182648903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:49 smithi022 ceph-mon[31360]: pgmap v12938: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:49 smithi022 ceph-mon[40250]: pgmap v12938: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:50] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T01:09:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:49 smithi122 ceph-mon[32255]: pgmap v12938: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:09:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:09:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:52 smithi122 ceph-mon[32255]: pgmap v12939: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:52 smithi022 ceph-mon[31360]: pgmap v12939: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:52 smithi022 ceph-mon[40250]: pgmap v12939: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:53.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:53.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:53.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:53.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2140472993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2140472993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2140472993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:55.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:54 smithi022 ceph-mon[31360]: pgmap v12940: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:55.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:54 smithi022 ceph-mon[40250]: pgmap v12940: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:55.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:54 smithi122 ceph-mon[32255]: pgmap v12940: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:09:55.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:09:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:09:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:09:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:56 smithi022 ceph-mon[31360]: pgmap v12941: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:56 smithi022 ceph-mon[40250]: pgmap v12941: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:56 smithi122 ceph-mon[32255]: pgmap v12941: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:09:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:57 smithi022 ceph-mon[31360]: pgmap v12942: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:57 smithi022 ceph-mon[40250]: pgmap v12942: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:57 smithi122 ceph-mon[32255]: pgmap v12942: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:09:58.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:09:59.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:09:59.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:09:59.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:09:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:09:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/638803908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:09:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/638803908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:09:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:09:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/638803908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:00] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T01:10:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:00 smithi122 ceph-mon[32255]: pgmap v12943: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T01:10:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:00 smithi022 ceph-mon[31360]: pgmap v12943: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T01:10:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:00 smithi022 ceph-mon[40250]: pgmap v12943: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T01:10:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:02 smithi122 ceph-mon[32255]: pgmap v12944: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:02 smithi022 ceph-mon[31360]: pgmap v12944: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:02 smithi022 ceph-mon[40250]: pgmap v12944: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:10:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:10:04.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:03 smithi022 ceph-mon[31360]: pgmap v12945: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:03 smithi022 ceph-mon[40250]: pgmap v12945: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:03 smithi122 ceph-mon[32255]: pgmap v12945: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:04.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:04.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:04.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:10:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:10:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3596203513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3596203513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3596203513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:05 smithi022 ceph-mon[31360]: pgmap v12946: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:06 smithi022 ceph-mon[40250]: pgmap v12946: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:06 smithi122 ceph-mon[32255]: pgmap v12946: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:10:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:07 smithi022 ceph-mon[31360]: pgmap v12947: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:07 smithi022 ceph-mon[40250]: pgmap v12947: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:07 smithi122 ceph-mon[32255]: pgmap v12947: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:09.465 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:09.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:09.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:09.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2294382959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2294382959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2294382959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:10] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:10:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[31360]: pgmap v12948: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:10:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:10:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:10:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[40250]: pgmap v12948: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:10:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:10:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:10:11.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:10 smithi122 ceph-mon[32255]: pgmap v12948: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:11.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:10:11.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:10:11.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:10:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:10:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:12 smithi122 ceph-mon[32255]: pgmap v12949: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:10:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:10:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:10:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:12 smithi022 ceph-mon[31360]: pgmap v12949: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:10:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:12 smithi022 ceph-mon[40250]: pgmap v12949: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:13 smithi022 ceph-mon[31360]: pgmap v12950: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:13 smithi022 ceph-mon[40250]: pgmap v12950: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:13 smithi122 ceph-mon[32255]: pgmap v12950: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:14.839 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:15.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:10:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:10:15.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:15.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:15.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:15.211 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:15.211 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2589968359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:16 smithi022 ceph-mon[31360]: pgmap v12951: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2589968359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:16 smithi022 ceph-mon[40250]: pgmap v12951: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2589968359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:16 smithi122 ceph-mon[32255]: pgmap v12951: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:10:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:17 smithi022 ceph-mon[31360]: pgmap v12952: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:17 smithi022 ceph-mon[40250]: pgmap v12952: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:17 smithi122 ceph-mon[32255]: pgmap v12952: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:20.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:19 smithi022 ceph-mon[31360]: pgmap v12953: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:19 smithi022 ceph-mon[40250]: pgmap v12953: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:20] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:10:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:19 smithi122 ceph-mon[32255]: pgmap v12953: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:20.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:20.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:20.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3087895696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3087895696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3087895696' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:22 smithi122 ceph-mon[32255]: pgmap v12954: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:10:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:10:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:22 smithi022 ceph-mon[31360]: pgmap v12954: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:22 smithi022 ceph-mon[40250]: pgmap v12954: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:23 smithi022 ceph-mon[31360]: pgmap v12955: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:23 smithi022 ceph-mon[40250]: pgmap v12955: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:23 smithi122 ceph-mon[32255]: pgmap v12955: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:10:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:10:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:26 smithi022 ceph-mon[31360]: pgmap v12956: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2660342625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:26 smithi022 ceph-mon[40250]: pgmap v12956: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2660342625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:26 smithi122 ceph-mon[32255]: pgmap v12956: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2660342625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:10:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:27 smithi022 ceph-mon[31360]: pgmap v12957: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:27 smithi022 ceph-mon[40250]: pgmap v12957: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:27 smithi122 ceph-mon[32255]: pgmap v12957: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:29 smithi022 ceph-mon[40250]: pgmap v12958: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:29 smithi022 ceph-mon[31360]: pgmap v12958: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:30 smithi122 ceph-mon[32255]: pgmap v12958: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:30] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:10:30.954 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:31.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:31.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:31.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:10:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:10:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1239760644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:32 smithi122 ceph-mon[32255]: pgmap v12959: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1239760644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:32 smithi022 ceph-mon[31360]: pgmap v12959: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1239760644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:32 smithi022 ceph-mon[40250]: pgmap v12959: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:10:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:10:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:10:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:10:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:10:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:10:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:10:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:10:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:10:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:10:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:10:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:10:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:34 smithi122 ceph-mon[32255]: pgmap v12960: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:35.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:10:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:10:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:34 smithi022 ceph-mon[31360]: pgmap v12960: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:34 smithi022 ceph-mon[40250]: pgmap v12960: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:36.327 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:36 smithi022 ceph-mon[31360]: pgmap v12961: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/917585144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:36 smithi022 ceph-mon[40250]: pgmap v12961: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/917585144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:36 smithi122 ceph-mon[32255]: pgmap v12961: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/917585144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:37.796 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:10:38.291 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:37 smithi022 ceph-mon[31360]: pgmap v12962: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:38.291 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:37 smithi022 ceph-mon[40250]: pgmap v12962: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:38.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:38 smithi122 ceph-mon[32255]: pgmap v12962: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:39 smithi022 ceph-mon[31360]: pgmap v12963: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:39 smithi022 ceph-mon[40250]: pgmap v12963: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:40 smithi122 ceph-mon[32255]: pgmap v12963: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:40] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:10:41.697 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:42 smithi122 ceph-mon[32255]: pgmap v12964: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3912518557' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:10:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:10:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:42 smithi022 ceph-mon[31360]: pgmap v12964: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3912518557' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:42 smithi022 ceph-mon[40250]: pgmap v12964: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3912518557' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:44 smithi022 ceph-mon[31360]: pgmap v12965: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:44 smithi022 ceph-mon[40250]: pgmap v12965: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:44 smithi122 ceph-mon[32255]: pgmap v12965: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:10:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:10:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:46 smithi022 ceph-mon[31360]: pgmap v12966: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:46 smithi022 ceph-mon[40250]: pgmap v12966: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:46 smithi122 ceph-mon[32255]: pgmap v12966: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:47.065 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:47.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:47.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:47.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3461468917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:10:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3461468917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3461468917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:48 smithi022 ceph-mon[31360]: pgmap v12967: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:48 smithi022 ceph-mon[40250]: pgmap v12967: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:48 smithi122 ceph-mon[32255]: pgmap v12967: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:50.021 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:50 smithi122 ceph-mon[32255]: pgmap v12968: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:50 smithi022 ceph-mon[31360]: pgmap v12968: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:50 smithi022 ceph-mon[40250]: pgmap v12968: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:10:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:52 smithi122 ceph-mon[32255]: pgmap v12969: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:10:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:10:52.438 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:52.438 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:52 smithi022 ceph-mon[31360]: pgmap v12969: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:52.439 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:52 smithi022 ceph-mon[40250]: pgmap v12969: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:52.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:52.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:52.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:53.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3800704950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:53.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3800704950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:53.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3800704950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:54 smithi122 ceph-mon[32255]: pgmap v12970: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:54 smithi022 ceph-mon[31360]: pgmap v12970: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:54 smithi022 ceph-mon[40250]: pgmap v12970: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:10:55.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:10:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:10:55.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:10:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:56 smithi122 ceph-mon[32255]: pgmap v12971: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:56 smithi022 ceph-mon[31360]: pgmap v12971: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:56 smithi022 ceph-mon[40250]: pgmap v12971: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:10:57.810 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:10:58.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:10:58.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:10:58.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:10:58.180 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:58 smithi022 ceph-mon[31360]: pgmap v12972: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:58.181 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:58 smithi022 ceph-mon[40250]: pgmap v12972: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:58 smithi122 ceph-mon[32255]: pgmap v12972: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:10:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:10:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/247936065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:10:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/247936065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:10:59.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:10:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/247936065' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:00 smithi022 ceph-mon[31360]: pgmap v12973: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:00 smithi022 ceph-mon[40250]: pgmap v12973: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:11:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:00 smithi122 ceph-mon[32255]: pgmap v12973: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:02.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:02 smithi122 ceph-mon[32255]: pgmap v12974: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:02.440 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:11:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:11:02.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:02 smithi022 ceph-mon[31360]: pgmap v12974: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:02.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:02 smithi022 ceph-mon[40250]: pgmap v12974: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:03.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:03.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:03.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:03.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:03.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3785986220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3785986220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3785986220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:04.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:11:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:11:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:04 smithi122 ceph-mon[32255]: pgmap v12975: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:04 smithi022 ceph-mon[31360]: pgmap v12975: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:04 smithi022 ceph-mon[40250]: pgmap v12975: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:06 smithi122 ceph-mon[32255]: pgmap v12976: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:06 smithi022 ceph-mon[31360]: pgmap v12976: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:06 smithi022 ceph-mon[40250]: pgmap v12976: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:11:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:08 smithi022 ceph-mon[31360]: pgmap v12977: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:08 smithi022 ceph-mon[40250]: pgmap v12977: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:08 smithi122 ceph-mon[32255]: pgmap v12977: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:08.555 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:08.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:08.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:08.923 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2239005238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2239005238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2239005238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:10 smithi022 ceph-mon[31360]: pgmap v12978: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:10 smithi022 ceph-mon[40250]: pgmap v12978: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:10] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T01:11:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:10 smithi122 ceph-mon[32255]: pgmap v12978: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:11:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:11:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:11:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:11:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:12 smithi122 ceph-mon[32255]: pgmap v12979: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:11:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:11:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:11:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:11:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:11:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:11:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[31360]: pgmap v12979: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:11:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:11:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:11:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:11:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:12 smithi022 ceph-mon[40250]: pgmap v12979: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:13.925 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:14 smithi022 ceph-mon[31360]: pgmap v12980: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:14 smithi022 ceph-mon[40250]: pgmap v12980: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:14.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:14.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:14.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:14 smithi122 ceph-mon[32255]: pgmap v12980: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:15.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:11:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:11:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/558906098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/558906098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/558906098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:16 smithi022 ceph-mon[31360]: pgmap v12981: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:16 smithi022 ceph-mon[40250]: pgmap v12981: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:16 smithi122 ceph-mon[32255]: pgmap v12981: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:11:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:18 smithi022 ceph-mon[31360]: pgmap v12982: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:18 smithi022 ceph-mon[40250]: pgmap v12982: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:18 smithi122 ceph-mon[32255]: pgmap v12982: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:19.294 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:19.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:19.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:19.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/233563214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/233563214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/233563214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:20] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:11:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:20 smithi022 ceph-mon[31360]: pgmap v12983: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:20 smithi022 ceph-mon[40250]: pgmap v12983: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:21.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:20 smithi122 ceph-mon[32255]: pgmap v12983: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:22 smithi122 ceph-mon[32255]: pgmap v12984: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:11:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:11:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:22 smithi022 ceph-mon[31360]: pgmap v12984: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:22 smithi022 ceph-mon[40250]: pgmap v12984: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:24 smithi022 ceph-mon[31360]: pgmap v12985: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:24 smithi022 ceph-mon[40250]: pgmap v12985: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:24 smithi122 ceph-mon[32255]: pgmap v12985: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:24.667 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:25.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:25.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:25.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:11:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:11:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/603719328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:26 smithi022 ceph-mon[31360]: pgmap v12986: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/603719328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:26 smithi022 ceph-mon[40250]: pgmap v12986: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/603719328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:26 smithi122 ceph-mon[32255]: pgmap v12986: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:11:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:28 smithi022 ceph-mon[31360]: pgmap v12987: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:28 smithi022 ceph-mon[40250]: pgmap v12987: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:28 smithi122 ceph-mon[32255]: pgmap v12987: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:30.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:30 smithi022 ceph-mon[31360]: pgmap v12988: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:30 smithi022 ceph-mon[40250]: pgmap v12988: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:30.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:30.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:30.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:30] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:11:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:30 smithi122 ceph-mon[32255]: pgmap v12988: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3826275443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3826275443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3826275443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:32.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:32 smithi122 ceph-mon[32255]: pgmap v12989: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:11:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:11:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:32 smithi022 ceph-mon[31360]: pgmap v12989: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:32 smithi022 ceph-mon[40250]: pgmap v12989: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:34.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:11:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:11:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:11:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:11:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:11:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:11:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:11:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:11:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:11:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:11:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:11:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:11:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:34 smithi122 ceph-mon[32255]: pgmap v12990: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:35.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:11:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:11:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:34 smithi022 ceph-mon[31360]: pgmap v12990: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:34 smithi022 ceph-mon[40250]: pgmap v12990: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:35.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:35.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:35.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:35.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2028874263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2028874263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:36.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:36.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2028874263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:36 smithi122 ceph-mon[32255]: pgmap v12991: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:36 smithi022 ceph-mon[31360]: pgmap v12991: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:36 smithi022 ceph-mon[40250]: pgmap v12991: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:11:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:38 smithi022 ceph-mon[31360]: pgmap v12992: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:38 smithi022 ceph-mon[40250]: pgmap v12992: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:38 smithi122 ceph-mon[32255]: pgmap v12992: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:40 smithi022 ceph-mon[31360]: pgmap v12993: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:40 smithi022 ceph-mon[40250]: pgmap v12993: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:40 smithi122 ceph-mon[32255]: pgmap v12993: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:40] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:11:40.785 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:41.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:41.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:41.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2079182564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:42 smithi122 ceph-mon[32255]: pgmap v12994: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:11:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:11:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2079182564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:42 smithi022 ceph-mon[31360]: pgmap v12994: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2079182564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:42 smithi022 ceph-mon[40250]: pgmap v12994: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:44 smithi022 ceph-mon[31360]: pgmap v12995: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:44 smithi022 ceph-mon[40250]: pgmap v12995: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:44 smithi122 ceph-mon[32255]: pgmap v12995: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:11:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:11:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:46.150 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:46 smithi022 ceph-mon[31360]: pgmap v12996: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:46 smithi022 ceph-mon[40250]: pgmap v12996: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:46 smithi122 ceph-mon[32255]: pgmap v12996: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:46.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:46.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:46.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:47.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/296408273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/296408273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/296408273' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:11:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:48 smithi122 ceph-mon[32255]: pgmap v12997: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:48 smithi022 ceph-mon[31360]: pgmap v12997: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:48 smithi022 ceph-mon[40250]: pgmap v12997: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:50 smithi022 ceph-mon[31360]: pgmap v12998: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:50 smithi022 ceph-mon[40250]: pgmap v12998: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:11:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:50 smithi122 ceph-mon[32255]: pgmap v12998: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:51.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:51.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:51.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:51.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/76851984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:52 smithi022 ceph-mon[31360]: pgmap v12999: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/76851984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:52 smithi022 ceph-mon[40250]: pgmap v12999: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/76851984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:52 smithi122 ceph-mon[32255]: pgmap v12999: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:11:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:11:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:54 smithi022 ceph-mon[31360]: pgmap v13000: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:54 smithi022 ceph-mon[40250]: pgmap v13000: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:54 smithi122 ceph-mon[32255]: pgmap v13000: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:11:55.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:11:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:11:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:11:56.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:56 smithi022 ceph-mon[31360]: pgmap v13001: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:56 smithi022 ceph-mon[40250]: pgmap v13001: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:56 smithi122 ceph-mon[32255]: pgmap v13001: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:56.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:11:57.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:11:57.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:11:57.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:11:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3390401013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:11:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3390401013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3390401013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:11:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:11:58 smithi122 ceph-mon[32255]: pgmap v13002: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:11:58 smithi022 ceph-mon[31360]: pgmap v13002: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:11:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:11:58 smithi022 ceph-mon[40250]: pgmap v13002: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:00 smithi022 ceph-mon[31360]: pgmap v13003: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:00 smithi022 ceph-mon[40250]: pgmap v13003: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:00 smithi122 ceph-mon[32255]: pgmap v13003: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:12:02.258 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:02 smithi122 ceph-mon[32255]: pgmap v13004: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:02.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:12:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:12:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:02 smithi022 ceph-mon[31360]: pgmap v13004: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:02 smithi022 ceph-mon[40250]: pgmap v13004: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:02.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:02.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:02.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/190511584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/190511584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/190511584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:04 smithi122 ceph-mon[32255]: pgmap v13005: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:04 smithi022 ceph-mon[31360]: pgmap v13005: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:04 smithi022 ceph-mon[40250]: pgmap v13005: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:12:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:12:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:06 smithi122 ceph-mon[32255]: pgmap v13006: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:06 smithi022 ceph-mon[31360]: pgmap v13006: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:06 smithi022 ceph-mon[40250]: pgmap v13006: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:07.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:12:07.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:07.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:07.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:08 smithi022 ceph-mon[31360]: pgmap v13007: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3581343055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:08 smithi022 ceph-mon[40250]: pgmap v13007: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3581343055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:08 smithi122 ceph-mon[32255]: pgmap v13007: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3581343055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:10 smithi022 ceph-mon[31360]: pgmap v13008: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:10 smithi022 ceph-mon[40250]: pgmap v13008: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:10] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T01:12:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:10 smithi122 ceph-mon[32255]: pgmap v13008: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:12:12.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:12:12.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:12:12.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[31360]: pgmap v13009: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:12.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:12.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:12.302 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:12:12.302 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:12:12.302 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:12:12.303 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[40250]: pgmap v13009: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:12.303 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:12.303 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:12:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:12:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:12:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:12 smithi122 ceph-mon[32255]: pgmap v13009: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:12:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:12:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:13.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:13.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:13.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:13.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:13.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/969598961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/969598961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:12:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/969598961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:14 smithi022 ceph-mon[31360]: pgmap v13010: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:14 smithi022 ceph-mon[40250]: pgmap v13010: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:14.879 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:14 smithi122 ceph-mon[32255]: pgmap v13010: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:15.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:12:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:12:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:16 smithi022 ceph-mon[31360]: pgmap v13011: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:16 smithi022 ceph-mon[40250]: pgmap v13011: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:16 smithi122 ceph-mon[32255]: pgmap v13011: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:12:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:18 smithi022 ceph-mon[31360]: pgmap v13012: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:18 smithi022 ceph-mon[40250]: pgmap v13012: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:18.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:18 smithi122 ceph-mon[32255]: pgmap v13012: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:18.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:18.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:18.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1249856574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1249856574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1249856574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:20 smithi022 ceph-mon[31360]: pgmap v13013: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:20 smithi022 ceph-mon[40250]: pgmap v13013: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:20 smithi122 ceph-mon[32255]: pgmap v13013: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:20] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:12:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:22 smithi122 ceph-mon[32255]: pgmap v13014: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:12:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:12:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:22 smithi022 ceph-mon[31360]: pgmap v13014: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:22 smithi022 ceph-mon[40250]: pgmap v13014: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:23.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:24.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:24.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:24.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:24 smithi022 ceph-mon[31360]: pgmap v13015: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:24 smithi022 ceph-mon[40250]: pgmap v13015: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:24 smithi122 ceph-mon[32255]: pgmap v13015: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:12:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:12:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/694749911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:25.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/694749911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/694749911' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:26 smithi022 ceph-mon[31360]: pgmap v13016: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:26 smithi022 ceph-mon[40250]: pgmap v13016: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:26 smithi122 ceph-mon[32255]: pgmap v13016: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:12:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:28 smithi022 ceph-mon[31360]: pgmap v13017: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:28 smithi022 ceph-mon[40250]: pgmap v13017: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:28 smithi122 ceph-mon[32255]: pgmap v13017: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:29.119 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:29.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:29.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:29.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1950412151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1950412151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1950412151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:30] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:12:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:30 smithi122 ceph-mon[32255]: pgmap v13018: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:30 smithi022 ceph-mon[31360]: pgmap v13018: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:30 smithi022 ceph-mon[40250]: pgmap v13018: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:32 smithi122 ceph-mon[32255]: pgmap v13019: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:12:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:12:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:32 smithi022 ceph-mon[31360]: pgmap v13019: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:32 smithi022 ceph-mon[40250]: pgmap v13019: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:12:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:12:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:12:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:12:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:12:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:12:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:12:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:12:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:12:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:12:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:12:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:12:34.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:34.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:34.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:34.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:35.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:34 smithi122 ceph-mon[32255]: pgmap v13020: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:35.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:12:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:12:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:34 smithi022 ceph-mon[31360]: pgmap v13020: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:34 smithi022 ceph-mon[40250]: pgmap v13020: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/392835330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/392835330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/392835330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:36 smithi122 ceph-mon[32255]: pgmap v13021: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:36 smithi022 ceph-mon[31360]: pgmap v13021: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:36 smithi022 ceph-mon[40250]: pgmap v13021: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:12:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:38 smithi022 ceph-mon[31360]: pgmap v13022: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:38 smithi022 ceph-mon[40250]: pgmap v13022: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:38 smithi122 ceph-mon[32255]: pgmap v13022: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:39.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:40.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:40.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:40.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:40.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:40 smithi022 ceph-mon[31360]: pgmap v13023: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:40.237 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:40 smithi022 ceph-mon[40250]: pgmap v13023: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:40 smithi122 ceph-mon[32255]: pgmap v13023: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:40] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:12:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1975687948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1975687948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1975687948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:42 smithi122 ceph-mon[32255]: pgmap v13024: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:12:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:12:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:42 smithi022 ceph-mon[31360]: pgmap v13024: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:42 smithi022 ceph-mon[40250]: pgmap v13024: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:44 smithi022 ceph-mon[31360]: pgmap v13025: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:44 smithi022 ceph-mon[40250]: pgmap v13025: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:44 smithi122 ceph-mon[32255]: pgmap v13025: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:45.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:45.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:12:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:12:45.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:45.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:45.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:45.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2245958076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:46 smithi022 ceph-mon[31360]: pgmap v13026: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2245958076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:46 smithi022 ceph-mon[40250]: pgmap v13026: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2245958076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:46 smithi122 ceph-mon[32255]: pgmap v13026: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:12:48.296 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:48 smithi022 ceph-mon[31360]: pgmap v13027: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:48.298 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:48 smithi022 ceph-mon[40250]: pgmap v13027: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:48 smithi122 ceph-mon[32255]: pgmap v13027: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:50 smithi022 ceph-mon[31360]: pgmap v13028: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:50 smithi022 ceph-mon[40250]: pgmap v13028: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:50 smithi122 ceph-mon[32255]: pgmap v13028: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:50] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:12:50.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:50.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:50.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:50.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1922959930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1922959930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1922959930' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:52 smithi122 ceph-mon[32255]: pgmap v13029: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:12:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:12:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:52 smithi022 ceph-mon[31360]: pgmap v13029: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:52 smithi022 ceph-mon[40250]: pgmap v13029: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:54 smithi022 ceph-mon[31360]: pgmap v13030: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:54 smithi022 ceph-mon[40250]: pgmap v13030: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:54 smithi122 ceph-mon[32255]: pgmap v13030: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:12:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:55.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:12:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:12:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:12:55.978 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:12:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:56 smithi022 ceph-mon[31360]: pgmap v13031: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:56 smithi022 ceph-mon[40250]: pgmap v13031: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:56.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:12:56.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:12:56.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:12:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:56 smithi122 ceph-mon[32255]: pgmap v13031: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3245059996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:57.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3245059996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3245059996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:12:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:12:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:12:58 smithi122 ceph-mon[32255]: pgmap v13032: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:58.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:12:58 smithi022 ceph-mon[31360]: pgmap v13032: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:12:58.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:12:58 smithi022 ceph-mon[40250]: pgmap v13032: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:00 smithi022 ceph-mon[31360]: pgmap v13033: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:00 smithi022 ceph-mon[40250]: pgmap v13033: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:00] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:13:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:00 smithi122 ceph-mon[32255]: pgmap v13033: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:01.349 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:01.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:01.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:01.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:02.439 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:13:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:13:02.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2011448475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:02.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:02 smithi122 ceph-mon[32255]: pgmap v13034: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2011448475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:02 smithi022 ceph-mon[31360]: pgmap v13034: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2011448475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:02 smithi022 ceph-mon[40250]: pgmap v13034: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:04 smithi022 ceph-mon[31360]: pgmap v13035: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:04 smithi022 ceph-mon[40250]: pgmap v13035: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:04 smithi122 ceph-mon[32255]: pgmap v13035: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:13:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:13:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:06 smithi122 ceph-mon[32255]: pgmap v13036: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:06.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:06 smithi022 ceph-mon[31360]: pgmap v13036: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:06 smithi022 ceph-mon[40250]: pgmap v13036: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:06.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:07.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:07.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:07.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3825864961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:07.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3825864961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:07.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3825864961' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:13:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:08 smithi122 ceph-mon[32255]: pgmap v13037: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:08 smithi022 ceph-mon[31360]: pgmap v13037: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:08 smithi022 ceph-mon[40250]: pgmap v13037: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:10 smithi022 ceph-mon[31360]: pgmap v13038: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:10 smithi022 ceph-mon[40250]: pgmap v13038: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:10] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:13:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:10 smithi122 ceph-mon[32255]: pgmap v13038: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:12.089 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:12 smithi122 ceph-mon[32255]: pgmap v13039: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:13:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:13:12.513 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:12 smithi022 ceph-mon[31360]: pgmap v13039: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:12.514 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:12 smithi022 ceph-mon[40250]: pgmap v13039: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:12.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:12.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:12.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:13.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:13:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:13:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:13:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4211301616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:13:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:13:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:13:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:13:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4211301616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:13:13.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:13:13.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:13:13.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:13:13.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4211301616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:13.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:13:14.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:14 smithi122 ceph-mon[32255]: pgmap v13040: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:14 smithi022 ceph-mon[31360]: pgmap v13040: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:14 smithi022 ceph-mon[40250]: pgmap v13040: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:15.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:13:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:13:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:16 smithi122 ceph-mon[32255]: pgmap v13041: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:16 smithi022 ceph-mon[31360]: pgmap v13041: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:16 smithi022 ceph-mon[40250]: pgmap v13041: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:17.464 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:13:17.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:17.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:17.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2830687760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2830687760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2830687760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:18 smithi122 ceph-mon[32255]: pgmap v13042: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:18 smithi022 ceph-mon[31360]: pgmap v13042: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:18 smithi022 ceph-mon[40250]: pgmap v13042: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:20 smithi022 ceph-mon[31360]: pgmap v13043: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:20 smithi022 ceph-mon[40250]: pgmap v13043: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:20 smithi122 ceph-mon[32255]: pgmap v13043: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:20] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:13:22.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:13:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:13:22.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:22 smithi122 ceph-mon[32255]: pgmap v13044: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:22 smithi022 ceph-mon[31360]: pgmap v13044: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:22 smithi022 ceph-mon[40250]: pgmap v13044: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:23.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:23.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:23.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3712188900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3712188900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3712188900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:24 smithi022 ceph-mon[31360]: pgmap v13045: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:24 smithi022 ceph-mon[40250]: pgmap v13045: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:24 smithi122 ceph-mon[32255]: pgmap v13045: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:25.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:13:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:13:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:26 smithi022 ceph-mon[31360]: pgmap v13046: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:26 smithi022 ceph-mon[40250]: pgmap v13046: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:26 smithi122 ceph-mon[32255]: pgmap v13046: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:13:28.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:28 smithi022 ceph-mon[31360]: pgmap v13047: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:28 smithi022 ceph-mon[40250]: pgmap v13047: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:28 smithi122 ceph-mon[32255]: pgmap v13047: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:28.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:28.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:28.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/442780935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/442780935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/442780935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:30] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:13:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:30 smithi122 ceph-mon[32255]: pgmap v13048: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:30 smithi022 ceph-mon[31360]: pgmap v13048: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:30 smithi022 ceph-mon[40250]: pgmap v13048: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:32 smithi122 ceph-mon[32255]: pgmap v13049: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:13:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:13:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:32 smithi022 ceph-mon[31360]: pgmap v13049: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:32 smithi022 ceph-mon[40250]: pgmap v13049: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:33.581 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:33.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:33.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:33.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:13:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:13:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:13:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:13:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:13:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:13:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:13:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:13:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:13:34.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:13:34.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:13:34.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:13:35.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:13:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:13:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2579609478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:34 smithi122 ceph-mon[32255]: pgmap v13050: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2579609478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:34 smithi022 ceph-mon[31360]: pgmap v13050: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2579609478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:34 smithi022 ceph-mon[40250]: pgmap v13050: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:36 smithi122 ceph-mon[32255]: pgmap v13051: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:36 smithi022 ceph-mon[31360]: pgmap v13051: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:36 smithi022 ceph-mon[40250]: pgmap v13051: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:13:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:38 smithi022 ceph-mon[31360]: pgmap v13052: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:38 smithi022 ceph-mon[40250]: pgmap v13052: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:38 smithi122 ceph-mon[32255]: pgmap v13052: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:38.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:39.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:39.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:39.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1164322776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1164322776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1164322776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:40 smithi122 ceph-mon[32255]: pgmap v13053: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:40] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:13:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:40 smithi022 ceph-mon[40250]: pgmap v13053: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:40 smithi022 ceph-mon[31360]: pgmap v13053: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:42 smithi122 ceph-mon[32255]: pgmap v13054: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:13:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:13:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:42 smithi022 ceph-mon[31360]: pgmap v13054: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:42 smithi022 ceph-mon[40250]: pgmap v13054: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:44.324 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:44 smithi022 ceph-mon[31360]: pgmap v13055: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:44.325 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:44 smithi022 ceph-mon[40250]: pgmap v13055: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:44 smithi122 ceph-mon[32255]: pgmap v13055: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:44.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:44.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:44.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:13:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:13:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2761764631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2761764631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2761764631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:46 smithi122 ceph-mon[32255]: pgmap v13056: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:46 smithi022 ceph-mon[31360]: pgmap v13056: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:46 smithi022 ceph-mon[40250]: pgmap v13056: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:13:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:48 smithi122 ceph-mon[32255]: pgmap v13057: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:48 smithi022 ceph-mon[31360]: pgmap v13057: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:48 smithi022 ceph-mon[40250]: pgmap v13057: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:49.697 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:50.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:50.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:50.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:50 smithi122 ceph-mon[32255]: pgmap v13058: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:50] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:13:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:50 smithi022 ceph-mon[31360]: pgmap v13058: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:50 smithi022 ceph-mon[40250]: pgmap v13058: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1575124064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1575124064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1575124064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:52 smithi122 ceph-mon[32255]: pgmap v13059: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:13:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:13:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:52 smithi022 ceph-mon[31360]: pgmap v13059: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:52 smithi022 ceph-mon[40250]: pgmap v13059: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:54 smithi122 ceph-mon[32255]: pgmap v13060: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:54 smithi022 ceph-mon[31360]: pgmap v13060: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:54 smithi022 ceph-mon[40250]: pgmap v13060: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:13:55.066 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:13:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:13:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:13:55.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:13:55.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:13:55.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:13:55.440 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:55.440 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:13:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3532136554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:56 smithi122 ceph-mon[32255]: pgmap v13061: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:56.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3532136554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:56.542 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:56 smithi022 ceph-mon[31360]: pgmap v13061: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:56.542 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3532136554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:13:56.543 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:56 smithi022 ceph-mon[40250]: pgmap v13061: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:13:58.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:13:58 smithi122 ceph-mon[32255]: pgmap v13062: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:13:58 smithi022 ceph-mon[31360]: pgmap v13062: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:13:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:13:58 smithi022 ceph-mon[40250]: pgmap v13062: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:00] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:14:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:00 smithi122 ceph-mon[32255]: pgmap v13063: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:00.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:00.441 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:00 smithi022 ceph-mon[31360]: pgmap v13063: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:00.442 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:00 smithi022 ceph-mon[40250]: pgmap v13063: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:00.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:00.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:00.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1510512428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1510512428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1510512428' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:02 smithi122 ceph-mon[32255]: pgmap v13064: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:14:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:14:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:02 smithi022 ceph-mon[31360]: pgmap v13064: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:02 smithi022 ceph-mon[40250]: pgmap v13064: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:04 smithi122 ceph-mon[32255]: pgmap v13065: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:04 smithi022 ceph-mon[31360]: pgmap v13065: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:04 smithi022 ceph-mon[40250]: pgmap v13065: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:05.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:14:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:14:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:05.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:06.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:06.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:06.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:06.184 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:06 smithi022 ceph-mon[31360]: pgmap v13066: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:06.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:06 smithi022 ceph-mon[40250]: pgmap v13066: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:06 smithi122 ceph-mon[32255]: pgmap v13066: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/35002115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/35002115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/35002115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:14:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:08 smithi122 ceph-mon[32255]: pgmap v13067: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:08 smithi022 ceph-mon[31360]: pgmap v13067: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:08 smithi022 ceph-mon[40250]: pgmap v13067: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:10] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:14:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:10 smithi122 ceph-mon[32255]: pgmap v13068: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:10 smithi022 ceph-mon[31360]: pgmap v13068: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:10 smithi022 ceph-mon[40250]: pgmap v13068: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:11.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:11.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:11.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:11.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:14:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:14:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/377434663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:12 smithi122 ceph-mon[32255]: pgmap v13069: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/377434663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:12 smithi022 ceph-mon[31360]: pgmap v13069: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/377434663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:12 smithi022 ceph-mon[40250]: pgmap v13069: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:14:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:14:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:14:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:14:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:14:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:14:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:14:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:14:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:14:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:14:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:14 smithi122 ceph-mon[32255]: pgmap v13070: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:14:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:14 smithi022 ceph-mon[31360]: pgmap v13070: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:14:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:14 smithi022 ceph-mon[40250]: pgmap v13070: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:15.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:14:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:14:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:16.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:16 smithi122 ceph-mon[32255]: pgmap v13071: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:16 smithi022 ceph-mon[31360]: pgmap v13071: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:16 smithi022 ceph-mon[40250]: pgmap v13071: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:16.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:16.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:16.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2339791510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2339791510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:14:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2339791510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:18 smithi122 ceph-mon[32255]: pgmap v13072: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:18 smithi022 ceph-mon[31360]: pgmap v13072: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:18 smithi022 ceph-mon[40250]: pgmap v13072: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:20] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:14:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:20 smithi122 ceph-mon[32255]: pgmap v13073: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:20 smithi022 ceph-mon[31360]: pgmap v13073: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:20 smithi022 ceph-mon[40250]: pgmap v13073: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:21.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:22 smithi022 ceph-mon[31360]: pgmap v13074: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:22 smithi022 ceph-mon[40250]: pgmap v13074: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:22.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:22.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:22.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:22 smithi122 ceph-mon[32255]: pgmap v13074: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:14:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:14:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/340923608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/340923608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/340923608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:24 smithi122 ceph-mon[32255]: pgmap v13075: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:24 smithi022 ceph-mon[31360]: pgmap v13075: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:24 smithi022 ceph-mon[40250]: pgmap v13075: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:14:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:14:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:26 smithi122 ceph-mon[32255]: pgmap v13076: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:26 smithi022 ceph-mon[31360]: pgmap v13076: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:26 smithi022 ceph-mon[40250]: pgmap v13076: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:27.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:27.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:27.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:27.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:27.671 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:14:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1990796433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1990796433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1990796433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:28 smithi022 ceph-mon[31360]: pgmap v13077: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:28 smithi022 ceph-mon[40250]: pgmap v13077: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:28 smithi122 ceph-mon[32255]: pgmap v13077: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:30] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:14:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:30 smithi122 ceph-mon[32255]: pgmap v13078: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:30 smithi022 ceph-mon[31360]: pgmap v13078: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:30 smithi022 ceph-mon[40250]: pgmap v13078: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:32.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:14:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:14:33.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:33.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:33.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:33 smithi122 ceph-mon[32255]: pgmap v13079: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2756509507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:33 smithi022 ceph-mon[31360]: pgmap v13079: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2756509507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:33 smithi022 ceph-mon[40250]: pgmap v13079: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2756509507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:14:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:14:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:14:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:14:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:14:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:14:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:14:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:14:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:14:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:14:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:14:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:14:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:14:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:14:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:35 smithi122 ceph-mon[32255]: pgmap v13080: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:35 smithi022 ceph-mon[31360]: pgmap v13080: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:35 smithi022 ceph-mon[40250]: pgmap v13080: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:37.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:37 smithi022 ceph-mon[31360]: pgmap v13081: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:37.399 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:37 smithi022 ceph-mon[40250]: pgmap v13081: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:37 smithi122 ceph-mon[32255]: pgmap v13081: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:14:38.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:38.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:38.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:38.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:39 smithi122 ceph-mon[32255]: pgmap v13082: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/757772622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:39 smithi022 ceph-mon[31360]: pgmap v13082: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/757772622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:39 smithi022 ceph-mon[40250]: pgmap v13082: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/757772622' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:40] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:14:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:41 smithi122 ceph-mon[32255]: pgmap v13083: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:41 smithi022 ceph-mon[31360]: pgmap v13083: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:41 smithi022 ceph-mon[40250]: pgmap v13083: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:14:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:14:43.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:43.410 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:43 smithi022 ceph-mon[31360]: pgmap v13084: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:43.410 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:43 smithi022 ceph-mon[40250]: pgmap v13084: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:43 smithi122 ceph-mon[32255]: pgmap v13084: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:43.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:43.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:43.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/463292210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/463292210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/463292210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:14:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:14:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:45 smithi122 ceph-mon[32255]: pgmap v13085: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:45 smithi022 ceph-mon[31360]: pgmap v13085: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:45 smithi022 ceph-mon[40250]: pgmap v13085: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:47 smithi122 ceph-mon[32255]: pgmap v13086: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:47 smithi022 ceph-mon[31360]: pgmap v13086: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:14:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:47 smithi022 ceph-mon[40250]: pgmap v13086: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:48.780 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:49.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:49.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:49.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:49 smithi122 ceph-mon[32255]: pgmap v13087: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/669455859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:49 smithi022 ceph-mon[40250]: pgmap v13087: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/669455859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:49 smithi022 ceph-mon[31360]: pgmap v13087: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/669455859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:50] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:14:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:51 smithi122 ceph-mon[32255]: pgmap v13088: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:51 smithi022 ceph-mon[31360]: pgmap v13088: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:51 smithi022 ceph-mon[40250]: pgmap v13088: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:14:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:14:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:53 smithi122 ceph-mon[32255]: pgmap v13089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:53 smithi022 ceph-mon[31360]: pgmap v13089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:53 smithi022 ceph-mon[40250]: pgmap v13089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:54.145 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:54.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:54.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:54.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:14:55.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:14:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:14:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:55 smithi122 ceph-mon[32255]: pgmap v13090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4258469928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:55.538 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:55 smithi022 ceph-mon[31360]: pgmap v13090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:55.538 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4258469928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:55.539 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:55 smithi022 ceph-mon[40250]: pgmap v13090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:14:55.539 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4258469928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:14:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:14:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:57 smithi122 ceph-mon[32255]: pgmap v13091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:57 smithi022 ceph-mon[31360]: pgmap v13091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:14:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:57 smithi022 ceph-mon[40250]: pgmap v13091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:14:59 smithi122 ceph-mon[32255]: pgmap v13092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:59.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:14:59.521 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:14:59 smithi022 ceph-mon[40250]: pgmap v13092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:59.522 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:14:59 smithi022 ceph-mon[31360]: pgmap v13092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:14:59.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:14:59.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:14:59.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:00] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:15:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1111271539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1111271539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1111271539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:01 smithi122 ceph-mon[32255]: pgmap v13093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:01 smithi022 ceph-mon[31360]: pgmap v13093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:01 smithi022 ceph-mon[40250]: pgmap v13093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:15:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:15:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:03 smithi122 ceph-mon[32255]: pgmap v13094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:03 smithi022 ceph-mon[31360]: pgmap v13094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:03 smithi022 ceph-mon[40250]: pgmap v13094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:04.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:05.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:05.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:05.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:05 smithi122 ceph-mon[32255]: pgmap v13095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:05.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:15:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:15:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:05 smithi022 ceph-mon[31360]: pgmap v13095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:05 smithi022 ceph-mon[40250]: pgmap v13095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2101096702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2101096702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2101096702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:07 smithi122 ceph-mon[32255]: pgmap v13096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:07 smithi022 ceph-mon[31360]: pgmap v13096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:15:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:07 smithi022 ceph-mon[40250]: pgmap v13096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:09 smithi122 ceph-mon[32255]: pgmap v13097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:09 smithi022 ceph-mon[31360]: pgmap v13097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:09 smithi022 ceph-mon[40250]: pgmap v13097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:10.261 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:10] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:15:10.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:10.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:10.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:11 smithi022 ceph-mon[31360]: pgmap v13098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3019201641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:11 smithi022 ceph-mon[40250]: pgmap v13098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3019201641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:11 smithi122 ceph-mon[32255]: pgmap v13098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3019201641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:15:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:15:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:13 smithi022 ceph-mon[31360]: pgmap v13099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:13 smithi022 ceph-mon[40250]: pgmap v13099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:13.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:13 smithi122 ceph-mon[32255]: pgmap v13099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:15:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:15:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:15:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:15:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:15:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:15:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:15:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:15:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:15:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:15:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:15:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:15:14.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:15:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:15:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:15:15.625 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:15.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:15.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:15.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:15.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:15 smithi022 ceph-mon[31360]: pgmap v13100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:15.994 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:15 smithi022 ceph-mon[40250]: pgmap v13100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:15 smithi122 ceph-mon[32255]: pgmap v13100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2073263740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2073263740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2073263740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:15:18.199 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:17 smithi122 ceph-mon[32255]: pgmap v13101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:17 smithi022 ceph-mon[31360]: pgmap v13101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:18.303 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:17 smithi022 ceph-mon[40250]: pgmap v13101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:19 smithi122 ceph-mon[32255]: pgmap v13102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:19 smithi022 ceph-mon[31360]: pgmap v13102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:19 smithi022 ceph-mon[40250]: pgmap v13102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:20] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:15:20.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:21.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:21 smithi122 ceph-mon[32255]: pgmap v13103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/306434057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:21 smithi022 ceph-mon[31360]: pgmap v13103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/306434057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:21 smithi022 ceph-mon[40250]: pgmap v13103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/306434057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:15:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:15:24.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:23 smithi122 ceph-mon[32255]: pgmap v13104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:23 smithi022 ceph-mon[31360]: pgmap v13104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:23 smithi022 ceph-mon[40250]: pgmap v13104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:15:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:15:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:25 smithi122 ceph-mon[32255]: pgmap v13105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:25 smithi022 ceph-mon[31360]: pgmap v13105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:25 smithi022 ceph-mon[40250]: pgmap v13105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:26.366 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:26.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:26.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:26.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3050660266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3050660266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:27.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:27.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3050660266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:15:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:27 smithi122 ceph-mon[32255]: pgmap v13106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:27 smithi022 ceph-mon[31360]: pgmap v13106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:27 smithi022 ceph-mon[40250]: pgmap v13106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:29 smithi122 ceph-mon[32255]: pgmap v13107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:29 smithi022 ceph-mon[31360]: pgmap v13107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:29 smithi022 ceph-mon[40250]: pgmap v13107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:30] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:15:31.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:32.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:31 smithi022 ceph-mon[31360]: pgmap v13108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:31 smithi022 ceph-mon[40250]: pgmap v13108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:32.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:32.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:32.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:31 smithi122 ceph-mon[32255]: pgmap v13108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:15:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:15:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1773549742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1773549742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1773549742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:34.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:33 smithi122 ceph-mon[32255]: pgmap v13109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:15:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:15:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:15:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:15:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[31360]: pgmap v13109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:15:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:15:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:15:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:15:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[40250]: pgmap v13109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:15:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:15:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:15:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:15:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:15:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:15:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:35 smithi122 ceph-mon[32255]: pgmap v13110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:35 smithi022 ceph-mon[31360]: pgmap v13110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:35 smithi022 ceph-mon[40250]: pgmap v13110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:37.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:37.111 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:37.112 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:37.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:37.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:37.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:37.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:15:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:37 smithi122 ceph-mon[32255]: pgmap v13111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/189535286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:37 smithi022 ceph-mon[31360]: pgmap v13111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/189535286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:37 smithi022 ceph-mon[40250]: pgmap v13111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/189535286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:39 smithi122 ceph-mon[32255]: pgmap v13112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:39 smithi022 ceph-mon[31360]: pgmap v13112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:39 smithi022 ceph-mon[40250]: pgmap v13112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:15:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:41 smithi122 ceph-mon[32255]: pgmap v13113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:41 smithi022 ceph-mon[31360]: pgmap v13113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:41 smithi022 ceph-mon[40250]: pgmap v13113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:42.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:15:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:15:42.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:42.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:42.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2888988090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2888988090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2888988090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:44.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:43 smithi122 ceph-mon[32255]: pgmap v13114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:43 smithi022 ceph-mon[31360]: pgmap v13114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:43 smithi022 ceph-mon[40250]: pgmap v13114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:15:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:15:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:45 smithi122 ceph-mon[32255]: pgmap v13115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:45 smithi022 ceph-mon[31360]: pgmap v13115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:46.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:45 smithi022 ceph-mon[40250]: pgmap v13115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:47.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:47.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:47.788 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:15:47.854 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:47 smithi122 ceph-mon[32255]: pgmap v13116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:48.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:48.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:48.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:48.224 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:47 smithi022 ceph-mon[31360]: pgmap v13116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:48.224 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:47 smithi022 ceph-mon[40250]: pgmap v13116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:49.196 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3597440937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3597440937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3597440937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:49 smithi122 ceph-mon[32255]: pgmap v13117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:50.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:49 smithi022 ceph-mon[31360]: pgmap v13117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:49 smithi022 ceph-mon[40250]: pgmap v13117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:50] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:15:52.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:51 smithi122 ceph-mon[32255]: pgmap v13118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:51 smithi022 ceph-mon[31360]: pgmap v13118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:51 smithi022 ceph-mon[40250]: pgmap v13118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:15:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:15:53.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:53.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:53.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:53.595 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:15:53.877 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:53 smithi022 ceph-mon[31360]: pgmap v13119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:53.877 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/105125959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:54.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:53 smithi122 ceph-mon[32255]: pgmap v13119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:54.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/105125959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:54.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:53 smithi022 ceph-mon[40250]: pgmap v13119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/105125959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:15:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:15:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:15:56.197 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:55 smithi122 ceph-mon[32255]: pgmap v13120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:55 smithi022 ceph-mon[31360]: pgmap v13120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:55 smithi022 ceph-mon[40250]: pgmap v13120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:15:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:15:57.795 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:15:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:57 smithi122 ceph-mon[32255]: pgmap v13121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:57 smithi022 ceph-mon[31360]: pgmap v13121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:57 smithi022 ceph-mon[40250]: pgmap v13121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:15:58.597 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:15:58.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:15:58.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:15:58.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:59 smithi122 ceph-mon[32255]: pgmap v13122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:15:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4110158296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:59 smithi022 ceph-mon[31360]: pgmap v13122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:15:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4110158296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:59 smithi022 ceph-mon[40250]: pgmap v13122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:15:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4110158296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:00] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:16:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:01 smithi122 ceph-mon[32255]: pgmap v13123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:01 smithi022 ceph-mon[31360]: pgmap v13123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:01 smithi022 ceph-mon[40250]: pgmap v13123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:16:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:16:03.970 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:04.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:03 smithi122 ceph-mon[32255]: pgmap v13124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:04.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:03 smithi022 ceph-mon[31360]: pgmap v13124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:03 smithi022 ceph-mon[40250]: pgmap v13124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:04.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:04.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:04.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/242938810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:05.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:16:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:16:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/242938810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/242938810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:06.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:05 smithi122 ceph-mon[32255]: pgmap v13125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:06.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:05 smithi022 ceph-mon[31360]: pgmap v13125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:05 smithi022 ceph-mon[40250]: pgmap v13125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:16:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:07 smithi122 ceph-mon[32255]: pgmap v13126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:07 smithi022 ceph-mon[31360]: pgmap v13126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:07 smithi022 ceph-mon[40250]: pgmap v13126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:09.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:09.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:09.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:09.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:10.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:09 smithi022 ceph-mon[31360]: pgmap v13127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/950897924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:09 smithi022 ceph-mon[40250]: pgmap v13127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/950897924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:09 smithi122 ceph-mon[32255]: pgmap v13127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/950897924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:10] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:16:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:11 smithi122 ceph-mon[32255]: pgmap v13128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:11 smithi022 ceph-mon[31360]: pgmap v13128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:11 smithi022 ceph-mon[40250]: pgmap v13128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:16:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:16:14.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:13 smithi122 ceph-mon[32255]: pgmap v13129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:16:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:16:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:16:14.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[31360]: pgmap v13129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:16:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:16:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:16:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[40250]: pgmap v13129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:16:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:16:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:16:14.710 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:15.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:15.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:15.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:16:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:16:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:15 smithi122 ceph-mon[32255]: pgmap v13130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2289475529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:16.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:15 smithi022 ceph-mon[31360]: pgmap v13130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2289475529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:15 smithi022 ceph-mon[40250]: pgmap v13130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2289475529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:17.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:17.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:16:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: pgmap v13131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: pgmap v13131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: pgmap v13131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3032M 2021-12-16T01:16:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3032M 2021-12-16T01:16:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3032M 2021-12-16T01:16:19.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:16:19.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:19.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:16:20.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:20.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:20] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:16:20.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:19 smithi122 ceph-mon[32255]: pgmap v13132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:20.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:19 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3248M 2021-12-16T01:16:20.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:19 smithi022 ceph-mon[31360]: pgmap v13132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:19 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3248M 2021-12-16T01:16:20.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:19 smithi022 ceph-mon[40250]: pgmap v13132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:20.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:19 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3248M 2021-12-16T01:16:20.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:20.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:20.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/328250284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/328250284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/328250284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:21 smithi122 ceph-mon[32255]: pgmap v13133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:21 smithi022 ceph-mon[31360]: pgmap v13133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:21 smithi022 ceph-mon[40250]: pgmap v13133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:16:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:16:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:23 smithi022 ceph-mon[31360]: pgmap v13134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:23 smithi022 ceph-mon[40250]: pgmap v13134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:23 smithi122 ceph-mon[32255]: pgmap v13134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:25.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:16:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:16:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:25.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:25.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:25.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:25 smithi022 ceph-mon[31360]: pgmap v13135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3170985916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:25 smithi022 ceph-mon[40250]: pgmap v13135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3170985916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:25 smithi122 ceph-mon[32255]: pgmap v13135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3170985916' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:16:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:27 smithi022 ceph-mon[31360]: pgmap v13136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:27 smithi022 ceph-mon[40250]: pgmap v13136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:27 smithi122 ceph-mon[32255]: pgmap v13136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:29 smithi022 ceph-mon[31360]: pgmap v13137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:29 smithi022 ceph-mon[40250]: pgmap v13137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:30] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:16:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:29 smithi122 ceph-mon[32255]: pgmap v13137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:30.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:31.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:31.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:31.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:32.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:31 smithi022 ceph-mon[31360]: pgmap v13138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3765844110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:31 smithi022 ceph-mon[40250]: pgmap v13138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3765844110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:31 smithi122 ceph-mon[32255]: pgmap v13138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3765844110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:16:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:16:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[31360]: pgmap v13139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:16:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:16:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:16:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:16:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[40250]: pgmap v13139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:16:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:16:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:16:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:16:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:33 smithi122 ceph-mon[32255]: pgmap v13139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:16:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:16:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:16:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:16:35.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:16:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:16:36.199 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:36.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:35 smithi022 ceph-mon[31360]: pgmap v13140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:35 smithi022 ceph-mon[40250]: pgmap v13140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:35 smithi122 ceph-mon[32255]: pgmap v13140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:36.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:36.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:36.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:37.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3121975654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3121975654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3121975654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:37.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:16:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:37 smithi022 ceph-mon[31360]: pgmap v13141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:37 smithi022 ceph-mon[40250]: pgmap v13141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:37 smithi122 ceph-mon[32255]: pgmap v13141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:39 smithi022 ceph-mon[31360]: pgmap v13142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:39 smithi022 ceph-mon[40250]: pgmap v13142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:40.457 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:39 smithi122 ceph-mon[32255]: pgmap v13142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:40.457 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:40] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:16:41.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:41.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:41.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:41.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:42.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:41 smithi022 ceph-mon[31360]: pgmap v13143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1698589571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:41 smithi022 ceph-mon[40250]: pgmap v13143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1698589571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:41 smithi122 ceph-mon[32255]: pgmap v13143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1698589571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:16:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:16:44.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:43 smithi022 ceph-mon[31360]: pgmap v13144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:43 smithi022 ceph-mon[40250]: pgmap v13144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:43 smithi122 ceph-mon[32255]: pgmap v13144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:16:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:16:46.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:45 smithi022 ceph-mon[31360]: pgmap v13145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:45 smithi022 ceph-mon[40250]: pgmap v13145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:45 smithi122 ceph-mon[32255]: pgmap v13145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:46.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:47.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:47.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:47.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:47.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:47.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:16:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:47 smithi022 ceph-mon[31360]: pgmap v13146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2339391477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:47 smithi022 ceph-mon[40250]: pgmap v13146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2339391477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:47 smithi122 ceph-mon[32255]: pgmap v13146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2339391477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:50.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:49 smithi022 ceph-mon[31360]: pgmap v13147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:49 smithi022 ceph-mon[40250]: pgmap v13147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:49 smithi122 ceph-mon[32255]: pgmap v13147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:50] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:16:52.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:51 smithi022 ceph-mon[31360]: pgmap v13148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:51 smithi022 ceph-mon[40250]: pgmap v13148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:52.310 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:51 smithi122 ceph-mon[32255]: pgmap v13148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:52.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:52.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:52.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:16:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:16:53.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3309658156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:53.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3309658156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3309658156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:54.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:53 smithi022 ceph-mon[31360]: pgmap v13149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:53 smithi022 ceph-mon[40250]: pgmap v13149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:53 smithi122 ceph-mon[32255]: pgmap v13149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:16:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:16:56.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:55 smithi022 ceph-mon[31360]: pgmap v13150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:55 smithi022 ceph-mon[40250]: pgmap v13150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:55 smithi122 ceph-mon[32255]: pgmap v13150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:16:57.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:16:57.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:16:57.683 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:16:58.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:16:58.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:16:58.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:16:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:58 smithi022 ceph-mon[31360]: pgmap v13151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:58 smithi022 ceph-mon[40250]: pgmap v13151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:58 smithi122 ceph-mon[32255]: pgmap v13151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:16:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:16:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3741220292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:59.341 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:16:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3741220292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:16:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:16:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3741220292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:00.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:00 smithi022 ceph-mon[31360]: pgmap v13152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:00 smithi022 ceph-mon[40250]: pgmap v13152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:00] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:17:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:00 smithi122 ceph-mon[32255]: pgmap v13152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:02.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:02 smithi022 ceph-mon[40250]: pgmap v13153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:02 smithi022 ceph-mon[31360]: pgmap v13153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:02.326 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:02 smithi122 ceph-mon[32255]: pgmap v13153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:17:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:17:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:03.457 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:03 smithi122 ceph-mon[32255]: pgmap v13154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:03.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:03.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:03.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:03 smithi022 ceph-mon[31360]: pgmap v13154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:03.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:03 smithi022 ceph-mon[40250]: pgmap v13154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:04.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3708751884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3708751884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3708751884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:05 smithi122 ceph-mon[32255]: pgmap v13155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:05.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:17:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:17:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:05 smithi022 ceph-mon[31360]: pgmap v13155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:05 smithi022 ceph-mon[40250]: pgmap v13155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:06.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:07 smithi122 ceph-mon[32255]: pgmap v13156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:07.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:07 smithi022 ceph-mon[31360]: pgmap v13156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:17:07.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:07 smithi022 ceph-mon[40250]: pgmap v13156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:08.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:08.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:08.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:08.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:09 smithi122 ceph-mon[32255]: pgmap v13157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/925831189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:09.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:09 smithi022 ceph-mon[31360]: pgmap v13157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/925831189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:09 smithi022 ceph-mon[40250]: pgmap v13157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/925831189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:10] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:17:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:11 smithi122 ceph-mon[32255]: pgmap v13158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:11.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:11 smithi022 ceph-mon[31360]: pgmap v13158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:11.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:11 smithi022 ceph-mon[40250]: pgmap v13158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:17:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:17:13.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:13 smithi122 ceph-mon[32255]: pgmap v13159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:13 smithi022 ceph-mon[31360]: pgmap v13159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:13 smithi022 ceph-mon[40250]: pgmap v13159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:13.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:14.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:14.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:14.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:14.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/830675389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/830675389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/830675389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:15 smithi122 ceph-mon[32255]: pgmap v13160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:15.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:17:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:17:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:15 smithi022 ceph-mon[31360]: pgmap v13160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:15 smithi022 ceph-mon[40250]: pgmap v13160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:16.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:17 smithi022 ceph-mon[31360]: pgmap v13161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:17:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:17 smithi022 ceph-mon[40250]: pgmap v13161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:17 smithi122 ceph-mon[32255]: pgmap v13161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:19.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:19.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[31360]: pgmap v13162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:17:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:17:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:17:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[40250]: pgmap v13162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:17:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:17:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:17:19.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:19.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:19.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:19 smithi122 ceph-mon[32255]: pgmap v13162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:17:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:17:19.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:17:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:17:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:17:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/461293853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:20.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:17:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/461293853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:17:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/461293853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:21 smithi022 ceph-mon[31360]: pgmap v13163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:21 smithi022 ceph-mon[40250]: pgmap v13163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:21 smithi122 ceph-mon[32255]: pgmap v13163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:17:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:17:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:23 smithi022 ceph-mon[31360]: pgmap v13164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:23 smithi022 ceph-mon[40250]: pgmap v13164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:23 smithi122 ceph-mon[32255]: pgmap v13164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:24.533 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:24.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:24.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:24.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:25.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:17:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:17:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:25 smithi122 ceph-mon[32255]: pgmap v13165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/296328096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:25.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:25 smithi022 ceph-mon[31360]: pgmap v13165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/296328096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:25 smithi022 ceph-mon[40250]: pgmap v13165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/296328096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:26.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:27.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:27 smithi022 ceph-mon[31360]: pgmap v13166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:17:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:27 smithi022 ceph-mon[40250]: pgmap v13166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:27 smithi122 ceph-mon[32255]: pgmap v13166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:29 smithi022 ceph-mon[31360]: pgmap v13167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:29 smithi022 ceph-mon[40250]: pgmap v13167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:29 smithi122 ceph-mon[32255]: pgmap v13167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:29.905 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:30.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:30.445 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:17:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:31 smithi022 ceph-mon[31360]: pgmap v13168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/336686142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:31 smithi022 ceph-mon[40250]: pgmap v13168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/336686142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:31 smithi122 ceph-mon[32255]: pgmap v13168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/336686142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:17:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:17:33.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:33 smithi022 ceph-mon[31360]: pgmap v13169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:33 smithi022 ceph-mon[40250]: pgmap v13169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:33 smithi122 ceph-mon[32255]: pgmap v13169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:34.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:17:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:17:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:17:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:17:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:17:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:17:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:17:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:17:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:17:35.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:35.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:35 smithi122 ceph-mon[32255]: pgmap v13170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:35.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:17:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:17:35.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:35 smithi022 ceph-mon[31360]: pgmap v13170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:35 smithi022 ceph-mon[40250]: pgmap v13170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4245434396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4245434396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4245434396' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:37 smithi022 ceph-mon[31360]: pgmap v13171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:17:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:37 smithi022 ceph-mon[40250]: pgmap v13171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:37 smithi122 ceph-mon[32255]: pgmap v13171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:39.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:39 smithi022 ceph-mon[31360]: pgmap v13172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:39 smithi022 ceph-mon[40250]: pgmap v13172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:39.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:39 smithi122 ceph-mon[32255]: pgmap v13172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:40] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:17:40.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:41.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:41.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:41.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:41 smithi022 ceph-mon[31360]: pgmap v13173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/8853432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:41 smithi022 ceph-mon[40250]: pgmap v13173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/8853432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:41 smithi122 ceph-mon[32255]: pgmap v13173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/8853432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:17:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:17:43.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:43 smithi022 ceph-mon[31360]: pgmap v13174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:43.541 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:43 smithi022 ceph-mon[40250]: pgmap v13174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:43 smithi122 ceph-mon[32255]: pgmap v13174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:45 smithi122 ceph-mon[32255]: pgmap v13175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:45.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:17:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:17:45.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:45 smithi022 ceph-mon[31360]: pgmap v13175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:45 smithi022 ceph-mon[40250]: pgmap v13175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:46.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:46.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:46.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:46.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:46.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:47 smithi122 ceph-mon[32255]: pgmap v13176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2178539635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:47.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:17:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:47 smithi022 ceph-mon[31360]: pgmap v13176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2178539635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:47 smithi022 ceph-mon[40250]: pgmap v13176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2178539635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:49 smithi122 ceph-mon[32255]: pgmap v13177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:49 smithi022 ceph-mon[31360]: pgmap v13177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:49 smithi022 ceph-mon[40250]: pgmap v13177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:50] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:17:51.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:51 smithi122 ceph-mon[32255]: pgmap v13178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:51.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:51.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:51.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:51.748 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:51 smithi022 ceph-mon[40250]: pgmap v13178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:51.748 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:51 smithi022 ceph-mon[31360]: pgmap v13178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:17:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3452768313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:17:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:17:52.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3452768313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3452768313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:53 smithi122 ceph-mon[32255]: pgmap v13179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:53 smithi022 ceph-mon[31360]: pgmap v13179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:53 smithi022 ceph-mon[40250]: pgmap v13179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:17:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:55 smithi122 ceph-mon[32255]: pgmap v13180: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:55.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:17:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:17:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:55 smithi022 ceph-mon[31360]: pgmap v13180: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:55 smithi022 ceph-mon[40250]: pgmap v13180: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:56.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:17:56.750 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:56.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:17:57.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:17:57.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:17:57.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:17:57.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:57 smithi022 ceph-mon[31360]: pgmap v13181: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:57.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/997798192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:57 smithi022 ceph-mon[40250]: pgmap v13181: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:57.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/997798192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:57 smithi122 ceph-mon[32255]: pgmap v13181: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/997798192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:17:57.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:17:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:17:59 smithi122 ceph-mon[32255]: pgmap v13182: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:17:59 smithi022 ceph-mon[31360]: pgmap v13182: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:17:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:17:59 smithi022 ceph-mon[40250]: pgmap v13182: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T01:18:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:00] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:18:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:01 smithi122 ceph-mon[32255]: pgmap v13183: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T01:18:01.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:01 smithi022 ceph-mon[31360]: pgmap v13183: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T01:18:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:01 smithi022 ceph-mon[40250]: pgmap v13183: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T01:18:02.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:02.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:02.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:02.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:18:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:18:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:03 smithi122 ceph-mon[32255]: pgmap v13184: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T01:18:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3423370964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:03 smithi022 ceph-mon[31360]: pgmap v13184: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T01:18:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3423370964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:03 smithi022 ceph-mon[40250]: pgmap v13184: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T01:18:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3423370964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:05 smithi122 ceph-mon[32255]: pgmap v13185: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T01:18:05.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:18:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:18:05.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:05 smithi022 ceph-mon[31360]: pgmap v13185: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T01:18:05.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:05 smithi022 ceph-mon[40250]: pgmap v13185: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T01:18:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:06.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:07.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:07 smithi122 ceph-mon[32255]: pgmap v13186: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:18:07.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:07 smithi022 ceph-mon[31360]: pgmap v13186: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:18:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:18:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:07 smithi022 ceph-mon[40250]: pgmap v13186: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:18:07.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:07.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:07.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:08.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3753705374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:08.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3753705374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3753705374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:09 smithi122 ceph-mon[32255]: pgmap v13187: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:18:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:09 smithi022 ceph-mon[31360]: pgmap v13187: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:18:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:09 smithi022 ceph-mon[40250]: pgmap v13187: 132 pgs: 132 active+clean; 582 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:18:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:10] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:18:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:11 smithi122 ceph-mon[32255]: pgmap v13188: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:18:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:11 smithi022 ceph-mon[31360]: pgmap v13188: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:18:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:11 smithi022 ceph-mon[40250]: pgmap v13188: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:18:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:18:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:18:12.860 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:13.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:13.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:13 smithi022 ceph-mon[31360]: pgmap v13189: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2377154888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:13 smithi022 ceph-mon[40250]: pgmap v13189: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2377154888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:13 smithi122 ceph-mon[32255]: pgmap v13189: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2377154888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:15.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:18:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:18:15.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:15 smithi122 ceph-mon[32255]: pgmap v13190: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:18:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:15 smithi022 ceph-mon[31360]: pgmap v13190: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:18:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:15 smithi022 ceph-mon[40250]: pgmap v13190: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:18:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:16.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:17 smithi122 ceph-mon[32255]: pgmap v13191: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:17 smithi022 ceph-mon[31360]: pgmap v13191: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:18:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:17 smithi022 ceph-mon[40250]: pgmap v13191: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:18.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[31360]: pgmap v13192: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1468485734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:18:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:18:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:18:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[40250]: pgmap v13192: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1468485734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:18:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:18:19.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:18:19.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:19 smithi122 ceph-mon[32255]: pgmap v13192: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:18:19.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1468485734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:19.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:18:19.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:18:19.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:18:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:20] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:18:21.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:18:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:18:21.204 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:18:22.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:21 smithi022 ceph-mon[31360]: pgmap v13193: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:18:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:21 smithi022 ceph-mon[40250]: pgmap v13193: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:18:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:21 smithi122 ceph-mon[32255]: pgmap v13193: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:18:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:18:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:18:23.601 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:23.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:23.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:23.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:23.971 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:23 smithi022 ceph-mon[40250]: pgmap v13194: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:23.971 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:23 smithi022 ceph-mon[31360]: pgmap v13194: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:24.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:23 smithi122 ceph-mon[32255]: pgmap v13194: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/460326989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/460326989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:25.158 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/460326989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:25.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:18:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:18:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:25 smithi022 ceph-mon[31360]: pgmap v13195: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:25 smithi022 ceph-mon[40250]: pgmap v13195: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:25 smithi122 ceph-mon[32255]: pgmap v13195: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:27.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:27.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:27.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:18:28.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:27 smithi022 ceph-mon[31360]: pgmap v13196: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:27 smithi022 ceph-mon[40250]: pgmap v13196: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:27 smithi122 ceph-mon[32255]: pgmap v13196: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:28.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:29.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:29 smithi022 ceph-mon[40250]: pgmap v13197: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1913697898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:29 smithi022 ceph-mon[31360]: pgmap v13197: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1913697898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:30.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:29 smithi122 ceph-mon[32255]: pgmap v13197: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1913697898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:30] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:18:32.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:31 smithi022 ceph-mon[31360]: pgmap v13198: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:31 smithi022 ceph-mon[40250]: pgmap v13198: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:31 smithi122 ceph-mon[32255]: pgmap v13198: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:18:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:18:34.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:33 smithi022 ceph-mon[31360]: pgmap v13199: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:33 smithi022 ceph-mon[40250]: pgmap v13199: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:34.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:33 smithi122 ceph-mon[32255]: pgmap v13199: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:34.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:34.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:34.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:34.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:18:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:18:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:18:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:18:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2203332529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:18:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:18:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2203332529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:35.162 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:18:35.163 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:18:35.163 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:18:35.163 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:18:35.163 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2203332529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:35.441 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:18:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:18:36.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:35 smithi022 ceph-mon[31360]: pgmap v13200: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:35 smithi022 ceph-mon[40250]: pgmap v13200: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:36.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:35 smithi122 ceph-mon[32255]: pgmap v13200: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:37.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:37.729 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:18:38.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:37 smithi022 ceph-mon[31360]: pgmap v13201: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:37 smithi022 ceph-mon[40250]: pgmap v13201: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:37 smithi122 ceph-mon[32255]: pgmap v13201: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:39.710 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:40.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:39 smithi022 ceph-mon[31360]: pgmap v13202: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:39 smithi022 ceph-mon[40250]: pgmap v13202: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:40.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:40.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:40.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:40.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:39 smithi122 ceph-mon[32255]: pgmap v13202: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:40] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:18:41.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3899491190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3899491190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3899491190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:41 smithi022 ceph-mon[31360]: pgmap v13203: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:41 smithi022 ceph-mon[40250]: pgmap v13203: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:41 smithi122 ceph-mon[32255]: pgmap v13203: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:18:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:18:44.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:43 smithi022 ceph-mon[31360]: pgmap v13204: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:43 smithi022 ceph-mon[40250]: pgmap v13204: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:44.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:43 smithi122 ceph-mon[32255]: pgmap v13204: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:45.085 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:45.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:18:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:18:45.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:45.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:45.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:45 smithi022 ceph-mon[31360]: pgmap v13205: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2530080751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:45 smithi022 ceph-mon[40250]: pgmap v13205: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2530080751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:45 smithi122 ceph-mon[32255]: pgmap v13205: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2530080751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:47.751 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:18:48.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:47 smithi022 ceph-mon[31360]: pgmap v13206: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:47 smithi022 ceph-mon[40250]: pgmap v13206: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:47 smithi122 ceph-mon[32255]: pgmap v13206: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:49 smithi022 ceph-mon[31360]: pgmap v13207: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:49 smithi022 ceph-mon[40250]: pgmap v13207: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:49 smithi122 ceph-mon[32255]: pgmap v13207: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:18:50.453 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:50.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:52.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:51 smithi022 ceph-mon[31360]: pgmap v13208: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/918943110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:51 smithi022 ceph-mon[40250]: pgmap v13208: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/918943110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:51 smithi122 ceph-mon[32255]: pgmap v13208: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/918943110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:18:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:18:54.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:53 smithi022 ceph-mon[31360]: pgmap v13209: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:53 smithi022 ceph-mon[40250]: pgmap v13209: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:54.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:53 smithi122 ceph-mon[32255]: pgmap v13209: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:55.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:18:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:18:55.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:18:56.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:55 smithi022 ceph-mon[31360]: pgmap v13210: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:55 smithi022 ceph-mon[40250]: pgmap v13210: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:55 smithi122 ceph-mon[32255]: pgmap v13210: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:18:56.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:18:56.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:18:56.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:18:57.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1523060145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1523060145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:57.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:18:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1523060145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:18:57.768 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:18:58.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:57 smithi022 ceph-mon[31360]: pgmap v13211: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:58.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:57 smithi022 ceph-mon[40250]: pgmap v13211: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:18:58.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:57 smithi122 ceph-mon[32255]: pgmap v13211: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:00.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:18:59 smithi022 ceph-mon[31360]: pgmap v13212: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:18:59 smithi022 ceph-mon[40250]: pgmap v13212: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:18:59 smithi122 ceph-mon[32255]: pgmap v13212: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:00.443 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:19:01.202 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:01.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:01.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:01.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:02.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:01 smithi022 ceph-mon[31360]: pgmap v13213: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3860336000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:01 smithi022 ceph-mon[40250]: pgmap v13213: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3860336000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:01 smithi122 ceph-mon[32255]: pgmap v13213: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3860336000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:19:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:19:04.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:03 smithi022 ceph-mon[31360]: pgmap v13214: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:03 smithi022 ceph-mon[40250]: pgmap v13214: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:04.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:03 smithi122 ceph-mon[32255]: pgmap v13214: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:05.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:19:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:19:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:05 smithi122 ceph-mon[32255]: pgmap v13215: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:06.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:05 smithi022 ceph-mon[31360]: pgmap v13215: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:05 smithi022 ceph-mon[40250]: pgmap v13215: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:06.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:06.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:06.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:06.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:06.940 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:06.941 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:19:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:07 smithi122 ceph-mon[32255]: pgmap v13216: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2606613560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:08.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:07 smithi022 ceph-mon[31360]: pgmap v13216: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2606613560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:07 smithi022 ceph-mon[40250]: pgmap v13216: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2606613560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:09 smithi122 ceph-mon[32255]: pgmap v13217: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:10.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:09 smithi022 ceph-mon[31360]: pgmap v13217: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:09 smithi022 ceph-mon[40250]: pgmap v13217: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:19:11.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:11 smithi122 ceph-mon[32255]: pgmap v13218: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:12.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:11 smithi022 ceph-mon[31360]: pgmap v13218: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:12.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:11 smithi022 ceph-mon[40250]: pgmap v13218: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:12.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:12.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:12.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:19:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:19:13.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/362299671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:13.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/362299671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:13.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/362299671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:13 smithi122 ceph-mon[32255]: pgmap v13219: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:14.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:13 smithi022 ceph-mon[31360]: pgmap v13219: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:13 smithi022 ceph-mon[40250]: pgmap v13219: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:19:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:19:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:15 smithi122 ceph-mon[32255]: pgmap v13220: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:16.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:15 smithi022 ceph-mon[31360]: pgmap v13220: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:15 smithi022 ceph-mon[40250]: pgmap v13220: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:17.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:17.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:17.314 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:17.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:17.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:17.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:17.684 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:19:18.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:17 smithi022 ceph-mon[31360]: pgmap v13221: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3895439555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:18.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:17 smithi022 ceph-mon[40250]: pgmap v13221: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3895439555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:17 smithi122 ceph-mon[32255]: pgmap v13221: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3895439555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:20.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[31360]: pgmap v13222: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:19:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:19:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:19:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[40250]: pgmap v13222: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:19:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:19:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:19:20.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:19 smithi122 ceph-mon[32255]: pgmap v13222: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:20.128 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:19:20.128 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:19:20.128 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:19:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:20] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:19:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:21 smithi122 ceph-mon[32255]: pgmap v13223: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:19:21.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:21 smithi022 ceph-mon[31360]: pgmap v13223: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:19:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:21 smithi022 ceph-mon[40250]: pgmap v13223: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:19:22.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:19:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:19:23.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:23.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:23.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:23 smithi122 ceph-mon[32255]: pgmap v13224: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/827599038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:23.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:23 smithi022 ceph-mon[31360]: pgmap v13224: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/827599038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:23 smithi022 ceph-mon[40250]: pgmap v13224: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/827599038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:25.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:25 smithi022 ceph-mon[31360]: pgmap v13225: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:25 smithi022 ceph-mon[40250]: pgmap v13225: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:25 smithi122 ceph-mon[32255]: pgmap v13225: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:19:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:19:26.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:27.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:27 smithi022 ceph-mon[31360]: pgmap v13226: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:19:27.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:27 smithi022 ceph-mon[40250]: pgmap v13226: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:27 smithi122 ceph-mon[32255]: pgmap v13226: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:28.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:28.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:28.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:28.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:29.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:29 smithi022 ceph-mon[31360]: pgmap v13227: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2274881872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:29 smithi022 ceph-mon[40250]: pgmap v13227: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2274881872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:29 smithi122 ceph-mon[32255]: pgmap v13227: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2274881872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:30] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:19:31.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:31 smithi022 ceph-mon[31360]: pgmap v13228: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:31 smithi022 ceph-mon[40250]: pgmap v13228: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:31.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:31 smithi122 ceph-mon[32255]: pgmap v13228: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:19:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:19:33.424 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:33.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:33 smithi022 ceph-mon[40250]: pgmap v13229: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:33 smithi022 ceph-mon[31360]: pgmap v13229: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:33 smithi122 ceph-mon[32255]: pgmap v13229: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:33.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:33.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:33.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1038103712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:19:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1038103712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:19:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:19:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:19:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:19:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1038103712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:19:35.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:35 smithi022 ceph-mon[31360]: pgmap v13230: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:35 smithi022 ceph-mon[40250]: pgmap v13230: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:35 smithi122 ceph-mon[32255]: pgmap v13230: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:19:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:19:36.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:37 smithi022 ceph-mon[31360]: pgmap v13231: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:19:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:37 smithi022 ceph-mon[40250]: pgmap v13231: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:37 smithi122 ceph-mon[32255]: pgmap v13231: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:38.796 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:39.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:39 smithi022 ceph-mon[31360]: pgmap v13232: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2040930830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:39 smithi022 ceph-mon[40250]: pgmap v13232: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2040930830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:39 smithi122 ceph-mon[32255]: pgmap v13232: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2040930830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:40] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:19:41.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:41 smithi022 ceph-mon[31360]: pgmap v13233: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:41 smithi022 ceph-mon[40250]: pgmap v13233: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:41 smithi122 ceph-mon[32255]: pgmap v13233: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:19:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:19:43.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:43 smithi022 ceph-mon[31360]: pgmap v13234: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:43 smithi022 ceph-mon[40250]: pgmap v13234: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:43 smithi122 ceph-mon[32255]: pgmap v13234: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:44.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:44.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:44.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:44.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:45.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:45 smithi022 ceph-mon[31360]: pgmap v13235: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3959320324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:45 smithi022 ceph-mon[40250]: pgmap v13235: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3959320324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:45.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:19:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:19:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:45 smithi122 ceph-mon[32255]: pgmap v13235: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3959320324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:47.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:47 smithi022 ceph-mon[31360]: pgmap v13236: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:19:47.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:47 smithi022 ceph-mon[40250]: pgmap v13236: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:47 smithi122 ceph-mon[32255]: pgmap v13236: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:49.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:49 smithi022 ceph-mon[31360]: pgmap v13237: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:49 smithi022 ceph-mon[40250]: pgmap v13237: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:49.533 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:49 smithi122 ceph-mon[32255]: pgmap v13237: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:49.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3032258668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:19:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3032258668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3032258668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:51.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:51 smithi022 ceph-mon[31360]: pgmap v13238: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:51 smithi022 ceph-mon[40250]: pgmap v13238: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:51.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:51 smithi122 ceph-mon[32255]: pgmap v13238: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:19:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:19:53.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:53 smithi022 ceph-mon[31360]: pgmap v13239: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:53 smithi022 ceph-mon[40250]: pgmap v13239: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:53 smithi122 ceph-mon[32255]: pgmap v13239: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:54.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:19:55.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:19:55.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:19:55.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:19:55.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:55 smithi022 ceph-mon[31360]: pgmap v13240: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:55 smithi022 ceph-mon[40250]: pgmap v13240: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:55 smithi122 ceph-mon[32255]: pgmap v13240: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:19:55.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:19:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:19:56.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3643027092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3643027092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:56.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:19:56.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3643027092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:19:57.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:57 smithi022 ceph-mon[31360]: pgmap v13241: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:19:57.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:57 smithi022 ceph-mon[40250]: pgmap v13241: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:57 smithi122 ceph-mon[32255]: pgmap v13241: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:59.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:19:59 smithi022 ceph-mon[31360]: pgmap v13242: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:19:59 smithi022 ceph-mon[40250]: pgmap v13242: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:19:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:19:59 smithi122 ceph-mon[32255]: pgmap v13242: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:00.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T01:20:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:20:00.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T01:20:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T01:20:00.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:00.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:00.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:01.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:01 smithi022 ceph-mon[31360]: pgmap v13243: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3932331733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:01 smithi022 ceph-mon[40250]: pgmap v13243: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3932331733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:01 smithi122 ceph-mon[32255]: pgmap v13243: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3932331733' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:20:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:20:03.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:03 smithi022 ceph-mon[31360]: pgmap v13244: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:03 smithi022 ceph-mon[40250]: pgmap v13244: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:03 smithi122 ceph-mon[32255]: pgmap v13244: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:05.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:05 smithi022 ceph-mon[31360]: pgmap v13245: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:05 smithi022 ceph-mon[40250]: pgmap v13245: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:05.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:05.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:20:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:20:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:05 smithi122 ceph-mon[32255]: pgmap v13245: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:06.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:06.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:06.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:06.265 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:06.265 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2152883901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:06.266 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:06.266 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2152883901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2152883901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:07.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:07 smithi022 ceph-mon[31360]: pgmap v13246: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:20:07.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:07 smithi022 ceph-mon[40250]: pgmap v13246: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:07 smithi122 ceph-mon[32255]: pgmap v13246: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:09.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:09 smithi022 ceph-mon[31360]: pgmap v13247: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:09 smithi022 ceph-mon[40250]: pgmap v13247: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:09 smithi122 ceph-mon[32255]: pgmap v13247: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:10] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:20:11.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:11.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:11 smithi022 ceph-mon[31360]: pgmap v13248: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:11 smithi022 ceph-mon[40250]: pgmap v13248: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:11.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:11 smithi122 ceph-mon[32255]: pgmap v13248: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3423355438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3423355438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3423355438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:20:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:20:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:13 smithi122 ceph-mon[32255]: pgmap v13249: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:13.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:13 smithi022 ceph-mon[31360]: pgmap v13249: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:13 smithi022 ceph-mon[40250]: pgmap v13249: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:15 smithi122 ceph-mon[32255]: pgmap v13250: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:20:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:20:15.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:15 smithi022 ceph-mon[31360]: pgmap v13250: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:15 smithi022 ceph-mon[40250]: pgmap v13250: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:16.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:16.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:16.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:16.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:16.773 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:16.773 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:17.588 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:20:17.589 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:17 smithi022 ceph-mon[31360]: pgmap v13251: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:17.589 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4036882171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:17.590 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:17 smithi022 ceph-mon[40250]: pgmap v13251: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:17.590 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4036882171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:17 smithi122 ceph-mon[32255]: pgmap v13251: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4036882171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:19 smithi122 ceph-mon[32255]: pgmap v13252: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:19 smithi022 ceph-mon[31360]: pgmap v13252: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:19 smithi022 ceph-mon[40250]: pgmap v13252: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:20:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:20:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:20:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:20:20.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:20:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:20:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:20:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:20:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:20:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:20:21.775 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:21 smithi122 ceph-mon[32255]: pgmap v13253: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:20:22.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:21 smithi022 ceph-mon[31360]: pgmap v13253: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:22.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:20:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:21 smithi022 ceph-mon[40250]: pgmap v13253: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:20:22.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:22.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:22.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:22.630 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:20:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:20:22.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2287862359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:23.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2287862359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2287862359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:23.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:23 smithi122 ceph-mon[32255]: pgmap v13254: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:24.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:23 smithi022 ceph-mon[31360]: pgmap v13254: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:23 smithi022 ceph-mon[40250]: pgmap v13254: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:25.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:20:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:20:25.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:25 smithi122 ceph-mon[32255]: pgmap v13255: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:26.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:25 smithi022 ceph-mon[31360]: pgmap v13255: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:25 smithi022 ceph-mon[40250]: pgmap v13255: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:27.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:27.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:27.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:27.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:27.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:27.512 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:20:27.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:27 smithi022 ceph-mon[31360]: pgmap v13256: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/598636535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:27.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:27 smithi022 ceph-mon[40250]: pgmap v13256: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:27.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/598636535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:27 smithi122 ceph-mon[32255]: pgmap v13256: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/598636535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:29 smithi122 ceph-mon[32255]: pgmap v13257: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:30.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:29 smithi022 ceph-mon[31360]: pgmap v13257: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:29 smithi022 ceph-mon[40250]: pgmap v13257: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:20:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:31 smithi122 ceph-mon[32255]: pgmap v13258: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:32.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:31 smithi022 ceph-mon[31360]: pgmap v13258: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:31 smithi022 ceph-mon[40250]: pgmap v13258: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:32.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:20:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:20:32.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:32.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:32.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:33 smithi122 ceph-mon[32255]: pgmap v13259: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2212831483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:34.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:33 smithi022 ceph-mon[31360]: pgmap v13259: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2212831483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:33 smithi022 ceph-mon[40250]: pgmap v13259: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2212831483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:20:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:20:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:20:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:20:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:20:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:20:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:20:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:20:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:20:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:20:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:20:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:20:35.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:20:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:20:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:35 smithi122 ceph-mon[32255]: pgmap v13260: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:35 smithi022 ceph-mon[31360]: pgmap v13260: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:35 smithi022 ceph-mon[40250]: pgmap v13260: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:37.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:37.656 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:20:37.886 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:37.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:37 smithi122 ceph-mon[32255]: pgmap v13261: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:38.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:37 smithi022 ceph-mon[31360]: pgmap v13261: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:37 smithi022 ceph-mon[40250]: pgmap v13261: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:38.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:38.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:38.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:38.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4090168770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:39.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4090168770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4090168770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:39 smithi122 ceph-mon[32255]: pgmap v13262: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:40.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:39 smithi022 ceph-mon[31360]: pgmap v13262: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:39 smithi022 ceph-mon[40250]: pgmap v13262: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:40] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:20:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:41 smithi122 ceph-mon[32255]: pgmap v13263: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:42.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:41 smithi022 ceph-mon[31360]: pgmap v13263: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:41 smithi022 ceph-mon[40250]: pgmap v13263: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:20:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:20:43.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:43.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:43.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:43.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:43.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:43 smithi122 ceph-mon[32255]: pgmap v13264: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:43.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/720284422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:44.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:43 smithi022 ceph-mon[31360]: pgmap v13264: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/720284422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:43 smithi022 ceph-mon[40250]: pgmap v13264: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/720284422' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:45.673 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:20:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:20:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:45 smithi122 ceph-mon[32255]: pgmap v13265: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:46.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:45 smithi022 ceph-mon[31360]: pgmap v13265: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:45 smithi022 ceph-mon[40250]: pgmap v13265: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:46.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:47.678 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:20:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:47 smithi122 ceph-mon[32255]: pgmap v13266: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:48.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:47 smithi022 ceph-mon[31360]: pgmap v13266: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:47 smithi022 ceph-mon[40250]: pgmap v13266: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:48.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:49.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:49.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:49.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:49.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:49 smithi122 ceph-mon[32255]: pgmap v13267: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2175578729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:50.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:49 smithi022 ceph-mon[31360]: pgmap v13267: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2175578729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:49 smithi022 ceph-mon[40250]: pgmap v13267: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2175578729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:20:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:51 smithi022 ceph-mon[31360]: pgmap v13268: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:51 smithi022 ceph-mon[40250]: pgmap v13268: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:51 smithi122 ceph-mon[32255]: pgmap v13268: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:20:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:20:54.008 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:54.009 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:53 smithi022 ceph-mon[31360]: pgmap v13269: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:54.010 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:53 smithi022 ceph-mon[40250]: pgmap v13269: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:53 smithi122 ceph-mon[32255]: pgmap v13269: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:54.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:54.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:54.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:20:54.698 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2591104257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:55.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2591104257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:55.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2591104257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:20:55.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:20:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:20:56.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:55 smithi022 ceph-mon[31360]: pgmap v13270: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:55 smithi022 ceph-mon[40250]: pgmap v13270: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:55 smithi122 ceph-mon[32255]: pgmap v13270: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:20:57.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:57.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:20:57.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:20:58.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:57 smithi022 ceph-mon[31360]: pgmap v13271: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:58.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:57 smithi022 ceph-mon[40250]: pgmap v13271: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:57 smithi122 ceph-mon[32255]: pgmap v13271: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:20:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:20:59.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:20:59.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:20:59.742 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:00.038 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:20:59 smithi022 ceph-mon[31360]: pgmap v13272: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:00.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:20:59 smithi022 ceph-mon[40250]: pgmap v13272: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:20:59 smithi122 ceph-mon[32255]: pgmap v13272: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:21:01.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/941551311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/941551311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/941551311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:02.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:01 smithi022 ceph-mon[31360]: pgmap v13273: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:01 smithi022 ceph-mon[40250]: pgmap v13273: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:01 smithi122 ceph-mon[32255]: pgmap v13273: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:21:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:21:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:03 smithi022 ceph-mon[31360]: pgmap v13274: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:03 smithi022 ceph-mon[40250]: pgmap v13274: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:04.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:03 smithi122 ceph-mon[32255]: pgmap v13274: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:04.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:05.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:21:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:21:06.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:05 smithi022 ceph-mon[31360]: pgmap v13275: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:06.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2275740165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:05 smithi022 ceph-mon[40250]: pgmap v13275: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2275740165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:05 smithi122 ceph-mon[32255]: pgmap v13275: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2275740165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:07.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:07.723 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:21:08.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:07 smithi022 ceph-mon[31360]: pgmap v13276: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:08.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:07 smithi022 ceph-mon[40250]: pgmap v13276: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:07 smithi122 ceph-mon[32255]: pgmap v13276: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:09 smithi022 ceph-mon[31360]: pgmap v13277: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:09 smithi022 ceph-mon[40250]: pgmap v13277: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:10.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:09 smithi122 ceph-mon[32255]: pgmap v13277: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:10] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:21:10.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:10.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:10.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:10.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3535192254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3535192254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3535192254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:11 smithi022 ceph-mon[31360]: pgmap v13278: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:11 smithi022 ceph-mon[40250]: pgmap v13278: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:11 smithi122 ceph-mon[32255]: pgmap v13278: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:21:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:21:14.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:13 smithi022 ceph-mon[31360]: pgmap v13279: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:13 smithi022 ceph-mon[40250]: pgmap v13279: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:14.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:13 smithi122 ceph-mon[32255]: pgmap v13279: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:15.483 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:21:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:21:15.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:15.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:15.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:16.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:15 smithi022 ceph-mon[31360]: pgmap v13280: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:15 smithi022 ceph-mon[40250]: pgmap v13280: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:15 smithi122 ceph-mon[32255]: pgmap v13280: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:17.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1895740507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1895740507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:17.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1895740507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:17.742 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:21:18.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:17 smithi022 ceph-mon[31360]: pgmap v13281: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:18.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:17 smithi022 ceph-mon[40250]: pgmap v13281: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:17 smithi122 ceph-mon[32255]: pgmap v13281: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:20.027 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:19 smithi022 ceph-mon[40250]: pgmap v13282: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:19 smithi022 ceph-mon[31360]: pgmap v13282: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:20.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:19 smithi122 ceph-mon[32255]: pgmap v13282: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:21:20.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:21:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:21:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:21:21.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:21:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:21:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:21:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:21:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:21:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:21:21.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:21.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:21.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:22.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:21 smithi022 ceph-mon[31360]: pgmap v13283: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:21:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2851798439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:21 smithi022 ceph-mon[40250]: pgmap v13283: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:21:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2851798439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:21 smithi122 ceph-mon[32255]: pgmap v13283: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:21:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2851798439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:21:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:21:24.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:23 smithi022 ceph-mon[31360]: pgmap v13284: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:23 smithi022 ceph-mon[40250]: pgmap v13284: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:24.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:23 smithi122 ceph-mon[32255]: pgmap v13284: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:25.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:21:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:21:26.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:25 smithi022 ceph-mon[31360]: pgmap v13285: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:25 smithi022 ceph-mon[40250]: pgmap v13285: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:25 smithi122 ceph-mon[32255]: pgmap v13285: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:26.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:26.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:27.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1960782179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1960782179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:27.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:27.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1960782179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:27.761 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:21:28.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:27 smithi022 ceph-mon[31360]: pgmap v13286: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:27 smithi022 ceph-mon[40250]: pgmap v13286: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:27 smithi122 ceph-mon[32255]: pgmap v13286: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:30.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:29 smithi022 ceph-mon[31360]: pgmap v13287: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:29 smithi022 ceph-mon[40250]: pgmap v13287: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:30.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:29 smithi122 ceph-mon[32255]: pgmap v13287: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:21:31.587 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:31.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:31.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:31.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:31.957 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:31 smithi022 ceph-mon[31360]: pgmap v13288: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:31.958 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:31 smithi022 ceph-mon[40250]: pgmap v13288: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:31 smithi122 ceph-mon[32255]: pgmap v13288: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:21:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:21:33.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4068685467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4068685467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:33.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4068685467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:34.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:33 smithi022 ceph-mon[31360]: pgmap v13289: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:33 smithi022 ceph-mon[40250]: pgmap v13289: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:34.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:33 smithi122 ceph-mon[32255]: pgmap v13289: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:35.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:21:35.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:21:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:21:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:21:35.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:21:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:21:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:21:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:21:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:21:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:21:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:21:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:21:35.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:21:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:21:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:35 smithi122 ceph-mon[32255]: pgmap v13290: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:36.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:35 smithi022 ceph-mon[31360]: pgmap v13290: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:35 smithi022 ceph-mon[40250]: pgmap v13290: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:36.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:37.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:37.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:37.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:37.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:37.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:21:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:37 smithi122 ceph-mon[32255]: pgmap v13291: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/276989632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:38.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:37 smithi022 ceph-mon[31360]: pgmap v13291: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/276989632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:37 smithi022 ceph-mon[40250]: pgmap v13291: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/276989632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:40.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:39 smithi122 ceph-mon[32255]: pgmap v13292: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:39 smithi022 ceph-mon[31360]: pgmap v13292: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:39 smithi022 ceph-mon[40250]: pgmap v13292: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:40] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:21:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:41 smithi122 ceph-mon[32255]: pgmap v13293: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:42.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:41 smithi022 ceph-mon[31360]: pgmap v13293: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:41 smithi022 ceph-mon[40250]: pgmap v13293: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:42.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:21:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:21:42.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:42.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:42.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:43.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2700648525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2700648525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:43.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2700648525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:44.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:43 smithi122 ceph-mon[32255]: pgmap v13294: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:44.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:43 smithi022 ceph-mon[31360]: pgmap v13294: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:43 smithi022 ceph-mon[40250]: pgmap v13294: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:45.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:21:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:21:46.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:45 smithi122 ceph-mon[32255]: pgmap v13295: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:46.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:45 smithi022 ceph-mon[31360]: pgmap v13295: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:45 smithi022 ceph-mon[40250]: pgmap v13295: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:47.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:47.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:47.700 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:21:48.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:47 smithi022 ceph-mon[31360]: pgmap v13296: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:47 smithi022 ceph-mon[40250]: pgmap v13296: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:48.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:48.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:48.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:47 smithi122 ceph-mon[32255]: pgmap v13296: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:49.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2475556836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:49.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2475556836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2475556836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:50.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:49 smithi122 ceph-mon[32255]: pgmap v13297: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:50.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:49 smithi022 ceph-mon[31360]: pgmap v13297: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:49 smithi022 ceph-mon[40250]: pgmap v13297: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:50] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:21:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:51 smithi122 ceph-mon[32255]: pgmap v13298: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:52.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:51 smithi022 ceph-mon[31360]: pgmap v13298: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:51 smithi022 ceph-mon[40250]: pgmap v13298: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:21:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:21:53.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:53.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:53.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:53.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:54.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:53 smithi122 ceph-mon[32255]: pgmap v13299: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:54.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3429896117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:54.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:53 smithi022 ceph-mon[31360]: pgmap v13299: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3429896117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:53 smithi022 ceph-mon[40250]: pgmap v13299: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3429896117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:55.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:21:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:21:56.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:55 smithi122 ceph-mon[32255]: pgmap v13300: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:56.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:55 smithi022 ceph-mon[31360]: pgmap v13300: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:55 smithi022 ceph-mon[40250]: pgmap v13300: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:21:57.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:57.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:21:57.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:21:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:57 smithi122 ceph-mon[32255]: pgmap v13301: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:58.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:57 smithi022 ceph-mon[31360]: pgmap v13301: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:57 smithi022 ceph-mon[40250]: pgmap v13301: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:21:58.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:21:58.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:21:58.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:21:58.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:21:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2607382001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:59.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2607382001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:21:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2607382001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:21:59 smithi122 ceph-mon[32255]: pgmap v13302: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:00.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:21:59 smithi022 ceph-mon[31360]: pgmap v13302: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:00.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:21:59 smithi022 ceph-mon[40250]: pgmap v13302: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:00] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:22:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:01 smithi122 ceph-mon[32255]: pgmap v13303: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:02.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:01 smithi022 ceph-mon[31360]: pgmap v13303: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:01 smithi022 ceph-mon[40250]: pgmap v13303: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:22:02.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:22:03.812 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:04.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:04.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:04.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:04.180 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:03 smithi022 ceph-mon[31360]: pgmap v13304: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:04.180 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:03 smithi022 ceph-mon[40250]: pgmap v13304: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:04.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:03 smithi122 ceph-mon[32255]: pgmap v13304: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:05.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2135661676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:05.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2135661676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2135661676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:05.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:22:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:22:06.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:05 smithi122 ceph-mon[32255]: pgmap v13305: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:06.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:05 smithi022 ceph-mon[31360]: pgmap v13305: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:05 smithi022 ceph-mon[40250]: pgmap v13305: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:07.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:07.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:22:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:07 smithi122 ceph-mon[32255]: pgmap v13306: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:07 smithi022 ceph-mon[31360]: pgmap v13306: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:07 smithi022 ceph-mon[40250]: pgmap v13306: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:09.180 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:09.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:09.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:09.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:09 smithi122 ceph-mon[32255]: pgmap v13307: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2990652008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:10.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:09 smithi022 ceph-mon[31360]: pgmap v13307: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2990652008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:09 smithi022 ceph-mon[40250]: pgmap v13307: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2990652008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:22:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:11 smithi122 ceph-mon[32255]: pgmap v13308: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:12.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:11 smithi022 ceph-mon[31360]: pgmap v13308: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:11 smithi022 ceph-mon[40250]: pgmap v13308: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:22:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:22:14.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:13 smithi122 ceph-mon[32255]: pgmap v13309: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:14.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:13 smithi022 ceph-mon[31360]: pgmap v13309: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:13 smithi022 ceph-mon[40250]: pgmap v13309: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:14.553 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:22:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:22:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:15 smithi122 ceph-mon[32255]: pgmap v13310: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:16.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4022566207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:15 smithi022 ceph-mon[31360]: pgmap v13310: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4022566207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:15 smithi022 ceph-mon[40250]: pgmap v13310: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4022566207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:17.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:17.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:17.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:22:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:17 smithi122 ceph-mon[32255]: pgmap v13311: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:18.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:17 smithi022 ceph-mon[31360]: pgmap v13311: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:17 smithi022 ceph-mon[40250]: pgmap v13311: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:19.926 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:20.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:19 smithi122 ceph-mon[32255]: pgmap v13312: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:20.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:19 smithi022 ceph-mon[31360]: pgmap v13312: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:19 smithi022 ceph-mon[40250]: pgmap v13312: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:20] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:22:21.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/794878831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:21.146 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/794878831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:21.146 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/794878831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:21 smithi122 ceph-mon[32255]: pgmap v13313: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:22:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:22:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:22:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:22.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[31360]: pgmap v13313: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:22:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:22:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:22:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[40250]: pgmap v13313: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:22:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:22:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:22:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:22:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:22:23.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:23.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:22:24.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:23 smithi022 ceph-mon[31360]: pgmap v13314: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:23 smithi022 ceph-mon[40250]: pgmap v13314: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:23 smithi122 ceph-mon[32255]: pgmap v13314: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:25.297 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:25.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:25.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:25.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:25.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:22:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:22:26.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:25 smithi022 ceph-mon[31360]: pgmap v13315: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3577863670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:25 smithi022 ceph-mon[40250]: pgmap v13315: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3577863670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:25 smithi122 ceph-mon[32255]: pgmap v13315: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3577863670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:27.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:27.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:22:28.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:27 smithi022 ceph-mon[31360]: pgmap v13316: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:27 smithi022 ceph-mon[40250]: pgmap v13316: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:27 smithi122 ceph-mon[32255]: pgmap v13316: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:29 smithi022 ceph-mon[31360]: pgmap v13317: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:29 smithi022 ceph-mon[40250]: pgmap v13317: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:29 smithi122 ceph-mon[32255]: pgmap v13317: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:30] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:22:30.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:31.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:31.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:31.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:32.297 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:31 smithi022 ceph-mon[31360]: pgmap v13318: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:32.298 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3077741605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:32.298 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:31 smithi022 ceph-mon[40250]: pgmap v13318: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:32.317 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3077741605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:32.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:31 smithi122 ceph-mon[32255]: pgmap v13318: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:32.324 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3077741605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:22:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:22:34.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[31360]: pgmap v13319: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:22:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:22:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:22:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:22:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[40250]: pgmap v13319: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:22:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:22:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:22:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:22:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:33 smithi122 ceph-mon[32255]: pgmap v13319: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:22:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:22:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:22:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:22:35.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:22:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:22:36.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:36.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:35 smithi022 ceph-mon[40250]: pgmap v13320: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:35 smithi022 ceph-mon[31360]: pgmap v13320: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:36.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:36.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:35 smithi122 ceph-mon[32255]: pgmap v13320: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:37.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2706902498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2706902498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2706902498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:37.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:22:38.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:37 smithi022 ceph-mon[31360]: pgmap v13321: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:37 smithi022 ceph-mon[40250]: pgmap v13321: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:37 smithi122 ceph-mon[32255]: pgmap v13321: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:40.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:39 smithi022 ceph-mon[31360]: pgmap v13322: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:39 smithi022 ceph-mon[40250]: pgmap v13322: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:39 smithi122 ceph-mon[32255]: pgmap v13322: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:40] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:22:41.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:41.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:41.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:41.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:42.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:41 smithi022 ceph-mon[31360]: pgmap v13323: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/190670996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:41 smithi022 ceph-mon[40250]: pgmap v13323: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/190670996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:41 smithi122 ceph-mon[32255]: pgmap v13323: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/190670996' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:22:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:22:44.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:43 smithi022 ceph-mon[31360]: pgmap v13324: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:43 smithi022 ceph-mon[40250]: pgmap v13324: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:43 smithi122 ceph-mon[32255]: pgmap v13324: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:45.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:22:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:22:46.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:45 smithi022 ceph-mon[31360]: pgmap v13325: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:45 smithi022 ceph-mon[40250]: pgmap v13325: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:45 smithi122 ceph-mon[32255]: pgmap v13325: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:46.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:47.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:47.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:47.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:47.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:47.156 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:47.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:22:48.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:48 smithi022 ceph-mon[31360]: pgmap v13326: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2702975497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:48 smithi022 ceph-mon[40250]: pgmap v13326: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2702975497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:48 smithi122 ceph-mon[32255]: pgmap v13326: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2702975497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:50.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:50 smithi022 ceph-mon[31360]: pgmap v13327: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:50 smithi022 ceph-mon[40250]: pgmap v13327: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:50 smithi122 ceph-mon[32255]: pgmap v13327: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:50] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:22:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:52.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:52 smithi022 ceph-mon[31360]: pgmap v13328: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:52 smithi022 ceph-mon[40250]: pgmap v13328: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:52.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:52 smithi122 ceph-mon[32255]: pgmap v13328: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:52.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:52.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:52.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:22:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:22:53.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3280406403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:53.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3280406403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3280406403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:54.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:54 smithi022 ceph-mon[31360]: pgmap v13329: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:54 smithi022 ceph-mon[40250]: pgmap v13329: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:54 smithi122 ceph-mon[32255]: pgmap v13329: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:55.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:22:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:22:56.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:56 smithi022 ceph-mon[31360]: pgmap v13330: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:56 smithi022 ceph-mon[40250]: pgmap v13330: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:56 smithi122 ceph-mon[32255]: pgmap v13330: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:22:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:22:57.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:22:57.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:22:57.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:22:57.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:22:57.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:22:58.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:58 smithi022 ceph-mon[31360]: pgmap v13331: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:58.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:22:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1914635607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:58.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:58 smithi022 ceph-mon[40250]: pgmap v13331: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:58.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:22:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1914635607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:22:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:58 smithi122 ceph-mon[32255]: pgmap v13331: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:22:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:22:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1914635607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:00.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:00 smithi022 ceph-mon[31360]: pgmap v13332: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:00 smithi022 ceph-mon[40250]: pgmap v13332: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:00] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:23:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:00 smithi122 ceph-mon[32255]: pgmap v13332: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:02 smithi122 ceph-mon[32255]: pgmap v13333: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:02 smithi022 ceph-mon[31360]: pgmap v13333: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:02 smithi022 ceph-mon[40250]: pgmap v13333: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:23:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:23:02.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:03.267 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:04 smithi122 ceph-mon[32255]: pgmap v13334: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2176199843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:04.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:04 smithi022 ceph-mon[31360]: pgmap v13334: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2176199843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:04 smithi022 ceph-mon[40250]: pgmap v13334: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2176199843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:05.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:23:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:23:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:06 smithi122 ceph-mon[32255]: pgmap v13335: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:06 smithi022 ceph-mon[31360]: pgmap v13335: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:06 smithi022 ceph-mon[40250]: pgmap v13335: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:23:08.269 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:08 smithi122 ceph-mon[32255]: pgmap v13336: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:08.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:08 smithi022 ceph-mon[31360]: pgmap v13336: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:08.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:08 smithi022 ceph-mon[40250]: pgmap v13336: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:08.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:08.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:08.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2216568109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:09.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2216568109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2216568109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:23:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:10 smithi122 ceph-mon[32255]: pgmap v13337: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:10.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:10 smithi022 ceph-mon[31360]: pgmap v13337: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:10 smithi022 ceph-mon[40250]: pgmap v13337: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:12 smithi122 ceph-mon[32255]: pgmap v13338: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:12 smithi022 ceph-mon[31360]: pgmap v13338: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:12 smithi022 ceph-mon[40250]: pgmap v13338: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:23:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:23:13.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:14.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:14.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:14 smithi022 ceph-mon[31360]: pgmap v13339: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3278444032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:14 smithi022 ceph-mon[40250]: pgmap v13339: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3278444032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:14.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:14 smithi122 ceph-mon[32255]: pgmap v13339: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:14.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3278444032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:23:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:23:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:16 smithi122 ceph-mon[32255]: pgmap v13340: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:16.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:16 smithi022 ceph-mon[31360]: pgmap v13340: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:16 smithi022 ceph-mon[40250]: pgmap v13340: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:17.391 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:17.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:17.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:23:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:18 smithi122 ceph-mon[32255]: pgmap v13341: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:18.591 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:18 smithi022 ceph-mon[31360]: pgmap v13341: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:18.591 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:18 smithi022 ceph-mon[40250]: pgmap v13341: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:19.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:19.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:19.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:23:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:20 smithi122 ceph-mon[32255]: pgmap v13342: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1536563454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:20.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:20 smithi022 ceph-mon[31360]: pgmap v13342: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1536563454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:20 smithi022 ceph-mon[40250]: pgmap v13342: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1536563454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:22.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[31360]: pgmap v13343: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:23:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:23:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:23:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[40250]: pgmap v13343: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:23:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:23:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:23:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:22 smithi122 ceph-mon[32255]: pgmap v13343: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:23:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:23:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:23:22.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:23:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:23:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:23 smithi122 ceph-mon[32255]: pgmap v13344: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:23:23.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:23 smithi022 ceph-mon[31360]: pgmap v13344: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:23:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:23 smithi022 ceph-mon[40250]: pgmap v13344: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:23:24.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:24.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:24.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:24.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:25 smithi122 ceph-mon[32255]: pgmap v13345: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/301778044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:23:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:23:25.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:25 smithi022 ceph-mon[31360]: pgmap v13345: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/301778044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:25 smithi022 ceph-mon[40250]: pgmap v13345: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/301778044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:26.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:27 smithi122 ceph-mon[32255]: pgmap v13346: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:27.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:27 smithi022 ceph-mon[31360]: pgmap v13346: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:23:27.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:27 smithi022 ceph-mon[40250]: pgmap v13346: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:29 smithi122 ceph-mon[32255]: pgmap v13347: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:29.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:29.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:29 smithi022 ceph-mon[40250]: pgmap v13347: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:29.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:29 smithi022 ceph-mon[31360]: pgmap v13347: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:30.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:30.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:30.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:23:30.437 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1322140072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:30.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1322140072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:30.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1322140072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:31.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:31 smithi022 ceph-mon[31360]: pgmap v13348: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:31 smithi022 ceph-mon[40250]: pgmap v13348: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:31 smithi122 ceph-mon[32255]: pgmap v13348: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:23:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:23:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:33 smithi022 ceph-mon[31360]: pgmap v13349: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:33 smithi022 ceph-mon[40250]: pgmap v13349: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:33.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:33 smithi122 ceph-mon[32255]: pgmap v13349: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:34.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:23:34.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:23:34.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:23:34.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:23:34.793 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:23:34.794 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:23:34.794 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:23:34.794 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:23:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:23:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:23:35.132 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:35.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:35.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:35.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:35 smithi122 ceph-mon[32255]: pgmap v13350: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:23:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:23:35.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:35 smithi022 ceph-mon[31360]: pgmap v13350: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:35 smithi022 ceph-mon[40250]: pgmap v13350: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:36.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3829088851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3829088851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3829088851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:37.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:37 smithi022 ceph-mon[31360]: pgmap v13351: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:23:37.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:37 smithi022 ceph-mon[40250]: pgmap v13351: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:37 smithi122 ceph-mon[32255]: pgmap v13351: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:39.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:39 smithi022 ceph-mon[31360]: pgmap v13352: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:39 smithi022 ceph-mon[40250]: pgmap v13352: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:39.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:39 smithi122 ceph-mon[32255]: pgmap v13352: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:40] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:23:40.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:40.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:40.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:40.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:41.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:41 smithi022 ceph-mon[31360]: pgmap v13353: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/79127991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:41 smithi022 ceph-mon[40250]: pgmap v13353: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/79127991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:41 smithi122 ceph-mon[32255]: pgmap v13353: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/79127991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:23:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:23:43.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:43 smithi022 ceph-mon[31360]: pgmap v13354: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:43 smithi022 ceph-mon[40250]: pgmap v13354: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:43.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:43 smithi122 ceph-mon[32255]: pgmap v13354: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:45 smithi122 ceph-mon[32255]: pgmap v13355: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:23:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:23:45.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:45 smithi022 ceph-mon[31360]: pgmap v13355: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:45.791 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:45 smithi022 ceph-mon[40250]: pgmap v13355: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:45.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:46.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:46.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:46.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:46.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/64521976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/64521976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:46.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:46.956 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/64521976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:47.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:47 smithi022 ceph-mon[31360]: pgmap v13356: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:23:47.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:47 smithi022 ceph-mon[40250]: pgmap v13356: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:47 smithi122 ceph-mon[32255]: pgmap v13356: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:49.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:49 smithi022 ceph-mon[31360]: pgmap v13357: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:49 smithi022 ceph-mon[40250]: pgmap v13357: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:49.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:49 smithi122 ceph-mon[32255]: pgmap v13357: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:50] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:23:51.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:51.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:51 smithi022 ceph-mon[31360]: pgmap v13358: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:51 smithi022 ceph-mon[40250]: pgmap v13358: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:51.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:51.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:51.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:51.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:51 smithi122 ceph-mon[32255]: pgmap v13358: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1912626059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:23:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:23:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1912626059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1912626059' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:53.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:53 smithi022 ceph-mon[31360]: pgmap v13359: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:53 smithi022 ceph-mon[40250]: pgmap v13359: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:53 smithi122 ceph-mon[32255]: pgmap v13359: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:55 smithi122 ceph-mon[32255]: pgmap v13360: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:23:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:23:55.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:55 smithi022 ceph-mon[31360]: pgmap v13360: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:55 smithi022 ceph-mon[40250]: pgmap v13360: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:23:56.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:23:56.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:56.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:56.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:23:56.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:23:56.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:23:56.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:23:57.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:57 smithi022 ceph-mon[31360]: pgmap v13361: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2456552343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:23:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:57 smithi022 ceph-mon[40250]: pgmap v13361: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2456552343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:57 smithi122 ceph-mon[32255]: pgmap v13361: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2456552343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:23:59.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:23:59 smithi022 ceph-mon[31360]: pgmap v13362: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:23:59 smithi022 ceph-mon[40250]: pgmap v13362: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:23:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:23:59 smithi122 ceph-mon[32255]: pgmap v13362: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:00] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:24:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:01 smithi022 ceph-mon[31360]: pgmap v13363: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:01 smithi022 ceph-mon[40250]: pgmap v13363: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:01 smithi122 ceph-mon[32255]: pgmap v13363: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:01.982 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:02.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:02.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:02.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/636750970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:24:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:24:02.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/636750970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/636750970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:03.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:03 smithi022 ceph-mon[31360]: pgmap v13364: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:03 smithi022 ceph-mon[40250]: pgmap v13364: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:03.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:03 smithi122 ceph-mon[32255]: pgmap v13364: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:05.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:24:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:24:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:05 smithi122 ceph-mon[32255]: pgmap v13365: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:05.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:05 smithi022 ceph-mon[31360]: pgmap v13365: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:05 smithi022 ceph-mon[40250]: pgmap v13365: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:06.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:06.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:07.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:07.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:07.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:07.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:07.722 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:24:07.722 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:07 smithi022 ceph-mon[31360]: pgmap v13366: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:07.723 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:07 smithi022 ceph-mon[40250]: pgmap v13366: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:07 smithi122 ceph-mon[32255]: pgmap v13366: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2621702367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2621702367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2621702367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:09.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:09 smithi022 ceph-mon[31360]: pgmap v13367: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:09 smithi022 ceph-mon[40250]: pgmap v13367: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:09 smithi122 ceph-mon[32255]: pgmap v13367: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:24:11.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:11 smithi022 ceph-mon[31360]: pgmap v13368: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:11 smithi022 ceph-mon[40250]: pgmap v13368: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:11.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:11 smithi122 ceph-mon[32255]: pgmap v13368: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:24:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:24:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:13.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:13.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:13.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:13.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:13 smithi022 ceph-mon[31360]: pgmap v13369: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:13.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/663126274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:13.777 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:13 smithi022 ceph-mon[40250]: pgmap v13369: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/663126274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:13 smithi122 ceph-mon[32255]: pgmap v13369: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/663126274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:24:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:24:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:15 smithi122 ceph-mon[32255]: pgmap v13370: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:16.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:15 smithi022 ceph-mon[31360]: pgmap v13370: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:15 smithi022 ceph-mon[40250]: pgmap v13370: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:17.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:17.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:17 smithi022 ceph-mon[31360]: pgmap v13371: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:24:17.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:17 smithi022 ceph-mon[40250]: pgmap v13371: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:17 smithi122 ceph-mon[32255]: pgmap v13371: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:18.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:18.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:18.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:18.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:18.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1015715513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1015715513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1015715513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:19 smithi122 ceph-mon[32255]: pgmap v13372: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:20.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:19 smithi022 ceph-mon[31360]: pgmap v13372: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:19 smithi022 ceph-mon[40250]: pgmap v13372: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:24:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:21 smithi122 ceph-mon[32255]: pgmap v13373: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:22.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:21 smithi022 ceph-mon[31360]: pgmap v13373: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:21 smithi022 ceph-mon[40250]: pgmap v13373: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:24:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:24:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:24:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:24:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:24:22.777 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:24:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:24:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:24:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:24:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:24:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:24:23.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:23.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:23.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:23.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:24.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:23 smithi122 ceph-mon[32255]: pgmap v13374: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:24:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2431732814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:24.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:23 smithi022 ceph-mon[31360]: pgmap v13374: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:24:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2431732814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:23 smithi022 ceph-mon[40250]: pgmap v13374: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:24:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2431732814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:25.887 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:24:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:24:26.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:25 smithi122 ceph-mon[32255]: pgmap v13375: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:26.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:25 smithi022 ceph-mon[31360]: pgmap v13375: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:25 smithi022 ceph-mon[40250]: pgmap v13375: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:27.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:27.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:24:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:27 smithi122 ceph-mon[32255]: pgmap v13376: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:28.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:27 smithi022 ceph-mon[31360]: pgmap v13376: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:28.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:27 smithi022 ceph-mon[40250]: pgmap v13376: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:28.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:29.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:29.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:29.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:29 smithi122 ceph-mon[32255]: pgmap v13377: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2428883677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:30.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:29 smithi022 ceph-mon[31360]: pgmap v13377: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2428883677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:29 smithi022 ceph-mon[40250]: pgmap v13377: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2428883677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:24:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:31 smithi122 ceph-mon[32255]: pgmap v13378: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:31 smithi022 ceph-mon[31360]: pgmap v13378: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:31 smithi022 ceph-mon[40250]: pgmap v13378: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:24:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:24:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:33 smithi122 ceph-mon[32255]: pgmap v13379: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:24:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:24:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:24:34.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:24:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:34.208 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[31360]: pgmap v13379: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:34.208 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:24:34.208 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:24:34.209 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:24:34.209 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:24:34.209 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[40250]: pgmap v13379: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:34.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:24:34.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:24:34.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:24:34.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:24:34.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:34.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:34.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4228780839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:35.295 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4228780839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:35.296 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4228780839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:35.908 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:24:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:24:36.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:35 smithi122 ceph-mon[32255]: pgmap v13380: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:35 smithi022 ceph-mon[31360]: pgmap v13380: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:35 smithi022 ceph-mon[40250]: pgmap v13380: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:37.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:37.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:24:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:37 smithi122 ceph-mon[32255]: pgmap v13381: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:37 smithi022 ceph-mon[31360]: pgmap v13381: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:37 smithi022 ceph-mon[40250]: pgmap v13381: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:39.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:39.924 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:39 smithi022 ceph-mon[40250]: pgmap v13382: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:39.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:39.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:39.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:39 smithi122 ceph-mon[32255]: pgmap v13382: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:40.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:39 smithi022 ceph-mon[31360]: pgmap v13382: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:40] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:24:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/630459570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/630459570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/630459570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:41 smithi122 ceph-mon[32255]: pgmap v13383: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:41 smithi022 ceph-mon[31360]: pgmap v13383: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:41 smithi022 ceph-mon[40250]: pgmap v13383: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:24:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:24:44.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:43 smithi122 ceph-mon[32255]: pgmap v13384: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:43 smithi022 ceph-mon[31360]: pgmap v13384: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:43 smithi022 ceph-mon[40250]: pgmap v13384: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:44.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:45.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:45.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:45.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:45.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:24:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:24:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:45 smithi122 ceph-mon[32255]: pgmap v13385: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:46.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4292349138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:45 smithi022 ceph-mon[31360]: pgmap v13385: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4292349138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:45 smithi022 ceph-mon[40250]: pgmap v13385: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4292349138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:47.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:24:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:47 smithi122 ceph-mon[32255]: pgmap v13386: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:47 smithi022 ceph-mon[31360]: pgmap v13386: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:47 smithi022 ceph-mon[40250]: pgmap v13386: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:49 smithi022 ceph-mon[31360]: pgmap v13387: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:49 smithi022 ceph-mon[40250]: pgmap v13387: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:50.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:50] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:24:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:49 smithi122 ceph-mon[32255]: pgmap v13387: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:50.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:50.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:50.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:51.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806134395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806134395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806134395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:51 smithi022 ceph-mon[31360]: pgmap v13388: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:51 smithi022 ceph-mon[40250]: pgmap v13388: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:51 smithi122 ceph-mon[32255]: pgmap v13388: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:24:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:24:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:53 smithi022 ceph-mon[31360]: pgmap v13389: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:53 smithi022 ceph-mon[40250]: pgmap v13389: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:53 smithi122 ceph-mon[32255]: pgmap v13389: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:55.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:24:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:24:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:24:56.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:24:56.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:24:56.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:24:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:55 smithi022 ceph-mon[31360]: pgmap v13390: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:55 smithi022 ceph-mon[40250]: pgmap v13390: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:55 smithi122 ceph-mon[32255]: pgmap v13390: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:24:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4105036498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4105036498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:24:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4105036498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:24:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:24:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:57 smithi022 ceph-mon[31360]: pgmap v13391: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:57 smithi022 ceph-mon[40250]: pgmap v13391: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:24:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:57 smithi122 ceph-mon[32255]: pgmap v13391: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:24:59 smithi022 ceph-mon[31360]: pgmap v13392: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:24:59 smithi022 ceph-mon[40250]: pgmap v13392: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:00] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:25:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:24:59 smithi122 ceph-mon[32255]: pgmap v13392: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:01.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:01.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:01.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:01.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:01 smithi022 ceph-mon[31360]: pgmap v13393: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2773254222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:01 smithi022 ceph-mon[40250]: pgmap v13393: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2773254222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:01 smithi122 ceph-mon[32255]: pgmap v13393: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2773254222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:25:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:25:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:03 smithi022 ceph-mon[31360]: pgmap v13394: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:03 smithi022 ceph-mon[40250]: pgmap v13394: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:04.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:03 smithi122 ceph-mon[32255]: pgmap v13394: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:05.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:25:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:25:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:05 smithi022 ceph-mon[31360]: pgmap v13395: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:05 smithi022 ceph-mon[40250]: pgmap v13395: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:06.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:05 smithi122 ceph-mon[32255]: pgmap v13395: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:06.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:06.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:06.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2824678345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2824678345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2824678345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:25:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:07 smithi022 ceph-mon[31360]: pgmap v13396: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:07 smithi022 ceph-mon[40250]: pgmap v13396: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:07 smithi122 ceph-mon[32255]: pgmap v13396: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:09 smithi022 ceph-mon[31360]: pgmap v13397: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:09 smithi022 ceph-mon[40250]: pgmap v13397: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:09 smithi122 ceph-mon[32255]: pgmap v13397: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:25:11.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:12.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:12.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:12.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:12.178 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:11 smithi022 ceph-mon[31360]: pgmap v13398: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:12.178 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:11 smithi022 ceph-mon[40250]: pgmap v13398: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:11 smithi122 ceph-mon[32255]: pgmap v13398: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:25:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:25:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2448584601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2448584601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2448584601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:13 smithi022 ceph-mon[31360]: pgmap v13399: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:13 smithi022 ceph-mon[40250]: pgmap v13399: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:13 smithi122 ceph-mon[32255]: pgmap v13399: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:25:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:25:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:15 smithi022 ceph-mon[31360]: pgmap v13400: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:15 smithi022 ceph-mon[40250]: pgmap v13400: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:15 smithi122 ceph-mon[32255]: pgmap v13400: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:17.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:17.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:17.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:17.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:25:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:17 smithi022 ceph-mon[31360]: pgmap v13401: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2486125795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:17 smithi022 ceph-mon[40250]: pgmap v13401: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2486125795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:17 smithi122 ceph-mon[32255]: pgmap v13401: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2486125795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:19 smithi022 ceph-mon[31360]: pgmap v13402: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:19 smithi022 ceph-mon[40250]: pgmap v13402: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:25:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:19 smithi122 ceph-mon[32255]: pgmap v13402: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:21 smithi022 ceph-mon[31360]: pgmap v13403: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:22 smithi022 ceph-mon[40250]: pgmap v13403: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:22 smithi122 ceph-mon[32255]: pgmap v13403: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:22.551 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:25:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:25:22.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:22.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:22.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:25:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:25:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:25:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3886319478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:25:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:25:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:25:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3886319478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:23.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:25:23.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:25:23.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:25:23.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3886319478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:24 smithi022 ceph-mon[31360]: pgmap v13404: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:25:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:24 smithi022 ceph-mon[40250]: pgmap v13404: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:25:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:24 smithi122 ceph-mon[32255]: pgmap v13404: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:25:25.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:25:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:25:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:26 smithi022 ceph-mon[31360]: pgmap v13405: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:26 smithi022 ceph-mon[40250]: pgmap v13405: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:26 smithi122 ceph-mon[32255]: pgmap v13405: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:25:27.924 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:28 smithi022 ceph-mon[40250]: pgmap v13406: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:28 smithi022 ceph-mon[31360]: pgmap v13406: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:28.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:28.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:28.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:28 smithi122 ceph-mon[32255]: pgmap v13406: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2522790334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2522790334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2522790334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:30 smithi022 ceph-mon[31360]: pgmap v13407: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:30 smithi022 ceph-mon[40250]: pgmap v13407: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:25:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:30 smithi122 ceph-mon[32255]: pgmap v13407: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:32 smithi022 ceph-mon[31360]: pgmap v13408: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:32 smithi022 ceph-mon[40250]: pgmap v13408: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:32 smithi122 ceph-mon[32255]: pgmap v13408: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:25:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:25:33.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:33.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:33.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:33.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[31360]: pgmap v13409: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/778879410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:25:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:25:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:25:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:25:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[40250]: pgmap v13409: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/778879410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:25:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:25:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:25:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:25:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:34 smithi122 ceph-mon[32255]: pgmap v13409: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/778879410' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:25:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:25:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:25:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:25:35.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:25:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:25:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:36 smithi022 ceph-mon[31360]: pgmap v13410: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:36 smithi022 ceph-mon[40250]: pgmap v13410: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:36 smithi122 ceph-mon[32255]: pgmap v13410: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:37.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:25:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:38 smithi122 ceph-mon[32255]: pgmap v13411: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:38 smithi022 ceph-mon[31360]: pgmap v13411: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:38 smithi022 ceph-mon[40250]: pgmap v13411: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:38.661 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:39.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:39.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:39.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/265720126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/265720126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/265720126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:40 smithi122 ceph-mon[32255]: pgmap v13412: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:40] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:25:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:40 smithi022 ceph-mon[31360]: pgmap v13412: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:40 smithi022 ceph-mon[40250]: pgmap v13412: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:42 smithi122 ceph-mon[32255]: pgmap v13413: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:42 smithi022 ceph-mon[31360]: pgmap v13413: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:42 smithi022 ceph-mon[40250]: pgmap v13413: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:25:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:25:44.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:44.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:44.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:44.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:44.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:44 smithi022 ceph-mon[31360]: pgmap v13414: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:44.402 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:44 smithi022 ceph-mon[40250]: pgmap v13414: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:44 smithi122 ceph-mon[32255]: pgmap v13414: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1058532731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1058532731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1058532731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:45.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:25:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:25:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:46 smithi122 ceph-mon[32255]: pgmap v13415: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:46 smithi022 ceph-mon[31360]: pgmap v13415: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:46 smithi022 ceph-mon[40250]: pgmap v13415: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:25:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:48 smithi122 ceph-mon[32255]: pgmap v13416: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:48 smithi022 ceph-mon[31360]: pgmap v13416: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:48 smithi022 ceph-mon[40250]: pgmap v13416: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:49.402 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:49.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:49.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:49.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:25:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:50 smithi122 ceph-mon[32255]: pgmap v13417: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1324367377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:50 smithi022 ceph-mon[31360]: pgmap v13417: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1324367377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:50 smithi022 ceph-mon[40250]: pgmap v13417: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1324367377' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:52 smithi122 ceph-mon[32255]: pgmap v13418: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:25:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:25:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:52 smithi022 ceph-mon[31360]: pgmap v13418: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:52 smithi022 ceph-mon[40250]: pgmap v13418: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:54 smithi122 ceph-mon[32255]: pgmap v13419: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:54 smithi022 ceph-mon[31360]: pgmap v13419: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:54 smithi022 ceph-mon[40250]: pgmap v13419: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:54.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:25:55.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:25:55.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:25:55.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:25:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:25:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:25:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:56 smithi122 ceph-mon[32255]: pgmap v13420: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1248792395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:56 smithi022 ceph-mon[31360]: pgmap v13420: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1248792395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:56 smithi022 ceph-mon[40250]: pgmap v13420: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:25:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1248792395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:25:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:25:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:25:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:25:58 smithi122 ceph-mon[32255]: pgmap v13421: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:25:58 smithi022 ceph-mon[31360]: pgmap v13421: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:25:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:25:58 smithi022 ceph-mon[40250]: pgmap v13421: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:00.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:00 smithi122 ceph-mon[32255]: pgmap v13422: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:26:00.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:00.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:00.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:00.510 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:00 smithi022 ceph-mon[31360]: pgmap v13422: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:00.512 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:00 smithi022 ceph-mon[40250]: pgmap v13422: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/913360475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/913360475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/913360475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:02 smithi122 ceph-mon[32255]: pgmap v13423: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:26:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:26:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:02 smithi022 ceph-mon[31360]: pgmap v13423: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:02 smithi022 ceph-mon[40250]: pgmap v13423: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:04 smithi122 ceph-mon[32255]: pgmap v13424: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:04 smithi022 ceph-mon[31360]: pgmap v13424: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:04 smithi022 ceph-mon[40250]: pgmap v13424: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:05.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:05.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:05.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:05.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:05.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:26:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:26:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:06 smithi022 ceph-mon[31360]: pgmap v13425: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3952562485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:06 smithi022 ceph-mon[40250]: pgmap v13425: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3952562485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:06 smithi122 ceph-mon[32255]: pgmap v13425: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3952562485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:26:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:08 smithi122 ceph-mon[32255]: pgmap v13426: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:08 smithi022 ceph-mon[31360]: pgmap v13426: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:08 smithi022 ceph-mon[40250]: pgmap v13426: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:10] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:26:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:10 smithi122 ceph-mon[32255]: pgmap v13427: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:10 smithi022 ceph-mon[31360]: pgmap v13427: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:10 smithi022 ceph-mon[40250]: pgmap v13427: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:10.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:11.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:11.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:11.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:12 smithi122 ceph-mon[32255]: pgmap v13428: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2807681407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:26:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:26:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:12 smithi022 ceph-mon[31360]: pgmap v13428: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2807681407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:12 smithi022 ceph-mon[40250]: pgmap v13428: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2807681407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:14 smithi122 ceph-mon[32255]: pgmap v13429: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:14 smithi022 ceph-mon[31360]: pgmap v13429: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:14 smithi022 ceph-mon[40250]: pgmap v13429: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:15 smithi122 ceph-mon[32255]: pgmap v13430: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:15 smithi022 ceph-mon[31360]: pgmap v13430: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:15 smithi022 ceph-mon[40250]: pgmap v13430: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:26:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:26:16.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:16.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:16.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:16.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:17 smithi022 ceph-mon[31360]: pgmap v13431: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/809042771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:26:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:17 smithi022 ceph-mon[40250]: pgmap v13431: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:17.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/809042771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:17 smithi122 ceph-mon[32255]: pgmap v13431: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/809042771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:19 smithi022 ceph-mon[31360]: pgmap v13432: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:19 smithi022 ceph-mon[40250]: pgmap v13432: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:19 smithi122 ceph-mon[32255]: pgmap v13432: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:26:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:21 smithi022 ceph-mon[31360]: pgmap v13433: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:21 smithi022 ceph-mon[40250]: pgmap v13433: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:21.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:21.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:21 smithi122 ceph-mon[32255]: pgmap v13433: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:21.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/753026399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:26:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:26:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/753026399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/753026399' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:23 smithi122 ceph-mon[32255]: pgmap v13434: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:23 smithi022 ceph-mon[31360]: pgmap v13434: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:23 smithi022 ceph-mon[40250]: pgmap v13434: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:26:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:26:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:26:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:26:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:26:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:26:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:26:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:26:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:26:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:26:25.549 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:25 smithi122 ceph-mon[32255]: pgmap v13435: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:25 smithi022 ceph-mon[31360]: pgmap v13435: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:25 smithi022 ceph-mon[40250]: pgmap v13435: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:25.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:26:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:26:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:26.996 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:27.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:27.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:27.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:26:28.395 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: pgmap v13436: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:28.396 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.396 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3246847958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:28.396 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.397 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.397 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.397 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.399 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.399 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: pgmap v13436: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3246847958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: pgmap v13436: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3246847958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:28.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:28.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:28.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3022M 2021-12-16T01:26:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:29.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3022M 2021-12-16T01:26:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3022M 2021-12-16T01:26:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:26:29.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:29.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:26:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:26:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:30 smithi122 ceph-mon[32255]: pgmap v13437: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:30 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3242M 2021-12-16T01:26:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:30 smithi022 ceph-mon[31360]: pgmap v13437: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:30 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3242M 2021-12-16T01:26:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:30 smithi022 ceph-mon[40250]: pgmap v13437: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:30 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3242M 2021-12-16T01:26:32.368 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:32 smithi022 ceph-mon[40250]: pgmap v13438: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:32.370 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:32 smithi022 ceph-mon[31360]: pgmap v13438: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:32.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:32 smithi122 ceph-mon[32255]: pgmap v13438: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:26:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:26:32.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:32.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:32.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1497184349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1497184349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1497184349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:34 smithi122 ceph-mon[32255]: pgmap v13439: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:26:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:26:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:26:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:26:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[31360]: pgmap v13439: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:26:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:26:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:26:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:26:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[40250]: pgmap v13439: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:26:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:26:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:26:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:26:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:35 smithi122 ceph-mon[32255]: pgmap v13440: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:35 smithi022 ceph-mon[31360]: pgmap v13440: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:35 smithi022 ceph-mon[40250]: pgmap v13440: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:35.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:26:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:26:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:37 smithi022 ceph-mon[31360]: pgmap v13441: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:26:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:37 smithi022 ceph-mon[40250]: pgmap v13441: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:37 smithi122 ceph-mon[32255]: pgmap v13441: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:37.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:38.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:38.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:38.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/708322610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/708322610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/708322610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:39 smithi122 ceph-mon[32255]: pgmap v13442: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:39 smithi022 ceph-mon[31360]: pgmap v13442: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:39 smithi022 ceph-mon[40250]: pgmap v13442: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:40] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:26:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:41 smithi122 ceph-mon[32255]: pgmap v13443: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:41 smithi022 ceph-mon[31360]: pgmap v13443: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:41 smithi022 ceph-mon[40250]: pgmap v13443: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:26:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:43.483 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:43 smithi022 ceph-mon[31360]: pgmap v13444: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:43.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:43 smithi022 ceph-mon[40250]: pgmap v13444: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:43 smithi122 ceph-mon[32255]: pgmap v13444: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2569417586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2569417586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:44.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2569417586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:45.570 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:45 smithi122 ceph-mon[32255]: pgmap v13445: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:45 smithi022 ceph-mon[31360]: pgmap v13445: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:45 smithi022 ceph-mon[40250]: pgmap v13445: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:45.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:26:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:26:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:47 smithi122 ceph-mon[32255]: pgmap v13446: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:47 smithi022 ceph-mon[31360]: pgmap v13446: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:26:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:47 smithi022 ceph-mon[40250]: pgmap v13446: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:48.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:48.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:48.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:48.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:49 smithi122 ceph-mon[32255]: pgmap v13447: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/784546125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:49 smithi022 ceph-mon[31360]: pgmap v13447: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/784546125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:49 smithi022 ceph-mon[40250]: pgmap v13447: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/784546125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:50] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T01:26:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:51 smithi122 ceph-mon[32255]: pgmap v13448: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:51 smithi022 ceph-mon[31360]: pgmap v13448: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:51 smithi022 ceph-mon[40250]: pgmap v13448: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:26:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:26:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:53 smithi122 ceph-mon[32255]: pgmap v13449: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:53 smithi022 ceph-mon[31360]: pgmap v13449: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:53 smithi022 ceph-mon[40250]: pgmap v13449: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:53.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:54.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:54.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:54.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/864880080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/864880080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/864880080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:26:55.581 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:55 smithi122 ceph-mon[32255]: pgmap v13450: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:55 smithi022 ceph-mon[31360]: pgmap v13450: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:55 smithi022 ceph-mon[40250]: pgmap v13450: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:26:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:26:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:26:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:26:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:57 smithi122 ceph-mon[32255]: pgmap v13451: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:57 smithi022 ceph-mon[31360]: pgmap v13451: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:26:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:57 smithi022 ceph-mon[40250]: pgmap v13451: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:59.226 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:26:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:26:59 smithi022 ceph-mon[31360]: pgmap v13452: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:26:59 smithi022 ceph-mon[40250]: pgmap v13452: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:26:59.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:26:59.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:26:59.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:26:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:26:59 smithi122 ceph-mon[32255]: pgmap v13452: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3301659147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:00] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T01:27:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3301659147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3301659147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:01.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:01 smithi122 ceph-mon[32255]: pgmap v13453: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:01 smithi022 ceph-mon[31360]: pgmap v13453: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:01 smithi022 ceph-mon[40250]: pgmap v13453: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:27:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:27:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:03 smithi122 ceph-mon[32255]: pgmap v13454: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:03 smithi022 ceph-mon[31360]: pgmap v13454: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:03 smithi022 ceph-mon[40250]: pgmap v13454: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:04.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:04.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:04.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:04.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:05.590 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:05 smithi122 ceph-mon[32255]: pgmap v13455: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:05.590 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4215700923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:05 smithi022 ceph-mon[31360]: pgmap v13455: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4215700923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:05 smithi022 ceph-mon[40250]: pgmap v13455: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4215700923' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:05.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:27:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:27:06.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:07 smithi122 ceph-mon[32255]: pgmap v13456: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:07 smithi022 ceph-mon[31360]: pgmap v13456: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:27:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:07 smithi022 ceph-mon[40250]: pgmap v13456: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:09 smithi122 ceph-mon[32255]: pgmap v13457: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:09 smithi022 ceph-mon[31360]: pgmap v13457: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:09 smithi022 ceph-mon[40250]: pgmap v13457: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:09.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:10.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:10.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:10.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:10] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T01:27:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2853794808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2853794808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2853794808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:11 smithi122 ceph-mon[32255]: pgmap v13458: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:11 smithi022 ceph-mon[31360]: pgmap v13458: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:11 smithi022 ceph-mon[40250]: pgmap v13458: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:27:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:27:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:13 smithi122 ceph-mon[32255]: pgmap v13459: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:13 smithi022 ceph-mon[31360]: pgmap v13459: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:13 smithi022 ceph-mon[40250]: pgmap v13459: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:15.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:15.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:27:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:27:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:15 smithi122 ceph-mon[32255]: pgmap v13460: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:15.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:15.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:15.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:15.708 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:15 smithi022 ceph-mon[31360]: pgmap v13460: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:15.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:15 smithi022 ceph-mon[40250]: pgmap v13460: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3340347765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3340347765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3340347765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:17 smithi122 ceph-mon[32255]: pgmap v13461: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:17 smithi022 ceph-mon[31360]: pgmap v13461: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:27:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:17 smithi022 ceph-mon[40250]: pgmap v13461: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:19 smithi122 ceph-mon[32255]: pgmap v13462: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:19 smithi022 ceph-mon[31360]: pgmap v13462: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:19 smithi022 ceph-mon[40250]: pgmap v13462: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:27:20.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:21.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:21.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:21.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:21 smithi122 ceph-mon[32255]: pgmap v13463: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2252605718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:21 smithi022 ceph-mon[31360]: pgmap v13463: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2252605718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:21 smithi022 ceph-mon[40250]: pgmap v13463: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2252605718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:27:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:27:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:23 smithi122 ceph-mon[32255]: pgmap v13464: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:23 smithi022 ceph-mon[31360]: pgmap v13464: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:23 smithi022 ceph-mon[40250]: pgmap v13464: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:25 smithi122 ceph-mon[32255]: pgmap v13465: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:27:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:27:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:25 smithi022 ceph-mon[31360]: pgmap v13465: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:25 smithi022 ceph-mon[40250]: pgmap v13465: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:26.078 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:27 smithi122 ceph-mon[32255]: pgmap v13466: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3734194683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:27:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:27 smithi022 ceph-mon[31360]: pgmap v13466: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3734194683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:27 smithi022 ceph-mon[40250]: pgmap v13466: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3734194683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:29 smithi122 ceph-mon[32255]: pgmap v13467: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:27:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:27:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:27:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:27:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[31360]: pgmap v13467: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:27:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:27:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:27:29.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:27:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[40250]: pgmap v13467: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:27:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:27:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:27:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:27:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:27:31.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:31 smithi122 ceph-mon[32255]: pgmap v13468: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:31 smithi022 ceph-mon[31360]: pgmap v13468: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:31 smithi022 ceph-mon[40250]: pgmap v13468: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:31.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:31.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:31.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3889949052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:27:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:27:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3889949052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3889949052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:33 smithi122 ceph-mon[32255]: pgmap v13469: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:33 smithi022 ceph-mon[31360]: pgmap v13469: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:33 smithi022 ceph-mon[40250]: pgmap v13469: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:27:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:27:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:27:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:27:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:27:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:27:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:27:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:27:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:27:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:35 smithi122 ceph-mon[32255]: pgmap v13470: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:35.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:27:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:27:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:35 smithi022 ceph-mon[31360]: pgmap v13470: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:35 smithi022 ceph-mon[40250]: pgmap v13470: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:36.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:37.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:37.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:37.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:27:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:37 smithi022 ceph-mon[31360]: pgmap v13471: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/323352102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:37 smithi022 ceph-mon[40250]: pgmap v13471: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/323352102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:37 smithi122 ceph-mon[32255]: pgmap v13471: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/323352102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:39 smithi122 ceph-mon[32255]: pgmap v13472: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:39 smithi022 ceph-mon[31360]: pgmap v13472: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:39 smithi022 ceph-mon[40250]: pgmap v13472: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:40] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T01:27:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:41 smithi122 ceph-mon[32255]: pgmap v13473: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:41 smithi022 ceph-mon[31360]: pgmap v13473: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:41 smithi022 ceph-mon[40250]: pgmap v13473: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:42.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:42.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:42.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:42.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:27:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:27:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:43 smithi122 ceph-mon[32255]: pgmap v13474: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2563373605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:43 smithi022 ceph-mon[31360]: pgmap v13474: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2563373605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:43 smithi022 ceph-mon[40250]: pgmap v13474: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2563373605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:45 smithi122 ceph-mon[32255]: pgmap v13475: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:45.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:27:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:27:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:45 smithi022 ceph-mon[31360]: pgmap v13475: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:45 smithi022 ceph-mon[40250]: pgmap v13475: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:47.556 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:47 smithi122 ceph-mon[32255]: pgmap v13476: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:47 smithi022 ceph-mon[31360]: pgmap v13476: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:27:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:47 smithi022 ceph-mon[40250]: pgmap v13476: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:47.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:47.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:47.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2731632221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2731632221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2731632221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:49 smithi122 ceph-mon[32255]: pgmap v13477: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:49 smithi022 ceph-mon[31360]: pgmap v13477: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:49 smithi022 ceph-mon[40250]: pgmap v13477: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:50] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T01:27:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:51 smithi122 ceph-mon[32255]: pgmap v13478: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:51 smithi022 ceph-mon[31360]: pgmap v13478: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:51 smithi022 ceph-mon[40250]: pgmap v13478: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:27:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:27:52.927 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:53.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:53.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:53.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:53 smithi122 ceph-mon[32255]: pgmap v13479: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/547809833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:53 smithi022 ceph-mon[31360]: pgmap v13479: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/547809833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:53 smithi022 ceph-mon[40250]: pgmap v13479: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/547809833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:55 smithi122 ceph-mon[32255]: pgmap v13480: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:27:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:27:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:55 smithi022 ceph-mon[31360]: pgmap v13480: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:55 smithi022 ceph-mon[40250]: pgmap v13480: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:27:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:56.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:27:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:57 smithi022 ceph-mon[31360]: pgmap v13481: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:27:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:57 smithi022 ceph-mon[40250]: pgmap v13481: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:57 smithi122 ceph-mon[32255]: pgmap v13481: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:27:58.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:27:58.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:27:58.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:27:58.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:27:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:59 smithi022 ceph-mon[31360]: pgmap v13482: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:27:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:27:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2509203647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:59 smithi022 ceph-mon[40250]: pgmap v13482: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:27:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:27:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2509203647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:27:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:59 smithi122 ceph-mon[32255]: pgmap v13482: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:27:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:27:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2509203647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:00] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T01:28:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:01 smithi022 ceph-mon[31360]: pgmap v13483: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:01 smithi022 ceph-mon[40250]: pgmap v13483: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:01 smithi122 ceph-mon[32255]: pgmap v13483: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:28:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:28:03.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:03 smithi022 ceph-mon[31360]: pgmap v13484: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:03 smithi022 ceph-mon[40250]: pgmap v13484: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:03 smithi122 ceph-mon[32255]: pgmap v13484: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/615558960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/615558960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/615558960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:05 smithi022 ceph-mon[31360]: pgmap v13485: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:05 smithi022 ceph-mon[40250]: pgmap v13485: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:05 smithi122 ceph-mon[32255]: pgmap v13485: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:05.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:28:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:28:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:07 smithi022 ceph-mon[31360]: pgmap v13486: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:28:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:07 smithi022 ceph-mon[40250]: pgmap v13486: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:07.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:07 smithi122 ceph-mon[32255]: pgmap v13486: 132 pgs: 132 active+clean; 582 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:09.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:09.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:09.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:09.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:09 smithi022 ceph-mon[40250]: pgmap v13487: 132 pgs: 132 active+clean; 582 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2119456685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:09 smithi022 ceph-mon[31360]: pgmap v13487: 132 pgs: 132 active+clean; 582 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2119456685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:09 smithi122 ceph-mon[32255]: pgmap v13487: 132 pgs: 132 active+clean; 582 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2119456685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:28:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:11 smithi022 ceph-mon[31360]: pgmap v13488: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:28:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:11 smithi022 ceph-mon[40250]: pgmap v13488: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:28:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:11 smithi122 ceph-mon[32255]: pgmap v13488: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:28:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:28:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:28:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:13 smithi022 ceph-mon[31360]: pgmap v13489: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:13 smithi022 ceph-mon[40250]: pgmap v13489: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:13 smithi122 ceph-mon[32255]: pgmap v13489: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:14.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:14.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:14.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:14.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:15 smithi022 ceph-mon[31360]: pgmap v13490: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:28:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1925712193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:15 smithi022 ceph-mon[40250]: pgmap v13490: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:28:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1925712193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:28:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:28:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:15 smithi122 ceph-mon[32255]: pgmap v13490: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:28:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1925712193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:17 smithi022 ceph-mon[31360]: pgmap v13491: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:28:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:17 smithi022 ceph-mon[40250]: pgmap v13491: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:17 smithi122 ceph-mon[32255]: pgmap v13491: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:19 smithi022 ceph-mon[31360]: pgmap v13492: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:19 smithi022 ceph-mon[40250]: pgmap v13492: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:19.783 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:19 smithi122 ceph-mon[32255]: pgmap v13492: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:28:20.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:20.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:20.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:20] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T01:28:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/812617260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:20.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/812617260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/812617260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:21 smithi022 ceph-mon[31360]: pgmap v13493: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:21 smithi022 ceph-mon[40250]: pgmap v13493: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:21 smithi122 ceph-mon[32255]: pgmap v13493: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:28:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:28:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:28:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:23 smithi022 ceph-mon[31360]: pgmap v13494: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:23 smithi022 ceph-mon[40250]: pgmap v13494: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:23.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:23 smithi122 ceph-mon[32255]: pgmap v13494: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:25.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:25.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:25 smithi022 ceph-mon[31360]: pgmap v13495: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:25.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:25.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:25.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:25 smithi022 ceph-mon[40250]: pgmap v13495: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:25 smithi122 ceph-mon[32255]: pgmap v13495: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:25.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:28:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:28:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2300972292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2300972292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2300972292' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:27 smithi022 ceph-mon[31360]: pgmap v13496: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:28:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:27 smithi022 ceph-mon[40250]: pgmap v13496: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:27 smithi122 ceph-mon[32255]: pgmap v13496: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:29 smithi122 ceph-mon[32255]: pgmap v13497: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:28:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:28:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:28:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:28:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[31360]: pgmap v13497: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:28:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:28:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:28:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:28:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[40250]: pgmap v13497: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:28:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:28:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:28:29.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:28:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:30] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T01:28:30.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:30.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:30.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:30.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:31 smithi022 ceph-mon[31360]: pgmap v13498: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/106853670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:31 smithi022 ceph-mon[40250]: pgmap v13498: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/106853670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:31 smithi122 ceph-mon[32255]: pgmap v13498: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/106853670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:28:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:28:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:33 smithi022 ceph-mon[31360]: pgmap v13499: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:33 smithi022 ceph-mon[40250]: pgmap v13499: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:33 smithi122 ceph-mon[32255]: pgmap v13499: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:28:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:28:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:28:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:28:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:28:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:28:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:28:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:28:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:28:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:28:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:28:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:28:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:35 smithi022 ceph-mon[31360]: pgmap v13500: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:35 smithi022 ceph-mon[40250]: pgmap v13500: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:35.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:35.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:28:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:28:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:35 smithi122 ceph-mon[32255]: pgmap v13500: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:36.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:36.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4039610659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4039610659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4039610659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:28:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:37 smithi022 ceph-mon[40250]: pgmap v13501: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:37 smithi022 ceph-mon[31360]: pgmap v13501: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:37 smithi122 ceph-mon[32255]: pgmap v13501: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:39 smithi022 ceph-mon[31360]: pgmap v13502: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:39 smithi022 ceph-mon[40250]: pgmap v13502: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:39 smithi122 ceph-mon[32255]: pgmap v13502: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:40] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:28:41.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:41.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:41.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:41.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:41 smithi022 ceph-mon[31360]: pgmap v13503: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:41 smithi022 ceph-mon[40250]: pgmap v13503: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:41 smithi122 ceph-mon[32255]: pgmap v13503: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2705931530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:28:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:28:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2705931530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2705931530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:43 smithi122 ceph-mon[32255]: pgmap v13504: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:43 smithi022 ceph-mon[31360]: pgmap v13504: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:43 smithi022 ceph-mon[40250]: pgmap v13504: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:45 smithi122 ceph-mon[32255]: pgmap v13505: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:45.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:28:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:28:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:45 smithi022 ceph-mon[31360]: pgmap v13505: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:45 smithi022 ceph-mon[40250]: pgmap v13505: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:46.635 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:47.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:47.005 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:28:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:47 smithi022 ceph-mon[31360]: pgmap v13506: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4282676365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:47 smithi022 ceph-mon[40250]: pgmap v13506: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4282676365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:47 smithi122 ceph-mon[32255]: pgmap v13506: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4282676365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:49 smithi122 ceph-mon[32255]: pgmap v13507: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:49 smithi022 ceph-mon[31360]: pgmap v13507: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:49 smithi022 ceph-mon[40250]: pgmap v13507: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:50] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:28:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:51 smithi122 ceph-mon[32255]: pgmap v13508: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:52.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:51 smithi022 ceph-mon[31360]: pgmap v13508: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:52.007 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:51 smithi022 ceph-mon[40250]: pgmap v13508: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:52.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:52.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:52.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/12740069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:28:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:28:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/12740069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/12740069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:53 smithi122 ceph-mon[32255]: pgmap v13509: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:53 smithi022 ceph-mon[31360]: pgmap v13509: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:53 smithi022 ceph-mon[40250]: pgmap v13509: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:55 smithi122 ceph-mon[32255]: pgmap v13510: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:28:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:28:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:55 smithi022 ceph-mon[31360]: pgmap v13510: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:55 smithi022 ceph-mon[40250]: pgmap v13510: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:28:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:28:57.377 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:28:57.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:28:57.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:28:57.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:28:57.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:28:57.747 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:57 smithi022 ceph-mon[40250]: pgmap v13511: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:57.748 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:57 smithi022 ceph-mon[31360]: pgmap v13511: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:57 smithi122 ceph-mon[32255]: pgmap v13511: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:28:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/13893080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/13893080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/13893080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:28:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:28:59 smithi122 ceph-mon[32255]: pgmap v13512: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:28:59 smithi022 ceph-mon[31360]: pgmap v13512: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:28:59 smithi022 ceph-mon[40250]: pgmap v13512: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:00] "GET /metrics HTTP/1.1" 200 202431 "" "Prometheus/2.18.1" 2021-12-16T01:29:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:01 smithi122 ceph-mon[32255]: pgmap v13513: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:01 smithi022 ceph-mon[31360]: pgmap v13513: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:01 smithi022 ceph-mon[40250]: pgmap v13513: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:29:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:29:02.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:03.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:03.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:03.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:03 smithi122 ceph-mon[32255]: pgmap v13514: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1592667008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:03 smithi022 ceph-mon[31360]: pgmap v13514: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1592667008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:03 smithi022 ceph-mon[40250]: pgmap v13514: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1592667008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:05 smithi122 ceph-mon[32255]: pgmap v13515: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:05.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:29:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:29:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:05 smithi022 ceph-mon[31360]: pgmap v13515: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:05 smithi022 ceph-mon[40250]: pgmap v13515: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:07 smithi022 ceph-mon[31360]: pgmap v13516: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:29:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:07 smithi022 ceph-mon[40250]: pgmap v13516: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:07 smithi122 ceph-mon[32255]: pgmap v13516: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:08.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:08.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:08.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:08.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1118244770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1118244770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:08.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1118244770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:09 smithi122 ceph-mon[32255]: pgmap v13517: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:09 smithi022 ceph-mon[31360]: pgmap v13517: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:09 smithi022 ceph-mon[40250]: pgmap v13517: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:10] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:29:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:11 smithi122 ceph-mon[32255]: pgmap v13518: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:11 smithi022 ceph-mon[31360]: pgmap v13518: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:11 smithi022 ceph-mon[40250]: pgmap v13518: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:29:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:29:13.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:13 smithi022 ceph-mon[31360]: pgmap v13519: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:13 smithi022 ceph-mon[40250]: pgmap v13519: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:13.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:13.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:13.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:13 smithi122 ceph-mon[32255]: pgmap v13519: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/541073146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/541073146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/541073146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:15.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:29:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:29:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:15 smithi122 ceph-mon[32255]: pgmap v13520: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:15 smithi022 ceph-mon[31360]: pgmap v13520: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:15 smithi022 ceph-mon[40250]: pgmap v13520: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:17 smithi022 ceph-mon[31360]: pgmap v13521: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:29:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:17 smithi022 ceph-mon[40250]: pgmap v13521: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:17 smithi122 ceph-mon[32255]: pgmap v13521: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:18.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:19.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:19.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:19.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:19 smithi122 ceph-mon[32255]: pgmap v13522: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1013024072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:19 smithi022 ceph-mon[31360]: pgmap v13522: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1013024072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:19 smithi022 ceph-mon[40250]: pgmap v13522: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1013024072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:20] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:29:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:21 smithi122 ceph-mon[32255]: pgmap v13523: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:21 smithi022 ceph-mon[31360]: pgmap v13523: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:21 smithi022 ceph-mon[40250]: pgmap v13523: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:29:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:29:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:23 smithi122 ceph-mon[32255]: pgmap v13524: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:23 smithi022 ceph-mon[31360]: pgmap v13524: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:23 smithi022 ceph-mon[40250]: pgmap v13524: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:24.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:24.596 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:24.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:24.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3508793084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3508793084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3508793084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:25.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:29:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:29:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:25 smithi122 ceph-mon[32255]: pgmap v13525: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:25 smithi022 ceph-mon[31360]: pgmap v13525: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:25 smithi022 ceph-mon[40250]: pgmap v13525: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:27 smithi022 ceph-mon[31360]: pgmap v13526: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:29:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:27 smithi022 ceph-mon[40250]: pgmap v13526: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:27 smithi122 ceph-mon[32255]: pgmap v13526: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:29.598 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:29 smithi122 ceph-mon[32255]: pgmap v13527: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:29:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:29:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:29:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[31360]: pgmap v13527: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:29:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:29:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:29:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[40250]: pgmap v13527: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:29:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:29:29.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:29:29.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:29.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:29.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:30] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:29:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:29:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3686008225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:29:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3686008225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:29:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3686008225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:31 smithi122 ceph-mon[32255]: pgmap v13528: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:31 smithi022 ceph-mon[31360]: pgmap v13528: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:31 smithi022 ceph-mon[40250]: pgmap v13528: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:29:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:29:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:33 smithi122 ceph-mon[32255]: pgmap v13529: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:33 smithi022 ceph-mon[31360]: pgmap v13529: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:33 smithi022 ceph-mon[40250]: pgmap v13529: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:34.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:29:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:29:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:29:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:29:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:29:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:29:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:29:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:29:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:29:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:29:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:29:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:29:35.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:35.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:35.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:36.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:29:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:29:36.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:35 smithi122 ceph-mon[32255]: pgmap v13530: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/762888626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:35 smithi022 ceph-mon[31360]: pgmap v13530: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/762888626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:35 smithi022 ceph-mon[40250]: pgmap v13530: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/762888626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:29:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:37 smithi122 ceph-mon[32255]: pgmap v13531: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:37 smithi022 ceph-mon[31360]: pgmap v13531: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:37 smithi022 ceph-mon[40250]: pgmap v13531: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:39 smithi122 ceph-mon[32255]: pgmap v13532: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:39 smithi022 ceph-mon[31360]: pgmap v13532: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:39 smithi022 ceph-mon[40250]: pgmap v13532: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:40.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:40] "GET /metrics HTTP/1.1" 200 202422 "" "Prometheus/2.18.1" 2021-12-16T01:29:40.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:40.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:40.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3774185210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3774185210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3774185210' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:41 smithi122 ceph-mon[32255]: pgmap v13533: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:41 smithi022 ceph-mon[31360]: pgmap v13533: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:41 smithi022 ceph-mon[40250]: pgmap v13533: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:29:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:29:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:43 smithi122 ceph-mon[32255]: pgmap v13534: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:43 smithi022 ceph-mon[31360]: pgmap v13534: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:43 smithi022 ceph-mon[40250]: pgmap v13534: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:45.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:45 smithi022 ceph-mon[31360]: pgmap v13535: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:45 smithi022 ceph-mon[40250]: pgmap v13535: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:45 smithi122 ceph-mon[32255]: pgmap v13535: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:29:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:29:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1794912558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1794912558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1794912558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:29:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:47 smithi122 ceph-mon[32255]: pgmap v13536: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:47 smithi022 ceph-mon[31360]: pgmap v13536: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:47 smithi022 ceph-mon[40250]: pgmap v13536: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:49 smithi122 ceph-mon[32255]: pgmap v13537: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:49 smithi022 ceph-mon[31360]: pgmap v13537: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:49 smithi022 ceph-mon[40250]: pgmap v13537: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:50] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:29:51.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:51.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:51.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:51.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:51 smithi122 ceph-mon[32255]: pgmap v13538: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1174541685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:51 smithi022 ceph-mon[31360]: pgmap v13538: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1174541685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:51 smithi022 ceph-mon[40250]: pgmap v13538: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1174541685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:29:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:29:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:53 smithi122 ceph-mon[32255]: pgmap v13539: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:53 smithi022 ceph-mon[31360]: pgmap v13539: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:53 smithi022 ceph-mon[40250]: pgmap v13539: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:29:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:29:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:55 smithi122 ceph-mon[32255]: pgmap v13540: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:55 smithi022 ceph-mon[31360]: pgmap v13540: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:55 smithi022 ceph-mon[40250]: pgmap v13540: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:29:56.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:29:56.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:29:56.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:29:56.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:29:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2333721231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2333721231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:29:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2333721231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:29:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:29:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:57 smithi122 ceph-mon[32255]: pgmap v13541: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:57 smithi022 ceph-mon[31360]: pgmap v13541: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:29:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:57 smithi022 ceph-mon[40250]: pgmap v13541: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:29:59 smithi122 ceph-mon[32255]: pgmap v13542: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:29:59 smithi022 ceph-mon[31360]: pgmap v13542: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:29:59 smithi022 ceph-mon[40250]: pgmap v13542: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:00] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:30:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T01:30:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T01:30:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T01:30:01.820 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:01 smithi122 ceph-mon[32255]: pgmap v13543: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:02.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:02.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:02.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:02.189 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:01 smithi022 ceph-mon[31360]: pgmap v13543: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:02.190 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:01 smithi022 ceph-mon[40250]: pgmap v13543: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:30:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:30:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3550639824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3550639824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3550639824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:03 smithi122 ceph-mon[32255]: pgmap v13544: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:03 smithi022 ceph-mon[31360]: pgmap v13544: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:03 smithi022 ceph-mon[40250]: pgmap v13544: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:05 smithi122 ceph-mon[32255]: pgmap v13545: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:06.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:30:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:30:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:05 smithi022 ceph-mon[40250]: pgmap v13545: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:06.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:05 smithi022 ceph-mon[31360]: pgmap v13545: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:07.191 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:07.191 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:07.192 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:30:07.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:07.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:07.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:07 smithi122 ceph-mon[32255]: pgmap v13546: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1243673749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:07 smithi022 ceph-mon[31360]: pgmap v13546: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1243673749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:07 smithi022 ceph-mon[40250]: pgmap v13546: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1243673749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:09 smithi122 ceph-mon[32255]: pgmap v13547: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:09 smithi022 ceph-mon[31360]: pgmap v13547: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:09 smithi022 ceph-mon[40250]: pgmap v13547: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:10] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:30:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:11 smithi122 ceph-mon[32255]: pgmap v13548: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:11 smithi022 ceph-mon[31360]: pgmap v13548: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:11 smithi022 ceph-mon[40250]: pgmap v13548: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:12.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:30:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:30:12.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:12.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:12.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:13 smithi122 ceph-mon[32255]: pgmap v13549: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/972034143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:13 smithi022 ceph-mon[40250]: pgmap v13549: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/972034143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:13 smithi022 ceph-mon[31360]: pgmap v13549: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/972034143' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:15 smithi122 ceph-mon[32255]: pgmap v13550: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:30:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:30:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:15 smithi022 ceph-mon[31360]: pgmap v13550: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:15 smithi022 ceph-mon[40250]: pgmap v13550: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:30:17.929 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:17 smithi122 ceph-mon[32255]: pgmap v13551: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:17 smithi022 ceph-mon[40250]: pgmap v13551: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:17 smithi022 ceph-mon[31360]: pgmap v13551: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:18.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:18.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:18.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4199051598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4199051598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4199051598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:20.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:19 smithi122 ceph-mon[32255]: pgmap v13552: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:20.186 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:20] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:30:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:19 smithi022 ceph-mon[31360]: pgmap v13552: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:19 smithi022 ceph-mon[40250]: pgmap v13552: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:21 smithi122 ceph-mon[32255]: pgmap v13553: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:21 smithi022 ceph-mon[31360]: pgmap v13553: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:21 smithi022 ceph-mon[40250]: pgmap v13553: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:30:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:30:23.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:23.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:23.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:23.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:23 smithi122 ceph-mon[32255]: pgmap v13554: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2567021710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:23 smithi022 ceph-mon[31360]: pgmap v13554: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2567021710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:23 smithi022 ceph-mon[40250]: pgmap v13554: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2567021710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:25 smithi122 ceph-mon[32255]: pgmap v13555: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:26.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:30:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:30:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:25 smithi022 ceph-mon[31360]: pgmap v13555: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:25 smithi022 ceph-mon[40250]: pgmap v13555: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:30:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:27 smithi022 ceph-mon[31360]: pgmap v13556: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:27 smithi022 ceph-mon[40250]: pgmap v13556: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:27 smithi122 ceph-mon[32255]: pgmap v13556: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:28.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:29.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:29.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:29.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:29 smithi122 ceph-mon[32255]: pgmap v13557: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2552306791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:30:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:30:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:30:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[31360]: pgmap v13557: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2552306791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:30:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:30:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:30:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[40250]: pgmap v13557: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2552306791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:30:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:30:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:30:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:30] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:30:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:31 smithi122 ceph-mon[32255]: pgmap v13558: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:30:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:31 smithi022 ceph-mon[31360]: pgmap v13558: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:30:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:31 smithi022 ceph-mon[40250]: pgmap v13558: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:30:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:30:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:30:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:33 smithi122 ceph-mon[32255]: pgmap v13559: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:33 smithi022 ceph-mon[31360]: pgmap v13559: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:33 smithi022 ceph-mon[40250]: pgmap v13559: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:34.044 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:34.335 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:30:34.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:30:34.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:30:34.336 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:30:34.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:34.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:30:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:30:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:30:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:30:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:30:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:35 smithi122 ceph-mon[32255]: pgmap v13560: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1255536062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:35 smithi022 ceph-mon[31360]: pgmap v13560: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1255536062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:35 smithi022 ceph-mon[40250]: pgmap v13560: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1255536062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:36.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:30:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:30:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:37 smithi122 ceph-mon[32255]: pgmap v13561: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:37 smithi022 ceph-mon[31360]: pgmap v13561: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:30:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:37 smithi022 ceph-mon[40250]: pgmap v13561: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:39.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:39 smithi122 ceph-mon[32255]: pgmap v13562: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:39 smithi022 ceph-mon[31360]: pgmap v13562: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:39 smithi022 ceph-mon[40250]: pgmap v13562: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:39.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:39.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:39.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:40] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:30:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1593204199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1593204199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1593204199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:41 smithi122 ceph-mon[32255]: pgmap v13563: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:41.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:41 smithi022 ceph-mon[31360]: pgmap v13563: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:41 smithi022 ceph-mon[40250]: pgmap v13563: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:30:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:30:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:43 smithi122 ceph-mon[32255]: pgmap v13564: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:43.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:43 smithi022 ceph-mon[31360]: pgmap v13564: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:43 smithi022 ceph-mon[40250]: pgmap v13564: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:44.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:45 smithi022 ceph-mon[31360]: pgmap v13565: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2907305782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:45 smithi022 ceph-mon[40250]: pgmap v13565: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2907305782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:45 smithi122 ceph-mon[32255]: pgmap v13565: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2907305782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:30:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:30:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:47 smithi122 ceph-mon[32255]: pgmap v13566: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:47 smithi022 ceph-mon[31360]: pgmap v13566: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:30:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:47 smithi022 ceph-mon[40250]: pgmap v13566: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:49 smithi122 ceph-mon[32255]: pgmap v13567: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:49 smithi022 ceph-mon[31360]: pgmap v13567: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:49 smithi022 ceph-mon[40250]: pgmap v13567: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:50.153 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:50] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:30:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:51 smithi122 ceph-mon[32255]: pgmap v13568: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1961633995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:51.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:51 smithi022 ceph-mon[31360]: pgmap v13568: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:51.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1961633995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:51 smithi022 ceph-mon[40250]: pgmap v13568: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:51.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1961633995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:30:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:30:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:53 smithi122 ceph-mon[32255]: pgmap v13569: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:53 smithi022 ceph-mon[31360]: pgmap v13569: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:53 smithi022 ceph-mon[40250]: pgmap v13569: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:55.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:30:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:55 smithi122 ceph-mon[32255]: pgmap v13570: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:55 smithi022 ceph-mon[31360]: pgmap v13570: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:55.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:55 smithi022 ceph-mon[40250]: pgmap v13570: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:55.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:30:55.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:30:55.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:30:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:30:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:30:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/117939634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/117939634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:30:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/117939634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:30:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:57 smithi122 ceph-mon[32255]: pgmap v13571: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:57 smithi022 ceph-mon[31360]: pgmap v13571: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:30:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:57 smithi022 ceph-mon[40250]: pgmap v13571: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:30:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:30:59 smithi122 ceph-mon[32255]: pgmap v13572: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:30:59 smithi022 ceph-mon[31360]: pgmap v13572: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:30:59.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:30:59 smithi022 ceph-mon[40250]: pgmap v13572: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:00] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:31:00.894 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:01.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:01.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:01.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:01 smithi022 ceph-mon[31360]: pgmap v13573: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3758635331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:01 smithi022 ceph-mon[40250]: pgmap v13573: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3758635331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:01 smithi122 ceph-mon[32255]: pgmap v13573: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3758635331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:31:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:31:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:03 smithi122 ceph-mon[32255]: pgmap v13574: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:03 smithi022 ceph-mon[31360]: pgmap v13574: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:03 smithi022 ceph-mon[40250]: pgmap v13574: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:05 smithi122 ceph-mon[32255]: pgmap v13575: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:05 smithi022 ceph-mon[31360]: pgmap v13575: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:05 smithi022 ceph-mon[40250]: pgmap v13575: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:06.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:31:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:31:06.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:06.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:06.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:06.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:07 smithi122 ceph-mon[32255]: pgmap v13576: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3927726411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:31:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:07 smithi022 ceph-mon[31360]: pgmap v13576: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3927726411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:07 smithi022 ceph-mon[40250]: pgmap v13576: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3927726411' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:09 smithi122 ceph-mon[32255]: pgmap v13577: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:09 smithi022 ceph-mon[31360]: pgmap v13577: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:09 smithi022 ceph-mon[40250]: pgmap v13577: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:10] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:31:11.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:11 smithi122 ceph-mon[32255]: pgmap v13578: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:11 smithi022 ceph-mon[31360]: pgmap v13578: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:11 smithi022 ceph-mon[40250]: pgmap v13578: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:12.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:12.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:12.008 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4140042423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:31:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:31:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4140042423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:12.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4140042423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:13 smithi122 ceph-mon[32255]: pgmap v13579: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:13 smithi022 ceph-mon[31360]: pgmap v13579: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:13 smithi022 ceph-mon[40250]: pgmap v13579: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:15 smithi122 ceph-mon[32255]: pgmap v13580: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:15 smithi022 ceph-mon[31360]: pgmap v13580: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:15 smithi022 ceph-mon[40250]: pgmap v13580: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:16.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:31:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:31:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:17.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:17.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:17.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:17.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:17 smithi122 ceph-mon[32255]: pgmap v13581: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3925058114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:31:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:17 smithi022 ceph-mon[31360]: pgmap v13581: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3925058114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:17 smithi022 ceph-mon[40250]: pgmap v13581: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3925058114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:19 smithi122 ceph-mon[32255]: pgmap v13582: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:19 smithi022 ceph-mon[31360]: pgmap v13582: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:19 smithi022 ceph-mon[40250]: pgmap v13582: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:31:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:21 smithi122 ceph-mon[32255]: pgmap v13583: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:21 smithi022 ceph-mon[31360]: pgmap v13583: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:21 smithi022 ceph-mon[40250]: pgmap v13583: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:22.379 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:31:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:31:22.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:22.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:22.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:23 smithi122 ceph-mon[32255]: pgmap v13584: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3506131409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:23 smithi022 ceph-mon[31360]: pgmap v13584: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3506131409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:23 smithi022 ceph-mon[40250]: pgmap v13584: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3506131409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:25 smithi122 ceph-mon[32255]: pgmap v13585: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:25 smithi022 ceph-mon[31360]: pgmap v13585: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:25 smithi022 ceph-mon[40250]: pgmap v13585: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:26.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:31:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:31:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:27 smithi122 ceph-mon[32255]: pgmap v13586: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:27.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:27.750 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:31:27.751 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:27 smithi022 ceph-mon[31360]: pgmap v13586: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:27.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:27 smithi022 ceph-mon[40250]: pgmap v13586: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:28.434 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2446674821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:28.435 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2446674821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2446674821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:29 smithi022 ceph-mon[31360]: pgmap v13587: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:29 smithi022 ceph-mon[40250]: pgmap v13587: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:29 smithi122 ceph-mon[32255]: pgmap v13587: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:31:30.722 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:31:30.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:31:30.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:31:30.740 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:31:30.740 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:31:30.741 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:31:30.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:31:30.741 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:31:30.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:31:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:31 smithi022 ceph-mon[31360]: pgmap v13588: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:31:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:31 smithi022 ceph-mon[40250]: pgmap v13588: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:31:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:31 smithi122 ceph-mon[32255]: pgmap v13588: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:31:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:31:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:31:33.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:33.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:33.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:33.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:33 smithi022 ceph-mon[31360]: pgmap v13589: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2683957164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:33 smithi022 ceph-mon[40250]: pgmap v13589: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2683957164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:33 smithi122 ceph-mon[32255]: pgmap v13589: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2683957164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:31:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:31:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:31:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:31:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:31:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:31:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:31:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:31:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:31:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:31:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:31:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:31:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:35 smithi022 ceph-mon[40250]: pgmap v13590: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:35 smithi022 ceph-mon[31360]: pgmap v13590: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:36.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:31:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:31:36.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:35 smithi122 ceph-mon[32255]: pgmap v13590: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:37.762 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:31:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:37 smithi022 ceph-mon[31360]: pgmap v13591: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:37 smithi022 ceph-mon[40250]: pgmap v13591: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:37 smithi122 ceph-mon[32255]: pgmap v13591: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:38.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:38.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:38.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:39 smithi022 ceph-mon[31360]: pgmap v13592: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4288489937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:39 smithi022 ceph-mon[40250]: pgmap v13592: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4288489937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:39 smithi122 ceph-mon[32255]: pgmap v13592: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4288489937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:40] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T01:31:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:41 smithi022 ceph-mon[31360]: pgmap v13593: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:41 smithi022 ceph-mon[40250]: pgmap v13593: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:41 smithi122 ceph-mon[32255]: pgmap v13593: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:31:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:31:43.863 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:43 smithi022 ceph-mon[31360]: pgmap v13594: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:43 smithi022 ceph-mon[40250]: pgmap v13594: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:43 smithi122 ceph-mon[32255]: pgmap v13594: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:44.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:44.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:44.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2088197203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2088197203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2088197203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:45 smithi022 ceph-mon[31360]: pgmap v13595: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:45 smithi022 ceph-mon[40250]: pgmap v13595: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:31:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:31:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:45 smithi122 ceph-mon[32255]: pgmap v13595: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:31:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:47 smithi122 ceph-mon[32255]: pgmap v13596: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:47 smithi022 ceph-mon[31360]: pgmap v13596: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:47 smithi022 ceph-mon[40250]: pgmap v13596: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:49.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:49.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:49.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:49.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:49 smithi022 ceph-mon[31360]: pgmap v13597: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3455988926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:49 smithi022 ceph-mon[40250]: pgmap v13597: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3455988926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:49 smithi122 ceph-mon[32255]: pgmap v13597: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3455988926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:50] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:31:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:51 smithi122 ceph-mon[32255]: pgmap v13598: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:51 smithi022 ceph-mon[31360]: pgmap v13598: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:51 smithi022 ceph-mon[40250]: pgmap v13598: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:31:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:31:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:53 smithi122 ceph-mon[32255]: pgmap v13599: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:53 smithi022 ceph-mon[31360]: pgmap v13599: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:53 smithi022 ceph-mon[40250]: pgmap v13599: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:54.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:31:54.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:31:54.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:31:54.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:31:56.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:31:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:31:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:55 smithi122 ceph-mon[32255]: pgmap v13600: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3381374423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:55 smithi022 ceph-mon[31360]: pgmap v13600: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3381374423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:55 smithi022 ceph-mon[40250]: pgmap v13600: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:31:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3381374423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:31:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:31:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:31:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:57 smithi122 ceph-mon[32255]: pgmap v13601: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:57 smithi022 ceph-mon[31360]: pgmap v13601: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:57 smithi022 ceph-mon[40250]: pgmap v13601: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:31:59.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:31:59 smithi122 ceph-mon[32255]: pgmap v13602: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:31:59 smithi022 ceph-mon[31360]: pgmap v13602: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:31:59 smithi022 ceph-mon[40250]: pgmap v13602: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:00.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:00.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:00.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:00] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:32:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1816701792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1816701792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1816701792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:01 smithi122 ceph-mon[32255]: pgmap v13603: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:01 smithi022 ceph-mon[31360]: pgmap v13603: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:01 smithi022 ceph-mon[40250]: pgmap v13603: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:32:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:32:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:03 smithi122 ceph-mon[32255]: pgmap v13604: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:03 smithi022 ceph-mon[31360]: pgmap v13604: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:03 smithi022 ceph-mon[40250]: pgmap v13604: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:05.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:05.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:05.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:05 smithi022 ceph-mon[31360]: pgmap v13605: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3837968341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:05 smithi022 ceph-mon[40250]: pgmap v13605: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3837968341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:32:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:32:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:05 smithi122 ceph-mon[32255]: pgmap v13605: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3837968341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:32:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:07 smithi122 ceph-mon[32255]: pgmap v13606: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:07 smithi022 ceph-mon[31360]: pgmap v13606: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:07 smithi022 ceph-mon[40250]: pgmap v13606: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:09 smithi122 ceph-mon[32255]: pgmap v13607: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:09 smithi022 ceph-mon[31360]: pgmap v13607: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:09 smithi022 ceph-mon[40250]: pgmap v13607: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:10] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:32:10.712 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:11.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:11.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:11 smithi122 ceph-mon[32255]: pgmap v13608: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2759049071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:11 smithi022 ceph-mon[31360]: pgmap v13608: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2759049071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:11 smithi022 ceph-mon[40250]: pgmap v13608: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2759049071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:32:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:32:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:13 smithi122 ceph-mon[32255]: pgmap v13609: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:13 smithi022 ceph-mon[31360]: pgmap v13609: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:13 smithi022 ceph-mon[40250]: pgmap v13609: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:16.083 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:15 smithi122 ceph-mon[32255]: pgmap v13610: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:32:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:32:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:15 smithi022 ceph-mon[31360]: pgmap v13610: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:15 smithi022 ceph-mon[40250]: pgmap v13610: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:16.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2286705155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2286705155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2286705155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:32:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:17 smithi122 ceph-mon[32255]: pgmap v13611: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:17 smithi022 ceph-mon[31360]: pgmap v13611: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:17 smithi022 ceph-mon[40250]: pgmap v13611: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:19 smithi122 ceph-mon[32255]: pgmap v13612: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:19 smithi022 ceph-mon[31360]: pgmap v13612: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:19 smithi022 ceph-mon[40250]: pgmap v13612: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:32:21.454 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:21.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:21 smithi122 ceph-mon[32255]: pgmap v13613: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1428764287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:21 smithi022 ceph-mon[31360]: pgmap v13613: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1428764287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:21 smithi022 ceph-mon[40250]: pgmap v13613: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1428764287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:32:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:32:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:23 smithi122 ceph-mon[32255]: pgmap v13614: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:23 smithi022 ceph-mon[31360]: pgmap v13614: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:23 smithi022 ceph-mon[40250]: pgmap v13614: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:26.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:32:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:32:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:25 smithi122 ceph-mon[32255]: pgmap v13615: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:25 smithi022 ceph-mon[31360]: pgmap v13615: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:25 smithi022 ceph-mon[40250]: pgmap v13615: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:27.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:27.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:27.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:27.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:27.197 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:32:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:27 smithi122 ceph-mon[32255]: pgmap v13616: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2556271576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:27 smithi022 ceph-mon[40250]: pgmap v13616: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2556271576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:27 smithi022 ceph-mon[31360]: pgmap v13616: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2556271576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:29 smithi122 ceph-mon[32255]: pgmap v13617: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:29 smithi022 ceph-mon[31360]: pgmap v13617: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:29 smithi022 ceph-mon[40250]: pgmap v13617: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:32:31.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:32:31.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:32:31.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:32:31.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:32:31.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:32:31.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:32:31.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:32:31.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:32:31.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:32:32.199 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:32:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:32:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:32 smithi122 ceph-mon[32255]: pgmap v13618: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[31360]: pgmap v13618: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[40250]: pgmap v13618: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:32:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3703433799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3703433799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3703433799' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:34 smithi122 ceph-mon[32255]: pgmap v13619: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:32:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:32:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:32:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:32:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[31360]: pgmap v13619: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:32:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:32:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:32:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:32:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[40250]: pgmap v13619: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:32:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:32:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:32:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:32:36.154 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:32:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:32:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:36 smithi122 ceph-mon[32255]: pgmap v13620: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:36 smithi022 ceph-mon[31360]: pgmap v13620: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:36 smithi022 ceph-mon[40250]: pgmap v13620: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:37.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:37.531 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:32:37.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:37.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:37.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:37.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:37.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:38 smithi022 ceph-mon[31360]: pgmap v13621: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3492029794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:38 smithi022 ceph-mon[40250]: pgmap v13621: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3492029794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:38 smithi122 ceph-mon[32255]: pgmap v13621: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3492029794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:40 smithi122 ceph-mon[32255]: pgmap v13622: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:40] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:32:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:40 smithi022 ceph-mon[31360]: pgmap v13622: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:40 smithi022 ceph-mon[40250]: pgmap v13622: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:42 smithi122 ceph-mon[32255]: pgmap v13623: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:32:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:32:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:42 smithi022 ceph-mon[31360]: pgmap v13623: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:42 smithi022 ceph-mon[40250]: pgmap v13623: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:42.936 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:44 smithi122 ceph-mon[32255]: pgmap v13624: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4262526344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:44 smithi022 ceph-mon[31360]: pgmap v13624: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4262526344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:44 smithi022 ceph-mon[40250]: pgmap v13624: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4262526344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:46.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:32:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:32:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:46 smithi122 ceph-mon[32255]: pgmap v13625: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:46 smithi022 ceph-mon[31360]: pgmap v13625: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:46 smithi022 ceph-mon[40250]: pgmap v13625: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:32:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:48.310 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:48 smithi122 ceph-mon[32255]: pgmap v13626: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:48 smithi022 ceph-mon[31360]: pgmap v13626: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:48 smithi022 ceph-mon[40250]: pgmap v13626: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:48.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:48.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:48.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:49 smithi122 ceph-mon[32255]: pgmap v13627: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1782018011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:49 smithi022 ceph-mon[31360]: pgmap v13627: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1782018011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:49 smithi022 ceph-mon[40250]: pgmap v13627: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1782018011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:50] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:32:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:51 smithi122 ceph-mon[32255]: pgmap v13628: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:51 smithi022 ceph-mon[31360]: pgmap v13628: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:51 smithi022 ceph-mon[40250]: pgmap v13628: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:32:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:32:53.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:53.684 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:53 smithi022 ceph-mon[31360]: pgmap v13629: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:53.685 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:53 smithi022 ceph-mon[40250]: pgmap v13629: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:53 smithi122 ceph-mon[32255]: pgmap v13629: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:54.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:54.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:54.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1001359775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1001359775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1001359775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:32:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:55 smithi122 ceph-mon[32255]: pgmap v13630: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:55 smithi022 ceph-mon[31360]: pgmap v13630: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:55 smithi022 ceph-mon[40250]: pgmap v13630: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:32:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:32:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:32:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:57 smithi122 ceph-mon[32255]: pgmap v13631: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:57 smithi022 ceph-mon[31360]: pgmap v13631: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:32:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:57 smithi022 ceph-mon[40250]: pgmap v13631: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:32:59.052 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:32:59.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:32:59.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:32:59.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:32:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:32:59 smithi122 ceph-mon[32255]: pgmap v13632: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:32:59 smithi022 ceph-mon[31360]: pgmap v13632: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:32:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:32:59 smithi022 ceph-mon[40250]: pgmap v13632: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1530333405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:00] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:33:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1530333405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1530333405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:01 smithi122 ceph-mon[32255]: pgmap v13633: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:01 smithi022 ceph-mon[31360]: pgmap v13633: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:01 smithi022 ceph-mon[40250]: pgmap v13633: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:33:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:33:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:03 smithi122 ceph-mon[32255]: pgmap v13634: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:03 smithi022 ceph-mon[31360]: pgmap v13634: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:03 smithi022 ceph-mon[40250]: pgmap v13634: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:04.425 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:04.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:04.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:04.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:05 smithi122 ceph-mon[32255]: pgmap v13635: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1574410328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:05 smithi022 ceph-mon[31360]: pgmap v13635: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1574410328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:05 smithi022 ceph-mon[40250]: pgmap v13635: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1574410328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:33:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:33:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:07 smithi122 ceph-mon[32255]: pgmap v13636: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:07 smithi022 ceph-mon[31360]: pgmap v13636: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:33:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:07 smithi022 ceph-mon[40250]: pgmap v13636: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:09 smithi122 ceph-mon[32255]: pgmap v13637: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:09 smithi022 ceph-mon[31360]: pgmap v13637: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:09 smithi022 ceph-mon[40250]: pgmap v13637: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:09.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:10.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:10] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:33:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3616009747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3616009747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3616009747' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:11 smithi122 ceph-mon[32255]: pgmap v13638: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:11 smithi022 ceph-mon[31360]: pgmap v13638: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:11 smithi022 ceph-mon[40250]: pgmap v13638: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:33:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:33:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:13 smithi122 ceph-mon[32255]: pgmap v13639: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:13 smithi022 ceph-mon[31360]: pgmap v13639: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:13 smithi022 ceph-mon[40250]: pgmap v13639: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:15.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:15.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:15.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:15.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:15 smithi122 ceph-mon[32255]: pgmap v13640: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:15 smithi022 ceph-mon[31360]: pgmap v13640: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:15 smithi022 ceph-mon[40250]: pgmap v13640: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:33:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:33:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/874509613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/874509613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/874509613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:17 smithi122 ceph-mon[32255]: pgmap v13641: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:17 smithi022 ceph-mon[31360]: pgmap v13641: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:33:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:17 smithi022 ceph-mon[40250]: pgmap v13641: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:19 smithi122 ceph-mon[32255]: pgmap v13642: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:19 smithi022 ceph-mon[31360]: pgmap v13642: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:19 smithi022 ceph-mon[40250]: pgmap v13642: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:33:20.541 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:20.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:20.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:20.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:21 smithi122 ceph-mon[32255]: pgmap v13643: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1257107603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:21 smithi022 ceph-mon[31360]: pgmap v13643: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1257107603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:21 smithi022 ceph-mon[40250]: pgmap v13643: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1257107603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:33:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:33:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:23 smithi022 ceph-mon[31360]: pgmap v13644: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:23 smithi022 ceph-mon[40250]: pgmap v13644: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:23 smithi122 ceph-mon[32255]: pgmap v13644: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:25 smithi022 ceph-mon[31360]: pgmap v13645: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:25 smithi022 ceph-mon[40250]: pgmap v13645: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:25.886 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:25 smithi122 ceph-mon[32255]: pgmap v13645: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:25.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:26.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:33:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:33:26.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:26.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:26.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1159335778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1159335778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1159335778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:27 smithi022 ceph-mon[31360]: pgmap v13646: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:33:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:27 smithi022 ceph-mon[40250]: pgmap v13646: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:27 smithi122 ceph-mon[32255]: pgmap v13646: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:29 smithi022 ceph-mon[31360]: pgmap v13647: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:29 smithi022 ceph-mon[40250]: pgmap v13647: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:29 smithi122 ceph-mon[32255]: pgmap v13647: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:33:31.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:31.640 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:31 smithi022 ceph-mon[31360]: pgmap v13648: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:31.640 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:31 smithi022 ceph-mon[40250]: pgmap v13648: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:31.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:31.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:31.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:31 smithi122 ceph-mon[32255]: pgmap v13648: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:33:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:33:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:33:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:33:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:33:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/766923397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:33:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:33:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:33:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:33:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/766923397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:33:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:33:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:33:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:33:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/766923397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:33:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:34 smithi122 ceph-mon[32255]: pgmap v13649: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:33:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:33:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[31360]: pgmap v13649: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:33:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:33:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:33:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:33:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[40250]: pgmap v13649: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:33:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:33:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:33:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:33:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:36 smithi122 ceph-mon[32255]: pgmap v13650: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:36.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:33:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:33:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:36 smithi022 ceph-mon[31360]: pgmap v13650: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:36 smithi022 ceph-mon[40250]: pgmap v13650: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:36.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:37.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:37.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:37.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/859018862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/859018862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/859018862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:33:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:38 smithi122 ceph-mon[32255]: pgmap v13651: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:38 smithi022 ceph-mon[31360]: pgmap v13651: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:38 smithi022 ceph-mon[40250]: pgmap v13651: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:40 smithi122 ceph-mon[32255]: pgmap v13652: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:40] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T01:33:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:40 smithi022 ceph-mon[31360]: pgmap v13652: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:40 smithi022 ceph-mon[40250]: pgmap v13652: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:42.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:42 smithi022 ceph-mon[40250]: pgmap v13653: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:42 smithi022 ceph-mon[31360]: pgmap v13653: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:42.385 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:42 smithi122 ceph-mon[32255]: pgmap v13653: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:33:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:33:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/984052946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/984052946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/984052946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:44 smithi122 ceph-mon[32255]: pgmap v13654: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:44 smithi022 ceph-mon[31360]: pgmap v13654: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:44 smithi022 ceph-mon[40250]: pgmap v13654: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:46 smithi122 ceph-mon[32255]: pgmap v13655: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:33:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:33:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:46 smithi022 ceph-mon[31360]: pgmap v13655: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:46 smithi022 ceph-mon[40250]: pgmap v13655: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:47.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:47.388 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:47.389 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:47.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:47.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:47.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:47.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:33:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:48 smithi122 ceph-mon[32255]: pgmap v13656: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4126053270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:48 smithi022 ceph-mon[31360]: pgmap v13656: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4126053270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:48 smithi022 ceph-mon[40250]: pgmap v13656: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4126053270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:50 smithi122 ceph-mon[32255]: pgmap v13657: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:50] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:33:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:50 smithi022 ceph-mon[31360]: pgmap v13657: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:50 smithi022 ceph-mon[40250]: pgmap v13657: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:52 smithi122 ceph-mon[32255]: pgmap v13658: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:33:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:33:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:52 smithi022 ceph-mon[31360]: pgmap v13658: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:52 smithi022 ceph-mon[40250]: pgmap v13658: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:52.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:53.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:53.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:53.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:54 smithi122 ceph-mon[32255]: pgmap v13659: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3723160576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:54 smithi022 ceph-mon[31360]: pgmap v13659: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3723160576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:54 smithi022 ceph-mon[40250]: pgmap v13659: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3723160576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:56 smithi122 ceph-mon[32255]: pgmap v13660: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:33:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:33:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:56 smithi022 ceph-mon[31360]: pgmap v13660: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:56 smithi022 ceph-mon[40250]: pgmap v13660: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:33:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:33:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:33:58.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:33:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:58 smithi122 ceph-mon[32255]: pgmap v13661: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:58.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:33:58.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:33:58.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:33:58.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:58 smithi022 ceph-mon[40250]: pgmap v13661: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:58.507 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:58 smithi022 ceph-mon[31360]: pgmap v13661: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:33:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:33:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/362559442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:33:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/362559442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:33:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:33:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/362559442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:00] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:34:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:00 smithi122 ceph-mon[32255]: pgmap v13662: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:00 smithi022 ceph-mon[31360]: pgmap v13662: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:00 smithi022 ceph-mon[40250]: pgmap v13662: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:02 smithi122 ceph-mon[32255]: pgmap v13663: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:34:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:34:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:02 smithi022 ceph-mon[31360]: pgmap v13663: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:02 smithi022 ceph-mon[40250]: pgmap v13663: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:03.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:03.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:03.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:04.126 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:04 smithi022 ceph-mon[40250]: pgmap v13664: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:04.126 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/956811905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:04.127 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:04 smithi022 ceph-mon[31360]: pgmap v13664: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:04.127 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/956811905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:04 smithi122 ceph-mon[32255]: pgmap v13664: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/956811905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:06 smithi122 ceph-mon[32255]: pgmap v13665: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:34:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:34:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:06 smithi022 ceph-mon[31360]: pgmap v13665: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:06 smithi022 ceph-mon[40250]: pgmap v13665: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:34:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:08 smithi122 ceph-mon[32255]: pgmap v13666: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:08 smithi022 ceph-mon[31360]: pgmap v13666: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:08 smithi022 ceph-mon[40250]: pgmap v13666: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:08.880 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:09.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:09.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:09.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:10] "GET /metrics HTTP/1.1" 200 202427 "" "Prometheus/2.18.1" 2021-12-16T01:34:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:10 smithi122 ceph-mon[32255]: pgmap v13667: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/983459041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:10 smithi022 ceph-mon[31360]: pgmap v13667: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/983459041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:10 smithi022 ceph-mon[40250]: pgmap v13667: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/983459041' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:12 smithi122 ceph-mon[32255]: pgmap v13668: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:34:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:34:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:12 smithi022 ceph-mon[31360]: pgmap v13668: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:12 smithi022 ceph-mon[40250]: pgmap v13668: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:14.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:14 smithi122 ceph-mon[32255]: pgmap v13669: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:14 smithi022 ceph-mon[31360]: pgmap v13669: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:14 smithi022 ceph-mon[40250]: pgmap v13669: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:14.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:14.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:14.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3067137426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3067137426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3067137426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:16 smithi122 ceph-mon[32255]: pgmap v13670: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:34:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:34:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:16 smithi022 ceph-mon[31360]: pgmap v13670: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:16 smithi022 ceph-mon[40250]: pgmap v13670: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:34:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:18 smithi122 ceph-mon[32255]: pgmap v13671: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:18 smithi022 ceph-mon[31360]: pgmap v13671: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:18 smithi022 ceph-mon[40250]: pgmap v13671: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:19.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:19.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:19.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:19.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:20 smithi022 ceph-mon[31360]: pgmap v13672: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1841596573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:20 smithi022 ceph-mon[40250]: pgmap v13672: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1841596573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:20] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T01:34:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:20 smithi122 ceph-mon[32255]: pgmap v13672: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1841596573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:34:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:34:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:22 smithi122 ceph-mon[32255]: pgmap v13673: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:22 smithi022 ceph-mon[31360]: pgmap v13673: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:22 smithi022 ceph-mon[40250]: pgmap v13673: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:24 smithi122 ceph-mon[32255]: pgmap v13674: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:24 smithi022 ceph-mon[31360]: pgmap v13674: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:24 smithi022 ceph-mon[40250]: pgmap v13674: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:24.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:25.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:25.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:25.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:26.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:34:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:34:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:26 smithi122 ceph-mon[32255]: pgmap v13675: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1099607368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:26 smithi022 ceph-mon[31360]: pgmap v13675: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1099607368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:26 smithi022 ceph-mon[40250]: pgmap v13675: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1099607368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:34:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:28 smithi122 ceph-mon[32255]: pgmap v13676: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:28 smithi022 ceph-mon[31360]: pgmap v13676: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:28 smithi022 ceph-mon[40250]: pgmap v13676: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:30.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:30 smithi122 ceph-mon[32255]: pgmap v13677: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:30] "GET /metrics HTTP/1.1" 200 202424 "" "Prometheus/2.18.1" 2021-12-16T01:34:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:30 smithi022 ceph-mon[31360]: pgmap v13677: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:30 smithi022 ceph-mon[40250]: pgmap v13677: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:30.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1668616086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1668616086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:31.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1668616086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:34:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:34:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:32 smithi122 ceph-mon[32255]: pgmap v13678: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:34:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:34:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:34:32.525 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[31360]: pgmap v13678: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:32.526 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:34:32.526 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:34:32.526 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:34:32.526 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[40250]: pgmap v13678: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:32.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:34:32.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:34:32.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:34:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:33 smithi122 ceph-mon[32255]: pgmap v13679: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:34:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:33 smithi022 ceph-mon[31360]: pgmap v13679: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:34:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:33 smithi022 ceph-mon[40250]: pgmap v13679: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:34:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:34:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:34:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:34:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:34:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:34:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:34:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:34:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:34:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:34:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:34:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:34:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:34:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:35.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:35 smithi122 ceph-mon[32255]: pgmap v13680: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:35 smithi022 ceph-mon[31360]: pgmap v13680: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:35 smithi022 ceph-mon[40250]: pgmap v13680: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:36.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:36.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:36.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:34:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:34:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1709283536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1709283536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1709283536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:37 smithi022 ceph-mon[31360]: pgmap v13681: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:34:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:37 smithi022 ceph-mon[40250]: pgmap v13681: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:37 smithi122 ceph-mon[32255]: pgmap v13681: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:39 smithi122 ceph-mon[32255]: pgmap v13682: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:39 smithi022 ceph-mon[31360]: pgmap v13682: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:39 smithi022 ceph-mon[40250]: pgmap v13682: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:40] "GET /metrics HTTP/1.1" 200 202423 "" "Prometheus/2.18.1" 2021-12-16T01:34:41.099 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:41.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:41 smithi022 ceph-mon[31360]: pgmap v13683: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4003127784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:41 smithi022 ceph-mon[40250]: pgmap v13683: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4003127784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:41 smithi122 ceph-mon[32255]: pgmap v13683: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4003127784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:34:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:34:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:43 smithi122 ceph-mon[32255]: pgmap v13684: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:43 smithi022 ceph-mon[31360]: pgmap v13684: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:43 smithi022 ceph-mon[40250]: pgmap v13684: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:45 smithi122 ceph-mon[32255]: pgmap v13685: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:45 smithi022 ceph-mon[31360]: pgmap v13685: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:45 smithi022 ceph-mon[40250]: pgmap v13685: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:34:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:34:46.468 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:46.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:46.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:46.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:47 smithi022 ceph-mon[31360]: pgmap v13686: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2445289160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:34:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:47 smithi022 ceph-mon[40250]: pgmap v13686: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2445289160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:47 smithi122 ceph-mon[32255]: pgmap v13686: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2445289160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:49 smithi122 ceph-mon[32255]: pgmap v13687: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:49 smithi022 ceph-mon[31360]: pgmap v13687: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:49 smithi022 ceph-mon[40250]: pgmap v13687: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:50] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:34:51.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:51 smithi022 ceph-mon[31360]: pgmap v13688: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:51.845 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:51 smithi022 ceph-mon[40250]: pgmap v13688: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:51.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:51 smithi122 ceph-mon[32255]: pgmap v13688: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:52.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:52.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:52.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:52.573 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:34:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:34:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2571105267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2571105267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:53.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2571105267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:53 smithi122 ceph-mon[32255]: pgmap v13689: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:53 smithi022 ceph-mon[31360]: pgmap v13689: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:53 smithi022 ceph-mon[40250]: pgmap v13689: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:55 smithi122 ceph-mon[32255]: pgmap v13690: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:55 smithi022 ceph-mon[31360]: pgmap v13690: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:55 smithi022 ceph-mon[40250]: pgmap v13690: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:34:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:34:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:34:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:34:57.214 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:34:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:34:57.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:34:57.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:34:57.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:34:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:57 smithi122 ceph-mon[32255]: pgmap v13691: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3258435246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:57 smithi022 ceph-mon[31360]: pgmap v13691: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3258435246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:57 smithi022 ceph-mon[40250]: pgmap v13691: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:34:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3258435246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:34:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:34:59 smithi122 ceph-mon[32255]: pgmap v13692: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:34:59 smithi022 ceph-mon[31360]: pgmap v13692: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:34:59 smithi022 ceph-mon[40250]: pgmap v13692: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:00] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:35:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:01 smithi122 ceph-mon[32255]: pgmap v13693: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:01 smithi022 ceph-mon[31360]: pgmap v13693: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:01 smithi022 ceph-mon[40250]: pgmap v13693: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:02.585 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:35:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:35:02.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:02.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:02.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:03 smithi122 ceph-mon[32255]: pgmap v13694: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2787260318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:03 smithi022 ceph-mon[31360]: pgmap v13694: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2787260318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:03 smithi022 ceph-mon[40250]: pgmap v13694: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2787260318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:05 smithi122 ceph-mon[32255]: pgmap v13695: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:05 smithi022 ceph-mon[31360]: pgmap v13695: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:05 smithi022 ceph-mon[40250]: pgmap v13695: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:35:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:35:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:07 smithi022 ceph-mon[31360]: pgmap v13696: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:35:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:07 smithi022 ceph-mon[40250]: pgmap v13696: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:07 smithi122 ceph-mon[32255]: pgmap v13696: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:07.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:08.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:08.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:08.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4036912997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4036912997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4036912997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:09 smithi122 ceph-mon[32255]: pgmap v13697: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:09 smithi022 ceph-mon[31360]: pgmap v13697: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:09 smithi022 ceph-mon[40250]: pgmap v13697: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:10] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:35:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:11 smithi122 ceph-mon[32255]: pgmap v13698: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:11 smithi022 ceph-mon[31360]: pgmap v13698: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:11 smithi022 ceph-mon[40250]: pgmap v13698: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:35:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:35:13.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:13.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:13.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:13.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:13 smithi122 ceph-mon[32255]: pgmap v13699: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:13 smithi022 ceph-mon[31360]: pgmap v13699: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:13 smithi022 ceph-mon[40250]: pgmap v13699: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4279805496' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4279805496' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4279805496' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:15.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:15 smithi122 ceph-mon[32255]: pgmap v13700: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:15 smithi022 ceph-mon[31360]: pgmap v13700: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:15 smithi022 ceph-mon[40250]: pgmap v13700: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:35:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:35:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:17 smithi022 ceph-mon[31360]: pgmap v13701: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:35:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:17 smithi022 ceph-mon[40250]: pgmap v13701: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:17 smithi122 ceph-mon[32255]: pgmap v13701: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:18.695 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:19.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:19.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:19.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:19 smithi122 ceph-mon[32255]: pgmap v13702: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3541097475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:19 smithi022 ceph-mon[31360]: pgmap v13702: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3541097475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:19 smithi022 ceph-mon[40250]: pgmap v13702: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3541097475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:20] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:35:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:21 smithi122 ceph-mon[32255]: pgmap v13703: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:21 smithi022 ceph-mon[31360]: pgmap v13703: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:21 smithi022 ceph-mon[40250]: pgmap v13703: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:35:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:35:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:23 smithi122 ceph-mon[32255]: pgmap v13704: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:23 smithi022 ceph-mon[31360]: pgmap v13704: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:23 smithi022 ceph-mon[40250]: pgmap v13704: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:24.069 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:24.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:24.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:24.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2530991170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2530991170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2530991170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:25 smithi122 ceph-mon[32255]: pgmap v13705: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:25 smithi022 ceph-mon[31360]: pgmap v13705: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:25 smithi022 ceph-mon[40250]: pgmap v13705: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:26.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:35:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:35:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:27.644 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:35:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:27 smithi122 ceph-mon[32255]: pgmap v13706: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:27 smithi022 ceph-mon[31360]: pgmap v13706: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:27 smithi022 ceph-mon[40250]: pgmap v13706: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:29.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:29 smithi022 ceph-mon[31360]: pgmap v13707: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:29 smithi022 ceph-mon[40250]: pgmap v13707: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:29 smithi122 ceph-mon[32255]: pgmap v13707: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:30] "GET /metrics HTTP/1.1" 200 202426 "" "Prometheus/2.18.1" 2021-12-16T01:35:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3817560460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3817560460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3817560460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:31 smithi122 ceph-mon[32255]: pgmap v13708: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:31 smithi022 ceph-mon[31360]: pgmap v13708: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:31 smithi022 ceph-mon[40250]: pgmap v13708: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:32.578 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:35:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:35:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:35:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:35:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:35:32.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:35:32.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:35:32.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:35:32.995 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:35:32.995 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:35:32.995 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:35:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[31360]: pgmap v13709: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:35:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:35:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:35:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:35:34.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:35:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[40250]: pgmap v13709: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:35:34.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:35:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:35:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:35:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:35:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:34 smithi122 ceph-mon[32255]: pgmap v13709: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:35:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:35:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:35:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:35:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:35:34.808 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:35.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:35.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:35.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:36 smithi022 ceph-mon[31360]: pgmap v13710: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3701327822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:36 smithi022 ceph-mon[40250]: pgmap v13710: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3701327822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:35:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:35:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:36 smithi122 ceph-mon[32255]: pgmap v13710: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3701327822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:35:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:38 smithi022 ceph-mon[31360]: pgmap v13711: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:38 smithi022 ceph-mon[40250]: pgmap v13711: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:38 smithi122 ceph-mon[32255]: pgmap v13711: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:35:40.182 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:40 smithi022 ceph-mon[31360]: pgmap v13712: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:40 smithi022 ceph-mon[40250]: pgmap v13712: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:40] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:35:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:40 smithi122 ceph-mon[32255]: pgmap v13712: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:40.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:40.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:40.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:41.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1454991161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:41.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1454991161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1454991161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:42 smithi022 ceph-mon[31360]: pgmap v13713: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:42 smithi022 ceph-mon[40250]: pgmap v13713: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:42 smithi122 ceph-mon[32255]: pgmap v13713: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:35:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:35:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:44 smithi022 ceph-mon[31360]: pgmap v13714: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:44 smithi022 ceph-mon[40250]: pgmap v13714: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:44 smithi122 ceph-mon[32255]: pgmap v13714: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:45.553 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:45.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:45.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:45.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:46 smithi022 ceph-mon[40250]: pgmap v13715: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1731450510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:46 smithi022 ceph-mon[31360]: pgmap v13715: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1731450510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:46 smithi122 ceph-mon[32255]: pgmap v13715: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:35:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1731450510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:35:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:35:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:35:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:48 smithi122 ceph-mon[32255]: pgmap v13716: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:48 smithi022 ceph-mon[40250]: pgmap v13716: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:48 smithi022 ceph-mon[31360]: pgmap v13716: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:35:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:50 smithi122 ceph-mon[32255]: pgmap v13717: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:35:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:50] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:35:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:50 smithi022 ceph-mon[40250]: pgmap v13717: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:35:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:50 smithi022 ceph-mon[31360]: pgmap v13717: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:35:50.925 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:51.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:51.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:51.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:52 smithi122 ceph-mon[32255]: pgmap v13718: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1662645239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:52 smithi022 ceph-mon[31360]: pgmap v13718: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1662645239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:52 smithi022 ceph-mon[40250]: pgmap v13718: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1662645239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:35:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:35:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:54 smithi122 ceph-mon[32255]: pgmap v13719: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:54 smithi022 ceph-mon[31360]: pgmap v13719: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:54 smithi022 ceph-mon[40250]: pgmap v13719: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:56.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:35:56.299 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:56 smithi022 ceph-mon[31360]: pgmap v13720: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:35:56.299 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:56 smithi022 ceph-mon[40250]: pgmap v13720: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:35:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:56 smithi122 ceph-mon[32255]: pgmap v13720: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:35:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:35:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:35:56.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:35:56.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:35:56.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:35:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3090260133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3090260133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:35:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3090260133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:35:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:35:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:35:58 smithi122 ceph-mon[32255]: pgmap v13721: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:35:58 smithi022 ceph-mon[31360]: pgmap v13721: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:35:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:35:58 smithi022 ceph-mon[40250]: pgmap v13721: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T01:36:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:00] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:36:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:00 smithi122 ceph-mon[32255]: pgmap v13722: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:36:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:00 smithi022 ceph-mon[31360]: pgmap v13722: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:36:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:00 smithi022 ceph-mon[40250]: pgmap v13722: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T01:36:01.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:02.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:02 smithi122 ceph-mon[32255]: pgmap v13723: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1675854839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:02 smithi022 ceph-mon[31360]: pgmap v13723: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1675854839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:02 smithi022 ceph-mon[40250]: pgmap v13723: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1675854839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:36:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:36:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:04 smithi122 ceph-mon[32255]: pgmap v13724: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:04 smithi022 ceph-mon[31360]: pgmap v13724: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:04 smithi022 ceph-mon[40250]: pgmap v13724: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:06 smithi122 ceph-mon[32255]: pgmap v13725: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:36:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:36:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:06 smithi022 ceph-mon[31360]: pgmap v13725: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:06 smithi022 ceph-mon[40250]: pgmap v13725: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:07.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:07.391 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:07.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:07.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:07.411 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:36:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:08 smithi122 ceph-mon[32255]: pgmap v13726: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/465200549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:08 smithi022 ceph-mon[31360]: pgmap v13726: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/465200549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:08 smithi022 ceph-mon[40250]: pgmap v13726: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/465200549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:10 smithi122 ceph-mon[32255]: pgmap v13727: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:10] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:36:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:10 smithi022 ceph-mon[31360]: pgmap v13727: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:10 smithi022 ceph-mon[40250]: pgmap v13727: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:12.413 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:12.413 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:12 smithi022 ceph-mon[31360]: pgmap v13728: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:12.414 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:12 smithi022 ceph-mon[40250]: pgmap v13728: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:12.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:36:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:36:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:12 smithi122 ceph-mon[32255]: pgmap v13728: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:12.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:12.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:12.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:13.073 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1542343137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1542343137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1542343137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:14 smithi122 ceph-mon[32255]: pgmap v13729: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:14 smithi022 ceph-mon[31360]: pgmap v13729: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:14 smithi022 ceph-mon[40250]: pgmap v13729: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:36:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:36:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:16 smithi122 ceph-mon[32255]: pgmap v13730: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:16 smithi022 ceph-mon[31360]: pgmap v13730: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:16 smithi022 ceph-mon[40250]: pgmap v13730: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:36:17.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:18.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:18.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:18.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:18 smithi122 ceph-mon[32255]: pgmap v13731: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:18 smithi022 ceph-mon[31360]: pgmap v13731: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:18 smithi022 ceph-mon[40250]: pgmap v13731: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/366919843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/366919843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/366919843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:20] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:36:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:20 smithi122 ceph-mon[32255]: pgmap v13732: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:20 smithi022 ceph-mon[31360]: pgmap v13732: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:20 smithi022 ceph-mon[40250]: pgmap v13732: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:22 smithi122 ceph-mon[32255]: pgmap v13733: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:36:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:36:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:22 smithi022 ceph-mon[40250]: pgmap v13733: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:22 smithi022 ceph-mon[31360]: pgmap v13733: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:23.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:23.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:23.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:24 smithi122 ceph-mon[32255]: pgmap v13734: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3487069558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:24 smithi022 ceph-mon[40250]: pgmap v13734: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3487069558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:24 smithi022 ceph-mon[31360]: pgmap v13734: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3487069558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:26 smithi122 ceph-mon[32255]: pgmap v13735: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:36:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:36:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:26 smithi022 ceph-mon[31360]: pgmap v13735: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:26 smithi022 ceph-mon[40250]: pgmap v13735: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:36:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:28 smithi122 ceph-mon[32255]: pgmap v13736: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:28.524 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:28 smithi022 ceph-mon[40250]: pgmap v13736: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:28.525 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:28 smithi022 ceph-mon[31360]: pgmap v13736: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:28.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:28.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:28.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1673272986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1673272986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1673272986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:30] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:36:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:30 smithi122 ceph-mon[32255]: pgmap v13737: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:30 smithi022 ceph-mon[31360]: pgmap v13737: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:30 smithi022 ceph-mon[40250]: pgmap v13737: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:32 smithi122 ceph-mon[32255]: pgmap v13738: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:36:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:36:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:32 smithi022 ceph-mon[31360]: pgmap v13738: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:32 smithi022 ceph-mon[40250]: pgmap v13738: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:36:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:36:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:36:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:36:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:36:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:36:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:36:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:36:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:36:33.897 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[31360]: pgmap v13739: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:36:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:36:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:36:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:36:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[40250]: pgmap v13739: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:36:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:36:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:36:34.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:36:34.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:34.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:34.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:34 smithi122 ceph-mon[32255]: pgmap v13739: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:36:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:36:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:36:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:36:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/513910613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/513910613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/513910613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:36 smithi122 ceph-mon[32255]: pgmap v13740: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:36:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:36:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:36 smithi022 ceph-mon[31360]: pgmap v13740: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:36 smithi022 ceph-mon[40250]: pgmap v13740: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:36:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:38 smithi122 ceph-mon[32255]: pgmap v13741: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:38.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:38.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:38.391 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:38 smithi022 ceph-mon[31360]: pgmap v13741: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:38.391 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:38.391 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:38.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:38 smithi022 ceph-mon[40250]: pgmap v13741: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:38.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:38.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:39.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:39.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:39.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:39.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:39.647 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:39 smithi022 ceph-mon[40250]: pgmap v13742: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:39.648 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:39.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:39 smithi022 ceph-mon[31360]: pgmap v13742: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:39.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:39 smithi122 ceph-mon[32255]: pgmap v13742: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:40] "GET /metrics HTTP/1.1" 200 202430 "" "Prometheus/2.18.1" 2021-12-16T01:36:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2478409000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3013M 2021-12-16T01:36:40.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2478409000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3013M 2021-12-16T01:36:40.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:40.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2478409000' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:40.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:40.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:40.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3013M 2021-12-16T01:36:40.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: pgmap v13743: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3237M 2021-12-16T01:36:42.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: pgmap v13743: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:42.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3237M 2021-12-16T01:36:42.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: pgmap v13743: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:36:42.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3237M 2021-12-16T01:36:42.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:36:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:36:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:36:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:43 smithi022 ceph-mon[31360]: pgmap v13744: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:43 smithi022 ceph-mon[40250]: pgmap v13744: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:43 smithi122 ceph-mon[32255]: pgmap v13744: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:44.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:45.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:45.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:45.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:45 smithi022 ceph-mon[31360]: pgmap v13745: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/642252388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:45 smithi022 ceph-mon[40250]: pgmap v13745: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/642252388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:36:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:36:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:45 smithi122 ceph-mon[32255]: pgmap v13745: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/642252388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:36:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:47 smithi022 ceph-mon[31360]: pgmap v13746: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:47 smithi022 ceph-mon[40250]: pgmap v13746: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:47 smithi122 ceph-mon[32255]: pgmap v13746: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:50.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:49 smithi022 ceph-mon[31360]: pgmap v13747: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:49 smithi022 ceph-mon[40250]: pgmap v13747: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:50.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:50.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:50.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:50] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:36:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:49 smithi122 ceph-mon[32255]: pgmap v13747: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2944653990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2944653990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2944653990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:51 smithi022 ceph-mon[31360]: pgmap v13748: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:51 smithi022 ceph-mon[40250]: pgmap v13748: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:51 smithi122 ceph-mon[32255]: pgmap v13748: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:36:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:36:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:53 smithi022 ceph-mon[40250]: pgmap v13749: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:53 smithi022 ceph-mon[31360]: pgmap v13749: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:53 smithi122 ceph-mon[32255]: pgmap v13749: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:55.393 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:36:55.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:36:55.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:36:55.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:36:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:55 smithi022 ceph-mon[31360]: pgmap v13750: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1603849822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:55 smithi022 ceph-mon[40250]: pgmap v13750: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1603849822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:36:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:36:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:55 smithi122 ceph-mon[32255]: pgmap v13750: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:36:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1603849822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:36:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:36:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:36:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:57 smithi022 ceph-mon[31360]: pgmap v13751: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:57 smithi022 ceph-mon[40250]: pgmap v13751: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:36:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:57 smithi122 ceph-mon[32255]: pgmap v13751: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:36:59 smithi022 ceph-mon[31360]: pgmap v13752: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:36:59 smithi022 ceph-mon[40250]: pgmap v13752: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:36:59 smithi122 ceph-mon[32255]: pgmap v13752: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:00] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:37:00.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:01.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:01.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:01.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:01 smithi022 ceph-mon[31360]: pgmap v13753: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3261974044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:01 smithi022 ceph-mon[40250]: pgmap v13753: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3261974044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:01 smithi122 ceph-mon[32255]: pgmap v13753: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3261974044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:37:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:37:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:03 smithi022 ceph-mon[31360]: pgmap v13754: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:03 smithi022 ceph-mon[40250]: pgmap v13754: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:03 smithi122 ceph-mon[32255]: pgmap v13754: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:06.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:06 smithi022 ceph-mon[31360]: pgmap v13755: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:06 smithi022 ceph-mon[40250]: pgmap v13755: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:06 smithi122 ceph-mon[32255]: pgmap v13755: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:06.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:37:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:37:06.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:06.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:06.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3263930323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3263930323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3263930323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:37:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:08 smithi022 ceph-mon[31360]: pgmap v13756: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:08.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:08 smithi022 ceph-mon[40250]: pgmap v13756: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:08 smithi122 ceph-mon[32255]: pgmap v13756: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:10 smithi022 ceph-mon[31360]: pgmap v13757: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:10 smithi022 ceph-mon[40250]: pgmap v13757: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:10] "GET /metrics HTTP/1.1" 200 202428 "" "Prometheus/2.18.1" 2021-12-16T01:37:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:10 smithi122 ceph-mon[32255]: pgmap v13757: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:11.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:11.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:11.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:11.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:12 smithi022 ceph-mon[31360]: pgmap v13758: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1631690504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:12 smithi022 ceph-mon[40250]: pgmap v13758: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1631690504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:12 smithi122 ceph-mon[32255]: pgmap v13758: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1631690504' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:37:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:37:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:14 smithi022 ceph-mon[31360]: pgmap v13759: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:14 smithi022 ceph-mon[40250]: pgmap v13759: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:14 smithi122 ceph-mon[32255]: pgmap v13759: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:16 smithi022 ceph-mon[31360]: pgmap v13760: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:16 smithi022 ceph-mon[40250]: pgmap v13760: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:16 smithi122 ceph-mon[32255]: pgmap v13760: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:37:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:37:16.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:17.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:17.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:17.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:17.242 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:17.243 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:37:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:18 smithi022 ceph-mon[31360]: pgmap v13761: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1301041047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:18 smithi022 ceph-mon[40250]: pgmap v13761: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1301041047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:18 smithi122 ceph-mon[32255]: pgmap v13761: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1301041047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:20 smithi022 ceph-mon[31360]: pgmap v13762: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:20 smithi022 ceph-mon[40250]: pgmap v13762: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:20 smithi122 ceph-mon[32255]: pgmap v13762: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:20] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:37:22.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:22 smithi122 ceph-mon[32255]: pgmap v13763: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:22 smithi022 ceph-mon[31360]: pgmap v13763: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:22 smithi022 ceph-mon[40250]: pgmap v13763: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:22.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:22.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:22.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:37:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:37:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/214946417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/214946417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/214946417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:24 smithi122 ceph-mon[32255]: pgmap v13764: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:24 smithi022 ceph-mon[31360]: pgmap v13764: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:24 smithi022 ceph-mon[40250]: pgmap v13764: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:26 smithi122 ceph-mon[32255]: pgmap v13765: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:26.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:37:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:37:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:26 smithi022 ceph-mon[31360]: pgmap v13765: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:26 smithi022 ceph-mon[40250]: pgmap v13765: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:27.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:37:27.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:27.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:27.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:28 smithi022 ceph-mon[31360]: pgmap v13766: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1057575644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:28 smithi022 ceph-mon[40250]: pgmap v13766: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1057575644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:28 smithi122 ceph-mon[32255]: pgmap v13766: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1057575644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:30] "GET /metrics HTTP/1.1" 200 202432 "" "Prometheus/2.18.1" 2021-12-16T01:37:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:30 smithi122 ceph-mon[32255]: pgmap v13767: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:30 smithi022 ceph-mon[31360]: pgmap v13767: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:30 smithi022 ceph-mon[40250]: pgmap v13767: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:32 smithi122 ceph-mon[32255]: pgmap v13768: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:32 smithi022 ceph-mon[31360]: pgmap v13768: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:32 smithi022 ceph-mon[40250]: pgmap v13768: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:37:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:37:32.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:33.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:33.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:33.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:34 smithi122 ceph-mon[32255]: pgmap v13769: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3487860301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:37:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:37:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:37:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:37:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[31360]: pgmap v13769: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3487860301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:37:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:37:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:37:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:37:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[40250]: pgmap v13769: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3487860301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:37:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:37:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:37:34.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:37:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:36 smithi122 ceph-mon[32255]: pgmap v13770: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:37:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:37:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:36 smithi022 ceph-mon[31360]: pgmap v13770: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:36 smithi022 ceph-mon[40250]: pgmap v13770: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:37:38.355 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:38.356 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:38 smithi022 ceph-mon[31360]: pgmap v13771: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:38.356 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:38 smithi022 ceph-mon[40250]: pgmap v13771: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:38 smithi122 ceph-mon[32255]: pgmap v13771: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:38.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:38.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:38.724 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1130209817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1130209817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1130209817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:40] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T01:37:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:40 smithi122 ceph-mon[32255]: pgmap v13772: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:40 smithi022 ceph-mon[31360]: pgmap v13772: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:40 smithi022 ceph-mon[40250]: pgmap v13772: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:37:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:37:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:37:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:37:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:37:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:37:41.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:37:41.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:37:41.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:37:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:42 smithi122 ceph-mon[32255]: pgmap v13773: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:37:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:37:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:37:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:42 smithi022 ceph-mon[31360]: pgmap v13773: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:37:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:42 smithi022 ceph-mon[40250]: pgmap v13773: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:37:43.725 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:44 smithi122 ceph-mon[32255]: pgmap v13774: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:44 smithi022 ceph-mon[31360]: pgmap v13774: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:44 smithi022 ceph-mon[40250]: pgmap v13774: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/504261896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/504261896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/504261896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:46 smithi122 ceph-mon[32255]: pgmap v13775: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:37:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:37:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:46 smithi022 ceph-mon[31360]: pgmap v13775: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:46 smithi022 ceph-mon[40250]: pgmap v13775: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:37:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:48 smithi122 ceph-mon[32255]: pgmap v13776: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:48 smithi022 ceph-mon[31360]: pgmap v13776: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:48 smithi022 ceph-mon[40250]: pgmap v13776: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:49.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:49.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:49.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:49.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:50] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T01:37:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:50 smithi122 ceph-mon[32255]: pgmap v13777: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2700463094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:50 smithi022 ceph-mon[31360]: pgmap v13777: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2700463094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:50 smithi022 ceph-mon[40250]: pgmap v13777: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2700463094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:52 smithi122 ceph-mon[32255]: pgmap v13778: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:37:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:37:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:52 smithi022 ceph-mon[31360]: pgmap v13778: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:52 smithi022 ceph-mon[40250]: pgmap v13778: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:54 smithi122 ceph-mon[32255]: pgmap v13779: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:54.469 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:37:54.469 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:54 smithi022 ceph-mon[31360]: pgmap v13779: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:54.470 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:54 smithi022 ceph-mon[40250]: pgmap v13779: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:54.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:37:54.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:37:54.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:37:55.103 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3129830274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:55.103 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3129830274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3129830274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:37:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:56 smithi122 ceph-mon[32255]: pgmap v13780: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:37:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:37:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:56 smithi022 ceph-mon[31360]: pgmap v13780: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:56 smithi022 ceph-mon[40250]: pgmap v13780: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:37:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:37:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:37:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:37:58 smithi122 ceph-mon[32255]: pgmap v13781: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:37:58 smithi022 ceph-mon[31360]: pgmap v13781: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:37:58 smithi022 ceph-mon[40250]: pgmap v13781: 132 pgs: 132 active+clean; 582 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:37:59.841 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:00] "GET /metrics HTTP/1.1" 200 202425 "" "Prometheus/2.18.1" 2021-12-16T01:38:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:00 smithi122 ceph-mon[32255]: pgmap v13782: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:00 smithi022 ceph-mon[31360]: pgmap v13782: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:00 smithi022 ceph-mon[40250]: pgmap v13782: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1462304493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1462304493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1462304493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:02 smithi122 ceph-mon[32255]: pgmap v13783: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:38:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:38:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:02 smithi022 ceph-mon[31360]: pgmap v13783: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:02 smithi022 ceph-mon[40250]: pgmap v13783: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:04 smithi122 ceph-mon[32255]: pgmap v13784: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:04 smithi022 ceph-mon[31360]: pgmap v13784: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:04 smithi022 ceph-mon[40250]: pgmap v13784: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:05.208 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:05.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:05.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:05.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:06 smithi122 ceph-mon[32255]: pgmap v13785: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2069474098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:06.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:38:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:38:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:06 smithi022 ceph-mon[31360]: pgmap v13785: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2069474098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:06 smithi022 ceph-mon[40250]: pgmap v13785: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2069474098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:07.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:38:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:08 smithi122 ceph-mon[32255]: pgmap v13786: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:08 smithi022 ceph-mon[31360]: pgmap v13786: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:08 smithi022 ceph-mon[40250]: pgmap v13786: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:38:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:10 smithi122 ceph-mon[32255]: pgmap v13787: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:38:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:10] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:38:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:10 smithi022 ceph-mon[31360]: pgmap v13787: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:38:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:10 smithi022 ceph-mon[40250]: pgmap v13787: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2021-12-16T01:38:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:10.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:10.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:10.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/480971971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/480971971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/480971971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:12 smithi122 ceph-mon[32255]: pgmap v13788: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:38:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:38:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:12 smithi022 ceph-mon[31360]: pgmap v13788: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:12 smithi022 ceph-mon[40250]: pgmap v13788: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:14 smithi122 ceph-mon[32255]: pgmap v13789: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:14 smithi022 ceph-mon[31360]: pgmap v13789: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:14 smithi022 ceph-mon[40250]: pgmap v13789: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:15.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:16 smithi022 ceph-mon[31360]: pgmap v13790: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:38:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:16 smithi022 ceph-mon[40250]: pgmap v13790: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:38:16.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:16.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:16.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:16 smithi122 ceph-mon[32255]: pgmap v13790: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:38:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:38:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:38:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2806778919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:38:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:17.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2806778919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2806778919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:18 smithi122 ceph-mon[32255]: pgmap v13791: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:18 smithi022 ceph-mon[31360]: pgmap v13791: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:18 smithi022 ceph-mon[40250]: pgmap v13791: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:38:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:20 smithi122 ceph-mon[32255]: pgmap v13792: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:38:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:20 smithi022 ceph-mon[31360]: pgmap v13792: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:38:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:20 smithi022 ceph-mon[40250]: pgmap v13792: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:38:21.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:21.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:21.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:21.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:38:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:38:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:22 smithi122 ceph-mon[32255]: pgmap v13793: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3183457262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:22 smithi022 ceph-mon[31360]: pgmap v13793: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3183457262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:22 smithi022 ceph-mon[40250]: pgmap v13793: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3183457262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:24 smithi122 ceph-mon[32255]: pgmap v13794: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:24 smithi022 ceph-mon[31360]: pgmap v13794: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:24 smithi022 ceph-mon[40250]: pgmap v13794: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:26 smithi122 ceph-mon[32255]: pgmap v13795: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:26.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:38:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:38:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:26 smithi022 ceph-mon[31360]: pgmap v13795: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:26 smithi022 ceph-mon[40250]: pgmap v13795: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:27.062 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3964182355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3964182355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3964182355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:38:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:28 smithi122 ceph-mon[32255]: pgmap v13796: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:28 smithi022 ceph-mon[31360]: pgmap v13796: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:28 smithi022 ceph-mon[40250]: pgmap v13796: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:38:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:30 smithi122 ceph-mon[32255]: pgmap v13797: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:30 smithi022 ceph-mon[31360]: pgmap v13797: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:30 smithi022 ceph-mon[40250]: pgmap v13797: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:32.064 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:32 smithi122 ceph-mon[32255]: pgmap v13798: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:38:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:38:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:32.438 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:32 smithi022 ceph-mon[31360]: pgmap v13798: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:32.439 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:32 smithi022 ceph-mon[40250]: pgmap v13798: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/893449158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/893449158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/893449158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:34 smithi122 ceph-mon[32255]: pgmap v13799: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:38:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:38:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:38:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:38:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[31360]: pgmap v13799: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:38:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:38:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:38:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:38:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[40250]: pgmap v13799: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:38:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:38:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:38:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:38:36.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:38:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:38:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:36 smithi122 ceph-mon[32255]: pgmap v13800: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:36 smithi022 ceph-mon[31360]: pgmap v13800: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:36 smithi022 ceph-mon[40250]: pgmap v13800: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:37.439 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:38:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:37.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:37.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:37.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:38 smithi022 ceph-mon[31360]: pgmap v13801: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4123863094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:38 smithi022 ceph-mon[40250]: pgmap v13801: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4123863094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:38 smithi122 ceph-mon[32255]: pgmap v13801: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4123863094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:38:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:40 smithi122 ceph-mon[32255]: pgmap v13802: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:40 smithi022 ceph-mon[31360]: pgmap v13802: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:40 smithi022 ceph-mon[40250]: pgmap v13802: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[31360]: pgmap v13803: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:38:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:38:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:38:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:38:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[40250]: pgmap v13803: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:38:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:38:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:38:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:38:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:38:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:38:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:42 smithi122 ceph-mon[32255]: pgmap v13803: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:38:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:38:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:38:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:38:42.808 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:43.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:43.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:43.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2683299514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2683299514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2683299514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:44 smithi022 ceph-mon[31360]: pgmap v13804: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:44 smithi022 ceph-mon[40250]: pgmap v13804: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:44 smithi122 ceph-mon[32255]: pgmap v13804: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:38:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:38:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:46 smithi122 ceph-mon[32255]: pgmap v13805: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:46 smithi022 ceph-mon[31360]: pgmap v13805: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:46 smithi022 ceph-mon[40250]: pgmap v13805: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:38:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:48.179 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:48 smithi022 ceph-mon[31360]: pgmap v13806: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:48 smithi022 ceph-mon[40250]: pgmap v13806: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:48.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:48.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:48.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:48 smithi122 ceph-mon[32255]: pgmap v13806: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1862782934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1862782934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1862782934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:50 smithi122 ceph-mon[32255]: pgmap v13807: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:50] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:38:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:50 smithi022 ceph-mon[31360]: pgmap v13807: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:50 smithi022 ceph-mon[40250]: pgmap v13807: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:52 smithi022 ceph-mon[31360]: pgmap v13808: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:52 smithi022 ceph-mon[40250]: pgmap v13808: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:52 smithi122 ceph-mon[32255]: pgmap v13808: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:38:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:38:53.555 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:53.924 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:53.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:53.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:38:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:54 smithi022 ceph-mon[31360]: pgmap v13809: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/573385303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:54 smithi022 ceph-mon[40250]: pgmap v13809: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/573385303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:54 smithi122 ceph-mon[32255]: pgmap v13809: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/573385303' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:38:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:56 smithi122 ceph-mon[32255]: pgmap v13810: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:38:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:38:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:56 smithi022 ceph-mon[31360]: pgmap v13810: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:56 smithi022 ceph-mon[40250]: pgmap v13810: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:38:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:38:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:38:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:38:58 smithi022 ceph-mon[31360]: pgmap v13811: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:38:58 smithi022 ceph-mon[40250]: pgmap v13811: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:38:58 smithi122 ceph-mon[32255]: pgmap v13811: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:38:58.926 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:38:59.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:38:59.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:38:59.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:00] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:00 smithi122 ceph-mon[32255]: pgmap v13812: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4068194908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:00 smithi022 ceph-mon[31360]: pgmap v13812: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4068194908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:00 smithi022 ceph-mon[40250]: pgmap v13812: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4068194908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:01 smithi022 ceph-mon[31360]: pgmap v13813: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:01 smithi022 ceph-mon[40250]: pgmap v13813: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:01 smithi122 ceph-mon[32255]: pgmap v13813: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:39:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:39:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:03 smithi022 ceph-mon[31360]: pgmap v13814: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:03 smithi022 ceph-mon[40250]: pgmap v13814: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:03 smithi122 ceph-mon[32255]: pgmap v13814: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:04.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:04.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:04.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:04.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:05 smithi022 ceph-mon[31360]: pgmap v13815: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1064701954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:05 smithi022 ceph-mon[40250]: pgmap v13815: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1064701954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:05 smithi122 ceph-mon[32255]: pgmap v13815: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1064701954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:06.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:39:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:39:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:07 smithi022 ceph-mon[31360]: pgmap v13816: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:39:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:07 smithi022 ceph-mon[40250]: pgmap v13816: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:07 smithi122 ceph-mon[32255]: pgmap v13816: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:09.663 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:09 smithi022 ceph-mon[31360]: pgmap v13817: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:09 smithi022 ceph-mon[40250]: pgmap v13817: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:09 smithi122 ceph-mon[32255]: pgmap v13817: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:10.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:10.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:10] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:39:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2860367833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2860367833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2860367833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:11 smithi022 ceph-mon[31360]: pgmap v13818: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:11 smithi022 ceph-mon[40250]: pgmap v13818: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:11 smithi122 ceph-mon[32255]: pgmap v13818: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:39:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:39:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:13 smithi022 ceph-mon[31360]: pgmap v13819: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:13 smithi022 ceph-mon[40250]: pgmap v13819: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:13 smithi122 ceph-mon[32255]: pgmap v13819: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:15.034 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:15.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:15.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:15.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:15 smithi022 ceph-mon[31360]: pgmap v13820: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/736545330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:15 smithi022 ceph-mon[40250]: pgmap v13820: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/736545330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:15 smithi122 ceph-mon[32255]: pgmap v13820: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/736545330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:16.470 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:39:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:39:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:17 smithi022 ceph-mon[31360]: pgmap v13821: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:39:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:17 smithi022 ceph-mon[40250]: pgmap v13821: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:17 smithi122 ceph-mon[32255]: pgmap v13821: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:19 smithi022 ceph-mon[31360]: pgmap v13822: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:19 smithi022 ceph-mon[40250]: pgmap v13822: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:19 smithi122 ceph-mon[32255]: pgmap v13822: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:20.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:20] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:39:20.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:20.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:20.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:21 smithi022 ceph-mon[31360]: pgmap v13823: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3260062573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:21 smithi022 ceph-mon[40250]: pgmap v13823: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3260062573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:21 smithi122 ceph-mon[32255]: pgmap v13823: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3260062573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:39:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:39:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:23 smithi022 ceph-mon[31360]: pgmap v13824: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:23 smithi022 ceph-mon[40250]: pgmap v13824: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:23 smithi122 ceph-mon[32255]: pgmap v13824: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:25 smithi022 ceph-mon[40250]: pgmap v13825: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:25 smithi022 ceph-mon[31360]: pgmap v13825: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:25.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:25 smithi122 ceph-mon[32255]: pgmap v13825: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:26.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:26.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:26.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:39:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:39:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4178168489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4178168489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4178168489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:27 smithi022 ceph-mon[31360]: pgmap v13826: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:39:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:27 smithi022 ceph-mon[40250]: pgmap v13826: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:27 smithi122 ceph-mon[32255]: pgmap v13826: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:29 smithi022 ceph-mon[31360]: pgmap v13827: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:29 smithi022 ceph-mon[40250]: pgmap v13827: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:29 smithi122 ceph-mon[32255]: pgmap v13827: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:30] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:39:31.149 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:31.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:31.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:31.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:31 smithi022 ceph-mon[31360]: pgmap v13828: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:31 smithi022 ceph-mon[40250]: pgmap v13828: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:31 smithi122 ceph-mon[32255]: pgmap v13828: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2941666241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:39:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:39:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2941666241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2941666241' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:33 smithi022 ceph-mon[31360]: pgmap v13829: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:33 smithi022 ceph-mon[40250]: pgmap v13829: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:33 smithi122 ceph-mon[32255]: pgmap v13829: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:39:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:39:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:39:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:39:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:39:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:39:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:39:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:39:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:39:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:39:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:39:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:39:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:35 smithi022 ceph-mon[31360]: pgmap v13830: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:35 smithi022 ceph-mon[40250]: pgmap v13830: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:35 smithi122 ceph-mon[32255]: pgmap v13830: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:36.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:39:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:39:36.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:36.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:36.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:36.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:39:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:37 smithi022 ceph-mon[31360]: pgmap v13831: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/325282568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:37 smithi022 ceph-mon[40250]: pgmap v13831: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/325282568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:37 smithi122 ceph-mon[32255]: pgmap v13831: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/325282568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:39 smithi022 ceph-mon[31360]: pgmap v13832: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:39 smithi022 ceph-mon[40250]: pgmap v13832: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:39 smithi122 ceph-mon[32255]: pgmap v13832: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:39:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:41 smithi022 ceph-mon[31360]: pgmap v13833: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:41.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:41 smithi022 ceph-mon[40250]: pgmap v13833: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:41.894 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:41 smithi122 ceph-mon[32255]: pgmap v13833: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:42.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:42.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:42.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:39:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:39:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:39:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2650616897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:39:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:39:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:39:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:39:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2650616897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:39:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:39:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:39:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:39:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2650616897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:39:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:39:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:39:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:43 smithi022 ceph-mon[31360]: pgmap v13834: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:43 smithi022 ceph-mon[40250]: pgmap v13834: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:43 smithi122 ceph-mon[32255]: pgmap v13834: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:45 smithi022 ceph-mon[31360]: pgmap v13835: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:45 smithi022 ceph-mon[40250]: pgmap v13835: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:45 smithi122 ceph-mon[32255]: pgmap v13835: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:46.529 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:39:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:39:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:46.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:47.271 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:39:47.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:47.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:47.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:47 smithi122 ceph-mon[32255]: pgmap v13836: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:47 smithi022 ceph-mon[31360]: pgmap v13836: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:47 smithi022 ceph-mon[40250]: pgmap v13836: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2965551677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2965551677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2965551677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:49 smithi122 ceph-mon[32255]: pgmap v13837: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:49 smithi022 ceph-mon[31360]: pgmap v13837: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:49 smithi022 ceph-mon[40250]: pgmap v13837: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:50] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:39:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:51 smithi122 ceph-mon[32255]: pgmap v13838: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:51 smithi022 ceph-mon[31360]: pgmap v13838: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:51 smithi022 ceph-mon[40250]: pgmap v13838: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:52.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:39:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:39:53.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:53.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:53.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:53 smithi122 ceph-mon[32255]: pgmap v13839: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4135696024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:53 smithi022 ceph-mon[31360]: pgmap v13839: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4135696024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:53 smithi022 ceph-mon[40250]: pgmap v13839: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4135696024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:55 smithi122 ceph-mon[32255]: pgmap v13840: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:55 smithi022 ceph-mon[31360]: pgmap v13840: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:55 smithi022 ceph-mon[40250]: pgmap v13840: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:39:56.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:39:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:39:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:39:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:57 smithi022 ceph-mon[31360]: pgmap v13841: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:39:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:57 smithi022 ceph-mon[40250]: pgmap v13841: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:57 smithi122 ceph-mon[32255]: pgmap v13841: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:39:58.014 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:39:58.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:39:58.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:39:58.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:39:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/420221905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/420221905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/420221905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:39:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:39:59 smithi122 ceph-mon[32255]: pgmap v13842: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:39:59 smithi022 ceph-mon[31360]: pgmap v13842: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:39:59 smithi022 ceph-mon[40250]: pgmap v13842: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:00] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:40:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T01:40:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T01:40:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T01:40:01.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:01 smithi122 ceph-mon[32255]: pgmap v13843: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:01 smithi022 ceph-mon[31360]: pgmap v13843: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:01 smithi022 ceph-mon[40250]: pgmap v13843: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:40:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:40:03.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:03.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:03.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:03.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:03.750 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:03 smithi022 ceph-mon[31360]: pgmap v13844: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:03.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:03 smithi022 ceph-mon[40250]: pgmap v13844: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:03 smithi122 ceph-mon[32255]: pgmap v13844: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3159599417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3159599417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3159599417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:05 smithi122 ceph-mon[32255]: pgmap v13845: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:05 smithi022 ceph-mon[31360]: pgmap v13845: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:05 smithi022 ceph-mon[40250]: pgmap v13845: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:06.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:40:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:40:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:07 smithi022 ceph-mon[31360]: pgmap v13846: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:07 smithi022 ceph-mon[40250]: pgmap v13846: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:40:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:07 smithi122 ceph-mon[32255]: pgmap v13846: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:08.751 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:09.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:09.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:09.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:09 smithi122 ceph-mon[32255]: pgmap v13847: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1841828482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:09 smithi022 ceph-mon[31360]: pgmap v13847: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1841828482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:09 smithi022 ceph-mon[40250]: pgmap v13847: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1841828482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:10] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:40:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:11 smithi122 ceph-mon[32255]: pgmap v13848: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:11 smithi022 ceph-mon[31360]: pgmap v13848: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:11 smithi022 ceph-mon[40250]: pgmap v13848: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:40:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:40:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:13 smithi122 ceph-mon[32255]: pgmap v13849: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:13 smithi022 ceph-mon[31360]: pgmap v13849: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:13 smithi022 ceph-mon[40250]: pgmap v13849: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:14.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:14.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:14.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2594793850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2594793850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2594793850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:15 smithi122 ceph-mon[32255]: pgmap v13850: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:15 smithi022 ceph-mon[31360]: pgmap v13850: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:15 smithi022 ceph-mon[40250]: pgmap v13850: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:16.590 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:40:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:40:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:17 smithi022 ceph-mon[31360]: pgmap v13851: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:40:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:17 smithi022 ceph-mon[40250]: pgmap v13851: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:17 smithi122 ceph-mon[32255]: pgmap v13851: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:19.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:19 smithi022 ceph-mon[31360]: pgmap v13852: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:19 smithi022 ceph-mon[40250]: pgmap v13852: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:19.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:19.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:19.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:19 smithi122 ceph-mon[32255]: pgmap v13852: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:40:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/405300057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/405300057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/405300057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:21 smithi122 ceph-mon[32255]: pgmap v13853: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:21 smithi022 ceph-mon[31360]: pgmap v13853: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:21 smithi022 ceph-mon[40250]: pgmap v13853: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:40:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:40:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:23 smithi122 ceph-mon[32255]: pgmap v13854: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:23 smithi022 ceph-mon[31360]: pgmap v13854: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:23 smithi022 ceph-mon[40250]: pgmap v13854: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:24.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:25.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:25.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:25.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:25 smithi122 ceph-mon[32255]: pgmap v13855: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2816102415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:25 smithi022 ceph-mon[31360]: pgmap v13855: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2816102415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:25 smithi022 ceph-mon[40250]: pgmap v13855: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2816102415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:26.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:40:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:40:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:27 smithi022 ceph-mon[31360]: pgmap v13856: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:40:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:27 smithi022 ceph-mon[40250]: pgmap v13856: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:27 smithi122 ceph-mon[32255]: pgmap v13856: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:29 smithi122 ceph-mon[32255]: pgmap v13857: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:29 smithi022 ceph-mon[31360]: pgmap v13857: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:29 smithi022 ceph-mon[40250]: pgmap v13857: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:30.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:40:30.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:30.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:30.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2137639172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2137639172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2137639172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:31 smithi122 ceph-mon[32255]: pgmap v13858: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:31 smithi022 ceph-mon[31360]: pgmap v13858: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:31 smithi022 ceph-mon[40250]: pgmap v13858: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:40:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:40:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:33 smithi122 ceph-mon[32255]: pgmap v13859: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:33 smithi022 ceph-mon[31360]: pgmap v13859: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:33 smithi022 ceph-mon[40250]: pgmap v13859: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:40:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:40:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:40:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:40:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:40:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:40:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:40:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:40:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:40:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:40:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:40:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:40:35.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:35 smithi122 ceph-mon[32255]: pgmap v13860: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:35.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:35.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:35.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:35.976 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:35 smithi022 ceph-mon[40250]: pgmap v13860: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:35.976 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:35 smithi022 ceph-mon[31360]: pgmap v13860: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:36.636 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:40:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:40:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/447404155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/447404155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/447404155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:40:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:37 smithi022 ceph-mon[31360]: pgmap v13861: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:37 smithi022 ceph-mon[40250]: pgmap v13861: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:37 smithi122 ceph-mon[32255]: pgmap v13861: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:39 smithi122 ceph-mon[32255]: pgmap v13862: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:39 smithi022 ceph-mon[31360]: pgmap v13862: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:39 smithi022 ceph-mon[40250]: pgmap v13862: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:40] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:40:40.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:41.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:41.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:41.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:41 smithi122 ceph-mon[32255]: pgmap v13863: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4218521753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:41 smithi022 ceph-mon[31360]: pgmap v13863: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4218521753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:41 smithi022 ceph-mon[40250]: pgmap v13863: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4218521753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:42.648 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:40:42.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:40:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:40:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:40:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:40:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:40:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:40:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:40:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:40:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:40:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:40:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:43 smithi122 ceph-mon[32255]: pgmap v13864: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:40:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:43 smithi022 ceph-mon[31360]: pgmap v13864: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:40:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:43 smithi022 ceph-mon[40250]: pgmap v13864: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:40:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:45 smithi122 ceph-mon[32255]: pgmap v13865: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:45 smithi022 ceph-mon[31360]: pgmap v13865: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:45 smithi022 ceph-mon[40250]: pgmap v13865: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:46.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:46.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:40:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:40:46.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:46.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:46.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:40:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:47 smithi022 ceph-mon[31360]: pgmap v13866: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1867550731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:47 smithi022 ceph-mon[40250]: pgmap v13866: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1867550731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:47 smithi122 ceph-mon[32255]: pgmap v13866: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1867550731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:49 smithi022 ceph-mon[31360]: pgmap v13867: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:49 smithi022 ceph-mon[40250]: pgmap v13867: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:49 smithi122 ceph-mon[32255]: pgmap v13867: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:50] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:40:51.717 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:51 smithi022 ceph-mon[31360]: pgmap v13868: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:51.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:51 smithi022 ceph-mon[40250]: pgmap v13868: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:51 smithi122 ceph-mon[32255]: pgmap v13868: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:52.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:52.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:52.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/632739593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:40:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:40:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/632739593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/632739593' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:53 smithi022 ceph-mon[31360]: pgmap v13869: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:53 smithi022 ceph-mon[40250]: pgmap v13869: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:53 smithi122 ceph-mon[32255]: pgmap v13869: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:55 smithi022 ceph-mon[31360]: pgmap v13870: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:55 smithi022 ceph-mon[40250]: pgmap v13870: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:55 smithi122 ceph-mon[32255]: pgmap v13870: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:56.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:40:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:40:57.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:40:57.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:40:57.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:40:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:40:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:57 smithi022 ceph-mon[31360]: pgmap v13871: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1509771590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:57 smithi022 ceph-mon[40250]: pgmap v13871: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1509771590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:40:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:57 smithi122 ceph-mon[32255]: pgmap v13871: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:40:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1509771590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:40:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:40:59 smithi022 ceph-mon[31360]: pgmap v13872: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:40:59 smithi022 ceph-mon[40250]: pgmap v13872: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:40:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:40:59 smithi122 ceph-mon[32255]: pgmap v13872: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:00] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:41:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:01 smithi022 ceph-mon[31360]: pgmap v13873: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:01.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:01 smithi022 ceph-mon[40250]: pgmap v13873: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:01 smithi122 ceph-mon[32255]: pgmap v13873: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:02.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:41:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:41:02.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:02.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:02.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:03 smithi022 ceph-mon[31360]: pgmap v13874: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2539802881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:03 smithi022 ceph-mon[40250]: pgmap v13874: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2539802881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:03 smithi122 ceph-mon[32255]: pgmap v13874: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2539802881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:05 smithi022 ceph-mon[31360]: pgmap v13875: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:05 smithi022 ceph-mon[40250]: pgmap v13875: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:05 smithi122 ceph-mon[32255]: pgmap v13875: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:06.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:41:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:41:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:41:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:07 smithi022 ceph-mon[31360]: pgmap v13876: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:07 smithi022 ceph-mon[40250]: pgmap v13876: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:07.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:07 smithi122 ceph-mon[32255]: pgmap v13876: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:08.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:08.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:08.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3166811516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3166811516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3166811516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:09 smithi022 ceph-mon[31360]: pgmap v13877: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:09 smithi022 ceph-mon[40250]: pgmap v13877: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:09 smithi122 ceph-mon[32255]: pgmap v13877: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:10] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:41:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:11 smithi022 ceph-mon[31360]: pgmap v13878: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:11 smithi022 ceph-mon[40250]: pgmap v13878: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:11 smithi122 ceph-mon[32255]: pgmap v13878: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:41:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:41:13.199 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:13.483 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:13 smithi022 ceph-mon[40250]: pgmap v13879: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:13 smithi022 ceph-mon[31360]: pgmap v13879: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:13 smithi122 ceph-mon[32255]: pgmap v13879: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3934785891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3934785891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3934785891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:15 smithi022 ceph-mon[31360]: pgmap v13880: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:15 smithi022 ceph-mon[40250]: pgmap v13880: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:15 smithi122 ceph-mon[32255]: pgmap v13880: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:16.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:41:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:41:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:41:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:17 smithi022 ceph-mon[31360]: pgmap v13881: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:17 smithi022 ceph-mon[40250]: pgmap v13881: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:17 smithi122 ceph-mon[32255]: pgmap v13881: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:18.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:18.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:18.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:18.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:19 smithi022 ceph-mon[31360]: pgmap v13882: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/73396796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:19 smithi022 ceph-mon[40250]: pgmap v13882: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/73396796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:19 smithi122 ceph-mon[32255]: pgmap v13882: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/73396796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:20] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:41:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:21 smithi022 ceph-mon[31360]: pgmap v13883: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:21 smithi022 ceph-mon[40250]: pgmap v13883: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:21 smithi122 ceph-mon[32255]: pgmap v13883: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:41:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:41:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:23 smithi022 ceph-mon[31360]: pgmap v13884: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:23 smithi022 ceph-mon[40250]: pgmap v13884: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:23 smithi122 ceph-mon[32255]: pgmap v13884: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:23.943 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:24.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:24.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:24.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2864704119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2864704119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2864704119' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:25 smithi022 ceph-mon[40250]: pgmap v13885: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:25 smithi022 ceph-mon[31360]: pgmap v13885: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:25 smithi122 ceph-mon[32255]: pgmap v13885: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:26.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:41:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:41:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:41:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:27 smithi022 ceph-mon[31360]: pgmap v13886: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:27 smithi022 ceph-mon[40250]: pgmap v13886: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:27 smithi122 ceph-mon[32255]: pgmap v13886: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:29.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:29.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:29.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:29.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:29.687 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:29 smithi022 ceph-mon[31360]: pgmap v13887: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:29.688 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:29 smithi022 ceph-mon[40250]: pgmap v13887: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:29 smithi122 ceph-mon[32255]: pgmap v13887: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:30] "GET /metrics HTTP/1.1" 200 202433 "" "Prometheus/2.18.1" 2021-12-16T01:41:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2632562194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2632562194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2632562194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:31 smithi122 ceph-mon[32255]: pgmap v13888: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:31 smithi022 ceph-mon[31360]: pgmap v13888: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:31 smithi022 ceph-mon[40250]: pgmap v13888: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:41:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:41:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:33 smithi122 ceph-mon[32255]: pgmap v13889: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:33 smithi022 ceph-mon[40250]: pgmap v13889: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:33 smithi022 ceph-mon[31360]: pgmap v13889: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:34.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:41:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:41:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:41:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:41:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:41:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:41:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:41:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:41:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:41:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:41:35.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:35.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:35.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:35 smithi122 ceph-mon[32255]: pgmap v13890: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3980205614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:35 smithi022 ceph-mon[31360]: pgmap v13890: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3980205614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:35 smithi022 ceph-mon[40250]: pgmap v13890: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3980205614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:36.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:41:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:41:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:37 smithi022 ceph-mon[31360]: pgmap v13891: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:41:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:37 smithi022 ceph-mon[40250]: pgmap v13891: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:37 smithi122 ceph-mon[32255]: pgmap v13891: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:39 smithi122 ceph-mon[32255]: pgmap v13892: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:39 smithi022 ceph-mon[31360]: pgmap v13892: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:39 smithi022 ceph-mon[40250]: pgmap v13892: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:40.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:40.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:40.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:40.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:40] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:41:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1526679968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1526679968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1526679968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:41 smithi122 ceph-mon[32255]: pgmap v13893: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:41 smithi022 ceph-mon[31360]: pgmap v13893: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:41 smithi022 ceph-mon[40250]: pgmap v13893: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:41:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:41:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[31360]: pgmap v13894: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:41:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:41:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:41:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:41:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[40250]: pgmap v13894: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:41:43.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:41:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:41:43.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:41:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:43 smithi122 ceph-mon[32255]: pgmap v13894: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:41:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:41:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:41:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:41:45.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:45 smithi022 ceph-mon[31360]: pgmap v13895: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:45 smithi022 ceph-mon[40250]: pgmap v13895: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:45.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:45.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:45.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:45 smithi122 ceph-mon[32255]: pgmap v13895: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/107721502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:46.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:41:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:41:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/107721502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/107721502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:41:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:47 smithi022 ceph-mon[31360]: pgmap v13896: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:47 smithi022 ceph-mon[40250]: pgmap v13896: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:47 smithi122 ceph-mon[32255]: pgmap v13896: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:49 smithi122 ceph-mon[32255]: pgmap v13897: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:49 smithi022 ceph-mon[31360]: pgmap v13897: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:49 smithi022 ceph-mon[40250]: pgmap v13897: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:50] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:41:50.801 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:51.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:51.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:51.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:51 smithi122 ceph-mon[32255]: pgmap v13898: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2043748616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:51 smithi022 ceph-mon[31360]: pgmap v13898: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2043748616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:51 smithi022 ceph-mon[40250]: pgmap v13898: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2043748616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:41:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:41:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:53 smithi122 ceph-mon[32255]: pgmap v13899: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:53 smithi022 ceph-mon[31360]: pgmap v13899: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:53 smithi022 ceph-mon[40250]: pgmap v13899: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:55 smithi122 ceph-mon[32255]: pgmap v13900: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:55 smithi022 ceph-mon[31360]: pgmap v13900: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:55 smithi022 ceph-mon[40250]: pgmap v13900: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:41:56.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:41:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:41:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:41:56.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:41:56.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:41:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:41:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:41:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:57 smithi022 ceph-mon[31360]: pgmap v13901: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2229797108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:57 smithi022 ceph-mon[40250]: pgmap v13901: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2229797108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:41:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:57 smithi122 ceph-mon[32255]: pgmap v13901: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:41:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2229797108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:41:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:41:59 smithi122 ceph-mon[32255]: pgmap v13902: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:41:59 smithi022 ceph-mon[31360]: pgmap v13902: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:41:59 smithi022 ceph-mon[40250]: pgmap v13902: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:00] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:42:01.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:01.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:01.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:01.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:01.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:01 smithi022 ceph-mon[31360]: pgmap v13903: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:01.910 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:01 smithi022 ceph-mon[40250]: pgmap v13903: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:01 smithi122 ceph-mon[32255]: pgmap v13903: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3799645189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:42:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:42:03.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3799645189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3799645189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:03 smithi122 ceph-mon[32255]: pgmap v13904: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:03 smithi022 ceph-mon[31360]: pgmap v13904: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:03 smithi022 ceph-mon[40250]: pgmap v13904: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:05 smithi122 ceph-mon[32255]: pgmap v13905: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:05 smithi022 ceph-mon[31360]: pgmap v13905: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:05 smithi022 ceph-mon[40250]: pgmap v13905: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:06.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:42:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:42:06.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:07.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:07.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:07.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:42:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:07 smithi022 ceph-mon[31360]: pgmap v13906: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1690628767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:07 smithi022 ceph-mon[40250]: pgmap v13906: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1690628767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:07 smithi122 ceph-mon[32255]: pgmap v13906: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1690628767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:09 smithi122 ceph-mon[32255]: pgmap v13907: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:09 smithi022 ceph-mon[31360]: pgmap v13907: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:09 smithi022 ceph-mon[40250]: pgmap v13907: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:10] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T01:42:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:11 smithi122 ceph-mon[32255]: pgmap v13908: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:11 smithi022 ceph-mon[31360]: pgmap v13908: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:11 smithi022 ceph-mon[40250]: pgmap v13908: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:12.282 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:12.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:12.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:12.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:42:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:42:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:13 smithi122 ceph-mon[32255]: pgmap v13909: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2579947592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:13 smithi022 ceph-mon[31360]: pgmap v13909: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2579947592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:13 smithi022 ceph-mon[40250]: pgmap v13909: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2579947592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:15 smithi122 ceph-mon[32255]: pgmap v13910: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:15 smithi022 ceph-mon[31360]: pgmap v13910: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:15 smithi022 ceph-mon[40250]: pgmap v13910: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:16.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:42:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:42:17.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:17.656 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:42:17.656 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:17.656 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:17 smithi022 ceph-mon[31360]: pgmap v13911: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:17.657 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:17.657 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:17 smithi022 ceph-mon[40250]: pgmap v13911: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:17 smithi122 ceph-mon[32255]: pgmap v13911: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:18.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2929928956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2929928956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2929928956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:19 smithi022 ceph-mon[31360]: pgmap v13912: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:19 smithi022 ceph-mon[40250]: pgmap v13912: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:19 smithi122 ceph-mon[32255]: pgmap v13912: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:20] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:42:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:21 smithi122 ceph-mon[32255]: pgmap v13913: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:21 smithi022 ceph-mon[31360]: pgmap v13913: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:21 smithi022 ceph-mon[40250]: pgmap v13913: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:42:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:42:23.026 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:23.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:23.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:23.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:23 smithi022 ceph-mon[31360]: pgmap v13914: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1182153482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:23 smithi022 ceph-mon[40250]: pgmap v13914: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1182153482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:23 smithi122 ceph-mon[32255]: pgmap v13914: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1182153482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:25 smithi122 ceph-mon[32255]: pgmap v13915: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:25 smithi022 ceph-mon[31360]: pgmap v13915: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:25 smithi022 ceph-mon[40250]: pgmap v13915: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:26.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:42:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:42:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:42:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:27 smithi022 ceph-mon[31360]: pgmap v13916: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:27 smithi022 ceph-mon[40250]: pgmap v13916: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:27 smithi122 ceph-mon[32255]: pgmap v13916: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:28.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:28.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:28.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:28.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:29.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:29 smithi122 ceph-mon[32255]: pgmap v13917: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:29.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/245227680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:29 smithi022 ceph-mon[31360]: pgmap v13917: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/245227680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:29 smithi022 ceph-mon[40250]: pgmap v13917: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/245227680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:30] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:42:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:31 smithi122 ceph-mon[32255]: pgmap v13918: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:32.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:31 smithi022 ceph-mon[31360]: pgmap v13918: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:32.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:31 smithi022 ceph-mon[40250]: pgmap v13918: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:42:32.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:42:33.775 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:33 smithi122 ceph-mon[32255]: pgmap v13919: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:33 smithi022 ceph-mon[31360]: pgmap v13919: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:33 smithi022 ceph-mon[40250]: pgmap v13919: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:34.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:34.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:34.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2555640691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:42:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:42:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:42:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:42:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2555640691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:42:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:42:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:42:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:42:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2555640691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:35 smithi122 ceph-mon[32255]: pgmap v13920: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:35 smithi022 ceph-mon[31360]: pgmap v13920: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:35 smithi022 ceph-mon[40250]: pgmap v13920: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:36.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:42:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:42:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:42:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:37 smithi022 ceph-mon[31360]: pgmap v13921: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:37 smithi022 ceph-mon[40250]: pgmap v13921: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:37 smithi122 ceph-mon[32255]: pgmap v13921: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:39 smithi022 ceph-mon[31360]: pgmap v13922: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3583613668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:39 smithi022 ceph-mon[40250]: pgmap v13922: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3583613668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:39 smithi122 ceph-mon[32255]: pgmap v13922: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3583613668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:42:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:41 smithi122 ceph-mon[32255]: pgmap v13923: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:41 smithi022 ceph-mon[40250]: pgmap v13923: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:41 smithi022 ceph-mon[31360]: pgmap v13923: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:42:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:42:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:43 smithi122 ceph-mon[32255]: pgmap v13924: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:42:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:42:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:42:43.733 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[31360]: pgmap v13924: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:43.734 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:42:43.734 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:42:43.734 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:42:43.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[40250]: pgmap v13924: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:43.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:42:43.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:42:43.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:42:44.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:44.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:44.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:44.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:42:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:45 smithi022 ceph-mon[31360]: pgmap v13925: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/344572836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:45 smithi022 ceph-mon[40250]: pgmap v13925: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/344572836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:45 smithi122 ceph-mon[32255]: pgmap v13925: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/344572836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:46.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:42:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:42:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:47 smithi022 ceph-mon[31360]: pgmap v13926: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:42:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:47 smithi022 ceph-mon[40250]: pgmap v13926: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:47 smithi122 ceph-mon[32255]: pgmap v13926: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:49.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:49.882 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:49 smithi022 ceph-mon[40250]: pgmap v13927: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:49.882 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:49 smithi022 ceph-mon[31360]: pgmap v13927: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:49 smithi122 ceph-mon[32255]: pgmap v13927: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:50.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:50.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:50.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:50] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:42:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4082215008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4082215008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4082215008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:51 smithi122 ceph-mon[32255]: pgmap v13928: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:51 smithi022 ceph-mon[31360]: pgmap v13928: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:51 smithi022 ceph-mon[40250]: pgmap v13928: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:42:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:42:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:53 smithi122 ceph-mon[32255]: pgmap v13929: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:53 smithi022 ceph-mon[31360]: pgmap v13929: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:53 smithi022 ceph-mon[40250]: pgmap v13929: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:55.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:42:55.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:42:55.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:42:55.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:42:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:55 smithi122 ceph-mon[32255]: pgmap v13930: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:55 smithi022 ceph-mon[31360]: pgmap v13930: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:55 smithi022 ceph-mon[40250]: pgmap v13930: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:42:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3745027855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:56.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:42:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:42:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3745027855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3745027855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:42:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:42:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:57 smithi022 ceph-mon[31360]: pgmap v13931: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:57 smithi022 ceph-mon[40250]: pgmap v13931: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:42:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:57 smithi122 ceph-mon[32255]: pgmap v13931: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:42:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:42:59 smithi122 ceph-mon[32255]: pgmap v13932: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:42:59 smithi022 ceph-mon[31360]: pgmap v13932: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:42:59 smithi022 ceph-mon[40250]: pgmap v13932: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:00] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:43:00.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:00.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:00.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:00.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:01 smithi122 ceph-mon[32255]: pgmap v13933: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/781983647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:01 smithi022 ceph-mon[31360]: pgmap v13933: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/781983647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:01 smithi022 ceph-mon[40250]: pgmap v13933: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/781983647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:43:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:43:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:03 smithi122 ceph-mon[32255]: pgmap v13934: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:03 smithi022 ceph-mon[31360]: pgmap v13934: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:03 smithi022 ceph-mon[40250]: pgmap v13934: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:05 smithi122 ceph-mon[32255]: pgmap v13935: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:05.991 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:05.992 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:05 smithi022 ceph-mon[31360]: pgmap v13935: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:05.992 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:05 smithi022 ceph-mon[40250]: pgmap v13935: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:06.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:06.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:06.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1965324438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:06.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:43:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:43:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1965324438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1965324438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:43:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:07 smithi022 ceph-mon[31360]: pgmap v13936: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:07 smithi022 ceph-mon[40250]: pgmap v13936: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:07 smithi122 ceph-mon[32255]: pgmap v13936: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:09 smithi122 ceph-mon[32255]: pgmap v13937: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:09 smithi022 ceph-mon[31360]: pgmap v13937: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:09 smithi022 ceph-mon[40250]: pgmap v13937: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:10] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:43:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:11.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:11.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:11.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:11.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:11 smithi022 ceph-mon[31360]: pgmap v13938: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:11.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:11 smithi022 ceph-mon[40250]: pgmap v13938: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:11 smithi122 ceph-mon[32255]: pgmap v13938: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:12.592 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:43:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:43:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1949220531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1949220531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1949220531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:13 smithi122 ceph-mon[32255]: pgmap v13939: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:13 smithi022 ceph-mon[31360]: pgmap v13939: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:13 smithi022 ceph-mon[40250]: pgmap v13939: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:15 smithi122 ceph-mon[32255]: pgmap v13940: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:15 smithi022 ceph-mon[31360]: pgmap v13940: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:15 smithi022 ceph-mon[40250]: pgmap v13940: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:16.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:43:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:43:16.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:17.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:17.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:17.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:43:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:17 smithi022 ceph-mon[31360]: pgmap v13941: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/872289924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:17 smithi022 ceph-mon[40250]: pgmap v13941: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/872289924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:17 smithi122 ceph-mon[32255]: pgmap v13941: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/872289924' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:19 smithi122 ceph-mon[32255]: pgmap v13942: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:19 smithi022 ceph-mon[31360]: pgmap v13942: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:19 smithi022 ceph-mon[40250]: pgmap v13942: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:20] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:43:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:21 smithi122 ceph-mon[32255]: pgmap v13943: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:21 smithi022 ceph-mon[31360]: pgmap v13943: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:21 smithi022 ceph-mon[40250]: pgmap v13943: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:22.106 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:22.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:22.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:22.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:22.570 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:43:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:43:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3554815538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3554815538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3554815538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:23 smithi122 ceph-mon[32255]: pgmap v13944: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:23 smithi022 ceph-mon[31360]: pgmap v13944: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:23 smithi022 ceph-mon[40250]: pgmap v13944: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:25 smithi122 ceph-mon[32255]: pgmap v13945: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:25 smithi022 ceph-mon[31360]: pgmap v13945: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:25 smithi022 ceph-mon[40250]: pgmap v13945: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:26.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:43:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:43:27.475 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:43:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:27 smithi022 ceph-mon[31360]: pgmap v13946: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:27 smithi022 ceph-mon[40250]: pgmap v13946: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:27.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:27.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:27.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:27 smithi122 ceph-mon[32255]: pgmap v13946: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2901154560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2901154560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2901154560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:29 smithi022 ceph-mon[31360]: pgmap v13947: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:29 smithi022 ceph-mon[40250]: pgmap v13947: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:29 smithi122 ceph-mon[32255]: pgmap v13947: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:30] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:43:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:31 smithi122 ceph-mon[32255]: pgmap v13948: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:31 smithi022 ceph-mon[31360]: pgmap v13948: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:31 smithi022 ceph-mon[40250]: pgmap v13948: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:43:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:43:32.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:33.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:33.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:33 smithi122 ceph-mon[32255]: pgmap v13949: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/93065795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:33 smithi022 ceph-mon[31360]: pgmap v13949: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/93065795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:33 smithi022 ceph-mon[40250]: pgmap v13949: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/93065795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:43:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:43:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:43:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:43:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:43:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:43:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:43:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:43:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:43:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:43:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:35 smithi122 ceph-mon[32255]: pgmap v13950: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:35 smithi022 ceph-mon[31360]: pgmap v13950: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:35 smithi022 ceph-mon[40250]: pgmap v13950: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:36.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:43:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:43:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:37 smithi022 ceph-mon[31360]: pgmap v13951: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:43:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:37 smithi022 ceph-mon[40250]: pgmap v13951: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:37 smithi122 ceph-mon[32255]: pgmap v13951: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:38.214 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:38.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:38.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:38.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:39 smithi122 ceph-mon[32255]: pgmap v13952: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2891125248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:39 smithi022 ceph-mon[31360]: pgmap v13952: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2891125248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:39 smithi022 ceph-mon[40250]: pgmap v13952: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2891125248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:40] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:43:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:41 smithi122 ceph-mon[32255]: pgmap v13953: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:41 smithi022 ceph-mon[31360]: pgmap v13953: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:41 smithi022 ceph-mon[40250]: pgmap v13953: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:43:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:43:43.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:43 smithi122 ceph-mon[32255]: pgmap v13954: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:43.958 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:43.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:43.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:43.959 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:43 smithi022 ceph-mon[31360]: pgmap v13954: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:43.960 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:43 smithi022 ceph-mon[40250]: pgmap v13954: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:44.588 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/784758866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:44.589 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:43:44.589 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:43:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:43:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/784758866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:43:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:43:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:43:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/784758866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:43:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:43:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:43:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:45 smithi122 ceph-mon[32255]: pgmap v13955: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:43:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:45 smithi022 ceph-mon[31360]: pgmap v13955: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:43:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:45 smithi022 ceph-mon[40250]: pgmap v13955: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:43:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:43:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:43:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:43:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:47 smithi022 ceph-mon[31360]: pgmap v13956: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:47 smithi022 ceph-mon[40250]: pgmap v13956: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:47 smithi122 ceph-mon[32255]: pgmap v13956: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:48.960 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:49.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:49.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:49.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:49 smithi022 ceph-mon[31360]: pgmap v13957: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2494616679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:49 smithi022 ceph-mon[40250]: pgmap v13957: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2494616679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:49 smithi122 ceph-mon[32255]: pgmap v13957: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2494616679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:50] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:43:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:51 smithi122 ceph-mon[32255]: pgmap v13958: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:51 smithi022 ceph-mon[31360]: pgmap v13958: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:51 smithi022 ceph-mon[40250]: pgmap v13958: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:43:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:43:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:53 smithi122 ceph-mon[32255]: pgmap v13959: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:53 smithi022 ceph-mon[31360]: pgmap v13959: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:53 smithi022 ceph-mon[40250]: pgmap v13959: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:54.327 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:54.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:43:54.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:43:54.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:43:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:55 smithi122 ceph-mon[32255]: pgmap v13960: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2299402491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:55 smithi022 ceph-mon[31360]: pgmap v13960: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2299402491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:55 smithi022 ceph-mon[40250]: pgmap v13960: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:43:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2299402491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:43:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:43:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:43:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:57 smithi022 ceph-mon[31360]: pgmap v13961: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:43:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:57 smithi022 ceph-mon[40250]: pgmap v13961: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:43:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:57 smithi122 ceph-mon[32255]: pgmap v13961: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:43:59.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:43:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:43:59 smithi122 ceph-mon[32255]: pgmap v13962: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:43:59 smithi022 ceph-mon[31360]: pgmap v13962: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:43:59 smithi022 ceph-mon[40250]: pgmap v13962: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:00.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:00.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:00.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:00] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:44:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2841211587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2841211587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2841211587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:01 smithi122 ceph-mon[32255]: pgmap v13963: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:01 smithi022 ceph-mon[31360]: pgmap v13963: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:01 smithi022 ceph-mon[40250]: pgmap v13963: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:44:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:44:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:03 smithi122 ceph-mon[32255]: pgmap v13964: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:03 smithi022 ceph-mon[31360]: pgmap v13964: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:03 smithi022 ceph-mon[40250]: pgmap v13964: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:05.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:05.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:05.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:05.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:05 smithi022 ceph-mon[31360]: pgmap v13965: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/217458183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:05 smithi022 ceph-mon[40250]: pgmap v13965: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/217458183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:05 smithi122 ceph-mon[32255]: pgmap v13965: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/217458183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:44:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:44:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:44:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:07 smithi022 ceph-mon[31360]: pgmap v13966: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:07 smithi022 ceph-mon[40250]: pgmap v13966: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:07 smithi122 ceph-mon[32255]: pgmap v13966: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:09 smithi122 ceph-mon[32255]: pgmap v13967: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:09 smithi022 ceph-mon[31360]: pgmap v13967: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:09 smithi022 ceph-mon[40250]: pgmap v13967: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:10] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:44:10.436 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:11 smithi122 ceph-mon[32255]: pgmap v13968: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/536554861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:11 smithi022 ceph-mon[31360]: pgmap v13968: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/536554861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:11 smithi022 ceph-mon[40250]: pgmap v13968: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/536554861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:44:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:44:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:13 smithi122 ceph-mon[32255]: pgmap v13969: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:13 smithi022 ceph-mon[31360]: pgmap v13969: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:13 smithi022 ceph-mon[40250]: pgmap v13969: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:15.811 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:15 smithi122 ceph-mon[32255]: pgmap v13970: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:15 smithi022 ceph-mon[40250]: pgmap v13970: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:15 smithi022 ceph-mon[31360]: pgmap v13970: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3744871870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:44:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:44:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3744871870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3744871870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:44:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:17 smithi022 ceph-mon[31360]: pgmap v13971: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:17 smithi022 ceph-mon[40250]: pgmap v13971: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:17 smithi122 ceph-mon[32255]: pgmap v13971: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:19 smithi122 ceph-mon[32255]: pgmap v13972: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:19 smithi022 ceph-mon[31360]: pgmap v13972: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:19 smithi022 ceph-mon[40250]: pgmap v13972: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:20] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:44:21.180 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:21.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:21.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:21.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:21 smithi122 ceph-mon[32255]: pgmap v13973: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/102980106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:21 smithi022 ceph-mon[31360]: pgmap v13973: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/102980106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:21 smithi022 ceph-mon[40250]: pgmap v13973: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/102980106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:44:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:44:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:23 smithi122 ceph-mon[32255]: pgmap v13974: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:23 smithi022 ceph-mon[31360]: pgmap v13974: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:23 smithi022 ceph-mon[40250]: pgmap v13974: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:25 smithi122 ceph-mon[32255]: pgmap v13975: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:25 smithi022 ceph-mon[31360]: pgmap v13975: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:25 smithi022 ceph-mon[40250]: pgmap v13975: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:26.549 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:26.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:44:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:44:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:44:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:27 smithi022 ceph-mon[31360]: pgmap v13976: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/495539608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:27 smithi022 ceph-mon[40250]: pgmap v13976: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/495539608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:27 smithi122 ceph-mon[32255]: pgmap v13976: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/495539608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:29 smithi122 ceph-mon[32255]: pgmap v13977: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:29 smithi022 ceph-mon[31360]: pgmap v13977: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:29 smithi022 ceph-mon[40250]: pgmap v13977: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:30] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:44:31.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:31.914 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:31 smithi022 ceph-mon[31360]: pgmap v13978: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:31.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:31 smithi022 ceph-mon[40250]: pgmap v13978: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:31 smithi122 ceph-mon[32255]: pgmap v13978: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:32.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:32.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:32.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:32.592 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:44:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:44:32.595 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2888354938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2888354938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2888354938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:33 smithi122 ceph-mon[32255]: pgmap v13979: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:33 smithi022 ceph-mon[31360]: pgmap v13979: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:33 smithi022 ceph-mon[40250]: pgmap v13979: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:44:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:44:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:44:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:44:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:44:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:44:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:44:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:44:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:44:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:44:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:44:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:44:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:35 smithi122 ceph-mon[32255]: pgmap v13980: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:35 smithi022 ceph-mon[31360]: pgmap v13980: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:35 smithi022 ceph-mon[40250]: pgmap v13980: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:36.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:44:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:44:37.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:37.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:37.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:37.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:37.656 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:44:37.657 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:37.657 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:37 smithi022 ceph-mon[31360]: pgmap v13981: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:37.657 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:37.658 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:37 smithi022 ceph-mon[40250]: pgmap v13981: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:37 smithi122 ceph-mon[32255]: pgmap v13981: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/874289632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/874289632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/874289632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:39 smithi022 ceph-mon[31360]: pgmap v13982: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:39 smithi022 ceph-mon[40250]: pgmap v13982: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:39 smithi122 ceph-mon[32255]: pgmap v13982: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:40] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:44:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:41 smithi122 ceph-mon[32255]: pgmap v13983: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:41 smithi022 ceph-mon[31360]: pgmap v13983: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:41 smithi022 ceph-mon[40250]: pgmap v13983: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:42.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:44:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:44:43.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:43.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:43.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:43 smithi122 ceph-mon[32255]: pgmap v13984: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4117709568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:43 smithi022 ceph-mon[31360]: pgmap v13984: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4117709568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:43 smithi022 ceph-mon[40250]: pgmap v13984: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4117709568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:45 smithi122 ceph-mon[32255]: pgmap v13985: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:44:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:44:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:44:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:44:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[31360]: pgmap v13985: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:44:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:44:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:44:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:44:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[40250]: pgmap v13985: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:44:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:44:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:44:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:44:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:44:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:44:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:44:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:47 smithi022 ceph-mon[31360]: pgmap v13986: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:47 smithi022 ceph-mon[40250]: pgmap v13986: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:47 smithi122 ceph-mon[32255]: pgmap v13986: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:48.029 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:48.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:48.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:48.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3830162656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3830162656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3830162656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:49 smithi022 ceph-mon[31360]: pgmap v13987: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:49 smithi022 ceph-mon[40250]: pgmap v13987: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:49 smithi122 ceph-mon[32255]: pgmap v13987: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:50] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:44:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:51 smithi122 ceph-mon[32255]: pgmap v13988: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:51 smithi022 ceph-mon[31360]: pgmap v13988: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:51 smithi022 ceph-mon[40250]: pgmap v13988: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:44:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:44:53.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:53.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:53.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:53.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:53.773 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:53 smithi022 ceph-mon[31360]: pgmap v13989: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:53.773 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:53 smithi022 ceph-mon[40250]: pgmap v13989: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:53 smithi122 ceph-mon[32255]: pgmap v13989: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3513562452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3513562452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3513562452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:44:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:55 smithi122 ceph-mon[32255]: pgmap v13990: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:55 smithi022 ceph-mon[31360]: pgmap v13990: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:55 smithi022 ceph-mon[40250]: pgmap v13990: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:44:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:44:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:44:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:57 smithi022 ceph-mon[31360]: pgmap v13991: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:57 smithi022 ceph-mon[40250]: pgmap v13991: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:44:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:57 smithi122 ceph-mon[32255]: pgmap v13991: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:44:58.774 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:44:59.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:44:59.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:44:59.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:44:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:59 smithi122 ceph-mon[32255]: pgmap v13992: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:44:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:44:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2316825252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:59 smithi022 ceph-mon[31360]: pgmap v13992: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:44:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2316825252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:59 smithi022 ceph-mon[40250]: pgmap v13992: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:44:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2316825252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:00] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:45:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:01 smithi122 ceph-mon[32255]: pgmap v13993: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:01 smithi022 ceph-mon[31360]: pgmap v13993: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:01 smithi022 ceph-mon[40250]: pgmap v13993: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:45:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:45:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:03 smithi122 ceph-mon[32255]: pgmap v13994: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:03 smithi022 ceph-mon[31360]: pgmap v13994: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:03 smithi022 ceph-mon[40250]: pgmap v13994: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:04.144 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:04.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:04.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:04.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2312950376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2312950376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2312950376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:05 smithi122 ceph-mon[32255]: pgmap v13995: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:05 smithi022 ceph-mon[31360]: pgmap v13995: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:05 smithi022 ceph-mon[40250]: pgmap v13995: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:45:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:45:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:07 smithi022 ceph-mon[31360]: pgmap v13996: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:45:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:07 smithi022 ceph-mon[40250]: pgmap v13996: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:07 smithi122 ceph-mon[32255]: pgmap v13996: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:09.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:09 smithi022 ceph-mon[31360]: pgmap v13997: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:09 smithi022 ceph-mon[40250]: pgmap v13997: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:09 smithi122 ceph-mon[32255]: pgmap v13997: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:10] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:45:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221865762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221865762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221865762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:11 smithi122 ceph-mon[32255]: pgmap v13998: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:11 smithi022 ceph-mon[31360]: pgmap v13998: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:11 smithi022 ceph-mon[40250]: pgmap v13998: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:45:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:45:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:13 smithi122 ceph-mon[32255]: pgmap v13999: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:13 smithi022 ceph-mon[31360]: pgmap v13999: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:13 smithi022 ceph-mon[40250]: pgmap v13999: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:14.886 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:15.251 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:15.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:15.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:15 smithi122 ceph-mon[32255]: pgmap v14000: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1057085994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:15 smithi022 ceph-mon[31360]: pgmap v14000: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1057085994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:15 smithi022 ceph-mon[40250]: pgmap v14000: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1057085994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:45:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:45:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:17 smithi022 ceph-mon[31360]: pgmap v14001: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:45:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:17 smithi022 ceph-mon[40250]: pgmap v14001: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:17 smithi122 ceph-mon[32255]: pgmap v14001: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:19 smithi122 ceph-mon[32255]: pgmap v14002: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:19 smithi022 ceph-mon[31360]: pgmap v14002: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:19 smithi022 ceph-mon[40250]: pgmap v14002: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:20.253 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:45:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:21 smithi122 ceph-mon[32255]: pgmap v14003: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1439611293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:21 smithi022 ceph-mon[31360]: pgmap v14003: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1439611293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:21 smithi022 ceph-mon[40250]: pgmap v14003: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1439611293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:45:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:45:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:23 smithi122 ceph-mon[32255]: pgmap v14004: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:23 smithi022 ceph-mon[31360]: pgmap v14004: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:23 smithi022 ceph-mon[40250]: pgmap v14004: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:25.625 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:25 smithi122 ceph-mon[32255]: pgmap v14005: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:25.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:25.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:25.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:25.994 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:25 smithi022 ceph-mon[31360]: pgmap v14005: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:25.994 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:25 smithi022 ceph-mon[40250]: pgmap v14005: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3612604402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:45:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:45:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3612604402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3612604402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:45:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:27 smithi022 ceph-mon[31360]: pgmap v14006: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:27 smithi022 ceph-mon[40250]: pgmap v14006: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:27 smithi122 ceph-mon[32255]: pgmap v14006: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:29 smithi122 ceph-mon[32255]: pgmap v14007: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:29 smithi022 ceph-mon[31360]: pgmap v14007: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:29 smithi022 ceph-mon[40250]: pgmap v14007: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:45:30.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:31.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:31.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:31.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:31 smithi022 ceph-mon[31360]: pgmap v14008: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2488042803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:31 smithi022 ceph-mon[40250]: pgmap v14008: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2488042803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:31 smithi122 ceph-mon[32255]: pgmap v14008: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2488042803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:45:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:45:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:33 smithi122 ceph-mon[32255]: pgmap v14009: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:33 smithi022 ceph-mon[31360]: pgmap v14009: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:33 smithi022 ceph-mon[40250]: pgmap v14009: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:45:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:45:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:45:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:45:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:45:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:45:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:45:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:45:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:45:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:45:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:45:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:45:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:35 smithi122 ceph-mon[32255]: pgmap v14010: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:35 smithi022 ceph-mon[31360]: pgmap v14010: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:35 smithi022 ceph-mon[40250]: pgmap v14010: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:36.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:36.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:36.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:36.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:36.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:45:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:45:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:45:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:37 smithi022 ceph-mon[31360]: pgmap v14011: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2080527586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:37 smithi022 ceph-mon[40250]: pgmap v14011: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2080527586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:37 smithi122 ceph-mon[32255]: pgmap v14011: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2080527586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:39 smithi122 ceph-mon[32255]: pgmap v14012: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:39 smithi022 ceph-mon[31360]: pgmap v14012: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:39 smithi022 ceph-mon[40250]: pgmap v14012: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:40] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:45:41.740 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:41.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:41 smithi122 ceph-mon[32255]: pgmap v14013: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:41 smithi022 ceph-mon[31360]: pgmap v14013: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:41 smithi022 ceph-mon[40250]: pgmap v14013: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:42.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:42.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:42.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:42.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:45:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:45:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4024884735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4024884735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4024884735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:43 smithi122 ceph-mon[32255]: pgmap v14014: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:43 smithi022 ceph-mon[31360]: pgmap v14014: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:43 smithi022 ceph-mon[40250]: pgmap v14014: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:45 smithi122 ceph-mon[32255]: pgmap v14015: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:45:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:45:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:45:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:45:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[31360]: pgmap v14015: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:45:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:45:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:45:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:45:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[40250]: pgmap v14015: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:45:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:45:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:45:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:45:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:45:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:45:47.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:47.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:47.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:47.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:45:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:47 smithi022 ceph-mon[31360]: pgmap v14016: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3694234977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:47 smithi022 ceph-mon[40250]: pgmap v14016: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3694234977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:47 smithi122 ceph-mon[32255]: pgmap v14016: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:45:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3694234977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:49 smithi122 ceph-mon[32255]: pgmap v14017: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:49 smithi022 ceph-mon[31360]: pgmap v14017: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:49 smithi022 ceph-mon[40250]: pgmap v14017: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:45:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:51 smithi122 ceph-mon[32255]: pgmap v14018: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:51 smithi022 ceph-mon[31360]: pgmap v14018: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:51 smithi022 ceph-mon[40250]: pgmap v14018: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:52.483 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:45:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:45:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:52.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:53 smithi122 ceph-mon[32255]: pgmap v14019: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1537416268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:53 smithi022 ceph-mon[31360]: pgmap v14019: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1537416268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:53 smithi022 ceph-mon[40250]: pgmap v14019: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1537416268' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:55 smithi122 ceph-mon[32255]: pgmap v14020: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:55 smithi022 ceph-mon[31360]: pgmap v14020: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:55 smithi022 ceph-mon[40250]: pgmap v14020: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:45:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:45:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:45:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:57 smithi022 ceph-mon[31360]: pgmap v14021: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:45:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:57 smithi022 ceph-mon[40250]: pgmap v14021: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:57.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:45:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:45:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:57 smithi122 ceph-mon[32255]: pgmap v14021: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:45:58.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:45:58.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:45:58.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:45:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2471735486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2471735486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2471735486' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:45:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:45:59 smithi122 ceph-mon[32255]: pgmap v14022: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:45:59 smithi022 ceph-mon[31360]: pgmap v14022: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:45:59 smithi022 ceph-mon[40250]: pgmap v14022: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:46:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:01 smithi122 ceph-mon[32255]: pgmap v14023: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:01 smithi022 ceph-mon[31360]: pgmap v14023: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:01 smithi022 ceph-mon[40250]: pgmap v14023: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:46:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:46:03.226 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:03.596 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:03.596 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:03.596 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:03 smithi122 ceph-mon[32255]: pgmap v14024: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1143394731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:03 smithi022 ceph-mon[31360]: pgmap v14024: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1143394731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:03 smithi022 ceph-mon[40250]: pgmap v14024: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1143394731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:05 smithi122 ceph-mon[32255]: pgmap v14025: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:05 smithi022 ceph-mon[31360]: pgmap v14025: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:05 smithi022 ceph-mon[40250]: pgmap v14025: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:06.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:46:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:46:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:46:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:07 smithi022 ceph-mon[31360]: pgmap v14026: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:07 smithi022 ceph-mon[40250]: pgmap v14026: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:07 smithi122 ceph-mon[32255]: pgmap v14026: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:08.598 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:08.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:08.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:08.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:09 smithi122 ceph-mon[32255]: pgmap v14027: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4214994998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:09 smithi022 ceph-mon[31360]: pgmap v14027: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4214994998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:09 smithi022 ceph-mon[40250]: pgmap v14027: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4214994998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:10] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:46:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:11 smithi122 ceph-mon[32255]: pgmap v14028: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:11 smithi022 ceph-mon[31360]: pgmap v14028: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:11 smithi022 ceph-mon[40250]: pgmap v14028: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:46:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:46:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:13 smithi122 ceph-mon[32255]: pgmap v14029: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:13.969 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:13 smithi022 ceph-mon[31360]: pgmap v14029: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:13.970 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:13 smithi022 ceph-mon[40250]: pgmap v14029: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:13.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:14.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:14.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:14.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1869072129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1869072129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:15.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1869072129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:15 smithi122 ceph-mon[32255]: pgmap v14030: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:15 smithi022 ceph-mon[31360]: pgmap v14030: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:15 smithi022 ceph-mon[40250]: pgmap v14030: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:46:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:46:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:17 smithi022 ceph-mon[31360]: pgmap v14031: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:46:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:17 smithi022 ceph-mon[40250]: pgmap v14031: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:17 smithi122 ceph-mon[32255]: pgmap v14031: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:19.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:19.617 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:19 smithi022 ceph-mon[40250]: pgmap v14032: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:19.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:19.709 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:19.710 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:19 smithi122 ceph-mon[32255]: pgmap v14032: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:19 smithi022 ceph-mon[31360]: pgmap v14032: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:20] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:46:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/954071678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/954071678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/954071678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:21 smithi122 ceph-mon[32255]: pgmap v14033: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:21 smithi022 ceph-mon[31360]: pgmap v14033: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:21 smithi022 ceph-mon[40250]: pgmap v14033: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:46:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:46:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:23 smithi122 ceph-mon[32255]: pgmap v14034: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:23 smithi022 ceph-mon[31360]: pgmap v14034: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:23 smithi022 ceph-mon[40250]: pgmap v14034: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:24.711 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:25.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:25.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:25.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:25 smithi122 ceph-mon[32255]: pgmap v14035: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2135628990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:25 smithi022 ceph-mon[31360]: pgmap v14035: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2135628990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:25 smithi022 ceph-mon[40250]: pgmap v14035: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2135628990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:46:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:46:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:46:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:27 smithi022 ceph-mon[31360]: pgmap v14036: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:27 smithi022 ceph-mon[40250]: pgmap v14036: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:27 smithi122 ceph-mon[32255]: pgmap v14036: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:29 smithi122 ceph-mon[32255]: pgmap v14037: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:29 smithi022 ceph-mon[31360]: pgmap v14037: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:29 smithi022 ceph-mon[40250]: pgmap v14037: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:30.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:30] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:46:30.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:30.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:30.445 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2449730757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2449730757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2449730757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:31 smithi122 ceph-mon[32255]: pgmap v14038: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:31 smithi022 ceph-mon[31360]: pgmap v14038: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:31 smithi022 ceph-mon[40250]: pgmap v14038: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:46:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:46:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:33 smithi122 ceph-mon[32255]: pgmap v14039: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:33 smithi022 ceph-mon[31360]: pgmap v14039: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:33 smithi022 ceph-mon[40250]: pgmap v14039: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:46:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:46:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:46:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:46:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:46:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:46:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:46:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:46:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:46:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:46:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:46:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:46:35.447 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:35 smithi022 ceph-mon[31360]: pgmap v14040: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:35 smithi022 ceph-mon[40250]: pgmap v14040: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:35.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:35.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:35.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:35 smithi122 ceph-mon[32255]: pgmap v14040: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1185507186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:36.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:46:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:46:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1185507186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1185507186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:46:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:37 smithi022 ceph-mon[31360]: pgmap v14041: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:37 smithi022 ceph-mon[40250]: pgmap v14041: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:37 smithi122 ceph-mon[32255]: pgmap v14041: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:39 smithi122 ceph-mon[32255]: pgmap v14042: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:39 smithi022 ceph-mon[31360]: pgmap v14042: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:39 smithi022 ceph-mon[40250]: pgmap v14042: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:46:40.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:41.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:41.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:41.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:41.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:41 smithi122 ceph-mon[32255]: pgmap v14043: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:41.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1846094170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:41 smithi022 ceph-mon[31360]: pgmap v14043: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1846094170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:41 smithi022 ceph-mon[40250]: pgmap v14043: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1846094170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:46:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:46:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:43 smithi122 ceph-mon[32255]: pgmap v14044: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:43 smithi022 ceph-mon[31360]: pgmap v14044: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:43 smithi022 ceph-mon[40250]: pgmap v14044: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:45 smithi122 ceph-mon[32255]: pgmap v14045: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:46:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:46:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:46:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[31360]: pgmap v14045: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:46:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:46:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:46:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[40250]: pgmap v14045: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:46:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:46:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:46:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:46.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:46.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:46.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:46:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:46:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:46:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/264214952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:47 smithi022 ceph-mon[31360]: pgmap v14046: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/264214952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:47 smithi022 ceph-mon[40250]: pgmap v14046: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/264214952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:47 smithi122 ceph-mon[32255]: pgmap v14046: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:50.397 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:50] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:46:50.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: pgmap v14047: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:50.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.398 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.399 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.399 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.399 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.399 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.400 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.400 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.400 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.400 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.400 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: pgmap v14047: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: pgmap v14047: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:50.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:50.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:50.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3003M 2021-12-16T01:46:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:51.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3003M 2021-12-16T01:46:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3003M 2021-12-16T01:46:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:46:51.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:51.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:46:51.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:51.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:51.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:51.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:52 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3230M 2021-12-16T01:46:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:52 smithi022 ceph-mon[31360]: pgmap v14048: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4276373034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:52 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3230M 2021-12-16T01:46:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:52 smithi022 ceph-mon[40250]: pgmap v14048: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4276373034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:46:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:46:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:52 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3230M 2021-12-16T01:46:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:52 smithi122 ceph-mon[32255]: pgmap v14048: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4276373034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:54 smithi122 ceph-mon[32255]: pgmap v14049: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:54 smithi022 ceph-mon[31360]: pgmap v14049: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:54 smithi022 ceph-mon[40250]: pgmap v14049: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:56 smithi122 ceph-mon[32255]: pgmap v14050: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:56 smithi022 ceph-mon[31360]: pgmap v14050: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:56 smithi022 ceph-mon[40250]: pgmap v14050: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:46:56.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:46:56.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:46:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:46:57.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:46:57.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:46:57.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:46:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:46:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:57 smithi022 ceph-mon[31360]: pgmap v14051: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1733251712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:57 smithi022 ceph-mon[40250]: pgmap v14051: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1733251712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:57 smithi122 ceph-mon[32255]: pgmap v14051: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:46:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:46:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1733251712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:46:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:46:59 smithi122 ceph-mon[32255]: pgmap v14052: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:46:59 smithi022 ceph-mon[31360]: pgmap v14052: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:46:59 smithi022 ceph-mon[40250]: pgmap v14052: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:00] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T01:47:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:01 smithi122 ceph-mon[32255]: pgmap v14053: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:01 smithi022 ceph-mon[31360]: pgmap v14053: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:01 smithi022 ceph-mon[40250]: pgmap v14053: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:02.303 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:02.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:02.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:02.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:47:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:47:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:03 smithi122 ceph-mon[32255]: pgmap v14054: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/987459814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:03 smithi022 ceph-mon[31360]: pgmap v14054: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/987459814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:03 smithi022 ceph-mon[40250]: pgmap v14054: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/987459814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:05 smithi122 ceph-mon[32255]: pgmap v14055: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:05 smithi022 ceph-mon[31360]: pgmap v14055: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:05 smithi022 ceph-mon[40250]: pgmap v14055: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:47:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:47:07.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:07.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:47:07.674 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:07 smithi022 ceph-mon[31360]: pgmap v14056: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:07.674 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:07.674 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:07 smithi022 ceph-mon[40250]: pgmap v14056: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:07.675 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:07 smithi122 ceph-mon[32255]: pgmap v14056: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:08.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:08.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:08.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2320444376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2320444376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2320444376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:09.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:09 smithi122 ceph-mon[32255]: pgmap v14057: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:09 smithi022 ceph-mon[31360]: pgmap v14057: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:09 smithi022 ceph-mon[40250]: pgmap v14057: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:47:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:11 smithi122 ceph-mon[32255]: pgmap v14058: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:11 smithi022 ceph-mon[31360]: pgmap v14058: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:11 smithi022 ceph-mon[40250]: pgmap v14058: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:47:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:47:13.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:13.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:13.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:13.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:13 smithi022 ceph-mon[31360]: pgmap v14059: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3196072325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:13 smithi022 ceph-mon[40250]: pgmap v14059: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3196072325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:13 smithi122 ceph-mon[32255]: pgmap v14059: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3196072325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:15 smithi122 ceph-mon[32255]: pgmap v14060: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:15 smithi022 ceph-mon[31360]: pgmap v14060: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:15 smithi022 ceph-mon[40250]: pgmap v14060: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:47:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:47:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:17 smithi022 ceph-mon[31360]: pgmap v14061: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:47:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:17 smithi022 ceph-mon[40250]: pgmap v14061: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:17 smithi122 ceph-mon[32255]: pgmap v14061: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:18.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:18.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:18.785 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:18.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:19 smithi122 ceph-mon[32255]: pgmap v14062: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3199376102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:19 smithi022 ceph-mon[31360]: pgmap v14062: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3199376102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:19 smithi022 ceph-mon[40250]: pgmap v14062: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3199376102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:20] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:47:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:21 smithi122 ceph-mon[32255]: pgmap v14063: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:21 smithi022 ceph-mon[31360]: pgmap v14063: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:21 smithi022 ceph-mon[40250]: pgmap v14063: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:47:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:47:23.787 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:23 smithi122 ceph-mon[32255]: pgmap v14064: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:23 smithi022 ceph-mon[31360]: pgmap v14064: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:23 smithi022 ceph-mon[40250]: pgmap v14064: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:24.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:24.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:24.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/355569078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/355569078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/355569078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:25 smithi122 ceph-mon[32255]: pgmap v14065: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:25 smithi022 ceph-mon[31360]: pgmap v14065: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:25 smithi022 ceph-mon[40250]: pgmap v14065: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:47:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:47:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:27 smithi022 ceph-mon[31360]: pgmap v14066: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:47:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:27 smithi022 ceph-mon[40250]: pgmap v14066: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:27 smithi122 ceph-mon[32255]: pgmap v14066: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:29.159 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:29.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:29.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:29.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:29 smithi022 ceph-mon[31360]: pgmap v14067: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1937981739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:29 smithi022 ceph-mon[40250]: pgmap v14067: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1937981739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:29 smithi122 ceph-mon[32255]: pgmap v14067: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1937981739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:30] "GET /metrics HTTP/1.1" 200 202434 "" "Prometheus/2.18.1" 2021-12-16T01:47:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:31 smithi122 ceph-mon[32255]: pgmap v14068: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:31 smithi022 ceph-mon[31360]: pgmap v14068: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:31 smithi022 ceph-mon[40250]: pgmap v14068: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:47:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:47:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:33 smithi122 ceph-mon[32255]: pgmap v14069: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:33 smithi022 ceph-mon[31360]: pgmap v14069: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:33 smithi022 ceph-mon[40250]: pgmap v14069: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:34.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:34.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:34.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:34.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:34.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:47:34.901 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:47:34.901 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:47:34.901 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:47:34.902 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:47:34.902 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:47:34.902 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:47:34.902 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:47:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:47:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:47:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:47:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:47:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:35 smithi122 ceph-mon[32255]: pgmap v14070: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1701765018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:35 smithi022 ceph-mon[31360]: pgmap v14070: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1701765018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:35 smithi022 ceph-mon[40250]: pgmap v14070: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1701765018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:36.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:47:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:47:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:47:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:37 smithi022 ceph-mon[31360]: pgmap v14071: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:37 smithi022 ceph-mon[40250]: pgmap v14071: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:37 smithi122 ceph-mon[32255]: pgmap v14071: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:39.903 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:39 smithi022 ceph-mon[31360]: pgmap v14072: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:39.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:39 smithi022 ceph-mon[40250]: pgmap v14072: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:39 smithi122 ceph-mon[32255]: pgmap v14072: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:40.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:40.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:40.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:40] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:47:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3872923960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3872923960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3872923960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:41 smithi122 ceph-mon[32255]: pgmap v14073: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:41 smithi022 ceph-mon[31360]: pgmap v14073: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:41 smithi022 ceph-mon[40250]: pgmap v14073: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:47:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:47:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:43 smithi122 ceph-mon[32255]: pgmap v14074: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:43 smithi022 ceph-mon[31360]: pgmap v14074: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:43 smithi022 ceph-mon[40250]: pgmap v14074: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:45.268 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:45.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:45.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:45.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:45 smithi122 ceph-mon[32255]: pgmap v14075: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4076372537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:45 smithi022 ceph-mon[31360]: pgmap v14075: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4076372537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:45 smithi022 ceph-mon[40250]: pgmap v14075: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4076372537' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:46.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:47:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:47:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:47:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:47 smithi022 ceph-mon[31360]: pgmap v14076: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:47 smithi022 ceph-mon[40250]: pgmap v14076: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:47 smithi122 ceph-mon[32255]: pgmap v14076: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:49 smithi122 ceph-mon[32255]: pgmap v14077: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:49 smithi022 ceph-mon[31360]: pgmap v14077: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:49 smithi022 ceph-mon[40250]: pgmap v14077: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:50] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:47:50.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:47:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:47:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:47:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:47:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:47:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:47:50.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:47:50.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:47:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:47:51.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:51.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:51.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:51 smithi122 ceph-mon[32255]: pgmap v14078: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:47:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1353766226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:51 smithi022 ceph-mon[31360]: pgmap v14078: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:47:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1353766226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:51 smithi022 ceph-mon[40250]: pgmap v14078: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:47:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1353766226' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:47:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:47:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:53 smithi122 ceph-mon[32255]: pgmap v14079: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:53 smithi022 ceph-mon[31360]: pgmap v14079: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:53 smithi022 ceph-mon[40250]: pgmap v14079: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:56.007 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:47:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:55 smithi122 ceph-mon[32255]: pgmap v14080: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:55 smithi022 ceph-mon[31360]: pgmap v14080: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:55 smithi022 ceph-mon[40250]: pgmap v14080: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:47:56.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:47:56.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:47:56.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:47:56.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:47:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:47:57.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1869652626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1869652626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1869652626' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:47:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:47:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:57 smithi122 ceph-mon[32255]: pgmap v14081: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:57 smithi022 ceph-mon[31360]: pgmap v14081: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:47:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:57 smithi022 ceph-mon[40250]: pgmap v14081: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:47:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:00.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:47:59 smithi122 ceph-mon[32255]: pgmap v14082: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:47:59 smithi022 ceph-mon[31360]: pgmap v14082: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:47:59 smithi022 ceph-mon[40250]: pgmap v14082: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:00] "GET /metrics HTTP/1.1" 200 202429 "" "Prometheus/2.18.1" 2021-12-16T01:48:01.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:01.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:01.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:01.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:01 smithi022 ceph-mon[31360]: pgmap v14083: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3932931329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:01 smithi022 ceph-mon[40250]: pgmap v14083: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3932931329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:01 smithi122 ceph-mon[32255]: pgmap v14083: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3932931329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:48:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:48:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:03 smithi122 ceph-mon[32255]: pgmap v14084: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:03 smithi022 ceph-mon[31360]: pgmap v14084: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:03 smithi022 ceph-mon[40250]: pgmap v14084: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:06.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:05 smithi122 ceph-mon[32255]: pgmap v14085: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:05 smithi022 ceph-mon[31360]: pgmap v14085: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:05 smithi022 ceph-mon[40250]: pgmap v14085: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:06.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:48:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:48:07.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:07.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:07.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:48:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:07 smithi022 ceph-mon[31360]: pgmap v14086: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3117759638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:07 smithi022 ceph-mon[40250]: pgmap v14086: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3117759638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:07 smithi122 ceph-mon[32255]: pgmap v14086: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:48:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3117759638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:09 smithi122 ceph-mon[32255]: pgmap v14087: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2021-12-16T01:48:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:09 smithi022 ceph-mon[31360]: pgmap v14087: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2021-12-16T01:48:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:09 smithi022 ceph-mon[40250]: pgmap v14087: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2021-12-16T01:48:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:48:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:11 smithi122 ceph-mon[32255]: pgmap v14088: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-16T01:48:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:11 smithi022 ceph-mon[31360]: pgmap v14088: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-16T01:48:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:11 smithi022 ceph-mon[40250]: pgmap v14088: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-16T01:48:12.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:12.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:12.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:12.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:12.659 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:48:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:48:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2475967621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:12.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2475967621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2475967621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:13 smithi122 ceph-mon[32255]: pgmap v14089: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-16T01:48:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:13 smithi022 ceph-mon[31360]: pgmap v14089: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-16T01:48:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:13 smithi022 ceph-mon[40250]: pgmap v14089: 132 pgs: 132 active+clean; 582 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2021-12-16T01:48:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:15 smithi122 ceph-mon[32255]: pgmap v14090: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:48:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:15 smithi022 ceph-mon[31360]: pgmap v14090: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:48:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:15 smithi022 ceph-mon[40250]: pgmap v14090: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:48:16.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:48:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:48:17.486 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:48:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:17 smithi022 ceph-mon[31360]: pgmap v14091: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:48:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:17 smithi022 ceph-mon[40250]: pgmap v14091: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:48:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:17.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:17.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:17.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:17 smithi122 ceph-mon[32255]: pgmap v14091: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:48:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2456244313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2456244313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2456244313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:19 smithi122 ceph-mon[32255]: pgmap v14092: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:48:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:19 smithi022 ceph-mon[31360]: pgmap v14092: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:48:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:19 smithi022 ceph-mon[40250]: pgmap v14092: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:48:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:20] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:48:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:21 smithi122 ceph-mon[32255]: pgmap v14093: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:21 smithi022 ceph-mon[31360]: pgmap v14093: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:21 smithi022 ceph-mon[40250]: pgmap v14093: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:48:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:48:22.856 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:23.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:23.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:23.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:23 smithi122 ceph-mon[32255]: pgmap v14094: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3496700297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:23 smithi022 ceph-mon[31360]: pgmap v14094: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3496700297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:23 smithi022 ceph-mon[40250]: pgmap v14094: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3496700297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:25 smithi122 ceph-mon[32255]: pgmap v14095: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:25 smithi022 ceph-mon[31360]: pgmap v14095: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:25 smithi022 ceph-mon[40250]: pgmap v14095: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2021-12-16T01:48:26.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:48:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:48:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:48:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:27 smithi022 ceph-mon[31360]: pgmap v14096: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:27 smithi022 ceph-mon[40250]: pgmap v14096: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:27 smithi122 ceph-mon[32255]: pgmap v14096: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:28.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/137854608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/137854608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/137854608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:29 smithi122 ceph-mon[32255]: pgmap v14097: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:29 smithi022 ceph-mon[31360]: pgmap v14097: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:29 smithi022 ceph-mon[40250]: pgmap v14097: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:30] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:48:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:31 smithi122 ceph-mon[32255]: pgmap v14098: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:31 smithi022 ceph-mon[31360]: pgmap v14098: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:31 smithi022 ceph-mon[40250]: pgmap v14098: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:48:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:48:33.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:33 smithi122 ceph-mon[32255]: pgmap v14099: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:33.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:33.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:33.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:33.967 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:33 smithi022 ceph-mon[31360]: pgmap v14099: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:33.967 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:33 smithi022 ceph-mon[40250]: pgmap v14099: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:48:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:48:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:48:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:48:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3453290438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:48:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:48:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:48:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:48:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3453290438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:48:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:48:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:48:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:48:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3453290438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:35 smithi122 ceph-mon[32255]: pgmap v14100: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:35 smithi022 ceph-mon[31360]: pgmap v14100: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:35 smithi022 ceph-mon[40250]: pgmap v14100: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:48:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:48:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:48:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:37 smithi022 ceph-mon[31360]: pgmap v14101: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:37 smithi022 ceph-mon[40250]: pgmap v14101: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:37 smithi122 ceph-mon[32255]: pgmap v14101: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:38.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:39.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:39.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:39.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:39 smithi122 ceph-mon[32255]: pgmap v14102: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4203895769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:39 smithi022 ceph-mon[31360]: pgmap v14102: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4203895769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:39 smithi022 ceph-mon[40250]: pgmap v14102: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4203895769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:40] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:48:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:41 smithi122 ceph-mon[32255]: pgmap v14103: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:41 smithi022 ceph-mon[31360]: pgmap v14103: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:41 smithi022 ceph-mon[40250]: pgmap v14103: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:48:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:48:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:43 smithi122 ceph-mon[32255]: pgmap v14104: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:43 smithi022 ceph-mon[31360]: pgmap v14104: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:43 smithi022 ceph-mon[40250]: pgmap v14104: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:44.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:44.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:44.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:44.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:45 smithi122 ceph-mon[32255]: pgmap v14105: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3576862636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:45 smithi022 ceph-mon[31360]: pgmap v14105: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3576862636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:45 smithi022 ceph-mon[40250]: pgmap v14105: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3576862636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:48:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:48:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:47 smithi022 ceph-mon[31360]: pgmap v14106: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:48:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:47 smithi022 ceph-mon[40250]: pgmap v14106: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:47 smithi122 ceph-mon[32255]: pgmap v14106: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:49.713 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:49 smithi122 ceph-mon[32255]: pgmap v14107: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:49 smithi022 ceph-mon[31360]: pgmap v14107: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:49 smithi022 ceph-mon[40250]: pgmap v14107: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:50.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:50.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:50.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:50] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:48:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1110908211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:50.937 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1110908211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:50.938 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1110908211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:51 smithi122 ceph-mon[32255]: pgmap v14108: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:48:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:48:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:48:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:48:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[31360]: pgmap v14108: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:48:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:48:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:48:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:48:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[40250]: pgmap v14108: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:48:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:48:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:48:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:48:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:48:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:48:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:53 smithi122 ceph-mon[32255]: pgmap v14109: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:53 smithi022 ceph-mon[31360]: pgmap v14109: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:53 smithi022 ceph-mon[40250]: pgmap v14109: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:55.084 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:48:55.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:48:55.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:48:55.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:48:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:55 smithi022 ceph-mon[31360]: pgmap v14110: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3282364750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:55 smithi022 ceph-mon[40250]: pgmap v14110: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3282364750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:55 smithi122 ceph-mon[32255]: pgmap v14110: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:48:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3282364750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:48:57.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:48:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:48:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:48:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:57 smithi022 ceph-mon[31360]: pgmap v14111: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:57 smithi022 ceph-mon[40250]: pgmap v14111: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:57 smithi122 ceph-mon[32255]: pgmap v14111: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:48:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:48:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:48:59 smithi122 ceph-mon[32255]: pgmap v14112: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:48:59 smithi022 ceph-mon[31360]: pgmap v14112: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:48:59 smithi022 ceph-mon[40250]: pgmap v14112: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:00] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:49:00.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:00.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:00.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:00.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:01 smithi122 ceph-mon[32255]: pgmap v14113: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2836212493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:01 smithi022 ceph-mon[31360]: pgmap v14113: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2836212493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:01 smithi022 ceph-mon[40250]: pgmap v14113: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2836212493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:49:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:49:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:03 smithi122 ceph-mon[32255]: pgmap v14114: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:03 smithi022 ceph-mon[31360]: pgmap v14114: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:03 smithi022 ceph-mon[40250]: pgmap v14114: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:05.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:05 smithi122 ceph-mon[32255]: pgmap v14115: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:05 smithi022 ceph-mon[31360]: pgmap v14115: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:05 smithi022 ceph-mon[40250]: pgmap v14115: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:06.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:06.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:06.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2698230162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:06.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:49:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:49:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2698230162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2698230162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:07.674 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:49:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:07 smithi122 ceph-mon[32255]: pgmap v14116: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:07.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:07 smithi022 ceph-mon[31360]: pgmap v14116: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:08.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:08.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:07 smithi022 ceph-mon[40250]: pgmap v14116: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:08.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:09 smithi122 ceph-mon[32255]: pgmap v14117: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:09 smithi022 ceph-mon[31360]: pgmap v14117: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:09 smithi022 ceph-mon[40250]: pgmap v14117: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:10] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:49:11.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:11.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:11.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:11.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:11 smithi122 ceph-mon[32255]: pgmap v14118: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3545979791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:11 smithi022 ceph-mon[31360]: pgmap v14118: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3545979791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:11 smithi022 ceph-mon[40250]: pgmap v14118: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3545979791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:49:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:49:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:13 smithi122 ceph-mon[32255]: pgmap v14119: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:13 smithi022 ceph-mon[31360]: pgmap v14119: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:13 smithi022 ceph-mon[40250]: pgmap v14119: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:15 smithi022 ceph-mon[31360]: pgmap v14120: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:15 smithi022 ceph-mon[40250]: pgmap v14120: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:15 smithi122 ceph-mon[32255]: pgmap v14120: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:16.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:16.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:16.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:16.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:17.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:49:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:49:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:49:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:17 smithi022 ceph-mon[31360]: pgmap v14121: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/552141782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:17 smithi022 ceph-mon[40250]: pgmap v14121: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/552141782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:17 smithi122 ceph-mon[32255]: pgmap v14121: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/552141782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:19 smithi122 ceph-mon[32255]: pgmap v14122: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:19 smithi022 ceph-mon[31360]: pgmap v14122: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:19 smithi022 ceph-mon[40250]: pgmap v14122: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:20] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:49:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:21 smithi122 ceph-mon[32255]: pgmap v14123: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:21.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:21.942 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:21 smithi022 ceph-mon[40250]: pgmap v14123: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:21.942 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:21 smithi022 ceph-mon[31360]: pgmap v14123: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:22.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:22.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:22.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:22.680 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:49:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:49:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4108180760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4108180760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4108180760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:23 smithi122 ceph-mon[32255]: pgmap v14124: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:23 smithi022 ceph-mon[31360]: pgmap v14124: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:23 smithi022 ceph-mon[40250]: pgmap v14124: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:25 smithi122 ceph-mon[32255]: pgmap v14125: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:25 smithi022 ceph-mon[40250]: pgmap v14125: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:25 smithi022 ceph-mon[31360]: pgmap v14125: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:49:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:49:27.314 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:27.580 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:49:27.581 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:27 smithi022 ceph-mon[31360]: pgmap v14126: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:27.581 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:27.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:27.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:27.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:27 smithi122 ceph-mon[32255]: pgmap v14126: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:27 smithi022 ceph-mon[40250]: pgmap v14126: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1295285882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1295285882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1295285882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:29 smithi122 ceph-mon[32255]: pgmap v14127: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:29 smithi022 ceph-mon[31360]: pgmap v14127: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:29 smithi022 ceph-mon[40250]: pgmap v14127: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:30] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:49:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:31 smithi122 ceph-mon[32255]: pgmap v14128: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:31 smithi022 ceph-mon[31360]: pgmap v14128: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:31 smithi022 ceph-mon[40250]: pgmap v14128: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:49:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:49:32.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:33.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:33.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:33.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:33 smithi122 ceph-mon[32255]: pgmap v14129: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1766636233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:33 smithi022 ceph-mon[31360]: pgmap v14129: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1766636233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:33 smithi022 ceph-mon[40250]: pgmap v14129: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1766636233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:49:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:49:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:49:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:49:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:49:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:49:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:49:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:49:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:49:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:49:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:49:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:49:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:35 smithi122 ceph-mon[32255]: pgmap v14130: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:35 smithi022 ceph-mon[31360]: pgmap v14130: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:35 smithi022 ceph-mon[40250]: pgmap v14130: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:49:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:49:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:37 smithi022 ceph-mon[31360]: pgmap v14131: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:49:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:37 smithi022 ceph-mon[40250]: pgmap v14131: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:37.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:37 smithi122 ceph-mon[32255]: pgmap v14131: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:38.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:38.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:38.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:38.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4279374470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4279374470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4279374470' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:39 smithi122 ceph-mon[32255]: pgmap v14132: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:39 smithi022 ceph-mon[31360]: pgmap v14132: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:39 smithi022 ceph-mon[40250]: pgmap v14132: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:40] "GET /metrics HTTP/1.1" 200 202435 "" "Prometheus/2.18.1" 2021-12-16T01:49:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:41 smithi122 ceph-mon[32255]: pgmap v14133: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:41 smithi022 ceph-mon[31360]: pgmap v14133: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:41 smithi022 ceph-mon[40250]: pgmap v14133: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:49:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:43 smithi122 ceph-mon[32255]: pgmap v14134: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:43 smithi022 ceph-mon[31360]: pgmap v14134: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:43 smithi022 ceph-mon[40250]: pgmap v14134: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1960383369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1960383369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1960383369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:45 smithi022 ceph-mon[31360]: pgmap v14135: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:45 smithi022 ceph-mon[40250]: pgmap v14135: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:45 smithi122 ceph-mon[32255]: pgmap v14135: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:49:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:49:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:49:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:47 smithi022 ceph-mon[31360]: pgmap v14136: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:47 smithi022 ceph-mon[40250]: pgmap v14136: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:47 smithi122 ceph-mon[32255]: pgmap v14136: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:48.794 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:49.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:49.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:49.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:49 smithi122 ceph-mon[32255]: pgmap v14137: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/325805415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:49 smithi022 ceph-mon[31360]: pgmap v14137: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/325805415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:49 smithi022 ceph-mon[40250]: pgmap v14137: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/325805415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:50] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:49:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:51 smithi122 ceph-mon[32255]: pgmap v14138: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:49:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:49:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:49:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[31360]: pgmap v14138: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:49:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:49:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:49:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[40250]: pgmap v14138: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:49:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:49:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:49:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:49:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:49:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:49:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:49:53.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:49:54.165 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:53 smithi022 ceph-mon[40250]: pgmap v14139: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:53 smithi022 ceph-mon[31360]: pgmap v14139: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:54.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:53 smithi122 ceph-mon[32255]: pgmap v14139: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:54.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:54.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:54.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1132906252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1132906252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:55 smithi122 ceph-mon[32255]: pgmap v14140: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:55 smithi022 ceph-mon[31360]: pgmap v14140: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1132906252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:49:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:55 smithi022 ceph-mon[40250]: pgmap v14140: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:57.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:49:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:49:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:49:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:57 smithi022 ceph-mon[31360]: pgmap v14141: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:57 smithi022 ceph-mon[40250]: pgmap v14141: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:57 smithi122 ceph-mon[32255]: pgmap v14141: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:49:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:49:59.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:49:59.905 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:49:59 smithi022 ceph-mon[31360]: pgmap v14142: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:59.905 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:49:59 smithi022 ceph-mon[40250]: pgmap v14142: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:49:59.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:49:59 smithi122 ceph-mon[32255]: pgmap v14142: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:00] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:50:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1903147029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T01:50:01.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1903147029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T01:50:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1903147029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T01:50:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:01 smithi022 ceph-mon[31360]: pgmap v14143: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:01 smithi022 ceph-mon[40250]: pgmap v14143: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:01 smithi122 ceph-mon[32255]: pgmap v14143: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:50:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:50:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:03 smithi022 ceph-mon[31360]: pgmap v14144: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:03 smithi022 ceph-mon[40250]: pgmap v14144: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:03 smithi122 ceph-mon[32255]: pgmap v14144: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:04.906 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:05.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:05.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:05.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:05 smithi022 ceph-mon[31360]: pgmap v14145: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1012176106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:05 smithi022 ceph-mon[40250]: pgmap v14145: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1012176106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:05 smithi122 ceph-mon[32255]: pgmap v14145: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1012176106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:07.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:50:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:50:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:50:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:07 smithi022 ceph-mon[31360]: pgmap v14146: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:07 smithi022 ceph-mon[40250]: pgmap v14146: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:07 smithi122 ceph-mon[32255]: pgmap v14146: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:09 smithi022 ceph-mon[31360]: pgmap v14147: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:09 smithi022 ceph-mon[40250]: pgmap v14147: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:09 smithi122 ceph-mon[32255]: pgmap v14147: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:10.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:10] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:50:10.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:10.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:10.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/862336155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/862336155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/862336155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:11 smithi022 ceph-mon[31360]: pgmap v14148: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:11 smithi022 ceph-mon[40250]: pgmap v14148: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:11 smithi122 ceph-mon[32255]: pgmap v14148: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:50:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:50:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:13 smithi022 ceph-mon[31360]: pgmap v14149: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:13 smithi022 ceph-mon[40250]: pgmap v14149: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:13 smithi122 ceph-mon[32255]: pgmap v14149: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:15.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:16.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:16.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:16.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:16.012 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:15 smithi022 ceph-mon[40250]: pgmap v14150: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:16.013 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:15 smithi022 ceph-mon[31360]: pgmap v14150: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:15 smithi122 ceph-mon[32255]: pgmap v14150: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1430226767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1430226767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1430226767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:17.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:50:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:50:17.702 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:50:17.703 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:17 smithi022 ceph-mon[40250]: pgmap v14151: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:17.703 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:17 smithi022 ceph-mon[31360]: pgmap v14151: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:17 smithi122 ceph-mon[32255]: pgmap v14151: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:19 smithi022 ceph-mon[31360]: pgmap v14152: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:19 smithi022 ceph-mon[40250]: pgmap v14152: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:19 smithi122 ceph-mon[32255]: pgmap v14152: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:20] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:50:21.014 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:21.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:21.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:21.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:21 smithi022 ceph-mon[31360]: pgmap v14153: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3035574188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:21 smithi022 ceph-mon[40250]: pgmap v14153: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3035574188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:21 smithi122 ceph-mon[32255]: pgmap v14153: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3035574188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:50:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:50:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:23 smithi022 ceph-mon[31360]: pgmap v14154: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:23 smithi022 ceph-mon[40250]: pgmap v14154: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:23 smithi122 ceph-mon[32255]: pgmap v14154: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:25 smithi022 ceph-mon[31360]: pgmap v14155: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:25 smithi022 ceph-mon[40250]: pgmap v14155: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:25 smithi122 ceph-mon[32255]: pgmap v14155: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:26.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:26.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:26.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:26.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:50:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:50:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:50:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:27 smithi022 ceph-mon[31360]: pgmap v14156: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/419408123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:27 smithi022 ceph-mon[40250]: pgmap v14156: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/419408123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:27 smithi122 ceph-mon[32255]: pgmap v14156: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/419408123' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:29 smithi022 ceph-mon[31360]: pgmap v14157: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:29 smithi022 ceph-mon[40250]: pgmap v14157: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:29 smithi122 ceph-mon[32255]: pgmap v14157: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:30] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:50:31.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:31 smithi022 ceph-mon[31360]: pgmap v14158: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:31 smithi022 ceph-mon[40250]: pgmap v14158: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:32.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:32.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:32.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:31 smithi122 ceph-mon[32255]: pgmap v14158: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:50:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:50:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/703200621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/703200621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/703200621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:33 smithi022 ceph-mon[31360]: pgmap v14159: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:33 smithi022 ceph-mon[40250]: pgmap v14159: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:33 smithi122 ceph-mon[32255]: pgmap v14159: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:50:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:50:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:50:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:50:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:50:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:50:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:50:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:50:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:50:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:50:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:50:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:50:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:35 smithi022 ceph-mon[31360]: pgmap v14160: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:35 smithi022 ceph-mon[40250]: pgmap v14160: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:35 smithi122 ceph-mon[32255]: pgmap v14160: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:37.122 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:50:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:50:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:50:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:37 smithi022 ceph-mon[31360]: pgmap v14161: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1711024640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:37 smithi022 ceph-mon[40250]: pgmap v14161: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1711024640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:37 smithi122 ceph-mon[32255]: pgmap v14161: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:37.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1711024640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:39 smithi022 ceph-mon[31360]: pgmap v14162: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:39 smithi022 ceph-mon[40250]: pgmap v14162: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:39 smithi122 ceph-mon[32255]: pgmap v14162: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:40] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T01:50:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:41 smithi022 ceph-mon[31360]: pgmap v14163: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:41 smithi022 ceph-mon[40250]: pgmap v14163: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:41 smithi122 ceph-mon[32255]: pgmap v14163: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:42.493 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:50:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:50:42.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:42.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:42.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:43 smithi022 ceph-mon[31360]: pgmap v14164: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4241962579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:43 smithi022 ceph-mon[40250]: pgmap v14164: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4241962579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:43 smithi122 ceph-mon[32255]: pgmap v14164: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4241962579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:45 smithi022 ceph-mon[31360]: pgmap v14165: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:45 smithi022 ceph-mon[40250]: pgmap v14165: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:45 smithi122 ceph-mon[32255]: pgmap v14165: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:50:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:50:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:50:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:47 smithi022 ceph-mon[31360]: pgmap v14166: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:47.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:47 smithi022 ceph-mon[40250]: pgmap v14166: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:47.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:47.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:47 smithi122 ceph-mon[32255]: pgmap v14166: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:48.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:48.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:48.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/464733514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/464733514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/464733514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:49 smithi122 ceph-mon[32255]: pgmap v14167: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:49 smithi022 ceph-mon[31360]: pgmap v14167: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:49 smithi022 ceph-mon[40250]: pgmap v14167: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:50] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:50:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:51 smithi022 ceph-mon[31360]: pgmap v14168: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:51 smithi022 ceph-mon[40250]: pgmap v14168: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:51 smithi122 ceph-mon[32255]: pgmap v14168: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:50:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:50:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:50:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:50:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:50:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:50:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:50:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:50:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:50:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:50:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:50:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:50:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:50:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:50:53.232 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:53.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:53.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:53.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:50:54.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:54 smithi122 ceph-mon[32255]: pgmap v14169: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:54.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2411388710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:54 smithi022 ceph-mon[31360]: pgmap v14169: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2411388710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:54 smithi022 ceph-mon[40250]: pgmap v14169: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2411388710' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:50:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:56 smithi122 ceph-mon[32255]: pgmap v14170: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:56 smithi022 ceph-mon[31360]: pgmap v14170: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:56 smithi022 ceph-mon[40250]: pgmap v14170: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:50:57.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:50:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:50:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:50:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:57 smithi022 ceph-mon[31360]: pgmap v14171: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:57 smithi022 ceph-mon[40250]: pgmap v14171: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:57 smithi122 ceph-mon[32255]: pgmap v14171: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:50:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:50:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:50:58.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:50:58.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:50:58.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:59 smithi022 ceph-mon[31360]: pgmap v14172: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:50:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2880364450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:59 smithi022 ceph-mon[40250]: pgmap v14172: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:50:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2880364450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:59 smithi122 ceph-mon[32255]: pgmap v14172: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:00.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:50:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2880364450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:00] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:51:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:01 smithi022 ceph-mon[31360]: pgmap v14173: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:01 smithi022 ceph-mon[40250]: pgmap v14173: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:01 smithi122 ceph-mon[32255]: pgmap v14173: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:51:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:51:03.976 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:03.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:03 smithi022 ceph-mon[31360]: pgmap v14174: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:03.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:03 smithi022 ceph-mon[40250]: pgmap v14174: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:03 smithi122 ceph-mon[32255]: pgmap v14174: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:04.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:04.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:04.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:04.713 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508998882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508998882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508998882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:05 smithi022 ceph-mon[31360]: pgmap v14175: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:05 smithi022 ceph-mon[40250]: pgmap v14175: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:05 smithi122 ceph-mon[32255]: pgmap v14175: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:07.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:51:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:51:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:51:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:07 smithi022 ceph-mon[31360]: pgmap v14176: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:07 smithi022 ceph-mon[40250]: pgmap v14176: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:07 smithi122 ceph-mon[32255]: pgmap v14176: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:09.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:09 smithi022 ceph-mon[31360]: pgmap v14177: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:09 smithi022 ceph-mon[40250]: pgmap v14177: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:09 smithi122 ceph-mon[32255]: pgmap v14177: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:51:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2589825156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2589825156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2589825156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:11 smithi022 ceph-mon[31360]: pgmap v14178: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:11 smithi022 ceph-mon[40250]: pgmap v14178: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:11 smithi122 ceph-mon[32255]: pgmap v14178: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:51:12.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:51:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:13 smithi022 ceph-mon[31360]: pgmap v14179: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:13 smithi022 ceph-mon[40250]: pgmap v14179: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:13 smithi122 ceph-mon[32255]: pgmap v14179: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:14.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:15.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:15.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:15.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:15 smithi022 ceph-mon[31360]: pgmap v14180: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/922836458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:15 smithi022 ceph-mon[40250]: pgmap v14180: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/922836458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:15 smithi122 ceph-mon[32255]: pgmap v14180: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/922836458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:17.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:51:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:51:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:17 smithi022 ceph-mon[31360]: pgmap v14181: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:51:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:17 smithi022 ceph-mon[40250]: pgmap v14181: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:17 smithi122 ceph-mon[32255]: pgmap v14181: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:19 smithi022 ceph-mon[31360]: pgmap v14182: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:19 smithi022 ceph-mon[40250]: pgmap v14182: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:20.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:19 smithi122 ceph-mon[32255]: pgmap v14182: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:20] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:20.714 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1802717023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:20.714 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1802717023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1802717023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:21 smithi022 ceph-mon[31360]: pgmap v14183: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:21 smithi022 ceph-mon[40250]: pgmap v14183: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:21 smithi122 ceph-mon[32255]: pgmap v14183: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:51:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:51:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:23 smithi022 ceph-mon[31360]: pgmap v14184: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:23 smithi022 ceph-mon[40250]: pgmap v14184: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:23 smithi122 ceph-mon[32255]: pgmap v14184: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:25.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:25.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:25.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:25.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:25 smithi022 ceph-mon[31360]: pgmap v14185: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:25 smithi022 ceph-mon[40250]: pgmap v14185: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:25 smithi122 ceph-mon[32255]: pgmap v14185: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1565222824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1565222824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1565222824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:27.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:51:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:51:27.722 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:51:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:27 smithi022 ceph-mon[31360]: pgmap v14186: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:27 smithi022 ceph-mon[40250]: pgmap v14186: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:27 smithi122 ceph-mon[32255]: pgmap v14186: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:29 smithi022 ceph-mon[31360]: pgmap v14187: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:29 smithi022 ceph-mon[40250]: pgmap v14187: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:29 smithi122 ceph-mon[32255]: pgmap v14187: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:30] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:31.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:31.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:31.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:31 smithi022 ceph-mon[31360]: pgmap v14188: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1479783255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:31 smithi022 ceph-mon[40250]: pgmap v14188: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1479783255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:31 smithi122 ceph-mon[32255]: pgmap v14188: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1479783255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:51:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:51:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:33 smithi022 ceph-mon[31360]: pgmap v14189: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:33 smithi022 ceph-mon[40250]: pgmap v14189: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:33 smithi122 ceph-mon[32255]: pgmap v14189: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:51:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:51:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:51:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:51:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:51:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:51:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:51:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:51:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:51:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:51:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:51:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:51:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:35 smithi022 ceph-mon[31360]: pgmap v14190: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:35 smithi022 ceph-mon[40250]: pgmap v14190: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:35 smithi122 ceph-mon[32255]: pgmap v14190: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:36.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:36.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:36.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:36.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3857311122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3857311122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3857311122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:37.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:51:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:51:37.739 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:51:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:37 smithi022 ceph-mon[31360]: pgmap v14191: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:37 smithi022 ceph-mon[40250]: pgmap v14191: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:38.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:37 smithi122 ceph-mon[32255]: pgmap v14191: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:39 smithi022 ceph-mon[31360]: pgmap v14192: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:39 smithi022 ceph-mon[40250]: pgmap v14192: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:39 smithi122 ceph-mon[32255]: pgmap v14192: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:40] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:51:41.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:41.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:41.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:41.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:41.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:41 smithi022 ceph-mon[31360]: pgmap v14193: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:41.950 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:41 smithi022 ceph-mon[40250]: pgmap v14193: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:41 smithi122 ceph-mon[32255]: pgmap v14193: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:51:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:51:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/610124395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/610124395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/610124395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:43 smithi022 ceph-mon[31360]: pgmap v14194: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:43 smithi022 ceph-mon[40250]: pgmap v14194: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:43 smithi122 ceph-mon[32255]: pgmap v14194: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:45 smithi022 ceph-mon[31360]: pgmap v14195: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:45 smithi022 ceph-mon[40250]: pgmap v14195: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:46.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:45 smithi122 ceph-mon[32255]: pgmap v14195: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:46.950 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:51:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:51:47.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:47.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:47.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:47.612 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:51:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:47 smithi122 ceph-mon[32255]: pgmap v14196: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:48.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:48.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1696268146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:47 smithi022 ceph-mon[31360]: pgmap v14196: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1696268146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:47 smithi022 ceph-mon[40250]: pgmap v14196: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1696268146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:49 smithi022 ceph-mon[31360]: pgmap v14197: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:49 smithi022 ceph-mon[40250]: pgmap v14197: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:49 smithi122 ceph-mon[32255]: pgmap v14197: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:50] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:51:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:51 smithi022 ceph-mon[31360]: pgmap v14198: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:51 smithi022 ceph-mon[40250]: pgmap v14198: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:51 smithi122 ceph-mon[32255]: pgmap v14198: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:51:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:51:52.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:52.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:52.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:51:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:51:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:51:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/222955267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:51:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:51:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:51:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/222955267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:51:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:51:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:51:53.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/222955267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:53 smithi122 ceph-mon[32255]: pgmap v14199: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:51:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:53 smithi022 ceph-mon[31360]: pgmap v14199: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:51:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:53 smithi022 ceph-mon[40250]: pgmap v14199: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:51:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:55 smithi122 ceph-mon[32255]: pgmap v14200: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:55 smithi022 ceph-mon[31360]: pgmap v14200: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:55 smithi022 ceph-mon[40250]: pgmap v14200: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:51:57.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:51:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:51:57.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:51:57.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:51:57.709 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:57 smithi022 ceph-mon[31360]: pgmap v14201: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:57.709 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:57.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:57 smithi022 ceph-mon[40250]: pgmap v14201: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:57.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:57 smithi122 ceph-mon[32255]: pgmap v14201: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:51:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:51:58.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:51:58.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:51:58.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:51:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/286837684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/286837684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/286837684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:51:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:51:59 smithi122 ceph-mon[32255]: pgmap v14202: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:51:59 smithi022 ceph-mon[31360]: pgmap v14202: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:51:59 smithi022 ceph-mon[40250]: pgmap v14202: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:00] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:52:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:01 smithi022 ceph-mon[31360]: pgmap v14203: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:01 smithi022 ceph-mon[40250]: pgmap v14203: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:01 smithi122 ceph-mon[32255]: pgmap v14203: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:52:02.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:52:03.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:03.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:03.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:03.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:03 smithi022 ceph-mon[31360]: pgmap v14204: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3478381165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:03 smithi022 ceph-mon[40250]: pgmap v14204: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3478381165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:03 smithi122 ceph-mon[32255]: pgmap v14204: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3478381165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:05 smithi022 ceph-mon[31360]: pgmap v14205: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:05 smithi022 ceph-mon[40250]: pgmap v14205: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:05 smithi122 ceph-mon[32255]: pgmap v14205: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:07.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:52:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:52:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:07 smithi022 ceph-mon[31360]: pgmap v14206: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:52:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:07 smithi022 ceph-mon[40250]: pgmap v14206: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:07 smithi122 ceph-mon[32255]: pgmap v14206: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:08.452 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:08.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:09 smithi022 ceph-mon[31360]: pgmap v14207: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/116266991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:09 smithi022 ceph-mon[40250]: pgmap v14207: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/116266991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:09 smithi122 ceph-mon[32255]: pgmap v14207: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/116266991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:10] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T01:52:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:11 smithi022 ceph-mon[31360]: pgmap v14208: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:11 smithi022 ceph-mon[40250]: pgmap v14208: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:11 smithi122 ceph-mon[32255]: pgmap v14208: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:52:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:52:13.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:13 smithi022 ceph-mon[31360]: pgmap v14209: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:13 smithi022 ceph-mon[40250]: pgmap v14209: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:13 smithi122 ceph-mon[32255]: pgmap v14209: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:14.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:14.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:14.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/932764350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/932764350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/932764350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:15 smithi022 ceph-mon[31360]: pgmap v14210: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:15 smithi022 ceph-mon[40250]: pgmap v14210: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:15 smithi122 ceph-mon[32255]: pgmap v14210: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:17.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:52:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:52:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:52:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:17 smithi022 ceph-mon[31360]: pgmap v14211: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:17 smithi022 ceph-mon[40250]: pgmap v14211: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:17 smithi122 ceph-mon[32255]: pgmap v14211: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:19.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:19.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:19.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:19.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:19 smithi022 ceph-mon[31360]: pgmap v14212: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/55368238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:19 smithi022 ceph-mon[40250]: pgmap v14212: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/55368238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:19 smithi122 ceph-mon[32255]: pgmap v14212: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/55368238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:20] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:52:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:21 smithi022 ceph-mon[31360]: pgmap v14213: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:21 smithi022 ceph-mon[40250]: pgmap v14213: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:21 smithi122 ceph-mon[32255]: pgmap v14213: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:52:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:52:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:23 smithi022 ceph-mon[31360]: pgmap v14214: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:23 smithi022 ceph-mon[40250]: pgmap v14214: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:23 smithi122 ceph-mon[32255]: pgmap v14214: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:24.565 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:25 smithi022 ceph-mon[31360]: pgmap v14215: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/631778889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:25 smithi022 ceph-mon[40250]: pgmap v14215: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/631778889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:25 smithi122 ceph-mon[32255]: pgmap v14215: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/631778889' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:52:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:27 smithi022 ceph-mon[31360]: pgmap v14216: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:27 smithi022 ceph-mon[40250]: pgmap v14216: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:52:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:27 smithi122 ceph-mon[32255]: pgmap v14216: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:29.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:29 smithi022 ceph-mon[31360]: pgmap v14217: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:29 smithi022 ceph-mon[40250]: pgmap v14217: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:29 smithi122 ceph-mon[32255]: pgmap v14217: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:30.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:30.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:30.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:30] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:52:30.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3224391001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3224391001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3224391001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:31 smithi022 ceph-mon[31360]: pgmap v14218: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:31 smithi022 ceph-mon[40250]: pgmap v14218: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:31 smithi122 ceph-mon[32255]: pgmap v14218: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:52:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:52:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:33 smithi022 ceph-mon[31360]: pgmap v14219: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:33 smithi022 ceph-mon[40250]: pgmap v14219: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:33 smithi122 ceph-mon[32255]: pgmap v14219: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:52:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:52:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:52:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:52:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:52:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:52:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:52:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:52:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:52:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:52:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:52:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:52:35.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:35 smithi022 ceph-mon[31360]: pgmap v14220: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3445509062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:35 smithi022 ceph-mon[40250]: pgmap v14220: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3445509062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:35 smithi122 ceph-mon[32255]: pgmap v14220: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3445509062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:52:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:52:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:52:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:37 smithi022 ceph-mon[31360]: pgmap v14221: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:37 smithi022 ceph-mon[40250]: pgmap v14221: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:37 smithi122 ceph-mon[32255]: pgmap v14221: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:39 smithi022 ceph-mon[31360]: pgmap v14222: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:39 smithi022 ceph-mon[40250]: pgmap v14222: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:39 smithi122 ceph-mon[32255]: pgmap v14222: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:40] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:52:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:41.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:41.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:41.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:41 smithi022 ceph-mon[31360]: pgmap v14223: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1408747847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:41 smithi022 ceph-mon[40250]: pgmap v14223: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1408747847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:41 smithi122 ceph-mon[32255]: pgmap v14223: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1408747847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:52:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:52:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:43 smithi022 ceph-mon[31360]: pgmap v14224: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:43 smithi022 ceph-mon[40250]: pgmap v14224: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:43 smithi122 ceph-mon[32255]: pgmap v14224: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:45 smithi022 ceph-mon[31360]: pgmap v14225: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:45 smithi022 ceph-mon[40250]: pgmap v14225: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:46.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:45 smithi122 ceph-mon[32255]: pgmap v14225: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:46.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:46.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:46.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1013792788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1013792788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1013792788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:52:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:52:47.742 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:52:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:47 smithi022 ceph-mon[31360]: pgmap v14226: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:47 smithi022 ceph-mon[40250]: pgmap v14226: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:47 smithi122 ceph-mon[32255]: pgmap v14226: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:49 smithi022 ceph-mon[31360]: pgmap v14227: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:49 smithi022 ceph-mon[40250]: pgmap v14227: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:49 smithi122 ceph-mon[32255]: pgmap v14227: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:50] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:52:51.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:51.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:51.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:51.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:51 smithi022 ceph-mon[31360]: pgmap v14228: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:51 smithi022 ceph-mon[40250]: pgmap v14228: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:51 smithi122 ceph-mon[32255]: pgmap v14228: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:52:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:52:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1156444099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:53.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1156444099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:53.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1156444099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:53 smithi122 ceph-mon[32255]: pgmap v14229: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:52:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:52:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:52:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[31360]: pgmap v14229: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:52:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:52:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:52:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[40250]: pgmap v14229: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:52:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:52:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:52:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:54.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:54.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:52:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:55 smithi022 ceph-mon[31360]: pgmap v14230: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:55 smithi022 ceph-mon[40250]: pgmap v14230: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:55 smithi122 ceph-mon[32255]: pgmap v14230: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:52:57.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:52:57.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:52:57.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:52:57.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:52:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:52:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:52:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:57 smithi122 ceph-mon[32255]: pgmap v14231: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4163599689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:57 smithi022 ceph-mon[31360]: pgmap v14231: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4163599689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:52:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:57 smithi022 ceph-mon[40250]: pgmap v14231: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:52:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:52:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4163599689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:52:59 smithi022 ceph-mon[31360]: pgmap v14232: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:52:59 smithi022 ceph-mon[40250]: pgmap v14232: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:52:59 smithi122 ceph-mon[32255]: pgmap v14232: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:00] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:53:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:01 smithi022 ceph-mon[31360]: pgmap v14233: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:01 smithi022 ceph-mon[40250]: pgmap v14233: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:02.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:01 smithi122 ceph-mon[32255]: pgmap v14233: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:02.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:02.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:02.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:53:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:53:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1422351940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:03.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1422351940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1422351940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:04.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:03 smithi022 ceph-mon[31360]: pgmap v14234: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:03 smithi022 ceph-mon[40250]: pgmap v14234: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:03 smithi122 ceph-mon[32255]: pgmap v14234: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:05 smithi022 ceph-mon[31360]: pgmap v14235: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:05 smithi022 ceph-mon[40250]: pgmap v14235: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:05 smithi122 ceph-mon[32255]: pgmap v14235: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:07.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:53:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:53:07.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:53:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:07 smithi022 ceph-mon[31360]: pgmap v14236: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:07 smithi022 ceph-mon[40250]: pgmap v14236: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:07.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:07.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:07.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:07 smithi122 ceph-mon[32255]: pgmap v14236: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2722792224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2722792224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2722792224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:09 smithi122 ceph-mon[32255]: pgmap v14237: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:09 smithi022 ceph-mon[31360]: pgmap v14237: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:09 smithi022 ceph-mon[40250]: pgmap v14237: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:53:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:11 smithi022 ceph-mon[31360]: pgmap v14238: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:11 smithi022 ceph-mon[40250]: pgmap v14238: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:11 smithi122 ceph-mon[32255]: pgmap v14238: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:53:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:53:12.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:13.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:13.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:13.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:13 smithi022 ceph-mon[31360]: pgmap v14239: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/92674243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:13 smithi022 ceph-mon[40250]: pgmap v14239: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/92674243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:13 smithi122 ceph-mon[32255]: pgmap v14239: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/92674243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:15 smithi022 ceph-mon[31360]: pgmap v14240: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:15 smithi022 ceph-mon[40250]: pgmap v14240: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:15 smithi122 ceph-mon[32255]: pgmap v14240: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:17.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:53:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:53:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:17 smithi022 ceph-mon[31360]: pgmap v14241: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:53:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:17 smithi022 ceph-mon[40250]: pgmap v14241: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:17 smithi122 ceph-mon[32255]: pgmap v14241: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:18.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:18.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:18.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:18.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2416880082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2416880082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2416880082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:19 smithi122 ceph-mon[32255]: pgmap v14242: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:19 smithi022 ceph-mon[31360]: pgmap v14242: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:19 smithi022 ceph-mon[40250]: pgmap v14242: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:20] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:53:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:21 smithi022 ceph-mon[31360]: pgmap v14243: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:21 smithi022 ceph-mon[40250]: pgmap v14243: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:21 smithi122 ceph-mon[32255]: pgmap v14243: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:53:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:53:23.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:24.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:24.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:24.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:24.015 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:23 smithi022 ceph-mon[40250]: pgmap v14244: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:24.016 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:23 smithi022 ceph-mon[31360]: pgmap v14244: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:23 smithi122 ceph-mon[32255]: pgmap v14244: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4062289833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4062289833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4062289833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:25 smithi022 ceph-mon[31360]: pgmap v14245: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:25 smithi022 ceph-mon[40250]: pgmap v14245: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:25 smithi122 ceph-mon[32255]: pgmap v14245: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:53:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:53:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:53:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:27 smithi022 ceph-mon[31360]: pgmap v14246: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:27 smithi022 ceph-mon[40250]: pgmap v14246: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:27 smithi122 ceph-mon[32255]: pgmap v14246: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:29.016 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:29.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:29.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:29.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:29 smithi022 ceph-mon[31360]: pgmap v14247: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2263792843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:29 smithi022 ceph-mon[40250]: pgmap v14247: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2263792843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:29 smithi122 ceph-mon[32255]: pgmap v14247: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2263792843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:30] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:53:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:31 smithi022 ceph-mon[31360]: pgmap v14248: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:31 smithi022 ceph-mon[40250]: pgmap v14248: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:31 smithi122 ceph-mon[32255]: pgmap v14248: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:53:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:53:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:33 smithi022 ceph-mon[31360]: pgmap v14249: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:33 smithi022 ceph-mon[40250]: pgmap v14249: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:33 smithi122 ceph-mon[32255]: pgmap v14249: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:34.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:34.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:53:34.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:53:34.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:53:34.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:53:34.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1462905850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:34.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:53:34.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:53:34.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:53:34.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:53:34.761 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1462905850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:34.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:34.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:34.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:53:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:53:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:53:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:53:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1462905850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:35 smithi022 ceph-mon[31360]: pgmap v14250: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:35 smithi022 ceph-mon[40250]: pgmap v14250: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:35 smithi122 ceph-mon[32255]: pgmap v14250: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:37.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:53:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:53:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:37 smithi022 ceph-mon[31360]: pgmap v14251: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:53:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:37 smithi022 ceph-mon[40250]: pgmap v14251: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:37 smithi122 ceph-mon[32255]: pgmap v14251: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:39.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:39 smithi022 ceph-mon[31360]: pgmap v14252: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:39 smithi022 ceph-mon[40250]: pgmap v14252: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:40.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:40.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:40.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:39 smithi122 ceph-mon[32255]: pgmap v14252: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:40] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:53:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1618408443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1618408443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1618408443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:41 smithi022 ceph-mon[31360]: pgmap v14253: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:41 smithi022 ceph-mon[40250]: pgmap v14253: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:41 smithi122 ceph-mon[32255]: pgmap v14253: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:53:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:53:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:43 smithi022 ceph-mon[31360]: pgmap v14254: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:43 smithi022 ceph-mon[40250]: pgmap v14254: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:43 smithi122 ceph-mon[32255]: pgmap v14254: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:45.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:45.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:45.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:45.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:45 smithi022 ceph-mon[31360]: pgmap v14255: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/98173252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:45 smithi022 ceph-mon[40250]: pgmap v14255: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/98173252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:45 smithi122 ceph-mon[32255]: pgmap v14255: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/98173252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:47.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:53:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:53:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:53:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:47 smithi022 ceph-mon[31360]: pgmap v14256: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:47 smithi022 ceph-mon[40250]: pgmap v14256: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:47.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:47 smithi122 ceph-mon[32255]: pgmap v14256: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:49 smithi022 ceph-mon[31360]: pgmap v14257: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:49 smithi022 ceph-mon[40250]: pgmap v14257: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:49 smithi122 ceph-mon[32255]: pgmap v14257: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:53:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:50.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:50.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:50.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:51 smithi022 ceph-mon[31360]: pgmap v14258: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4105060385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:51 smithi022 ceph-mon[40250]: pgmap v14258: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4105060385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:51 smithi122 ceph-mon[32255]: pgmap v14258: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4105060385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:53:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:53:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:53 smithi122 ceph-mon[32255]: pgmap v14259: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:53:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:53:53.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:53:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[31360]: pgmap v14259: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:53:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:53:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:53:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[40250]: pgmap v14259: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:53:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:53:54.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:53:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:53:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:53:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:53:55.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:53:56.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:53:56.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:53:56.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:53:56.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:56 smithi022 ceph-mon[31360]: pgmap v14260: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:56.245 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:56 smithi022 ceph-mon[40250]: pgmap v14260: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:56 smithi122 ceph-mon[32255]: pgmap v14260: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:53:57.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3392822993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:57.398 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3392822993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:57.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:53:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:53:57.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3392822993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:53:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:53:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:58 smithi122 ceph-mon[32255]: pgmap v14261: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:53:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:58 smithi022 ceph-mon[31360]: pgmap v14261: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:53:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:53:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:58 smithi022 ceph-mon[40250]: pgmap v14261: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:53:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:53:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:54:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:00 smithi122 ceph-mon[32255]: pgmap v14262: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:00 smithi022 ceph-mon[31360]: pgmap v14262: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:00 smithi022 ceph-mon[40250]: pgmap v14262: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:01.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:01.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:01.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:01.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:02 smithi122 ceph-mon[32255]: pgmap v14263: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2251119988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:54:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:54:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:02 smithi022 ceph-mon[40250]: pgmap v14263: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2251119988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:02 smithi022 ceph-mon[31360]: pgmap v14263: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2251119988' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:04 smithi122 ceph-mon[32255]: pgmap v14264: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:04 smithi022 ceph-mon[31360]: pgmap v14264: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:04 smithi022 ceph-mon[40250]: pgmap v14264: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:06 smithi122 ceph-mon[32255]: pgmap v14265: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:06 smithi022 ceph-mon[31360]: pgmap v14265: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:06 smithi022 ceph-mon[40250]: pgmap v14265: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:06.616 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:06.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:06.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:07.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:54:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:54:07.642 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:54:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:07 smithi122 ceph-mon[32255]: pgmap v14266: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3737815775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:07 smithi022 ceph-mon[31360]: pgmap v14266: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3737815775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:07 smithi022 ceph-mon[40250]: pgmap v14266: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3737815775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:09 smithi022 ceph-mon[31360]: pgmap v14267: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:09 smithi022 ceph-mon[40250]: pgmap v14267: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:09 smithi122 ceph-mon[32255]: pgmap v14267: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:10] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T01:54:11.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:11 smithi022 ceph-mon[31360]: pgmap v14268: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:12.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:11 smithi022 ceph-mon[40250]: pgmap v14268: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:11 smithi122 ceph-mon[32255]: pgmap v14268: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:12.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:12.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:12.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:54:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:54:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2192431850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2192431850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2192431850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:13 smithi022 ceph-mon[31360]: pgmap v14269: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:13 smithi022 ceph-mon[40250]: pgmap v14269: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:13 smithi122 ceph-mon[32255]: pgmap v14269: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:15 smithi022 ceph-mon[31360]: pgmap v14270: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:15 smithi022 ceph-mon[40250]: pgmap v14270: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:15 smithi122 ceph-mon[32255]: pgmap v14270: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:17.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:17.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:54:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:54:17.644 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:54:17.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:17.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:17.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:17 smithi122 ceph-mon[32255]: pgmap v14271: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:17 smithi022 ceph-mon[31360]: pgmap v14271: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:17 smithi022 ceph-mon[40250]: pgmap v14271: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1506272676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1506272676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1506272676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:19.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:19 smithi122 ceph-mon[32255]: pgmap v14272: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:19 smithi022 ceph-mon[31360]: pgmap v14272: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:19 smithi022 ceph-mon[40250]: pgmap v14272: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:20] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:54:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:21 smithi022 ceph-mon[31360]: pgmap v14273: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:22.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:21 smithi022 ceph-mon[40250]: pgmap v14273: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:21 smithi122 ceph-mon[32255]: pgmap v14273: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:54:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:54:22.727 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:23.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:23.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:23.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:23 smithi022 ceph-mon[31360]: pgmap v14274: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3966489202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:23 smithi022 ceph-mon[40250]: pgmap v14274: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3966489202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:23 smithi122 ceph-mon[32255]: pgmap v14274: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3966489202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:25 smithi022 ceph-mon[31360]: pgmap v14275: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:25 smithi022 ceph-mon[40250]: pgmap v14275: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:25 smithi122 ceph-mon[32255]: pgmap v14275: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:27.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:54:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:54:27.647 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:54:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:27 smithi122 ceph-mon[32255]: pgmap v14276: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:27.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:27 smithi022 ceph-mon[31360]: pgmap v14276: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:27 smithi022 ceph-mon[40250]: pgmap v14276: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:28.105 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:28.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:28.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:28.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1321615618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1321615618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1321615618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:29 smithi122 ceph-mon[32255]: pgmap v14277: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:29 smithi022 ceph-mon[31360]: pgmap v14277: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:29 smithi022 ceph-mon[40250]: pgmap v14277: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:30] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T01:54:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:31 smithi022 ceph-mon[31360]: pgmap v14278: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:31 smithi022 ceph-mon[40250]: pgmap v14278: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:31 smithi122 ceph-mon[32255]: pgmap v14278: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:54:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:54:33.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:33.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:33.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:33.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:33 smithi022 ceph-mon[31360]: pgmap v14279: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:33 smithi022 ceph-mon[40250]: pgmap v14279: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:33 smithi122 ceph-mon[32255]: pgmap v14279: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1402661529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:54:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:54:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:54:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:54:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1402661529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:54:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:54:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:54:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:54:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1402661529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:54:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:54:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:54:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:54:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:35 smithi022 ceph-mon[31360]: pgmap v14280: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:35 smithi022 ceph-mon[40250]: pgmap v14280: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:35 smithi122 ceph-mon[32255]: pgmap v14280: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:37.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:54:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:54:37.650 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:54:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:37 smithi122 ceph-mon[32255]: pgmap v14281: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:38.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:37 smithi022 ceph-mon[31360]: pgmap v14281: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:37 smithi022 ceph-mon[40250]: pgmap v14281: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:38.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:39.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:39.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:39.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:39 smithi022 ceph-mon[31360]: pgmap v14282: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2028284332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:39 smithi022 ceph-mon[40250]: pgmap v14282: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2028284332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:39 smithi122 ceph-mon[32255]: pgmap v14282: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2028284332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:40] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:54:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:41 smithi022 ceph-mon[31360]: pgmap v14283: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:41 smithi022 ceph-mon[40250]: pgmap v14283: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:41 smithi122 ceph-mon[32255]: pgmap v14283: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:54:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:54:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:43 smithi022 ceph-mon[31360]: pgmap v14284: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:44.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:43 smithi022 ceph-mon[40250]: pgmap v14284: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:43 smithi122 ceph-mon[32255]: pgmap v14284: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:44.219 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:44.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:44.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:44.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2421276426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2421276426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2421276426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:45 smithi122 ceph-mon[32255]: pgmap v14285: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:45 smithi022 ceph-mon[31360]: pgmap v14285: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:45 smithi022 ceph-mon[40250]: pgmap v14285: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:54:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:54:47.652 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:54:47.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:47 smithi022 ceph-mon[40250]: pgmap v14286: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:47.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:47 smithi122 ceph-mon[32255]: pgmap v14286: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:47 smithi022 ceph-mon[31360]: pgmap v14286: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:49.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:49.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:49.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:49.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:49.965 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:49 smithi022 ceph-mon[31360]: pgmap v14287: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:49.966 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:49 smithi022 ceph-mon[40250]: pgmap v14287: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:49 smithi122 ceph-mon[32255]: pgmap v14287: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:50] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:54:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/368962669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/368962669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/368962669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:51 smithi122 ceph-mon[32255]: pgmap v14288: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:51 smithi022 ceph-mon[31360]: pgmap v14288: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:51 smithi022 ceph-mon[40250]: pgmap v14288: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:54:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:54:54.116 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:53 smithi122 ceph-mon[32255]: pgmap v14289: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:54.118 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:53 smithi022 ceph-mon[31360]: pgmap v14289: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:54.118 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:53 smithi022 ceph-mon[40250]: pgmap v14289: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:54.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:54:54.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:54:54.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:54:54.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:54:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:54:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:54:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:54:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:54:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:54:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:54:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:54:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:54:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:54:55.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:54:55.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:54:55.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:54:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:55 smithi122 ceph-mon[32255]: pgmap v14290: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2776932307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:55 smithi022 ceph-mon[31360]: pgmap v14290: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2776932307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:55 smithi022 ceph-mon[40250]: pgmap v14290: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:54:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2776932307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:54:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:54:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:54:57.654 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:54:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:57 smithi122 ceph-mon[32255]: pgmap v14291: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:57 smithi022 ceph-mon[31360]: pgmap v14291: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:54:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:57 smithi022 ceph-mon[40250]: pgmap v14291: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:54:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:54:59 smithi022 ceph-mon[31360]: pgmap v14292: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:54:59 smithi022 ceph-mon[40250]: pgmap v14292: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:54:59 smithi122 ceph-mon[32255]: pgmap v14292: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:00.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:00] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:55:00.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:00.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:00.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/194608852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/194608852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/194608852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:01 smithi122 ceph-mon[32255]: pgmap v14293: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:01 smithi022 ceph-mon[31360]: pgmap v14293: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:01 smithi022 ceph-mon[40250]: pgmap v14293: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:55:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:55:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:03 smithi122 ceph-mon[32255]: pgmap v14294: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:03 smithi022 ceph-mon[31360]: pgmap v14294: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:03 smithi022 ceph-mon[40250]: pgmap v14294: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:05.692 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:05 smithi022 ceph-mon[31360]: pgmap v14295: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:05 smithi022 ceph-mon[40250]: pgmap v14295: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:06.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:06.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:06.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:05 smithi122 ceph-mon[32255]: pgmap v14295: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3548782638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:07.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:55:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:55:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3548782638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3548782638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:55:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:07 smithi122 ceph-mon[32255]: pgmap v14296: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:07 smithi022 ceph-mon[31360]: pgmap v14296: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:07 smithi022 ceph-mon[40250]: pgmap v14296: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:09 smithi122 ceph-mon[32255]: pgmap v14297: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:09 smithi022 ceph-mon[31360]: pgmap v14297: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:09 smithi022 ceph-mon[40250]: pgmap v14297: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T01:55:11.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:11.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:11 smithi122 ceph-mon[32255]: pgmap v14298: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/304293129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:11 smithi022 ceph-mon[31360]: pgmap v14298: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/304293129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:11 smithi022 ceph-mon[40250]: pgmap v14298: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/304293129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:55:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:55:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:13 smithi122 ceph-mon[32255]: pgmap v14299: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:13 smithi022 ceph-mon[31360]: pgmap v14299: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:13 smithi022 ceph-mon[40250]: pgmap v14299: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:15 smithi122 ceph-mon[32255]: pgmap v14300: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:15 smithi022 ceph-mon[31360]: pgmap v14300: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:15 smithi022 ceph-mon[40250]: pgmap v14300: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:16.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:16.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:16.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:17.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:55:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:55:17.657 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:55:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:17 smithi122 ceph-mon[32255]: pgmap v14301: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1751317018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:17 smithi022 ceph-mon[31360]: pgmap v14301: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1751317018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:18.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:18.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:17 smithi022 ceph-mon[40250]: pgmap v14301: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:18.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1751317018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:18.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:19 smithi122 ceph-mon[32255]: pgmap v14302: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:19 smithi022 ceph-mon[31360]: pgmap v14302: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:19 smithi022 ceph-mon[40250]: pgmap v14302: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:20] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:55:21.794 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:22.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:22.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:22.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:22.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:21 smithi022 ceph-mon[31360]: pgmap v14303: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:22.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:21 smithi022 ceph-mon[40250]: pgmap v14303: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:21 smithi122 ceph-mon[32255]: pgmap v14303: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:55:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:55:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2119138925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2119138925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:23.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2119138925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:23 smithi122 ceph-mon[32255]: pgmap v14304: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:23 smithi022 ceph-mon[31360]: pgmap v14304: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:23 smithi022 ceph-mon[40250]: pgmap v14304: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:25 smithi122 ceph-mon[32255]: pgmap v14305: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:25 smithi022 ceph-mon[31360]: pgmap v14305: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:25 smithi022 ceph-mon[40250]: pgmap v14305: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:27.163 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:27.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:55:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:55:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:55:27.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:27.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:27.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:27 smithi122 ceph-mon[32255]: pgmap v14306: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1557685330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:27 smithi022 ceph-mon[31360]: pgmap v14306: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1557685330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:27 smithi022 ceph-mon[40250]: pgmap v14306: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1557685330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:29 smithi122 ceph-mon[32255]: pgmap v14307: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:29 smithi022 ceph-mon[31360]: pgmap v14307: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:29 smithi022 ceph-mon[40250]: pgmap v14307: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:30] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:55:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:31 smithi122 ceph-mon[32255]: pgmap v14308: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:31 smithi022 ceph-mon[31360]: pgmap v14308: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:31 smithi022 ceph-mon[40250]: pgmap v14308: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:32.533 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:55:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:55:32.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:32.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:32.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:33 smithi122 ceph-mon[32255]: pgmap v14309: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3185758954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:33 smithi022 ceph-mon[31360]: pgmap v14309: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3185758954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:33 smithi022 ceph-mon[40250]: pgmap v14309: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3185758954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:55:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:55:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:55:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:55:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:55:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:55:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:55:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:55:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:55:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:55:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:55:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:55:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:35 smithi122 ceph-mon[32255]: pgmap v14310: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:35 smithi022 ceph-mon[31360]: pgmap v14310: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:35 smithi022 ceph-mon[40250]: pgmap v14310: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:37.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:55:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:55:37.662 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:55:37.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:37 smithi122 ceph-mon[32255]: pgmap v14311: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:37 smithi022 ceph-mon[31360]: pgmap v14311: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:37 smithi022 ceph-mon[40250]: pgmap v14311: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:38.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:38.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:38.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2727282336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2727282336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2727282336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:39 smithi122 ceph-mon[32255]: pgmap v14312: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:39 smithi022 ceph-mon[31360]: pgmap v14312: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:39 smithi022 ceph-mon[40250]: pgmap v14312: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:40] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:55:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:41 smithi122 ceph-mon[32255]: pgmap v14313: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:41 smithi022 ceph-mon[31360]: pgmap v14313: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:41 smithi022 ceph-mon[40250]: pgmap v14313: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:55:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:55:43.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:43.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:43.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:43.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:43 smithi022 ceph-mon[31360]: pgmap v14314: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2281158167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:43 smithi022 ceph-mon[40250]: pgmap v14314: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2281158167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:43 smithi122 ceph-mon[32255]: pgmap v14314: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2281158167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:45 smithi122 ceph-mon[32255]: pgmap v14315: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:45 smithi022 ceph-mon[31360]: pgmap v14315: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:45 smithi022 ceph-mon[40250]: pgmap v14315: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:55:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:55:47.664 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:55:47.665 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:47 smithi022 ceph-mon[40250]: pgmap v14316: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:47.665 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:47.665 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:47 smithi022 ceph-mon[31360]: pgmap v14316: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:47.665 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:47 smithi122 ceph-mon[32255]: pgmap v14316: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:48.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:49 smithi122 ceph-mon[32255]: pgmap v14317: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2919088611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:49 smithi022 ceph-mon[31360]: pgmap v14317: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2919088611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:49 smithi022 ceph-mon[40250]: pgmap v14317: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2919088611' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:55:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:51 smithi122 ceph-mon[32255]: pgmap v14318: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:51 smithi022 ceph-mon[31360]: pgmap v14318: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:51 smithi022 ceph-mon[40250]: pgmap v14318: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:55:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:55:54.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:53 smithi122 ceph-mon[32255]: pgmap v14319: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:53 smithi022 ceph-mon[31360]: pgmap v14319: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:53 smithi022 ceph-mon[40250]: pgmap v14319: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:54.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:54.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:54.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:55:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2980334384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:55:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:55:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:55:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2980334384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:55:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:55:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:55:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2980334384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:55:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:55:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:55:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:55:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:55 smithi022 ceph-mon[31360]: pgmap v14320: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:55:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:55 smithi022 ceph-mon[40250]: pgmap v14320: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:55:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:55 smithi122 ceph-mon[32255]: pgmap v14320: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:55:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:55:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:55:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:55:57.666 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:55:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:57 smithi122 ceph-mon[32255]: pgmap v14321: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:57 smithi022 ceph-mon[31360]: pgmap v14321: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:57 smithi022 ceph-mon[40250]: pgmap v14321: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:55:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:55:59.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:55:59.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:55:59.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:55:59.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:59 smithi022 ceph-mon[31360]: pgmap v14322: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:55:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1072032257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:59 smithi022 ceph-mon[40250]: pgmap v14322: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:55:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1072032257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:59 smithi122 ceph-mon[32255]: pgmap v14322: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:55:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1072032257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:56:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:01 smithi122 ceph-mon[32255]: pgmap v14323: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:01 smithi022 ceph-mon[31360]: pgmap v14323: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:01 smithi022 ceph-mon[40250]: pgmap v14323: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:56:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:56:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:03 smithi122 ceph-mon[32255]: pgmap v14324: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:03 smithi022 ceph-mon[31360]: pgmap v14324: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:03 smithi022 ceph-mon[40250]: pgmap v14324: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:04.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:05.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:05.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:05.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:05 smithi122 ceph-mon[32255]: pgmap v14325: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2636450334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:05 smithi022 ceph-mon[31360]: pgmap v14325: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2636450334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:05 smithi022 ceph-mon[40250]: pgmap v14325: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2636450334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:07.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:56:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:56:07.668 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:56:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:07 smithi122 ceph-mon[32255]: pgmap v14326: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:07 smithi022 ceph-mon[31360]: pgmap v14326: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:07 smithi022 ceph-mon[40250]: pgmap v14326: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:10.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:10.120 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:09 smithi022 ceph-mon[31360]: pgmap v14327: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:10.121 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:09 smithi022 ceph-mon[40250]: pgmap v14327: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:09 smithi122 ceph-mon[32255]: pgmap v14327: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:10] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:56:10.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:10.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:10.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4235773701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4235773701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4235773701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:11 smithi122 ceph-mon[32255]: pgmap v14328: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:11 smithi022 ceph-mon[31360]: pgmap v14328: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:11 smithi022 ceph-mon[40250]: pgmap v14328: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:56:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:56:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:13 smithi122 ceph-mon[32255]: pgmap v14329: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:13 smithi022 ceph-mon[31360]: pgmap v14329: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:13 smithi022 ceph-mon[40250]: pgmap v14329: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:15.490 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:15 smithi122 ceph-mon[32255]: pgmap v14330: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:15 smithi022 ceph-mon[31360]: pgmap v14330: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:15 smithi022 ceph-mon[40250]: pgmap v14330: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:17.059 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1004815426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1004815426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1004815426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:17.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:56:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:56:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:56:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:17 smithi122 ceph-mon[32255]: pgmap v14331: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:17 smithi022 ceph-mon[31360]: pgmap v14331: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:17 smithi022 ceph-mon[40250]: pgmap v14331: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:19 smithi122 ceph-mon[32255]: pgmap v14332: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:19 smithi022 ceph-mon[31360]: pgmap v14332: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:19 smithi022 ceph-mon[40250]: pgmap v14332: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:56:20.863 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:21.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:21.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:21.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:21 smithi122 ceph-mon[32255]: pgmap v14333: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3720718346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:21 smithi022 ceph-mon[31360]: pgmap v14333: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3720718346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:21 smithi022 ceph-mon[40250]: pgmap v14333: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3720718346' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:56:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:56:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:23 smithi122 ceph-mon[32255]: pgmap v14334: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:23 smithi022 ceph-mon[31360]: pgmap v14334: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:23 smithi022 ceph-mon[40250]: pgmap v14334: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:25 smithi122 ceph-mon[32255]: pgmap v14335: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:26.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:26.233 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:25 smithi022 ceph-mon[31360]: pgmap v14335: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:26.233 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:25 smithi022 ceph-mon[40250]: pgmap v14335: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:26.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:26.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:26.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3677263135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3677263135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3677263135' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:27.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:56:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:56:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:56:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:27 smithi122 ceph-mon[32255]: pgmap v14336: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:27 smithi022 ceph-mon[31360]: pgmap v14336: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:27 smithi022 ceph-mon[40250]: pgmap v14336: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:29 smithi122 ceph-mon[32255]: pgmap v14337: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:29 smithi022 ceph-mon[31360]: pgmap v14337: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:29 smithi022 ceph-mon[40250]: pgmap v14337: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T01:56:31.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:31.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:31.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:31.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:31.974 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:31 smithi022 ceph-mon[31360]: pgmap v14338: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:31.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:31 smithi022 ceph-mon[40250]: pgmap v14338: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:31 smithi122 ceph-mon[32255]: pgmap v14338: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:56:32.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:56:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4109502657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4109502657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4109502657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:33 smithi122 ceph-mon[32255]: pgmap v14339: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:33 smithi022 ceph-mon[31360]: pgmap v14339: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:33 smithi022 ceph-mon[40250]: pgmap v14339: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:56:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:56:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:56:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:56:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:56:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:56:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:56:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:56:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:56:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:56:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:56:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:56:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:35 smithi122 ceph-mon[32255]: pgmap v14340: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:35 smithi022 ceph-mon[31360]: pgmap v14340: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:35 smithi022 ceph-mon[40250]: pgmap v14340: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:36.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:37.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:37.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:37.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:37.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:56:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:56:37.673 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:56:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:37 smithi122 ceph-mon[32255]: pgmap v14341: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2013639078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:37 smithi022 ceph-mon[31360]: pgmap v14341: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2013639078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:37 smithi022 ceph-mon[40250]: pgmap v14341: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2013639078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:39 smithi122 ceph-mon[32255]: pgmap v14342: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:39 smithi022 ceph-mon[31360]: pgmap v14342: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:39 smithi022 ceph-mon[40250]: pgmap v14342: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:40] "GET /metrics HTTP/1.1" 200 202436 "" "Prometheus/2.18.1" 2021-12-16T01:56:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:41 smithi122 ceph-mon[32255]: pgmap v14343: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:41 smithi022 ceph-mon[31360]: pgmap v14343: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:41 smithi022 ceph-mon[40250]: pgmap v14343: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:42.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:56:42.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:56:42.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:42.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:42.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/438221267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/438221267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/438221267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:43 smithi122 ceph-mon[32255]: pgmap v14344: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:43 smithi022 ceph-mon[31360]: pgmap v14344: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:43 smithi022 ceph-mon[40250]: pgmap v14344: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:45 smithi122 ceph-mon[32255]: pgmap v14345: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:45 smithi022 ceph-mon[31360]: pgmap v14345: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:45 smithi022 ceph-mon[40250]: pgmap v14345: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:56:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:56:47.676 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:56:47.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:47 smithi122 ceph-mon[32255]: pgmap v14346: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:47 smithi022 ceph-mon[31360]: pgmap v14346: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:47 smithi022 ceph-mon[40250]: pgmap v14346: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:48.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:48.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:48.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2616016851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2616016851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2616016851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:49 smithi022 ceph-mon[31360]: pgmap v14347: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:49 smithi022 ceph-mon[40250]: pgmap v14347: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:49 smithi122 ceph-mon[32255]: pgmap v14347: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:50] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:56:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:51 smithi122 ceph-mon[32255]: pgmap v14348: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:51 smithi022 ceph-mon[31360]: pgmap v14348: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:51 smithi022 ceph-mon[40250]: pgmap v14348: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:56:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:56:53.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:53.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:53.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:53.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:53 smithi122 ceph-mon[32255]: pgmap v14349: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/499686570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:53 smithi022 ceph-mon[31360]: pgmap v14349: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/499686570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:53 smithi022 ceph-mon[40250]: pgmap v14349: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T01:56:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/499686570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:55 smithi122 ceph-mon[32255]: pgmap v14350: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:56:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:56:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:56:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[31360]: pgmap v14350: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:56:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:56:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:56:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[40250]: pgmap v14350: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:56:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:56:56.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:56:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:56:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:56:57.677 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:56:57.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:57 smithi022 ceph-mon[40250]: pgmap v14351: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:57.678 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:57 smithi122 ceph-mon[32255]: pgmap v14351: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:57 smithi022 ceph-mon[31360]: pgmap v14351: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:56:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:56:58.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:56:58.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:56:58.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:56:58.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:56:59.659 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:56:59.660 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:56:59.660 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: pgmap v14352: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:59.660 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1479703496' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:59.660 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:56:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: pgmap v14352: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:56:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1479703496' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:56:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:56:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2992M 2021-12-16T01:56:59.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:56:59.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:56:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2992M 2021-12-16T01:57:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: pgmap v14352: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1479703496' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2992M 2021-12-16T01:57:00.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:00.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:56:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:00] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:57:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3225M 2021-12-16T01:57:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3225M 2021-12-16T01:57:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T01:57:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3225M 2021-12-16T01:57:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:57:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:01 smithi122 ceph-mon[32255]: pgmap v14353: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:01 smithi022 ceph-mon[31360]: pgmap v14353: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:01 smithi022 ceph-mon[40250]: pgmap v14353: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:57:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:57:03.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:04.177 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:04.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:04.178 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:04.179 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:03 smithi022 ceph-mon[31360]: pgmap v14354: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:04.179 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:03 smithi022 ceph-mon[40250]: pgmap v14354: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:04.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:03 smithi122 ceph-mon[32255]: pgmap v14354: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1896319289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1896319289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1896319289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:05 smithi122 ceph-mon[32255]: pgmap v14355: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:05 smithi022 ceph-mon[31360]: pgmap v14355: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:05 smithi022 ceph-mon[40250]: pgmap v14355: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:07.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:57:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:57:07.680 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:57:07.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:07 smithi022 ceph-mon[40250]: pgmap v14356: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:07.682 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:07 smithi122 ceph-mon[32255]: pgmap v14356: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:07 smithi022 ceph-mon[31360]: pgmap v14356: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:09.180 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:09.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:09.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:09.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:09 smithi122 ceph-mon[32255]: pgmap v14357: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1251836338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:09 smithi022 ceph-mon[31360]: pgmap v14357: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1251836338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:09 smithi022 ceph-mon[40250]: pgmap v14357: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1251836338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:10] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:57:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:11 smithi122 ceph-mon[32255]: pgmap v14358: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:11 smithi022 ceph-mon[31360]: pgmap v14358: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:11 smithi022 ceph-mon[40250]: pgmap v14358: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:57:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:57:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:13 smithi122 ceph-mon[32255]: pgmap v14359: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:13 smithi022 ceph-mon[31360]: pgmap v14359: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:13 smithi022 ceph-mon[40250]: pgmap v14359: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:14.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:14.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:14.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:14.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:15 smithi122 ceph-mon[32255]: pgmap v14360: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/661747932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:15 smithi022 ceph-mon[31360]: pgmap v14360: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/661747932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:15 smithi022 ceph-mon[40250]: pgmap v14360: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/661747932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:17.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:57:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:57:17.682 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:57:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:17 smithi122 ceph-mon[32255]: pgmap v14361: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:17 smithi022 ceph-mon[31360]: pgmap v14361: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:17 smithi022 ceph-mon[40250]: pgmap v14361: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:19.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:19 smithi122 ceph-mon[32255]: pgmap v14362: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:19 smithi022 ceph-mon[31360]: pgmap v14362: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:19 smithi022 ceph-mon[40250]: pgmap v14362: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:20] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:57:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1062318312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1062318312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1062318312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:21 smithi122 ceph-mon[32255]: pgmap v14363: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:21 smithi022 ceph-mon[31360]: pgmap v14363: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:21 smithi022 ceph-mon[40250]: pgmap v14363: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:57:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:57:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:23 smithi122 ceph-mon[32255]: pgmap v14364: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:23 smithi022 ceph-mon[31360]: pgmap v14364: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:23 smithi022 ceph-mon[40250]: pgmap v14364: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:25.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:25.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:25.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:25.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:25 smithi022 ceph-mon[31360]: pgmap v14365: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/188446290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:25 smithi022 ceph-mon[40250]: pgmap v14365: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/188446290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:25 smithi122 ceph-mon[32255]: pgmap v14365: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/188446290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:27.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:57:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:57:27.684 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:57:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:27 smithi022 ceph-mon[31360]: pgmap v14366: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:27 smithi022 ceph-mon[40250]: pgmap v14366: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:27 smithi122 ceph-mon[32255]: pgmap v14366: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:29 smithi122 ceph-mon[32255]: pgmap v14367: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:29 smithi022 ceph-mon[31360]: pgmap v14367: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:29 smithi022 ceph-mon[40250]: pgmap v14367: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:30] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:57:30.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:31.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:31.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:31.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:31 smithi122 ceph-mon[32255]: pgmap v14368: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/930711895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:31 smithi022 ceph-mon[31360]: pgmap v14368: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/930711895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:31 smithi022 ceph-mon[40250]: pgmap v14368: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/930711895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:57:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:57:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:33 smithi122 ceph-mon[32255]: pgmap v14369: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:33 smithi022 ceph-mon[31360]: pgmap v14369: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:33 smithi022 ceph-mon[40250]: pgmap v14369: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:57:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:57:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:57:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:57:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:57:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:57:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:57:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:57:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:57:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:57:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:57:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:57:36.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:35 smithi122 ceph-mon[32255]: pgmap v14370: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:35 smithi022 ceph-mon[31360]: pgmap v14370: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:35 smithi022 ceph-mon[40250]: pgmap v14370: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:36.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:36.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:36.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:37.128 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2776364824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2776364824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2776364824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:37.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:57:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:57:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:57:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:37 smithi122 ceph-mon[32255]: pgmap v14371: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:37 smithi022 ceph-mon[31360]: pgmap v14371: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:37 smithi022 ceph-mon[40250]: pgmap v14371: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:39 smithi122 ceph-mon[32255]: pgmap v14372: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:39 smithi022 ceph-mon[31360]: pgmap v14372: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:39 smithi022 ceph-mon[40250]: pgmap v14372: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:40] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:57:41.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:41.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:41.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:41.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:41 smithi022 ceph-mon[31360]: pgmap v14373: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2269045430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:41 smithi022 ceph-mon[40250]: pgmap v14373: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2269045430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:41 smithi122 ceph-mon[32255]: pgmap v14373: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2269045430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:57:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:57:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:43 smithi122 ceph-mon[32255]: pgmap v14374: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:43 smithi022 ceph-mon[31360]: pgmap v14374: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:43 smithi022 ceph-mon[40250]: pgmap v14374: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:45 smithi122 ceph-mon[32255]: pgmap v14375: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:45 smithi022 ceph-mon[31360]: pgmap v14375: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:45 smithi022 ceph-mon[40250]: pgmap v14375: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:46.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:47.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:47.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:47.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:57:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:57:47.689 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:57:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:47 smithi022 ceph-mon[31360]: pgmap v14376: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4257257191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:47 smithi022 ceph-mon[40250]: pgmap v14376: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4257257191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:47 smithi122 ceph-mon[32255]: pgmap v14376: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4257257191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:48.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:49 smithi122 ceph-mon[32255]: pgmap v14377: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:49 smithi022 ceph-mon[31360]: pgmap v14377: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:49 smithi022 ceph-mon[40250]: pgmap v14377: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:57:52.142 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:52.142 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:51 smithi022 ceph-mon[31360]: pgmap v14378: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:52.143 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:51 smithi022 ceph-mon[40250]: pgmap v14378: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:51 smithi122 ceph-mon[32255]: pgmap v14378: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:52.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:52.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:52.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:57:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:57:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3026079566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3026079566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:53.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3026079566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:53 smithi122 ceph-mon[32255]: pgmap v14379: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:53 smithi022 ceph-mon[31360]: pgmap v14379: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:53 smithi022 ceph-mon[40250]: pgmap v14379: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:55 smithi122 ceph-mon[32255]: pgmap v14380: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:55 smithi022 ceph-mon[31360]: pgmap v14380: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:55 smithi022 ceph-mon[40250]: pgmap v14380: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:57:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:57:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:57:57.514 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:57:57.691 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:57:57.691 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:57 smithi022 ceph-mon[31360]: pgmap v14381: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:57.691 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:57.692 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:57 smithi022 ceph-mon[40250]: pgmap v14381: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:57.692 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:57.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:57:57.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:57:57.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:57:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:57 smithi122 ceph-mon[32255]: pgmap v14381: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:57:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:57:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/416947049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/416947049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:57:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/416947049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:57:59 smithi022 ceph-mon[31360]: pgmap v14382: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:57:59 smithi022 ceph-mon[40250]: pgmap v14382: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:57:59 smithi122 ceph-mon[32255]: pgmap v14382: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:58:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:58:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:58:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:58:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:58:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:58:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:58:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:58:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:58:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:58:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:58:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:58:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:58:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:01 smithi022 ceph-mon[31360]: pgmap v14383: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:02.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:01 smithi022 ceph-mon[40250]: pgmap v14383: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:01 smithi122 ceph-mon[32255]: pgmap v14383: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:58:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:58:02.886 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:03.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:03.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:03.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:03 smithi122 ceph-mon[32255]: pgmap v14384: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/835119927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:03 smithi022 ceph-mon[31360]: pgmap v14384: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/835119927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:03 smithi022 ceph-mon[40250]: pgmap v14384: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/835119927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:05 smithi122 ceph-mon[32255]: pgmap v14385: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:05 smithi022 ceph-mon[31360]: pgmap v14385: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:05 smithi022 ceph-mon[40250]: pgmap v14385: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:07.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:58:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:58:07.695 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:58:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:07 smithi022 ceph-mon[31360]: pgmap v14386: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:07 smithi022 ceph-mon[40250]: pgmap v14386: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:07 smithi122 ceph-mon[32255]: pgmap v14386: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:08.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:08.675 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:08.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:08.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2035916947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2035916947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2035916947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:09 smithi022 ceph-mon[40250]: pgmap v14387: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:09 smithi022 ceph-mon[31360]: pgmap v14387: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:09 smithi122 ceph-mon[32255]: pgmap v14387: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T01:58:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:10] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T01:58:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:11 smithi122 ceph-mon[32255]: pgmap v14388: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:11 smithi022 ceph-mon[31360]: pgmap v14388: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:11 smithi022 ceph-mon[40250]: pgmap v14388: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:58:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:58:13.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:13 smithi022 ceph-mon[31360]: pgmap v14389: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:13 smithi022 ceph-mon[40250]: pgmap v14389: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:14.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:14.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:14.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:13 smithi122 ceph-mon[32255]: pgmap v14389: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1002047887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1002047887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1002047887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:15 smithi122 ceph-mon[32255]: pgmap v14390: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:15 smithi022 ceph-mon[31360]: pgmap v14390: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:15 smithi022 ceph-mon[40250]: pgmap v14390: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:17.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:58:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:58:17.697 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:58:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:17 smithi022 ceph-mon[31360]: pgmap v14391: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:17 smithi022 ceph-mon[40250]: pgmap v14391: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:17 smithi122 ceph-mon[32255]: pgmap v14391: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:19.050 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:19.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:19.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:19.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:19 smithi122 ceph-mon[32255]: pgmap v14392: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3591575063' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:19 smithi022 ceph-mon[31360]: pgmap v14392: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3591575063' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:19 smithi022 ceph-mon[40250]: pgmap v14392: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3591575063' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:20] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:58:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:21 smithi122 ceph-mon[32255]: pgmap v14393: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:21 smithi022 ceph-mon[31360]: pgmap v14393: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:21 smithi022 ceph-mon[40250]: pgmap v14393: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:58:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:58:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:23 smithi122 ceph-mon[32255]: pgmap v14394: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:23 smithi022 ceph-mon[31360]: pgmap v14394: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:23 smithi022 ceph-mon[40250]: pgmap v14394: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T01:58:24.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:24.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:24.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:24.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2219063352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2219063352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2219063352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:25 smithi122 ceph-mon[32255]: pgmap v14395: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:25 smithi022 ceph-mon[31360]: pgmap v14395: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:25 smithi022 ceph-mon[40250]: pgmap v14395: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T01:58:27.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:58:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:58:27.700 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:58:27.701 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:27 smithi022 ceph-mon[40250]: pgmap v14396: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:27.701 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:27 smithi022 ceph-mon[31360]: pgmap v14396: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:27 smithi122 ceph-mon[32255]: pgmap v14396: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:29.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:30.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:30.159 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:30.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:30.160 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:29 smithi022 ceph-mon[31360]: pgmap v14397: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:30.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:29 smithi022 ceph-mon[40250]: pgmap v14397: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:29 smithi122 ceph-mon[32255]: pgmap v14397: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:30] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:58:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3978503693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3978503693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3978503693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:31 smithi122 ceph-mon[32255]: pgmap v14398: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:31 smithi022 ceph-mon[31360]: pgmap v14398: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:31 smithi022 ceph-mon[40250]: pgmap v14398: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:58:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:58:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:33 smithi122 ceph-mon[32255]: pgmap v14399: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:33 smithi022 ceph-mon[31360]: pgmap v14399: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:33 smithi022 ceph-mon[40250]: pgmap v14399: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:35.161 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:35.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:58:35.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:58:35.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:58:35.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:58:35.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:58:35.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:58:35.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:58:35.164 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:58:35.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:58:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:58:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:58:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:58:35.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:35.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:35.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:35 smithi122 ceph-mon[32255]: pgmap v14400: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/561671393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:35 smithi022 ceph-mon[31360]: pgmap v14400: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/561671393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:35 smithi022 ceph-mon[40250]: pgmap v14400: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/561671393' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:37.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:58:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:58:37.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:58:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:37 smithi022 ceph-mon[31360]: pgmap v14401: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:37 smithi022 ceph-mon[40250]: pgmap v14401: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:37 smithi122 ceph-mon[32255]: pgmap v14401: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:39 smithi122 ceph-mon[32255]: pgmap v14402: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:39 smithi022 ceph-mon[31360]: pgmap v14402: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:39 smithi022 ceph-mon[40250]: pgmap v14402: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:40] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T01:58:40.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:40.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:40.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:40.901 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:41 smithi122 ceph-mon[32255]: pgmap v14403: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/182110128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:41 smithi022 ceph-mon[31360]: pgmap v14403: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/182110128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:41 smithi022 ceph-mon[40250]: pgmap v14403: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/182110128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:58:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:58:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:43 smithi122 ceph-mon[32255]: pgmap v14404: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:43 smithi022 ceph-mon[31360]: pgmap v14404: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:43 smithi022 ceph-mon[40250]: pgmap v14404: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:45.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:45 smithi122 ceph-mon[32255]: pgmap v14405: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:46.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:46.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:46.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:46.271 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:45 smithi022 ceph-mon[40250]: pgmap v14405: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:46.271 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:45 smithi022 ceph-mon[31360]: pgmap v14405: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:47.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2142395274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2142395274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2142395274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:58:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:58:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:58:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:47 smithi122 ceph-mon[32255]: pgmap v14406: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:47 smithi022 ceph-mon[31360]: pgmap v14406: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:47 smithi022 ceph-mon[40250]: pgmap v14406: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:49 smithi122 ceph-mon[32255]: pgmap v14407: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:49 smithi022 ceph-mon[31360]: pgmap v14407: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:49 smithi022 ceph-mon[40250]: pgmap v14407: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:50] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:58:51.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:51.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:51 smithi022 ceph-mon[31360]: pgmap v14408: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/973220497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:51 smithi022 ceph-mon[40250]: pgmap v14408: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/973220497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:51 smithi122 ceph-mon[32255]: pgmap v14408: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/973220497' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:58:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:58:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:53 smithi122 ceph-mon[32255]: pgmap v14409: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:53 smithi022 ceph-mon[31360]: pgmap v14409: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:53 smithi022 ceph-mon[40250]: pgmap v14409: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:55 smithi122 ceph-mon[32255]: pgmap v14410: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:55 smithi022 ceph-mon[31360]: pgmap v14410: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:55 smithi022 ceph-mon[40250]: pgmap v14410: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:58:56.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:58:57.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:58:57.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:58:57.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:58:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:58:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:58:57.704 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:58:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:57 smithi022 ceph-mon[31360]: pgmap v14411: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1088092072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:57 smithi022 ceph-mon[40250]: pgmap v14411: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1088092072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:58.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:58:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:57 smithi122 ceph-mon[32255]: pgmap v14411: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:58:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1088092072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:58:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:58:59 smithi122 ceph-mon[32255]: pgmap v14412: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:58:59 smithi022 ceph-mon[31360]: pgmap v14412: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:58:59 smithi022 ceph-mon[40250]: pgmap v14412: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:00] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T01:59:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:59:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:59:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:59:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:59:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:59:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:59:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:59:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:59:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T01:59:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T01:59:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T01:59:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T01:59:02.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:01 smithi122 ceph-mon[32255]: pgmap v14413: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:01 smithi022 ceph-mon[31360]: pgmap v14413: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:01 smithi022 ceph-mon[40250]: pgmap v14413: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:02.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:02.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:02.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:59:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:59:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/303194509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/303194509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/303194509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:03 smithi122 ceph-mon[32255]: pgmap v14414: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:03 smithi022 ceph-mon[31360]: pgmap v14414: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:03 smithi022 ceph-mon[40250]: pgmap v14414: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:05 smithi122 ceph-mon[32255]: pgmap v14415: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:05 smithi022 ceph-mon[31360]: pgmap v14415: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:05 smithi022 ceph-mon[40250]: pgmap v14415: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:07.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:59:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:59:07.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:59:07.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:07.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:07.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:07 smithi022 ceph-mon[31360]: pgmap v14416: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:08.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:07 smithi022 ceph-mon[40250]: pgmap v14416: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:07 smithi122 ceph-mon[32255]: pgmap v14416: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/808044301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/808044301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/808044301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:09 smithi022 ceph-mon[31360]: pgmap v14417: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:10.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:09 smithi022 ceph-mon[40250]: pgmap v14417: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:10.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:09 smithi122 ceph-mon[32255]: pgmap v14417: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:10] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T01:59:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:11 smithi122 ceph-mon[32255]: pgmap v14418: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:11 smithi022 ceph-mon[31360]: pgmap v14418: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:11 smithi022 ceph-mon[40250]: pgmap v14418: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:59:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:59:12.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:13.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:13.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:13.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:13 smithi122 ceph-mon[32255]: pgmap v14419: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1745344354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:13 smithi022 ceph-mon[31360]: pgmap v14419: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1745344354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:13 smithi022 ceph-mon[40250]: pgmap v14419: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1745344354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:15 smithi122 ceph-mon[32255]: pgmap v14420: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:15 smithi022 ceph-mon[31360]: pgmap v14420: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:15 smithi022 ceph-mon[40250]: pgmap v14420: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:59:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:59:17.710 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:59:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:17 smithi022 ceph-mon[31360]: pgmap v14421: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:17 smithi022 ceph-mon[40250]: pgmap v14421: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:18.137 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:17 smithi122 ceph-mon[32255]: pgmap v14421: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:18.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:18.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:18.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2252762334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2252762334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2252762334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:19 smithi022 ceph-mon[31360]: pgmap v14422: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:19 smithi022 ceph-mon[40250]: pgmap v14422: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:19 smithi122 ceph-mon[32255]: pgmap v14422: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:20] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:59:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:21 smithi122 ceph-mon[32255]: pgmap v14423: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:21 smithi022 ceph-mon[31360]: pgmap v14423: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:21 smithi022 ceph-mon[40250]: pgmap v14423: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:59:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:59:23.503 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:23.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:23.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:23.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:23 smithi122 ceph-mon[32255]: pgmap v14424: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:23 smithi022 ceph-mon[31360]: pgmap v14424: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:23 smithi022 ceph-mon[40250]: pgmap v14424: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2957022484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2957022484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2957022484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:25 smithi122 ceph-mon[32255]: pgmap v14425: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:25 smithi022 ceph-mon[31360]: pgmap v14425: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:25 smithi022 ceph-mon[40250]: pgmap v14425: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:59:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:59:27.712 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:59:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:27 smithi022 ceph-mon[31360]: pgmap v14426: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:27 smithi022 ceph-mon[40250]: pgmap v14426: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:27 smithi122 ceph-mon[32255]: pgmap v14426: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:28.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:29.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:29.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:29.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:29 smithi122 ceph-mon[32255]: pgmap v14427: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2828967048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:29 smithi022 ceph-mon[31360]: pgmap v14427: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2828967048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:29 smithi022 ceph-mon[40250]: pgmap v14427: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2828967048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:30] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T01:59:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:31 smithi122 ceph-mon[32255]: pgmap v14428: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:31 smithi022 ceph-mon[31360]: pgmap v14428: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:31 smithi022 ceph-mon[40250]: pgmap v14428: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:59:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:59:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:33 smithi122 ceph-mon[32255]: pgmap v14429: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:34.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:34.252 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:33 smithi022 ceph-mon[31360]: pgmap v14429: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:34.253 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:33 smithi022 ceph-mon[40250]: pgmap v14429: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:34.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:34.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:34.618 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:34.868 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:59:34.868 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:59:34.868 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:59:34.869 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:59:34.869 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2369845199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:35.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:59:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:59:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:59:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:59:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2369845199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:59:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:59:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T01:59:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T01:59:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2369845199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:35 smithi122 ceph-mon[32255]: pgmap v14430: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:35 smithi022 ceph-mon[31360]: pgmap v14430: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:35 smithi022 ceph-mon[40250]: pgmap v14430: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:59:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:59:37.715 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:59:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:37 smithi022 ceph-mon[31360]: pgmap v14431: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:37 smithi022 ceph-mon[40250]: pgmap v14431: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:37 smithi122 ceph-mon[32255]: pgmap v14431: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:39.620 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:39.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:39.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:39.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:39.987 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:39 smithi022 ceph-mon[31360]: pgmap v14432: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:39.988 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:39 smithi022 ceph-mon[40250]: pgmap v14432: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:39 smithi122 ceph-mon[32255]: pgmap v14432: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:40] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T01:59:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1684757142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1684757142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:41.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1684757142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:41 smithi122 ceph-mon[32255]: pgmap v14433: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:41 smithi022 ceph-mon[31360]: pgmap v14433: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:41 smithi022 ceph-mon[40250]: pgmap v14433: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:59:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:59:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:43 smithi122 ceph-mon[32255]: pgmap v14434: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:43 smithi022 ceph-mon[31360]: pgmap v14434: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:43 smithi022 ceph-mon[40250]: pgmap v14434: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:44.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:45.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:45.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:45.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:45 smithi122 ceph-mon[32255]: pgmap v14435: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3025019511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:45 smithi022 ceph-mon[31360]: pgmap v14435: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3025019511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:45 smithi022 ceph-mon[40250]: pgmap v14435: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3025019511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:47.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:59:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:59:47.718 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:59:47.719 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:47 smithi022 ceph-mon[40250]: pgmap v14436: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:47.719 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:47 smithi022 ceph-mon[31360]: pgmap v14436: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:47 smithi122 ceph-mon[32255]: pgmap v14436: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:49 smithi122 ceph-mon[32255]: pgmap v14437: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:49 smithi022 ceph-mon[31360]: pgmap v14437: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:49 smithi022 ceph-mon[40250]: pgmap v14437: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:50.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 01:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:50] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T01:59:50.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:50.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:50.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/353129007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/353129007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/353129007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:51 smithi122 ceph-mon[32255]: pgmap v14438: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:51 smithi022 ceph-mon[31360]: pgmap v14438: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:51 smithi022 ceph-mon[40250]: pgmap v14438: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 01:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T01:59:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T01:59:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:53 smithi122 ceph-mon[32255]: pgmap v14439: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:53 smithi022 ceph-mon[31360]: pgmap v14439: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:53 smithi022 ceph-mon[40250]: pgmap v14439: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:55.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T01:59:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:55 smithi022 ceph-mon[31360]: pgmap v14440: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:56.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:55 smithi022 ceph-mon[40250]: pgmap v14440: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:56.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T01:59:56.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T01:59:56.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T01:59:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:55 smithi122 ceph-mon[32255]: pgmap v14440: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T01:59:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2338935694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2338935694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2338935694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T01:59:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 01:59:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T01:59:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 01:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:01:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T01:59:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:57 smithi122 ceph-mon[32255]: pgmap v14441: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:57 smithi022 ceph-mon[31360]: pgmap v14441: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T01:59:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:57 smithi022 ceph-mon[40250]: pgmap v14441: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T01:59:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 01:59:59 smithi122 ceph-mon[32255]: pgmap v14442: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 01:59:59 smithi022 ceph-mon[31360]: pgmap v14442: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 01:59:59 smithi022 ceph-mon[40250]: pgmap v14442: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:00] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:00:01.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:01.094 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T02:00:01.094 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:00:01.094 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:00:01.095 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:00:01.095 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T02:00:01.095 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:00:01.095 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:00:01.096 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:00:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T02:00:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:00:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:00:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:00:01.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:01.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:01.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:02 smithi022 ceph-mon[31360]: pgmap v14443: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:00:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3944463827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:02 smithi022 ceph-mon[40250]: pgmap v14443: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:00:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3944463827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:00:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:00:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:02 smithi122 ceph-mon[32255]: pgmap v14443: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:00:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3944463827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:04 smithi022 ceph-mon[31360]: pgmap v14444: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:04 smithi022 ceph-mon[40250]: pgmap v14444: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:04 smithi122 ceph-mon[32255]: pgmap v14444: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:06.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:06 smithi022 ceph-mon[31360]: pgmap v14445: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:06 smithi022 ceph-mon[40250]: pgmap v14445: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:06 smithi122 ceph-mon[32255]: pgmap v14445: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:06.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:06.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:06.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:00:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:00:07.722 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:00:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:07 smithi022 ceph-mon[31360]: pgmap v14446: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2892314350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:08.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:07 smithi022 ceph-mon[40250]: pgmap v14446: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2892314350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:08.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:07 smithi122 ceph-mon[32255]: pgmap v14446: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2892314350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:09 smithi122 ceph-mon[32255]: pgmap v14447: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:09 smithi022 ceph-mon[31360]: pgmap v14447: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:09 smithi022 ceph-mon[40250]: pgmap v14447: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:10] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:00:11.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:11 smithi122 ceph-mon[32255]: pgmap v14448: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:12.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:12.203 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:12.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:12.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:11 smithi022 ceph-mon[31360]: pgmap v14448: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:12.204 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:11 smithi022 ceph-mon[40250]: pgmap v14448: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:00:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:00:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/198681388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/198681388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/198681388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:13 smithi122 ceph-mon[32255]: pgmap v14449: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:13 smithi022 ceph-mon[31360]: pgmap v14449: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:13 smithi022 ceph-mon[40250]: pgmap v14449: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:15 smithi122 ceph-mon[32255]: pgmap v14450: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:15 smithi022 ceph-mon[31360]: pgmap v14450: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:15 smithi022 ceph-mon[40250]: pgmap v14450: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:17.205 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:00:17.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:17.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:17.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:00:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:00:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:17 smithi022 ceph-mon[31360]: pgmap v14451: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1857767947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:17 smithi022 ceph-mon[40250]: pgmap v14451: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1857767947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:18.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:17 smithi122 ceph-mon[32255]: pgmap v14451: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:18.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:18.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1857767947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:19 smithi122 ceph-mon[32255]: pgmap v14452: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:19 smithi022 ceph-mon[31360]: pgmap v14452: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:19 smithi022 ceph-mon[40250]: pgmap v14452: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:00:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:21 smithi122 ceph-mon[32255]: pgmap v14453: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:21 smithi022 ceph-mon[31360]: pgmap v14453: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:21 smithi022 ceph-mon[40250]: pgmap v14453: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:22.577 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:00:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:00:22.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:22.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:22.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:23 smithi122 ceph-mon[32255]: pgmap v14454: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3653064082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:23 smithi022 ceph-mon[31360]: pgmap v14454: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3653064082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:23 smithi022 ceph-mon[40250]: pgmap v14454: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3653064082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:25 smithi122 ceph-mon[32255]: pgmap v14455: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:25 smithi022 ceph-mon[31360]: pgmap v14455: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:25 smithi022 ceph-mon[40250]: pgmap v14455: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:00:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:00:27.731 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:00:27.732 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:27 smithi022 ceph-mon[40250]: pgmap v14456: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:27.733 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:27.733 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:27 smithi022 ceph-mon[31360]: pgmap v14456: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:27.733 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:27.948 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:27 smithi122 ceph-mon[32255]: pgmap v14456: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:28.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:28.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:28.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3275783475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3275783475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3275783475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:29 smithi022 ceph-mon[31360]: pgmap v14457: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:29 smithi022 ceph-mon[40250]: pgmap v14457: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:29 smithi122 ceph-mon[32255]: pgmap v14457: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:00:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:31 smithi122 ceph-mon[32255]: pgmap v14458: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:31 smithi022 ceph-mon[31360]: pgmap v14458: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:31 smithi022 ceph-mon[40250]: pgmap v14458: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:00:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:00:33.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:33.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:33.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:33.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:33 smithi022 ceph-mon[31360]: pgmap v14459: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/536345860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:33 smithi022 ceph-mon[40250]: pgmap v14459: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/536345860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:33 smithi122 ceph-mon[32255]: pgmap v14459: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/536345860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:00:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:00:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:00:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:00:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:00:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:00:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:00:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:00:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:00:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:00:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:00:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:00:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:35 smithi122 ceph-mon[32255]: pgmap v14460: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:35 smithi022 ceph-mon[31360]: pgmap v14460: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:35 smithi022 ceph-mon[40250]: pgmap v14460: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:00:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:00:37.730 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:00:37.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:37 smithi022 ceph-mon[40250]: pgmap v14461: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:37 smithi022 ceph-mon[31360]: pgmap v14461: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:37 smithi122 ceph-mon[32255]: pgmap v14461: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:38.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:39.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:39.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:39 smithi122 ceph-mon[32255]: pgmap v14462: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3818344621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:39 smithi022 ceph-mon[31360]: pgmap v14462: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3818344621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:39 smithi022 ceph-mon[40250]: pgmap v14462: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3818344621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:40] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:00:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:41 smithi122 ceph-mon[32255]: pgmap v14463: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:41 smithi022 ceph-mon[31360]: pgmap v14463: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:41 smithi022 ceph-mon[40250]: pgmap v14463: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:00:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:00:44.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:43 smithi122 ceph-mon[32255]: pgmap v14464: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:43 smithi022 ceph-mon[31360]: pgmap v14464: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:43 smithi022 ceph-mon[40250]: pgmap v14464: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:44.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:44.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:44.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/398283648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/398283648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/398283648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:45 smithi122 ceph-mon[32255]: pgmap v14465: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:45 smithi022 ceph-mon[31360]: pgmap v14465: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:45 smithi022 ceph-mon[40250]: pgmap v14465: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:47.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:00:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:00:47.733 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:00:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:47 smithi022 ceph-mon[31360]: pgmap v14466: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:47 smithi022 ceph-mon[40250]: pgmap v14466: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:47 smithi122 ceph-mon[32255]: pgmap v14466: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:49.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:49.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:49.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:49.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:49 smithi122 ceph-mon[32255]: pgmap v14467: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3250635798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:49 smithi022 ceph-mon[31360]: pgmap v14467: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3250635798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:49 smithi022 ceph-mon[40250]: pgmap v14467: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3250635798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:00:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:51 smithi122 ceph-mon[32255]: pgmap v14468: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:51 smithi022 ceph-mon[31360]: pgmap v14468: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:51 smithi022 ceph-mon[40250]: pgmap v14468: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:00:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:00:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:53 smithi122 ceph-mon[32255]: pgmap v14469: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:53 smithi022 ceph-mon[31360]: pgmap v14469: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:53 smithi022 ceph-mon[40250]: pgmap v14469: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:54.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:00:55.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:00:55.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:00:55.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:00:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:55 smithi122 ceph-mon[32255]: pgmap v14470: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1658858454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:55 smithi022 ceph-mon[31360]: pgmap v14470: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1658858454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:55 smithi022 ceph-mon[40250]: pgmap v14470: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:00:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1658858454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:00:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:00:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:00:57.734 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:00:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:57 smithi022 ceph-mon[31360]: pgmap v14471: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:57 smithi022 ceph-mon[40250]: pgmap v14471: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:00:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:57 smithi122 ceph-mon[32255]: pgmap v14471: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:00:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:00:59 smithi122 ceph-mon[32255]: pgmap v14472: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:00.173 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:00:59 smithi022 ceph-mon[31360]: pgmap v14472: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:00.174 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:00:59 smithi022 ceph-mon[40250]: pgmap v14472: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:01:00.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:00.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:00.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:01.152 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3519994640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3519994640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3519994640' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[31360]: pgmap v14473: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:01:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:01:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:01:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:01:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[40250]: pgmap v14473: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:01:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:01:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:01:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:01:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:01 smithi122 ceph-mon[32255]: pgmap v14473: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:01:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:01:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:01:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:01:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:01:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:01:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:03 smithi122 ceph-mon[32255]: pgmap v14474: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:03 smithi022 ceph-mon[31360]: pgmap v14474: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:03 smithi022 ceph-mon[40250]: pgmap v14474: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:05.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:05.901 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:05 smithi022 ceph-mon[40250]: pgmap v14475: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:05.901 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2400545006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:05.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:05 smithi022 ceph-mon[31360]: pgmap v14475: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:05.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2400545006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:05.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:05.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:05.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:05 smithi122 ceph-mon[32255]: pgmap v14475: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2400545006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:01:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:01:07.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:01:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:07 smithi022 ceph-mon[31360]: pgmap v14476: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:07 smithi022 ceph-mon[40250]: pgmap v14476: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:07 smithi122 ceph-mon[32255]: pgmap v14476: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:08.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:09 smithi122 ceph-mon[32255]: pgmap v14477: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:09 smithi022 ceph-mon[31360]: pgmap v14477: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:09 smithi022 ceph-mon[40250]: pgmap v14477: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:10] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:01:10.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:11.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:11.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:11.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:11 smithi122 ceph-mon[32255]: pgmap v14478: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1618863741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:11 smithi022 ceph-mon[31360]: pgmap v14478: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1618863741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:11 smithi022 ceph-mon[40250]: pgmap v14478: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1618863741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:01:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:01:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:13 smithi122 ceph-mon[32255]: pgmap v14479: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:13 smithi022 ceph-mon[31360]: pgmap v14479: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:13 smithi022 ceph-mon[40250]: pgmap v14479: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:15 smithi122 ceph-mon[32255]: pgmap v14480: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:15 smithi022 ceph-mon[31360]: pgmap v14480: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:15 smithi022 ceph-mon[40250]: pgmap v14480: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:16.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:16.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:16.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:16.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2720860719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2720860719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:17.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2720860719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:01:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:01:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:01:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:17 smithi122 ceph-mon[32255]: pgmap v14481: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:17 smithi022 ceph-mon[31360]: pgmap v14481: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:17 smithi022 ceph-mon[40250]: pgmap v14481: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:19 smithi122 ceph-mon[32255]: pgmap v14482: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:19 smithi022 ceph-mon[31360]: pgmap v14482: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:19 smithi022 ceph-mon[40250]: pgmap v14482: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:01:21.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:22.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:22.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:22.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:21 smithi122 ceph-mon[32255]: pgmap v14483: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:21 smithi022 ceph-mon[31360]: pgmap v14483: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:21 smithi022 ceph-mon[40250]: pgmap v14483: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:01:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:01:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/932335033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/932335033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:23.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/932335033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:23 smithi122 ceph-mon[32255]: pgmap v14484: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:23 smithi022 ceph-mon[31360]: pgmap v14484: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:23 smithi022 ceph-mon[40250]: pgmap v14484: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:25 smithi122 ceph-mon[32255]: pgmap v14485: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:25 smithi022 ceph-mon[31360]: pgmap v14485: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:25 smithi022 ceph-mon[40250]: pgmap v14485: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:27.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:27.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:27.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:01:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:01:27.739 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:01:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:27 smithi022 ceph-mon[31360]: pgmap v14486: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1142852887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:27 smithi022 ceph-mon[40250]: pgmap v14486: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:28.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1142852887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:27 smithi122 ceph-mon[32255]: pgmap v14486: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1142852887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:29 smithi122 ceph-mon[32255]: pgmap v14487: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:29 smithi022 ceph-mon[31360]: pgmap v14487: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:29 smithi022 ceph-mon[40250]: pgmap v14487: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:01:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:31 smithi122 ceph-mon[32255]: pgmap v14488: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:31 smithi022 ceph-mon[31360]: pgmap v14488: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:31 smithi022 ceph-mon[40250]: pgmap v14488: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:32.387 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:01:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:01:32.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:32.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:32.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2145002960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2145002960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2145002960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:33 smithi122 ceph-mon[32255]: pgmap v14489: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:33 smithi022 ceph-mon[31360]: pgmap v14489: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:33 smithi022 ceph-mon[40250]: pgmap v14489: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:01:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:01:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:01:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:01:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:01:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:01:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:01:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:01:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:01:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:35 smithi122 ceph-mon[32255]: pgmap v14490: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:35 smithi022 ceph-mon[31360]: pgmap v14490: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:35 smithi022 ceph-mon[40250]: pgmap v14490: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:01:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:01:37.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:01:37.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:37 smithi022 ceph-mon[40250]: pgmap v14491: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:37.744 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:37.757 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:37 smithi022 ceph-mon[31360]: pgmap v14491: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:38.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:38.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:38.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:37 smithi122 ceph-mon[32255]: pgmap v14491: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1594951921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:39.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1594951921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1594951921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:39 smithi022 ceph-mon[31360]: pgmap v14492: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:40.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:39 smithi022 ceph-mon[40250]: pgmap v14492: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:39 smithi122 ceph-mon[32255]: pgmap v14492: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:40] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T02:01:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:41 smithi122 ceph-mon[32255]: pgmap v14493: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:41 smithi022 ceph-mon[31360]: pgmap v14493: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:41 smithi022 ceph-mon[40250]: pgmap v14493: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:01:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:01:43.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:43 smithi122 ceph-mon[32255]: pgmap v14494: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/413505866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:43 smithi022 ceph-mon[31360]: pgmap v14494: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/413505866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:43 smithi022 ceph-mon[40250]: pgmap v14494: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/413505866' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:45 smithi122 ceph-mon[32255]: pgmap v14495: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:45 smithi022 ceph-mon[31360]: pgmap v14495: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:45 smithi022 ceph-mon[40250]: pgmap v14495: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:47.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:01:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:01:47.744 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:01:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:47 smithi022 ceph-mon[31360]: pgmap v14496: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:47 smithi022 ceph-mon[40250]: pgmap v14496: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:47 smithi122 ceph-mon[32255]: pgmap v14496: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:48.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:48.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:48.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:48.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3748205902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3748205902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3748205902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:49 smithi122 ceph-mon[32255]: pgmap v14497: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:49 smithi022 ceph-mon[31360]: pgmap v14497: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:49 smithi022 ceph-mon[40250]: pgmap v14497: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:50] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T02:01:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:51 smithi122 ceph-mon[32255]: pgmap v14498: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:52.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:51 smithi022 ceph-mon[31360]: pgmap v14498: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:51 smithi022 ceph-mon[40250]: pgmap v14498: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:01:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:01:53.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:53 smithi122 ceph-mon[32255]: pgmap v14499: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:54.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:54.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:54.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:01:54.241 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:53 smithi022 ceph-mon[31360]: pgmap v14499: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:54.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:53 smithi022 ceph-mon[40250]: pgmap v14499: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1577678121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1577678121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1577678121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:01:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:55 smithi122 ceph-mon[32255]: pgmap v14500: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:55 smithi022 ceph-mon[31360]: pgmap v14500: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:55 smithi022 ceph-mon[40250]: pgmap v14500: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:01:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:01:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:01:57.747 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:01:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:57 smithi022 ceph-mon[31360]: pgmap v14501: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:57 smithi022 ceph-mon[40250]: pgmap v14501: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:57 smithi122 ceph-mon[32255]: pgmap v14501: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:01:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:01:59.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:01:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:01:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:01:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:59 smithi122 ceph-mon[32255]: pgmap v14502: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:01:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3228458485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:59 smithi022 ceph-mon[31360]: pgmap v14502: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:01:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3228458485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:59 smithi022 ceph-mon[40250]: pgmap v14502: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:01:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3228458485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:00] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T02:02:02.180 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:01 smithi122 ceph-mon[32255]: pgmap v14503: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:02.181 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:02:02.181 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:02:02.181 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:02:02.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[31360]: pgmap v14503: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:02.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:02:02.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:02:02.204 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:02:02.204 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[40250]: pgmap v14503: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:02.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:02:02.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:02:02.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:02:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:02:02.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:02:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:02:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:02:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:02:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:04 smithi022 ceph-mon[31360]: pgmap v14504: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:04 smithi022 ceph-mon[40250]: pgmap v14504: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:04.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:04 smithi122 ceph-mon[32255]: pgmap v14504: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:04.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:04.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:04.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4292035100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:05.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4292035100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4292035100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:06 smithi022 ceph-mon[31360]: pgmap v14505: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:06 smithi022 ceph-mon[40250]: pgmap v14505: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:06 smithi122 ceph-mon[32255]: pgmap v14505: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:02:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:02:07.749 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:02:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:07 smithi022 ceph-mon[31360]: pgmap v14506: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:07 smithi022 ceph-mon[40250]: pgmap v14506: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:07 smithi122 ceph-mon[32255]: pgmap v14506: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:09.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:09 smithi122 ceph-mon[32255]: pgmap v14507: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:09 smithi022 ceph-mon[31360]: pgmap v14507: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:09 smithi022 ceph-mon[40250]: pgmap v14507: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:10.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:10.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:10.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:10] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T02:02:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4148267462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4148267462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4148267462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:11 smithi122 ceph-mon[32255]: pgmap v14508: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:11 smithi022 ceph-mon[31360]: pgmap v14508: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:11 smithi022 ceph-mon[40250]: pgmap v14508: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:02:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:02:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:13 smithi122 ceph-mon[32255]: pgmap v14509: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:13 smithi022 ceph-mon[31360]: pgmap v14509: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:13 smithi022 ceph-mon[40250]: pgmap v14509: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:15.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:15.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:15.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:15.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:15 smithi022 ceph-mon[31360]: pgmap v14510: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/981043302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:15 smithi022 ceph-mon[40250]: pgmap v14510: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/981043302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:15 smithi122 ceph-mon[32255]: pgmap v14510: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/981043302' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:02:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:02:17.751 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:02:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:17 smithi022 ceph-mon[31360]: pgmap v14511: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:17 smithi022 ceph-mon[40250]: pgmap v14511: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:17 smithi122 ceph-mon[32255]: pgmap v14511: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:19 smithi122 ceph-mon[32255]: pgmap v14512: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:19 smithi022 ceph-mon[31360]: pgmap v14512: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:19 smithi022 ceph-mon[40250]: pgmap v14512: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:02:20.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:21.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:21.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:21.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:21 smithi122 ceph-mon[32255]: pgmap v14513: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1792659868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:21 smithi022 ceph-mon[31360]: pgmap v14513: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1792659868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:21 smithi022 ceph-mon[40250]: pgmap v14513: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1792659868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:02:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:02:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:23 smithi122 ceph-mon[32255]: pgmap v14514: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:23 smithi022 ceph-mon[31360]: pgmap v14514: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:23 smithi022 ceph-mon[40250]: pgmap v14514: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:26.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:25 smithi122 ceph-mon[32255]: pgmap v14515: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:25 smithi022 ceph-mon[31360]: pgmap v14515: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:25 smithi022 ceph-mon[40250]: pgmap v14515: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:26.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:26.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:26.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1867362890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1867362890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1867362890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:27.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:02:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:02:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:02:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:27 smithi122 ceph-mon[32255]: pgmap v14516: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:27 smithi022 ceph-mon[31360]: pgmap v14516: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:27 smithi022 ceph-mon[40250]: pgmap v14516: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:30.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:02:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:29 smithi122 ceph-mon[32255]: pgmap v14517: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:29 smithi022 ceph-mon[31360]: pgmap v14517: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:29 smithi022 ceph-mon[40250]: pgmap v14517: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:31.840 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:31.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:31.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:31 smithi122 ceph-mon[32255]: pgmap v14518: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3861445283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:31 smithi022 ceph-mon[31360]: pgmap v14518: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3861445283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:31 smithi022 ceph-mon[40250]: pgmap v14518: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3861445283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:02:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:02:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:33 smithi122 ceph-mon[32255]: pgmap v14519: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:33 smithi022 ceph-mon[31360]: pgmap v14519: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:33 smithi022 ceph-mon[40250]: pgmap v14519: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:02:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:02:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:02:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:02:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:02:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:02:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:02:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:02:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:02:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:02:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:02:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:02:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:35 smithi122 ceph-mon[32255]: pgmap v14520: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:35 smithi022 ceph-mon[31360]: pgmap v14520: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:35 smithi022 ceph-mon[40250]: pgmap v14520: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:36.842 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:37.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:37.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:37.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:02:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:02:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:02:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:37 smithi022 ceph-mon[31360]: pgmap v14521: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1787124261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:37 smithi022 ceph-mon[40250]: pgmap v14521: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1787124261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:37 smithi122 ceph-mon[32255]: pgmap v14521: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1787124261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:39 smithi122 ceph-mon[32255]: pgmap v14522: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:39 smithi022 ceph-mon[31360]: pgmap v14522: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:39 smithi022 ceph-mon[40250]: pgmap v14522: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:40] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:02:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:41 smithi122 ceph-mon[32255]: pgmap v14523: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:42.213 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:42.213 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:41 smithi022 ceph-mon[31360]: pgmap v14523: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:42.214 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:41 smithi022 ceph-mon[40250]: pgmap v14523: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:42.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:42.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:42.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:02:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:02:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3609397862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3609397862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3609397862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:43 smithi122 ceph-mon[32255]: pgmap v14524: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:43 smithi022 ceph-mon[31360]: pgmap v14524: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:43 smithi022 ceph-mon[40250]: pgmap v14524: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:45 smithi122 ceph-mon[32255]: pgmap v14525: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:45 smithi022 ceph-mon[31360]: pgmap v14525: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:45 smithi022 ceph-mon[40250]: pgmap v14525: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:47.581 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:47.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:02:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:02:47.756 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:02:47.757 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:47 smithi022 ceph-mon[31360]: pgmap v14526: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:47.757 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:47.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:47 smithi022 ceph-mon[40250]: pgmap v14526: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:47.758 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:47.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:47.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:47.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:47 smithi122 ceph-mon[32255]: pgmap v14526: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1685723158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1685723158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1685723158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:49 smithi022 ceph-mon[31360]: pgmap v14527: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:50.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:49 smithi022 ceph-mon[40250]: pgmap v14527: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:49 smithi122 ceph-mon[32255]: pgmap v14527: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:02:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:51 smithi122 ceph-mon[32255]: pgmap v14528: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:51 smithi022 ceph-mon[31360]: pgmap v14528: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:51 smithi022 ceph-mon[40250]: pgmap v14528: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:02:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:02:52.951 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:53 smithi122 ceph-mon[32255]: pgmap v14529: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3768194583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:53 smithi022 ceph-mon[31360]: pgmap v14529: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3768194583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:53 smithi022 ceph-mon[40250]: pgmap v14529: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3768194583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:55 smithi122 ceph-mon[32255]: pgmap v14530: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:55 smithi022 ceph-mon[31360]: pgmap v14530: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:55 smithi022 ceph-mon[40250]: pgmap v14530: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:02:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:02:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:02:57.764 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:02:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:57 smithi022 ceph-mon[31360]: pgmap v14531: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:57 smithi022 ceph-mon[40250]: pgmap v14531: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:57 smithi122 ceph-mon[32255]: pgmap v14531: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:02:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:02:58.324 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:02:58.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:02:58.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:02:58.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:02:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2305644427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2305644427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:02:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2305644427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:02:59 smithi022 ceph-mon[31360]: pgmap v14532: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:00.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:02:59 smithi022 ceph-mon[40250]: pgmap v14532: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:02:59 smithi122 ceph-mon[32255]: pgmap v14532: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:03:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:01 smithi122 ceph-mon[32255]: pgmap v14533: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:02.258 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:01 smithi022 ceph-mon[31360]: pgmap v14533: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:02.258 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:01 smithi022 ceph-mon[40250]: pgmap v14533: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:02.603 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:03:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:03:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:03:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:03:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:03:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:03:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:03:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:03:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:03:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:03:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:03:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:04.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:04.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:04.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:04 smithi122 ceph-mon[32255]: pgmap v14534: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4270680701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:04 smithi022 ceph-mon[31360]: pgmap v14534: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4270680701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:04 smithi022 ceph-mon[40250]: pgmap v14534: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:03:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4270680701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:06 smithi122 ceph-mon[32255]: pgmap v14535: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:06 smithi022 ceph-mon[31360]: pgmap v14535: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:06 smithi022 ceph-mon[40250]: pgmap v14535: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:03:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:03:07.760 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:03:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:07 smithi022 ceph-mon[31360]: pgmap v14536: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:07 smithi022 ceph-mon[40250]: pgmap v14536: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:07 smithi122 ceph-mon[32255]: pgmap v14536: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:09.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:10.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:10] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:03:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:09 smithi122 ceph-mon[32255]: pgmap v14537: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1985889085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:09 smithi022 ceph-mon[31360]: pgmap v14537: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1985889085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:09 smithi022 ceph-mon[40250]: pgmap v14537: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1985889085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:11 smithi122 ceph-mon[32255]: pgmap v14538: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:11 smithi022 ceph-mon[31360]: pgmap v14538: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:11 smithi022 ceph-mon[40250]: pgmap v14538: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:03:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:03:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:13 smithi122 ceph-mon[32255]: pgmap v14539: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:13 smithi022 ceph-mon[31360]: pgmap v14539: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:13 smithi022 ceph-mon[40250]: pgmap v14539: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:14.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:14.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:14.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3177731318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:15.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3177731318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3177731318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:15 smithi022 ceph-mon[31360]: pgmap v14540: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:15 smithi022 ceph-mon[40250]: pgmap v14540: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:15 smithi122 ceph-mon[32255]: pgmap v14540: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:17.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:03:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:03:17.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:03:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:17 smithi022 ceph-mon[31360]: pgmap v14541: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:17 smithi022 ceph-mon[40250]: pgmap v14541: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:17 smithi122 ceph-mon[32255]: pgmap v14541: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:19.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:20.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:20.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:20.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:20.169 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:19 smithi022 ceph-mon[31360]: pgmap v14542: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:20.169 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:19 smithi022 ceph-mon[40250]: pgmap v14542: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:20.184 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:03:20.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:19 smithi122 ceph-mon[32255]: pgmap v14542: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1684797403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1684797403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1684797403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:21 smithi122 ceph-mon[32255]: pgmap v14543: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:21 smithi022 ceph-mon[31360]: pgmap v14543: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:21 smithi022 ceph-mon[40250]: pgmap v14543: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:03:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:03:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:23 smithi022 ceph-mon[31360]: pgmap v14544: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:23 smithi022 ceph-mon[40250]: pgmap v14544: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:23 smithi122 ceph-mon[32255]: pgmap v14544: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:25.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:25.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:25.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:25.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:25 smithi022 ceph-mon[40250]: pgmap v14545: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/426120989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:25 smithi022 ceph-mon[31360]: pgmap v14545: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/426120989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:25 smithi122 ceph-mon[32255]: pgmap v14545: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/426120989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:27.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:03:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:03:27.766 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:03:27.766 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:27 smithi022 ceph-mon[31360]: pgmap v14546: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:27.767 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:27 smithi022 ceph-mon[40250]: pgmap v14546: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:27 smithi122 ceph-mon[32255]: pgmap v14546: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:29 smithi122 ceph-mon[32255]: pgmap v14547: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:30.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:03:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:29 smithi022 ceph-mon[31360]: pgmap v14547: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:29 smithi022 ceph-mon[40250]: pgmap v14547: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:30.541 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:30.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:30.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:30.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1800312707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1800312707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1800312707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:31 smithi022 ceph-mon[31360]: pgmap v14548: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:31 smithi022 ceph-mon[40250]: pgmap v14548: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:31 smithi122 ceph-mon[32255]: pgmap v14548: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:03:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:03:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:33 smithi022 ceph-mon[31360]: pgmap v14549: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:33 smithi022 ceph-mon[40250]: pgmap v14549: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:33 smithi122 ceph-mon[32255]: pgmap v14549: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:03:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:03:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:03:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:03:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:03:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:03:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:03:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:03:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:03:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:03:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:03:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:03:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:35 smithi022 ceph-mon[31360]: pgmap v14550: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:35 smithi022 ceph-mon[40250]: pgmap v14550: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:36.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:36.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:36.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:35 smithi122 ceph-mon[32255]: pgmap v14550: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1452014374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:37.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1452014374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:37.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1452014374' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:37.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:03:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:03:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:03:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:37 smithi022 ceph-mon[31360]: pgmap v14551: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:37 smithi022 ceph-mon[40250]: pgmap v14551: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:37 smithi122 ceph-mon[32255]: pgmap v14551: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:39 smithi022 ceph-mon[31360]: pgmap v14552: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:39 smithi022 ceph-mon[40250]: pgmap v14552: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:39 smithi122 ceph-mon[32255]: pgmap v14552: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:40] "GET /metrics HTTP/1.1" 200 202437 "" "Prometheus/2.18.1" 2021-12-16T02:03:41.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:41.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:41.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:41.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:41 smithi022 ceph-mon[31360]: pgmap v14553: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4214305382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:41 smithi022 ceph-mon[40250]: pgmap v14553: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4214305382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:41 smithi122 ceph-mon[32255]: pgmap v14553: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4214305382' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:03:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:03:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:43 smithi022 ceph-mon[31360]: pgmap v14554: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:43 smithi022 ceph-mon[40250]: pgmap v14554: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:43 smithi122 ceph-mon[32255]: pgmap v14554: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:45 smithi022 ceph-mon[31360]: pgmap v14555: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:46.335 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:45 smithi022 ceph-mon[40250]: pgmap v14555: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:45 smithi122 ceph-mon[32255]: pgmap v14555: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:46.652 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:47.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:47.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:47.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:47.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:03:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:03:47.772 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:03:47.772 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:47 smithi022 ceph-mon[40250]: pgmap v14556: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:47.773 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4281395716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:47.773 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:47 smithi022 ceph-mon[31360]: pgmap v14556: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4281395716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:47 smithi122 ceph-mon[32255]: pgmap v14556: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4281395716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:49 smithi022 ceph-mon[31360]: pgmap v14557: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:49 smithi022 ceph-mon[40250]: pgmap v14557: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:49 smithi122 ceph-mon[32255]: pgmap v14557: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:03:52.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:51 smithi022 ceph-mon[31360]: pgmap v14558: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:51 smithi022 ceph-mon[40250]: pgmap v14558: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:51 smithi122 ceph-mon[32255]: pgmap v14558: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:52.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:52.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:52.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:03:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:03:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4249939654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:53.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4249939654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4249939654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:53 smithi022 ceph-mon[31360]: pgmap v14559: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:53 smithi022 ceph-mon[40250]: pgmap v14559: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:53 smithi122 ceph-mon[32255]: pgmap v14559: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:55 smithi022 ceph-mon[31360]: pgmap v14560: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:55 smithi022 ceph-mon[40250]: pgmap v14560: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:55 smithi122 ceph-mon[32255]: pgmap v14560: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:03:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:03:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:03:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:03:57.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:03:57.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:03:57.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:03:57.763 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:03:58.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:57 smithi022 ceph-mon[40250]: pgmap v14561: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:58.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:58.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3629559027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:57 smithi022 ceph-mon[31360]: pgmap v14561: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3629559027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:03:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:57 smithi122 ceph-mon[32255]: pgmap v14561: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:03:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:03:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3629559027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:03:59 smithi022 ceph-mon[31360]: pgmap v14562: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:03:59 smithi022 ceph-mon[40250]: pgmap v14562: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:03:59 smithi122 ceph-mon[32255]: pgmap v14562: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:04:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:01 smithi022 ceph-mon[31360]: pgmap v14563: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:01 smithi022 ceph-mon[40250]: pgmap v14563: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:01 smithi122 ceph-mon[32255]: pgmap v14563: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:04:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:04:02.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:03.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:03.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:03.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[31360]: pgmap v14564: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:04:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:04:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:04:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1853869735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:04:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[40250]: pgmap v14564: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:04:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:04:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:04:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1853869735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:04:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:03 smithi122 ceph-mon[32255]: pgmap v14564: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:04:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:04:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:04:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1853869735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:04:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:05 smithi022 ceph-mon[31360]: pgmap v14565: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:05 smithi022 ceph-mon[40250]: pgmap v14565: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:05 smithi122 ceph-mon[32255]: pgmap v14565: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:07.777 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:04:07.778 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:04:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:04:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:07 smithi022 ceph-mon[40250]: pgmap v14566: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:07.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:07 smithi022 ceph-mon[31360]: pgmap v14566: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:08.139 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:07 smithi122 ceph-mon[32255]: pgmap v14566: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:08.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:08.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:08.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4039690967' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4039690967' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4039690967' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:09 smithi122 ceph-mon[32255]: pgmap v14567: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:09 smithi022 ceph-mon[31360]: pgmap v14567: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:09 smithi022 ceph-mon[40250]: pgmap v14567: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:10] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:04:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:11 smithi022 ceph-mon[31360]: pgmap v14568: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:11 smithi022 ceph-mon[40250]: pgmap v14568: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:11 smithi122 ceph-mon[32255]: pgmap v14568: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:04:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:04:13.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:13.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:13.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:13.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:13 smithi022 ceph-mon[31360]: pgmap v14569: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3186609250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:13 smithi022 ceph-mon[40250]: pgmap v14569: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3186609250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:13 smithi122 ceph-mon[32255]: pgmap v14569: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3186609250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:15 smithi022 ceph-mon[31360]: pgmap v14570: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:15 smithi022 ceph-mon[40250]: pgmap v14570: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:15 smithi122 ceph-mon[32255]: pgmap v14570: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:04:17.778 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:04:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:04:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:17 smithi122 ceph-mon[32255]: pgmap v14571: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:17 smithi022 ceph-mon[31360]: pgmap v14571: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:17 smithi022 ceph-mon[40250]: pgmap v14571: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:18.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:19.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:19.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:19.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:19 smithi022 ceph-mon[31360]: pgmap v14572: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4096182329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:19 smithi022 ceph-mon[40250]: pgmap v14572: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4096182329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:20] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T02:04:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:19 smithi122 ceph-mon[32255]: pgmap v14572: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4096182329' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:21 smithi022 ceph-mon[31360]: pgmap v14573: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:21 smithi022 ceph-mon[40250]: pgmap v14573: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:21 smithi122 ceph-mon[32255]: pgmap v14573: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:04:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:04:24.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:24.246 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:23 smithi022 ceph-mon[31360]: pgmap v14574: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:24.247 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:23 smithi022 ceph-mon[40250]: pgmap v14574: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:23 smithi122 ceph-mon[32255]: pgmap v14574: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:24.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:24.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:24.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2700851176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:25.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2700851176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2700851176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:25 smithi022 ceph-mon[31360]: pgmap v14575: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:25 smithi022 ceph-mon[40250]: pgmap v14575: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:25 smithi122 ceph-mon[32255]: pgmap v14575: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:04:27.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:04:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:04:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:27 smithi122 ceph-mon[32255]: pgmap v14576: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:27 smithi022 ceph-mon[31360]: pgmap v14576: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:27 smithi022 ceph-mon[40250]: pgmap v14576: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:29.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:29.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:29.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:29 smithi022 ceph-mon[31360]: pgmap v14577: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:29 smithi022 ceph-mon[40250]: pgmap v14577: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:29 smithi122 ceph-mon[32255]: pgmap v14577: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:30] "GET /metrics HTTP/1.1" 200 202439 "" "Prometheus/2.18.1" 2021-12-16T02:04:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1908044174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1908044174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1908044174' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:31 smithi022 ceph-mon[31360]: pgmap v14578: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:31 smithi022 ceph-mon[40250]: pgmap v14578: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:31 smithi122 ceph-mon[32255]: pgmap v14578: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:04:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:04:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:33 smithi022 ceph-mon[31360]: pgmap v14579: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:33 smithi022 ceph-mon[40250]: pgmap v14579: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:33 smithi122 ceph-mon[32255]: pgmap v14579: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:34.986 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:04:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:04:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:04:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:04:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:04:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:04:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:04:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:04:35.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:35.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:35.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:04:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:04:35.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:04:35.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:04:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:35 smithi022 ceph-mon[31360]: pgmap v14580: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/289053925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:35 smithi022 ceph-mon[40250]: pgmap v14580: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/289053925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:35 smithi122 ceph-mon[32255]: pgmap v14580: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/289053925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:04:37.783 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:04:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:04:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:37 smithi122 ceph-mon[32255]: pgmap v14581: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:37 smithi022 ceph-mon[31360]: pgmap v14581: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:37 smithi022 ceph-mon[40250]: pgmap v14581: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:39 smithi022 ceph-mon[31360]: pgmap v14582: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:39 smithi022 ceph-mon[40250]: pgmap v14582: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:40.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:39 smithi122 ceph-mon[32255]: pgmap v14582: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:40] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:04:40.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:40.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:40.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4237487027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4237487027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4237487027' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:41 smithi022 ceph-mon[31360]: pgmap v14583: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:41 smithi022 ceph-mon[40250]: pgmap v14583: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:41 smithi122 ceph-mon[32255]: pgmap v14583: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:04:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:04:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:43 smithi022 ceph-mon[31360]: pgmap v14584: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:43 smithi022 ceph-mon[40250]: pgmap v14584: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:43 smithi122 ceph-mon[32255]: pgmap v14584: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:45.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:45 smithi022 ceph-mon[31360]: pgmap v14585: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:45 smithi022 ceph-mon[40250]: pgmap v14585: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:46.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:46.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:46.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:45 smithi122 ceph-mon[32255]: pgmap v14585: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2506630955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2506630955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2506630955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:04:47.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:04:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:04:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:47 smithi022 ceph-mon[31360]: pgmap v14586: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:47 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:47 smithi022 ceph-mon[40250]: pgmap v14586: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:47 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:47 smithi122 ceph-mon[32255]: pgmap v14586: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:47 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:49 smithi022 ceph-mon[31360]: pgmap v14587: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:49 smithi022 ceph-mon[40250]: pgmap v14587: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:49 smithi122 ceph-mon[32255]: pgmap v14587: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T02:04:51.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:51.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:51.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:51.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:51 smithi022 ceph-mon[31360]: pgmap v14588: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/898520438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:51 smithi022 ceph-mon[40250]: pgmap v14588: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/898520438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:51 smithi122 ceph-mon[32255]: pgmap v14588: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/898520438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:04:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:04:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:53 smithi022 ceph-mon[31360]: pgmap v14589: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:53 smithi022 ceph-mon[40250]: pgmap v14589: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:53 smithi122 ceph-mon[32255]: pgmap v14589: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:55 smithi022 ceph-mon[31360]: pgmap v14590: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:55 smithi022 ceph-mon[40250]: pgmap v14590: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:55 smithi122 ceph-mon[32255]: pgmap v14590: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:04:56.462 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:04:56.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:04:56.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:04:56.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:04:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:04:57.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:04:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:04:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/903430194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:57 smithi122 ceph-mon[32255]: pgmap v14591: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/903430194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:57 smithi022 ceph-mon[31360]: pgmap v14591: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:04:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/903430194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:04:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:57 smithi022 ceph-mon[40250]: pgmap v14591: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:04:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:04:59 smithi022 ceph-mon[31360]: pgmap v14592: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:04:59 smithi022 ceph-mon[40250]: pgmap v14592: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T02:05:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:04:59 smithi122 ceph-mon[32255]: pgmap v14592: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:01.834 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:02.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:02.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:02.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:02.202 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:01 smithi022 ceph-mon[40250]: pgmap v14593: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:02.202 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:01 smithi022 ceph-mon[31360]: pgmap v14593: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:01 smithi122 ceph-mon[32255]: pgmap v14593: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:05:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:05:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1150823318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1150823318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1150823318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[31360]: pgmap v14594: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:05:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:05:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:05:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:05:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[40250]: pgmap v14594: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:05:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:05:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:05:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:05:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:03 smithi122 ceph-mon[32255]: pgmap v14594: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:05:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:05:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:05:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:05:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:05 smithi022 ceph-mon[31360]: pgmap v14595: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:05 smithi022 ceph-mon[40250]: pgmap v14595: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:05 smithi122 ceph-mon[32255]: pgmap v14595: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:07.204 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:05:07.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:07.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:07.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:07.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:05:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:05:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:07 smithi022 ceph-mon[31360]: pgmap v14596: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:07 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1779648031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:07 smithi022 ceph-mon[40250]: pgmap v14596: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:07 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1779648031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:07 smithi122 ceph-mon[32255]: pgmap v14596: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:07 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1779648031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:09 smithi022 ceph-mon[31360]: pgmap v14597: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:09 smithi022 ceph-mon[40250]: pgmap v14597: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:05:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:09 smithi122 ceph-mon[32255]: pgmap v14597: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:11 smithi022 ceph-mon[31360]: pgmap v14598: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:11 smithi022 ceph-mon[40250]: pgmap v14598: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:11 smithi122 ceph-mon[32255]: pgmap v14598: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:12.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:05:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:05:12.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:12.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:12.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2085314320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2085314320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2085314320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:13 smithi022 ceph-mon[31360]: pgmap v14599: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:14.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:13 smithi022 ceph-mon[40250]: pgmap v14599: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:13 smithi122 ceph-mon[32255]: pgmap v14599: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:15 smithi022 ceph-mon[31360]: pgmap v14600: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:15 smithi022 ceph-mon[40250]: pgmap v14600: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:15 smithi122 ceph-mon[32255]: pgmap v14600: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:05:17.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:05:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:05:17.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:17 smithi122 ceph-mon[32255]: pgmap v14601: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:17 smithi022 ceph-mon[31360]: pgmap v14601: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:17 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:17 smithi022 ceph-mon[40250]: pgmap v14601: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:18.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:18.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:18.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3097706468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3097706468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3097706468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:19 smithi122 ceph-mon[32255]: pgmap v14602: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:19 smithi022 ceph-mon[31360]: pgmap v14602: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:19 smithi022 ceph-mon[40250]: pgmap v14602: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:20] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T02:05:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:21 smithi022 ceph-mon[31360]: pgmap v14603: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:21 smithi022 ceph-mon[40250]: pgmap v14603: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:21 smithi122 ceph-mon[32255]: pgmap v14603: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:05:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:05:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:23 smithi022 ceph-mon[31360]: pgmap v14604: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/653892950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:23 smithi022 ceph-mon[40250]: pgmap v14604: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/653892950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:23 smithi122 ceph-mon[32255]: pgmap v14604: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/653892950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:25 smithi022 ceph-mon[31360]: pgmap v14605: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:25 smithi022 ceph-mon[40250]: pgmap v14605: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:25 smithi122 ceph-mon[32255]: pgmap v14605: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:05:27.796 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:05:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:05:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:27 smithi122 ceph-mon[32255]: pgmap v14606: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:27 smithi022 ceph-mon[31360]: pgmap v14606: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:27 smithi022 ceph-mon[40250]: pgmap v14606: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:28.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:29.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:29.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:29.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:29 smithi022 ceph-mon[31360]: pgmap v14607: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4176781663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:29 smithi022 ceph-mon[40250]: pgmap v14607: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4176781663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:30] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T02:05:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:29 smithi122 ceph-mon[32255]: pgmap v14607: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4176781663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:31 smithi022 ceph-mon[31360]: pgmap v14608: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:31 smithi022 ceph-mon[40250]: pgmap v14608: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:31 smithi122 ceph-mon[32255]: pgmap v14608: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:05:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:05:34.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:33 smithi022 ceph-mon[31360]: pgmap v14609: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:33 smithi022 ceph-mon[40250]: pgmap v14609: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:33 smithi122 ceph-mon[32255]: pgmap v14609: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:05:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:05:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:05:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:05:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/306693237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:05:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:05:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:05:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:05:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/306693237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:05:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:05:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:05:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:05:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/306693237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:35 smithi022 ceph-mon[31360]: pgmap v14610: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:35 smithi022 ceph-mon[40250]: pgmap v14610: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:35 smithi122 ceph-mon[32255]: pgmap v14610: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:05:37.798 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:05:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:05:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:37 smithi122 ceph-mon[32255]: pgmap v14611: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:37 smithi022 ceph-mon[31360]: pgmap v14611: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:37 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:37 smithi022 ceph-mon[40250]: pgmap v14611: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:37 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:39.421 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:39.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:39.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:39.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:39 smithi022 ceph-mon[31360]: pgmap v14612: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1868182740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:39 smithi022 ceph-mon[40250]: pgmap v14612: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1868182740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:40] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T02:05:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:39 smithi122 ceph-mon[32255]: pgmap v14612: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1868182740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:41 smithi022 ceph-mon[31360]: pgmap v14613: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:41 smithi022 ceph-mon[40250]: pgmap v14613: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:41 smithi122 ceph-mon[32255]: pgmap v14613: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:05:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:05:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:43 smithi022 ceph-mon[31360]: pgmap v14614: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:43 smithi022 ceph-mon[40250]: pgmap v14614: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:43 smithi122 ceph-mon[32255]: pgmap v14614: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:44.790 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:45 smithi022 ceph-mon[31360]: pgmap v14615: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/516648154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:45 smithi022 ceph-mon[40250]: pgmap v14615: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/516648154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:45 smithi122 ceph-mon[32255]: pgmap v14615: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/516648154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:05:47.800 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:05:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:05:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:47 smithi122 ceph-mon[32255]: pgmap v14616: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:47 smithi022 ceph-mon[31360]: pgmap v14616: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:47 smithi022 ceph-mon[40250]: pgmap v14616: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:50.160 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:50.160 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:49 smithi022 ceph-mon[40250]: pgmap v14617: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:50.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:49 smithi022 ceph-mon[31360]: pgmap v14617: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:49 smithi122 ceph-mon[32255]: pgmap v14617: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:50] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:05:50.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:50.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:50.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3469497800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3469497800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3469497800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:51 smithi122 ceph-mon[32255]: pgmap v14618: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:51 smithi022 ceph-mon[31360]: pgmap v14618: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:51 smithi022 ceph-mon[40250]: pgmap v14618: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:05:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:05:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:53 smithi022 ceph-mon[31360]: pgmap v14619: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:53 smithi022 ceph-mon[40250]: pgmap v14619: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:53 smithi122 ceph-mon[32255]: pgmap v14619: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:05:55.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:05:55.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:05:55.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:05:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:55 smithi022 ceph-mon[31360]: pgmap v14620: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/607523076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:55 smithi022 ceph-mon[40250]: pgmap v14620: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/607523076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:55 smithi122 ceph-mon[32255]: pgmap v14620: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:05:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/607523076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:05:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:05:57.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:05:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:05:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:57 smithi122 ceph-mon[32255]: pgmap v14621: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:57 smithi022 ceph-mon[31360]: pgmap v14621: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:57 smithi022 ceph-mon[40250]: pgmap v14621: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:05:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:05:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:05:59 smithi122 ceph-mon[32255]: pgmap v14622: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:05:59 smithi022 ceph-mon[31360]: pgmap v14622: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:05:59 smithi022 ceph-mon[40250]: pgmap v14622: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:00] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:06:00.899 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:01.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:01.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:01.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:01 smithi022 ceph-mon[31360]: pgmap v14623: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1983559176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:01 smithi022 ceph-mon[40250]: pgmap v14623: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1983559176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:01 smithi122 ceph-mon[32255]: pgmap v14623: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1983559176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:06:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:06:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[31360]: pgmap v14624: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:06:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:06:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:06:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[40250]: pgmap v14624: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:06:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:06:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:06:04.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:03 smithi122 ceph-mon[32255]: pgmap v14624: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:04.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:06:04.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:06:04.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:06:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:06:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:06:05.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:06:06.271 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:06 smithi022 ceph-mon[31360]: pgmap v14625: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:06 smithi022 ceph-mon[40250]: pgmap v14625: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:06.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:06.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:06.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:06 smithi122 ceph-mon[32255]: pgmap v14625: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2229436424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:06:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2229436424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2229436424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:06:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:06:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:08 smithi022 ceph-mon[31360]: pgmap v14626: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:08 smithi022 ceph-mon[40250]: pgmap v14626: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:08.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:08 smithi122 ceph-mon[32255]: pgmap v14626: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:10] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:06:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:10 smithi022 ceph-mon[31360]: pgmap v14627: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:10 smithi022 ceph-mon[40250]: pgmap v14627: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:10 smithi122 ceph-mon[32255]: pgmap v14627: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:11.641 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:12.012 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:12.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:12.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:06:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:06:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:12 smithi122 ceph-mon[32255]: pgmap v14628: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1313074235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:12 smithi022 ceph-mon[31360]: pgmap v14628: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1313074235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:12 smithi022 ceph-mon[40250]: pgmap v14628: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:12.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1313074235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:14 smithi022 ceph-mon[31360]: pgmap v14629: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:14 smithi022 ceph-mon[40250]: pgmap v14629: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:14 smithi122 ceph-mon[32255]: pgmap v14629: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:16 smithi022 ceph-mon[31360]: pgmap v14630: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:16 smithi022 ceph-mon[40250]: pgmap v14630: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:16 smithi122 ceph-mon[32255]: pgmap v14630: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:17.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:17.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:17.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:06:17.806 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:06:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:06:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:17 smithi122 ceph-mon[32255]: pgmap v14631: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3038875488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:17 smithi022 ceph-mon[31360]: pgmap v14631: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3038875488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:17 smithi022 ceph-mon[40250]: pgmap v14631: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3038875488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:19 smithi122 ceph-mon[32255]: pgmap v14632: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:19 smithi022 ceph-mon[31360]: pgmap v14632: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:19 smithi022 ceph-mon[40250]: pgmap v14632: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:20] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:06:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:21 smithi022 ceph-mon[31360]: pgmap v14633: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:21 smithi022 ceph-mon[40250]: pgmap v14633: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:21 smithi122 ceph-mon[32255]: pgmap v14633: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:22.384 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:06:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:06:22.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:22.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:22.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1732560110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1732560110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1732560110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:23 smithi022 ceph-mon[31360]: pgmap v14634: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:23 smithi022 ceph-mon[40250]: pgmap v14634: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:23 smithi122 ceph-mon[32255]: pgmap v14634: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:25 smithi022 ceph-mon[31360]: pgmap v14635: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:25 smithi022 ceph-mon[40250]: pgmap v14635: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:25 smithi122 ceph-mon[32255]: pgmap v14635: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:27.754 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:06:27.808 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:06:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:06:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:27 smithi022 ceph-mon[31360]: pgmap v14636: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:27 smithi022 ceph-mon[40250]: pgmap v14636: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:27 smithi122 ceph-mon[32255]: pgmap v14636: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/161945491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/161945491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/161945491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:29 smithi122 ceph-mon[32255]: pgmap v14637: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:29 smithi022 ceph-mon[31360]: pgmap v14637: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:29 smithi022 ceph-mon[40250]: pgmap v14637: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:30] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:06:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:31 smithi022 ceph-mon[31360]: pgmap v14638: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:31 smithi022 ceph-mon[40250]: pgmap v14638: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:31 smithi122 ceph-mon[32255]: pgmap v14638: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:06:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:06:33.124 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:33.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:33.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:33.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:33 smithi022 ceph-mon[31360]: pgmap v14639: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/12459163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:33 smithi022 ceph-mon[40250]: pgmap v14639: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/12459163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:33 smithi122 ceph-mon[32255]: pgmap v14639: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/12459163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:06:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:06:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:06:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:06:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:06:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:06:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:06:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:06:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:06:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:06:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:06:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:06:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:35 smithi022 ceph-mon[31360]: pgmap v14640: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:36.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:35 smithi022 ceph-mon[40250]: pgmap v14640: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:36 smithi122 ceph-mon[32255]: pgmap v14640: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:06:37.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:06:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:06:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:37 smithi122 ceph-mon[32255]: pgmap v14641: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:37 smithi022 ceph-mon[31360]: pgmap v14641: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:37 smithi022 ceph-mon[40250]: pgmap v14641: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:38.498 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:38.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:38.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:38.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3915907972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:39 smithi122 ceph-mon[32255]: pgmap v14642: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3915907972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:39 smithi022 ceph-mon[31360]: pgmap v14642: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3915907972' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:39 smithi022 ceph-mon[40250]: pgmap v14642: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:40] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:06:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:41 smithi022 ceph-mon[31360]: pgmap v14643: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:41 smithi022 ceph-mon[40250]: pgmap v14643: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:41 smithi122 ceph-mon[32255]: pgmap v14643: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:06:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:06:43.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:44.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:44.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:44.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:44.236 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:43 smithi022 ceph-mon[31360]: pgmap v14644: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:44.237 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:43 smithi022 ceph-mon[40250]: pgmap v14644: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:43 smithi122 ceph-mon[32255]: pgmap v14644: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3231112034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3231112034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3231112034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:46 smithi022 ceph-mon[31360]: pgmap v14645: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:46 smithi022 ceph-mon[40250]: pgmap v14645: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:46 smithi122 ceph-mon[32255]: pgmap v14645: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:06:47.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:06:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:06:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:47 smithi122 ceph-mon[32255]: pgmap v14646: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:47 smithi022 ceph-mon[31360]: pgmap v14646: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:47 smithi022 ceph-mon[40250]: pgmap v14646: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:49.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:49.238 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:49.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:49.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:49.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:49.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:49 smithi022 ceph-mon[31360]: pgmap v14647: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4266251882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:49 smithi022 ceph-mon[40250]: pgmap v14647: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4266251882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:49 smithi122 ceph-mon[32255]: pgmap v14647: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4266251882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:06:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:51 smithi022 ceph-mon[31360]: pgmap v14648: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:51 smithi022 ceph-mon[40250]: pgmap v14648: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:51 smithi122 ceph-mon[32255]: pgmap v14648: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:06:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:06:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:53 smithi022 ceph-mon[31360]: pgmap v14649: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:53 smithi022 ceph-mon[40250]: pgmap v14649: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:53 smithi122 ceph-mon[32255]: pgmap v14649: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:54.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:06:54.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:06:54.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:06:54.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:06:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/226377872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/226377872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/226377872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:06:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:56 smithi022 ceph-mon[31360]: pgmap v14650: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:56.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:56 smithi022 ceph-mon[40250]: pgmap v14650: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:56 smithi122 ceph-mon[32255]: pgmap v14650: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:06:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:06:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:57 smithi122 ceph-mon[32255]: pgmap v14651: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:06:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:06:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:57 smithi022 ceph-mon[31360]: pgmap v14651: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:57 smithi022 ceph-mon[40250]: pgmap v14651: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:06:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:06:59.973 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:06:59 smithi122 ceph-mon[32255]: pgmap v14652: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:06:59 smithi022 ceph-mon[31360]: pgmap v14652: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:06:59 smithi022 ceph-mon[40250]: pgmap v14652: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:00.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:00.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:00.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:07:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1210295807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1210295807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1210295807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:01 smithi122 ceph-mon[32255]: pgmap v14653: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:01 smithi022 ceph-mon[31360]: pgmap v14653: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:01 smithi022 ceph-mon[40250]: pgmap v14653: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:07:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:07:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:03 smithi022 ceph-mon[31360]: pgmap v14654: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:03 smithi022 ceph-mon[40250]: pgmap v14654: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:03 smithi122 ceph-mon[32255]: pgmap v14654: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:07:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:07:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:07:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:07:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:07:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:07:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:07:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:07:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:07:05.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:05.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:05.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:06.002 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:06 smithi022 ceph-mon[31360]: pgmap v14655: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:06.103 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3261110804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:06 smithi022 ceph-mon[40250]: pgmap v14655: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3261110804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:06 smithi122 ceph-mon[32255]: pgmap v14655: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3261110804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:07:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:07:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:07:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:07 smithi122 ceph-mon[32255]: pgmap v14656: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:08.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:07 smithi022 ceph-mon[31360]: pgmap v14656: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:08.084 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:07 smithi022 ceph-mon[40250]: pgmap v14656: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:07:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:09.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:10.135 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:10 smithi022 ceph-mon[31360]: pgmap v14657: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:10.135 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:10.136 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:10 smithi022 ceph-mon[40250]: pgmap v14657: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:10.136 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:10.420 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:10 smithi122 ceph-mon[32255]: pgmap v14657: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:10.420 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:10.420 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:10] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T02:07:10.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:11.273 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1519330630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1519330630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:11.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1519330630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:12.432 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:07:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:07:12.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: pgmap v14658: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:12.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:12.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2978M 2021-12-16T02:07:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:12 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: pgmap v14658: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2978M 2021-12-16T02:07:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:12.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: pgmap v14658: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:12.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:12.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:12.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2978M 2021-12-16T02:07:12.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:12 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3219M 2021-12-16T02:07:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3219M 2021-12-16T02:07:13.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:07:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3219M 2021-12-16T02:07:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:13.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:07:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:14 smithi022 ceph-mon[31360]: pgmap v14659: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:14 smithi022 ceph-mon[40250]: pgmap v14659: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:14 smithi122 ceph-mon[32255]: pgmap v14659: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:16.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:16.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:16.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:16.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:16 smithi022 ceph-mon[31360]: pgmap v14660: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2229293582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:16 smithi022 ceph-mon[40250]: pgmap v14660: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2229293582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:16 smithi122 ceph-mon[32255]: pgmap v14660: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2229293582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:07:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:17 smithi122 ceph-mon[32255]: pgmap v14661: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:17.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:07:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:07:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:17 smithi022 ceph-mon[31360]: pgmap v14661: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:17 smithi022 ceph-mon[40250]: pgmap v14661: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:19 smithi122 ceph-mon[32255]: pgmap v14662: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:19 smithi022 ceph-mon[31360]: pgmap v14662: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:19 smithi022 ceph-mon[40250]: pgmap v14662: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:20] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T02:07:21.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:22 smithi022 ceph-mon[31360]: pgmap v14663: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/84278182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:22 smithi022 ceph-mon[40250]: pgmap v14663: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/84278182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:22 smithi122 ceph-mon[32255]: pgmap v14663: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/84278182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:07:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:07:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:24 smithi022 ceph-mon[31360]: pgmap v14664: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:24 smithi022 ceph-mon[40250]: pgmap v14664: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:24 smithi122 ceph-mon[32255]: pgmap v14664: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:26 smithi022 ceph-mon[31360]: pgmap v14665: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:26.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:26 smithi022 ceph-mon[40250]: pgmap v14665: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:26 smithi122 ceph-mon[32255]: pgmap v14665: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:27.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:27.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:27.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:07:27.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:07:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:07:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:27 smithi122 ceph-mon[32255]: pgmap v14666: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1865946149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:27 smithi022 ceph-mon[31360]: pgmap v14666: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1865946149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:27 smithi022 ceph-mon[40250]: pgmap v14666: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1865946149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:30.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:29 smithi122 ceph-mon[32255]: pgmap v14667: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:29 smithi022 ceph-mon[31360]: pgmap v14667: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:29 smithi022 ceph-mon[40250]: pgmap v14667: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:30] "GET /metrics HTTP/1.1" 200 202438 "" "Prometheus/2.18.1" 2021-12-16T02:07:32.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:32 smithi022 ceph-mon[31360]: pgmap v14668: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:32 smithi022 ceph-mon[40250]: pgmap v14668: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:32 smithi122 ceph-mon[32255]: pgmap v14668: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:32.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:32.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:32.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:07:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:07:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4061990456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4061990456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4061990456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:34 smithi022 ceph-mon[31360]: pgmap v14669: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:34 smithi022 ceph-mon[40250]: pgmap v14669: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:34 smithi122 ceph-mon[32255]: pgmap v14669: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:07:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:07:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:07:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:07:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:07:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:07:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:07:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:07:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:07:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:07:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:07:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:07:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:36 smithi022 ceph-mon[31360]: pgmap v14670: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:36 smithi022 ceph-mon[40250]: pgmap v14670: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:36 smithi122 ceph-mon[32255]: pgmap v14670: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:37.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:07:37.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:07:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:07:37.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:37 smithi122 ceph-mon[32255]: pgmap v14671: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:37.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:37.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:37.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:37 smithi022 ceph-mon[31360]: pgmap v14671: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:37 smithi022 ceph-mon[40250]: pgmap v14671: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4151474365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4151474365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4151474365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:39 smithi122 ceph-mon[32255]: pgmap v14672: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:39 smithi022 ceph-mon[31360]: pgmap v14672: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:39 smithi022 ceph-mon[40250]: pgmap v14672: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:40] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:07:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:42 smithi022 ceph-mon[31360]: pgmap v14673: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:42 smithi022 ceph-mon[40250]: pgmap v14673: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:42 smithi122 ceph-mon[32255]: pgmap v14673: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:07:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:07:42.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:44 smithi022 ceph-mon[31360]: pgmap v14674: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1853716145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:44 smithi022 ceph-mon[40250]: pgmap v14674: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1853716145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:44 smithi122 ceph-mon[32255]: pgmap v14674: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1853716145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:46 smithi022 ceph-mon[31360]: pgmap v14675: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:46 smithi022 ceph-mon[40250]: pgmap v14675: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:46 smithi122 ceph-mon[32255]: pgmap v14675: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:07:47.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:47 smithi122 ceph-mon[32255]: pgmap v14676: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:47.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:07:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:07:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:47 smithi022 ceph-mon[31360]: pgmap v14676: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:47 smithi022 ceph-mon[40250]: pgmap v14676: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:48.316 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:48.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:48.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:48.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3432117287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3432117287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3432117287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:49 smithi122 ceph-mon[32255]: pgmap v14677: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:49 smithi022 ceph-mon[31360]: pgmap v14677: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:49 smithi022 ceph-mon[40250]: pgmap v14677: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:50] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:07:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:52 smithi022 ceph-mon[31360]: pgmap v14678: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:52 smithi022 ceph-mon[40250]: pgmap v14678: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:52 smithi122 ceph-mon[32255]: pgmap v14678: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:07:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:07:53.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:54.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:54.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:54.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:07:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:54 smithi022 ceph-mon[31360]: pgmap v14679: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:54.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:54 smithi022 ceph-mon[40250]: pgmap v14679: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:54 smithi122 ceph-mon[32255]: pgmap v14679: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3916511823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3916511823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3916511823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:07:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:56 smithi022 ceph-mon[31360]: pgmap v14680: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:56 smithi022 ceph-mon[40250]: pgmap v14680: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:56 smithi122 ceph-mon[32255]: pgmap v14680: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:07:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:07:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:57 smithi122 ceph-mon[32255]: pgmap v14681: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:07:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:07:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:57 smithi022 ceph-mon[31360]: pgmap v14681: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:57 smithi022 ceph-mon[40250]: pgmap v14681: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:07:59.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:07:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:07:59.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:07:59.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:07:59.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:59 smithi122 ceph-mon[32255]: pgmap v14682: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:07:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/619716522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:59 smithi022 ceph-mon[31360]: pgmap v14682: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:07:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/619716522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:59 smithi022 ceph-mon[40250]: pgmap v14682: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:07:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/619716522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:00] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:08:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:02 smithi022 ceph-mon[31360]: pgmap v14683: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:02 smithi022 ceph-mon[40250]: pgmap v14683: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:02 smithi122 ceph-mon[32255]: pgmap v14683: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:08:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:08:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:04 smithi022 ceph-mon[31360]: pgmap v14684: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:04 smithi022 ceph-mon[40250]: pgmap v14684: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:04.425 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:04 smithi122 ceph-mon[32255]: pgmap v14684: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:04.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:04.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:04.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/654597797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/654597797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/654597797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:06 smithi022 ceph-mon[31360]: pgmap v14685: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:06 smithi022 ceph-mon[40250]: pgmap v14685: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:06 smithi122 ceph-mon[32255]: pgmap v14685: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:08:07.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:08:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:08:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:07 smithi122 ceph-mon[32255]: pgmap v14686: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:07 smithi022 ceph-mon[31360]: pgmap v14686: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:07 smithi022 ceph-mon[40250]: pgmap v14686: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:09.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:10.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:10.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:09 smithi022 ceph-mon[31360]: pgmap v14687: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:10.166 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:09 smithi022 ceph-mon[40250]: pgmap v14687: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:09 smithi122 ceph-mon[32255]: pgmap v14687: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:08:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:10] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:08:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/411842045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/411842045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/411842045' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:11 smithi122 ceph-mon[32255]: pgmap v14688: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:11 smithi022 ceph-mon[40250]: pgmap v14688: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:11 smithi022 ceph-mon[31360]: pgmap v14688: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:12.565 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:08:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:08:13.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:08:13.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:08:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:12 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:08:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:08:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:08:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:12 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:08:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:08:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:08:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:12 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:08:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:14 smithi022 ceph-mon[31360]: pgmap v14689: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:08:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:14 smithi022 ceph-mon[40250]: pgmap v14689: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:08:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:14 smithi122 ceph-mon[32255]: pgmap v14689: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:08:15.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:15.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:15.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:15.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:16 smithi022 ceph-mon[31360]: pgmap v14690: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3400495190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:16 smithi022 ceph-mon[40250]: pgmap v14690: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3400495190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:16 smithi122 ceph-mon[32255]: pgmap v14690: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3400495190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:08:17.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:08:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:08:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:17 smithi122 ceph-mon[32255]: pgmap v14691: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:17 smithi022 ceph-mon[31360]: pgmap v14691: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:17 smithi022 ceph-mon[40250]: pgmap v14691: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:20.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:19 smithi122 ceph-mon[32255]: pgmap v14692: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:19 smithi022 ceph-mon[40250]: pgmap v14692: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:19 smithi022 ceph-mon[31360]: pgmap v14692: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:20] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:08:20.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:20.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:20.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:20.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1143587652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1143587652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1143587652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:21 smithi022 ceph-mon[31360]: pgmap v14693: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:22.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:21 smithi022 ceph-mon[40250]: pgmap v14693: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:21 smithi122 ceph-mon[32255]: pgmap v14693: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:08:22.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:08:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:24 smithi022 ceph-mon[31360]: pgmap v14694: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:24.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:24 smithi022 ceph-mon[40250]: pgmap v14694: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:24 smithi122 ceph-mon[32255]: pgmap v14694: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:08:25.907 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:26.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:26.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:26.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:26.274 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:26 smithi022 ceph-mon[31360]: pgmap v14695: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:26.275 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:26 smithi022 ceph-mon[40250]: pgmap v14695: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:26 smithi122 ceph-mon[32255]: pgmap v14695: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:08:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2965068477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2965068477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:27.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2965068477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:08:27.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:08:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:08:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:28 smithi022 ceph-mon[31360]: pgmap v14696: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:28 smithi022 ceph-mon[40250]: pgmap v14696: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:28 smithi122 ceph-mon[32255]: pgmap v14696: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:30] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:08:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:30 smithi122 ceph-mon[32255]: pgmap v14697: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:30 smithi022 ceph-mon[31360]: pgmap v14697: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:30 smithi022 ceph-mon[40250]: pgmap v14697: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:31.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:31.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:31.642 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:31.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:32 smithi122 ceph-mon[32255]: pgmap v14698: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4103291271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:32 smithi022 ceph-mon[31360]: pgmap v14698: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4103291271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:32 smithi022 ceph-mon[40250]: pgmap v14698: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4103291271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:08:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:08:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:34 smithi122 ceph-mon[32255]: pgmap v14699: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:08:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:08:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:08:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:08:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[31360]: pgmap v14699: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:08:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:08:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:08:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:08:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[40250]: pgmap v14699: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:08:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:08:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:08:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:36 smithi122 ceph-mon[32255]: pgmap v14700: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:36 smithi022 ceph-mon[31360]: pgmap v14700: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:36 smithi022 ceph-mon[40250]: pgmap v14700: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:36.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:37.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:37.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:37.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:08:37.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:08:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:08:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:37 smithi122 ceph-mon[32255]: pgmap v14701: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/834025220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:37 smithi022 ceph-mon[31360]: pgmap v14701: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/834025220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:37 smithi022 ceph-mon[40250]: pgmap v14701: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/834025220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:40.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:39 smithi122 ceph-mon[32255]: pgmap v14702: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:39 smithi022 ceph-mon[31360]: pgmap v14702: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:39 smithi022 ceph-mon[40250]: pgmap v14702: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T02:08:42.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:42 smithi022 ceph-mon[31360]: pgmap v14703: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:42 smithi022 ceph-mon[40250]: pgmap v14703: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:42 smithi122 ceph-mon[32255]: pgmap v14703: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:08:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:08:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2803862278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2803862278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2803862278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:44 smithi122 ceph-mon[32255]: pgmap v14704: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:44 smithi022 ceph-mon[31360]: pgmap v14704: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:44 smithi022 ceph-mon[40250]: pgmap v14704: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:46 smithi122 ceph-mon[32255]: pgmap v14705: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:46 smithi022 ceph-mon[31360]: pgmap v14705: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:46 smithi022 ceph-mon[40250]: pgmap v14705: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:47.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:47.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:47.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:47.755 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:47.756 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:08:47.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:08:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:08:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:47 smithi122 ceph-mon[32255]: pgmap v14706: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1240857636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:47 smithi022 ceph-mon[31360]: pgmap v14706: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1240857636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:47 smithi022 ceph-mon[40250]: pgmap v14706: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:48.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1240857636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:50.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:49 smithi122 ceph-mon[32255]: pgmap v14707: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:49 smithi022 ceph-mon[31360]: pgmap v14707: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:49 smithi022 ceph-mon[40250]: pgmap v14707: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:50] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:08:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:52 smithi122 ceph-mon[32255]: pgmap v14708: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:52.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:52 smithi022 ceph-mon[31360]: pgmap v14708: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:52.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:52 smithi022 ceph-mon[40250]: pgmap v14708: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:08:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:08:52.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:53.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:53.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:53.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:54 smithi122 ceph-mon[32255]: pgmap v14709: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/644770902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:54 smithi022 ceph-mon[31360]: pgmap v14709: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/644770902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:54 smithi022 ceph-mon[40250]: pgmap v14709: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/644770902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:56 smithi122 ceph-mon[32255]: pgmap v14710: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:56 smithi022 ceph-mon[31360]: pgmap v14710: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:56 smithi022 ceph-mon[40250]: pgmap v14710: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:08:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:08:57.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:57 smithi122 ceph-mon[32255]: pgmap v14711: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:08:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:08:58.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:08:58.128 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:57 smithi022 ceph-mon[31360]: pgmap v14711: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:58.129 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:57 smithi022 ceph-mon[40250]: pgmap v14711: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:08:58.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:08:58.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:08:58.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:08:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3329146230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3329146230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:08:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:08:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3329146230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:08:59 smithi122 ceph-mon[32255]: pgmap v14712: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:08:59 smithi022 ceph-mon[31360]: pgmap v14712: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:08:59 smithi022 ceph-mon[40250]: pgmap v14712: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:00] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:09:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:02 smithi122 ceph-mon[32255]: pgmap v14713: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:02 smithi022 ceph-mon[31360]: pgmap v14713: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:02 smithi022 ceph-mon[40250]: pgmap v14713: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:09:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:09:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:03.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:03.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:03.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:04 smithi022 ceph-mon[31360]: pgmap v14714: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3684913519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:04 smithi022 ceph-mon[40250]: pgmap v14714: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3684913519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:04 smithi122 ceph-mon[32255]: pgmap v14714: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3684913519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:06 smithi122 ceph-mon[32255]: pgmap v14715: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:06 smithi022 ceph-mon[31360]: pgmap v14715: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:06 smithi022 ceph-mon[40250]: pgmap v14715: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:09:08.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:07 smithi122 ceph-mon[32255]: pgmap v14716: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:08.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:09:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:09:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:07 smithi022 ceph-mon[31360]: pgmap v14716: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:07 smithi022 ceph-mon[40250]: pgmap v14716: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:08.867 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:09.233 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:09.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:09.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:09.234 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:09.235 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:09 smithi122 ceph-mon[32255]: pgmap v14717: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/920365206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:09 smithi022 ceph-mon[31360]: pgmap v14717: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/920365206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:09 smithi022 ceph-mon[40250]: pgmap v14717: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/920365206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:10] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:09:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:12 smithi122 ceph-mon[32255]: pgmap v14718: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:12 smithi022 ceph-mon[31360]: pgmap v14718: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:12 smithi022 ceph-mon[40250]: pgmap v14718: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:09:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:09:13.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:09:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:09:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:13 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:09:13.452 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:09:13.452 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:09:13.452 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:13 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:09:13.453 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:09:13.453 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:09:13.453 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:13 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:09:14.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:14 smithi022 ceph-mon[31360]: pgmap v14719: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:09:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:14 smithi022 ceph-mon[40250]: pgmap v14719: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:09:14.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:14.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:14.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:14 smithi122 ceph-mon[32255]: pgmap v14719: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:09:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1152928301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1152928301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:15.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1152928301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:16 smithi022 ceph-mon[31360]: pgmap v14720: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:16 smithi022 ceph-mon[40250]: pgmap v14720: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:16.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:16 smithi122 ceph-mon[32255]: pgmap v14720: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:09:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:17 smithi122 ceph-mon[32255]: pgmap v14721: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:18.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:09:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:09:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:17 smithi022 ceph-mon[31360]: pgmap v14721: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:17 smithi022 ceph-mon[40250]: pgmap v14721: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:19.605 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:19.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:19.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:19.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:19.974 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:19 smithi022 ceph-mon[31360]: pgmap v14722: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:19.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:19 smithi022 ceph-mon[40250]: pgmap v14722: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:20.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:19 smithi122 ceph-mon[32255]: pgmap v14722: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:20] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:09:21.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2084360573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2084360573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2084360573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:22.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:21 smithi122 ceph-mon[32255]: pgmap v14723: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:21 smithi022 ceph-mon[31360]: pgmap v14723: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:21 smithi022 ceph-mon[40250]: pgmap v14723: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:09:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:09:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:24 smithi122 ceph-mon[32255]: pgmap v14724: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:24 smithi022 ceph-mon[31360]: pgmap v14724: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:24 smithi022 ceph-mon[40250]: pgmap v14724: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:24.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:25.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:25.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:25.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:26 smithi122 ceph-mon[32255]: pgmap v14725: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/53719675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:26 smithi022 ceph-mon[31360]: pgmap v14725: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/53719675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:26 smithi022 ceph-mon[40250]: pgmap v14725: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/53719675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:09:28.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:09:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:09:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:27 smithi122 ceph-mon[32255]: pgmap v14726: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:27 smithi022 ceph-mon[31360]: pgmap v14726: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:27 smithi022 ceph-mon[40250]: pgmap v14726: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:29 smithi122 ceph-mon[32255]: pgmap v14727: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:29 smithi022 ceph-mon[31360]: pgmap v14727: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:29 smithi022 ceph-mon[40250]: pgmap v14727: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:30.345 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:30] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:09:30.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:30.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:30.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3267025501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3267025501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:31.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3267025501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:32.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:31 smithi122 ceph-mon[32255]: pgmap v14728: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:31 smithi022 ceph-mon[31360]: pgmap v14728: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:31 smithi022 ceph-mon[40250]: pgmap v14728: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:09:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:09:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:34 smithi122 ceph-mon[32255]: pgmap v14729: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:09:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:09:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:09:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:09:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[31360]: pgmap v14729: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:09:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:09:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:09:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:09:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[40250]: pgmap v14729: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:09:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:09:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:09:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:09:35.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:36.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:36.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:36.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:36 smithi122 ceph-mon[32255]: pgmap v14730: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:36 smithi022 ceph-mon[31360]: pgmap v14730: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:36 smithi022 ceph-mon[40250]: pgmap v14730: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:37.391 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2743192370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:37.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2743192370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2743192370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:09:38.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:09:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:09:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:38 smithi122 ceph-mon[32255]: pgmap v14731: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:38 smithi022 ceph-mon[31360]: pgmap v14731: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:38 smithi022 ceph-mon[40250]: pgmap v14731: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:40] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T02:09:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:40 smithi122 ceph-mon[32255]: pgmap v14732: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:40 smithi022 ceph-mon[31360]: pgmap v14732: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:40 smithi022 ceph-mon[40250]: pgmap v14732: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:41.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:41.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:41.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:41.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:42 smithi122 ceph-mon[32255]: pgmap v14733: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3132601685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:42 smithi022 ceph-mon[31360]: pgmap v14733: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3132601685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:42 smithi022 ceph-mon[40250]: pgmap v14733: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3132601685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:09:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:09:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:44 smithi122 ceph-mon[32255]: pgmap v14734: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:44 smithi022 ceph-mon[40250]: pgmap v14734: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:44 smithi022 ceph-mon[31360]: pgmap v14734: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:46 smithi122 ceph-mon[32255]: pgmap v14735: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:46.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:46.458 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:46 smithi022 ceph-mon[31360]: pgmap v14735: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:46.458 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:46 smithi022 ceph-mon[40250]: pgmap v14735: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:46.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:46.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:46.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3227665353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:47.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3227665353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3227665353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:09:48.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:09:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:09:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:48 smithi122 ceph-mon[32255]: pgmap v14736: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:48 smithi022 ceph-mon[31360]: pgmap v14736: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:48 smithi022 ceph-mon[40250]: pgmap v14736: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:50] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:09:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:50 smithi122 ceph-mon[32255]: pgmap v14737: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:50 smithi022 ceph-mon[31360]: pgmap v14737: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:50 smithi022 ceph-mon[40250]: pgmap v14737: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:51.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:52.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:52.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:52.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:52.202 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:52 smithi022 ceph-mon[31360]: pgmap v14738: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:52.205 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:52 smithi022 ceph-mon[40250]: pgmap v14738: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:52.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:52 smithi122 ceph-mon[32255]: pgmap v14738: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:09:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:09:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/874694660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/874694660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/874694660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:54 smithi122 ceph-mon[32255]: pgmap v14739: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:54 smithi022 ceph-mon[31360]: pgmap v14739: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:54 smithi022 ceph-mon[40250]: pgmap v14739: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:56 smithi122 ceph-mon[32255]: pgmap v14740: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:56 smithi022 ceph-mon[40250]: pgmap v14740: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:56 smithi022 ceph-mon[31360]: pgmap v14740: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:09:57.202 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:09:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:09:57.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:09:57.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:09:57.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:09:58.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:09:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:09:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:57 smithi122 ceph-mon[32255]: pgmap v14741: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/755437252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:57 smithi022 ceph-mon[31360]: pgmap v14741: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/755437252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:57 smithi022 ceph-mon[40250]: pgmap v14741: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:09:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/755437252' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:09:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:09:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:09:59 smithi122 ceph-mon[32255]: pgmap v14742: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:09:59 smithi022 ceph-mon[31360]: pgmap v14742: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:09:59 smithi022 ceph-mon[40250]: pgmap v14742: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:00] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:10:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T02:10:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T02:10:01.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T02:10:02.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:01 smithi122 ceph-mon[32255]: pgmap v14743: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:01 smithi022 ceph-mon[31360]: pgmap v14743: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:01 smithi022 ceph-mon[40250]: pgmap v14743: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:02.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:10:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:10:02.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1778719330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1778719330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1778719330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:03 smithi022 ceph-mon[31360]: pgmap v14744: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:03 smithi022 ceph-mon[40250]: pgmap v14744: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:03 smithi122 ceph-mon[32255]: pgmap v14744: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:06 smithi122 ceph-mon[32255]: pgmap v14745: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:06 smithi022 ceph-mon[31360]: pgmap v14745: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:06 smithi022 ceph-mon[40250]: pgmap v14745: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:10:07.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:10:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:10:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:07 smithi122 ceph-mon[32255]: pgmap v14746: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:07 smithi022 ceph-mon[31360]: pgmap v14746: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:07 smithi022 ceph-mon[40250]: pgmap v14746: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:08.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:08.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:08.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1419666570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1419666570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1419666570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:10.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:09 smithi122 ceph-mon[32255]: pgmap v14747: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:09 smithi022 ceph-mon[31360]: pgmap v14747: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:09 smithi022 ceph-mon[40250]: pgmap v14747: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:10] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:10:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:12 smithi122 ceph-mon[32255]: pgmap v14748: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:12 smithi022 ceph-mon[31360]: pgmap v14748: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:12 smithi022 ceph-mon[40250]: pgmap v14748: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:10:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:10:13.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:13.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:13.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:13.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:14.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:14 smithi122 ceph-mon[32255]: pgmap v14749: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:10:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:10:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:10:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2892589801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:14.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:14 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:10:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[31360]: pgmap v14749: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:10:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:10:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:10:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2892589801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:10:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[40250]: pgmap v14749: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:10:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:10:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:10:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2892589801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:14 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:10:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:16 smithi122 ceph-mon[32255]: pgmap v14750: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:16 smithi022 ceph-mon[31360]: pgmap v14750: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:16 smithi022 ceph-mon[40250]: pgmap v14750: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:10:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:17 smithi122 ceph-mon[32255]: pgmap v14751: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:18.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:10:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:10:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:17 smithi022 ceph-mon[31360]: pgmap v14751: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:17 smithi022 ceph-mon[40250]: pgmap v14751: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:18.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:19.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:19.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:19.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:19 smithi122 ceph-mon[32255]: pgmap v14752: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3645270687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:19 smithi022 ceph-mon[40250]: pgmap v14752: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3645270687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:19 smithi022 ceph-mon[31360]: pgmap v14752: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3645270687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:20] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:10:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:22 smithi122 ceph-mon[32255]: pgmap v14753: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:22 smithi022 ceph-mon[31360]: pgmap v14753: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:22 smithi022 ceph-mon[40250]: pgmap v14753: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:10:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:10:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:24.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:24.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:24.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:24.429 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:24 smithi022 ceph-mon[31360]: pgmap v14754: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:24.429 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:24 smithi022 ceph-mon[40250]: pgmap v14754: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:24 smithi122 ceph-mon[32255]: pgmap v14754: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2595961650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2595961650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2595961650' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:26 smithi122 ceph-mon[32255]: pgmap v14755: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:26 smithi022 ceph-mon[31360]: pgmap v14755: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:26 smithi022 ceph-mon[40250]: pgmap v14755: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:10:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:27 smithi122 ceph-mon[32255]: pgmap v14756: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:28.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:10:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:10:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:27 smithi022 ceph-mon[31360]: pgmap v14756: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:27 smithi022 ceph-mon[40250]: pgmap v14756: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:29.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:29.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:29.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:29.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:29 smithi122 ceph-mon[32255]: pgmap v14757: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2413022829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:29 smithi022 ceph-mon[31360]: pgmap v14757: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2413022829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:29 smithi022 ceph-mon[40250]: pgmap v14757: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:30.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2413022829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:30] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:10:32.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:32 smithi122 ceph-mon[32255]: pgmap v14758: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:10:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:10:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:32 smithi022 ceph-mon[31360]: pgmap v14758: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:32 smithi022 ceph-mon[40250]: pgmap v14758: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:34 smithi122 ceph-mon[32255]: pgmap v14759: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:10:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:10:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:10:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:10:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[31360]: pgmap v14759: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:10:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:10:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:10:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:10:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[40250]: pgmap v14759: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:10:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:10:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:10:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:10:34.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:35.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:35.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:35.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:36 smithi122 ceph-mon[32255]: pgmap v14760: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4146400654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:36 smithi022 ceph-mon[31360]: pgmap v14760: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4146400654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:36 smithi022 ceph-mon[40250]: pgmap v14760: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4146400654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:10:38.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:10:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:10:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:37 smithi122 ceph-mon[32255]: pgmap v14761: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:37 smithi022 ceph-mon[31360]: pgmap v14761: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:37 smithi022 ceph-mon[40250]: pgmap v14761: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:40.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:40.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:39 smithi022 ceph-mon[31360]: pgmap v14762: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:40.171 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:39 smithi022 ceph-mon[40250]: pgmap v14762: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:39 smithi122 ceph-mon[32255]: pgmap v14762: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:40] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:10:40.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:40.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:40.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:41.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/388616523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/388616523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/388616523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:42.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:41 smithi122 ceph-mon[32255]: pgmap v14763: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:41 smithi022 ceph-mon[31360]: pgmap v14763: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:41 smithi022 ceph-mon[40250]: pgmap v14763: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:10:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:10:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:44 smithi122 ceph-mon[32255]: pgmap v14764: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:44 smithi022 ceph-mon[31360]: pgmap v14764: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:44 smithi022 ceph-mon[40250]: pgmap v14764: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:45.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:45.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:45.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:45.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:46 smithi022 ceph-mon[31360]: pgmap v14765: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/232595102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:46 smithi022 ceph-mon[40250]: pgmap v14765: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/232595102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:46 smithi122 ceph-mon[32255]: pgmap v14765: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/232595102' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:10:48.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:10:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:10:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:47 smithi122 ceph-mon[32255]: pgmap v14766: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:47 smithi022 ceph-mon[31360]: pgmap v14766: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:47 smithi022 ceph-mon[40250]: pgmap v14766: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:50.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:49 smithi122 ceph-mon[32255]: pgmap v14767: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:49 smithi022 ceph-mon[31360]: pgmap v14767: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:49 smithi022 ceph-mon[40250]: pgmap v14767: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:50] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:10:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:51.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:52.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:52 smithi122 ceph-mon[32255]: pgmap v14768: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1443898835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:10:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:10:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:52 smithi022 ceph-mon[31360]: pgmap v14768: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1443898835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:52 smithi022 ceph-mon[40250]: pgmap v14768: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1443898835' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:54 smithi122 ceph-mon[32255]: pgmap v14769: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:54 smithi022 ceph-mon[31360]: pgmap v14769: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:54 smithi022 ceph-mon[40250]: pgmap v14769: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:56.279 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:10:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:56 smithi122 ceph-mon[32255]: pgmap v14770: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:56 smithi022 ceph-mon[31360]: pgmap v14770: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:56 smithi022 ceph-mon[40250]: pgmap v14770: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:10:56.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:10:56.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:10:56.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:10:57.391 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1514640134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:57.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1514640134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1514640134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:10:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:10:58.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:10:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:10:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:58 smithi122 ceph-mon[32255]: pgmap v14771: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:58 smithi022 ceph-mon[31360]: pgmap v14771: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:58 smithi022 ceph-mon[40250]: pgmap v14771: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:10:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:10:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:10:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:10:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:10:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:00] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:11:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:00 smithi122 ceph-mon[32255]: pgmap v14772: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:00 smithi022 ceph-mon[31360]: pgmap v14772: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:00 smithi022 ceph-mon[40250]: pgmap v14772: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:01.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:02.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:02.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:02.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:02 smithi022 ceph-mon[31360]: pgmap v14773: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2593351318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:02 smithi022 ceph-mon[40250]: pgmap v14773: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2593351318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:02.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:02 smithi122 ceph-mon[32255]: pgmap v14773: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2593351318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:11:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:11:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:04 smithi122 ceph-mon[32255]: pgmap v14774: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:04 smithi022 ceph-mon[31360]: pgmap v14774: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:04 smithi022 ceph-mon[40250]: pgmap v14774: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:06 smithi122 ceph-mon[32255]: pgmap v14775: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:06 smithi022 ceph-mon[31360]: pgmap v14775: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:06 smithi022 ceph-mon[40250]: pgmap v14775: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:07.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:07.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:07.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:07.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:11:08.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:11:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:11:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:07 smithi122 ceph-mon[32255]: pgmap v14776: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/671667171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:07 smithi022 ceph-mon[31360]: pgmap v14776: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/671667171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:07 smithi022 ceph-mon[40250]: pgmap v14776: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/671667171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:09 smithi122 ceph-mon[32255]: pgmap v14777: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:09 smithi022 ceph-mon[31360]: pgmap v14777: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:09 smithi022 ceph-mon[40250]: pgmap v14777: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:10] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:11:12.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:12.390 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:12 smithi022 ceph-mon[31360]: pgmap v14778: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:12.391 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:12 smithi022 ceph-mon[40250]: pgmap v14778: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:12 smithi122 ceph-mon[32255]: pgmap v14778: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:11:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:11:12.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:12.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:12.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:13.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3681781401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3681781401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3681781401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[31360]: pgmap v14779: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:11:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:11:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:11:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[40250]: pgmap v14779: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:11:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:11:14.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:14 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:11:14.351 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:14 smithi122 ceph-mon[32255]: pgmap v14779: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:14.351 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:11:14.351 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:11:14.352 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:14 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:11:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:11:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:11:15.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:11:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:16 smithi122 ceph-mon[32255]: pgmap v14780: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:16 smithi022 ceph-mon[31360]: pgmap v14780: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:16 smithi022 ceph-mon[40250]: pgmap v14780: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:17.765 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:17.766 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:11:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:17 smithi022 ceph-mon[31360]: pgmap v14781: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:18.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:17 smithi022 ceph-mon[40250]: pgmap v14781: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:18.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:18.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:18.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:18.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:11:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:11:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:17 smithi122 ceph-mon[32255]: pgmap v14781: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/622604736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/622604736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/622604736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:19 smithi122 ceph-mon[32255]: pgmap v14782: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:19 smithi022 ceph-mon[31360]: pgmap v14782: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:19 smithi022 ceph-mon[40250]: pgmap v14782: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:20] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T02:11:22.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:22 smithi122 ceph-mon[32255]: pgmap v14783: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:11:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:11:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:22 smithi022 ceph-mon[31360]: pgmap v14783: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:22 smithi022 ceph-mon[40250]: pgmap v14783: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:23.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:23.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:23.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:23.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:24 smithi122 ceph-mon[32255]: pgmap v14784: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/637880942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:24 smithi022 ceph-mon[31360]: pgmap v14784: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/637880942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:24 smithi022 ceph-mon[40250]: pgmap v14784: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/637880942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:26 smithi122 ceph-mon[32255]: pgmap v14785: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:26 smithi022 ceph-mon[31360]: pgmap v14785: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:26 smithi022 ceph-mon[40250]: pgmap v14785: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:11:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:27 smithi122 ceph-mon[32255]: pgmap v14786: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:28.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:11:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:11:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:27 smithi022 ceph-mon[31360]: pgmap v14786: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:27 smithi022 ceph-mon[40250]: pgmap v14786: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:28.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:28.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:28.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:28.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2247983554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2247983554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2247983554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:30.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:29 smithi122 ceph-mon[32255]: pgmap v14787: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:29 smithi022 ceph-mon[31360]: pgmap v14787: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:29 smithi022 ceph-mon[40250]: pgmap v14787: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:30] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T02:11:32.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:32 smithi122 ceph-mon[32255]: pgmap v14788: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:11:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:11:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:32 smithi022 ceph-mon[31360]: pgmap v14788: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:32 smithi022 ceph-mon[40250]: pgmap v14788: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:33.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:34.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:34.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:34.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:34.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[31360]: pgmap v14789: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:34.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:11:34.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:11:34.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:11:34.252 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:11:34.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[40250]: pgmap v14789: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:34.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:11:34.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:11:34.253 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:11:34.253 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:11:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:34 smithi122 ceph-mon[32255]: pgmap v14789: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:11:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:11:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:11:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:11:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2354910551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2354910551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:35.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2354910551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:36 smithi122 ceph-mon[32255]: pgmap v14790: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:36 smithi022 ceph-mon[31360]: pgmap v14790: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:36 smithi022 ceph-mon[40250]: pgmap v14790: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:11:38.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:11:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:11:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:37 smithi122 ceph-mon[32255]: pgmap v14791: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:37 smithi022 ceph-mon[31360]: pgmap v14791: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:37 smithi022 ceph-mon[40250]: pgmap v14791: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:39.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:39.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:39.253 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:39.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:39.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:39.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:39.885 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:39 smithi022 ceph-mon[40250]: pgmap v14792: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:39.886 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3152408641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:40.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:39 smithi122 ceph-mon[32255]: pgmap v14792: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3152408641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:39 smithi022 ceph-mon[31360]: pgmap v14792: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3152408641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:40] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:11:42.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:42 smithi122 ceph-mon[32255]: pgmap v14793: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:11:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:11:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:42 smithi022 ceph-mon[31360]: pgmap v14793: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:42 smithi022 ceph-mon[40250]: pgmap v14793: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:44 smithi122 ceph-mon[32255]: pgmap v14794: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:44 smithi022 ceph-mon[31360]: pgmap v14794: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:44 smithi022 ceph-mon[40250]: pgmap v14794: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:44.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:44.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:44.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:44.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/911711698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:45.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/911711698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:45.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/911711698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:46 smithi122 ceph-mon[32255]: pgmap v14795: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:46 smithi022 ceph-mon[31360]: pgmap v14795: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:46 smithi022 ceph-mon[40250]: pgmap v14795: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:11:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:47 smithi122 ceph-mon[32255]: pgmap v14796: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:48.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:11:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:11:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:47 smithi022 ceph-mon[31360]: pgmap v14796: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:47 smithi022 ceph-mon[40250]: pgmap v14796: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:49.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:50.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:49 smithi122 ceph-mon[32255]: pgmap v14797: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:49 smithi022 ceph-mon[31360]: pgmap v14797: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:49 smithi022 ceph-mon[40250]: pgmap v14797: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:50.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:50.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:50.360 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:50] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:11:51.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3533264092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3533264092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3533264092' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:52.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:51 smithi122 ceph-mon[32255]: pgmap v14798: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:51 smithi022 ceph-mon[31360]: pgmap v14798: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:51 smithi022 ceph-mon[40250]: pgmap v14798: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:11:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:11:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:54 smithi122 ceph-mon[32255]: pgmap v14799: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:54 smithi022 ceph-mon[31360]: pgmap v14799: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:54 smithi022 ceph-mon[40250]: pgmap v14799: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:55.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:11:55.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:11:55.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:11:55.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:11:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:56 smithi122 ceph-mon[32255]: pgmap v14800: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/357964591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:56 smithi022 ceph-mon[31360]: pgmap v14800: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/357964591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:56 smithi022 ceph-mon[40250]: pgmap v14800: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:11:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/357964591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:11:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:11:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:57 smithi122 ceph-mon[32255]: pgmap v14801: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:58.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:11:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:11:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:57 smithi022 ceph-mon[31360]: pgmap v14801: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:57 smithi022 ceph-mon[40250]: pgmap v14801: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:11:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:11:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:11:59 smithi122 ceph-mon[32255]: pgmap v14802: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:11:59 smithi022 ceph-mon[31360]: pgmap v14802: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:11:59 smithi022 ceph-mon[40250]: pgmap v14802: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:00] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:12:00.733 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:01.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:01.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:01.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:02.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:12:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:12:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:02 smithi122 ceph-mon[32255]: pgmap v14803: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2730148545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:02 smithi022 ceph-mon[31360]: pgmap v14803: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2730148545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:02 smithi022 ceph-mon[40250]: pgmap v14803: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2730148545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:04 smithi122 ceph-mon[32255]: pgmap v14804: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:04 smithi022 ceph-mon[31360]: pgmap v14804: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:04 smithi022 ceph-mon[40250]: pgmap v14804: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:06.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:06 smithi122 ceph-mon[32255]: pgmap v14805: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:06.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:06.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:06.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:06.475 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:06 smithi022 ceph-mon[31360]: pgmap v14805: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:06.475 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:06 smithi022 ceph-mon[40250]: pgmap v14805: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2658399043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2658399043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:07.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2658399043' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:12:08.097 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:12:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:12:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:08 smithi122 ceph-mon[32255]: pgmap v14806: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:08 smithi022 ceph-mon[31360]: pgmap v14806: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:08 smithi022 ceph-mon[40250]: pgmap v14806: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:10] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:12:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:10 smithi122 ceph-mon[32255]: pgmap v14807: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:10 smithi022 ceph-mon[31360]: pgmap v14807: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:10 smithi022 ceph-mon[40250]: pgmap v14807: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:11.476 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:11.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:11.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:11.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:12.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:12:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:12:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:12 smithi122 ceph-mon[32255]: pgmap v14808: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/834801109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:12 smithi022 ceph-mon[31360]: pgmap v14808: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/834801109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:12 smithi022 ceph-mon[40250]: pgmap v14808: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/834801109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:14.393 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:14 smithi122 ceph-mon[32255]: pgmap v14809: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:14.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:14 smithi022 ceph-mon[31360]: pgmap v14809: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:14.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:14 smithi022 ceph-mon[40250]: pgmap v14809: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:12:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:12:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:12:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:12:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:12:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:12:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:12:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:12:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:12:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:12:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:12:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:12:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:16 smithi122 ceph-mon[32255]: pgmap v14810: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:16 smithi022 ceph-mon[31360]: pgmap v14810: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:16 smithi022 ceph-mon[40250]: pgmap v14810: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:16.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:17.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:17.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:17.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:12:18.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:12:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:12:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:17 smithi122 ceph-mon[32255]: pgmap v14811: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1792364594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:17 smithi022 ceph-mon[31360]: pgmap v14811: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1792364594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:17 smithi022 ceph-mon[40250]: pgmap v14811: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1792364594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:19.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:20.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:19 smithi122 ceph-mon[32255]: pgmap v14812: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:19 smithi022 ceph-mon[31360]: pgmap v14812: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:19 smithi022 ceph-mon[40250]: pgmap v14812: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:20] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:12:22.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:22.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:22 smithi122 ceph-mon[32255]: pgmap v14813: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:12:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:12:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:22 smithi022 ceph-mon[31360]: pgmap v14813: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:22 smithi022 ceph-mon[40250]: pgmap v14813: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:23.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2834801761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2834801761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2834801761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:24 smithi122 ceph-mon[32255]: pgmap v14814: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:24 smithi022 ceph-mon[31360]: pgmap v14814: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:24 smithi022 ceph-mon[40250]: pgmap v14814: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:26 smithi122 ceph-mon[32255]: pgmap v14815: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:26 smithi022 ceph-mon[31360]: pgmap v14815: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:26 smithi022 ceph-mon[40250]: pgmap v14815: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:12:27.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:27.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:27.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:27 smithi122 ceph-mon[32255]: pgmap v14816: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:28.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:12:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:12:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:27 smithi022 ceph-mon[31360]: pgmap v14816: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:27 smithi022 ceph-mon[40250]: pgmap v14816: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2835333022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2835333022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2835333022' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:30.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:29 smithi122 ceph-mon[32255]: pgmap v14817: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:29 smithi022 ceph-mon[31360]: pgmap v14817: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:29 smithi022 ceph-mon[40250]: pgmap v14817: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:30] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:12:32.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:12:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:12:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:32 smithi122 ceph-mon[32255]: pgmap v14818: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:32 smithi022 ceph-mon[31360]: pgmap v14818: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:32 smithi022 ceph-mon[40250]: pgmap v14818: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:32.955 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:33.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:33.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:33.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:34 smithi122 ceph-mon[32255]: pgmap v14819: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/722323929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:12:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:12:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:12:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:12:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[31360]: pgmap v14819: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/722323929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:12:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:12:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:12:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:12:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[40250]: pgmap v14819: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/722323929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:12:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:12:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:12:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:12:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:36 smithi122 ceph-mon[32255]: pgmap v14820: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:36 smithi022 ceph-mon[31360]: pgmap v14820: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:36 smithi022 ceph-mon[40250]: pgmap v14820: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:12:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:37 smithi122 ceph-mon[32255]: pgmap v14821: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:38.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:12:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:12:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:37 smithi022 ceph-mon[31360]: pgmap v14821: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:37 smithi022 ceph-mon[40250]: pgmap v14821: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:38.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:38.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:38.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:38.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:39.056 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/169453328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:39.057 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:39.058 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/169453328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/169453328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:40.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:39 smithi122 ceph-mon[32255]: pgmap v14822: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:39 smithi022 ceph-mon[31360]: pgmap v14822: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:39 smithi022 ceph-mon[40250]: pgmap v14822: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:40] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:12:42.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:42 smithi122 ceph-mon[32255]: pgmap v14823: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:12:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:12:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:42 smithi022 ceph-mon[31360]: pgmap v14823: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:42 smithi022 ceph-mon[40250]: pgmap v14823: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:43.703 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:44 smithi122 ceph-mon[32255]: pgmap v14824: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1813349295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:44 smithi022 ceph-mon[31360]: pgmap v14824: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1813349295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:44 smithi022 ceph-mon[40250]: pgmap v14824: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1813349295' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:46 smithi122 ceph-mon[32255]: pgmap v14825: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:46 smithi022 ceph-mon[31360]: pgmap v14825: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:46 smithi022 ceph-mon[40250]: pgmap v14825: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:12:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:47 smithi122 ceph-mon[32255]: pgmap v14826: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:48.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:12:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:12:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:47 smithi022 ceph-mon[31360]: pgmap v14826: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:47 smithi022 ceph-mon[40250]: pgmap v14826: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:49.438 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:49.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:49.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:50.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:49 smithi122 ceph-mon[32255]: pgmap v14827: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1543359146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:49 smithi022 ceph-mon[31360]: pgmap v14827: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1543359146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:49 smithi022 ceph-mon[40250]: pgmap v14827: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1543359146' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:50] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:12:52.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:52 smithi122 ceph-mon[32255]: pgmap v14828: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:12:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:12:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:52 smithi022 ceph-mon[31360]: pgmap v14828: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:52 smithi022 ceph-mon[40250]: pgmap v14828: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:54 smithi122 ceph-mon[32255]: pgmap v14829: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:12:54.441 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:54 smithi022 ceph-mon[31360]: pgmap v14829: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:54.442 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:54 smithi022 ceph-mon[40250]: pgmap v14829: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:54.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:12:54.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:12:54.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:12:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3643360444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3643360444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3643360444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:12:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:56 smithi122 ceph-mon[32255]: pgmap v14830: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:56 smithi022 ceph-mon[31360]: pgmap v14830: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:56 smithi022 ceph-mon[40250]: pgmap v14830: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:12:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:12:58.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:12:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:12:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:57 smithi122 ceph-mon[32255]: pgmap v14831: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:57 smithi022 ceph-mon[31360]: pgmap v14831: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:57 smithi022 ceph-mon[40250]: pgmap v14831: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:12:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:12:59.812 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:12:59 smithi122 ceph-mon[32255]: pgmap v14832: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:00.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:00.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:00.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:00.185 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:12:59 smithi022 ceph-mon[31360]: pgmap v14832: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:00.185 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:12:59 smithi022 ceph-mon[40250]: pgmap v14832: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:00] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:13:01.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1637946851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1637946851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1637946851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:01 smithi122 ceph-mon[32255]: pgmap v14833: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:01 smithi022 ceph-mon[31360]: pgmap v14833: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:01 smithi022 ceph-mon[40250]: pgmap v14833: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:13:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:13:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:04 smithi122 ceph-mon[32255]: pgmap v14834: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:04 smithi022 ceph-mon[31360]: pgmap v14834: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:04 smithi022 ceph-mon[40250]: pgmap v14834: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:05.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:05.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:05.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:05.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:06 smithi122 ceph-mon[32255]: pgmap v14835: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1703565331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:06 smithi022 ceph-mon[31360]: pgmap v14835: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1703565331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:06 smithi022 ceph-mon[40250]: pgmap v14835: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1703565331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:13:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:07 smithi122 ceph-mon[32255]: pgmap v14836: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:13:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:13:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:07 smithi022 ceph-mon[31360]: pgmap v14836: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:07 smithi022 ceph-mon[40250]: pgmap v14836: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:09 smithi122 ceph-mon[32255]: pgmap v14837: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:09 smithi022 ceph-mon[31360]: pgmap v14837: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:09 smithi022 ceph-mon[40250]: pgmap v14837: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:10] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:13:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:10.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:10.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:10.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/856812817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/856812817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/856812817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:11 smithi022 ceph-mon[31360]: pgmap v14838: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:12.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:11 smithi022 ceph-mon[40250]: pgmap v14838: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:12.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:11 smithi122 ceph-mon[32255]: pgmap v14838: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:13:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:13:14.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:14 smithi122 ceph-mon[32255]: pgmap v14839: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:14 smithi022 ceph-mon[31360]: pgmap v14839: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:14 smithi022 ceph-mon[40250]: pgmap v14839: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:15.236 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:13:15.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:13:15.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:13:15.238 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:13:15.239 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:13:15.239 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:13:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:13:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:13:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:13:15.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:16.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:16.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:16.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[31360]: pgmap v14840: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[40250]: pgmap v14840: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:16 smithi122 ceph-mon[32255]: pgmap v14840: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:13:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3829780200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:13:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3829780200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3829780200' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:18.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:13:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:13:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:18 smithi022 ceph-mon[31360]: pgmap v14841: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:18 smithi022 ceph-mon[40250]: pgmap v14841: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:18 smithi122 ceph-mon[32255]: pgmap v14841: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:20] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:13:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:20 smithi122 ceph-mon[32255]: pgmap v14842: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:20 smithi022 ceph-mon[31360]: pgmap v14842: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:20 smithi022 ceph-mon[40250]: pgmap v14842: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:21.294 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:21.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:21.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:21.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:22 smithi022 ceph-mon[31360]: pgmap v14843: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/359468049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:22 smithi022 ceph-mon[40250]: pgmap v14843: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/359468049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:22 smithi122 ceph-mon[32255]: pgmap v14843: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/359468049' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:13:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:13:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:24 smithi022 ceph-mon[31360]: pgmap v14844: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:24 smithi022 ceph-mon[40250]: pgmap v14844: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:24 smithi122 ceph-mon[32255]: pgmap v14844: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:26 smithi022 ceph-mon[31360]: pgmap v14845: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:26 smithi022 ceph-mon[40250]: pgmap v14845: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:26.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:26 smithi122 ceph-mon[32255]: pgmap v14845: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:13:28.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:13:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:13:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3344504356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:27 smithi122 ceph-mon[32255]: pgmap v14846: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3344504356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:27 smithi022 ceph-mon[31360]: pgmap v14846: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3344504356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:27 smithi022 ceph-mon[40250]: pgmap v14846: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:30.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:29 smithi122 ceph-mon[32255]: pgmap v14847: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:29 smithi022 ceph-mon[31360]: pgmap v14847: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:29 smithi022 ceph-mon[40250]: pgmap v14847: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:30] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:13:32.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:32.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:32.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:32.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:32.407 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:32 smithi022 ceph-mon[31360]: pgmap v14848: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:32.407 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:32 smithi022 ceph-mon[40250]: pgmap v14848: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:32.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:32 smithi122 ceph-mon[32255]: pgmap v14848: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:32.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:13:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:13:33.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/191734097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/191734097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/191734097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:34 smithi122 ceph-mon[32255]: pgmap v14849: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:13:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:13:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:13:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:13:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[31360]: pgmap v14849: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:13:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:13:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:13:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:13:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[40250]: pgmap v14849: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:13:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:13:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:13:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:13:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:36 smithi122 ceph-mon[32255]: pgmap v14850: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:36 smithi022 ceph-mon[31360]: pgmap v14850: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:36 smithi022 ceph-mon[40250]: pgmap v14850: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:37.407 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:37.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:37.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:37.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:13:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:37 smithi022 ceph-mon[31360]: pgmap v14851: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4074180369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:37 smithi022 ceph-mon[40250]: pgmap v14851: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4074180369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:38.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:13:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:13:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:37 smithi122 ceph-mon[32255]: pgmap v14851: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4074180369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:40.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:39 smithi122 ceph-mon[32255]: pgmap v14852: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:39 smithi022 ceph-mon[31360]: pgmap v14852: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:39 smithi022 ceph-mon[40250]: pgmap v14852: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:40] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T02:13:42.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:42 smithi122 ceph-mon[32255]: pgmap v14853: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:13:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:13:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:42 smithi022 ceph-mon[31360]: pgmap v14853: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:42 smithi022 ceph-mon[40250]: pgmap v14853: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:42.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:43.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:43.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:43.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:44 smithi122 ceph-mon[32255]: pgmap v14854: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3761019613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:44 smithi022 ceph-mon[31360]: pgmap v14854: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3761019613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:44 smithi022 ceph-mon[40250]: pgmap v14854: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3761019613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:46 smithi122 ceph-mon[32255]: pgmap v14855: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:46 smithi022 ceph-mon[31360]: pgmap v14855: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:46 smithi022 ceph-mon[40250]: pgmap v14855: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:13:48.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:48.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:47 smithi122 ceph-mon[32255]: pgmap v14856: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:48.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:13:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:13:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:47 smithi022 ceph-mon[31360]: pgmap v14856: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:47 smithi022 ceph-mon[40250]: pgmap v14856: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:48.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2999776084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2999776084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2999776084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:49 smithi122 ceph-mon[32255]: pgmap v14857: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:49 smithi022 ceph-mon[31360]: pgmap v14857: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:49 smithi022 ceph-mon[40250]: pgmap v14857: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:13:52.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:52 smithi122 ceph-mon[32255]: pgmap v14858: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:13:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:13:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:52 smithi022 ceph-mon[31360]: pgmap v14858: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:52 smithi022 ceph-mon[40250]: pgmap v14858: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:53.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:53.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:53.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:53.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:54 smithi022 ceph-mon[31360]: pgmap v14859: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3706073153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:54 smithi022 ceph-mon[40250]: pgmap v14859: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3706073153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:54 smithi122 ceph-mon[32255]: pgmap v14859: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3706073153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:13:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:56 smithi122 ceph-mon[32255]: pgmap v14860: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:56 smithi022 ceph-mon[31360]: pgmap v14860: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:56 smithi022 ceph-mon[40250]: pgmap v14860: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:13:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:13:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:57 smithi122 ceph-mon[32255]: pgmap v14861: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:58.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:13:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:13:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:57 smithi022 ceph-mon[31360]: pgmap v14861: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:57 smithi022 ceph-mon[40250]: pgmap v14861: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:13:58.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:13:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:59.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:13:59.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:13:59.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:13:59.260 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:13:59.260 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:00.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:59 smithi122 ceph-mon[32255]: pgmap v14862: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:13:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/835187476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:59 smithi022 ceph-mon[31360]: pgmap v14862: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:13:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/835187476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:59 smithi022 ceph-mon[40250]: pgmap v14862: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:13:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/835187476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:14:02.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:02 smithi122 ceph-mon[32255]: pgmap v14863: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:14:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:14:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:02 smithi022 ceph-mon[31360]: pgmap v14863: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:02 smithi022 ceph-mon[40250]: pgmap v14863: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:04.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:04 smithi122 ceph-mon[32255]: pgmap v14864: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:04 smithi022 ceph-mon[31360]: pgmap v14864: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:04 smithi022 ceph-mon[40250]: pgmap v14864: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:04.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:04.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:04.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:05.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2334187040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2334187040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2334187040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:06 smithi122 ceph-mon[32255]: pgmap v14865: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:06 smithi022 ceph-mon[31360]: pgmap v14865: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:06 smithi022 ceph-mon[40250]: pgmap v14865: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:14:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:07 smithi122 ceph-mon[32255]: pgmap v14866: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:14:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:14:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:07 smithi022 ceph-mon[31360]: pgmap v14866: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:07 smithi022 ceph-mon[40250]: pgmap v14866: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:09.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:09.918 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:09 smithi022 ceph-mon[40250]: pgmap v14867: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:10.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:10.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:10.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:10.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:09 smithi122 ceph-mon[32255]: pgmap v14867: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:09 smithi022 ceph-mon[31360]: pgmap v14867: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:10] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:14:11.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2363167809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2363167809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2363167809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:12.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:11 smithi122 ceph-mon[32255]: pgmap v14868: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:11 smithi022 ceph-mon[31360]: pgmap v14868: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:11 smithi022 ceph-mon[40250]: pgmap v14868: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:14:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:14:14.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:14 smithi122 ceph-mon[32255]: pgmap v14869: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:14 smithi022 ceph-mon[31360]: pgmap v14869: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:14 smithi022 ceph-mon[40250]: pgmap v14869: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:15.002 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:16 smithi122 ceph-mon[32255]: pgmap v14870: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4221056621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:14:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:14:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:14:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[31360]: pgmap v14870: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4221056621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:14:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:14:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:14:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[40250]: pgmap v14870: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4221056621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:14:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:14:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:14:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:14:17.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:14:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:14:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:14:18.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:18 smithi122 ceph-mon[32255]: pgmap v14871: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:18.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:14:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:14:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:18 smithi022 ceph-mon[31360]: pgmap v14871: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:18 smithi022 ceph-mon[40250]: pgmap v14871: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:20.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:20] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:14:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:20 smithi122 ceph-mon[32255]: pgmap v14872: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:20 smithi022 ceph-mon[40250]: pgmap v14872: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:20 smithi022 ceph-mon[31360]: pgmap v14872: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:20.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:20.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:20.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3128785706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3128785706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3128785706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:22.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:22 smithi122 ceph-mon[32255]: pgmap v14873: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:14:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:14:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:22 smithi022 ceph-mon[31360]: pgmap v14873: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:22 smithi022 ceph-mon[40250]: pgmap v14873: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:24 smithi122 ceph-mon[32255]: pgmap v14874: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:24 smithi022 ceph-mon[31360]: pgmap v14874: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:24 smithi022 ceph-mon[40250]: pgmap v14874: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:25.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:26.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:26.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:26.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:26 smithi122 ceph-mon[32255]: pgmap v14875: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3663052881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:26 smithi022 ceph-mon[31360]: pgmap v14875: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3663052881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:26 smithi022 ceph-mon[40250]: pgmap v14875: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3663052881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:14:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:27 smithi122 ceph-mon[32255]: pgmap v14876: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:28.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:14:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:14:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:27 smithi022 ceph-mon[31360]: pgmap v14876: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:27 smithi022 ceph-mon[40250]: pgmap v14876: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:30.184 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:30] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:14:30.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:29 smithi122 ceph-mon[32255]: pgmap v14877: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:29 smithi022 ceph-mon[31360]: pgmap v14877: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:29 smithi022 ceph-mon[40250]: pgmap v14877: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:31.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:31.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:32.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:32 smithi122 ceph-mon[32255]: pgmap v14878: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1085867624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:14:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:14:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:32 smithi022 ceph-mon[31360]: pgmap v14878: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1085867624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:32 smithi022 ceph-mon[40250]: pgmap v14878: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1085867624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:34 smithi122 ceph-mon[32255]: pgmap v14879: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:14:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:14:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:14:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:14:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[31360]: pgmap v14879: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:14:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:14:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:14:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:14:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[40250]: pgmap v14879: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:14:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:14:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:14:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:14:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:36 smithi122 ceph-mon[32255]: pgmap v14880: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:36.479 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:36 smithi022 ceph-mon[31360]: pgmap v14880: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:36.479 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:36 smithi022 ceph-mon[40250]: pgmap v14880: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:36.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:36.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:36.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:37.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4089690540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4089690540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:14:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4089690540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:38 smithi122 ceph-mon[32255]: pgmap v14881: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:38.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:14:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:14:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:38 smithi022 ceph-mon[31360]: pgmap v14881: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:38 smithi022 ceph-mon[40250]: pgmap v14881: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:40] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:14:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:40 smithi122 ceph-mon[32255]: pgmap v14882: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:40 smithi022 ceph-mon[31360]: pgmap v14882: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:40 smithi022 ceph-mon[40250]: pgmap v14882: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:42.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:42.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:42.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:42.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:42 smithi122 ceph-mon[32255]: pgmap v14883: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:14:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:14:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:42 smithi022 ceph-mon[31360]: pgmap v14883: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:42 smithi022 ceph-mon[40250]: pgmap v14883: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:43.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2801853446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2801853446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2801853446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:44 smithi122 ceph-mon[32255]: pgmap v14884: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:44 smithi022 ceph-mon[31360]: pgmap v14884: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:44 smithi022 ceph-mon[40250]: pgmap v14884: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:46 smithi122 ceph-mon[32255]: pgmap v14885: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:46 smithi022 ceph-mon[31360]: pgmap v14885: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:46 smithi022 ceph-mon[40250]: pgmap v14885: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:47.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:14:47.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:47.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:47.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:48.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:14:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:14:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:47 smithi122 ceph-mon[32255]: pgmap v14886: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1766476847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:47 smithi022 ceph-mon[31360]: pgmap v14886: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1766476847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:47 smithi022 ceph-mon[40250]: pgmap v14886: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1766476847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:49.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:50.184 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:50] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T02:14:50.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:49 smithi122 ceph-mon[32255]: pgmap v14887: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:49 smithi022 ceph-mon[31360]: pgmap v14887: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:49 smithi022 ceph-mon[40250]: pgmap v14887: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:52.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:52 smithi122 ceph-mon[32255]: pgmap v14888: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:14:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:14:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:52 smithi022 ceph-mon[31360]: pgmap v14888: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:52 smithi022 ceph-mon[40250]: pgmap v14888: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:52.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:52.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:52.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1886778824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1886778824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:53.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1886778824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:54 smithi122 ceph-mon[32255]: pgmap v14889: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:54 smithi022 ceph-mon[31360]: pgmap v14889: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:54 smithi022 ceph-mon[40250]: pgmap v14889: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:14:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:56 smithi122 ceph-mon[32255]: pgmap v14890: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:56 smithi022 ceph-mon[31360]: pgmap v14890: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:56 smithi022 ceph-mon[40250]: pgmap v14890: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:14:57.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:14:58.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:57 smithi122 ceph-mon[32255]: pgmap v14891: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:58.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:14:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:14:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:57 smithi022 ceph-mon[31360]: pgmap v14891: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:57 smithi022 ceph-mon[40250]: pgmap v14891: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:14:59.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3670492872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3670492872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:14:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:14:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3670492872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:00.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:14:59 smithi122 ceph-mon[32255]: pgmap v14892: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:00.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:00] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T02:15:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:14:59 smithi022 ceph-mon[31360]: pgmap v14892: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:14:59 smithi022 ceph-mon[40250]: pgmap v14892: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:02.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:02 smithi122 ceph-mon[32255]: pgmap v14893: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:15:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:15:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:02 smithi022 ceph-mon[31360]: pgmap v14893: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:02 smithi022 ceph-mon[40250]: pgmap v14893: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:04 smithi122 ceph-mon[32255]: pgmap v14894: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2828592442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:04 smithi022 ceph-mon[31360]: pgmap v14894: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2828592442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:04 smithi022 ceph-mon[40250]: pgmap v14894: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2828592442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:06 smithi122 ceph-mon[32255]: pgmap v14895: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:06 smithi022 ceph-mon[31360]: pgmap v14895: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:06 smithi022 ceph-mon[40250]: pgmap v14895: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:15:08.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:07 smithi122 ceph-mon[32255]: pgmap v14896: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:08.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:15:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:15:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:07 smithi022 ceph-mon[31360]: pgmap v14896: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:07 smithi022 ceph-mon[40250]: pgmap v14896: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:08.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:09.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:09.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:09.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:09.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:09.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:10.184 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:10] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:15:10.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/36169597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:10.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:09 smithi122 ceph-mon[32255]: pgmap v14897: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/36169597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:09 smithi022 ceph-mon[40250]: pgmap v14897: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/36169597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:09 smithi022 ceph-mon[31360]: pgmap v14897: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:12 smithi122 ceph-mon[32255]: pgmap v14898: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:15:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:15:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:12 smithi022 ceph-mon[31360]: pgmap v14898: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:12 smithi022 ceph-mon[40250]: pgmap v14898: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:14.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:14 smithi122 ceph-mon[32255]: pgmap v14899: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:14.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:14.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:14.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:14.468 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:14 smithi022 ceph-mon[40250]: pgmap v14899: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:14.468 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:14 smithi022 ceph-mon[31360]: pgmap v14899: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:15.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3857042356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3857042356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3857042356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:16 smithi122 ceph-mon[32255]: pgmap v14900: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:15:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:15:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:15:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[31360]: pgmap v14900: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:15:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:15:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:15:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[40250]: pgmap v14900: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:15:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:15:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:15:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:15:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:15:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:15:17.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:15:18.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:15:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:15:18.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:18 smithi122 ceph-mon[32255]: pgmap v14901: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:18 smithi022 ceph-mon[31360]: pgmap v14901: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:19.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:18 smithi022 ceph-mon[40250]: pgmap v14901: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:19.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:19.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:19.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:19.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:20] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:15:20.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:20 smithi122 ceph-mon[32255]: pgmap v14902: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1956783898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:20 smithi022 ceph-mon[31360]: pgmap v14902: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1956783898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:20 smithi022 ceph-mon[40250]: pgmap v14902: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1956783898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:22.611 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:15:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:15:22.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:22 smithi122 ceph-mon[32255]: pgmap v14903: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:22 smithi022 ceph-mon[31360]: pgmap v14903: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:22 smithi022 ceph-mon[40250]: pgmap v14903: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:23.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:23 smithi122 ceph-mon[32255]: pgmap v14904: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:23 smithi022 ceph-mon[31360]: pgmap v14904: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:24.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:23 smithi022 ceph-mon[40250]: pgmap v14904: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:24.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:26 smithi122 ceph-mon[32255]: pgmap v14905: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1750395830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:26 smithi022 ceph-mon[31360]: pgmap v14905: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1750395830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:26 smithi022 ceph-mon[40250]: pgmap v14905: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1750395830' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:15:28.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:27 smithi122 ceph-mon[32255]: pgmap v14906: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:28.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:15:27 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:15:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:27 smithi022 ceph-mon[31360]: pgmap v14906: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:27 smithi022 ceph-mon[40250]: pgmap v14906: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:29.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:30.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:30.207 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:29 smithi022 ceph-mon[40250]: pgmap v14907: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:30.207 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:29 smithi022 ceph-mon[31360]: pgmap v14907: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:29 smithi122 ceph-mon[32255]: pgmap v14907: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:30] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2261597718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2261597718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2261597718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:31 smithi022 ceph-mon[31360]: pgmap v14908: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:31 smithi022 ceph-mon[40250]: pgmap v14908: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:31 smithi122 ceph-mon[32255]: pgmap v14908: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:15:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:15:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:34 smithi122 ceph-mon[32255]: pgmap v14909: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:15:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:15:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:15:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:15:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[31360]: pgmap v14909: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:15:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:15:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:15:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:15:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[40250]: pgmap v14909: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:15:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:15:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:15:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:15:35.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:35.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:35.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:35.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:36 smithi022 ceph-mon[31360]: pgmap v14910: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2839957578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:36 smithi022 ceph-mon[40250]: pgmap v14910: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2839957578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:36 smithi122 ceph-mon[32255]: pgmap v14910: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2839957578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:15:38.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:37 smithi122 ceph-mon[32255]: pgmap v14911: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:38.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:15:37 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:15:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:37 smithi022 ceph-mon[31360]: pgmap v14911: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:37 smithi022 ceph-mon[40250]: pgmap v14911: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:39.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:39 smithi022 ceph-mon[31360]: pgmap v14912: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:39 smithi022 ceph-mon[40250]: pgmap v14912: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:40] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:15:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:39 smithi122 ceph-mon[32255]: pgmap v14912: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:40.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:41.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:41.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:41.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:42.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:15:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:15:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:42 smithi122 ceph-mon[32255]: pgmap v14913: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/226026919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:42 smithi022 ceph-mon[31360]: pgmap v14913: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/226026919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:42 smithi022 ceph-mon[40250]: pgmap v14913: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/226026919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:44 smithi122 ceph-mon[32255]: pgmap v14914: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:44 smithi022 ceph-mon[31360]: pgmap v14914: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:44 smithi022 ceph-mon[40250]: pgmap v14914: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:46.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:46 smithi122 ceph-mon[32255]: pgmap v14915: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:46 smithi022 ceph-mon[31360]: pgmap v14915: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:46 smithi022 ceph-mon[40250]: pgmap v14915: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:46.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:46.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:46.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:47.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1756035967' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1756035967' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:15:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1756035967' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:48 smithi122 ceph-mon[32255]: pgmap v14916: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:48.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:15:47 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:15:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:48 smithi022 ceph-mon[31360]: pgmap v14916: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:48 smithi022 ceph-mon[40250]: pgmap v14916: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:15:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:50 smithi122 ceph-mon[32255]: pgmap v14917: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:50 smithi022 ceph-mon[31360]: pgmap v14917: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:50 smithi022 ceph-mon[40250]: pgmap v14917: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:52.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:52.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:52.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:52.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:15:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:15:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:52 smithi122 ceph-mon[32255]: pgmap v14918: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/637623603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:52 smithi022 ceph-mon[31360]: pgmap v14918: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/637623603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:52 smithi022 ceph-mon[40250]: pgmap v14918: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/637623603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:54.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:54 smithi122 ceph-mon[32255]: pgmap v14919: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:54 smithi022 ceph-mon[31360]: pgmap v14919: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:54 smithi022 ceph-mon[40250]: pgmap v14919: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:15:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:56 smithi122 ceph-mon[32255]: pgmap v14920: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:56 smithi022 ceph-mon[31360]: pgmap v14920: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:56 smithi022 ceph-mon[40250]: pgmap v14920: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:57.062 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:15:57.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:15:57.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:15:57.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:15:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:15:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:57 smithi022 ceph-mon[31360]: pgmap v14921: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2021089664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:57 smithi022 ceph-mon[40250]: pgmap v14921: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2021089664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:58.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:15:57 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:15:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:57 smithi122 ceph-mon[32255]: pgmap v14921: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:15:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2021089664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:15:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:15:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:15:59 smithi022 ceph-mon[31360]: pgmap v14922: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:15:59 smithi022 ceph-mon[40250]: pgmap v14922: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:15:59 smithi122 ceph-mon[32255]: pgmap v14922: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:16:02.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:02.434 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:02 smithi022 ceph-mon[31360]: pgmap v14923: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:02.434 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:02 smithi022 ceph-mon[40250]: pgmap v14923: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:02 smithi122 ceph-mon[32255]: pgmap v14923: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:16:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:16:02.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:02.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:02.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:03.167 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4071024580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:03.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4071024580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4071024580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:04 smithi122 ceph-mon[32255]: pgmap v14924: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:04 smithi022 ceph-mon[31360]: pgmap v14924: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:04 smithi022 ceph-mon[40250]: pgmap v14924: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:06.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:06 smithi122 ceph-mon[32255]: pgmap v14925: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:06 smithi022 ceph-mon[31360]: pgmap v14925: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:06 smithi022 ceph-mon[40250]: pgmap v14925: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:16:07.804 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:08.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:08.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:08.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:08.176 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:07 smithi022 ceph-mon[40250]: pgmap v14926: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:08.177 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:07 smithi022 ceph-mon[31360]: pgmap v14926: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:07 smithi122 ceph-mon[32255]: pgmap v14926: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:16:07 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:16:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3700084159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3700084159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3700084159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:09 smithi022 ceph-mon[31360]: pgmap v14927: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:09 smithi022 ceph-mon[40250]: pgmap v14927: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:10] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:16:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:09 smithi122 ceph-mon[32255]: pgmap v14927: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:12.434 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:16:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:16:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:12 smithi122 ceph-mon[32255]: pgmap v14928: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:12 smithi022 ceph-mon[31360]: pgmap v14928: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:12 smithi022 ceph-mon[40250]: pgmap v14928: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:13.178 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:13.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:14.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:14 smithi122 ceph-mon[32255]: pgmap v14929: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:14.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/720683526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:14 smithi022 ceph-mon[31360]: pgmap v14929: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/720683526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:14 smithi022 ceph-mon[40250]: pgmap v14929: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/720683526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:16.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:16 smithi122 ceph-mon[32255]: pgmap v14930: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:16 smithi022 ceph-mon[31360]: pgmap v14930: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:16 smithi022 ceph-mon[40250]: pgmap v14930: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:16:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:16:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:16:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:16:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:16:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:16:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:16:17.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:16:17.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:16:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:16:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:16:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:16:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:16:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:18 smithi122 ceph-mon[32255]: pgmap v14931: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:16:17 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:16:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:18 smithi022 ceph-mon[31360]: pgmap v14931: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:18 smithi022 ceph-mon[40250]: pgmap v14931: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:18.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:18.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:18.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:18.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2591022110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2591022110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2591022110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:20 smithi122 ceph-mon[32255]: pgmap v14932: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:20] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:16:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:20 smithi022 ceph-mon[31360]: pgmap v14932: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:20 smithi022 ceph-mon[40250]: pgmap v14932: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:22.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:22 smithi122 ceph-mon[32255]: pgmap v14933: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:16:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:16:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:22 smithi022 ceph-mon[31360]: pgmap v14933: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:22 smithi022 ceph-mon[40250]: pgmap v14933: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:23.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:24.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:24.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:24.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:24.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:24 smithi122 ceph-mon[32255]: pgmap v14934: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:24 smithi022 ceph-mon[31360]: pgmap v14934: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:24.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:24 smithi022 ceph-mon[40250]: pgmap v14934: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4003388189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4003388189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4003388189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:26 smithi022 ceph-mon[31360]: pgmap v14935: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:26 smithi022 ceph-mon[40250]: pgmap v14935: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:26 smithi122 ceph-mon[32255]: pgmap v14935: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:16:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:27 smithi022 ceph-mon[40250]: pgmap v14936: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:27 smithi022 ceph-mon[31360]: pgmap v14936: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:27 smithi122 ceph-mon[32255]: pgmap v14936: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:28.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:16:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:16:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:29.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:29.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:29.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:29.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:29.953 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:29 smithi022 ceph-mon[31360]: pgmap v14937: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:29.953 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3803577527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:29 smithi022 ceph-mon[40250]: pgmap v14937: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3803577527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:30] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:16:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:29 smithi122 ceph-mon[32255]: pgmap v14937: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3803577527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:32.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:32 smithi122 ceph-mon[32255]: pgmap v14938: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:16:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:16:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:32 smithi022 ceph-mon[31360]: pgmap v14938: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:32 smithi022 ceph-mon[40250]: pgmap v14938: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:34 smithi122 ceph-mon[32255]: pgmap v14939: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:16:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:16:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:16:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:16:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[31360]: pgmap v14939: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:16:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:16:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:16:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:16:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[40250]: pgmap v14939: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:16:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:16:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:16:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:16:34.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:35.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:35.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:35.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3507174983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3507174983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:35.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3507174983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:36 smithi122 ceph-mon[32255]: pgmap v14940: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:36 smithi022 ceph-mon[31360]: pgmap v14940: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:36 smithi022 ceph-mon[40250]: pgmap v14940: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:16:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:37 smithi022 ceph-mon[31360]: pgmap v14941: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:37 smithi022 ceph-mon[40250]: pgmap v14941: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:37 smithi122 ceph-mon[32255]: pgmap v14941: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:38.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:16:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:16:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:40.029 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:39 smithi022 ceph-mon[31360]: pgmap v14942: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:39 smithi022 ceph-mon[40250]: pgmap v14942: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:40.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:40.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:40.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:39 smithi122 ceph-mon[32255]: pgmap v14942: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:40] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:16:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3459550613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3459550613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3459550613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:41 smithi022 ceph-mon[31360]: pgmap v14943: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:42.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:41 smithi022 ceph-mon[40250]: pgmap v14943: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:42.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:41 smithi122 ceph-mon[32255]: pgmap v14943: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:16:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:16:44.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:44 smithi122 ceph-mon[32255]: pgmap v14944: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:44 smithi022 ceph-mon[31360]: pgmap v14944: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:44 smithi022 ceph-mon[40250]: pgmap v14944: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:45.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:45.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:45.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:45.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:46 smithi122 ceph-mon[32255]: pgmap v14945: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:46.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3791754857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:46 smithi022 ceph-mon[31360]: pgmap v14945: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3791754857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:46 smithi022 ceph-mon[40250]: pgmap v14945: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3791754857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:16:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:47 smithi022 ceph-mon[31360]: pgmap v14946: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:47 smithi022 ceph-mon[40250]: pgmap v14946: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:48.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:16:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:16:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:47 smithi122 ceph-mon[32255]: pgmap v14946: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:49 smithi022 ceph-mon[31360]: pgmap v14947: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:49 smithi022 ceph-mon[40250]: pgmap v14947: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:50] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T02:16:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:49 smithi122 ceph-mon[32255]: pgmap v14947: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:50.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:51.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:51.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:51.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2849285214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2849285214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2849285214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:52.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:52 smithi122 ceph-mon[32255]: pgmap v14948: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:16:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:16:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:52 smithi022 ceph-mon[31360]: pgmap v14948: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:52 smithi022 ceph-mon[40250]: pgmap v14948: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:54 smithi122 ceph-mon[32255]: pgmap v14949: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:54 smithi022 ceph-mon[31360]: pgmap v14949: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:54 smithi022 ceph-mon[40250]: pgmap v14949: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:16:56.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:16:56.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:16:56.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:16:56.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:16:56.513 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:56 smithi022 ceph-mon[40250]: pgmap v14950: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:56.513 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:56 smithi022 ceph-mon[31360]: pgmap v14950: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:56 smithi122 ceph-mon[32255]: pgmap v14950: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:16:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3226946836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3226946836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3226946836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:16:58.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:16:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:16:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:58 smithi122 ceph-mon[32255]: pgmap v14951: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:58 smithi022 ceph-mon[31360]: pgmap v14951: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:58 smithi022 ceph-mon[40250]: pgmap v14951: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:16:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:16:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:16:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:16:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:16:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:00 smithi122 ceph-mon[32255]: pgmap v14952: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:00] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T02:17:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:00 smithi022 ceph-mon[31360]: pgmap v14952: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:00 smithi022 ceph-mon[40250]: pgmap v14952: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:01.514 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:02 smithi022 ceph-mon[31360]: pgmap v14953: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3524145580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:02 smithi022 ceph-mon[40250]: pgmap v14953: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3524145580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:17:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:17:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:02 smithi122 ceph-mon[32255]: pgmap v14953: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3524145580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:04 smithi022 ceph-mon[31360]: pgmap v14954: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:04 smithi022 ceph-mon[40250]: pgmap v14954: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:04 smithi122 ceph-mon[32255]: pgmap v14954: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:06 smithi022 ceph-mon[31360]: pgmap v14955: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:06 smithi022 ceph-mon[40250]: pgmap v14955: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:06 smithi122 ceph-mon[32255]: pgmap v14955: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:06.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:07.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:07.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:07.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:17:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:07 smithi022 ceph-mon[31360]: pgmap v14956: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/978926938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:07 smithi022 ceph-mon[40250]: pgmap v14956: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/978926938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:07 smithi122 ceph-mon[32255]: pgmap v14956: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/978926938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:17:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:17:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:09 smithi022 ceph-mon[31360]: pgmap v14957: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:09 smithi022 ceph-mon[40250]: pgmap v14957: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:10] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:17:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:09 smithi122 ceph-mon[32255]: pgmap v14957: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:12.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:12 smithi022 ceph-mon[31360]: pgmap v14958: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:12 smithi022 ceph-mon[40250]: pgmap v14958: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:12.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:12.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:12.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:12 smithi122 ceph-mon[32255]: pgmap v14958: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:17:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:17:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2504258817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2504258817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2504258817' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:14 smithi022 ceph-mon[31360]: pgmap v14959: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:14 smithi022 ceph-mon[40250]: pgmap v14959: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:14 smithi122 ceph-mon[32255]: pgmap v14959: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:16.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:16 smithi022 ceph-mon[31360]: pgmap v14960: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:16.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:16 smithi022 ceph-mon[40250]: pgmap v14960: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:16 smithi122 ceph-mon[32255]: pgmap v14960: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:17.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:17:17.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:17.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:18.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:17:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:17:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:17:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[31360]: pgmap v14961: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:17:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:17:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:17:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:17 smithi022 ceph-mon[40250]: pgmap v14961: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:18.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:17:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:17:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:17:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:17:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:17:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:17 smithi122 ceph-mon[32255]: pgmap v14961: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:17:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2224337156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2224337156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2224337156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:19 smithi022 ceph-mon[31360]: pgmap v14962: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:19 smithi022 ceph-mon[40250]: pgmap v14962: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:19 smithi122 ceph-mon[32255]: pgmap v14962: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:20] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:17:21.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:21.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:21.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:21.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:22.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:17:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:17:22.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2964M 2021-12-16T02:17:22.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: pgmap v14963: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:23.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:23.001 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2964M 2021-12-16T02:17:23.001 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: pgmap v14963: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:23.002 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:23.002 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.002 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.002 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.003 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.003 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.003 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.003 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.004 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.004 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:23.004 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:23.004 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2964M 2021-12-16T02:17:23.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: pgmap v14963: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:23.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:23.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:17:23.007 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:23.007 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:17:23.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:23.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:23.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:23 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3214M 2021-12-16T02:17:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2118723551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:23 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3214M 2021-12-16T02:17:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2118723551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:23.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:23 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3214M 2021-12-16T02:17:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2118723551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:24 smithi122 ceph-mon[32255]: pgmap v14964: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:24 smithi022 ceph-mon[31360]: pgmap v14964: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:24 smithi022 ceph-mon[40250]: pgmap v14964: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:26 smithi122 ceph-mon[32255]: pgmap v14965: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:26 smithi022 ceph-mon[31360]: pgmap v14965: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:26 smithi022 ceph-mon[40250]: pgmap v14965: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:17:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:27 smithi022 ceph-mon[31360]: pgmap v14966: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:27 smithi022 ceph-mon[40250]: pgmap v14966: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:28.376 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:28.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:17:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:17:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:27 smithi122 ceph-mon[32255]: pgmap v14966: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:28.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:28.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:28.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3847540416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3847540416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3847540416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:29 smithi022 ceph-mon[31360]: pgmap v14967: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:29 smithi022 ceph-mon[40250]: pgmap v14967: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:29 smithi122 ceph-mon[32255]: pgmap v14967: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:30] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:17:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:32 smithi022 ceph-mon[31360]: pgmap v14968: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:32 smithi022 ceph-mon[40250]: pgmap v14968: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:17:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:17:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:32 smithi122 ceph-mon[32255]: pgmap v14968: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:33.746 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:34.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:34.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[31360]: pgmap v14969: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:17:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:17:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:17:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:17:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/798621336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[40250]: pgmap v14969: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:17:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:17:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:17:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:17:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/798621336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:34 smithi122 ceph-mon[32255]: pgmap v14969: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:17:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/798621336' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:36 smithi022 ceph-mon[31360]: pgmap v14970: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:36.557 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:36 smithi022 ceph-mon[40250]: pgmap v14970: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:36.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:36 smithi122 ceph-mon[32255]: pgmap v14970: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:17:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:37 smithi022 ceph-mon[31360]: pgmap v14971: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:37 smithi022 ceph-mon[40250]: pgmap v14971: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:37 smithi122 ceph-mon[32255]: pgmap v14971: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:38.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:17:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:17:39.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:39.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:39.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:39.488 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:39 smithi022 ceph-mon[31360]: pgmap v14972: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2296395976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:39 smithi022 ceph-mon[40250]: pgmap v14972: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2296395976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:40] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:17:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:39 smithi122 ceph-mon[32255]: pgmap v14972: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2296395976' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:42 smithi022 ceph-mon[31360]: pgmap v14973: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:42 smithi022 ceph-mon[40250]: pgmap v14973: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:17:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:17:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:42 smithi122 ceph-mon[32255]: pgmap v14973: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:44.490 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:44.491 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:44 smithi022 ceph-mon[31360]: pgmap v14974: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:44.491 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:44 smithi022 ceph-mon[40250]: pgmap v14974: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:44 smithi122 ceph-mon[32255]: pgmap v14974: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:44.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:44.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:44.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/994034746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/994034746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/994034746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:46 smithi022 ceph-mon[31360]: pgmap v14975: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:46 smithi022 ceph-mon[40250]: pgmap v14975: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:46 smithi122 ceph-mon[32255]: pgmap v14975: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:17:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:47 smithi022 ceph-mon[31360]: pgmap v14976: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:47 smithi022 ceph-mon[40250]: pgmap v14976: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:47 smithi122 ceph-mon[32255]: pgmap v14976: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:48.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:17:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:17:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:49.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:50.230 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:49 smithi022 ceph-mon[31360]: pgmap v14977: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:50.231 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:49 smithi022 ceph-mon[40250]: pgmap v14977: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:50] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:17:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:49 smithi122 ceph-mon[32255]: pgmap v14977: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:17:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2271489245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2271489245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2271489245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:51 smithi022 ceph-mon[31360]: pgmap v14978: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:51 smithi022 ceph-mon[40250]: pgmap v14978: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:51 smithi122 ceph-mon[32255]: pgmap v14978: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:17:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:17:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:17:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:54 smithi022 ceph-mon[31360]: pgmap v14979: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:54 smithi022 ceph-mon[40250]: pgmap v14979: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:54 smithi122 ceph-mon[32255]: pgmap v14979: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:55.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:17:55.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:17:55.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:17:55.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:17:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:56 smithi022 ceph-mon[31360]: pgmap v14980: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2369189981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:56 smithi022 ceph-mon[40250]: pgmap v14980: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2369189981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:56 smithi122 ceph-mon[32255]: pgmap v14980: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2369189981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:17:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:17:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:57 smithi022 ceph-mon[31360]: pgmap v14981: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:57 smithi022 ceph-mon[40250]: pgmap v14981: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:57 smithi122 ceph-mon[32255]: pgmap v14981: 132 pgs: 132 active+clean; 582 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T02:17:58.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:17:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:17:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:17:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:17:59 smithi022 ceph-mon[31360]: pgmap v14982: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T02:18:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:17:59 smithi022 ceph-mon[40250]: pgmap v14982: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T02:18:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:00] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:18:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:17:59 smithi122 ceph-mon[32255]: pgmap v14982: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T02:18:00.601 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:00.972 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3206942245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:00.972 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3206942245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3206942245' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:01 smithi022 ceph-mon[31360]: pgmap v14983: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T02:18:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:01 smithi022 ceph-mon[40250]: pgmap v14983: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T02:18:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:01 smithi122 ceph-mon[32255]: pgmap v14983: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T02:18:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:18:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:18:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:04 smithi022 ceph-mon[31360]: pgmap v14984: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T02:18:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:04 smithi022 ceph-mon[40250]: pgmap v14984: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T02:18:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:04 smithi122 ceph-mon[32255]: pgmap v14984: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T02:18:05.974 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:06 smithi022 ceph-mon[31360]: pgmap v14985: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:06 smithi022 ceph-mon[40250]: pgmap v14985: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:06.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:06.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:06.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:06 smithi122 ceph-mon[32255]: pgmap v14985: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:18:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/14787698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/14787698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:07.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/14787698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:08 smithi122 ceph-mon[32255]: pgmap v14986: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:18:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:18:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:08 smithi022 ceph-mon[31360]: pgmap v14986: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:08 smithi022 ceph-mon[40250]: pgmap v14986: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:09.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:10 smithi122 ceph-mon[32255]: pgmap v14987: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:10] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:18:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:10 smithi022 ceph-mon[31360]: pgmap v14987: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:10 smithi022 ceph-mon[40250]: pgmap v14987: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:18:11.347 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:11.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:11.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:11.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:12 smithi022 ceph-mon[31360]: pgmap v14988: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3178451168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:12 smithi022 ceph-mon[40250]: pgmap v14988: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3178451168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:12 smithi122 ceph-mon[32255]: pgmap v14988: 132 pgs: 132 active+clean; 582 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3178451168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:18:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:18:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:14 smithi022 ceph-mon[31360]: pgmap v14989: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:14 smithi022 ceph-mon[40250]: pgmap v14989: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:14 smithi122 ceph-mon[32255]: pgmap v14989: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:16 smithi022 ceph-mon[31360]: pgmap v14990: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:16 smithi022 ceph-mon[40250]: pgmap v14990: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:16 smithi122 ceph-mon[32255]: pgmap v14990: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:16.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:17.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:17.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:17.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:18:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2246525707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:17 smithi022 ceph-mon[31360]: pgmap v14991: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2246525707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:17 smithi022 ceph-mon[40250]: pgmap v14991: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:18.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:18:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:18:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2246525707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:17 smithi122 ceph-mon[32255]: pgmap v14991: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:19 smithi022 ceph-mon[31360]: pgmap v14992: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:19 smithi022 ceph-mon[40250]: pgmap v14992: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:19 smithi122 ceph-mon[32255]: pgmap v14992: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:20] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:18:22.091 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:22 smithi122 ceph-mon[32255]: pgmap v14993: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:18:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:18:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:18:22.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:22.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:22.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:22.461 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[31360]: pgmap v14993: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:22.461 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:18:22.461 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:18:22.462 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:18:22.462 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[40250]: pgmap v14993: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:18:22.462 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:18:22.463 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:18:22.463 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:18:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:18:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:18:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2137912740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:18:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2137912740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:18:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2137912740' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:18:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:24 smithi022 ceph-mon[31360]: pgmap v14994: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:24 smithi022 ceph-mon[40250]: pgmap v14994: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:24.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:24 smithi122 ceph-mon[32255]: pgmap v14994: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:18:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:26 smithi022 ceph-mon[31360]: pgmap v14995: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:26 smithi022 ceph-mon[40250]: pgmap v14995: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:26 smithi122 ceph-mon[32255]: pgmap v14995: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:27.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:18:27.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:27.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:27.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:27 smithi022 ceph-mon[31360]: pgmap v14996: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1804541392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:27 smithi022 ceph-mon[40250]: pgmap v14996: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1804541392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:28.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:18:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:18:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:27 smithi122 ceph-mon[32255]: pgmap v14996: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1804541392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:28 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:28 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:28 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:29 smithi022 ceph-mon[31360]: pgmap v14997: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:29 smithi022 ceph-mon[40250]: pgmap v14997: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:30] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:18:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:29 smithi122 ceph-mon[32255]: pgmap v14997: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:32 smithi022 ceph-mon[31360]: pgmap v14998: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:32 smithi022 ceph-mon[40250]: pgmap v14998: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:32 smithi122 ceph-mon[32255]: pgmap v14998: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:18:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:18:32.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:33.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:33.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:33.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1574772559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1574772559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1574772559' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[31360]: pgmap v14999: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:18:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:18:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:18:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:18:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[40250]: pgmap v14999: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:18:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:18:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:18:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:18:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:34 smithi122 ceph-mon[32255]: pgmap v14999: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:18:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:18:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:18:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:18:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:36 smithi022 ceph-mon[31360]: pgmap v15000: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:36 smithi022 ceph-mon[40250]: pgmap v15000: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:36 smithi122 ceph-mon[32255]: pgmap v15000: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:18:38.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:37 smithi022 ceph-mon[31360]: pgmap v15001: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:37 smithi022 ceph-mon[40250]: pgmap v15001: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:37 smithi122 ceph-mon[32255]: pgmap v15001: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:38.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:18:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:18:38.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:38.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:38.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2650520940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2650520940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2650520940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:39 smithi022 ceph-mon[31360]: pgmap v15002: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:39 smithi022 ceph-mon[40250]: pgmap v15002: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:39 smithi122 ceph-mon[32255]: pgmap v15002: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:40] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:18:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:42 smithi022 ceph-mon[31360]: pgmap v15003: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:42 smithi022 ceph-mon[40250]: pgmap v15003: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:42 smithi122 ceph-mon[32255]: pgmap v15003: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:18:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:18:43.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:43.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:43.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:43.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:44 smithi022 ceph-mon[31360]: pgmap v15004: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1661469613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:44 smithi022 ceph-mon[40250]: pgmap v15004: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1661469613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:44 smithi122 ceph-mon[32255]: pgmap v15004: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1661469613' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:46 smithi022 ceph-mon[31360]: pgmap v15005: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:46 smithi022 ceph-mon[40250]: pgmap v15005: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:46 smithi122 ceph-mon[32255]: pgmap v15005: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:18:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:47 smithi022 ceph-mon[31360]: pgmap v15006: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:47 smithi022 ceph-mon[40250]: pgmap v15006: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:48.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:18:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:18:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:47 smithi122 ceph-mon[32255]: pgmap v15006: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:48.940 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:49.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:49.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:49.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:49 smithi022 ceph-mon[31360]: pgmap v15007: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3318454785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:49 smithi022 ceph-mon[40250]: pgmap v15007: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3318454785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:50] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:18:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:49 smithi122 ceph-mon[32255]: pgmap v15007: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3318454785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:52 smithi022 ceph-mon[31360]: pgmap v15008: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:52 smithi022 ceph-mon[40250]: pgmap v15008: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:52 smithi122 ceph-mon[32255]: pgmap v15008: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:18:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:18:54.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:54 smithi022 ceph-mon[31360]: pgmap v15009: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:54 smithi022 ceph-mon[40250]: pgmap v15009: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:54 smithi122 ceph-mon[32255]: pgmap v15009: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:18:54.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:18:54.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:18:54.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:18:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/427596584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:55.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/427596584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/427596584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:18:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:56 smithi022 ceph-mon[31360]: pgmap v15010: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:56 smithi022 ceph-mon[40250]: pgmap v15010: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:56 smithi122 ceph-mon[32255]: pgmap v15010: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:18:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:57 smithi022 ceph-mon[31360]: pgmap v15011: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:57 smithi022 ceph-mon[40250]: pgmap v15011: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:58.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:18:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:18:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:57 smithi122 ceph-mon[32255]: pgmap v15011: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:18:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:18:59.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:18:59.983 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:18:59 smithi022 ceph-mon[40250]: pgmap v15012: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:00.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:00.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:18:59 smithi022 ceph-mon[31360]: pgmap v15012: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:00] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:19:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:18:59 smithi122 ceph-mon[32255]: pgmap v15012: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3821417057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3821417057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3821417057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:01 smithi022 ceph-mon[31360]: pgmap v15013: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:01 smithi022 ceph-mon[40250]: pgmap v15013: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:01 smithi122 ceph-mon[32255]: pgmap v15013: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:19:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:19:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:04 smithi022 ceph-mon[31360]: pgmap v15014: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:04 smithi022 ceph-mon[40250]: pgmap v15014: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:04 smithi122 ceph-mon[32255]: pgmap v15014: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:05.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:05.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:05.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:05.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:06 smithi022 ceph-mon[31360]: pgmap v15015: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1144082653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:06 smithi022 ceph-mon[40250]: pgmap v15015: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1144082653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:06 smithi122 ceph-mon[32255]: pgmap v15015: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1144082653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:19:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:07 smithi022 ceph-mon[31360]: pgmap v15016: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:07 smithi022 ceph-mon[40250]: pgmap v15016: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:07 smithi122 ceph-mon[32255]: pgmap v15016: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:19:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:19:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:09 smithi022 ceph-mon[31360]: pgmap v15017: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:09 smithi022 ceph-mon[40250]: pgmap v15017: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:10.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:10] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:19:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:09 smithi122 ceph-mon[32255]: pgmap v15017: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:10.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:10.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:10.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3644302786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:11.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3644302786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3644302786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:11 smithi022 ceph-mon[31360]: pgmap v15018: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:11 smithi022 ceph-mon[40250]: pgmap v15018: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:12.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:11 smithi122 ceph-mon[32255]: pgmap v15018: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:19:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:19:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:14 smithi022 ceph-mon[31360]: pgmap v15019: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:14 smithi022 ceph-mon[40250]: pgmap v15019: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:14 smithi122 ceph-mon[32255]: pgmap v15019: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:15.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:16.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:16.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:16.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:16 smithi022 ceph-mon[31360]: pgmap v15020: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3871921789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:16 smithi022 ceph-mon[40250]: pgmap v15020: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3871921789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:16 smithi122 ceph-mon[32255]: pgmap v15020: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3871921789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:19:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:17 smithi022 ceph-mon[31360]: pgmap v15021: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:17 smithi022 ceph-mon[40250]: pgmap v15021: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:17 smithi122 ceph-mon[32255]: pgmap v15021: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:19:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:19:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:19 smithi022 ceph-mon[31360]: pgmap v15022: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:19 smithi022 ceph-mon[40250]: pgmap v15022: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:19 smithi122 ceph-mon[32255]: pgmap v15022: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:20] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:19:21.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:21.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:21.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:21.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:22 smithi022 ceph-mon[31360]: pgmap v15023: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1773841351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:22 smithi022 ceph-mon[40250]: pgmap v15023: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1773841351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:22.546 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:19:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:19:22.547 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:22 smithi122 ceph-mon[32255]: pgmap v15023: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:22.547 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1773841351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:19:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:19:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:19:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:19:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:19:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:19:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:19:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:19:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:19:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:19:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:19:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:19:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:24 smithi022 ceph-mon[31360]: pgmap v15024: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:24 smithi022 ceph-mon[40250]: pgmap v15024: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:24 smithi122 ceph-mon[32255]: pgmap v15024: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:26 smithi022 ceph-mon[31360]: pgmap v15025: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:26 smithi022 ceph-mon[40250]: pgmap v15025: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:26.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:26 smithi122 ceph-mon[32255]: pgmap v15025: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:26.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3663668644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:19:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3663668644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3663668644' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:28 smithi122 ceph-mon[32255]: pgmap v15026: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:28.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:19:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:19:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:28 smithi022 ceph-mon[31360]: pgmap v15026: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:28 smithi022 ceph-mon[40250]: pgmap v15026: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:30] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:19:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:30 smithi122 ceph-mon[32255]: pgmap v15027: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:30 smithi022 ceph-mon[31360]: pgmap v15027: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:30 smithi022 ceph-mon[40250]: pgmap v15027: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:31.905 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:32.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:32.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:32.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:32 smithi022 ceph-mon[31360]: pgmap v15028: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:32 smithi022 ceph-mon[40250]: pgmap v15028: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:32 smithi122 ceph-mon[32255]: pgmap v15028: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:19:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:19:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1079592155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1079592155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1079592155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[31360]: pgmap v15029: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:19:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:19:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[40250]: pgmap v15029: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:19:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:19:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:34 smithi122 ceph-mon[32255]: pgmap v15029: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:19:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:19:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:36 smithi022 ceph-mon[31360]: pgmap v15030: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:36 smithi022 ceph-mon[40250]: pgmap v15030: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:36 smithi122 ceph-mon[32255]: pgmap v15030: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:37.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:19:37.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:37.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:37.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:37 smithi022 ceph-mon[31360]: pgmap v15031: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/852888461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:37 smithi022 ceph-mon[40250]: pgmap v15031: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/852888461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:38.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:19:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:19:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:37 smithi122 ceph-mon[32255]: pgmap v15031: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/852888461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:38 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:38 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:38 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:39 smithi022 ceph-mon[31360]: pgmap v15032: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:39 smithi022 ceph-mon[40250]: pgmap v15032: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:39 smithi122 ceph-mon[32255]: pgmap v15032: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:40] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:19:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:42 smithi022 ceph-mon[31360]: pgmap v15033: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:42 smithi022 ceph-mon[40250]: pgmap v15033: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:42.641 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:42 smithi122 ceph-mon[32255]: pgmap v15033: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:19:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:19:43.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:43.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:43.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3871176236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3871176236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3871176236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:44 smithi022 ceph-mon[31360]: pgmap v15034: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:44 smithi022 ceph-mon[40250]: pgmap v15034: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:44 smithi122 ceph-mon[32255]: pgmap v15034: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:46 smithi022 ceph-mon[31360]: pgmap v15035: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:46 smithi022 ceph-mon[40250]: pgmap v15035: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:46 smithi122 ceph-mon[32255]: pgmap v15035: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:19:48.016 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:47 smithi022 ceph-mon[31360]: pgmap v15036: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:47 smithi022 ceph-mon[40250]: pgmap v15036: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:48.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:48.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:48.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:47 smithi122 ceph-mon[32255]: pgmap v15036: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:48.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:19:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:19:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:48 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3185518984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:48 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3185518984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:48 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3185518984' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:49 smithi022 ceph-mon[31360]: pgmap v15037: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:49 smithi022 ceph-mon[40250]: pgmap v15037: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:49 smithi122 ceph-mon[32255]: pgmap v15037: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:50] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:19:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:52 smithi022 ceph-mon[31360]: pgmap v15038: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:52 smithi022 ceph-mon[40250]: pgmap v15038: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:52 smithi122 ceph-mon[32255]: pgmap v15038: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:19:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:19:53.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:53.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:53.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:53.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:54 smithi022 ceph-mon[31360]: pgmap v15039: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3948702248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:54 smithi022 ceph-mon[40250]: pgmap v15039: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3948702248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:54 smithi122 ceph-mon[32255]: pgmap v15039: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:19:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3948702248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:19:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:56 smithi022 ceph-mon[31360]: pgmap v15040: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:56 smithi022 ceph-mon[40250]: pgmap v15040: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:56 smithi122 ceph-mon[32255]: pgmap v15040: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:19:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:57 smithi022 ceph-mon[31360]: pgmap v15041: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:57 smithi022 ceph-mon[40250]: pgmap v15041: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:57 smithi122 ceph-mon[32255]: pgmap v15041: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:19:58.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:19:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:19:58.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:19:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:58 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:58 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:19:59.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:19:59.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:19:59.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:19:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:58 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1183789675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:19:59 smithi022 ceph-mon[31360]: pgmap v15042: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1183789675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:19:59 smithi022 ceph-mon[40250]: pgmap v15042: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:00] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:20:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1183789675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:19:59 smithi122 ceph-mon[32255]: pgmap v15042: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T02:20:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T02:20:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T02:20:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:02 smithi122 ceph-mon[32255]: pgmap v15043: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:02 smithi022 ceph-mon[31360]: pgmap v15043: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:02.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:02 smithi022 ceph-mon[40250]: pgmap v15043: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:20:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:20:04.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:04.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:04 smithi022 ceph-mon[40250]: pgmap v15044: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:04.496 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:04 smithi022 ceph-mon[31360]: pgmap v15044: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:04 smithi122 ceph-mon[32255]: pgmap v15044: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2014791117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2014791117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2014791117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:06 smithi022 ceph-mon[31360]: pgmap v15045: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:06 smithi022 ceph-mon[40250]: pgmap v15045: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:06 smithi122 ceph-mon[32255]: pgmap v15045: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:20:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:07 smithi022 ceph-mon[31360]: pgmap v15046: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:07 smithi022 ceph-mon[40250]: pgmap v15046: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:07 smithi122 ceph-mon[32255]: pgmap v15046: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:20:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:20:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:08 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:08 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:08 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:09.497 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:09.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:09.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:09.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:09 smithi022 ceph-mon[31360]: pgmap v15047: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2382497936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:09 smithi022 ceph-mon[40250]: pgmap v15047: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2382497936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:09 smithi122 ceph-mon[32255]: pgmap v15047: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2382497936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:20:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:12 smithi022 ceph-mon[31360]: pgmap v15048: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:12 smithi022 ceph-mon[40250]: pgmap v15048: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:12 smithi122 ceph-mon[32255]: pgmap v15048: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:20:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:20:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:14 smithi022 ceph-mon[31360]: pgmap v15049: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:14 smithi022 ceph-mon[40250]: pgmap v15049: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:14 smithi122 ceph-mon[32255]: pgmap v15049: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:14.868 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:15.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:15.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3720930944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3720930944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3720930944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:16 smithi022 ceph-mon[31360]: pgmap v15050: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:16 smithi022 ceph-mon[40250]: pgmap v15050: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:16 smithi122 ceph-mon[32255]: pgmap v15050: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:20:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:17 smithi022 ceph-mon[31360]: pgmap v15051: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:17 smithi022 ceph-mon[40250]: pgmap v15051: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:17 smithi122 ceph-mon[32255]: pgmap v15051: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:18.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:20:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:20:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:18 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:18 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:18 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:20.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:20 smithi022 ceph-mon[31360]: pgmap v15052: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:20 smithi022 ceph-mon[40250]: pgmap v15052: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:20 smithi122 ceph-mon[32255]: pgmap v15052: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:20] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:20:20.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:20.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:20.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2627812671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:21.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2627812671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2627812671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:22 smithi022 ceph-mon[31360]: pgmap v15053: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:22 smithi022 ceph-mon[40250]: pgmap v15053: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:22.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:22 smithi122 ceph-mon[32255]: pgmap v15053: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:20:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:20:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:20:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:20:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:20:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:20:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:20:23.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:20:23.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:20:23.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:20:23.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:20:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:24 smithi022 ceph-mon[31360]: pgmap v15054: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:20:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:24 smithi022 ceph-mon[40250]: pgmap v15054: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:20:24.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:24 smithi122 ceph-mon[32255]: pgmap v15054: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:20:25.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:25.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:25.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:25.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:26 smithi022 ceph-mon[31360]: pgmap v15055: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3144293038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:26 smithi022 ceph-mon[40250]: pgmap v15055: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3144293038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:26 smithi122 ceph-mon[32255]: pgmap v15055: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3144293038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:20:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:28 smithi022 ceph-mon[31360]: pgmap v15056: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:28 smithi022 ceph-mon[40250]: pgmap v15056: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:28.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:28 smithi122 ceph-mon[32255]: pgmap v15056: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:28.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:20:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:20:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:30 smithi022 ceph-mon[31360]: pgmap v15057: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:30 smithi022 ceph-mon[40250]: pgmap v15057: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:30 smithi122 ceph-mon[32255]: pgmap v15057: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:30] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:20:30.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:31.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:31.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:31.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:32 smithi022 ceph-mon[31360]: pgmap v15058: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2656923487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:32 smithi022 ceph-mon[40250]: pgmap v15058: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2656923487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:20:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:20:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:32 smithi122 ceph-mon[32255]: pgmap v15058: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2656923487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[31360]: pgmap v15059: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:20:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:20:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:20:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:20:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[40250]: pgmap v15059: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:20:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:20:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:20:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:20:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:34 smithi122 ceph-mon[32255]: pgmap v15059: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:20:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:20:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:20:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:20:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:36 smithi022 ceph-mon[31360]: pgmap v15060: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:36 smithi022 ceph-mon[40250]: pgmap v15060: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:36 smithi122 ceph-mon[32255]: pgmap v15060: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/196513534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:20:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/196513534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:37.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/196513534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:38 smithi022 ceph-mon[31360]: pgmap v15061: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:38 smithi022 ceph-mon[40250]: pgmap v15061: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:38.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:20:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:20:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:38 smithi122 ceph-mon[32255]: pgmap v15061: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:40] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:20:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:40 smithi122 ceph-mon[32255]: pgmap v15062: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:40 smithi022 ceph-mon[31360]: pgmap v15062: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:40 smithi022 ceph-mon[40250]: pgmap v15062: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:41.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:42.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:42 smithi022 ceph-mon[31360]: pgmap v15063: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1855245202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:42 smithi022 ceph-mon[40250]: pgmap v15063: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1855245202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:20:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:20:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:42 smithi122 ceph-mon[32255]: pgmap v15063: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1855245202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:44 smithi022 ceph-mon[31360]: pgmap v15064: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:44 smithi022 ceph-mon[40250]: pgmap v15064: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:44 smithi122 ceph-mon[32255]: pgmap v15064: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:46 smithi022 ceph-mon[31360]: pgmap v15065: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:46 smithi022 ceph-mon[40250]: pgmap v15065: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:46 smithi122 ceph-mon[32255]: pgmap v15065: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:47.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:47.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:47.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:47.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:20:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:47 smithi022 ceph-mon[31360]: pgmap v15066: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3079429181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:47 smithi022 ceph-mon[40250]: pgmap v15066: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3079429181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:48.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:20:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:20:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:47 smithi122 ceph-mon[32255]: pgmap v15066: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3079429181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:49.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:50 smithi022 ceph-mon[31360]: pgmap v15067: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:50 smithi022 ceph-mon[40250]: pgmap v15067: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:50] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:20:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:50 smithi122 ceph-mon[32255]: pgmap v15067: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:52.461 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:52 smithi022 ceph-mon[31360]: pgmap v15068: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:52 smithi022 ceph-mon[40250]: pgmap v15068: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:52 smithi122 ceph-mon[32255]: pgmap v15068: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:20:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:20:52.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:52.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:52.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4273043573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4273043573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4273043573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:54 smithi022 ceph-mon[31360]: pgmap v15069: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:54 smithi022 ceph-mon[40250]: pgmap v15069: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:54 smithi122 ceph-mon[32255]: pgmap v15069: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:20:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:56 smithi022 ceph-mon[31360]: pgmap v15070: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:56 smithi022 ceph-mon[40250]: pgmap v15070: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:56 smithi122 ceph-mon[32255]: pgmap v15070: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:20:57.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:20:58.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:20:58.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:20:58.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:20:58.203 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:58 smithi022 ceph-mon[31360]: pgmap v15071: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:58.203 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:58 smithi022 ceph-mon[40250]: pgmap v15071: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:58.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:20:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:20:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:58 smithi122 ceph-mon[32255]: pgmap v15071: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:20:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3065508431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:20:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3065508431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:20:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:20:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3065508431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:20:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:20:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:00 smithi022 ceph-mon[31360]: pgmap v15072: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:00 smithi022 ceph-mon[40250]: pgmap v15072: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:00 smithi122 ceph-mon[32255]: pgmap v15072: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:00] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:21:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:02 smithi022 ceph-mon[31360]: pgmap v15073: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:02 smithi022 ceph-mon[40250]: pgmap v15073: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:02 smithi122 ceph-mon[32255]: pgmap v15073: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:21:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:21:03.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:03.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:03.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:03.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:04 smithi022 ceph-mon[31360]: pgmap v15074: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3690852375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:04 smithi022 ceph-mon[40250]: pgmap v15074: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3690852375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:04 smithi122 ceph-mon[32255]: pgmap v15074: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3690852375' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:06 smithi022 ceph-mon[31360]: pgmap v15075: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:06 smithi022 ceph-mon[40250]: pgmap v15075: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:06 smithi122 ceph-mon[32255]: pgmap v15075: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:21:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:08 smithi022 ceph-mon[31360]: pgmap v15076: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:08 smithi022 ceph-mon[40250]: pgmap v15076: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:08 smithi122 ceph-mon[32255]: pgmap v15076: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:21:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:21:08.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:08.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:08.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:08.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/675446761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/675446761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/675446761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:10 smithi022 ceph-mon[31360]: pgmap v15077: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:10 smithi022 ceph-mon[40250]: pgmap v15077: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:10 smithi122 ceph-mon[32255]: pgmap v15077: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:10] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:21:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:12 smithi022 ceph-mon[31360]: pgmap v15078: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:12 smithi022 ceph-mon[40250]: pgmap v15078: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:12 smithi122 ceph-mon[32255]: pgmap v15078: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:21:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:21:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:14.258 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:14 smithi022 ceph-mon[40250]: pgmap v15079: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:14.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:14.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:14.316 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:14 smithi022 ceph-mon[31360]: pgmap v15079: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:14 smithi122 ceph-mon[32255]: pgmap v15079: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3285377528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3285377528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3285377528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:16 smithi022 ceph-mon[31360]: pgmap v15080: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:16 smithi022 ceph-mon[40250]: pgmap v15080: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:16 smithi122 ceph-mon[32255]: pgmap v15080: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:21:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:18 smithi022 ceph-mon[31360]: pgmap v15081: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:18 smithi022 ceph-mon[40250]: pgmap v15081: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:18 smithi122 ceph-mon[32255]: pgmap v15081: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:18.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:21:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:21:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:19.318 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:19.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:19.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:19.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:20 smithi022 ceph-mon[31360]: pgmap v15082: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4075031086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:20 smithi022 ceph-mon[40250]: pgmap v15082: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4075031086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:21:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:20 smithi122 ceph-mon[32255]: pgmap v15082: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4075031086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:22 smithi022 ceph-mon[31360]: pgmap v15083: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:22 smithi022 ceph-mon[40250]: pgmap v15083: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:21:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:21:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:22 smithi122 ceph-mon[32255]: pgmap v15083: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[31360]: pgmap v15084: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:21:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:21:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:21:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:21:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[40250]: pgmap v15084: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:21:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:21:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:21:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:21:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:24 smithi122 ceph-mon[32255]: pgmap v15084: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:21:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:21:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:21:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:21:24.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:25.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:25.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:25.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4152592196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4152592196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4152592196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:26 smithi022 ceph-mon[31360]: pgmap v15085: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:26 smithi022 ceph-mon[40250]: pgmap v15085: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:26 smithi122 ceph-mon[32255]: pgmap v15085: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:21:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:28 smithi022 ceph-mon[31360]: pgmap v15086: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:28 smithi022 ceph-mon[40250]: pgmap v15086: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:28.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:21:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:21:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:28 smithi122 ceph-mon[32255]: pgmap v15086: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:30.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:30 smithi022 ceph-mon[31360]: pgmap v15087: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:30.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:30 smithi022 ceph-mon[40250]: pgmap v15087: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:30.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:30.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:30.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:21:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:30 smithi122 ceph-mon[32255]: pgmap v15087: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3602819314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:31.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3602819314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:31.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3602819314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:32.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:32 smithi022 ceph-mon[40250]: pgmap v15088: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:32 smithi022 ceph-mon[31360]: pgmap v15088: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:32 smithi122 ceph-mon[32255]: pgmap v15088: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:21:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:21:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[31360]: pgmap v15089: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:21:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:21:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:21:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:21:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[40250]: pgmap v15089: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:21:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:21:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:21:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:21:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:34 smithi122 ceph-mon[32255]: pgmap v15089: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:21:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:21:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:21:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:21:35.430 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:35.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:35.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:35.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:36 smithi022 ceph-mon[31360]: pgmap v15090: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/223216689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:36 smithi022 ceph-mon[40250]: pgmap v15090: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/223216689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:36 smithi122 ceph-mon[32255]: pgmap v15090: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/223216689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:21:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:38 smithi022 ceph-mon[31360]: pgmap v15091: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:38 smithi022 ceph-mon[40250]: pgmap v15091: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:38.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:21:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:21:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:38 smithi122 ceph-mon[32255]: pgmap v15091: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:40 smithi022 ceph-mon[31360]: pgmap v15092: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:40 smithi022 ceph-mon[40250]: pgmap v15092: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:40] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:21:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:40 smithi122 ceph-mon[32255]: pgmap v15092: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:40.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:41.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:41.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:41.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2317450938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2317450938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:41.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2317450938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:42 smithi022 ceph-mon[31360]: pgmap v15093: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:42 smithi022 ceph-mon[40250]: pgmap v15093: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:42 smithi122 ceph-mon[32255]: pgmap v15093: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:21:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:21:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:44 smithi022 ceph-mon[31360]: pgmap v15094: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:44 smithi022 ceph-mon[40250]: pgmap v15094: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:44 smithi122 ceph-mon[32255]: pgmap v15094: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:46.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:46 smithi022 ceph-mon[31360]: pgmap v15095: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:46 smithi022 ceph-mon[40250]: pgmap v15095: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:46.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:46.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:46.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:46 smithi122 ceph-mon[32255]: pgmap v15095: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3582485535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:21:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3582485535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:47.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3582485535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:48 smithi022 ceph-mon[31360]: pgmap v15096: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:48 smithi022 ceph-mon[40250]: pgmap v15096: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:48.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:21:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:21:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:48 smithi122 ceph-mon[32255]: pgmap v15096: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:49.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:21:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:50 smithi122 ceph-mon[32255]: pgmap v15097: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:50 smithi022 ceph-mon[31360]: pgmap v15097: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:50 smithi022 ceph-mon[40250]: pgmap v15097: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:51.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:51.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:51.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:51.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:52 smithi022 ceph-mon[31360]: pgmap v15098: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/33856801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:52 smithi022 ceph-mon[40250]: pgmap v15098: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/33856801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:21:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:21:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:52 smithi122 ceph-mon[32255]: pgmap v15098: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/33856801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:54 smithi022 ceph-mon[31360]: pgmap v15099: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:54 smithi022 ceph-mon[40250]: pgmap v15099: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:54 smithi122 ceph-mon[32255]: pgmap v15099: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:21:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:56 smithi022 ceph-mon[31360]: pgmap v15100: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:56 smithi022 ceph-mon[40250]: pgmap v15100: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:56 smithi122 ceph-mon[32255]: pgmap v15100: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:56.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:21:57.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:21:57.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:21:57.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:21:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:21:58.267 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:58 smithi122 ceph-mon[32255]: pgmap v15101: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:58.267 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/511262966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:58 smithi022 ceph-mon[31360]: pgmap v15101: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/511262966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:58 smithi022 ceph-mon[40250]: pgmap v15101: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:21:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/511262966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:21:58.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:21:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:21:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:21:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:59.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:21:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:21:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:21:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:00 smithi022 ceph-mon[31360]: pgmap v15102: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:00 smithi022 ceph-mon[40250]: pgmap v15102: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:00 smithi122 ceph-mon[32255]: pgmap v15102: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:22:02.282 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:02 smithi022 ceph-mon[31360]: pgmap v15103: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:02 smithi022 ceph-mon[40250]: pgmap v15103: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:02.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:02.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:02.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:02 smithi122 ceph-mon[32255]: pgmap v15103: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:22:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:22:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1069763888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1069763888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1069763888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:04 smithi022 ceph-mon[31360]: pgmap v15104: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:04.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:04 smithi022 ceph-mon[40250]: pgmap v15104: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:04 smithi122 ceph-mon[32255]: pgmap v15104: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:06 smithi122 ceph-mon[32255]: pgmap v15105: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:06 smithi022 ceph-mon[31360]: pgmap v15105: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:06 smithi022 ceph-mon[40250]: pgmap v15105: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:07.652 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:07.653 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:22:08.018 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:08 smithi022 ceph-mon[31360]: pgmap v15106: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:08.018 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/951683483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:08.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:08.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:08.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:08.272 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:08 smithi122 ceph-mon[32255]: pgmap v15106: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:08.272 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/951683483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:08.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:08 smithi022 ceph-mon[40250]: pgmap v15106: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:08.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/951683483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:08.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:22:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:22:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:09.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:10 smithi022 ceph-mon[31360]: pgmap v15107: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:10 smithi022 ceph-mon[40250]: pgmap v15107: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:22:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:10 smithi122 ceph-mon[32255]: pgmap v15107: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:12 smithi022 ceph-mon[31360]: pgmap v15108: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:12 smithi022 ceph-mon[40250]: pgmap v15108: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:12 smithi122 ceph-mon[32255]: pgmap v15108: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:22:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:22:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:13.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:13.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:13.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:14 smithi022 ceph-mon[31360]: pgmap v15109: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/913844477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:14 smithi022 ceph-mon[40250]: pgmap v15109: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/913844477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:14 smithi122 ceph-mon[32255]: pgmap v15109: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/913844477' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:16 smithi122 ceph-mon[32255]: pgmap v15110: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:16 smithi022 ceph-mon[31360]: pgmap v15110: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:16 smithi022 ceph-mon[40250]: pgmap v15110: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:22:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:18 smithi022 ceph-mon[31360]: pgmap v15111: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:18 smithi022 ceph-mon[40250]: pgmap v15111: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:18.279 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:18 smithi122 ceph-mon[32255]: pgmap v15111: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:18.398 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:18.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:22:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:22:18.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:18.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:18.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4158647917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4158647917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4158647917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:20 smithi022 ceph-mon[31360]: pgmap v15112: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:20 smithi022 ceph-mon[40250]: pgmap v15112: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:20 smithi122 ceph-mon[32255]: pgmap v15112: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:20] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:22:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:22 smithi022 ceph-mon[31360]: pgmap v15113: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:22 smithi022 ceph-mon[40250]: pgmap v15113: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:22 smithi122 ceph-mon[32255]: pgmap v15113: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:22:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:22:23.768 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:24.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:24.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:24.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[31360]: pgmap v15114: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:22:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:22:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:22:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/210331088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[40250]: pgmap v15114: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:22:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:22:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:22:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/210331088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:24 smithi122 ceph-mon[32255]: pgmap v15114: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:22:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:22:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:22:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/210331088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:25.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:22:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:22:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:22:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:26 smithi122 ceph-mon[32255]: pgmap v15115: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:26 smithi022 ceph-mon[31360]: pgmap v15115: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:26 smithi022 ceph-mon[40250]: pgmap v15115: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:22:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:28 smithi022 ceph-mon[31360]: pgmap v15116: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:28 smithi022 ceph-mon[40250]: pgmap v15116: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:28.283 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:28 smithi122 ceph-mon[32255]: pgmap v15116: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:28.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:22:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:22:29.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:29.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:29.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:29.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:29.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:30 smithi022 ceph-mon[31360]: pgmap v15117: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/396201726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:30 smithi022 ceph-mon[40250]: pgmap v15117: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/396201726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:30] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:22:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:30 smithi122 ceph-mon[32255]: pgmap v15117: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/396201726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:32 smithi122 ceph-mon[32255]: pgmap v15118: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:22:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:22:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:32 smithi022 ceph-mon[31360]: pgmap v15118: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:32 smithi022 ceph-mon[40250]: pgmap v15118: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:34.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:34 smithi122 ceph-mon[32255]: pgmap v15119: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:22:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:22:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:22:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:22:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[31360]: pgmap v15119: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:22:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:22:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:22:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:22:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[40250]: pgmap v15119: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:22:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:22:34.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:34.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:34.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:35.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1306242054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1306242054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1306242054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:36 smithi122 ceph-mon[32255]: pgmap v15120: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:36 smithi022 ceph-mon[31360]: pgmap v15120: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:36 smithi022 ceph-mon[40250]: pgmap v15120: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:22:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:38 smithi022 ceph-mon[31360]: pgmap v15121: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:38 smithi022 ceph-mon[40250]: pgmap v15121: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:38.294 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:38 smithi122 ceph-mon[32255]: pgmap v15121: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:38.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:22:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:22:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:39.888 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:40.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:40.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:40.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:40.258 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:40 smithi022 ceph-mon[31360]: pgmap v15122: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:40.258 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:40 smithi022 ceph-mon[40250]: pgmap v15122: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:40] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:22:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:40 smithi122 ceph-mon[32255]: pgmap v15122: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:41.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3976531168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3976531168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3976531168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:42.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:42 smithi122 ceph-mon[32255]: pgmap v15123: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:42 smithi022 ceph-mon[31360]: pgmap v15123: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:42 smithi022 ceph-mon[40250]: pgmap v15123: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:22:42.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:22:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:44 smithi122 ceph-mon[32255]: pgmap v15124: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:44 smithi022 ceph-mon[31360]: pgmap v15124: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:44.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:44 smithi022 ceph-mon[40250]: pgmap v15124: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:45.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:45.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:45.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:45.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:46 smithi122 ceph-mon[32255]: pgmap v15125: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3341616948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:46 smithi022 ceph-mon[31360]: pgmap v15125: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3341616948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:46 smithi022 ceph-mon[40250]: pgmap v15125: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3341616948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:22:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:48 smithi022 ceph-mon[31360]: pgmap v15126: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:48 smithi022 ceph-mon[40250]: pgmap v15126: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:48.296 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:48 smithi122 ceph-mon[32255]: pgmap v15126: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:48.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:22:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:22:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:50 smithi122 ceph-mon[32255]: pgmap v15127: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:50] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:22:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:50 smithi022 ceph-mon[31360]: pgmap v15127: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:50 smithi022 ceph-mon[40250]: pgmap v15127: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:50.628 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:50.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:50.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:50.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3401768409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3401768409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3401768409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:52 smithi122 ceph-mon[32255]: pgmap v15128: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:52 smithi022 ceph-mon[31360]: pgmap v15128: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:52 smithi022 ceph-mon[40250]: pgmap v15128: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:22:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:22:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:54 smithi122 ceph-mon[32255]: pgmap v15129: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:54 smithi022 ceph-mon[31360]: pgmap v15129: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:54 smithi022 ceph-mon[40250]: pgmap v15129: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:22:55.999 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:22:56.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:22:56.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:22:56.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:22:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:56 smithi122 ceph-mon[32255]: pgmap v15130: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:56 smithi022 ceph-mon[31360]: pgmap v15130: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:56 smithi022 ceph-mon[40250]: pgmap v15130: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3121541218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3121541218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:22:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3121541218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:22:58.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:22:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:22:58.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:58 smithi122 ceph-mon[32255]: pgmap v15131: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:58 smithi022 ceph-mon[31360]: pgmap v15131: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:58 smithi022 ceph-mon[40250]: pgmap v15131: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:22:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:22:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:22:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:22:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:22:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:00] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:23:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:00 smithi122 ceph-mon[32255]: pgmap v15132: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:00 smithi022 ceph-mon[31360]: pgmap v15132: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:00.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:00 smithi022 ceph-mon[40250]: pgmap v15132: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:01.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:02 smithi122 ceph-mon[32255]: pgmap v15133: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1391581791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:23:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:23:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:02 smithi022 ceph-mon[31360]: pgmap v15133: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1391581791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:02 smithi022 ceph-mon[40250]: pgmap v15133: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1391581791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:04 smithi122 ceph-mon[32255]: pgmap v15134: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:04 smithi022 ceph-mon[31360]: pgmap v15134: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:04 smithi022 ceph-mon[40250]: pgmap v15134: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:06 smithi122 ceph-mon[32255]: pgmap v15135: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:06.745 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:06.745 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:06 smithi022 ceph-mon[31360]: pgmap v15135: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:06.746 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:06 smithi022 ceph-mon[40250]: pgmap v15135: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:07.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:07.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:07.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:23:08.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2726809501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:08.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:08 smithi122 ceph-mon[32255]: pgmap v15136: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2726809501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:08 smithi022 ceph-mon[31360]: pgmap v15136: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2726809501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:08 smithi022 ceph-mon[40250]: pgmap v15136: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:08.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:23:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:23:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:10 smithi122 ceph-mon[32255]: pgmap v15137: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:10] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:23:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:10 smithi022 ceph-mon[31360]: pgmap v15137: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:10 smithi022 ceph-mon[40250]: pgmap v15137: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:12.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:12.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:12.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:12.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:12.486 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:12 smithi022 ceph-mon[31360]: pgmap v15138: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:12.487 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:12 smithi022 ceph-mon[40250]: pgmap v15138: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:23:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:23:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:12 smithi122 ceph-mon[32255]: pgmap v15138: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:13.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1840337213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1840337213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1840337213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:14 smithi122 ceph-mon[32255]: pgmap v15139: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:14 smithi022 ceph-mon[31360]: pgmap v15139: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:14.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:14 smithi022 ceph-mon[40250]: pgmap v15139: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:16 smithi122 ceph-mon[32255]: pgmap v15140: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:16 smithi022 ceph-mon[31360]: pgmap v15140: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:16 smithi022 ceph-mon[40250]: pgmap v15140: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:17.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:23:17.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:17.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:17.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:18 smithi022 ceph-mon[31360]: pgmap v15141: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2220009505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:18 smithi022 ceph-mon[40250]: pgmap v15141: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2220009505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:18.326 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:18 smithi122 ceph-mon[32255]: pgmap v15141: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:18.326 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2220009505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:18.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:23:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:23:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:20 smithi122 ceph-mon[32255]: pgmap v15142: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:20] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:23:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:20 smithi022 ceph-mon[31360]: pgmap v15142: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:20 smithi022 ceph-mon[40250]: pgmap v15142: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:22 smithi122 ceph-mon[32255]: pgmap v15143: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:22.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:23:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:23:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:22 smithi022 ceph-mon[31360]: pgmap v15143: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:22 smithi022 ceph-mon[40250]: pgmap v15143: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:22.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:23.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:23.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:23.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3920032487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3920032487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:23.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3920032487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:24 smithi122 ceph-mon[32255]: pgmap v15144: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:24.771 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:24 smithi022 ceph-mon[40250]: pgmap v15144: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:24.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:24 smithi022 ceph-mon[31360]: pgmap v15144: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:23:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:23:25.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:23:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:23:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:23:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:23:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:23:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:23:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:23:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:23:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:26 smithi122 ceph-mon[32255]: pgmap v15145: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:26 smithi022 ceph-mon[31360]: pgmap v15145: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:26 smithi022 ceph-mon[40250]: pgmap v15145: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:23:28.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:28.336 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:28 smithi122 ceph-mon[32255]: pgmap v15146: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:28 smithi022 ceph-mon[31360]: pgmap v15146: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:28 smithi022 ceph-mon[40250]: pgmap v15146: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:28.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:28.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:28.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:28.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:23:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:23:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1110076891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1110076891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1110076891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:30 smithi122 ceph-mon[32255]: pgmap v15147: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:30] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:23:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:30 smithi022 ceph-mon[40250]: pgmap v15147: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:30 smithi022 ceph-mon[31360]: pgmap v15147: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:32 smithi122 ceph-mon[32255]: pgmap v15148: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:23:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:23:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:32 smithi022 ceph-mon[31360]: pgmap v15148: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:32 smithi022 ceph-mon[40250]: pgmap v15148: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:33.601 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:34 smithi122 ceph-mon[32255]: pgmap v15149: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3640435144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:23:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:23:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:23:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:23:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[31360]: pgmap v15149: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3640435144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:23:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[40250]: pgmap v15149: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3640435144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:23:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:23:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:23:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:36 smithi122 ceph-mon[32255]: pgmap v15150: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:36 smithi022 ceph-mon[31360]: pgmap v15150: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:36 smithi022 ceph-mon[40250]: pgmap v15150: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:23:38.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:38 smithi122 ceph-mon[32255]: pgmap v15151: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:38 smithi022 ceph-mon[31360]: pgmap v15151: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:38 smithi022 ceph-mon[40250]: pgmap v15151: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:38.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:23:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:23:38.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:39.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:39.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:39.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:39.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:40] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:23:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:40 smithi122 ceph-mon[32255]: pgmap v15152: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1961741531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:40 smithi022 ceph-mon[31360]: pgmap v15152: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1961741531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:40 smithi022 ceph-mon[40250]: pgmap v15152: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1961741531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:42 smithi122 ceph-mon[32255]: pgmap v15153: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:23:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:23:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:42 smithi022 ceph-mon[31360]: pgmap v15153: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:42 smithi022 ceph-mon[40250]: pgmap v15153: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:44.343 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:44 smithi122 ceph-mon[32255]: pgmap v15154: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:44.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:44.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:44.713 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:44.713 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:44 smithi022 ceph-mon[31360]: pgmap v15154: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:44.714 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:44 smithi022 ceph-mon[40250]: pgmap v15154: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:45.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2229505331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2229505331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2229505331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:46 smithi122 ceph-mon[32255]: pgmap v15155: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:46 smithi022 ceph-mon[31360]: pgmap v15155: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:46 smithi022 ceph-mon[40250]: pgmap v15155: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:23:48.356 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:48 smithi122 ceph-mon[32255]: pgmap v15156: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:48 smithi022 ceph-mon[31360]: pgmap v15156: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:48 smithi022 ceph-mon[40250]: pgmap v15156: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:48.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:23:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:23:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:49.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:50.082 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:50.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:50.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:50] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:23:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:50 smithi122 ceph-mon[32255]: pgmap v15157: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:50 smithi022 ceph-mon[31360]: pgmap v15157: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:50 smithi022 ceph-mon[40250]: pgmap v15157: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:51.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4290754199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4290754199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4290754199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:52.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:52 smithi122 ceph-mon[32255]: pgmap v15158: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:52 smithi022 ceph-mon[31360]: pgmap v15158: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:52 smithi022 ceph-mon[40250]: pgmap v15158: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:23:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:23:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:54 smithi122 ceph-mon[32255]: pgmap v15159: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:54 smithi022 ceph-mon[31360]: pgmap v15159: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:54 smithi022 ceph-mon[40250]: pgmap v15159: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:23:55.084 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:23:55.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:23:55.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:23:55.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:23:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:56 smithi122 ceph-mon[32255]: pgmap v15160: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3814357256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:56 smithi022 ceph-mon[31360]: pgmap v15160: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3814357256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:56 smithi022 ceph-mon[40250]: pgmap v15160: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3814357256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:23:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:23:58.364 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:58 smithi122 ceph-mon[32255]: pgmap v15161: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:58 smithi022 ceph-mon[31360]: pgmap v15161: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:58 smithi022 ceph-mon[40250]: pgmap v15161: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:23:58.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:23:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:23:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:23:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:23:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:23:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:23:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:00] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:24:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:00 smithi122 ceph-mon[32255]: pgmap v15162: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:00.455 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:00.456 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:00 smithi022 ceph-mon[31360]: pgmap v15162: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:00.456 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:00 smithi022 ceph-mon[40250]: pgmap v15162: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:00.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:00.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:00.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3200617863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3200617863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:01.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3200617863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:02.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:02 smithi122 ceph-mon[32255]: pgmap v15163: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:02 smithi022 ceph-mon[31360]: pgmap v15163: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:02 smithi022 ceph-mon[40250]: pgmap v15163: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:24:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:24:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:04 smithi122 ceph-mon[32255]: pgmap v15164: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:04 smithi022 ceph-mon[31360]: pgmap v15164: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:04 smithi022 ceph-mon[40250]: pgmap v15164: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:05.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:06.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:06.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:06.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:06 smithi022 ceph-mon[31360]: pgmap v15165: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1635755634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:06 smithi022 ceph-mon[40250]: pgmap v15165: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1635755634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:06 smithi122 ceph-mon[32255]: pgmap v15165: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1635755634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:24:08.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:08 smithi122 ceph-mon[32255]: pgmap v15166: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:08 smithi022 ceph-mon[31360]: pgmap v15166: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:08 smithi022 ceph-mon[40250]: pgmap v15166: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:08.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:24:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:24:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:10 smithi122 ceph-mon[32255]: pgmap v15167: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:10] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T02:24:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:10 smithi022 ceph-mon[31360]: pgmap v15167: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:10 smithi022 ceph-mon[40250]: pgmap v15167: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:11.199 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:11.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:11.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:11.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:24:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:24:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:12 smithi122 ceph-mon[32255]: pgmap v15168: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3534257064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:12 smithi022 ceph-mon[31360]: pgmap v15168: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3534257064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:12 smithi022 ceph-mon[40250]: pgmap v15168: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3534257064' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:14 smithi122 ceph-mon[32255]: pgmap v15169: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:14 smithi022 ceph-mon[31360]: pgmap v15169: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:14 smithi022 ceph-mon[40250]: pgmap v15169: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:16.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:16.603 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:16 smithi022 ceph-mon[31360]: pgmap v15170: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:16.604 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:16 smithi022 ceph-mon[40250]: pgmap v15170: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:16 smithi122 ceph-mon[32255]: pgmap v15170: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:16.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:16.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:16.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:17.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/491119714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/491119714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:24:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/491119714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:18 smithi122 ceph-mon[32255]: pgmap v15171: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:18.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:24:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:24:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:18 smithi022 ceph-mon[31360]: pgmap v15171: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:18 smithi022 ceph-mon[40250]: pgmap v15171: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:19.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:19.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:20.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:20 smithi122 ceph-mon[32255]: pgmap v15172: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:20] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:24:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:20 smithi022 ceph-mon[31360]: pgmap v15172: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:20.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:20 smithi022 ceph-mon[40250]: pgmap v15172: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:21.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:22.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:22.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:22.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:22 smithi122 ceph-mon[32255]: pgmap v15173: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3432478286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:24:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:24:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:22 smithi022 ceph-mon[31360]: pgmap v15173: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3432478286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:22 smithi022 ceph-mon[40250]: pgmap v15173: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3432478286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:24 smithi122 ceph-mon[32255]: pgmap v15174: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:24 smithi022 ceph-mon[40250]: pgmap v15174: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:24 smithi022 ceph-mon[31360]: pgmap v15174: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:24:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:24:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:24:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:24:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:24:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:24:25.670 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:24:25.670 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:24:25.670 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:24:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:26 smithi122 ceph-mon[32255]: pgmap v15175: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:24:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:26 smithi022 ceph-mon[31360]: pgmap v15175: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:24:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:26 smithi022 ceph-mon[40250]: pgmap v15175: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:24:27.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:27.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:27.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:27.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:27.686 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:24:28.389 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:28 smithi122 ceph-mon[32255]: pgmap v15176: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:28.389 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3399870776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:28 smithi022 ceph-mon[31360]: pgmap v15176: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3399870776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:28 smithi022 ceph-mon[40250]: pgmap v15176: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3399870776' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:28.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:24:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:24:29.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:30] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:24:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:30 smithi122 ceph-mon[32255]: pgmap v15177: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:30 smithi022 ceph-mon[31360]: pgmap v15177: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:30 smithi022 ceph-mon[40250]: pgmap v15177: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:32 smithi122 ceph-mon[32255]: pgmap v15178: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:24:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:24:32.687 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:32.687 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:32 smithi022 ceph-mon[31360]: pgmap v15178: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:32.688 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:32 smithi022 ceph-mon[40250]: pgmap v15178: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:33.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:33.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:33.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/82858858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/82858858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/82858858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:34 smithi122 ceph-mon[32255]: pgmap v15179: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:24:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:24:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:24:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:24:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[31360]: pgmap v15179: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:24:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:24:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:24:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:24:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[40250]: pgmap v15179: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:24:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:24:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:24:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:24:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:36 smithi122 ceph-mon[32255]: pgmap v15180: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:36 smithi022 ceph-mon[31360]: pgmap v15180: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:36 smithi022 ceph-mon[40250]: pgmap v15180: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:24:38.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:38.397 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:38 smithi122 ceph-mon[32255]: pgmap v15181: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:38.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:38.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:38.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:38.432 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:38 smithi022 ceph-mon[31360]: pgmap v15181: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:38.432 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:38 smithi022 ceph-mon[40250]: pgmap v15181: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:38.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:24:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:24:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1028225788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1028225788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1028225788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:40] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:24:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:40 smithi122 ceph-mon[32255]: pgmap v15182: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:40 smithi022 ceph-mon[31360]: pgmap v15182: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:40 smithi022 ceph-mon[40250]: pgmap v15182: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:42 smithi122 ceph-mon[32255]: pgmap v15183: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:24:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:24:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:42 smithi022 ceph-mon[31360]: pgmap v15183: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:42.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:42 smithi022 ceph-mon[40250]: pgmap v15183: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:43.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:43.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:43.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:43.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:44 smithi122 ceph-mon[32255]: pgmap v15184: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3122964874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:44 smithi022 ceph-mon[31360]: pgmap v15184: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3122964874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:44 smithi022 ceph-mon[40250]: pgmap v15184: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3122964874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:46 smithi122 ceph-mon[32255]: pgmap v15185: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:46 smithi022 ceph-mon[31360]: pgmap v15185: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:46 smithi022 ceph-mon[40250]: pgmap v15185: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:24:48.400 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:48 smithi122 ceph-mon[32255]: pgmap v15186: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:48 smithi022 ceph-mon[31360]: pgmap v15186: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:48 smithi022 ceph-mon[40250]: pgmap v15186: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:48.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:24:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:24:48.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:49.059 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:49.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:49.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:49.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1094795501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:50 smithi122 ceph-mon[32255]: pgmap v15187: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:50] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:24:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1094795501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:50 smithi022 ceph-mon[31360]: pgmap v15187: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1094795501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:50 smithi022 ceph-mon[40250]: pgmap v15187: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:52 smithi122 ceph-mon[32255]: pgmap v15188: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:24:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:24:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:52 smithi022 ceph-mon[31360]: pgmap v15188: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:52 smithi022 ceph-mon[40250]: pgmap v15188: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:54 smithi022 ceph-mon[31360]: pgmap v15189: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:54 smithi022 ceph-mon[40250]: pgmap v15189: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:54.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:54.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:54.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:24:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:54 smithi122 ceph-mon[32255]: pgmap v15189: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:24:55.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2568139311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2568139311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2568139311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:24:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:56 smithi122 ceph-mon[32255]: pgmap v15190: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:56 smithi022 ceph-mon[31360]: pgmap v15190: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:56 smithi022 ceph-mon[40250]: pgmap v15190: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:24:58.411 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:58 smithi122 ceph-mon[32255]: pgmap v15191: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:58 smithi022 ceph-mon[31360]: pgmap v15191: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:58 smithi022 ceph-mon[40250]: pgmap v15191: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:24:58.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:24:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:24:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:24:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:24:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:24:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:24:59.545 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:24:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:24:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:24:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:00 smithi022 ceph-mon[31360]: pgmap v15192: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2735043363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:00 smithi022 ceph-mon[40250]: pgmap v15192: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2735043363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:00 smithi122 ceph-mon[32255]: pgmap v15192: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2735043363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:00] "GET /metrics HTTP/1.1" 200 202441 "" "Prometheus/2.18.1" 2021-12-16T02:25:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:02 smithi122 ceph-mon[32255]: pgmap v15193: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:25:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:25:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:02 smithi022 ceph-mon[31360]: pgmap v15193: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:02 smithi022 ceph-mon[40250]: pgmap v15193: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:04 smithi122 ceph-mon[32255]: pgmap v15194: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:04 smithi022 ceph-mon[31360]: pgmap v15194: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:04 smithi022 ceph-mon[40250]: pgmap v15194: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:05.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:05.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:05.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/378970512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/378970512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/378970512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:06 smithi122 ceph-mon[32255]: pgmap v15195: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:06 smithi022 ceph-mon[31360]: pgmap v15195: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:06 smithi022 ceph-mon[40250]: pgmap v15195: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:25:08.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:08 smithi122 ceph-mon[32255]: pgmap v15196: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:08 smithi022 ceph-mon[31360]: pgmap v15196: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:08 smithi022 ceph-mon[40250]: pgmap v15196: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:08.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:25:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:25:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:10.282 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:10.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:10] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:25:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:10 smithi122 ceph-mon[32255]: pgmap v15197: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:10 smithi022 ceph-mon[31360]: pgmap v15197: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:10 smithi022 ceph-mon[40250]: pgmap v15197: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:10.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:10.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:10.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:11.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/873558466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/873558466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/873558466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:12.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:12 smithi122 ceph-mon[32255]: pgmap v15198: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:25:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:25:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:12 smithi022 ceph-mon[31360]: pgmap v15198: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:12 smithi022 ceph-mon[40250]: pgmap v15198: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:14 smithi122 ceph-mon[32255]: pgmap v15199: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:14 smithi022 ceph-mon[31360]: pgmap v15199: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:14 smithi022 ceph-mon[40250]: pgmap v15199: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:16.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:16.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:16.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:16 smithi122 ceph-mon[32255]: pgmap v15200: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1362842894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:16 smithi022 ceph-mon[31360]: pgmap v15200: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1362842894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:16 smithi022 ceph-mon[40250]: pgmap v15200: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1362842894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:25:18.431 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:18 smithi122 ceph-mon[32255]: pgmap v15201: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:18 smithi022 ceph-mon[31360]: pgmap v15201: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:18 smithi022 ceph-mon[40250]: pgmap v15201: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:18.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:25:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:25:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:25:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:20 smithi122 ceph-mon[32255]: pgmap v15202: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:20 smithi022 ceph-mon[31360]: pgmap v15202: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:20 smithi022 ceph-mon[40250]: pgmap v15202: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:21.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:21.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:21.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:21.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:22 smithi122 ceph-mon[32255]: pgmap v15203: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:22.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1715711877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:25:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:25:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:22 smithi022 ceph-mon[31360]: pgmap v15203: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1715711877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:22 smithi022 ceph-mon[40250]: pgmap v15203: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1715711877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:24 smithi122 ceph-mon[32255]: pgmap v15204: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:24 smithi022 ceph-mon[31360]: pgmap v15204: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:24 smithi022 ceph-mon[40250]: pgmap v15204: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:26.396 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:26.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[31360]: pgmap v15205: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:26.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:25:26.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:25:26.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:25:26.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:25:26.398 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[40250]: pgmap v15205: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:26.398 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:25:26.398 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:25:26.398 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:25:26.398 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:25:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:26 smithi122 ceph-mon[32255]: pgmap v15205: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:25:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:25:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:25:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:25:26.756 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:26.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:26.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3994412944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3994412944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:25:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3994412944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:28 smithi122 ceph-mon[32255]: pgmap v15206: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:28.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:25:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:25:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:28 smithi022 ceph-mon[31360]: pgmap v15206: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:28 smithi022 ceph-mon[40250]: pgmap v15206: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:29.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:30.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:30 smithi122 ceph-mon[32255]: pgmap v15207: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:25:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:30 smithi022 ceph-mon[31360]: pgmap v15207: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:30 smithi022 ceph-mon[40250]: pgmap v15207: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:31.758 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:32.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:32.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:32.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:32 smithi022 ceph-mon[31360]: pgmap v15208: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4229843751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:32 smithi022 ceph-mon[40250]: pgmap v15208: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4229843751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:32.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:25:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:25:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:32 smithi122 ceph-mon[32255]: pgmap v15208: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4229843751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:34 smithi122 ceph-mon[32255]: pgmap v15209: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:25:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:25:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:25:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:25:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[31360]: pgmap v15209: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:25:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:25:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:25:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:25:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[40250]: pgmap v15209: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:25:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:25:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:25:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:25:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:36 smithi122 ceph-mon[32255]: pgmap v15210: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:36 smithi022 ceph-mon[31360]: pgmap v15210: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:36 smithi022 ceph-mon[40250]: pgmap v15210: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:37.132 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:37.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:25:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:38 smithi122 ceph-mon[32255]: pgmap v15211: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2726658068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:38 smithi022 ceph-mon[31360]: pgmap v15211: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2726658068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:38 smithi022 ceph-mon[40250]: pgmap v15211: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2726658068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:38.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:25:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:25:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:40 smithi122 ceph-mon[32255]: pgmap v15212: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:40] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:25:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:40 smithi022 ceph-mon[31360]: pgmap v15212: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:40 smithi022 ceph-mon[40250]: pgmap v15212: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:42.502 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:42 smithi122 ceph-mon[32255]: pgmap v15213: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:25:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:25:42.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:42 smithi022 ceph-mon[40250]: pgmap v15213: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:42 smithi022 ceph-mon[31360]: pgmap v15213: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:42.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:42.873 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:42.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3619777746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3619777746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3619777746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:44 smithi122 ceph-mon[32255]: pgmap v15214: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:44 smithi022 ceph-mon[31360]: pgmap v15214: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:44.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:44 smithi022 ceph-mon[40250]: pgmap v15214: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:46 smithi122 ceph-mon[32255]: pgmap v15215: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:46 smithi022 ceph-mon[31360]: pgmap v15215: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:46 smithi022 ceph-mon[40250]: pgmap v15215: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:25:47.875 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:48.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:48.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:48.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:48.249 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:48 smithi022 ceph-mon[31360]: pgmap v15216: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:48.249 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:48 smithi022 ceph-mon[40250]: pgmap v15216: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:48 smithi122 ceph-mon[32255]: pgmap v15216: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:48.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:25:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:25:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4175919774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4175919774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4175919774' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:50] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:25:50.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:50 smithi122 ceph-mon[32255]: pgmap v15217: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:50 smithi022 ceph-mon[31360]: pgmap v15217: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:50 smithi022 ceph-mon[40250]: pgmap v15217: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:52 smithi122 ceph-mon[32255]: pgmap v15218: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:52.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:25:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:25:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:52 smithi022 ceph-mon[31360]: pgmap v15218: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:52 smithi022 ceph-mon[40250]: pgmap v15218: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:53.250 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:53.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:53.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:53.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:54 smithi122 ceph-mon[32255]: pgmap v15219: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/864787229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:54 smithi022 ceph-mon[31360]: pgmap v15219: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/864787229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:54 smithi022 ceph-mon[40250]: pgmap v15219: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:25:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/864787229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:56 smithi122 ceph-mon[32255]: pgmap v15220: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:56 smithi022 ceph-mon[31360]: pgmap v15220: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:56 smithi022 ceph-mon[40250]: pgmap v15220: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:25:58.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:58 smithi122 ceph-mon[32255]: pgmap v15221: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:58 smithi022 ceph-mon[31360]: pgmap v15221: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:58 smithi022 ceph-mon[40250]: pgmap v15221: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:25:58.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:25:58.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:25:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:25:58.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:25:58.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:25:58.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:25:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:25:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3612080354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:25:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3612080354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:25:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:25:59.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:25:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3612080354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:00 smithi122 ceph-mon[32255]: pgmap v15222: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:00] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:26:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:00 smithi022 ceph-mon[31360]: pgmap v15222: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:00 smithi022 ceph-mon[40250]: pgmap v15222: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:02.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:02 smithi122 ceph-mon[32255]: pgmap v15223: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:26:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:26:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:02 smithi022 ceph-mon[31360]: pgmap v15223: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:02 smithi022 ceph-mon[40250]: pgmap v15223: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:03.993 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:04.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:04.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:04.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:04 smithi122 ceph-mon[32255]: pgmap v15224: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:04 smithi022 ceph-mon[31360]: pgmap v15224: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:04 smithi022 ceph-mon[40250]: pgmap v15224: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:05.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1977967570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1977967570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1977967570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:06.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:06 smithi122 ceph-mon[32255]: pgmap v15225: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:06 smithi022 ceph-mon[31360]: pgmap v15225: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:06 smithi022 ceph-mon[40250]: pgmap v15225: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:26:08.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:08 smithi122 ceph-mon[32255]: pgmap v15226: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:08 smithi022 ceph-mon[31360]: pgmap v15226: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:08 smithi022 ceph-mon[40250]: pgmap v15226: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:08.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:26:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:26:09.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:09.366 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:09.366 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:09.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:09.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:09.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:09.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:10.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:10 smithi122 ceph-mon[32255]: pgmap v15227: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1815754943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:26:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:10 smithi022 ceph-mon[31360]: pgmap v15227: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1815754943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:10 smithi022 ceph-mon[40250]: pgmap v15227: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1815754943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:12.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:12 smithi122 ceph-mon[32255]: pgmap v15228: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:26:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:26:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:12 smithi022 ceph-mon[31360]: pgmap v15228: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:12.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:12 smithi022 ceph-mon[40250]: pgmap v15228: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:14 smithi122 ceph-mon[32255]: pgmap v15229: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:14.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:14.736 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:14 smithi022 ceph-mon[31360]: pgmap v15229: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:14.737 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:14 smithi022 ceph-mon[40250]: pgmap v15229: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:15.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:15.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:15.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4047631652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:15.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4047631652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:15.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4047631652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:16 smithi122 ceph-mon[32255]: pgmap v15230: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:16 smithi022 ceph-mon[31360]: pgmap v15230: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:16 smithi022 ceph-mon[40250]: pgmap v15230: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:26:18.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:18 smithi122 ceph-mon[32255]: pgmap v15231: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:18 smithi022 ceph-mon[31360]: pgmap v15231: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:18 smithi022 ceph-mon[40250]: pgmap v15231: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:18.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:26:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:26:19.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:19.668 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:20.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:20.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:20] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:26:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:20 smithi122 ceph-mon[32255]: pgmap v15232: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:20.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:20.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:20.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:20.476 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:20 smithi022 ceph-mon[40250]: pgmap v15232: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:20.476 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:20 smithi022 ceph-mon[31360]: pgmap v15232: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:21.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4028577332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4028577332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4028577332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:22.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:22 smithi122 ceph-mon[32255]: pgmap v15233: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:26:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:26:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:22 smithi022 ceph-mon[31360]: pgmap v15233: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:22 smithi022 ceph-mon[40250]: pgmap v15233: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:24.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:24 smithi122 ceph-mon[32255]: pgmap v15234: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:24 smithi022 ceph-mon[31360]: pgmap v15234: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:24 smithi022 ceph-mon[40250]: pgmap v15234: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:25.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:26 smithi122 ceph-mon[32255]: pgmap v15235: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3956800373' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:26:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:26:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:26:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[40250]: pgmap v15235: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3956800373' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:26:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:26:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:26:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[31360]: pgmap v15235: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3956800373' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:26:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:26:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:26:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:26:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:26:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:26:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:26:28.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:28 smithi122 ceph-mon[32255]: pgmap v15236: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:28.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:26:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:26:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:28 smithi022 ceph-mon[31360]: pgmap v15236: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:28 smithi022 ceph-mon[40250]: pgmap v15236: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:30] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:26:30.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:30.846 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:30 smithi022 ceph-mon[31360]: pgmap v15237: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:30.846 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:30 smithi022 ceph-mon[40250]: pgmap v15237: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:30.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:30 smithi122 ceph-mon[32255]: pgmap v15237: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:31.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/754411960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/754411960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/754411960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:32.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:26:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:26:32.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:32 smithi122 ceph-mon[32255]: pgmap v15238: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:32 smithi022 ceph-mon[31360]: pgmap v15238: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:32 smithi022 ceph-mon[40250]: pgmap v15238: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:34.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:34 smithi122 ceph-mon[32255]: pgmap v15239: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:26:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:26:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[31360]: pgmap v15239: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:26:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:26:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:26:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:26:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[40250]: pgmap v15239: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:26:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:26:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:26:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:26:36.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:36.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:36.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:36.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:36.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:36 smithi122 ceph-mon[32255]: pgmap v15240: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1671244525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:36 smithi022 ceph-mon[40250]: pgmap v15240: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1671244525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:36 smithi022 ceph-mon[31360]: pgmap v15240: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1671244525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:26:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:38 smithi122 ceph-mon[32255]: pgmap v15241: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:38 smithi022 ceph-mon[31360]: pgmap v15241: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:38 smithi022 ceph-mon[40250]: pgmap v15241: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:38.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:26:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:26:39.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:40 smithi122 ceph-mon[32255]: pgmap v15242: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:40] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:26:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:40 smithi022 ceph-mon[31360]: pgmap v15242: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:40 smithi022 ceph-mon[40250]: pgmap v15242: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:41.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:41.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:41.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:41.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:42.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:42 smithi122 ceph-mon[32255]: pgmap v15243: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2734052551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:26:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:26:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:42 smithi022 ceph-mon[31360]: pgmap v15243: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2734052551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:42 smithi022 ceph-mon[40250]: pgmap v15243: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2734052551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:44.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:44 smithi122 ceph-mon[32255]: pgmap v15244: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:44 smithi022 ceph-mon[31360]: pgmap v15244: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:44 smithi022 ceph-mon[40250]: pgmap v15244: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:46.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:46 smithi122 ceph-mon[32255]: pgmap v15245: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:46 smithi022 ceph-mon[31360]: pgmap v15245: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:46 smithi022 ceph-mon[40250]: pgmap v15245: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:46.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:47.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:47.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:47.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:26:48.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:48 smithi122 ceph-mon[32255]: pgmap v15246: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3408636896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:48 smithi022 ceph-mon[31360]: pgmap v15246: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3408636896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:48 smithi022 ceph-mon[40250]: pgmap v15246: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3408636896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:48.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:26:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:26:49.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:50.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:50] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:26:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:50 smithi122 ceph-mon[32255]: pgmap v15247: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:50 smithi022 ceph-mon[31360]: pgmap v15247: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:50 smithi022 ceph-mon[40250]: pgmap v15247: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:52.340 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:52.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:52 smithi122 ceph-mon[32255]: pgmap v15248: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:26:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:26:52.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:52.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:52.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:52.705 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:52 smithi022 ceph-mon[31360]: pgmap v15248: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:52.705 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:52 smithi022 ceph-mon[40250]: pgmap v15248: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/205351795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/205351795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/205351795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:54 smithi122 ceph-mon[32255]: pgmap v15249: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:54 smithi022 ceph-mon[31360]: pgmap v15249: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:54 smithi022 ceph-mon[40250]: pgmap v15249: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:26:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:56 smithi122 ceph-mon[32255]: pgmap v15250: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:56 smithi022 ceph-mon[31360]: pgmap v15250: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:56 smithi022 ceph-mon[40250]: pgmap v15250: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:57.706 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:26:57.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:26:58.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:26:58.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:26:58.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:26:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:58 smithi122 ceph-mon[32255]: pgmap v15251: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3125541018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:58 smithi022 ceph-mon[31360]: pgmap v15251: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3125541018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:58 smithi022 ceph-mon[40250]: pgmap v15251: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:26:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3125541018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:26:58.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:26:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:26:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:26:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:26:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:26:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:26:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:00] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:27:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:00 smithi122 ceph-mon[32255]: pgmap v15252: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:00 smithi022 ceph-mon[31360]: pgmap v15252: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:00 smithi022 ceph-mon[40250]: pgmap v15252: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:02 smithi122 ceph-mon[32255]: pgmap v15253: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:27:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:27:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:02 smithi022 ceph-mon[31360]: pgmap v15253: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:02 smithi022 ceph-mon[40250]: pgmap v15253: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:03.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:04 smithi122 ceph-mon[32255]: pgmap v15254: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2164147144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:04 smithi022 ceph-mon[31360]: pgmap v15254: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2164147144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:04 smithi022 ceph-mon[40250]: pgmap v15254: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2164147144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:06 smithi122 ceph-mon[32255]: pgmap v15255: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:06 smithi022 ceph-mon[31360]: pgmap v15255: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:06 smithi022 ceph-mon[40250]: pgmap v15255: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:27:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:08 smithi122 ceph-mon[32255]: pgmap v15256: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:08.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:08.450 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:08 smithi022 ceph-mon[40250]: pgmap v15256: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:08.450 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:08 smithi022 ceph-mon[31360]: pgmap v15256: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:08.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:27:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:27:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1863448853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1863448853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1863448853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:10] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:27:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:10 smithi122 ceph-mon[32255]: pgmap v15257: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:10 smithi022 ceph-mon[31360]: pgmap v15257: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:10 smithi022 ceph-mon[40250]: pgmap v15257: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:12 smithi122 ceph-mon[32255]: pgmap v15258: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:27:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:27:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:12 smithi022 ceph-mon[31360]: pgmap v15258: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:12 smithi022 ceph-mon[40250]: pgmap v15258: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:13.825 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:14.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:14.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:14.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:14 smithi022 ceph-mon[31360]: pgmap v15259: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/802441060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:14 smithi022 ceph-mon[40250]: pgmap v15259: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/802441060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:14 smithi122 ceph-mon[32255]: pgmap v15259: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/802441060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:16 smithi122 ceph-mon[32255]: pgmap v15260: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:16 smithi022 ceph-mon[31360]: pgmap v15260: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:16 smithi022 ceph-mon[40250]: pgmap v15260: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:27:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:18 smithi122 ceph-mon[32255]: pgmap v15261: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:18 smithi022 ceph-mon[31360]: pgmap v15261: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:18 smithi022 ceph-mon[40250]: pgmap v15261: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:18.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:27:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:27:19.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:19.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:19.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:19.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:20 smithi122 ceph-mon[32255]: pgmap v15262: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1420396213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:20] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:27:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:20 smithi022 ceph-mon[31360]: pgmap v15262: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1420396213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:20 smithi022 ceph-mon[40250]: pgmap v15262: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1420396213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:22 smithi122 ceph-mon[32255]: pgmap v15263: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:27:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:27:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:22 smithi022 ceph-mon[31360]: pgmap v15263: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:22 smithi022 ceph-mon[40250]: pgmap v15263: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:24.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:24 smithi122 ceph-mon[32255]: pgmap v15264: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:24 smithi022 ceph-mon[31360]: pgmap v15264: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:24 smithi022 ceph-mon[40250]: pgmap v15264: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:24.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:24.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:24.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2736864054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2736864054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2736864054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:26.646 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:26 smithi122 ceph-mon[32255]: pgmap v15265: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:26.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:26 smithi022 ceph-mon[31360]: pgmap v15265: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:26.649 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:26 smithi022 ceph-mon[40250]: pgmap v15265: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:27:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:27:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:27:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:27:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:27:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:27:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:27:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:27:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:27:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:27:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:28 smithi122 ceph-mon[32255]: pgmap v15266: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:28.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:27:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:27:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:28 smithi022 ceph-mon[31360]: pgmap v15266: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:28 smithi022 ceph-mon[40250]: pgmap v15266: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:29.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:29.935 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:30.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:30.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:30.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:30] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:27:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:30 smithi122 ceph-mon[32255]: pgmap v15267: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/147790540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:30 smithi022 ceph-mon[31360]: pgmap v15267: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/147790540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:30 smithi022 ceph-mon[40250]: pgmap v15267: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/147790540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2949M 2021-12-16T02:27:31.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:31.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2949M 2021-12-16T02:27:31.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:31.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2949M 2021-12-16T02:27:31.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:31.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:27:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:27:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: pgmap v15268: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3208M 2021-12-16T02:27:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: pgmap v15268: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3208M 2021-12-16T02:27:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:32.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: pgmap v15268: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:27:32.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3208M 2021-12-16T02:27:32.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:32.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:27:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:34 smithi122 ceph-mon[32255]: pgmap v15269: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:27:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:27:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:27:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:27:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[31360]: pgmap v15269: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:27:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:27:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:27:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[40250]: pgmap v15269: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:27:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:27:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:27:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:35.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:35.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:35.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:36 smithi122 ceph-mon[32255]: pgmap v15270: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2490838354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:36 smithi022 ceph-mon[31360]: pgmap v15270: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2490838354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:36 smithi022 ceph-mon[40250]: pgmap v15270: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2490838354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:27:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:38 smithi122 ceph-mon[32255]: pgmap v15271: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:38 smithi022 ceph-mon[31360]: pgmap v15271: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:38 smithi022 ceph-mon[40250]: pgmap v15271: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:38.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:27:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:27:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:40] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:27:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:40 smithi122 ceph-mon[32255]: pgmap v15272: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:40 smithi022 ceph-mon[31360]: pgmap v15272: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:40 smithi022 ceph-mon[40250]: pgmap v15272: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:40.673 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:41.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:41.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:41.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2572707526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2572707526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2572707526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:42 smithi122 ceph-mon[32255]: pgmap v15273: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:27:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:27:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:42 smithi022 ceph-mon[31360]: pgmap v15273: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:42 smithi022 ceph-mon[40250]: pgmap v15273: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:44 smithi122 ceph-mon[32255]: pgmap v15274: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:44 smithi022 ceph-mon[31360]: pgmap v15274: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:44 smithi022 ceph-mon[40250]: pgmap v15274: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:46.041 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:46.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:46.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:46.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:46 smithi122 ceph-mon[32255]: pgmap v15275: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:46 smithi022 ceph-mon[31360]: pgmap v15275: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:46 smithi022 ceph-mon[40250]: pgmap v15275: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2363118582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2363118582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:27:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2363118582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:48 smithi122 ceph-mon[32255]: pgmap v15276: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:48.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:27:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:27:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:48 smithi022 ceph-mon[31360]: pgmap v15276: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:48 smithi022 ceph-mon[40250]: pgmap v15276: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:50] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:27:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:50 smithi122 ceph-mon[32255]: pgmap v15277: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:50 smithi022 ceph-mon[31360]: pgmap v15277: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:50 smithi022 ceph-mon[40250]: pgmap v15277: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:51.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:51.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:51.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:51.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:52 smithi122 ceph-mon[32255]: pgmap v15278: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3779500714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:27:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:27:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:52 smithi022 ceph-mon[31360]: pgmap v15278: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3779500714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:52 smithi022 ceph-mon[40250]: pgmap v15278: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3779500714' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:54 smithi122 ceph-mon[32255]: pgmap v15279: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:54 smithi022 ceph-mon[31360]: pgmap v15279: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:54 smithi022 ceph-mon[40250]: pgmap v15279: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:27:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:56 smithi122 ceph-mon[32255]: pgmap v15280: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:56 smithi022 ceph-mon[31360]: pgmap v15280: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:56 smithi022 ceph-mon[40250]: pgmap v15280: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:56.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:27:57.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:27:57.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:27:57.152 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:27:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:27:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2047580080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:58 smithi122 ceph-mon[32255]: pgmap v15281: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2047580080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:58 smithi022 ceph-mon[31360]: pgmap v15281: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2047580080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:27:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:58 smithi022 ceph-mon[40250]: pgmap v15281: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:27:58.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:27:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:27:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:27:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:27:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:27:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:27:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:00] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:28:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:00 smithi122 ceph-mon[32255]: pgmap v15282: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:00 smithi022 ceph-mon[31360]: pgmap v15282: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:00 smithi022 ceph-mon[40250]: pgmap v15282: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:02.153 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:02.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:02.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:02.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:02.527 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:02 smithi022 ceph-mon[31360]: pgmap v15283: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:02 smithi022 ceph-mon[40250]: pgmap v15283: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:02 smithi122 ceph-mon[32255]: pgmap v15283: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:28:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:28:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1876464054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1876464054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:03.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1876464054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:04 smithi122 ceph-mon[32255]: pgmap v15284: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:04 smithi022 ceph-mon[31360]: pgmap v15284: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:04 smithi022 ceph-mon[40250]: pgmap v15284: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:06 smithi122 ceph-mon[32255]: pgmap v15285: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:06 smithi022 ceph-mon[31360]: pgmap v15285: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:06 smithi022 ceph-mon[40250]: pgmap v15285: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:07.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:28:07.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:07.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:07.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:08 smithi022 ceph-mon[31360]: pgmap v15286: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/561247749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:08 smithi022 ceph-mon[40250]: pgmap v15286: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/561247749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:08 smithi122 ceph-mon[32255]: pgmap v15286: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/561247749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:08.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:28:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:28:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:10] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:28:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:10 smithi122 ceph-mon[32255]: pgmap v15287: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:10 smithi022 ceph-mon[31360]: pgmap v15287: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:10 smithi022 ceph-mon[40250]: pgmap v15287: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:28:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:12 smithi122 ceph-mon[32255]: pgmap v15288: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:28:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:28:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:12 smithi022 ceph-mon[31360]: pgmap v15288: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:12 smithi022 ceph-mon[40250]: pgmap v15288: 132 pgs: 132 active+clean; 582 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:12.903 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:13.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:13.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:13.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/901709487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/901709487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/901709487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:14 smithi122 ceph-mon[32255]: pgmap v15289: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:14 smithi022 ceph-mon[31360]: pgmap v15289: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:14 smithi022 ceph-mon[40250]: pgmap v15289: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:16 smithi122 ceph-mon[32255]: pgmap v15290: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:16 smithi022 ceph-mon[40250]: pgmap v15290: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:16 smithi022 ceph-mon[31360]: pgmap v15290: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:28:18.274 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:18 smithi122 ceph-mon[32255]: pgmap v15291: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:18 smithi022 ceph-mon[40250]: pgmap v15291: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:18 smithi022 ceph-mon[31360]: pgmap v15291: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:18.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:18.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:18.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:18.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:28:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:28:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/721024093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/721024093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/721024093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:20 smithi122 ceph-mon[32255]: pgmap v15292: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:20] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:28:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:20 smithi022 ceph-mon[31360]: pgmap v15292: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:20 smithi022 ceph-mon[40250]: pgmap v15292: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:22 smithi122 ceph-mon[32255]: pgmap v15293: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:28:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:28:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:22 smithi022 ceph-mon[31360]: pgmap v15293: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:22 smithi022 ceph-mon[40250]: pgmap v15293: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:28:23.641 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:24.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:24.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:24.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:24 smithi122 ceph-mon[32255]: pgmap v15294: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1288915719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:24 smithi022 ceph-mon[31360]: pgmap v15294: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1288915719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:24 smithi022 ceph-mon[40250]: pgmap v15294: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:28:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1288915719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:26 smithi122 ceph-mon[32255]: pgmap v15295: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:26 smithi022 ceph-mon[31360]: pgmap v15295: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:26 smithi022 ceph-mon[40250]: pgmap v15295: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:28:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:28 smithi122 ceph-mon[32255]: pgmap v15296: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:28 smithi022 ceph-mon[31360]: pgmap v15296: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:28 smithi022 ceph-mon[40250]: pgmap v15296: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:28.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:28:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:28:29.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:29.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:29.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:29.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:30] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:28:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:30 smithi122 ceph-mon[32255]: pgmap v15297: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/312766603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:30 smithi022 ceph-mon[31360]: pgmap v15297: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/312766603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:30 smithi022 ceph-mon[40250]: pgmap v15297: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/312766603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:28:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:32 smithi122 ceph-mon[32255]: pgmap v15298: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:28:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:28:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:28:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[31360]: pgmap v15298: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:28:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:28:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:28:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:28:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[40250]: pgmap v15298: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:28:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:28:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:28:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:28:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:34 smithi122 ceph-mon[32255]: pgmap v15299: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:28:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:28:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:28:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:28:34.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:34.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:34.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:34.751 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[31360]: pgmap v15299: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:34.751 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:28:34.751 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:28:34.751 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:28:34.752 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:28:34.752 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[40250]: pgmap v15299: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:34.752 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:28:34.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:28:34.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:28:34.753 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:28:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3753818387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3753818387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3753818387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:36 smithi122 ceph-mon[32255]: pgmap v15300: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:36 smithi022 ceph-mon[31360]: pgmap v15300: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:36 smithi022 ceph-mon[40250]: pgmap v15300: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:28:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:38 smithi122 ceph-mon[32255]: pgmap v15301: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:38 smithi022 ceph-mon[31360]: pgmap v15301: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:38 smithi022 ceph-mon[40250]: pgmap v15301: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:38.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:28:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:28:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:39.751 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:40.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:40.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:40.124 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:40] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:28:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:40 smithi122 ceph-mon[32255]: pgmap v15302: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:40 smithi022 ceph-mon[31360]: pgmap v15302: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:40 smithi022 ceph-mon[40250]: pgmap v15302: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1969082682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1969082682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1969082682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:42 smithi122 ceph-mon[32255]: pgmap v15303: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:28:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:28:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:42 smithi022 ceph-mon[31360]: pgmap v15303: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:42 smithi022 ceph-mon[40250]: pgmap v15303: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:28:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:44 smithi122 ceph-mon[32255]: pgmap v15304: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:44 smithi022 ceph-mon[31360]: pgmap v15304: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:44 smithi022 ceph-mon[40250]: pgmap v15304: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:45.126 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:45.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:45.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:45.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:46 smithi122 ceph-mon[32255]: pgmap v15305: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/8770670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:46 smithi022 ceph-mon[31360]: pgmap v15305: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/8770670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:46 smithi022 ceph-mon[40250]: pgmap v15305: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/8770670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:28:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:48 smithi122 ceph-mon[32255]: pgmap v15306: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:48 smithi022 ceph-mon[31360]: pgmap v15306: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:48 smithi022 ceph-mon[40250]: pgmap v15306: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:48.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:28:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:28:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:50 smithi122 ceph-mon[32255]: pgmap v15307: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:50] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:28:50.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:50.502 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:50 smithi022 ceph-mon[31360]: pgmap v15307: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:50.503 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:50 smithi022 ceph-mon[40250]: pgmap v15307: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2877409091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:51.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2877409091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2877409091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:52 smithi122 ceph-mon[32255]: pgmap v15308: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:28:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:28:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:52 smithi022 ceph-mon[31360]: pgmap v15308: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:52 smithi022 ceph-mon[40250]: pgmap v15308: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:54 smithi122 ceph-mon[32255]: pgmap v15309: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:54 smithi022 ceph-mon[31360]: pgmap v15309: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:54 smithi022 ceph-mon[40250]: pgmap v15309: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:28:55.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:28:56.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:28:56.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:28:56.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:28:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:56 smithi022 ceph-mon[31360]: pgmap v15310: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3357893950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:56 smithi022 ceph-mon[40250]: pgmap v15310: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3357893950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:56 smithi122 ceph-mon[32255]: pgmap v15310: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3357893950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:28:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:28:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:58 smithi122 ceph-mon[32255]: pgmap v15311: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:58 smithi022 ceph-mon[31360]: pgmap v15311: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:58 smithi022 ceph-mon[40250]: pgmap v15311: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:28:58.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:28:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:28:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:28:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:28:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:28:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:28:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:00 smithi122 ceph-mon[32255]: pgmap v15312: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:00] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:29:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:00 smithi022 ceph-mon[31360]: pgmap v15312: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:00 smithi022 ceph-mon[40250]: pgmap v15312: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:01.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:01.607 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:01.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:01.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:02 smithi122 ceph-mon[32255]: pgmap v15313: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3096646415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:29:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:29:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:02 smithi022 ceph-mon[31360]: pgmap v15313: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3096646415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:02 smithi022 ceph-mon[40250]: pgmap v15313: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3096646415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:04 smithi122 ceph-mon[32255]: pgmap v15314: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:04 smithi022 ceph-mon[31360]: pgmap v15314: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:04 smithi022 ceph-mon[40250]: pgmap v15314: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:06.609 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:06 smithi122 ceph-mon[32255]: pgmap v15315: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:06 smithi022 ceph-mon[31360]: pgmap v15315: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:06 smithi022 ceph-mon[40250]: pgmap v15315: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:06.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:06.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:06.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/217522580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/217522580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:29:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/217522580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:08.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:29:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:29:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:08 smithi122 ceph-mon[32255]: pgmap v15316: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:08 smithi022 ceph-mon[31360]: pgmap v15316: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:08 smithi022 ceph-mon[40250]: pgmap v15316: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:10 smithi122 ceph-mon[32255]: pgmap v15317: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:10] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:29:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:10 smithi022 ceph-mon[31360]: pgmap v15317: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:10 smithi022 ceph-mon[40250]: pgmap v15317: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:11.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:12.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:12.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:12.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:29:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:29:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:12 smithi122 ceph-mon[32255]: pgmap v15318: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3722833251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:12 smithi022 ceph-mon[31360]: pgmap v15318: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3722833251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:12 smithi022 ceph-mon[40250]: pgmap v15318: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3722833251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:14 smithi122 ceph-mon[32255]: pgmap v15319: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:14 smithi022 ceph-mon[31360]: pgmap v15319: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:14 smithi022 ceph-mon[40250]: pgmap v15319: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:16 smithi122 ceph-mon[32255]: pgmap v15320: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:16 smithi022 ceph-mon[31360]: pgmap v15320: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:16 smithi022 ceph-mon[40250]: pgmap v15320: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:17.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:17.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:17.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:17.751 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:17.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:29:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:18 smithi122 ceph-mon[32255]: pgmap v15321: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2233340179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:18 smithi022 ceph-mon[31360]: pgmap v15321: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2233340179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:18 smithi022 ceph-mon[40250]: pgmap v15321: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2233340179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:18.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:29:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:29:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:20] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:29:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:20 smithi122 ceph-mon[32255]: pgmap v15322: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:20 smithi022 ceph-mon[31360]: pgmap v15322: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:20 smithi022 ceph-mon[40250]: pgmap v15322: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:22 smithi122 ceph-mon[32255]: pgmap v15323: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:29:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:29:22.753 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:22.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:22 smithi022 ceph-mon[31360]: pgmap v15323: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:22.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:22 smithi022 ceph-mon[40250]: pgmap v15323: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:23.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:23.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:23.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2852871313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2852871313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2852871313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:24 smithi122 ceph-mon[32255]: pgmap v15324: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:24 smithi022 ceph-mon[31360]: pgmap v15324: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:24 smithi022 ceph-mon[40250]: pgmap v15324: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:26 smithi122 ceph-mon[32255]: pgmap v15325: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:26 smithi022 ceph-mon[31360]: pgmap v15325: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:26 smithi022 ceph-mon[40250]: pgmap v15325: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:29:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:28 smithi122 ceph-mon[32255]: pgmap v15326: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:28.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:28.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:28.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:28.495 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:28 smithi022 ceph-mon[31360]: pgmap v15326: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:28.495 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:28 smithi022 ceph-mon[40250]: pgmap v15326: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:28.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:29:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:29:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2571471073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2571471073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2571471073' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:30 smithi122 ceph-mon[32255]: pgmap v15327: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:30] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:29:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:30 smithi022 ceph-mon[31360]: pgmap v15327: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:30 smithi022 ceph-mon[40250]: pgmap v15327: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:32 smithi122 ceph-mon[32255]: pgmap v15328: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:29:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:29:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:29:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:29:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:29:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[31360]: pgmap v15328: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:29:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:29:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:29:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[40250]: pgmap v15328: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:29:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:29:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:29:33.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:29:33.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:29:33.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:33.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:33.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:29:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[31360]: pgmap v15329: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3394485905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:29:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:29:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:29:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:29:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[40250]: pgmap v15329: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3394485905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:29:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:29:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:29:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:29:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:34 smithi122 ceph-mon[32255]: pgmap v15329: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3394485905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:29:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:29:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:29:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:29:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:36 smithi022 ceph-mon[31360]: pgmap v15330: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:36 smithi022 ceph-mon[40250]: pgmap v15330: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:36 smithi122 ceph-mon[32255]: pgmap v15330: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:29:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:38 smithi122 ceph-mon[32255]: pgmap v15331: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:38 smithi022 ceph-mon[31360]: pgmap v15331: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:38 smithi022 ceph-mon[40250]: pgmap v15331: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:38.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:38.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:29:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:29:39.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:39.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:39.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:40] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:29:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2092030127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:40 smithi122 ceph-mon[32255]: pgmap v15332: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2092030127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:40 smithi022 ceph-mon[31360]: pgmap v15332: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2092030127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:40 smithi022 ceph-mon[40250]: pgmap v15332: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:42 smithi122 ceph-mon[32255]: pgmap v15333: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:29:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:29:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:42 smithi022 ceph-mon[31360]: pgmap v15333: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:42 smithi022 ceph-mon[40250]: pgmap v15333: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:44.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:44 smithi022 ceph-mon[31360]: pgmap v15334: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:44 smithi022 ceph-mon[40250]: pgmap v15334: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:44.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:44.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:44 smithi122 ceph-mon[32255]: pgmap v15334: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3844417970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3844417970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3844417970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:46 smithi122 ceph-mon[32255]: pgmap v15335: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:46 smithi022 ceph-mon[31360]: pgmap v15335: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:46 smithi022 ceph-mon[40250]: pgmap v15335: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:29:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:48 smithi122 ceph-mon[32255]: pgmap v15336: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:48 smithi022 ceph-mon[31360]: pgmap v15336: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:48 smithi022 ceph-mon[40250]: pgmap v15336: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:48.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:29:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:29:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:49.610 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:49.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:49.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:49.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:50 smithi022 ceph-mon[31360]: pgmap v15337: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/280801574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:50 smithi022 ceph-mon[40250]: pgmap v15337: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/280801574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:50] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:29:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:50 smithi122 ceph-mon[32255]: pgmap v15337: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/280801574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:52 smithi122 ceph-mon[32255]: pgmap v15338: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:29:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:29:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:52 smithi022 ceph-mon[31360]: pgmap v15338: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:52 smithi022 ceph-mon[40250]: pgmap v15338: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:54 smithi122 ceph-mon[32255]: pgmap v15339: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:54 smithi022 ceph-mon[31360]: pgmap v15339: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:54 smithi022 ceph-mon[40250]: pgmap v15339: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:29:54.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:29:55.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:29:55.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:29:55.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:29:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3644112638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3644112638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3644112638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:29:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:56 smithi122 ceph-mon[32255]: pgmap v15340: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:56 smithi022 ceph-mon[31360]: pgmap v15340: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:56 smithi022 ceph-mon[40250]: pgmap v15340: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:29:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:58 smithi122 ceph-mon[32255]: pgmap v15341: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:58 smithi022 ceph-mon[31360]: pgmap v15341: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:58 smithi022 ceph-mon[40250]: pgmap v15341: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:29:58.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:29:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:29:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:29:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:29:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:29:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:29:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:00.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:00] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:30:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:00 smithi122 ceph-mon[32255]: pgmap v15342: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T02:30:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:00 smithi022 ceph-mon[40250]: pgmap v15342: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T02:30:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:00 smithi022 ceph-mon[31360]: pgmap v15342: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T02:30:00.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:00.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:00.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/396084164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/396084164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/396084164' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:02 smithi122 ceph-mon[32255]: pgmap v15343: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:30:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:30:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:02 smithi022 ceph-mon[31360]: pgmap v15343: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:02 smithi022 ceph-mon[40250]: pgmap v15343: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:04 smithi122 ceph-mon[32255]: pgmap v15344: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:04 smithi022 ceph-mon[31360]: pgmap v15344: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:04 smithi022 ceph-mon[40250]: pgmap v15344: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:05.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:06.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:06.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:06.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:06 smithi122 ceph-mon[32255]: pgmap v15345: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1712966718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:06 smithi022 ceph-mon[31360]: pgmap v15345: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1712966718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:06 smithi022 ceph-mon[40250]: pgmap v15345: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1712966718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:30:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:08 smithi122 ceph-mon[32255]: pgmap v15346: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:08 smithi022 ceph-mon[31360]: pgmap v15346: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:08 smithi022 ceph-mon[40250]: pgmap v15346: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:08.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:30:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:30:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:10 smithi122 ceph-mon[32255]: pgmap v15347: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:10] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T02:30:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:10 smithi022 ceph-mon[31360]: pgmap v15347: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:10 smithi022 ceph-mon[40250]: pgmap v15347: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:11.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:11.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:11.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:11.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:12 smithi122 ceph-mon[32255]: pgmap v15348: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1226562612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:30:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:30:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:12 smithi022 ceph-mon[31360]: pgmap v15348: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1226562612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:12 smithi022 ceph-mon[40250]: pgmap v15348: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1226562612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:14 smithi122 ceph-mon[32255]: pgmap v15349: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:14 smithi022 ceph-mon[31360]: pgmap v15349: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:14 smithi022 ceph-mon[40250]: pgmap v15349: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:16.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:16 smithi122 ceph-mon[32255]: pgmap v15350: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:16 smithi022 ceph-mon[31360]: pgmap v15350: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:16 smithi022 ceph-mon[40250]: pgmap v15350: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:16.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:16.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:16.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2200762035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2200762035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:30:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2200762035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:18 smithi122 ceph-mon[32255]: pgmap v15351: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:18.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:30:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:30:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:18 smithi022 ceph-mon[31360]: pgmap v15351: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:18.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:18 smithi022 ceph-mon[40250]: pgmap v15351: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:20.433 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:20] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:30:20.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:20 smithi122 ceph-mon[32255]: pgmap v15352: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:20 smithi022 ceph-mon[31360]: pgmap v15352: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:20.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:20 smithi022 ceph-mon[40250]: pgmap v15352: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:21.839 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:22.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:22.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:22.212 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:22 smithi022 ceph-mon[31360]: pgmap v15353: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1902202170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:22 smithi022 ceph-mon[40250]: pgmap v15353: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1902202170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:30:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:30:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:22 smithi122 ceph-mon[32255]: pgmap v15353: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1902202170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:24 smithi022 ceph-mon[31360]: pgmap v15354: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:24 smithi022 ceph-mon[40250]: pgmap v15354: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:24 smithi122 ceph-mon[32255]: pgmap v15354: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:26 smithi022 ceph-mon[31360]: pgmap v15355: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:26 smithi022 ceph-mon[40250]: pgmap v15355: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:26 smithi122 ceph-mon[32255]: pgmap v15355: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:27.213 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:30:27.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:27.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:27.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:28 smithi122 ceph-mon[32255]: pgmap v15356: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4238433518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:28 smithi022 ceph-mon[31360]: pgmap v15356: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4238433518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:28 smithi022 ceph-mon[40250]: pgmap v15356: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4238433518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:28.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:30:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:30:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:30] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:30:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:30 smithi122 ceph-mon[32255]: pgmap v15357: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:30 smithi022 ceph-mon[31360]: pgmap v15357: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:30 smithi022 ceph-mon[40250]: pgmap v15357: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:32.586 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:32 smithi122 ceph-mon[32255]: pgmap v15358: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:30:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:30:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:32 smithi022 ceph-mon[31360]: pgmap v15358: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:32 smithi022 ceph-mon[40250]: pgmap v15358: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:32.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:32.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:32.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:30:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:30:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:30:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1945414725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:30:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:30:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:30:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:30:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1945414725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:30:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:30:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:30:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:30:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1945414725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:33.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:34 smithi122 ceph-mon[32255]: pgmap v15359: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:30:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:30:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:30:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:30:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[31360]: pgmap v15359: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:30:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:30:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:30:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:30:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[40250]: pgmap v15359: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:30:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:30:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:30:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:30:36.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:36 smithi122 ceph-mon[32255]: pgmap v15360: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:36 smithi022 ceph-mon[31360]: pgmap v15360: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:36 smithi022 ceph-mon[40250]: pgmap v15360: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:30:37.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:38 smithi022 ceph-mon[31360]: pgmap v15361: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:38 smithi022 ceph-mon[40250]: pgmap v15361: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:38.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:38.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:38.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:38.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:38 smithi122 ceph-mon[32255]: pgmap v15361: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:38.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:30:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:30:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1476872404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1476872404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1476872404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:40] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:30:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:40 smithi122 ceph-mon[32255]: pgmap v15362: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:40 smithi022 ceph-mon[31360]: pgmap v15362: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:40 smithi022 ceph-mon[40250]: pgmap v15362: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:42 smithi122 ceph-mon[32255]: pgmap v15363: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:30:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:30:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:42 smithi022 ceph-mon[31360]: pgmap v15363: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:42 smithi022 ceph-mon[40250]: pgmap v15363: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:43.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:43.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:43.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:43.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:44 smithi122 ceph-mon[32255]: pgmap v15364: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3510866861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:44 smithi022 ceph-mon[31360]: pgmap v15364: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3510866861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:44 smithi022 ceph-mon[40250]: pgmap v15364: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3510866861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:46 smithi122 ceph-mon[32255]: pgmap v15365: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:46 smithi022 ceph-mon[31360]: pgmap v15365: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:46 smithi022 ceph-mon[40250]: pgmap v15365: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:30:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:48 smithi122 ceph-mon[32255]: pgmap v15366: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:48 smithi022 ceph-mon[31360]: pgmap v15366: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:48 smithi022 ceph-mon[40250]: pgmap v15366: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:48.698 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:48.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:30:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:30:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3156293269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3156293269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3156293269' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:50 smithi122 ceph-mon[32255]: pgmap v15367: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:50] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:30:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:50 smithi022 ceph-mon[31360]: pgmap v15367: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:50 smithi022 ceph-mon[40250]: pgmap v15367: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:52 smithi122 ceph-mon[32255]: pgmap v15368: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:30:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:30:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:52 smithi022 ceph-mon[31360]: pgmap v15368: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:52 smithi022 ceph-mon[40250]: pgmap v15368: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:54.072 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:54.434 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:54 smithi022 ceph-mon[31360]: pgmap v15369: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:54.434 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3347631925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:54.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:54.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:54.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:30:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:54 smithi122 ceph-mon[32255]: pgmap v15369: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3347631925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:54 smithi022 ceph-mon[40250]: pgmap v15369: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:30:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3347631925' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:30:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:56 smithi022 ceph-mon[31360]: pgmap v15370: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:56 smithi022 ceph-mon[40250]: pgmap v15370: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:56 smithi122 ceph-mon[32255]: pgmap v15370: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:30:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:58 smithi122 ceph-mon[32255]: pgmap v15371: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:58 smithi022 ceph-mon[31360]: pgmap v15371: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:58 smithi022 ceph-mon[40250]: pgmap v15371: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:30:58.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:30:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:30:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:30:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:59.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:30:59.446 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:30:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:59.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:30:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:30:59.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:30:59.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:30:59.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:00] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:31:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:00 smithi122 ceph-mon[32255]: pgmap v15372: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1216346855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:00 smithi022 ceph-mon[31360]: pgmap v15372: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1216346855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:00 smithi022 ceph-mon[40250]: pgmap v15372: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1216346855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:02 smithi122 ceph-mon[32255]: pgmap v15373: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:31:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:31:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:02 smithi022 ceph-mon[31360]: pgmap v15373: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:02.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:02 smithi022 ceph-mon[40250]: pgmap v15373: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:04 smithi022 ceph-mon[31360]: pgmap v15374: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:04 smithi022 ceph-mon[40250]: pgmap v15374: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:04.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:04 smithi122 ceph-mon[32255]: pgmap v15374: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:05.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:05.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:05.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:05.442 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/449252538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/449252538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/449252538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:06 smithi022 ceph-mon[31360]: pgmap v15375: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:06 smithi022 ceph-mon[40250]: pgmap v15375: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:06 smithi122 ceph-mon[32255]: pgmap v15375: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:31:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:08 smithi122 ceph-mon[32255]: pgmap v15376: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:08 smithi022 ceph-mon[31360]: pgmap v15376: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:08 smithi022 ceph-mon[40250]: pgmap v15376: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:09.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:31:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:31:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:10.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:10 smithi122 ceph-mon[32255]: pgmap v15377: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:10] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:31:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:10 smithi022 ceph-mon[31360]: pgmap v15377: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:10 smithi022 ceph-mon[40250]: pgmap v15377: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:10.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:10.555 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:10.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3305860839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3305860839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3305860839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:12 smithi122 ceph-mon[32255]: pgmap v15378: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:31:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:31:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:12 smithi022 ceph-mon[31360]: pgmap v15378: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:12 smithi022 ceph-mon[40250]: pgmap v15378: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:14 smithi022 ceph-mon[31360]: pgmap v15379: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:14 smithi022 ceph-mon[40250]: pgmap v15379: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:14.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:14 smithi122 ceph-mon[32255]: pgmap v15379: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:15.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:15.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:15.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:16 smithi022 ceph-mon[31360]: pgmap v15380: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2451917718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:16 smithi022 ceph-mon[40250]: pgmap v15380: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2451917718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:16 smithi122 ceph-mon[32255]: pgmap v15380: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2451917718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:31:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:18 smithi122 ceph-mon[32255]: pgmap v15381: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:18 smithi022 ceph-mon[31360]: pgmap v15381: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:18 smithi022 ceph-mon[40250]: pgmap v15381: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T02:31:19.147 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:31:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:31:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:20] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:31:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:20 smithi122 ceph-mon[32255]: pgmap v15382: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:20 smithi022 ceph-mon[31360]: pgmap v15382: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:20 smithi022 ceph-mon[40250]: pgmap v15382: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T02:31:20.928 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:21.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:21.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:21.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2231928599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2231928599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2231928599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:22 smithi122 ceph-mon[32255]: pgmap v15383: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:31:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:31:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:22 smithi022 ceph-mon[31360]: pgmap v15383: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:22 smithi022 ceph-mon[40250]: pgmap v15383: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:24 smithi022 ceph-mon[31360]: pgmap v15384: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:24 smithi022 ceph-mon[40250]: pgmap v15384: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:24 smithi122 ceph-mon[32255]: pgmap v15384: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:26.296 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:26.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:26.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:26.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:26.668 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:26 smithi022 ceph-mon[40250]: pgmap v15385: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:26.669 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:26 smithi022 ceph-mon[31360]: pgmap v15385: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:26 smithi122 ceph-mon[32255]: pgmap v15385: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1136849821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:31:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1136849821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1136849821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:28.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:28 smithi122 ceph-mon[32255]: pgmap v15386: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:28 smithi022 ceph-mon[31360]: pgmap v15386: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:28 smithi022 ceph-mon[40250]: pgmap v15386: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:29.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:31:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:31:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:30] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:31:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:30 smithi022 ceph-mon[31360]: pgmap v15387: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:30 smithi022 ceph-mon[40250]: pgmap v15387: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:30 smithi122 ceph-mon[32255]: pgmap v15387: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:31.669 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:32.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:32.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:32.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:32 smithi122 ceph-mon[32255]: pgmap v15388: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/340076701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:31:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:31:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:32 smithi022 ceph-mon[31360]: pgmap v15388: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/340076701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:32 smithi022 ceph-mon[40250]: pgmap v15388: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/340076701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:31:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:31:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:31:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:31:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:31:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:31:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:31:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:31:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:31:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:31:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:31:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:31:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[31360]: pgmap v15389: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:31:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:31:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:31:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:31:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[40250]: pgmap v15389: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:31:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:31:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:31:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:31:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:34 smithi122 ceph-mon[32255]: pgmap v15389: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:31:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:31:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:31:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:31:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:36 smithi022 ceph-mon[31360]: pgmap v15390: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:36 smithi022 ceph-mon[40250]: pgmap v15390: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:36 smithi122 ceph-mon[32255]: pgmap v15390: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:37.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:37.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:37.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:37.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:31:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:38 smithi122 ceph-mon[32255]: pgmap v15391: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1534419694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:38 smithi022 ceph-mon[31360]: pgmap v15391: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1534419694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:38 smithi022 ceph-mon[40250]: pgmap v15391: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1534419694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:39.149 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:31:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:31:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:40 smithi122 ceph-mon[32255]: pgmap v15392: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:40] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:31:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:40 smithi022 ceph-mon[31360]: pgmap v15392: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:40 smithi022 ceph-mon[40250]: pgmap v15392: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:42.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:42 smithi122 ceph-mon[32255]: pgmap v15393: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:31:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:31:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:42 smithi022 ceph-mon[31360]: pgmap v15393: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:42 smithi022 ceph-mon[40250]: pgmap v15393: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:42.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:42.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:42.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2548023500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2548023500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2548023500' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:44 smithi022 ceph-mon[31360]: pgmap v15394: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:44 smithi022 ceph-mon[40250]: pgmap v15394: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:44 smithi122 ceph-mon[32255]: pgmap v15394: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:46 smithi022 ceph-mon[31360]: pgmap v15395: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:46 smithi022 ceph-mon[40250]: pgmap v15395: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:46 smithi122 ceph-mon[32255]: pgmap v15395: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:31:47.780 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:48.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:48.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:48.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:48 smithi122 ceph-mon[32255]: pgmap v15396: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3074984750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:48 smithi022 ceph-mon[31360]: pgmap v15396: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3074984750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:48 smithi022 ceph-mon[40250]: pgmap v15396: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3074984750' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:49.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:31:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:31:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:50] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:31:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:50 smithi122 ceph-mon[32255]: pgmap v15397: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:50 smithi022 ceph-mon[31360]: pgmap v15397: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:50 smithi022 ceph-mon[40250]: pgmap v15397: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:52 smithi122 ceph-mon[32255]: pgmap v15398: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:31:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:31:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:52 smithi022 ceph-mon[31360]: pgmap v15398: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:52 smithi022 ceph-mon[40250]: pgmap v15398: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:53.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:53.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:53.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:53.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:54 smithi022 ceph-mon[31360]: pgmap v15399: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/500933702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:54 smithi022 ceph-mon[40250]: pgmap v15399: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/500933702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:54 smithi122 ceph-mon[32255]: pgmap v15399: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:31:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/500933702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:56 smithi022 ceph-mon[31360]: pgmap v15400: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:56 smithi022 ceph-mon[40250]: pgmap v15400: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:56 smithi122 ceph-mon[32255]: pgmap v15400: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:31:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:58 smithi122 ceph-mon[32255]: pgmap v15401: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:31:58.521 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:58 smithi022 ceph-mon[31360]: pgmap v15401: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:58.522 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:58 smithi022 ceph-mon[40250]: pgmap v15401: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:31:58.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:31:58.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:31:58.892 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:31:59.154 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:31:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:31:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:31:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/209155184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:31:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/209155184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:31:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:31:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:31:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/209155184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:00] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:32:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:00 smithi122 ceph-mon[32255]: pgmap v15402: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:00 smithi022 ceph-mon[31360]: pgmap v15402: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:00 smithi022 ceph-mon[40250]: pgmap v15402: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:02 smithi122 ceph-mon[32255]: pgmap v15403: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:32:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:32:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:02 smithi022 ceph-mon[31360]: pgmap v15403: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:02 smithi022 ceph-mon[40250]: pgmap v15403: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:03.893 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:04.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:04.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:04.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:04 smithi022 ceph-mon[31360]: pgmap v15404: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3233451931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:04 smithi022 ceph-mon[40250]: pgmap v15404: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3233451931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:04 smithi122 ceph-mon[32255]: pgmap v15404: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3233451931' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:06 smithi022 ceph-mon[31360]: pgmap v15405: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:06 smithi022 ceph-mon[40250]: pgmap v15405: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:06 smithi122 ceph-mon[32255]: pgmap v15405: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:32:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:08 smithi122 ceph-mon[32255]: pgmap v15406: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:08 smithi022 ceph-mon[31360]: pgmap v15406: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:08 smithi022 ceph-mon[40250]: pgmap v15406: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:09.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:32:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:32:09.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:09.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:09.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:09.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:10 smithi122 ceph-mon[32255]: pgmap v15407: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1561811199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:10] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:32:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:10 smithi022 ceph-mon[31360]: pgmap v15407: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1561811199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:10 smithi022 ceph-mon[40250]: pgmap v15407: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1561811199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:12 smithi122 ceph-mon[32255]: pgmap v15408: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:32:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:32:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:12 smithi022 ceph-mon[31360]: pgmap v15408: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:12 smithi022 ceph-mon[40250]: pgmap v15408: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:14.636 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:14 smithi022 ceph-mon[31360]: pgmap v15409: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:14 smithi022 ceph-mon[40250]: pgmap v15409: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:14 smithi122 ceph-mon[32255]: pgmap v15409: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:15.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:15.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:15.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3886059983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3886059983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3886059983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:16 smithi022 ceph-mon[31360]: pgmap v15410: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:16 smithi022 ceph-mon[40250]: pgmap v15410: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:16 smithi122 ceph-mon[32255]: pgmap v15410: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:32:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:18 smithi122 ceph-mon[32255]: pgmap v15411: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:18 smithi022 ceph-mon[31360]: pgmap v15411: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:18 smithi022 ceph-mon[40250]: pgmap v15411: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:19.159 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:32:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:32:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:19 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:19 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:19.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:19 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:20.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:20 smithi022 ceph-mon[31360]: pgmap v15412: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:20 smithi022 ceph-mon[40250]: pgmap v15412: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:20.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:20.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:20.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:20] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:32:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:20 smithi122 ceph-mon[32255]: pgmap v15412: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/260298126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:21.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/260298126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:21.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/260298126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:22 smithi122 ceph-mon[32255]: pgmap v15413: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:32:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:32:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:22 smithi022 ceph-mon[31360]: pgmap v15413: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:22 smithi022 ceph-mon[40250]: pgmap v15413: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:24 smithi022 ceph-mon[31360]: pgmap v15414: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:24 smithi022 ceph-mon[40250]: pgmap v15414: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:24 smithi122 ceph-mon[32255]: pgmap v15414: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:25.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:25.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:25.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:25.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:26 smithi022 ceph-mon[31360]: pgmap v15415: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2135477682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:26 smithi022 ceph-mon[40250]: pgmap v15415: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2135477682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:26 smithi122 ceph-mon[32255]: pgmap v15415: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2135477682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:32:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:28 smithi122 ceph-mon[32255]: pgmap v15416: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:28 smithi022 ceph-mon[31360]: pgmap v15416: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:28 smithi022 ceph-mon[40250]: pgmap v15416: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:29.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:32:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:32:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:29 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:29 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:29 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:30] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:32:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:30 smithi122 ceph-mon[32255]: pgmap v15417: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:30 smithi022 ceph-mon[31360]: pgmap v15417: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:30 smithi022 ceph-mon[40250]: pgmap v15417: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:30.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:31.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:31.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/771350334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/771350334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/771350334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:32 smithi122 ceph-mon[32255]: pgmap v15418: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:32:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:32:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:32 smithi022 ceph-mon[31360]: pgmap v15418: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:32 smithi022 ceph-mon[40250]: pgmap v15418: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:32:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:32:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:32:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:32:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:32:33.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:32:33.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:32:33.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:32:33.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:32:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[31360]: pgmap v15419: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:32:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:32:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:32:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:32:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:32:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[40250]: pgmap v15419: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:32:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:32:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:32:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:32:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:32:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:34 smithi122 ceph-mon[32255]: pgmap v15419: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:32:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:32:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:32:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:32:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:36.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:36.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:36.496 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:36 smithi022 ceph-mon[31360]: pgmap v15420: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:36.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:36 smithi022 ceph-mon[40250]: pgmap v15420: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:36 smithi122 ceph-mon[32255]: pgmap v15420: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1086225179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:32:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1086225179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1086225179' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:38 smithi022 ceph-mon[31360]: pgmap v15421: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:38 smithi022 ceph-mon[40250]: pgmap v15421: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:38.779 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:38 smithi122 ceph-mon[32255]: pgmap v15421: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:39.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:32:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:32:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:39 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:39 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:39 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:40] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:32:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:40 smithi022 ceph-mon[31360]: pgmap v15422: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:40 smithi022 ceph-mon[40250]: pgmap v15422: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:40 smithi122 ceph-mon[32255]: pgmap v15422: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:41.498 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:41.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:41.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:41.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:32:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:32:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:42 smithi122 ceph-mon[32255]: pgmap v15423: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3448736759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:42 smithi022 ceph-mon[31360]: pgmap v15423: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3448736759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:42 smithi022 ceph-mon[40250]: pgmap v15423: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3448736759' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:44 smithi022 ceph-mon[31360]: pgmap v15424: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:44 smithi022 ceph-mon[40250]: pgmap v15424: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:44 smithi122 ceph-mon[32255]: pgmap v15424: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:46 smithi022 ceph-mon[31360]: pgmap v15425: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:46 smithi022 ceph-mon[40250]: pgmap v15425: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:46.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:46 smithi122 ceph-mon[32255]: pgmap v15425: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:47.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:47.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:47.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:32:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1064715495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:48 smithi122 ceph-mon[32255]: pgmap v15426: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1064715495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:48 smithi022 ceph-mon[31360]: pgmap v15426: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1064715495' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:48 smithi022 ceph-mon[40250]: pgmap v15426: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:49.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:32:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:32:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:49 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:49 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:49 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:50] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:32:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:50 smithi122 ceph-mon[32255]: pgmap v15427: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:50 smithi022 ceph-mon[31360]: pgmap v15427: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:50 smithi022 ceph-mon[40250]: pgmap v15427: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:52.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:52 smithi022 ceph-mon[31360]: pgmap v15428: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:52 smithi022 ceph-mon[40250]: pgmap v15428: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:52 smithi122 ceph-mon[32255]: pgmap v15428: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:32:52.314Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:32:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2469159259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2469159259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2469159259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:54 smithi022 ceph-mon[31360]: pgmap v15429: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:54 smithi022 ceph-mon[40250]: pgmap v15429: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:54 smithi122 ceph-mon[32255]: pgmap v15429: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:32:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:56 smithi022 ceph-mon[31360]: pgmap v15430: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:56 smithi022 ceph-mon[40250]: pgmap v15430: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:56 smithi122 ceph-mon[32255]: pgmap v15430: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:57.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:32:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:32:57.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:32:57.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:32:57.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:32:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:58 smithi022 ceph-mon[31360]: pgmap v15431: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2034710560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:58 smithi022 ceph-mon[40250]: pgmap v15431: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2034710560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:58 smithi122 ceph-mon[32255]: pgmap v15431: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:32:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2034710560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:32:59.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:32:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:32:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:32:59 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:32:59 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:32:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:32:59 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:00] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:33:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:00 smithi122 ceph-mon[32255]: pgmap v15432: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:00 smithi022 ceph-mon[31360]: pgmap v15432: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:00 smithi022 ceph-mon[40250]: pgmap v15432: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:02 smithi122 ceph-mon[32255]: pgmap v15433: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:33:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:33:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:02 smithi022 ceph-mon[31360]: pgmap v15433: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:02 smithi022 ceph-mon[40250]: pgmap v15433: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:03.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:03.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:03.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3836400044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3836400044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3836400044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:04 smithi022 ceph-mon[31360]: pgmap v15434: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:04 smithi022 ceph-mon[40250]: pgmap v15434: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:04 smithi122 ceph-mon[32255]: pgmap v15434: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:06 smithi022 ceph-mon[31360]: pgmap v15435: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:06 smithi022 ceph-mon[40250]: pgmap v15435: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:06 smithi122 ceph-mon[32255]: pgmap v15435: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:33:08.358 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:08 smithi122 ceph-mon[32255]: pgmap v15436: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:08 smithi022 ceph-mon[31360]: pgmap v15436: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:08 smithi022 ceph-mon[40250]: pgmap v15436: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:08.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:08.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:08.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:09.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:33:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:33:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/134049166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:09 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/134049166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:09 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/134049166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:09 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:10] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:33:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:10 smithi122 ceph-mon[32255]: pgmap v15437: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:10 smithi022 ceph-mon[31360]: pgmap v15437: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:10 smithi022 ceph-mon[40250]: pgmap v15437: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:12 smithi122 ceph-mon[32255]: pgmap v15438: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:33:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:33:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:12 smithi022 ceph-mon[31360]: pgmap v15438: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:12 smithi022 ceph-mon[40250]: pgmap v15438: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:13.731 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:14.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:14.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:14.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:14 smithi022 ceph-mon[31360]: pgmap v15439: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/98624867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:14 smithi022 ceph-mon[40250]: pgmap v15439: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/98624867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:14 smithi122 ceph-mon[32255]: pgmap v15439: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/98624867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:16 smithi022 ceph-mon[31360]: pgmap v15440: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:16 smithi022 ceph-mon[40250]: pgmap v15440: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:16 smithi122 ceph-mon[32255]: pgmap v15440: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:33:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:18 smithi122 ceph-mon[32255]: pgmap v15441: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:18 smithi022 ceph-mon[31360]: pgmap v15441: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:18 smithi022 ceph-mon[40250]: pgmap v15441: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:19.093 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:19.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:33:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:33:19.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:19.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:19.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3676066032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3676066032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3676066032' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:20] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:33:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:20 smithi022 ceph-mon[31360]: pgmap v15442: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:20 smithi022 ceph-mon[40250]: pgmap v15442: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:20 smithi122 ceph-mon[32255]: pgmap v15442: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:22 smithi122 ceph-mon[32255]: pgmap v15443: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:33:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:33:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:22 smithi022 ceph-mon[31360]: pgmap v15443: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:22 smithi022 ceph-mon[40250]: pgmap v15443: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:24.462 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:24 smithi022 ceph-mon[31360]: pgmap v15444: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:24 smithi022 ceph-mon[40250]: pgmap v15444: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:24 smithi122 ceph-mon[32255]: pgmap v15444: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3892445333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3892445333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3892445333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:26 smithi022 ceph-mon[31360]: pgmap v15445: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:26 smithi022 ceph-mon[40250]: pgmap v15445: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:26 smithi122 ceph-mon[32255]: pgmap v15445: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:33:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:28 smithi122 ceph-mon[32255]: pgmap v15446: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:28 smithi022 ceph-mon[31360]: pgmap v15446: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:28 smithi022 ceph-mon[40250]: pgmap v15446: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:29.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:33:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:33:29.834 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:30.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:30.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:30.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:30] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:33:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:30 smithi022 ceph-mon[40250]: pgmap v15447: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/614340472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:30 smithi022 ceph-mon[31360]: pgmap v15447: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:30.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/614340472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:30 smithi122 ceph-mon[32255]: pgmap v15447: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/614340472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:32 smithi122 ceph-mon[32255]: pgmap v15448: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:33:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:33:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:32 smithi022 ceph-mon[31360]: pgmap v15448: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:32 smithi022 ceph-mon[40250]: pgmap v15448: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: pgmap v15449: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:33:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:33:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:33:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:33:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: pgmap v15449: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:33:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:33:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:33:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:33:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:33:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:34.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: pgmap v15449: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:34.661 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:33:34.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:33:34.662 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:33:34.663 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:33:34.663 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:33:34.663 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:33:34.664 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:33:34.664 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:34.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:35.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:35.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:35.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:35.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1572119811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1572119811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:33:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1572119811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:36 smithi022 ceph-mon[31360]: pgmap v15450: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:36 smithi022 ceph-mon[40250]: pgmap v15450: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:36 smithi122 ceph-mon[32255]: pgmap v15450: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:33:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:38 smithi122 ceph-mon[32255]: pgmap v15451: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:38 smithi022 ceph-mon[31360]: pgmap v15451: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:38.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:38 smithi022 ceph-mon[40250]: pgmap v15451: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:39.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:33:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:33:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:40] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:33:40.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:40 smithi022 ceph-mon[40250]: pgmap v15452: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:40 smithi022 ceph-mon[31360]: pgmap v15452: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:40 smithi122 ceph-mon[32255]: pgmap v15452: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:40.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:40.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:40.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/440158250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:41.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/440158250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/440158250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:42 smithi122 ceph-mon[32255]: pgmap v15453: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:33:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:33:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:42 smithi022 ceph-mon[31360]: pgmap v15453: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:42 smithi022 ceph-mon[40250]: pgmap v15453: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:44 smithi022 ceph-mon[31360]: pgmap v15454: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:44 smithi022 ceph-mon[40250]: pgmap v15454: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:44 smithi122 ceph-mon[32255]: pgmap v15454: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:45.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:46.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:46.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:46.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:46 smithi022 ceph-mon[31360]: pgmap v15455: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/770090192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:46 smithi022 ceph-mon[40250]: pgmap v15455: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/770090192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:46 smithi122 ceph-mon[32255]: pgmap v15455: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/770090192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:33:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:48 smithi122 ceph-mon[32255]: pgmap v15456: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:48 smithi022 ceph-mon[31360]: pgmap v15456: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:48 smithi022 ceph-mon[40250]: pgmap v15456: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:49.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:33:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:33:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:50] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:33:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:50 smithi022 ceph-mon[31360]: pgmap v15457: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:50 smithi022 ceph-mon[40250]: pgmap v15457: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:33:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:50 smithi122 ceph-mon[32255]: pgmap v15457: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:51.308 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:52 smithi122 ceph-mon[32255]: pgmap v15458: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2258850109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:33:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:33:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:52 smithi022 ceph-mon[31360]: pgmap v15458: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2258850109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:52 smithi022 ceph-mon[40250]: pgmap v15458: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2258850109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:54 smithi022 ceph-mon[31360]: pgmap v15459: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:54 smithi022 ceph-mon[40250]: pgmap v15459: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:54 smithi122 ceph-mon[32255]: pgmap v15459: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:33:56.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:33:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:56 smithi022 ceph-mon[31360]: pgmap v15460: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:56 smithi022 ceph-mon[40250]: pgmap v15460: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:56 smithi122 ceph-mon[32255]: pgmap v15460: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:57.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:33:57.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:33:57.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:33:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1720933276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:33:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1720933276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1720933276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:33:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:33:58 smithi022 ceph-mon[31360]: pgmap v15461: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:33:58 smithi022 ceph-mon[40250]: pgmap v15461: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:33:58 smithi122 ceph-mon[32255]: pgmap v15461: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:33:59.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:33:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:34:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:00] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:34:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:00 smithi022 ceph-mon[31360]: pgmap v15462: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:00 smithi022 ceph-mon[40250]: pgmap v15462: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:00 smithi122 ceph-mon[32255]: pgmap v15462: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:02.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:02.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:02.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:02 smithi122 ceph-mon[32255]: pgmap v15463: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3614359466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:34:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:34:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:02 smithi022 ceph-mon[31360]: pgmap v15463: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3614359466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:02 smithi022 ceph-mon[40250]: pgmap v15463: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3614359466' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:04 smithi022 ceph-mon[31360]: pgmap v15464: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:04 smithi022 ceph-mon[40250]: pgmap v15464: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:04 smithi122 ceph-mon[32255]: pgmap v15464: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:06 smithi022 ceph-mon[31360]: pgmap v15465: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:06 smithi022 ceph-mon[40250]: pgmap v15465: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:06 smithi122 ceph-mon[32255]: pgmap v15465: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:07.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:34:07.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:07.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:07.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:08 smithi022 ceph-mon[31360]: pgmap v15466: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3662790108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:08 smithi022 ceph-mon[40250]: pgmap v15466: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3662790108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:08 smithi122 ceph-mon[32255]: pgmap v15466: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3662790108' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:09.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:34:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:34:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:10] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:34:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:10 smithi022 ceph-mon[31360]: pgmap v15467: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:10 smithi022 ceph-mon[40250]: pgmap v15467: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:10 smithi122 ceph-mon[32255]: pgmap v15467: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:12 smithi122 ceph-mon[32255]: pgmap v15468: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:34:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:34:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:12 smithi022 ceph-mon[31360]: pgmap v15468: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:12.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:12 smithi022 ceph-mon[40250]: pgmap v15468: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:12.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:13.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:13.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:13.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1883163847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1883163847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1883163847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:14 smithi022 ceph-mon[31360]: pgmap v15469: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:14 smithi022 ceph-mon[40250]: pgmap v15469: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:14 smithi122 ceph-mon[32255]: pgmap v15469: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:16 smithi022 ceph-mon[31360]: pgmap v15470: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:16 smithi022 ceph-mon[40250]: pgmap v15470: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:16 smithi122 ceph-mon[32255]: pgmap v15470: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:34:18.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:18 smithi022 ceph-mon[40250]: pgmap v15471: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:18 smithi022 ceph-mon[31360]: pgmap v15471: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:18.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:18.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:18.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:18 smithi122 ceph-mon[32255]: pgmap v15471: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:19.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:34:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:34:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/777585395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/777585395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/777585395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:20] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:34:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:20 smithi122 ceph-mon[32255]: pgmap v15472: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:20 smithi022 ceph-mon[31360]: pgmap v15472: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:20 smithi022 ceph-mon[40250]: pgmap v15472: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:22 smithi122 ceph-mon[32255]: pgmap v15473: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:34:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:34:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:22 smithi022 ceph-mon[31360]: pgmap v15473: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:22 smithi022 ceph-mon[40250]: pgmap v15473: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:23.544 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:23.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:23.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:23.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:24 smithi022 ceph-mon[31360]: pgmap v15474: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2169804084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:24 smithi022 ceph-mon[40250]: pgmap v15474: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2169804084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:24 smithi122 ceph-mon[32255]: pgmap v15474: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2169804084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:26 smithi022 ceph-mon[31360]: pgmap v15475: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:26 smithi022 ceph-mon[40250]: pgmap v15475: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:26 smithi122 ceph-mon[32255]: pgmap v15475: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:34:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:28 smithi022 ceph-mon[31360]: pgmap v15476: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:28 smithi022 ceph-mon[40250]: pgmap v15476: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:28 smithi122 ceph-mon[32255]: pgmap v15476: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:28.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:29.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:34:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:34:29.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:29.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:29.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3522531157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3522531157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3522531157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:30] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:34:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:30 smithi122 ceph-mon[32255]: pgmap v15477: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:30 smithi022 ceph-mon[31360]: pgmap v15477: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:30 smithi022 ceph-mon[40250]: pgmap v15477: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:34:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:34:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:32 smithi122 ceph-mon[32255]: pgmap v15478: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:32 smithi022 ceph-mon[31360]: pgmap v15478: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:32 smithi022 ceph-mon[40250]: pgmap v15478: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:34.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:34.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:34.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:34.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:34.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[31360]: pgmap v15479: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:34.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:34:34.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:34:34.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:34:34.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:34:34.651 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[40250]: pgmap v15479: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:34.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:34:34.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:34:34.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:34:34.652 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:34:34.765 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:34 smithi122 ceph-mon[32255]: pgmap v15479: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:34.765 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:34:34.766 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:34:34.766 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:34:34.766 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:34:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1188777515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:34:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:34:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:34:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:34:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1188777515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:34:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:34:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:34:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:34:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1188777515' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:34:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:34:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:34:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:34:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:36 smithi022 ceph-mon[31360]: pgmap v15480: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:36 smithi022 ceph-mon[40250]: pgmap v15480: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:36 smithi122 ceph-mon[32255]: pgmap v15480: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:34:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:38 smithi022 ceph-mon[31360]: pgmap v15481: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:38 smithi022 ceph-mon[40250]: pgmap v15481: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:38 smithi122 ceph-mon[32255]: pgmap v15481: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:39.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:34:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:34:39.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:40.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:40.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:40.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:40] "GET /metrics HTTP/1.1" 200 202440 "" "Prometheus/2.18.1" 2021-12-16T02:34:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:40 smithi022 ceph-mon[31360]: pgmap v15482: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/992400417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:40 smithi022 ceph-mon[40250]: pgmap v15482: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/992400417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:40 smithi122 ceph-mon[32255]: pgmap v15482: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/992400417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:42 smithi122 ceph-mon[32255]: pgmap v15483: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:34:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:34:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:42 smithi022 ceph-mon[31360]: pgmap v15483: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:42 smithi022 ceph-mon[40250]: pgmap v15483: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:44 smithi022 ceph-mon[31360]: pgmap v15484: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:44 smithi022 ceph-mon[40250]: pgmap v15484: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:44 smithi122 ceph-mon[32255]: pgmap v15484: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:45.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:45.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:45.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:45.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2526006351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2526006351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2526006351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:46 smithi022 ceph-mon[31360]: pgmap v15485: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:46 smithi022 ceph-mon[40250]: pgmap v15485: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:46 smithi122 ceph-mon[32255]: pgmap v15485: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:34:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:48 smithi022 ceph-mon[31360]: pgmap v15486: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:48 smithi022 ceph-mon[40250]: pgmap v15486: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:48 smithi122 ceph-mon[32255]: pgmap v15486: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:49.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:34:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:34:50.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:50] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:34:50.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:50.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:50.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:50.759 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:50.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:50 smithi022 ceph-mon[31360]: pgmap v15487: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:50.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:50.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:50 smithi022 ceph-mon[40250]: pgmap v15487: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:34:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:50 smithi122 ceph-mon[32255]: pgmap v15487: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4080227655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4080227655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:51.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4080227655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:52 smithi122 ceph-mon[32255]: pgmap v15488: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:34:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:34:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:52 smithi022 ceph-mon[31360]: pgmap v15488: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:52 smithi022 ceph-mon[40250]: pgmap v15488: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:54 smithi022 ceph-mon[31360]: pgmap v15489: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:54 smithi022 ceph-mon[40250]: pgmap v15489: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:54 smithi122 ceph-mon[32255]: pgmap v15489: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:34:55.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:34:56.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:34:56.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:34:56.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:34:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:56 smithi022 ceph-mon[31360]: pgmap v15490: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3714581577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:56 smithi022 ceph-mon[40250]: pgmap v15490: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3714581577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:56 smithi122 ceph-mon[32255]: pgmap v15490: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3714581577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:34:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:34:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:34:58 smithi022 ceph-mon[31360]: pgmap v15491: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:34:58 smithi022 ceph-mon[40250]: pgmap v15491: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:34:58 smithi122 ceph-mon[32255]: pgmap v15491: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:34:59.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:34:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:35:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:00] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:35:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:00 smithi022 ceph-mon[31360]: pgmap v15492: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:00 smithi022 ceph-mon[40250]: pgmap v15492: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:00 smithi122 ceph-mon[32255]: pgmap v15492: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:01.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:01.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:01.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:01.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3228056734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3228056734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3228056734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:02 smithi122 ceph-mon[32255]: pgmap v15493: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:35:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:35:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:02 smithi022 ceph-mon[31360]: pgmap v15493: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:02 smithi022 ceph-mon[40250]: pgmap v15493: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:04 smithi022 ceph-mon[31360]: pgmap v15494: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:04 smithi022 ceph-mon[40250]: pgmap v15494: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:04 smithi122 ceph-mon[32255]: pgmap v15494: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:06.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:06 smithi022 ceph-mon[31360]: pgmap v15495: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:06 smithi022 ceph-mon[40250]: pgmap v15495: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:06.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:06.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:06.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:06 smithi122 ceph-mon[32255]: pgmap v15495: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/346150513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:35:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/346150513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/346150513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:08 smithi022 ceph-mon[31360]: pgmap v15496: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:08 smithi022 ceph-mon[40250]: pgmap v15496: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:08.892 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:08 smithi122 ceph-mon[32255]: pgmap v15496: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:09.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:35:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:35:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:10] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:35:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:10 smithi022 ceph-mon[31360]: pgmap v15497: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:10 smithi022 ceph-mon[40250]: pgmap v15497: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:10 smithi122 ceph-mon[32255]: pgmap v15497: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:11.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:12 smithi122 ceph-mon[32255]: pgmap v15498: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2856134281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:35:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:35:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:12 smithi022 ceph-mon[31360]: pgmap v15498: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2856134281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:12 smithi022 ceph-mon[40250]: pgmap v15498: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2856134281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:14 smithi122 ceph-mon[32255]: pgmap v15499: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:14 smithi022 ceph-mon[31360]: pgmap v15499: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:14 smithi022 ceph-mon[40250]: pgmap v15499: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:16 smithi122 ceph-mon[32255]: pgmap v15500: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:16 smithi022 ceph-mon[31360]: pgmap v15500: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:16 smithi022 ceph-mon[40250]: pgmap v15500: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:17.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:35:17.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:17.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:17.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:18 smithi022 ceph-mon[31360]: pgmap v15501: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/66952697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:18 smithi022 ceph-mon[40250]: pgmap v15501: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/66952697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:18 smithi122 ceph-mon[32255]: pgmap v15501: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/66952697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:19.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:35:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:35:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:35:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:20 smithi022 ceph-mon[31360]: pgmap v15502: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:20 smithi022 ceph-mon[40250]: pgmap v15502: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:20 smithi122 ceph-mon[32255]: pgmap v15502: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:22.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:22 smithi122 ceph-mon[32255]: pgmap v15503: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:35:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:35:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:22 smithi022 ceph-mon[31360]: pgmap v15503: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:22 smithi022 ceph-mon[40250]: pgmap v15503: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:22.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:22.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:22.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/558672455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/558672455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/558672455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:24 smithi022 ceph-mon[31360]: pgmap v15504: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:24 smithi022 ceph-mon[40250]: pgmap v15504: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:24 smithi122 ceph-mon[32255]: pgmap v15504: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:26 smithi122 ceph-mon[32255]: pgmap v15505: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:26 smithi022 ceph-mon[31360]: pgmap v15505: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:26 smithi022 ceph-mon[40250]: pgmap v15505: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:35:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:28 smithi022 ceph-mon[31360]: pgmap v15506: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:28.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:28 smithi022 ceph-mon[40250]: pgmap v15506: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:28.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:28.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:28.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:28 smithi122 ceph-mon[32255]: pgmap v15506: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:29.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:35:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:35:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4010470903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4010470903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4010470903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:35:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:30 smithi122 ceph-mon[32255]: pgmap v15507: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:30 smithi022 ceph-mon[31360]: pgmap v15507: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:30 smithi022 ceph-mon[40250]: pgmap v15507: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:32 smithi122 ceph-mon[32255]: pgmap v15508: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:35:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:35:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:32 smithi022 ceph-mon[31360]: pgmap v15508: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:32 smithi022 ceph-mon[40250]: pgmap v15508: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:33.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:33.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:33.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:33.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[31360]: pgmap v15509: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2495045358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:35:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:35:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:35:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:35:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[40250]: pgmap v15509: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2495045358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:35:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:35:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:35:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:35:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:34 smithi122 ceph-mon[32255]: pgmap v15509: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2495045358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:35:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:35:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:35:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:35:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:35:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:35:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:35:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:35:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:35:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:35:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:35:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:35:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:35:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:36 smithi122 ceph-mon[32255]: pgmap v15510: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:35:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:36 smithi022 ceph-mon[31360]: pgmap v15510: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:35:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:36 smithi022 ceph-mon[40250]: pgmap v15510: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:35:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:35:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:38 smithi022 ceph-mon[31360]: pgmap v15511: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:38 smithi022 ceph-mon[40250]: pgmap v15511: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:38 smithi122 ceph-mon[32255]: pgmap v15511: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:38.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:39.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:39.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:39.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:39.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:35:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:35:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/825020081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/825020081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/825020081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:40] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:35:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:40 smithi122 ceph-mon[32255]: pgmap v15512: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:40 smithi022 ceph-mon[31360]: pgmap v15512: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:40 smithi022 ceph-mon[40250]: pgmap v15512: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:35:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:35:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:42 smithi122 ceph-mon[32255]: pgmap v15513: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:42 smithi022 ceph-mon[31360]: pgmap v15513: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:42 smithi022 ceph-mon[40250]: pgmap v15513: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:44.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:44.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:44.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:44.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:44 smithi022 ceph-mon[31360]: pgmap v15514: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/606194663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:44 smithi022 ceph-mon[40250]: pgmap v15514: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/606194663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:44 smithi122 ceph-mon[32255]: pgmap v15514: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/606194663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:46 smithi022 ceph-mon[31360]: pgmap v15515: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:46 smithi022 ceph-mon[40250]: pgmap v15515: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:46 smithi122 ceph-mon[32255]: pgmap v15515: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:35:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:48 smithi022 ceph-mon[31360]: pgmap v15516: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:48 smithi022 ceph-mon[40250]: pgmap v15516: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:48 smithi122 ceph-mon[32255]: pgmap v15516: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:49.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:35:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:35:49.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:50] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:35:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:50 smithi022 ceph-mon[31360]: pgmap v15517: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/111798420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:50 smithi022 ceph-mon[40250]: pgmap v15517: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/111798420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:50.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:35:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:50 smithi122 ceph-mon[32255]: pgmap v15517: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/111798420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:52 smithi122 ceph-mon[32255]: pgmap v15518: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:35:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:35:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:52 smithi022 ceph-mon[31360]: pgmap v15518: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:52 smithi022 ceph-mon[40250]: pgmap v15518: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:54.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:35:54.846 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:54 smithi022 ceph-mon[40250]: pgmap v15519: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:54.846 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:54 smithi022 ceph-mon[31360]: pgmap v15519: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:54 smithi122 ceph-mon[32255]: pgmap v15519: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:35:55.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:35:55.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:35:55.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:35:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2712491673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2712491673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2712491673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:35:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:56 smithi122 ceph-mon[32255]: pgmap v15520: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:56 smithi022 ceph-mon[31360]: pgmap v15520: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:56 smithi022 ceph-mon[40250]: pgmap v15520: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:35:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:35:58 smithi022 ceph-mon[31360]: pgmap v15521: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:35:58 smithi022 ceph-mon[40250]: pgmap v15521: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:35:58 smithi122 ceph-mon[32255]: pgmap v15521: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:35:59.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:35:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:36:00.217 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:00] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:36:00.526 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:00.527 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:00 smithi022 ceph-mon[40250]: pgmap v15522: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:00.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:00.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:00.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:00 smithi022 ceph-mon[31360]: pgmap v15522: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:00 smithi122 ceph-mon[32255]: pgmap v15522: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2932451047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2932451047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2932451047' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:02 smithi122 ceph-mon[32255]: pgmap v15523: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:36:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:36:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:02 smithi022 ceph-mon[31360]: pgmap v15523: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:03.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:02 smithi022 ceph-mon[40250]: pgmap v15523: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:04 smithi122 ceph-mon[32255]: pgmap v15524: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:04 smithi022 ceph-mon[31360]: pgmap v15524: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:04 smithi022 ceph-mon[40250]: pgmap v15524: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:05.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:05.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:05.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:05.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:06 smithi122 ceph-mon[32255]: pgmap v15525: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/797505557' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:06 smithi022 ceph-mon[31360]: pgmap v15525: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/797505557' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:06 smithi022 ceph-mon[40250]: pgmap v15525: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/797505557' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:36:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:08 smithi022 ceph-mon[31360]: pgmap v15526: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:08 smithi022 ceph-mon[40250]: pgmap v15526: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:08 smithi122 ceph-mon[32255]: pgmap v15526: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:09.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:36:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:36:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:36:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:10 smithi122 ceph-mon[32255]: pgmap v15527: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:10.957 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:10.958 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:10.958 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:10 smithi022 ceph-mon[31360]: pgmap v15527: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:10.959 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:10.959 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:10 smithi022 ceph-mon[40250]: pgmap v15527: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:11.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:11.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:11.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1789271087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1789271087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1789271087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:36:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:36:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:12 smithi122 ceph-mon[32255]: pgmap v15528: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:12 smithi022 ceph-mon[31360]: pgmap v15528: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:12 smithi022 ceph-mon[40250]: pgmap v15528: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:14 smithi122 ceph-mon[32255]: pgmap v15529: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:14 smithi022 ceph-mon[31360]: pgmap v15529: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:14 smithi022 ceph-mon[40250]: pgmap v15529: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:16.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:16.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:16.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:16.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:16.695 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:16 smithi022 ceph-mon[40250]: pgmap v15530: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:16.695 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:16 smithi022 ceph-mon[31360]: pgmap v15530: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:16 smithi122 ceph-mon[32255]: pgmap v15530: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1037214130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:36:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1037214130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1037214130' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:18 smithi122 ceph-mon[32255]: pgmap v15531: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:18 smithi022 ceph-mon[31360]: pgmap v15531: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:18 smithi022 ceph-mon[40250]: pgmap v15531: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:19.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:36:18 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:36:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:20] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:36:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:20 smithi122 ceph-mon[32255]: pgmap v15532: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:20 smithi022 ceph-mon[40250]: pgmap v15532: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:20 smithi022 ceph-mon[31360]: pgmap v15532: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:21.695 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:22.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:22.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:22.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:22 smithi122 ceph-mon[32255]: pgmap v15533: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3201338980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:36:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:36:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:22 smithi022 ceph-mon[31360]: pgmap v15533: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3201338980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:22 smithi022 ceph-mon[40250]: pgmap v15533: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3201338980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:24 smithi122 ceph-mon[32255]: pgmap v15534: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:24 smithi022 ceph-mon[31360]: pgmap v15534: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:24 smithi022 ceph-mon[40250]: pgmap v15534: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:26 smithi122 ceph-mon[32255]: pgmap v15535: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:26 smithi022 ceph-mon[31360]: pgmap v15535: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:26 smithi022 ceph-mon[40250]: pgmap v15535: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:27.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:27.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:27.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:27.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:36:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3021204550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:28 smithi022 ceph-mon[31360]: pgmap v15536: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3021204550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:28 smithi022 ceph-mon[40250]: pgmap v15536: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3021204550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:28 smithi122 ceph-mon[32255]: pgmap v15536: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:29.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:36:28 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:36:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:30] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:36:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:30 smithi122 ceph-mon[32255]: pgmap v15537: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:30 smithi022 ceph-mon[31360]: pgmap v15537: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:30 smithi022 ceph-mon[40250]: pgmap v15537: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:32.442 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:36:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:36:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:32 smithi122 ceph-mon[32255]: pgmap v15538: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:32 smithi022 ceph-mon[31360]: pgmap v15538: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:32 smithi022 ceph-mon[40250]: pgmap v15538: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:32.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:32.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:32.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/387976705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/387976705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/387976705' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:34 smithi122 ceph-mon[32255]: pgmap v15539: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:36:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:36:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:36:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:36:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[31360]: pgmap v15539: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:36:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:36:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:36:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:36:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[40250]: pgmap v15539: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:36:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:36:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:36:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:36:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:36 smithi122 ceph-mon[32255]: pgmap v15540: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:36:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:36:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:36:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:36:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[31360]: pgmap v15540: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:36:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:36:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:36:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:36:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[40250]: pgmap v15540: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:36:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:36:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:36:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:36:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:36:37.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:38.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:38.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:38.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:38 smithi022 ceph-mon[31360]: pgmap v15541: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/314546992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:38 smithi022 ceph-mon[40250]: pgmap v15541: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/314546992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:38 smithi122 ceph-mon[32255]: pgmap v15541: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/314546992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:39.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:36:38 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:36:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:40] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:36:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:40 smithi122 ceph-mon[32255]: pgmap v15542: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:40 smithi022 ceph-mon[31360]: pgmap v15542: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:40 smithi022 ceph-mon[40250]: pgmap v15542: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:42 smithi122 ceph-mon[32255]: pgmap v15543: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:36:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:36:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:42 smithi022 ceph-mon[31360]: pgmap v15543: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:42 smithi022 ceph-mon[40250]: pgmap v15543: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:43.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:43.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:43.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:43.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:44 smithi122 ceph-mon[32255]: pgmap v15544: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1715046612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:44 smithi022 ceph-mon[31360]: pgmap v15544: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1715046612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:44 smithi022 ceph-mon[40250]: pgmap v15544: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1715046612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:46 smithi122 ceph-mon[32255]: pgmap v15545: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:46 smithi022 ceph-mon[31360]: pgmap v15545: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:46 smithi022 ceph-mon[40250]: pgmap v15545: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:36:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:48 smithi022 ceph-mon[31360]: pgmap v15546: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:48 smithi022 ceph-mon[40250]: pgmap v15546: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:48.558 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:48 smithi122 ceph-mon[32255]: pgmap v15546: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:48.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:48.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:48.926 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:49.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:36:48 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:36:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/322029229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/322029229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/322029229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:50] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:36:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:50 smithi122 ceph-mon[32255]: pgmap v15547: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:50 smithi022 ceph-mon[31360]: pgmap v15547: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:36:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:50 smithi022 ceph-mon[40250]: pgmap v15547: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:52 smithi122 ceph-mon[32255]: pgmap v15548: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:36:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:36:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:52 smithi022 ceph-mon[31360]: pgmap v15548: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:52 smithi022 ceph-mon[40250]: pgmap v15548: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:53.927 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:54.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:54.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:54.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:36:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:54 smithi022 ceph-mon[31360]: pgmap v15549: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3233089617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:54 smithi022 ceph-mon[40250]: pgmap v15549: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3233089617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:54 smithi122 ceph-mon[32255]: pgmap v15549: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:36:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3233089617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:36:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:56 smithi122 ceph-mon[32255]: pgmap v15550: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:56 smithi022 ceph-mon[31360]: pgmap v15550: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:56 smithi022 ceph-mon[40250]: pgmap v15550: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:36:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:36:58 smithi022 ceph-mon[31360]: pgmap v15551: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:36:58 smithi022 ceph-mon[40250]: pgmap v15551: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:36:58 smithi122 ceph-mon[32255]: pgmap v15551: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:36:59.299 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:36:59.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:36:58 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:36:59.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:36:59.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:36:59.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:00] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:37:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:00 smithi122 ceph-mon[32255]: pgmap v15552: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2857995801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:00 smithi022 ceph-mon[31360]: pgmap v15552: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2857995801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:00 smithi022 ceph-mon[40250]: pgmap v15552: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2857995801' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:02 smithi122 ceph-mon[32255]: pgmap v15553: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:37:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:37:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:02 smithi022 ceph-mon[31360]: pgmap v15553: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:02 smithi022 ceph-mon[40250]: pgmap v15553: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:04.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:04 smithi122 ceph-mon[32255]: pgmap v15554: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:04 smithi022 ceph-mon[31360]: pgmap v15554: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:04 smithi022 ceph-mon[40250]: pgmap v15554: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:05.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:05.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:05.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4139926448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4139926448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4139926448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:06 smithi122 ceph-mon[32255]: pgmap v15555: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:06 smithi022 ceph-mon[31360]: pgmap v15555: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:06 smithi022 ceph-mon[40250]: pgmap v15555: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:37:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:08 smithi022 ceph-mon[31360]: pgmap v15556: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:08 smithi022 ceph-mon[40250]: pgmap v15556: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:08 smithi122 ceph-mon[32255]: pgmap v15556: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:09.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:37:08 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:37:10.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:10.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:10.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:10.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:10] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:37:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:10 smithi022 ceph-mon[31360]: pgmap v15557: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1440831475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:10 smithi022 ceph-mon[40250]: pgmap v15557: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1440831475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:10 smithi122 ceph-mon[32255]: pgmap v15557: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1440831475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:12 smithi122 ceph-mon[32255]: pgmap v15558: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:37:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:37:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:12 smithi022 ceph-mon[31360]: pgmap v15558: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:12 smithi022 ceph-mon[40250]: pgmap v15558: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:14 smithi122 ceph-mon[32255]: pgmap v15559: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:14 smithi022 ceph-mon[31360]: pgmap v15559: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:14 smithi022 ceph-mon[40250]: pgmap v15559: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:15.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:15.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:15.792 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:15.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:16 smithi122 ceph-mon[32255]: pgmap v15560: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3227149869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:16 smithi022 ceph-mon[31360]: pgmap v15560: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3227149869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:16 smithi022 ceph-mon[40250]: pgmap v15560: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3227149869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:37:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:18 smithi022 ceph-mon[31360]: pgmap v15561: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:18 smithi022 ceph-mon[40250]: pgmap v15561: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:18 smithi122 ceph-mon[32255]: pgmap v15561: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:19.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:37:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:37:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:20] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:37:20.794 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:20 smithi122 ceph-mon[32255]: pgmap v15562: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:20 smithi022 ceph-mon[31360]: pgmap v15562: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:20 smithi022 ceph-mon[40250]: pgmap v15562: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:21.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:21.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:21.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2515667253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2515667253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2515667253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:22 smithi122 ceph-mon[32255]: pgmap v15563: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:37:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:37:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:22 smithi022 ceph-mon[31360]: pgmap v15563: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:22 smithi022 ceph-mon[40250]: pgmap v15563: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:24 smithi122 ceph-mon[32255]: pgmap v15564: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:24 smithi022 ceph-mon[31360]: pgmap v15564: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:24 smithi022 ceph-mon[40250]: pgmap v15564: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:26.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:26.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:26.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:26 smithi122 ceph-mon[32255]: pgmap v15565: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1887402114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:26 smithi022 ceph-mon[31360]: pgmap v15565: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1887402114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:26 smithi022 ceph-mon[40250]: pgmap v15565: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1887402114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:37:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:28 smithi022 ceph-mon[31360]: pgmap v15566: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:28 smithi022 ceph-mon[40250]: pgmap v15566: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:28 smithi122 ceph-mon[32255]: pgmap v15566: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:29.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:37:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:37:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:30] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:37:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:30 smithi122 ceph-mon[32255]: pgmap v15567: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:31.192 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:30 smithi022 ceph-mon[31360]: pgmap v15567: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:31.192 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:31.193 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:30 smithi022 ceph-mon[40250]: pgmap v15567: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:31.538 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:31.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:31.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:31.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:37:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:37:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:32 smithi122 ceph-mon[32255]: pgmap v15568: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2512622259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:32 smithi022 ceph-mon[31360]: pgmap v15568: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2512622259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:32 smithi022 ceph-mon[40250]: pgmap v15568: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2512622259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:34 smithi122 ceph-mon[32255]: pgmap v15569: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:37:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:37:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:37:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:37:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[31360]: pgmap v15569: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:37:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:37:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:37:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:37:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[40250]: pgmap v15569: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:37:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:37:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:37:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:37:36.909 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[31360]: pgmap v15570: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:36.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:37:36.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:37:36.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:37:36.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[40250]: pgmap v15570: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:36.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:37:36.911 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:37:36.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:37:36.912 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:36 smithi122 ceph-mon[32255]: pgmap v15570: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:37:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:37:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:37:37.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:37.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:37.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:37:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/33390550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:38 smithi022 ceph-mon[31360]: pgmap v15571: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/33390550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:38 smithi022 ceph-mon[40250]: pgmap v15571: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/33390550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:38 smithi122 ceph-mon[32255]: pgmap v15571: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:39.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:37:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:37:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:40] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:37:40.741 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:40.742 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:40 smithi022 ceph-mon[31360]: pgmap v15572: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:40.742 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:40.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:40.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:40 smithi022 ceph-mon[40250]: pgmap v15572: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:40.743 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:40 smithi122 ceph-mon[32255]: pgmap v15572: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.605 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.605 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.605 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.606 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.607 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.607 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:42.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:42.508 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:37:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:37:42.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:42.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:42.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:42.820 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:42 smithi122 ceph-mon[32255]: pgmap v15573: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:42.861 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:42 smithi022 ceph-mon[40250]: pgmap v15573: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:42.862 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:42 smithi022 ceph-mon[31360]: pgmap v15573: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/576400344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/576400344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/576400344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:44.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2935M 2021-12-16T02:37:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: pgmap v15574: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2935M 2021-12-16T02:37:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: pgmap v15574: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:45.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2935M 2021-12-16T02:37:45.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: pgmap v15574: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:45.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:45.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:37:45.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:45.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:37:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:45 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3202M 2021-12-16T02:37:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:45 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3202M 2021-12-16T02:37:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:45 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3202M 2021-12-16T02:37:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:46 smithi122 ceph-mon[32255]: pgmap v15575: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:46 smithi022 ceph-mon[31360]: pgmap v15575: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:46 smithi022 ceph-mon[40250]: pgmap v15575: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:47.657 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:37:47.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:48.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:48.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:48.027 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:48 smithi022 ceph-mon[31360]: pgmap v15576: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3257490833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:48 smithi022 ceph-mon[40250]: pgmap v15576: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3257490833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:48 smithi122 ceph-mon[32255]: pgmap v15576: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3257490833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:49.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:37:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:37:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:50] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:37:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:50 smithi122 ceph-mon[32255]: pgmap v15577: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:50 smithi022 ceph-mon[31360]: pgmap v15577: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:37:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:50 smithi022 ceph-mon[40250]: pgmap v15577: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:52 smithi122 ceph-mon[32255]: pgmap v15578: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:37:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:37:53.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:52 smithi022 ceph-mon[31360]: pgmap v15578: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:52 smithi022 ceph-mon[40250]: pgmap v15578: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:53.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:53.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:53.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3414249256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3414249256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3414249256' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:54 smithi122 ceph-mon[32255]: pgmap v15579: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:54 smithi022 ceph-mon[31360]: pgmap v15579: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:54 smithi022 ceph-mon[40250]: pgmap v15579: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:37:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:56 smithi122 ceph-mon[32255]: pgmap v15580: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:56 smithi022 ceph-mon[31360]: pgmap v15580: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:56 smithi022 ceph-mon[40250]: pgmap v15580: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:37:58.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:37:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:58 smithi022 ceph-mon[31360]: pgmap v15581: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:58 smithi022 ceph-mon[40250]: pgmap v15581: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:58 smithi122 ceph-mon[32255]: pgmap v15581: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:37:58.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:37:58.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:37:58.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:37:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:37:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2246755291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:59.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:37:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:37:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:37:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2246755291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:37:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:37:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2246755291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:00] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T02:38:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:00 smithi122 ceph-mon[32255]: pgmap v15582: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:00 smithi022 ceph-mon[31360]: pgmap v15582: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:00 smithi022 ceph-mon[40250]: pgmap v15582: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:02 smithi122 ceph-mon[32255]: pgmap v15583: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:38:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:38:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:02 smithi022 ceph-mon[31360]: pgmap v15583: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:02 smithi022 ceph-mon[40250]: pgmap v15583: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:03.770 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:04.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:04.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:04.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:04 smithi122 ceph-mon[32255]: pgmap v15584: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2536202692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:04 smithi022 ceph-mon[31360]: pgmap v15584: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2536202692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:04 smithi022 ceph-mon[40250]: pgmap v15584: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2536202692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:06 smithi122 ceph-mon[32255]: pgmap v15585: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:06 smithi022 ceph-mon[31360]: pgmap v15585: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:06 smithi022 ceph-mon[40250]: pgmap v15585: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:38:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:08 smithi022 ceph-mon[31360]: pgmap v15586: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:08 smithi022 ceph-mon[40250]: pgmap v15586: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:08 smithi122 ceph-mon[32255]: pgmap v15586: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:09.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:09.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:38:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:38:09.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:09.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:09.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/967402075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:09.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/967402075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/967402075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:10] "GET /metrics HTTP/1.1" 200 202443 "" "Prometheus/2.18.1" 2021-12-16T02:38:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:10 smithi122 ceph-mon[32255]: pgmap v15587: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:10 smithi022 ceph-mon[31360]: pgmap v15587: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:10 smithi022 ceph-mon[40250]: pgmap v15587: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:38:12.568 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:38:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:38:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:12 smithi122 ceph-mon[32255]: pgmap v15588: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:12 smithi022 ceph-mon[31360]: pgmap v15588: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:12 smithi022 ceph-mon[40250]: pgmap v15588: 132 pgs: 132 active+clean; 582 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:14.516 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:14 smithi022 ceph-mon[31360]: pgmap v15589: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:14 smithi022 ceph-mon[40250]: pgmap v15589: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:14.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:14.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:14.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:14 smithi122 ceph-mon[32255]: pgmap v15589: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/928882341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/928882341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/928882341' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:16 smithi122 ceph-mon[32255]: pgmap v15590: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:16 smithi022 ceph-mon[31360]: pgmap v15590: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:16 smithi022 ceph-mon[40250]: pgmap v15590: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:38:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:18 smithi022 ceph-mon[31360]: pgmap v15591: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:18 smithi022 ceph-mon[40250]: pgmap v15591: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:18 smithi122 ceph-mon[32255]: pgmap v15591: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:19.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:38:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:38:19.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:20.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:20.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:20.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:20] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:38:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:20 smithi122 ceph-mon[32255]: pgmap v15592: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1926217565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:20 smithi022 ceph-mon[31360]: pgmap v15592: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1926217565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:20 smithi022 ceph-mon[40250]: pgmap v15592: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1926217565' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:22.577 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:38:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:38:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:22 smithi122 ceph-mon[32255]: pgmap v15593: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:22 smithi022 ceph-mon[31360]: pgmap v15593: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:22 smithi022 ceph-mon[40250]: pgmap v15593: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:38:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:24 smithi122 ceph-mon[32255]: pgmap v15594: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:24 smithi022 ceph-mon[31360]: pgmap v15594: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:24 smithi022 ceph-mon[40250]: pgmap v15594: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:38:25.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:25.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:25.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:25.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:26 smithi122 ceph-mon[32255]: pgmap v15595: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/500863528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:26 smithi022 ceph-mon[31360]: pgmap v15595: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/500863528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:26 smithi022 ceph-mon[40250]: pgmap v15595: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/500863528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:38:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:28 smithi022 ceph-mon[31360]: pgmap v15596: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:28 smithi022 ceph-mon[40250]: pgmap v15596: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:28 smithi122 ceph-mon[32255]: pgmap v15596: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:29.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:38:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:38:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:30] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T02:38:30.625 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:30 smithi122 ceph-mon[32255]: pgmap v15597: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:30.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:30.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:30.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:30.995 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:30.995 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:30 smithi022 ceph-mon[31360]: pgmap v15597: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:30.996 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:30.996 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:30 smithi022 ceph-mon[40250]: pgmap v15597: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3813128715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3813128715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3813128715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:32.580 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:38:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:38:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:32 smithi122 ceph-mon[32255]: pgmap v15598: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:32 smithi022 ceph-mon[31360]: pgmap v15598: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:32 smithi022 ceph-mon[40250]: pgmap v15598: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:34 smithi122 ceph-mon[32255]: pgmap v15599: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:38:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:38:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:38:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:38:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[31360]: pgmap v15599: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:38:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:38:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:38:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:38:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[40250]: pgmap v15599: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:38:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:38:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:38:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:38:35.996 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:36.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:36.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:36.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:36 smithi022 ceph-mon[31360]: pgmap v15600: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1717987839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:36 smithi022 ceph-mon[40250]: pgmap v15600: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1717987839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:36 smithi122 ceph-mon[32255]: pgmap v15600: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1717987839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:38:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:38 smithi022 ceph-mon[31360]: pgmap v15601: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:38 smithi022 ceph-mon[40250]: pgmap v15601: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:38 smithi122 ceph-mon[32255]: pgmap v15601: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:39.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:38:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:38:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:40] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:38:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:40 smithi122 ceph-mon[32255]: pgmap v15602: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:40 smithi022 ceph-mon[31360]: pgmap v15602: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:40 smithi022 ceph-mon[40250]: pgmap v15602: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:41.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:41.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:42.577 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:38:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:38:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:42 smithi122 ceph-mon[32255]: pgmap v15603: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1897633571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:42 smithi022 ceph-mon[31360]: pgmap v15603: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1897633571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:42 smithi022 ceph-mon[40250]: pgmap v15603: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1897633571' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[31360]: pgmap v15604: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:38:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:38:44.787 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:38:44.788 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:38:44.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[40250]: pgmap v15604: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:44.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:38:44.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:38:44.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:38:44.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:38:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:44 smithi122 ceph-mon[32255]: pgmap v15604: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:38:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:38:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:38:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:38:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:46 smithi122 ceph-mon[32255]: pgmap v15605: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:46 smithi022 ceph-mon[31360]: pgmap v15605: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:46 smithi022 ceph-mon[40250]: pgmap v15605: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:47.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:47.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:47.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4292611595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:38:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4292611595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4292611595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:48 smithi122 ceph-mon[32255]: pgmap v15606: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:48 smithi022 ceph-mon[31360]: pgmap v15606: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:48 smithi022 ceph-mon[40250]: pgmap v15606: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:38:49.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:38:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:38:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:50] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:38:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:50 smithi122 ceph-mon[32255]: pgmap v15607: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:50 smithi022 ceph-mon[31360]: pgmap v15607: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:38:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:50 smithi022 ceph-mon[40250]: pgmap v15607: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:52.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:52.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:52.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:52.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:52.590 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:38:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:38:52.590 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:52 smithi122 ceph-mon[32255]: pgmap v15608: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:52 smithi022 ceph-mon[31360]: pgmap v15608: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1930591843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:52 smithi022 ceph-mon[40250]: pgmap v15608: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1930591843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1930591843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:54 smithi122 ceph-mon[32255]: pgmap v15609: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:54 smithi022 ceph-mon[31360]: pgmap v15609: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:54 smithi022 ceph-mon[40250]: pgmap v15609: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:38:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:56 smithi122 ceph-mon[32255]: pgmap v15610: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:56 smithi022 ceph-mon[31360]: pgmap v15610: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:56 smithi022 ceph-mon[40250]: pgmap v15610: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:57.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:38:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:38:57.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:38:57.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:38:57.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:38:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:58 smithi022 ceph-mon[31360]: pgmap v15611: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:38:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/180916031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:58 smithi022 ceph-mon[40250]: pgmap v15611: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:38:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/180916031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:58 smithi122 ceph-mon[32255]: pgmap v15611: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:38:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:38:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/180916031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:38:59.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:38:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:39:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:00] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:39:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:00 smithi122 ceph-mon[32255]: pgmap v15612: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:00 smithi022 ceph-mon[31360]: pgmap v15612: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:00 smithi022 ceph-mon[40250]: pgmap v15612: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:02.586 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:39:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:39:02.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:02.846 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:02 smithi022 ceph-mon[31360]: pgmap v15613: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:02.846 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:02 smithi022 ceph-mon[40250]: pgmap v15613: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:02 smithi122 ceph-mon[32255]: pgmap v15613: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:03.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:03.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:03.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4013071057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:04.293 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4013071057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:04.294 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4013071057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:04 smithi122 ceph-mon[32255]: pgmap v15614: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:04 smithi022 ceph-mon[31360]: pgmap v15614: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:04 smithi022 ceph-mon[40250]: pgmap v15614: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:06 smithi122 ceph-mon[32255]: pgmap v15615: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:06 smithi022 ceph-mon[31360]: pgmap v15615: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:06 smithi022 ceph-mon[40250]: pgmap v15615: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:39:08.220 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:08 smithi022 ceph-mon[31360]: pgmap v15616: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:08 smithi022 ceph-mon[40250]: pgmap v15616: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:08.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:08.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:08.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:08 smithi122 ceph-mon[32255]: pgmap v15616: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:09.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:39:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:39:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1582279212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1582279212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1582279212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:10] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:39:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:10 smithi122 ceph-mon[32255]: pgmap v15617: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:10 smithi022 ceph-mon[31360]: pgmap v15617: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:10 smithi022 ceph-mon[40250]: pgmap v15617: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:12.587 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:39:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:39:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:12 smithi122 ceph-mon[32255]: pgmap v15618: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:12 smithi022 ceph-mon[31360]: pgmap v15618: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:12 smithi022 ceph-mon[40250]: pgmap v15618: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:13.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:13.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:13.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:13.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:14 smithi122 ceph-mon[32255]: pgmap v15619: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1202773417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:14 smithi022 ceph-mon[31360]: pgmap v15619: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1202773417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:14 smithi022 ceph-mon[40250]: pgmap v15619: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1202773417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:16 smithi122 ceph-mon[32255]: pgmap v15620: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:16 smithi022 ceph-mon[31360]: pgmap v15620: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:16 smithi022 ceph-mon[40250]: pgmap v15620: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:39:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:18 smithi022 ceph-mon[31360]: pgmap v15621: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:18 smithi022 ceph-mon[40250]: pgmap v15621: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:18 smithi122 ceph-mon[32255]: pgmap v15621: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:18.966 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:19.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:19.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:19.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:19.371 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:39:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:39:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1555282409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1555282409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1555282409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:20] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T02:39:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:20 smithi122 ceph-mon[32255]: pgmap v15622: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:20 smithi022 ceph-mon[31360]: pgmap v15622: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:20 smithi022 ceph-mon[40250]: pgmap v15622: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:22.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:39:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:39:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:22 smithi122 ceph-mon[32255]: pgmap v15623: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:22 smithi022 ceph-mon[31360]: pgmap v15623: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:22 smithi022 ceph-mon[40250]: pgmap v15623: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:24.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:24.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:24.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:24 smithi122 ceph-mon[32255]: pgmap v15624: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:24 smithi022 ceph-mon[31360]: pgmap v15624: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:24 smithi022 ceph-mon[40250]: pgmap v15624: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3571890220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3571890220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3571890220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:26 smithi122 ceph-mon[32255]: pgmap v15625: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:26 smithi022 ceph-mon[31360]: pgmap v15625: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:26 smithi022 ceph-mon[40250]: pgmap v15625: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:39:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:28 smithi022 ceph-mon[31360]: pgmap v15626: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:28 smithi022 ceph-mon[40250]: pgmap v15626: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:28 smithi122 ceph-mon[32255]: pgmap v15626: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:29.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:39:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:39:29.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:30] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T02:39:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:30 smithi122 ceph-mon[32255]: pgmap v15627: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1802997843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:30 smithi022 ceph-mon[31360]: pgmap v15627: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1802997843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:30 smithi022 ceph-mon[40250]: pgmap v15627: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1802997843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:32.593 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:39:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:39:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:32 smithi122 ceph-mon[32255]: pgmap v15628: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:32 smithi022 ceph-mon[31360]: pgmap v15628: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:32 smithi022 ceph-mon[40250]: pgmap v15628: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:34 smithi122 ceph-mon[32255]: pgmap v15629: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:34.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:39:34.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:39:34.954 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:39:34.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:39:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[31360]: pgmap v15629: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:39:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:39:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:39:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:39:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[40250]: pgmap v15629: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:39:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:39:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:39:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:39:35.071 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:35.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:35.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:35.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2568112315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2568112315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2568112315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:36 smithi122 ceph-mon[32255]: pgmap v15630: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:36 smithi022 ceph-mon[31360]: pgmap v15630: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:36 smithi022 ceph-mon[40250]: pgmap v15630: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:39:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:38 smithi022 ceph-mon[31360]: pgmap v15631: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:38 smithi022 ceph-mon[40250]: pgmap v15631: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:38 smithi122 ceph-mon[32255]: pgmap v15631: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:39.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:39:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:39:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:40] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:39:40.442 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:40 smithi022 ceph-mon[31360]: pgmap v15632: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:40 smithi022 ceph-mon[40250]: pgmap v15632: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:40.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:40.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:40.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:40 smithi122 ceph-mon[32255]: pgmap v15632: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2672016484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2672016484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2672016484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:42.601 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:39:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:39:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:42 smithi122 ceph-mon[32255]: pgmap v15633: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:42 smithi022 ceph-mon[31360]: pgmap v15633: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:42 smithi022 ceph-mon[40250]: pgmap v15633: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:44 smithi122 ceph-mon[32255]: pgmap v15634: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:39:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:39:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:44 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:39:44.984 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[40250]: pgmap v15634: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:44.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:39:44.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:39:44.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:39:44.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[31360]: pgmap v15634: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:44.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:39:44.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:39:44.986 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:44 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:39:45.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:46.186 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:39:46.186 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:45 smithi022 ceph-mon[40250]: pgmap v15635: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:46.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:39:46.187 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:45 smithi022 ceph-mon[31360]: pgmap v15635: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:39:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:45 smithi122 ceph-mon[32255]: pgmap v15635: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3402989905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3402989905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3402989905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:39:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:48 smithi022 ceph-mon[31360]: pgmap v15636: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:48 smithi022 ceph-mon[40250]: pgmap v15636: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:48 smithi122 ceph-mon[32255]: pgmap v15636: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:49.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:39:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:39:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:50] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T02:39:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:50 smithi122 ceph-mon[32255]: pgmap v15637: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:50 smithi022 ceph-mon[31360]: pgmap v15637: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:39:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:50 smithi022 ceph-mon[40250]: pgmap v15637: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:51.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:51.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:51.553 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:51.554 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3977645044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3977645044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3977645044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:52.602 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:39:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:39:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:52 smithi122 ceph-mon[32255]: pgmap v15638: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:52 smithi022 ceph-mon[31360]: pgmap v15638: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:52 smithi022 ceph-mon[40250]: pgmap v15638: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:54 smithi122 ceph-mon[32255]: pgmap v15639: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:54 smithi022 ceph-mon[31360]: pgmap v15639: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:54 smithi022 ceph-mon[40250]: pgmap v15639: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:39:56.555 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:39:56.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:39:56.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:39:56.927 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:39:56.928 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:56 smithi022 ceph-mon[31360]: pgmap v15640: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:56.928 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:56 smithi022 ceph-mon[40250]: pgmap v15640: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:56 smithi122 ceph-mon[32255]: pgmap v15640: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1877054089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1877054089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:39:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1877054089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:39:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:39:58 smithi122 ceph-mon[32255]: pgmap v15641: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:39:58 smithi022 ceph-mon[31360]: pgmap v15641: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:39:58 smithi022 ceph-mon[40250]: pgmap v15641: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:39:59.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:39:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:40:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:00] "GET /metrics HTTP/1.1" 200 202442 "" "Prometheus/2.18.1" 2021-12-16T02:40:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:00 smithi122 ceph-mon[32255]: pgmap v15642: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T02:40:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:00 smithi022 ceph-mon[31360]: pgmap v15642: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T02:40:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:00 smithi022 ceph-mon[40250]: pgmap v15642: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T02:40:01.930 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:02.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:02.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:02.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:02.616 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:40:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:40:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:02 smithi122 ceph-mon[32255]: pgmap v15643: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1763839544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:02 smithi022 ceph-mon[31360]: pgmap v15643: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1763839544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:02 smithi022 ceph-mon[40250]: pgmap v15643: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1763839544' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:04 smithi122 ceph-mon[32255]: pgmap v15644: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:04 smithi022 ceph-mon[31360]: pgmap v15644: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:04 smithi022 ceph-mon[40250]: pgmap v15644: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:06 smithi122 ceph-mon[32255]: pgmap v15645: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:06 smithi022 ceph-mon[31360]: pgmap v15645: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:06 smithi022 ceph-mon[40250]: pgmap v15645: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:07.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:07.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:07.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:07.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:07.667 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:40:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:08 smithi022 ceph-mon[31360]: pgmap v15646: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2488027551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:08 smithi022 ceph-mon[40250]: pgmap v15646: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2488027551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:08 smithi122 ceph-mon[32255]: pgmap v15646: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2488027551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:09.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:40:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:40:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:10] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:40:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:10 smithi122 ceph-mon[32255]: pgmap v15647: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:10 smithi022 ceph-mon[31360]: pgmap v15647: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:10 smithi022 ceph-mon[40250]: pgmap v15647: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:12.608 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:40:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:40:12.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:12 smithi122 ceph-mon[32255]: pgmap v15648: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:12 smithi022 ceph-mon[31360]: pgmap v15648: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:12 smithi022 ceph-mon[40250]: pgmap v15648: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:13.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:13.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:13.035 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3131156215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3131156215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3131156215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:14 smithi122 ceph-mon[32255]: pgmap v15649: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:14 smithi022 ceph-mon[31360]: pgmap v15649: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:14 smithi022 ceph-mon[40250]: pgmap v15649: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:16 smithi122 ceph-mon[32255]: pgmap v15650: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:16 smithi022 ceph-mon[31360]: pgmap v15650: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:16 smithi022 ceph-mon[40250]: pgmap v15650: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:40:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:18.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:18.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:18.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:18.410 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:18 smithi022 ceph-mon[31360]: pgmap v15651: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:18.410 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:18 smithi022 ceph-mon[40250]: pgmap v15651: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:18 smithi122 ceph-mon[32255]: pgmap v15651: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3426240689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:19.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:40:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:40:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3426240689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3426240689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:20] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:40:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:20 smithi122 ceph-mon[32255]: pgmap v15652: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:20 smithi022 ceph-mon[31360]: pgmap v15652: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:20 smithi022 ceph-mon[40250]: pgmap v15652: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:22.608 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:40:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:40:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:22 smithi122 ceph-mon[32255]: pgmap v15653: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:22 smithi022 ceph-mon[31360]: pgmap v15653: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:22 smithi022 ceph-mon[40250]: pgmap v15653: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:23.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:23.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:23.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:23.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:24 smithi122 ceph-mon[32255]: pgmap v15654: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2224686440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:24 smithi022 ceph-mon[31360]: pgmap v15654: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2224686440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:24 smithi022 ceph-mon[40250]: pgmap v15654: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2224686440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:26 smithi122 ceph-mon[32255]: pgmap v15655: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:26 smithi022 ceph-mon[31360]: pgmap v15655: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:26 smithi022 ceph-mon[40250]: pgmap v15655: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:40:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:28 smithi022 ceph-mon[31360]: pgmap v15656: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:28 smithi022 ceph-mon[40250]: pgmap v15656: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:28 smithi122 ceph-mon[32255]: pgmap v15656: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:28.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:29.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:29.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:29.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/238620172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:29.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:40:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:40:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/238620172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/238620172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:30] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:40:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:30 smithi122 ceph-mon[32255]: pgmap v15657: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:30 smithi022 ceph-mon[31360]: pgmap v15657: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:30 smithi022 ceph-mon[40250]: pgmap v15657: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:32.610 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:40:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:40:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:32 smithi122 ceph-mon[32255]: pgmap v15658: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:32 smithi022 ceph-mon[31360]: pgmap v15658: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:32 smithi022 ceph-mon[40250]: pgmap v15658: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:34.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:34.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:34.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:34.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[40250]: pgmap v15659: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:40:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:40:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:40:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:40:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/989058964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[31360]: pgmap v15659: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:40:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:40:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:40:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:40:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/989058964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:34 smithi122 ceph-mon[32255]: pgmap v15659: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:40:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:40:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:40:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:40:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/989058964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:36 smithi122 ceph-mon[32255]: pgmap v15660: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:37.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:36 smithi022 ceph-mon[31360]: pgmap v15660: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:37.072 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:36 smithi022 ceph-mon[40250]: pgmap v15660: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:40:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:38 smithi022 ceph-mon[31360]: pgmap v15661: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:38 smithi022 ceph-mon[40250]: pgmap v15661: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:38 smithi122 ceph-mon[32255]: pgmap v15661: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:39.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:40:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:40:39.523 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:39.893 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:39.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:39.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:40] "GET /metrics HTTP/1.1" 200 202447 "" "Prometheus/2.18.1" 2021-12-16T02:40:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:40 smithi122 ceph-mon[32255]: pgmap v15662: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1585156698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:40 smithi022 ceph-mon[31360]: pgmap v15662: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1585156698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:40 smithi022 ceph-mon[40250]: pgmap v15662: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1585156698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:42.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:40:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:40:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:42 smithi122 ceph-mon[32255]: pgmap v15663: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:42 smithi022 ceph-mon[31360]: pgmap v15663: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:42 smithi022 ceph-mon[40250]: pgmap v15663: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:44.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:44.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:44 smithi022 ceph-mon[31360]: pgmap v15664: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:44.897 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:44 smithi022 ceph-mon[40250]: pgmap v15664: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:44 smithi122 ceph-mon[32255]: pgmap v15664: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:45.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:45.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:45.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:40:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:40:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:40:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/250530106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:40:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:40:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:40:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:40:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/250530106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:40:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:40:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:40:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:40:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/250530106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:40:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:46 smithi122 ceph-mon[32255]: pgmap v15665: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:46 smithi022 ceph-mon[31360]: pgmap v15665: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:46 smithi022 ceph-mon[40250]: pgmap v15665: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:40:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:48 smithi022 ceph-mon[31360]: pgmap v15666: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:48 smithi022 ceph-mon[40250]: pgmap v15666: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:48 smithi122 ceph-mon[32255]: pgmap v15666: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:49.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:40:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:40:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:50] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:40:50.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:50.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:50.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:50 smithi122 ceph-mon[32255]: pgmap v15667: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:50 smithi022 ceph-mon[31360]: pgmap v15667: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:40:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:50 smithi022 ceph-mon[40250]: pgmap v15667: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4009668400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4009668400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4009668400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:52.621 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:40:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:40:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:52 smithi122 ceph-mon[32255]: pgmap v15668: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:52 smithi022 ceph-mon[31360]: pgmap v15668: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:52 smithi022 ceph-mon[40250]: pgmap v15668: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:54 smithi122 ceph-mon[32255]: pgmap v15669: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:54 smithi022 ceph-mon[31360]: pgmap v15669: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:54 smithi022 ceph-mon[40250]: pgmap v15669: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:40:55.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:40:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:56 smithi122 ceph-mon[32255]: pgmap v15670: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1776327729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:56 smithi022 ceph-mon[31360]: pgmap v15670: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1776327729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:56 smithi022 ceph-mon[40250]: pgmap v15670: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1776327729' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:40:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:40:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:40:58 smithi122 ceph-mon[32255]: pgmap v15671: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:40:58 smithi022 ceph-mon[31360]: pgmap v15671: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:40:58 smithi022 ceph-mon[40250]: pgmap v15671: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:40:59.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:40:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:41:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:00] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:41:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:00 smithi122 ceph-mon[32255]: pgmap v15672: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:01.015 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:01.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:01.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:00 smithi022 ceph-mon[40250]: pgmap v15672: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:01.016 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:01.017 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:00 smithi022 ceph-mon[31360]: pgmap v15672: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:01.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3999578708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3999578708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3999578708' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:02.315 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:41:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:41:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:02 smithi122 ceph-mon[32255]: pgmap v15673: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:02 smithi022 ceph-mon[31360]: pgmap v15673: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:02 smithi022 ceph-mon[40250]: pgmap v15673: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:04 smithi122 ceph-mon[32255]: pgmap v15674: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:04 smithi022 ceph-mon[31360]: pgmap v15674: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:04 smithi022 ceph-mon[40250]: pgmap v15674: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:06.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:06.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:06.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:06.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:06.759 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:06 smithi022 ceph-mon[40250]: pgmap v15675: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:06.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:06 smithi022 ceph-mon[31360]: pgmap v15675: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:06 smithi122 ceph-mon[32255]: pgmap v15675: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:41:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3942608082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3942608082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3942608082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:08 smithi122 ceph-mon[32255]: pgmap v15676: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:08 smithi022 ceph-mon[31360]: pgmap v15676: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:08 smithi022 ceph-mon[40250]: pgmap v15676: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:09.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:41:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:41:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:10] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:41:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:10 smithi122 ceph-mon[32255]: pgmap v15677: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:10 smithi022 ceph-mon[31360]: pgmap v15677: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:10 smithi022 ceph-mon[40250]: pgmap v15677: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:11.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:12.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:12.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:12.128 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:12.637 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:41:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:41:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:12 smithi122 ceph-mon[32255]: pgmap v15678: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/338339405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:12 smithi022 ceph-mon[31360]: pgmap v15678: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/338339405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:12 smithi022 ceph-mon[40250]: pgmap v15678: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/338339405' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:14 smithi122 ceph-mon[32255]: pgmap v15679: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:14 smithi022 ceph-mon[31360]: pgmap v15679: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:14 smithi022 ceph-mon[40250]: pgmap v15679: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:16 smithi122 ceph-mon[32255]: pgmap v15680: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:16 smithi022 ceph-mon[31360]: pgmap v15680: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:16 smithi022 ceph-mon[40250]: pgmap v15680: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:17.130 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:17.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:17.499 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:17.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:41:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1320673890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:18 smithi122 ceph-mon[32255]: pgmap v15681: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1320673890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:18 smithi022 ceph-mon[31360]: pgmap v15681: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1320673890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:18 smithi022 ceph-mon[40250]: pgmap v15681: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:19.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:41:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:41:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:20] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:41:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:20 smithi122 ceph-mon[32255]: pgmap v15682: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:20 smithi022 ceph-mon[31360]: pgmap v15682: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:20 smithi022 ceph-mon[40250]: pgmap v15682: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:22.626 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:41:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:41:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:22 smithi022 ceph-mon[31360]: pgmap v15683: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:22 smithi022 ceph-mon[40250]: pgmap v15683: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:22.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:22.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:22.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:22 smithi122 ceph-mon[32255]: pgmap v15683: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/711523401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/711523401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/711523401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:24 smithi122 ceph-mon[32255]: pgmap v15684: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:24 smithi022 ceph-mon[31360]: pgmap v15684: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:24 smithi022 ceph-mon[40250]: pgmap v15684: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:26 smithi122 ceph-mon[32255]: pgmap v15685: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:26 smithi022 ceph-mon[31360]: pgmap v15685: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:26 smithi022 ceph-mon[40250]: pgmap v15685: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:41:27.873 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:28.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:28.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:28.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:28 smithi022 ceph-mon[31360]: pgmap v15686: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3294199987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:28 smithi022 ceph-mon[40250]: pgmap v15686: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3294199987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:28 smithi122 ceph-mon[32255]: pgmap v15686: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3294199987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:29.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:41:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:41:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:30] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:41:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:30 smithi122 ceph-mon[32255]: pgmap v15687: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:30 smithi022 ceph-mon[31360]: pgmap v15687: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:30 smithi022 ceph-mon[40250]: pgmap v15687: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:32.636 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:41:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:41:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:32 smithi122 ceph-mon[32255]: pgmap v15688: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:32 smithi022 ceph-mon[31360]: pgmap v15688: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:32 smithi022 ceph-mon[40250]: pgmap v15688: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:33.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:33.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:33.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:33.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2136135860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2136135860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2136135860' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:34 smithi122 ceph-mon[32255]: pgmap v15689: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:41:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:41:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:41:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:41:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[31360]: pgmap v15689: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:41:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:41:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:41:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:41:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[40250]: pgmap v15689: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:41:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:41:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:41:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:41:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:36 smithi122 ceph-mon[32255]: pgmap v15690: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:36 smithi022 ceph-mon[31360]: pgmap v15690: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:36 smithi022 ceph-mon[40250]: pgmap v15690: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:41:38.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:38.618 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:38 smithi022 ceph-mon[31360]: pgmap v15691: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:38.619 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:38 smithi022 ceph-mon[40250]: pgmap v15691: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:38 smithi122 ceph-mon[32255]: pgmap v15691: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:38.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:38.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:38.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:39.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:41:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:41:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3315233582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3315233582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3315233582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:40] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:41:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:40 smithi122 ceph-mon[32255]: pgmap v15692: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:40 smithi022 ceph-mon[31360]: pgmap v15692: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:40 smithi022 ceph-mon[40250]: pgmap v15692: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:42.630 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:41:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:41:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:42 smithi122 ceph-mon[32255]: pgmap v15693: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:42 smithi022 ceph-mon[31360]: pgmap v15693: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:42 smithi022 ceph-mon[40250]: pgmap v15693: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:43.987 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:44.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:44.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:44.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:44 smithi122 ceph-mon[32255]: pgmap v15694: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2409194683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:44 smithi022 ceph-mon[31360]: pgmap v15694: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2409194683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:44 smithi022 ceph-mon[40250]: pgmap v15694: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2409194683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:41:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:41:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:45 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:41:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:41:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:41:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:45 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:41:45.902 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:41:45.902 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:41:45.903 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:45 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:41:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:46 smithi122 ceph-mon[32255]: pgmap v15695: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:41:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:46 smithi022 ceph-mon[31360]: pgmap v15695: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:41:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:46 smithi022 ceph-mon[40250]: pgmap v15695: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:41:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:41:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:48 smithi122 ceph-mon[32255]: pgmap v15696: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:48 smithi022 ceph-mon[31360]: pgmap v15696: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:48 smithi022 ceph-mon[40250]: pgmap v15696: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:49.356 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:49.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:41:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:41:49.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:49.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:49.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:50] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:41:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:50 smithi122 ceph-mon[32255]: pgmap v15697: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/504651625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:50 smithi022 ceph-mon[31360]: pgmap v15697: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/504651625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:41:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:50 smithi022 ceph-mon[40250]: pgmap v15697: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/504651625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:52.635 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:41:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:41:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:52 smithi122 ceph-mon[32255]: pgmap v15698: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:52 smithi022 ceph-mon[31360]: pgmap v15698: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:52 smithi022 ceph-mon[40250]: pgmap v15698: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:54.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:41:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:54 smithi122 ceph-mon[32255]: pgmap v15699: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:54 smithi022 ceph-mon[31360]: pgmap v15699: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:55.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:54 smithi022 ceph-mon[40250]: pgmap v15699: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:41:55.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:41:55.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:41:55.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:41:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3077439841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3077439841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3077439841' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:41:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:56 smithi122 ceph-mon[32255]: pgmap v15700: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:56 smithi022 ceph-mon[31360]: pgmap v15700: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:56 smithi022 ceph-mon[40250]: pgmap v15700: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:41:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:41:58 smithi122 ceph-mon[32255]: pgmap v15701: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:41:58 smithi022 ceph-mon[31360]: pgmap v15701: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:41:58 smithi022 ceph-mon[40250]: pgmap v15701: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:41:59.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:41:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:42:00.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:00] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:42:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:00 smithi022 ceph-mon[31360]: pgmap v15702: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2106511128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:00 smithi022 ceph-mon[40250]: pgmap v15702: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2106511128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:00 smithi122 ceph-mon[32255]: pgmap v15702: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2106511128' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:02.639 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:42:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:42:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:02 smithi122 ceph-mon[32255]: pgmap v15703: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:02 smithi022 ceph-mon[31360]: pgmap v15703: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:02 smithi022 ceph-mon[40250]: pgmap v15703: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:04 smithi122 ceph-mon[32255]: pgmap v15704: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:04 smithi022 ceph-mon[31360]: pgmap v15704: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:04 smithi022 ceph-mon[40250]: pgmap v15704: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:05.468 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:05.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:05.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:05.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:06 smithi122 ceph-mon[32255]: pgmap v15705: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2281480773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:06 smithi022 ceph-mon[31360]: pgmap v15705: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2281480773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:06 smithi022 ceph-mon[40250]: pgmap v15705: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2281480773' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:42:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:08 smithi122 ceph-mon[32255]: pgmap v15706: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:08 smithi022 ceph-mon[31360]: pgmap v15706: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:08 smithi022 ceph-mon[40250]: pgmap v15706: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:09.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:42:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:42:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:10] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:42:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:10 smithi122 ceph-mon[32255]: pgmap v15707: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:11.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:10 smithi022 ceph-mon[40250]: pgmap v15707: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:10 smithi022 ceph-mon[31360]: pgmap v15707: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:11.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:11.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:11.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/978824964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/978824964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/978824964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:12.645 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:42:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:42:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:12 smithi122 ceph-mon[32255]: pgmap v15708: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:12 smithi022 ceph-mon[31360]: pgmap v15708: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:12 smithi022 ceph-mon[40250]: pgmap v15708: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:14 smithi122 ceph-mon[32255]: pgmap v15709: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:14 smithi022 ceph-mon[31360]: pgmap v15709: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:14 smithi022 ceph-mon[40250]: pgmap v15709: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:16.210 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:16.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:16.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:16.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:16 smithi122 ceph-mon[32255]: pgmap v15710: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3021545620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:17.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:16 smithi022 ceph-mon[40250]: pgmap v15710: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3021545620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:16 smithi022 ceph-mon[31360]: pgmap v15710: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3021545620' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:42:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:18 smithi122 ceph-mon[32255]: pgmap v15711: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:18 smithi022 ceph-mon[31360]: pgmap v15711: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:18 smithi022 ceph-mon[40250]: pgmap v15711: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:19.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:42:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:42:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:20] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:42:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:20 smithi122 ceph-mon[32255]: pgmap v15712: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:20 smithi022 ceph-mon[40250]: pgmap v15712: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:20 smithi022 ceph-mon[31360]: pgmap v15712: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:21.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:21.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:21.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:21.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:22.644 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:42:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:42:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:22 smithi122 ceph-mon[32255]: pgmap v15713: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2299467800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:22 smithi022 ceph-mon[31360]: pgmap v15713: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2299467800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:22 smithi022 ceph-mon[40250]: pgmap v15713: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2299467800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:24 smithi122 ceph-mon[32255]: pgmap v15714: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:24 smithi022 ceph-mon[31360]: pgmap v15714: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:25.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:24 smithi022 ceph-mon[40250]: pgmap v15714: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:26 smithi122 ceph-mon[32255]: pgmap v15715: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:26.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:26.947 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:26 smithi022 ceph-mon[40250]: pgmap v15715: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:26.947 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:26 smithi022 ceph-mon[31360]: pgmap v15715: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:27.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:42:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2742833225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:28 smithi122 ceph-mon[32255]: pgmap v15716: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2742833225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:28 smithi022 ceph-mon[31360]: pgmap v15716: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2742833225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:28 smithi022 ceph-mon[40250]: pgmap v15716: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:29.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:42:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:42:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:30] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:42:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:30 smithi122 ceph-mon[32255]: pgmap v15717: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:30 smithi022 ceph-mon[31360]: pgmap v15717: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:31.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:30 smithi022 ceph-mon[40250]: pgmap v15717: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:32.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:32.650 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:42:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:42:32.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:32.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:32.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:32 smithi122 ceph-mon[32255]: pgmap v15718: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:32 smithi022 ceph-mon[31360]: pgmap v15718: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:32 smithi022 ceph-mon[40250]: pgmap v15718: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221548508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221548508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221548508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:34 smithi122 ceph-mon[32255]: pgmap v15719: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:42:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:42:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[31360]: pgmap v15719: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:42:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:42:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:42:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:42:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[40250]: pgmap v15719: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:42:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:42:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:36 smithi122 ceph-mon[32255]: pgmap v15720: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:36 smithi022 ceph-mon[31360]: pgmap v15720: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:36 smithi022 ceph-mon[40250]: pgmap v15720: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:37.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:37.685 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:42:38.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:38.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:38.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:38 smithi022 ceph-mon[31360]: pgmap v15721: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2437069388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:38 smithi022 ceph-mon[40250]: pgmap v15721: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2437069388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:38 smithi122 ceph-mon[32255]: pgmap v15721: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2437069388' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:39.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:42:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:42:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:40] "GET /metrics HTTP/1.1" 200 202445 "" "Prometheus/2.18.1" 2021-12-16T02:42:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:40 smithi122 ceph-mon[32255]: pgmap v15722: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:40 smithi022 ceph-mon[31360]: pgmap v15722: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:40 smithi022 ceph-mon[40250]: pgmap v15722: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:42.653 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:42:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:42:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:42 smithi122 ceph-mon[32255]: pgmap v15723: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:42 smithi022 ceph-mon[31360]: pgmap v15723: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:42 smithi022 ceph-mon[40250]: pgmap v15723: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:43.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:43.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:43.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:43.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3561408044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3561408044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3561408044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:44 smithi122 ceph-mon[32255]: pgmap v15724: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:44 smithi022 ceph-mon[31360]: pgmap v15724: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:44 smithi022 ceph-mon[40250]: pgmap v15724: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:46 smithi122 ceph-mon[32255]: pgmap v15725: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:42:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:42:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:42:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:42:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[31360]: pgmap v15725: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:42:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:42:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:42:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:42:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[40250]: pgmap v15725: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:42:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:42:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:42:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:42:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:42:48.425 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:48 smithi122 ceph-mon[32255]: pgmap v15726: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:48 smithi022 ceph-mon[40250]: pgmap v15726: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:48 smithi022 ceph-mon[31360]: pgmap v15726: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:48.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:48.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:48.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2628969069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:42:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:42:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2628969069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2628969069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:50 smithi122 ceph-mon[32255]: pgmap v15727: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:50] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:42:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:50 smithi022 ceph-mon[31360]: pgmap v15727: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:42:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:50 smithi022 ceph-mon[40250]: pgmap v15727: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:52.652 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:42:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:42:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:52 smithi122 ceph-mon[32255]: pgmap v15728: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:52 smithi022 ceph-mon[31360]: pgmap v15728: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:52 smithi022 ceph-mon[40250]: pgmap v15728: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:53.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:54.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:54.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:54.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:54 smithi122 ceph-mon[32255]: pgmap v15729: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2076300361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:54 smithi022 ceph-mon[31360]: pgmap v15729: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2076300361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:54 smithi022 ceph-mon[40250]: pgmap v15729: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:42:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2076300361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:42:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:56 smithi122 ceph-mon[32255]: pgmap v15730: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:56 smithi022 ceph-mon[31360]: pgmap v15730: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:56 smithi022 ceph-mon[40250]: pgmap v15730: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:42:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:58 smithi122 ceph-mon[32255]: pgmap v15731: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:58 smithi022 ceph-mon[31360]: pgmap v15731: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:58 smithi022 ceph-mon[40250]: pgmap v15731: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:42:59.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:42:59.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:42:59.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:42:59.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:42:59.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:42:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:42:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:42:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/788703344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:42:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/788703344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:42:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/788703344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:00] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:43:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:00 smithi122 ceph-mon[32255]: pgmap v15732: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:00 smithi022 ceph-mon[31360]: pgmap v15732: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:00 smithi022 ceph-mon[40250]: pgmap v15732: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:02.655 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:43:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:43:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:02 smithi122 ceph-mon[32255]: pgmap v15733: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:02 smithi022 ceph-mon[31360]: pgmap v15733: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:02 smithi022 ceph-mon[40250]: pgmap v15733: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:04.535 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:04.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:04.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:04.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:04.904 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:04 smithi022 ceph-mon[40250]: pgmap v15734: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:04.905 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:04 smithi022 ceph-mon[31360]: pgmap v15734: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:04 smithi122 ceph-mon[32255]: pgmap v15734: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3095257007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3095257007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3095257007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:06 smithi122 ceph-mon[32255]: pgmap v15735: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:06 smithi022 ceph-mon[31360]: pgmap v15735: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:06 smithi022 ceph-mon[40250]: pgmap v15735: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:43:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:08 smithi122 ceph-mon[32255]: pgmap v15736: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:08 smithi022 ceph-mon[31360]: pgmap v15736: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:08 smithi022 ceph-mon[40250]: pgmap v15736: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:09.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:43:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:43:09.905 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:10.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:10.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:10] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:43:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:10 smithi122 ceph-mon[32255]: pgmap v15737: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1066728091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:10 smithi022 ceph-mon[31360]: pgmap v15737: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1066728091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:10 smithi022 ceph-mon[40250]: pgmap v15737: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1066728091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:12.660 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:43:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:43:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:12 smithi122 ceph-mon[32255]: pgmap v15738: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:12 smithi022 ceph-mon[31360]: pgmap v15738: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:12 smithi022 ceph-mon[40250]: pgmap v15738: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:14 smithi122 ceph-mon[32255]: pgmap v15739: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:14 smithi022 ceph-mon[31360]: pgmap v15739: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:14 smithi022 ceph-mon[40250]: pgmap v15739: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:15.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:15.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:15.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:15.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/377642876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/377642876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/377642876' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:16 smithi122 ceph-mon[32255]: pgmap v15740: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:16 smithi022 ceph-mon[31360]: pgmap v15740: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:16 smithi022 ceph-mon[40250]: pgmap v15740: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:43:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:18 smithi122 ceph-mon[32255]: pgmap v15741: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:18 smithi022 ceph-mon[31360]: pgmap v15741: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:18 smithi022 ceph-mon[40250]: pgmap v15741: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:19.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:43:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:43:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:20] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:43:20.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:20 smithi122 ceph-mon[32255]: pgmap v15742: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:21.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:21.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:21.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:21.018 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:21.019 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:20 smithi022 ceph-mon[40250]: pgmap v15742: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:21.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:21.019 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:20 smithi022 ceph-mon[31360]: pgmap v15742: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/816645387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/816645387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/816645387' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:22.667 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:43:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:43:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:22 smithi122 ceph-mon[32255]: pgmap v15743: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:22 smithi022 ceph-mon[31360]: pgmap v15743: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:22 smithi022 ceph-mon[40250]: pgmap v15743: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:24 smithi122 ceph-mon[32255]: pgmap v15744: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:24 smithi022 ceph-mon[31360]: pgmap v15744: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:24 smithi022 ceph-mon[40250]: pgmap v15744: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:26.019 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:26.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:26.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:26.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:26 smithi122 ceph-mon[32255]: pgmap v15745: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3904264726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:26 smithi022 ceph-mon[31360]: pgmap v15745: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3904264726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:26 smithi022 ceph-mon[40250]: pgmap v15745: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3904264726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:43:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:28 smithi122 ceph-mon[32255]: pgmap v15746: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:28 smithi022 ceph-mon[31360]: pgmap v15746: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:28 smithi022 ceph-mon[40250]: pgmap v15746: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:29.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:43:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:43:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:30] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:43:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:30 smithi122 ceph-mon[32255]: pgmap v15747: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:30 smithi022 ceph-mon[31360]: pgmap v15747: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:30 smithi022 ceph-mon[40250]: pgmap v15747: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:31.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:31.757 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:31.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:31.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:32.669 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:43:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:43:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:32 smithi122 ceph-mon[32255]: pgmap v15748: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1378548861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:32 smithi022 ceph-mon[31360]: pgmap v15748: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1378548861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:32 smithi022 ceph-mon[40250]: pgmap v15748: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1378548861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:34 smithi122 ceph-mon[32255]: pgmap v15749: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:43:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:43:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:43:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:43:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[31360]: pgmap v15749: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:43:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:43:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:43:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:43:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[40250]: pgmap v15749: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:43:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:43:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:43:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:43:36.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:36 smithi122 ceph-mon[32255]: pgmap v15750: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:36 smithi022 ceph-mon[31360]: pgmap v15750: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:36 smithi022 ceph-mon[40250]: pgmap v15750: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:37.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:37.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:37.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:37.674 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:43:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2097383942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2097383942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2097383942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:38 smithi122 ceph-mon[32255]: pgmap v15751: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:38 smithi022 ceph-mon[31360]: pgmap v15751: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:38 smithi022 ceph-mon[40250]: pgmap v15751: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:39.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:43:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:43:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:40] "GET /metrics HTTP/1.1" 200 202444 "" "Prometheus/2.18.1" 2021-12-16T02:43:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:40 smithi122 ceph-mon[32255]: pgmap v15752: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:40 smithi022 ceph-mon[31360]: pgmap v15752: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:40 smithi022 ceph-mon[40250]: pgmap v15752: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:42.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:42.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:42.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:42.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:42.683 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:43:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:43:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:42 smithi022 ceph-mon[31360]: pgmap v15753: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/491844661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:42 smithi022 ceph-mon[40250]: pgmap v15753: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/491844661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:42 smithi122 ceph-mon[32255]: pgmap v15753: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/491844661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:44 smithi122 ceph-mon[32255]: pgmap v15754: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:44 smithi022 ceph-mon[31360]: pgmap v15754: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:44 smithi022 ceph-mon[40250]: pgmap v15754: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[31360]: pgmap v15755: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:43:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:43:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:43:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[40250]: pgmap v15755: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:43:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:43:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:43:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:46 smithi122 ceph-mon[32255]: pgmap v15755: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:43:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:43:47.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:43:47.503 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:47.504 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:43:47.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:47.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:47.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:43:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:48 smithi122 ceph-mon[32255]: pgmap v15756: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3089810168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:48 smithi022 ceph-mon[31360]: pgmap v15756: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3089810168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:48 smithi022 ceph-mon[40250]: pgmap v15756: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:49.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3089810168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:49.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:43:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:43:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:50] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:43:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:50 smithi122 ceph-mon[32255]: pgmap v15757: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:50 smithi022 ceph-mon[31360]: pgmap v15757: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:43:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:50 smithi022 ceph-mon[40250]: pgmap v15757: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:43:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:43:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:52 smithi122 ceph-mon[32255]: pgmap v15758: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:53.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:53.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:53.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:53.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:52 smithi022 ceph-mon[40250]: pgmap v15758: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:53.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:52 smithi022 ceph-mon[31360]: pgmap v15758: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3813592284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3813592284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3813592284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:54 smithi122 ceph-mon[32255]: pgmap v15759: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:54 smithi022 ceph-mon[31360]: pgmap v15759: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:54 smithi022 ceph-mon[40250]: pgmap v15759: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:43:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:56 smithi122 ceph-mon[32255]: pgmap v15760: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:56 smithi022 ceph-mon[31360]: pgmap v15760: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:56 smithi022 ceph-mon[40250]: pgmap v15760: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:43:58.238 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:43:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:58 smithi022 ceph-mon[31360]: pgmap v15761: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:58 smithi022 ceph-mon[40250]: pgmap v15761: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:43:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:43:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:43:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:58 smithi122 ceph-mon[32255]: pgmap v15761: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:43:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:43:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/708264286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:59.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:43:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:43:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:43:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/708264286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:43:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:43:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/708264286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:00] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:44:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:00 smithi122 ceph-mon[32255]: pgmap v15762: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:00 smithi022 ceph-mon[31360]: pgmap v15762: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:00 smithi022 ceph-mon[40250]: pgmap v15762: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:02.674 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:44:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:44:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:02 smithi122 ceph-mon[32255]: pgmap v15763: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:02 smithi022 ceph-mon[31360]: pgmap v15763: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:02 smithi022 ceph-mon[40250]: pgmap v15763: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:03.607 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:03.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:03.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:03.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:04 smithi122 ceph-mon[32255]: pgmap v15764: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4128880320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:04 smithi022 ceph-mon[31360]: pgmap v15764: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:05.040 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4128880320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:05.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:04 smithi022 ceph-mon[40250]: pgmap v15764: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:05.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4128880320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:06 smithi122 ceph-mon[32255]: pgmap v15765: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:06 smithi022 ceph-mon[31360]: pgmap v15765: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:06 smithi022 ceph-mon[40250]: pgmap v15765: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:44:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:08 smithi122 ceph-mon[32255]: pgmap v15766: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:08 smithi022 ceph-mon[31360]: pgmap v15766: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:08 smithi022 ceph-mon[40250]: pgmap v15766: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:08.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:09.345 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:09.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:09.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:09.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:44:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:44:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3096102013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3096102013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3096102013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:10] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T02:44:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:10 smithi122 ceph-mon[32255]: pgmap v15767: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:10 smithi022 ceph-mon[31360]: pgmap v15767: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:10 smithi022 ceph-mon[40250]: pgmap v15767: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:12.679 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:44:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:44:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:12 smithi122 ceph-mon[32255]: pgmap v15768: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:12 smithi022 ceph-mon[31360]: pgmap v15768: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:12 smithi022 ceph-mon[40250]: pgmap v15768: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:14.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:14 smithi022 ceph-mon[40250]: pgmap v15769: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:14.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:14 smithi022 ceph-mon[31360]: pgmap v15769: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:14.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:14.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:14.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:14 smithi122 ceph-mon[32255]: pgmap v15769: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1214092936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1214092936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1214092936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:16 smithi022 ceph-mon[31360]: pgmap v15770: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:16 smithi022 ceph-mon[40250]: pgmap v15770: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:16 smithi122 ceph-mon[32255]: pgmap v15770: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:44:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:18 smithi122 ceph-mon[32255]: pgmap v15771: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:18 smithi022 ceph-mon[31360]: pgmap v15771: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:18 smithi022 ceph-mon[40250]: pgmap v15771: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:19.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:44:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:44:19.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:20.085 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:20.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:20.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:20] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:44:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:20 smithi122 ceph-mon[32255]: pgmap v15772: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3939500782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:20 smithi022 ceph-mon[31360]: pgmap v15772: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3939500782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:20 smithi022 ceph-mon[40250]: pgmap v15772: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3939500782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:22.683 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:44:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:44:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:22 smithi122 ceph-mon[32255]: pgmap v15773: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:22 smithi022 ceph-mon[31360]: pgmap v15773: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:22 smithi022 ceph-mon[40250]: pgmap v15773: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:24 smithi122 ceph-mon[32255]: pgmap v15774: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:24 smithi022 ceph-mon[31360]: pgmap v15774: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:24 smithi022 ceph-mon[40250]: pgmap v15774: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:25.087 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:25.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:25.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1744675322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1744675322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1744675322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:26 smithi022 ceph-mon[31360]: pgmap v15775: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:26 smithi022 ceph-mon[40250]: pgmap v15775: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:26 smithi122 ceph-mon[32255]: pgmap v15775: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:44:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:28 smithi122 ceph-mon[32255]: pgmap v15776: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:28 smithi022 ceph-mon[31360]: pgmap v15776: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:28 smithi022 ceph-mon[40250]: pgmap v15776: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:29.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:44:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:44:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:30] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:44:30.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:30 smithi022 ceph-mon[40250]: pgmap v15777: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:30 smithi022 ceph-mon[31360]: pgmap v15777: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:30.825 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:30.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:30.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:30 smithi122 ceph-mon[32255]: pgmap v15777: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2041998482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2041998482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2041998482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:44:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:44:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:32 smithi022 ceph-mon[31360]: pgmap v15778: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:32 smithi022 ceph-mon[40250]: pgmap v15778: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:32 smithi122 ceph-mon[32255]: pgmap v15778: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[31360]: pgmap v15779: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:44:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:44:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:44:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:44:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[40250]: pgmap v15779: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:44:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:44:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:44:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:44:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:34 smithi122 ceph-mon[32255]: pgmap v15779: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:44:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:44:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:44:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:44:35.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:36.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:36.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:36.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:36 smithi022 ceph-mon[31360]: pgmap v15780: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1917147582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:36 smithi022 ceph-mon[40250]: pgmap v15780: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1917147582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:36 smithi122 ceph-mon[32255]: pgmap v15780: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1917147582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:44:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:38 smithi122 ceph-mon[32255]: pgmap v15781: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:38 smithi022 ceph-mon[31360]: pgmap v15781: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:38 smithi022 ceph-mon[40250]: pgmap v15781: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:39.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:44:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:44:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:40] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T02:44:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:40 smithi022 ceph-mon[31360]: pgmap v15782: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:40 smithi022 ceph-mon[40250]: pgmap v15782: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:40 smithi122 ceph-mon[32255]: pgmap v15782: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:41.197 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:41.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:41.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:41.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1614629173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1614629173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1614629173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:44:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:44:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:42 smithi022 ceph-mon[31360]: pgmap v15783: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:42 smithi022 ceph-mon[40250]: pgmap v15783: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:42 smithi122 ceph-mon[32255]: pgmap v15783: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:44 smithi022 ceph-mon[31360]: pgmap v15784: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:44 smithi022 ceph-mon[40250]: pgmap v15784: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:44 smithi122 ceph-mon[32255]: pgmap v15784: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:46.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:46.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:46.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:46.943 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:46 smithi022 ceph-mon[31360]: pgmap v15785: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:46.944 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:46 smithi022 ceph-mon[40250]: pgmap v15785: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:46 smithi122 ceph-mon[32255]: pgmap v15785: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:44:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2859507023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:44:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:44:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:44:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:47 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:44:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2859507023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:44:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:44:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:44:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:44:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2859507023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:44:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:44:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:44:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:47 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:44:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:48 smithi022 ceph-mon[31360]: pgmap v15786: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:48 smithi022 ceph-mon[40250]: pgmap v15786: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:48 smithi122 ceph-mon[32255]: pgmap v15786: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:49.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:44:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:44:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:50] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:44:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:50 smithi022 ceph-mon[31360]: pgmap v15787: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:50 smithi022 ceph-mon[40250]: pgmap v15787: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:44:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:50 smithi122 ceph-mon[32255]: pgmap v15787: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:51.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:52.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:52.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:52.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:44:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:44:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:52 smithi022 ceph-mon[31360]: pgmap v15788: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2215217672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:52 smithi022 ceph-mon[40250]: pgmap v15788: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2215217672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:52 smithi122 ceph-mon[32255]: pgmap v15788: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2215217672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:55.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:54 smithi022 ceph-mon[40250]: pgmap v15789: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:55.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:54 smithi022 ceph-mon[31360]: pgmap v15789: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:54 smithi122 ceph-mon[32255]: pgmap v15789: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:44:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:56 smithi022 ceph-mon[31360]: pgmap v15790: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:56 smithi022 ceph-mon[40250]: pgmap v15790: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:56 smithi122 ceph-mon[32255]: pgmap v15790: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:57.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:44:57.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:44:57.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:44:57.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:44:57.682 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:44:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:58 smithi122 ceph-mon[32255]: pgmap v15791: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:44:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1285068599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:58 smithi022 ceph-mon[31360]: pgmap v15791: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:44:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1285068599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:58 smithi022 ceph-mon[40250]: pgmap v15791: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:44:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:44:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1285068599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:44:59.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:44:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:45:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:00] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T02:45:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:00 smithi022 ceph-mon[31360]: pgmap v15792: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:00 smithi022 ceph-mon[40250]: pgmap v15792: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:00 smithi122 ceph-mon[32255]: pgmap v15792: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:02.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:45:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:45:03.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:02 smithi022 ceph-mon[40250]: pgmap v15793: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:02 smithi022 ceph-mon[31360]: pgmap v15793: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:03.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:03.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:03.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:02 smithi122 ceph-mon[32255]: pgmap v15793: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1006468311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1006468311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1006468311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:04 smithi022 ceph-mon[31360]: pgmap v15794: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:04 smithi022 ceph-mon[40250]: pgmap v15794: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:04 smithi122 ceph-mon[32255]: pgmap v15794: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:06 smithi022 ceph-mon[31360]: pgmap v15795: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:06 smithi022 ceph-mon[40250]: pgmap v15795: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:06 smithi122 ceph-mon[32255]: pgmap v15795: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:45:08.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:08.328 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:08 smithi022 ceph-mon[40250]: pgmap v15796: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:08.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:08.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:08.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:08 smithi122 ceph-mon[32255]: pgmap v15796: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:08 smithi022 ceph-mon[31360]: pgmap v15796: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3129716415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:09.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:45:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:45:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3129716415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3129716415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:10] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T02:45:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:10 smithi122 ceph-mon[32255]: pgmap v15797: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:10 smithi022 ceph-mon[31360]: pgmap v15797: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:10 smithi022 ceph-mon[40250]: pgmap v15797: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:45:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:45:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:12 smithi022 ceph-mon[31360]: pgmap v15798: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:12 smithi022 ceph-mon[40250]: pgmap v15798: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:12 smithi122 ceph-mon[32255]: pgmap v15798: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:13.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:13.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:13.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:13.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:14 smithi022 ceph-mon[31360]: pgmap v15799: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4034711949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:14 smithi022 ceph-mon[40250]: pgmap v15799: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4034711949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:14 smithi122 ceph-mon[32255]: pgmap v15799: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4034711949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:16 smithi022 ceph-mon[31360]: pgmap v15800: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:16 smithi022 ceph-mon[40250]: pgmap v15800: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:16 smithi122 ceph-mon[32255]: pgmap v15800: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:45:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:18 smithi122 ceph-mon[32255]: pgmap v15801: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:18 smithi022 ceph-mon[31360]: pgmap v15801: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:18 smithi022 ceph-mon[40250]: pgmap v15801: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:18.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:19.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:19.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:19.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/890652148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/890652148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:19.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:45:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:45:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/890652148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:20] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:45:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:20 smithi122 ceph-mon[32255]: pgmap v15802: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:20 smithi022 ceph-mon[31360]: pgmap v15802: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:20 smithi022 ceph-mon[40250]: pgmap v15802: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:45:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:45:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:22 smithi022 ceph-mon[31360]: pgmap v15803: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:22 smithi022 ceph-mon[40250]: pgmap v15803: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:22 smithi122 ceph-mon[32255]: pgmap v15803: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:24.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:24 smithi022 ceph-mon[31360]: pgmap v15804: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3674459227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:24 smithi022 ceph-mon[40250]: pgmap v15804: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3674459227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:24 smithi122 ceph-mon[32255]: pgmap v15804: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3674459227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:26 smithi022 ceph-mon[31360]: pgmap v15805: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:26 smithi022 ceph-mon[40250]: pgmap v15805: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:26 smithi122 ceph-mon[32255]: pgmap v15805: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:45:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:28 smithi122 ceph-mon[32255]: pgmap v15806: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:28 smithi022 ceph-mon[31360]: pgmap v15806: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:28 smithi022 ceph-mon[40250]: pgmap v15806: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:29.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:29.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:45:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:45:29.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:29.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:29.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:30] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:45:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:30 smithi022 ceph-mon[31360]: pgmap v15807: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1137101810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:30 smithi022 ceph-mon[40250]: pgmap v15807: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1137101810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:30 smithi122 ceph-mon[32255]: pgmap v15807: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1137101810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:45:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:32 smithi022 ceph-mon[31360]: pgmap v15808: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:32 smithi022 ceph-mon[40250]: pgmap v15808: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:32 smithi122 ceph-mon[32255]: pgmap v15808: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:34.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[40250]: pgmap v15809: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:45:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:45:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:45:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:45:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[31360]: pgmap v15809: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:45:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:45:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:45:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:45:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:34 smithi122 ceph-mon[32255]: pgmap v15809: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:45:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:45:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:45:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:45:35.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:35.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:35.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:35.712 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3691076756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:35.712 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3691076756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3691076756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:36 smithi022 ceph-mon[31360]: pgmap v15810: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:36 smithi022 ceph-mon[40250]: pgmap v15810: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:36 smithi122 ceph-mon[32255]: pgmap v15810: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:45:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:38 smithi122 ceph-mon[32255]: pgmap v15811: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:38 smithi022 ceph-mon[31360]: pgmap v15811: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:38 smithi022 ceph-mon[40250]: pgmap v15811: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:39.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:45:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:45:40.292 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:40] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T02:45:40.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:40.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:40.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:40 smithi022 ceph-mon[40250]: pgmap v15812: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2574401959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:40 smithi022 ceph-mon[31360]: pgmap v15812: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2574401959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:40 smithi122 ceph-mon[32255]: pgmap v15812: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2574401959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:45:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:45:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:42 smithi022 ceph-mon[31360]: pgmap v15813: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:42 smithi022 ceph-mon[40250]: pgmap v15813: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:42 smithi122 ceph-mon[32255]: pgmap v15813: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:44 smithi022 ceph-mon[31360]: pgmap v15814: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:44 smithi022 ceph-mon[40250]: pgmap v15814: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:44 smithi122 ceph-mon[32255]: pgmap v15814: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:45.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:46.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:46.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:46.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:46 smithi022 ceph-mon[31360]: pgmap v15815: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3563581445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:46 smithi022 ceph-mon[40250]: pgmap v15815: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3563581445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:46 smithi122 ceph-mon[32255]: pgmap v15815: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3563581445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:47.734 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:45:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:48 smithi122 ceph-mon[32255]: pgmap v15816: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:45:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:45:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:45:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:45:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[31360]: pgmap v15816: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:45:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:45:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:45:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:45:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[40250]: pgmap v15816: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:45:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:45:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:45:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:45:49.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:45:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:45:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:50] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:45:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:51.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:50 smithi022 ceph-mon[31360]: pgmap v15817: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:50 smithi022 ceph-mon[40250]: pgmap v15817: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:51.036 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:45:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:50 smithi122 ceph-mon[32255]: pgmap v15817: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:51.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:51.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:51.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2790547635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2790547635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2790547635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:45:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:45:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:52 smithi022 ceph-mon[31360]: pgmap v15818: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:52 smithi022 ceph-mon[40250]: pgmap v15818: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:52 smithi122 ceph-mon[32255]: pgmap v15818: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:54 smithi022 ceph-mon[31360]: pgmap v15819: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:54 smithi022 ceph-mon[40250]: pgmap v15819: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:54 smithi122 ceph-mon[32255]: pgmap v15819: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:45:56.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:45:56.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:45:56.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:45:56.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:45:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:56 smithi022 ceph-mon[31360]: pgmap v15820: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:56 smithi022 ceph-mon[40250]: pgmap v15820: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:56 smithi122 ceph-mon[32255]: pgmap v15820: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:57.720 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:45:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1765219619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1765219619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1765219619' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:45:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:45:58 smithi022 ceph-mon[31360]: pgmap v15821: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:45:58 smithi022 ceph-mon[40250]: pgmap v15821: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:45:58 smithi122 ceph-mon[32255]: pgmap v15821: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:45:59.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:45:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:46:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:00] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:46:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:00 smithi022 ceph-mon[31360]: pgmap v15822: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:00 smithi022 ceph-mon[40250]: pgmap v15822: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:00 smithi122 ceph-mon[32255]: pgmap v15822: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:01.778 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:02.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:02.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:02.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:46:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:46:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:02 smithi022 ceph-mon[31360]: pgmap v15823: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/425761783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:02 smithi022 ceph-mon[40250]: pgmap v15823: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/425761783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:02 smithi122 ceph-mon[32255]: pgmap v15823: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/425761783' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:04 smithi022 ceph-mon[31360]: pgmap v15824: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:04 smithi022 ceph-mon[40250]: pgmap v15824: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:04 smithi122 ceph-mon[32255]: pgmap v15824: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:06 smithi022 ceph-mon[31360]: pgmap v15825: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:06 smithi022 ceph-mon[40250]: pgmap v15825: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:07.149 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:06 smithi122 ceph-mon[32255]: pgmap v15825: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:07.519 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:46:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/864124138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:08 smithi122 ceph-mon[32255]: pgmap v15826: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/864124138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:08 smithi022 ceph-mon[31360]: pgmap v15826: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/864124138' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:08 smithi022 ceph-mon[40250]: pgmap v15826: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:09.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:46:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:46:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:10] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:46:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:10 smithi022 ceph-mon[31360]: pgmap v15827: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:10 smithi022 ceph-mon[40250]: pgmap v15827: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:10 smithi122 ceph-mon[32255]: pgmap v15827: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:12.520 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:46:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:46:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:12 smithi022 ceph-mon[31360]: pgmap v15828: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:12.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:12 smithi022 ceph-mon[40250]: pgmap v15828: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:12.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:12.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:12.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:12 smithi122 ceph-mon[32255]: pgmap v15828: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3351121426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3351121426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3351121426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:14 smithi022 ceph-mon[31360]: pgmap v15829: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:14 smithi022 ceph-mon[40250]: pgmap v15829: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:14 smithi122 ceph-mon[32255]: pgmap v15829: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:16 smithi022 ceph-mon[31360]: pgmap v15830: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:16 smithi022 ceph-mon[40250]: pgmap v15830: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:17.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:16 smithi122 ceph-mon[32255]: pgmap v15830: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:46:17.893 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:18.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:18.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:18.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:18 smithi022 ceph-mon[31360]: pgmap v15831: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2154671968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:18 smithi022 ceph-mon[40250]: pgmap v15831: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2154671968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:18 smithi122 ceph-mon[32255]: pgmap v15831: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2154671968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:19.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:46:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:46:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:20] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:46:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:20 smithi022 ceph-mon[31360]: pgmap v15832: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:20 smithi022 ceph-mon[40250]: pgmap v15832: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:20 smithi122 ceph-mon[32255]: pgmap v15832: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:46:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:46:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:22 smithi022 ceph-mon[31360]: pgmap v15833: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:22 smithi022 ceph-mon[40250]: pgmap v15833: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:22 smithi122 ceph-mon[32255]: pgmap v15833: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:23.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:23.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:23.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:23.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/395145308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/395145308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/395145308' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:24 smithi022 ceph-mon[31360]: pgmap v15834: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:24 smithi022 ceph-mon[40250]: pgmap v15834: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:24 smithi122 ceph-mon[32255]: pgmap v15834: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:26 smithi022 ceph-mon[31360]: pgmap v15835: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:26 smithi022 ceph-mon[40250]: pgmap v15835: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:26 smithi122 ceph-mon[32255]: pgmap v15835: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:46:28.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:28.634 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:28 smithi022 ceph-mon[31360]: pgmap v15836: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:28.635 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:28 smithi022 ceph-mon[40250]: pgmap v15836: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:28 smithi122 ceph-mon[32255]: pgmap v15836: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:29.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:29.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:29.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:29.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:46:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:46:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2310775412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2310775412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2310775412' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:30] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:46:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:30 smithi122 ceph-mon[32255]: pgmap v15837: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:30 smithi022 ceph-mon[31360]: pgmap v15837: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:30 smithi022 ceph-mon[40250]: pgmap v15837: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:46:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:46:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:32 smithi022 ceph-mon[31360]: pgmap v15838: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:32 smithi022 ceph-mon[40250]: pgmap v15838: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:32 smithi122 ceph-mon[32255]: pgmap v15838: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:34.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:34.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:34.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:34.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[40250]: pgmap v15839: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:46:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:46:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:46:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:46:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2826195957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[31360]: pgmap v15839: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:46:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:46:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:46:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:46:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2826195957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:34 smithi122 ceph-mon[32255]: pgmap v15839: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:46:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:46:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:46:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:46:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2826195957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:36 smithi022 ceph-mon[40250]: pgmap v15840: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:36 smithi022 ceph-mon[31360]: pgmap v15840: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:36 smithi122 ceph-mon[32255]: pgmap v15840: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:46:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:38 smithi122 ceph-mon[32255]: pgmap v15841: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:38 smithi022 ceph-mon[40250]: pgmap v15841: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:38 smithi022 ceph-mon[31360]: pgmap v15841: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:39.372 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:39.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:39.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:39.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:39.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:46:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:46:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:40] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:46:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:40 smithi022 ceph-mon[31360]: pgmap v15842: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1440442994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:40 smithi022 ceph-mon[40250]: pgmap v15842: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1440442994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:40 smithi122 ceph-mon[32255]: pgmap v15842: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1440442994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:46:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:46:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:42 smithi022 ceph-mon[31360]: pgmap v15843: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:42 smithi022 ceph-mon[40250]: pgmap v15843: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:42 smithi122 ceph-mon[32255]: pgmap v15843: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:44.739 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:44 smithi022 ceph-mon[31360]: pgmap v15844: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:44 smithi022 ceph-mon[40250]: pgmap v15844: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:45.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:45.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:45.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:44 smithi122 ceph-mon[32255]: pgmap v15844: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3186592343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3186592343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3186592343' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:46 smithi022 ceph-mon[31360]: pgmap v15845: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:46 smithi022 ceph-mon[40250]: pgmap v15845: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:46 smithi122 ceph-mon[32255]: pgmap v15845: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:46:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[31360]: pgmap v15846: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:46:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:46:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:46:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[40250]: pgmap v15846: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:48.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:46:48.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:46:48.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:46:48.609 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:48 smithi122 ceph-mon[32255]: pgmap v15846: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:48.609 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:46:48.610 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:46:48.610 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:46:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:46:49.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:46:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:46:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:46:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:46:50.110 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:50] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:46:50.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:50.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:50.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:50 smithi022 ceph-mon[31360]: pgmap v15847: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:50.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4157282902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:50 smithi022 ceph-mon[40250]: pgmap v15847: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4157282902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:46:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:50 smithi122 ceph-mon[32255]: pgmap v15847: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4157282902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:46:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:46:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:52 smithi022 ceph-mon[31360]: pgmap v15848: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:52 smithi022 ceph-mon[40250]: pgmap v15848: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:52 smithi122 ceph-mon[32255]: pgmap v15848: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:54 smithi022 ceph-mon[31360]: pgmap v15849: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:54 smithi022 ceph-mon[40250]: pgmap v15849: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:54 smithi122 ceph-mon[32255]: pgmap v15849: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:46:55.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:46:55.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:46:55.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:46:55.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:46:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:56 smithi022 ceph-mon[31360]: pgmap v15850: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4094606153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:56 smithi022 ceph-mon[40250]: pgmap v15850: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4094606153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:56 smithi122 ceph-mon[32255]: pgmap v15850: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4094606153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:46:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:46:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:46:58 smithi122 ceph-mon[32255]: pgmap v15851: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:46:58 smithi022 ceph-mon[31360]: pgmap v15851: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:46:58 smithi022 ceph-mon[40250]: pgmap v15851: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:46:59.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:46:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:47:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:00] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:47:00.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:00 smithi022 ceph-mon[31360]: pgmap v15852: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:00 smithi022 ceph-mon[40250]: pgmap v15852: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:00 smithi122 ceph-mon[32255]: pgmap v15852: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:01.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:01.223 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:01.224 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1529182700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1529182700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1529182700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:47:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:47:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:02 smithi022 ceph-mon[31360]: pgmap v15853: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:02 smithi022 ceph-mon[40250]: pgmap v15853: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:02 smithi122 ceph-mon[32255]: pgmap v15853: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:04 smithi022 ceph-mon[31360]: pgmap v15854: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:04 smithi022 ceph-mon[40250]: pgmap v15854: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:04 smithi122 ceph-mon[32255]: pgmap v15854: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:06.225 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:06.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:06.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:06.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:06 smithi022 ceph-mon[31360]: pgmap v15855: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2912268172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:06 smithi022 ceph-mon[40250]: pgmap v15855: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2912268172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:06 smithi122 ceph-mon[32255]: pgmap v15855: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2912268172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:47:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:08 smithi122 ceph-mon[32255]: pgmap v15856: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:08 smithi022 ceph-mon[31360]: pgmap v15856: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:08 smithi022 ceph-mon[40250]: pgmap v15856: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:09.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:47:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:47:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:10] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:47:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:10 smithi022 ceph-mon[31360]: pgmap v15857: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:10 smithi022 ceph-mon[40250]: pgmap v15857: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:10 smithi122 ceph-mon[32255]: pgmap v15857: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:11.596 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:11.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:11.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:11.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:47:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:47:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:12 smithi022 ceph-mon[31360]: pgmap v15858: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1495389765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:12 smithi022 ceph-mon[40250]: pgmap v15858: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1495389765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:12 smithi122 ceph-mon[32255]: pgmap v15858: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1495389765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:14 smithi022 ceph-mon[31360]: pgmap v15859: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:14 smithi022 ceph-mon[40250]: pgmap v15859: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:14 smithi122 ceph-mon[32255]: pgmap v15859: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:16.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:16 smithi022 ceph-mon[31360]: pgmap v15860: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:16 smithi022 ceph-mon[40250]: pgmap v15860: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:16 smithi122 ceph-mon[32255]: pgmap v15860: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:17.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:17.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:17.740 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:47:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2947142677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2947142677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2947142677' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:18 smithi022 ceph-mon[31360]: pgmap v15861: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:18 smithi022 ceph-mon[40250]: pgmap v15861: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:18 smithi122 ceph-mon[32255]: pgmap v15861: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:19.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:47:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:47:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:20] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:47:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:20 smithi022 ceph-mon[31360]: pgmap v15862: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:20 smithi022 ceph-mon[40250]: pgmap v15862: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:20 smithi122 ceph-mon[32255]: pgmap v15862: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:22.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:47:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:47:22.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:22.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:22.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:22 smithi022 ceph-mon[40250]: pgmap v15863: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2459651689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:22 smithi022 ceph-mon[31360]: pgmap v15863: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2459651689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:22 smithi122 ceph-mon[32255]: pgmap v15863: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2459651689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:24 smithi022 ceph-mon[31360]: pgmap v15864: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:24 smithi022 ceph-mon[40250]: pgmap v15864: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:24 smithi122 ceph-mon[32255]: pgmap v15864: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:26 smithi022 ceph-mon[31360]: pgmap v15865: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:26 smithi022 ceph-mon[40250]: pgmap v15865: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:26 smithi122 ceph-mon[32255]: pgmap v15865: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:27.704 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:27.705 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:47:28.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:28.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:28.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:28 smithi122 ceph-mon[32255]: pgmap v15866: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3362713285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:28 smithi022 ceph-mon[31360]: pgmap v15866: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3362713285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:28 smithi022 ceph-mon[40250]: pgmap v15866: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3362713285' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:29.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:47:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:47:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:30] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:47:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:30 smithi022 ceph-mon[40250]: pgmap v15867: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:30 smithi022 ceph-mon[31360]: pgmap v15867: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:30 smithi122 ceph-mon[32255]: pgmap v15867: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:47:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:47:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:32 smithi022 ceph-mon[31360]: pgmap v15868: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:32 smithi022 ceph-mon[40250]: pgmap v15868: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:33.071 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:32 smithi122 ceph-mon[32255]: pgmap v15868: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3285088836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3285088836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3285088836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[31360]: pgmap v15869: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:47:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[40250]: pgmap v15869: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:47:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:47:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:47:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:34 smithi122 ceph-mon[32255]: pgmap v15869: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:47:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:47:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:47:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:47:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:36 smithi022 ceph-mon[31360]: pgmap v15870: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:36 smithi022 ceph-mon[40250]: pgmap v15870: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:36 smithi122 ceph-mon[32255]: pgmap v15870: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:47:38.444 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:38 smithi122 ceph-mon[32255]: pgmap v15871: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:38 smithi022 ceph-mon[31360]: pgmap v15871: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:38 smithi022 ceph-mon[40250]: pgmap v15871: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:38.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:38.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:38.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:39.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:47:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:47:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4159589881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4159589881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4159589881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:40] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:47:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:40 smithi122 ceph-mon[32255]: pgmap v15872: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:40 smithi022 ceph-mon[31360]: pgmap v15872: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:40.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:40 smithi022 ceph-mon[40250]: pgmap v15872: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:47:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:47:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:42 smithi022 ceph-mon[31360]: pgmap v15873: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:42 smithi022 ceph-mon[40250]: pgmap v15873: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:42 smithi122 ceph-mon[32255]: pgmap v15873: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:43.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:44.182 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:44.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:44.183 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:44 smithi022 ceph-mon[31360]: pgmap v15874: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/551392147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:44 smithi022 ceph-mon[40250]: pgmap v15874: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/551392147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:44 smithi122 ceph-mon[32255]: pgmap v15874: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/551392147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:46 smithi022 ceph-mon[31360]: pgmap v15875: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:46 smithi022 ceph-mon[40250]: pgmap v15875: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:46 smithi122 ceph-mon[32255]: pgmap v15875: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:47:48.660 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:48 smithi122 ceph-mon[32255]: pgmap v15876: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:48.663 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:48 smithi022 ceph-mon[31360]: pgmap v15876: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:48.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:48 smithi022 ceph-mon[40250]: pgmap v15876: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:49.185 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:47:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:47:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:47:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:47:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:47:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:47:49.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:49.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:49.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:49.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:47:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:47:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:47:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:47:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:47:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:50] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:47:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1565591867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:50 smithi122 ceph-mon[32255]: pgmap v15877: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1565591867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:50 smithi022 ceph-mon[31360]: pgmap v15877: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1565591867' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:47:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:50 smithi022 ceph-mon[40250]: pgmap v15877: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:47:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:47:52.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:52 smithi122 ceph-mon[32255]: pgmap v15878: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:52.997 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:52 smithi022 ceph-mon[31360]: pgmap v15878: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:52 smithi022 ceph-mon[40250]: pgmap v15878: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:47:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2921M 2021-12-16T02:47:54.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2921M 2021-12-16T02:47:54.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.038 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.039 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2921M 2021-12-16T02:47:54.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:47:54.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:47:54.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:47:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:47:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:47:54.933 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:54 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3196M 2021-12-16T02:47:54.934 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:54 smithi022 ceph-mon[31360]: pgmap v15879: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:54.934 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:54 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3196M 2021-12-16T02:47:54.934 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:54 smithi022 ceph-mon[40250]: pgmap v15879: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:54 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3196M 2021-12-16T02:47:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:54 smithi122 ceph-mon[32255]: pgmap v15879: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:47:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/744161334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/744161334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/744161334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:47:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:56 smithi022 ceph-mon[31360]: pgmap v15880: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-16T02:47:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:56 smithi022 ceph-mon[40250]: pgmap v15880: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-16T02:47:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:56 smithi122 ceph-mon[32255]: pgmap v15880: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-16T02:47:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:47:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:47:58 smithi122 ceph-mon[32255]: pgmap v15881: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-16T02:47:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:47:58 smithi022 ceph-mon[31360]: pgmap v15881: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-16T02:47:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:47:58 smithi022 ceph-mon[40250]: pgmap v15881: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-12-16T02:47:59.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:47:59.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:47:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:48:00.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:00.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:00.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:00] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:48:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:00 smithi022 ceph-mon[31360]: pgmap v15882: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1848212145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:00 smithi022 ceph-mon[40250]: pgmap v15882: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1848212145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:00 smithi122 ceph-mon[32255]: pgmap v15882: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1848212145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:48:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:48:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:02 smithi022 ceph-mon[31360]: pgmap v15883: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:02 smithi022 ceph-mon[40250]: pgmap v15883: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:02 smithi122 ceph-mon[32255]: pgmap v15883: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:04 smithi022 ceph-mon[31360]: pgmap v15884: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:04 smithi022 ceph-mon[40250]: pgmap v15884: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:04 smithi122 ceph-mon[32255]: pgmap v15884: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:05.303 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:05.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:05.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:05.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2897673942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2897673942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2897673942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:06 smithi022 ceph-mon[31360]: pgmap v15885: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:06 smithi022 ceph-mon[40250]: pgmap v15885: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:06 smithi122 ceph-mon[32255]: pgmap v15885: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:48:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:08 smithi122 ceph-mon[32255]: pgmap v15886: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:08 smithi022 ceph-mon[31360]: pgmap v15886: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:08 smithi022 ceph-mon[40250]: pgmap v15886: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:09.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:48:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:48:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:10] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T02:48:10.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:10 smithi022 ceph-mon[31360]: pgmap v15887: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:10 smithi022 ceph-mon[40250]: pgmap v15887: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:11.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:11.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:11.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:10 smithi122 ceph-mon[32255]: pgmap v15887: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:48:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2646156133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2646156133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2646156133' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:48:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:48:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:12 smithi022 ceph-mon[31360]: pgmap v15888: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:12 smithi022 ceph-mon[40250]: pgmap v15888: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:12 smithi122 ceph-mon[32255]: pgmap v15888: 132 pgs: 132 active+clean; 582 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:14 smithi022 ceph-mon[31360]: pgmap v15889: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:14 smithi022 ceph-mon[40250]: pgmap v15889: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:14 smithi122 ceph-mon[32255]: pgmap v15889: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:16.033 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:16.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:16.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:16 smithi022 ceph-mon[31360]: pgmap v15890: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/444676433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:16 smithi022 ceph-mon[40250]: pgmap v15890: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/444676433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:16 smithi122 ceph-mon[32255]: pgmap v15890: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/444676433' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:48:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:18 smithi122 ceph-mon[32255]: pgmap v15891: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:18 smithi022 ceph-mon[31360]: pgmap v15891: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:18 smithi022 ceph-mon[40250]: pgmap v15891: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:19.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:48:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:48:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:20] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T02:48:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:20 smithi022 ceph-mon[31360]: pgmap v15892: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:20 smithi022 ceph-mon[40250]: pgmap v15892: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:20 smithi122 ceph-mon[32255]: pgmap v15892: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:21.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:21.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:21.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:21.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:48:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:48:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:22 smithi022 ceph-mon[31360]: pgmap v15893: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4005330190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:22 smithi022 ceph-mon[40250]: pgmap v15893: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4005330190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:22 smithi122 ceph-mon[32255]: pgmap v15893: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:48:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4005330190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:24 smithi022 ceph-mon[31360]: pgmap v15894: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:24 smithi022 ceph-mon[40250]: pgmap v15894: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:24 smithi122 ceph-mon[32255]: pgmap v15894: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:48:26.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:26 smithi022 ceph-mon[31360]: pgmap v15895: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:26 smithi022 ceph-mon[40250]: pgmap v15895: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:27.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:27.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:27.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:26 smithi122 ceph-mon[32255]: pgmap v15895: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:27.762 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:48:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3641288124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3641288124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3641288124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:28 smithi022 ceph-mon[31360]: pgmap v15896: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:28 smithi022 ceph-mon[40250]: pgmap v15896: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:28 smithi122 ceph-mon[32255]: pgmap v15896: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:29.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:48:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:48:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:30] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T02:48:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:30 smithi022 ceph-mon[31360]: pgmap v15897: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:30 smithi022 ceph-mon[40250]: pgmap v15897: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:31.286 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:30 smithi122 ceph-mon[32255]: pgmap v15897: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:32.143 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:32.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:32.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:32.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:48:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:48:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:32 smithi022 ceph-mon[31360]: pgmap v15898: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1962142284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:32 smithi022 ceph-mon[40250]: pgmap v15898: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1962142284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:32 smithi122 ceph-mon[32255]: pgmap v15898: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1962142284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[31360]: pgmap v15899: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:48:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:48:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:48:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:48:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[40250]: pgmap v15899: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:48:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:48:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:48:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:48:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:34 smithi122 ceph-mon[32255]: pgmap v15899: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:48:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:48:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:48:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:48:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:36 smithi022 ceph-mon[31360]: pgmap v15900: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:36 smithi022 ceph-mon[40250]: pgmap v15900: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:36 smithi122 ceph-mon[32255]: pgmap v15900: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:37.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:48:37.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:37.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:37.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:38 smithi122 ceph-mon[32255]: pgmap v15901: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/875777667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:38 smithi022 ceph-mon[31360]: pgmap v15901: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/875777667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:38 smithi022 ceph-mon[40250]: pgmap v15901: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/875777667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:39.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:48:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:48:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:40] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T02:48:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:40 smithi022 ceph-mon[31360]: pgmap v15902: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:40 smithi022 ceph-mon[40250]: pgmap v15902: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:40 smithi122 ceph-mon[32255]: pgmap v15902: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:48:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:48:42.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:42 smithi022 ceph-mon[31360]: pgmap v15903: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:42 smithi022 ceph-mon[40250]: pgmap v15903: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:42 smithi122 ceph-mon[32255]: pgmap v15903: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1307806568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1307806568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1307806568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:44 smithi022 ceph-mon[31360]: pgmap v15904: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:44 smithi022 ceph-mon[40250]: pgmap v15904: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:44 smithi122 ceph-mon[32255]: pgmap v15904: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:46 smithi022 ceph-mon[31360]: pgmap v15905: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:46 smithi022 ceph-mon[40250]: pgmap v15905: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:46 smithi122 ceph-mon[32255]: pgmap v15905: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:48:48.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:48 smithi022 ceph-mon[31360]: pgmap v15906: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:48 smithi022 ceph-mon[40250]: pgmap v15906: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:48.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:48.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:48.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:48 smithi122 ceph-mon[32255]: pgmap v15906: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2983302332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:49.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:48:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:48:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2983302332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2983302332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:50] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:48:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:50 smithi122 ceph-mon[32255]: pgmap v15907: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:50 smithi022 ceph-mon[31360]: pgmap v15907: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:48:50.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:50 smithi022 ceph-mon[40250]: pgmap v15907: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:48:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:48:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:52 smithi022 ceph-mon[31360]: pgmap v15908: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:52 smithi022 ceph-mon[40250]: pgmap v15908: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:52 smithi122 ceph-mon[32255]: pgmap v15908: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:53.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:53.768 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:48:53.769 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:48:53.769 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:48:53.769 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:48:53.770 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:48:53.770 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:48:53.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:53.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:53.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:48:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:48:54.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:48:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:54 smithi122 ceph-mon[32255]: pgmap v15909: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:48:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2060562469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:54 smithi022 ceph-mon[31360]: pgmap v15909: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:48:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2060562469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:54 smithi022 ceph-mon[40250]: pgmap v15909: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:48:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:48:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2060562469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:56 smithi122 ceph-mon[32255]: pgmap v15910: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:56 smithi022 ceph-mon[31360]: pgmap v15910: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:56 smithi022 ceph-mon[40250]: pgmap v15910: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:48:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:58 smithi122 ceph-mon[32255]: pgmap v15911: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:58 smithi022 ceph-mon[31360]: pgmap v15911: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:58 smithi022 ceph-mon[40250]: pgmap v15911: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:48:58.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:48:59.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:48:59.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:48:59.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:48:59.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:48:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:48:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:48:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1016954502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:48:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1016954502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:48:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:48:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1016954502' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:00] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:49:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:00 smithi122 ceph-mon[32255]: pgmap v15912: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:00 smithi022 ceph-mon[31360]: pgmap v15912: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:00 smithi022 ceph-mon[40250]: pgmap v15912: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:49:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:49:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:02 smithi022 ceph-mon[31360]: pgmap v15913: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:02 smithi022 ceph-mon[40250]: pgmap v15913: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:02 smithi122 ceph-mon[32255]: pgmap v15913: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:04.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:04.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:04.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:04.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:04 smithi022 ceph-mon[31360]: pgmap v15914: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1626103956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:04 smithi022 ceph-mon[40250]: pgmap v15914: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1626103956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:04 smithi122 ceph-mon[32255]: pgmap v15914: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1626103956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:06 smithi022 ceph-mon[31360]: pgmap v15915: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:06 smithi022 ceph-mon[40250]: pgmap v15915: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:06 smithi122 ceph-mon[32255]: pgmap v15915: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:49:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:08 smithi122 ceph-mon[32255]: pgmap v15916: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:08 smithi022 ceph-mon[31360]: pgmap v15916: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:08 smithi022 ceph-mon[40250]: pgmap v15916: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:49:09.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:09.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:49:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:49:10.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:10.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:10.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:10] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T02:49:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:10 smithi022 ceph-mon[31360]: pgmap v15917: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/824069297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:10 smithi022 ceph-mon[40250]: pgmap v15917: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/824069297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:10 smithi122 ceph-mon[32255]: pgmap v15917: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/824069297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:49:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:49:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:12 smithi022 ceph-mon[31360]: pgmap v15918: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:12 smithi022 ceph-mon[40250]: pgmap v15918: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:12 smithi122 ceph-mon[32255]: pgmap v15918: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:14 smithi022 ceph-mon[31360]: pgmap v15919: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:14 smithi022 ceph-mon[40250]: pgmap v15919: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:15.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:14 smithi122 ceph-mon[32255]: pgmap v15919: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:15.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:15.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:15.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1121279592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1121279592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1121279592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:16.955 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:16 smithi122 ceph-mon[32255]: pgmap v15920: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:16 smithi022 ceph-mon[31360]: pgmap v15920: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:16 smithi022 ceph-mon[40250]: pgmap v15920: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:49:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:18 smithi122 ceph-mon[32255]: pgmap v15921: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:18 smithi022 ceph-mon[31360]: pgmap v15921: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:18 smithi022 ceph-mon[40250]: pgmap v15921: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:19.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:49:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:49:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:20] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:49:20.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:20.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:20.772 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:20 smithi022 ceph-mon[31360]: pgmap v15922: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:20 smithi022 ceph-mon[40250]: pgmap v15922: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:20 smithi122 ceph-mon[32255]: pgmap v15922: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1376547985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1376547985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1376547985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:49:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:49:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:22 smithi022 ceph-mon[31360]: pgmap v15923: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:22 smithi022 ceph-mon[40250]: pgmap v15923: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:22 smithi122 ceph-mon[32255]: pgmap v15923: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:24 smithi122 ceph-mon[32255]: pgmap v15924: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:24 smithi022 ceph-mon[31360]: pgmap v15924: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:24 smithi022 ceph-mon[40250]: pgmap v15924: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:25.848 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:26.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:26.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:26.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:26 smithi122 ceph-mon[32255]: pgmap v15925: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1511225440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:26 smithi022 ceph-mon[31360]: pgmap v15925: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1511225440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:26 smithi022 ceph-mon[40250]: pgmap v15925: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1511225440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:49:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:28 smithi122 ceph-mon[32255]: pgmap v15926: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:28 smithi022 ceph-mon[31360]: pgmap v15926: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:28 smithi022 ceph-mon[40250]: pgmap v15926: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:29.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:49:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:49:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:30] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:49:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:30 smithi022 ceph-mon[31360]: pgmap v15927: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:30 smithi022 ceph-mon[40250]: pgmap v15927: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:30 smithi122 ceph-mon[32255]: pgmap v15927: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:31.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:31.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:31.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:31.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/958768056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/958768056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/958768056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:49:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:49:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:32 smithi122 ceph-mon[32255]: pgmap v15928: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:32 smithi022 ceph-mon[31360]: pgmap v15928: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:32 smithi022 ceph-mon[40250]: pgmap v15928: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:34 smithi122 ceph-mon[32255]: pgmap v15929: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:49:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:49:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:49:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:49:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[31360]: pgmap v15929: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:49:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:49:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:49:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:49:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[40250]: pgmap v15929: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:49:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:49:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:49:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:49:36.587 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:36.962 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:36 smithi022 ceph-mon[31360]: pgmap v15930: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:36.963 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:36 smithi022 ceph-mon[40250]: pgmap v15930: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:36 smithi122 ceph-mon[32255]: pgmap v15930: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:49:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2697243838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2697243838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2697243838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:38 smithi122 ceph-mon[32255]: pgmap v15931: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:38 smithi022 ceph-mon[31360]: pgmap v15931: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:38 smithi022 ceph-mon[40250]: pgmap v15931: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:39.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:49:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:49:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T02:49:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:40 smithi122 ceph-mon[32255]: pgmap v15932: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:40 smithi022 ceph-mon[31360]: pgmap v15932: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:40 smithi022 ceph-mon[40250]: pgmap v15932: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:41.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:42.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:42.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:42.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:49:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:49:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:42 smithi122 ceph-mon[32255]: pgmap v15933: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2406881447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:42 smithi022 ceph-mon[31360]: pgmap v15933: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2406881447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:42 smithi022 ceph-mon[40250]: pgmap v15933: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2406881447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:44 smithi122 ceph-mon[32255]: pgmap v15934: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:44 smithi022 ceph-mon[31360]: pgmap v15934: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:44 smithi022 ceph-mon[40250]: pgmap v15934: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:46 smithi122 ceph-mon[32255]: pgmap v15935: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:46 smithi022 ceph-mon[31360]: pgmap v15935: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:46 smithi022 ceph-mon[40250]: pgmap v15935: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:47.335 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:47.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:47.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:47.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:47.705 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:49:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2450437551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:48 smithi122 ceph-mon[32255]: pgmap v15936: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2450437551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:48 smithi022 ceph-mon[31360]: pgmap v15936: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2450437551' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:48 smithi022 ceph-mon[40250]: pgmap v15936: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:49.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:49:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:49:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:49:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:50 smithi122 ceph-mon[32255]: pgmap v15937: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:50 smithi022 ceph-mon[31360]: pgmap v15937: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:49:51.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:50 smithi022 ceph-mon[40250]: pgmap v15937: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:49:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:49:52.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:52 smithi022 ceph-mon[31360]: pgmap v15938: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:53.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:52 smithi022 ceph-mon[40250]: pgmap v15938: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:53.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:53.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:53.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:52 smithi122 ceph-mon[32255]: pgmap v15938: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2821157467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2821157467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2821157467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:54 smithi122 ceph-mon[32255]: pgmap v15939: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:49:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:49:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:49:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:49:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[31360]: pgmap v15939: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:49:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:49:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:49:55.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:49:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[40250]: pgmap v15939: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:49:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:49:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:49:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:49:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:49:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:56 smithi122 ceph-mon[32255]: pgmap v15940: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:56 smithi022 ceph-mon[31360]: pgmap v15940: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:56 smithi022 ceph-mon[40250]: pgmap v15940: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:49:58.081 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:49:58.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:49:58.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:49:58.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:49:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:58 smithi122 ceph-mon[32255]: pgmap v15941: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:58 smithi022 ceph-mon[31360]: pgmap v15941: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:58 smithi022 ceph-mon[40250]: pgmap v15941: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:49:59.646 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:49:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1397893223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:49:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1397893223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:49:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1397893223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:49:59.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:49:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:50:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:00 smithi122 ceph-mon[32255]: pgmap v15942: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T02:50:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:00 smithi022 ceph-mon[31360]: pgmap v15942: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T02:50:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:00 smithi022 ceph-mon[40250]: pgmap v15942: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T02:50:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:50:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:50:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:02 smithi122 ceph-mon[32255]: pgmap v15943: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:02 smithi022 ceph-mon[31360]: pgmap v15943: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:02 smithi022 ceph-mon[40250]: pgmap v15943: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:03.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:03.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:03.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:03.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:04 smithi122 ceph-mon[32255]: pgmap v15944: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4045923807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:04 smithi022 ceph-mon[31360]: pgmap v15944: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4045923807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:04 smithi022 ceph-mon[40250]: pgmap v15944: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4045923807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:06 smithi122 ceph-mon[32255]: pgmap v15945: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:06 smithi022 ceph-mon[31360]: pgmap v15945: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:06 smithi022 ceph-mon[40250]: pgmap v15945: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:50:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:08 smithi122 ceph-mon[32255]: pgmap v15946: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:08 smithi022 ceph-mon[31360]: pgmap v15946: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:08 smithi022 ceph-mon[40250]: pgmap v15946: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:09.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:09.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:09.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2401926584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2401926584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:09.649 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2401926584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:09.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:50:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:50:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:10] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T02:50:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:10 smithi122 ceph-mon[32255]: pgmap v15947: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:10 smithi022 ceph-mon[31360]: pgmap v15947: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:10 smithi022 ceph-mon[40250]: pgmap v15947: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:50:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:50:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:12 smithi122 ceph-mon[32255]: pgmap v15948: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:12 smithi022 ceph-mon[31360]: pgmap v15948: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:12 smithi022 ceph-mon[40250]: pgmap v15948: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:14.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:14.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:14.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:14.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:14 smithi022 ceph-mon[31360]: pgmap v15949: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/534348034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:14 smithi022 ceph-mon[40250]: pgmap v15949: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/534348034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:14 smithi122 ceph-mon[32255]: pgmap v15949: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/534348034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:16 smithi122 ceph-mon[32255]: pgmap v15950: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:16 smithi022 ceph-mon[31360]: pgmap v15950: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:16 smithi022 ceph-mon[40250]: pgmap v15950: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:50:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:18 smithi122 ceph-mon[32255]: pgmap v15951: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:18 smithi022 ceph-mon[31360]: pgmap v15951: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:18 smithi022 ceph-mon[40250]: pgmap v15951: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:19.563 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:19.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:19.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:19.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:19.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:50:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:50:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:20] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T02:50:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:20 smithi122 ceph-mon[32255]: pgmap v15952: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1666502703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:20 smithi022 ceph-mon[31360]: pgmap v15952: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1666502703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:20 smithi022 ceph-mon[40250]: pgmap v15952: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1666502703' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:50:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:50:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:22 smithi122 ceph-mon[32255]: pgmap v15953: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:22 smithi022 ceph-mon[31360]: pgmap v15953: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:22 smithi022 ceph-mon[40250]: pgmap v15953: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:24.933 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:24 smithi122 ceph-mon[32255]: pgmap v15954: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:24 smithi022 ceph-mon[31360]: pgmap v15954: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:24 smithi022 ceph-mon[40250]: pgmap v15954: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:25.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:25.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:25.303 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2481033294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2481033294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2481033294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:26 smithi122 ceph-mon[32255]: pgmap v15955: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:26 smithi022 ceph-mon[31360]: pgmap v15955: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:26 smithi022 ceph-mon[40250]: pgmap v15955: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:50:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:28 smithi122 ceph-mon[32255]: pgmap v15956: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:28 smithi022 ceph-mon[31360]: pgmap v15956: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:28 smithi022 ceph-mon[40250]: pgmap v15956: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:29.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:50:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:50:30.306 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:30] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T02:50:30.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:30.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:30.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:30 smithi022 ceph-mon[31360]: pgmap v15957: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/536631443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:30 smithi022 ceph-mon[40250]: pgmap v15957: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/536631443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:30 smithi122 ceph-mon[32255]: pgmap v15957: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/536631443' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:50:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:50:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:32 smithi122 ceph-mon[32255]: pgmap v15958: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:32 smithi022 ceph-mon[31360]: pgmap v15958: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:32 smithi022 ceph-mon[40250]: pgmap v15958: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:34 smithi122 ceph-mon[32255]: pgmap v15959: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:50:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:50:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:50:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:50:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[31360]: pgmap v15959: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:50:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:50:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:50:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:50:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[40250]: pgmap v15959: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:50:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:50:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:50:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:50:35.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:36.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:36.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:36.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:36 smithi122 ceph-mon[32255]: pgmap v15960: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2408603697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:36 smithi022 ceph-mon[31360]: pgmap v15960: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2408603697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:36 smithi022 ceph-mon[40250]: pgmap v15960: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2408603697' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:50:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:38 smithi122 ceph-mon[32255]: pgmap v15961: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:38 smithi022 ceph-mon[31360]: pgmap v15961: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:38 smithi022 ceph-mon[40250]: pgmap v15961: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:39.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:50:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:50:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:40] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T02:50:41.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:41.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:41.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:40 smithi022 ceph-mon[31360]: pgmap v15962: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:41.047 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:41.047 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:40 smithi022 ceph-mon[40250]: pgmap v15962: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:40 smithi122 ceph-mon[32255]: pgmap v15962: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:41.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:41.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:41.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/387036731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/387036731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/387036731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:50:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:50:43.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:42 smithi122 ceph-mon[32255]: pgmap v15963: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:42 smithi022 ceph-mon[31360]: pgmap v15963: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:42 smithi022 ceph-mon[40250]: pgmap v15963: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:44 smithi122 ceph-mon[32255]: pgmap v15964: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:44 smithi022 ceph-mon[31360]: pgmap v15964: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:44 smithi022 ceph-mon[40250]: pgmap v15964: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:46.419 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:46.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:46.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:46.788 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:46 smithi122 ceph-mon[32255]: pgmap v15965: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3707704797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:46 smithi022 ceph-mon[31360]: pgmap v15965: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3707704797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:46 smithi022 ceph-mon[40250]: pgmap v15965: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3707704797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:50:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:48 smithi122 ceph-mon[32255]: pgmap v15966: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:48 smithi022 ceph-mon[31360]: pgmap v15966: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:48 smithi022 ceph-mon[40250]: pgmap v15966: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:49.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:50:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:50:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:50] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:50:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:50 smithi122 ceph-mon[32255]: pgmap v15967: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:50 smithi022 ceph-mon[31360]: pgmap v15967: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:50:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:50 smithi022 ceph-mon[40250]: pgmap v15967: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:51.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:52.161 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:52.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:52.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:50:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:50:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:52 smithi122 ceph-mon[32255]: pgmap v15968: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3831270078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:52 smithi022 ceph-mon[31360]: pgmap v15968: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3831270078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:52 smithi022 ceph-mon[40250]: pgmap v15968: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:53.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3831270078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:54 smithi122 ceph-mon[32255]: pgmap v15969: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:50:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:50:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:50:55.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:50:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[31360]: pgmap v15969: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:50:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:50:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:50:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:50:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[40250]: pgmap v15969: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:50:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:50:55.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:50:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:50:55.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:50:57.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:50:57.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:56 smithi022 ceph-mon[31360]: pgmap v15970: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:57.165 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:56 smithi022 ceph-mon[40250]: pgmap v15970: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:56 smithi122 ceph-mon[32255]: pgmap v15970: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:50:57.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:50:57.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:50:57.533 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:50:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2717982960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:50:58 smithi122 ceph-mon[32255]: pgmap v15971: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2717982960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:50:58 smithi022 ceph-mon[31360]: pgmap v15971: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:50:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2717982960' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:50:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:50:58 smithi022 ceph-mon[40250]: pgmap v15971: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:00.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:50:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:51:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:00] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:51:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:00 smithi122 ceph-mon[32255]: pgmap v15972: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:00 smithi022 ceph-mon[31360]: pgmap v15972: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:00 smithi022 ceph-mon[40250]: pgmap v15972: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:02.534 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:51:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:51:02.902 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:02.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:02.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:02 smithi122 ceph-mon[32255]: pgmap v15973: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:02 smithi022 ceph-mon[31360]: pgmap v15973: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:02 smithi022 ceph-mon[40250]: pgmap v15973: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/331833823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/331833823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/331833823' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:04 smithi122 ceph-mon[32255]: pgmap v15974: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:04 smithi022 ceph-mon[31360]: pgmap v15974: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:04 smithi022 ceph-mon[40250]: pgmap v15974: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:06 smithi122 ceph-mon[32255]: pgmap v15975: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:06 smithi022 ceph-mon[31360]: pgmap v15975: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:06 smithi022 ceph-mon[40250]: pgmap v15975: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:51:07.904 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:08.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:08.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:08.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:08 smithi022 ceph-mon[31360]: pgmap v15976: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4215322595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:08 smithi022 ceph-mon[40250]: pgmap v15976: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4215322595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:08 smithi122 ceph-mon[32255]: pgmap v15976: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4215322595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:10.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:51:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:51:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:10] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:51:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:10 smithi122 ceph-mon[32255]: pgmap v15977: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:10 smithi022 ceph-mon[31360]: pgmap v15977: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:10 smithi022 ceph-mon[40250]: pgmap v15977: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:51:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:51:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:12 smithi122 ceph-mon[32255]: pgmap v15978: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:13.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:13.276 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:12 smithi022 ceph-mon[40250]: pgmap v15978: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:13.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:12 smithi022 ceph-mon[31360]: pgmap v15978: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:13.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:13.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:13.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2903132225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2903132225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2903132225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:14 smithi122 ceph-mon[32255]: pgmap v15979: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:14 smithi022 ceph-mon[31360]: pgmap v15979: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:14 smithi022 ceph-mon[40250]: pgmap v15979: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:16 smithi122 ceph-mon[32255]: pgmap v15980: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:16 smithi022 ceph-mon[31360]: pgmap v15980: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:16 smithi022 ceph-mon[40250]: pgmap v15980: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:51:18.646 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:18 smithi122 ceph-mon[32255]: pgmap v15981: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:18 smithi022 ceph-mon[31360]: pgmap v15981: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:18 smithi022 ceph-mon[40250]: pgmap v15981: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3984514247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3984514247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3984514247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:20.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:51:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:51:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:20] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:51:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:20 smithi122 ceph-mon[32255]: pgmap v15982: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:20 smithi022 ceph-mon[31360]: pgmap v15982: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:20 smithi022 ceph-mon[40250]: pgmap v15982: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:51:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:51:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:22 smithi122 ceph-mon[32255]: pgmap v15983: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:22 smithi022 ceph-mon[31360]: pgmap v15983: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:22 smithi022 ceph-mon[40250]: pgmap v15983: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:24.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:24.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:24.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:24.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:24 smithi122 ceph-mon[32255]: pgmap v15984: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3797872166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:24 smithi022 ceph-mon[31360]: pgmap v15984: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3797872166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:24 smithi022 ceph-mon[40250]: pgmap v15984: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3797872166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:26 smithi122 ceph-mon[32255]: pgmap v15985: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:26 smithi022 ceph-mon[31360]: pgmap v15985: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:26 smithi022 ceph-mon[40250]: pgmap v15985: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:51:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:28 smithi122 ceph-mon[32255]: pgmap v15986: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:28 smithi022 ceph-mon[31360]: pgmap v15986: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:28 smithi022 ceph-mon[40250]: pgmap v15986: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:29.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:29.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:29.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:29.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2763616929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2763616929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2763616929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:30.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:51:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:51:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:30] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:51:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:31.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:30 smithi122 ceph-mon[32255]: pgmap v15987: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:30 smithi022 ceph-mon[31360]: pgmap v15987: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:30 smithi022 ceph-mon[40250]: pgmap v15987: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:51:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:51:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:32 smithi122 ceph-mon[32255]: pgmap v15988: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:32 smithi022 ceph-mon[31360]: pgmap v15988: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:32 smithi022 ceph-mon[40250]: pgmap v15988: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:34.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[31360]: pgmap v15989: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:51:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:51:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:51:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:51:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[40250]: pgmap v15989: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:51:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:51:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:51:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:51:35.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:35.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:35.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:34 smithi122 ceph-mon[32255]: pgmap v15989: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:51:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:51:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:51:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:51:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1732937062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1732937062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1732937062' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:36 smithi122 ceph-mon[32255]: pgmap v15990: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:36 smithi022 ceph-mon[31360]: pgmap v15990: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:36 smithi022 ceph-mon[40250]: pgmap v15990: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:51:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:38 smithi122 ceph-mon[32255]: pgmap v15991: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:38 smithi022 ceph-mon[31360]: pgmap v15991: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:38 smithi022 ceph-mon[40250]: pgmap v15991: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:40.132 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:40.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:51:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:51:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:40] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T02:51:40.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:40.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:40.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:40 smithi122 ceph-mon[32255]: pgmap v15992: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1713162770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:40 smithi022 ceph-mon[31360]: pgmap v15992: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1713162770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:40 smithi022 ceph-mon[40250]: pgmap v15992: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:41.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1713162770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:51:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:51:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:42 smithi122 ceph-mon[32255]: pgmap v15993: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:42 smithi022 ceph-mon[31360]: pgmap v15993: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:42 smithi022 ceph-mon[40250]: pgmap v15993: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:44 smithi122 ceph-mon[32255]: pgmap v15994: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:44 smithi022 ceph-mon[31360]: pgmap v15994: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:44 smithi022 ceph-mon[40250]: pgmap v15994: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:45.502 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:45.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:45.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:45.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:46 smithi122 ceph-mon[32255]: pgmap v15995: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2214296441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:46 smithi022 ceph-mon[31360]: pgmap v15995: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2214296441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:46 smithi022 ceph-mon[40250]: pgmap v15995: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2214296441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:51:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:48 smithi122 ceph-mon[32255]: pgmap v15996: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:48 smithi022 ceph-mon[31360]: pgmap v15996: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:48 smithi022 ceph-mon[40250]: pgmap v15996: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:50.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:51:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:51:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:50] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T02:51:50.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:50 smithi122 ceph-mon[32255]: pgmap v15997: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:51.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:51.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:51.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:51.242 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:51.242 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:50 smithi022 ceph-mon[40250]: pgmap v15997: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:51.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:51:51.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:50 smithi022 ceph-mon[31360]: pgmap v15997: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3719453780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3719453780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3719453780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:51:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:51:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:52 smithi122 ceph-mon[32255]: pgmap v15998: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:52 smithi022 ceph-mon[31360]: pgmap v15998: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:52 smithi022 ceph-mon[40250]: pgmap v15998: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:54 smithi122 ceph-mon[32255]: pgmap v15999: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:51:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:51:55.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:51:55.257 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[31360]: pgmap v15999: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:55.258 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:51:55.258 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:51:55.258 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:51:55.258 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[40250]: pgmap v15999: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:51:55.259 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:51:55.259 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:51:55.259 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:51:56.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:51:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:51:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:56 smithi022 ceph-mon[31360]: pgmap v16000: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:51:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:56 smithi022 ceph-mon[40250]: pgmap v16000: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:56.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:51:56.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:51:56.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:51:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:51:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:56 smithi122 ceph-mon[32255]: pgmap v16000: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:51:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1444975813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1444975813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1444975813' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:51:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:51:58 smithi122 ceph-mon[32255]: pgmap v16001: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:51:58 smithi022 ceph-mon[31360]: pgmap v16001: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:51:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:51:58 smithi022 ceph-mon[40250]: pgmap v16001: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:00.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:51:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:52:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:00] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T02:52:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:00 smithi122 ceph-mon[32255]: pgmap v16002: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:00 smithi022 ceph-mon[31360]: pgmap v16002: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:00 smithi022 ceph-mon[40250]: pgmap v16002: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:01.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:52:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:52:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:02 smithi122 ceph-mon[32255]: pgmap v16003: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3112435524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:02 smithi022 ceph-mon[31360]: pgmap v16003: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3112435524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:02 smithi022 ceph-mon[40250]: pgmap v16003: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3112435524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:04 smithi122 ceph-mon[32255]: pgmap v16004: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:04 smithi022 ceph-mon[31360]: pgmap v16004: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:04 smithi022 ceph-mon[40250]: pgmap v16004: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:06.986 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:07.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:06 smithi122 ceph-mon[32255]: pgmap v16005: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:06 smithi022 ceph-mon[31360]: pgmap v16005: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:06 smithi022 ceph-mon[40250]: pgmap v16005: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:07.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:07.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:07.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:52:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2530916827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2530916827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2530916827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:08 smithi122 ceph-mon[32255]: pgmap v16006: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:08 smithi022 ceph-mon[31360]: pgmap v16006: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:08 smithi022 ceph-mon[40250]: pgmap v16006: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:10.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:52:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:52:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:10] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T02:52:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:10 smithi122 ceph-mon[32255]: pgmap v16007: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:10 smithi022 ceph-mon[31360]: pgmap v16007: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:10 smithi022 ceph-mon[40250]: pgmap v16007: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:12.354 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:52:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:52:12.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:12 smithi022 ceph-mon[31360]: pgmap v16008: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/54934314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:12 smithi022 ceph-mon[40250]: pgmap v16008: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/54934314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:12 smithi122 ceph-mon[32255]: pgmap v16008: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/54934314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:14 smithi122 ceph-mon[32255]: pgmap v16009: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:14 smithi022 ceph-mon[31360]: pgmap v16009: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:14 smithi022 ceph-mon[40250]: pgmap v16009: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:16 smithi122 ceph-mon[32255]: pgmap v16010: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:16 smithi022 ceph-mon[31360]: pgmap v16010: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:16 smithi022 ceph-mon[40250]: pgmap v16010: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:17.723 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:52:17.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:18.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:18.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:18.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:18.421 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:18 smithi022 ceph-mon[40250]: pgmap v16011: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:18.422 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3595651767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:18 smithi122 ceph-mon[32255]: pgmap v16011: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3595651767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:18 smithi022 ceph-mon[31360]: pgmap v16011: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3595651767' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:20.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:52:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:52:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:20] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:52:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:20 smithi122 ceph-mon[32255]: pgmap v16012: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:20 smithi022 ceph-mon[31360]: pgmap v16012: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:20 smithi022 ceph-mon[40250]: pgmap v16012: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:52:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:52:23.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:23.093 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:22 smithi022 ceph-mon[40250]: pgmap v16013: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:23.093 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:22 smithi022 ceph-mon[31360]: pgmap v16013: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:22 smithi122 ceph-mon[32255]: pgmap v16013: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:23.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:23.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:23.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3716171648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3716171648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3716171648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:24 smithi122 ceph-mon[32255]: pgmap v16014: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:24 smithi022 ceph-mon[31360]: pgmap v16014: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:24 smithi022 ceph-mon[40250]: pgmap v16014: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:26 smithi122 ceph-mon[32255]: pgmap v16015: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:26 smithi022 ceph-mon[31360]: pgmap v16015: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:26 smithi022 ceph-mon[40250]: pgmap v16015: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:52:28.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:28 smithi122 ceph-mon[32255]: pgmap v16016: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:28 smithi022 ceph-mon[31360]: pgmap v16016: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:28 smithi022 ceph-mon[40250]: pgmap v16016: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:28.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:28.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:28.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2189749442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2189749442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2189749442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:30.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:52:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:52:30.433 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:30] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:52:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:30 smithi122 ceph-mon[32255]: pgmap v16017: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:30 smithi022 ceph-mon[31360]: pgmap v16017: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:30 smithi022 ceph-mon[40250]: pgmap v16017: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:52:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:52:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:32 smithi122 ceph-mon[32255]: pgmap v16018: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:32 smithi022 ceph-mon[31360]: pgmap v16018: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:32 smithi022 ceph-mon[40250]: pgmap v16018: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:33.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:34.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:34 smithi122 ceph-mon[32255]: pgmap v16019: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:52:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:52:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:52:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:52:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/168462095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[31360]: pgmap v16019: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:52:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:52:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:52:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:52:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/168462095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[40250]: pgmap v16019: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:52:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:52:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:52:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:52:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/168462095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:36 smithi122 ceph-mon[32255]: pgmap v16020: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:36 smithi022 ceph-mon[31360]: pgmap v16020: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:36 smithi022 ceph-mon[40250]: pgmap v16020: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:52:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:38 smithi122 ceph-mon[32255]: pgmap v16021: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:38 smithi022 ceph-mon[31360]: pgmap v16021: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:38 smithi022 ceph-mon[40250]: pgmap v16021: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:39.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:39.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:52:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:52:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/304641354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/304641354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/304641354' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:40] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:52:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:40 smithi122 ceph-mon[32255]: pgmap v16022: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:40 smithi022 ceph-mon[31360]: pgmap v16022: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:40 smithi022 ceph-mon[40250]: pgmap v16022: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:52:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:52:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:42 smithi122 ceph-mon[32255]: pgmap v16023: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:42 smithi022 ceph-mon[31360]: pgmap v16023: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:42 smithi022 ceph-mon[40250]: pgmap v16023: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:44.575 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:44.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:44.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:44.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:44 smithi122 ceph-mon[32255]: pgmap v16024: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:44 smithi022 ceph-mon[31360]: pgmap v16024: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:44 smithi022 ceph-mon[40250]: pgmap v16024: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/63020212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/63020212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/63020212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:46 smithi122 ceph-mon[32255]: pgmap v16025: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:46 smithi022 ceph-mon[31360]: pgmap v16025: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:46 smithi022 ceph-mon[40250]: pgmap v16025: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:52:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:48 smithi122 ceph-mon[32255]: pgmap v16026: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:48 smithi022 ceph-mon[31360]: pgmap v16026: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:48 smithi022 ceph-mon[40250]: pgmap v16026: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:49.950 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:50.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:52:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:52:50.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:50.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:50.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:50] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T02:52:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:50 smithi122 ceph-mon[32255]: pgmap v16027: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:51.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/763359447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:50 smithi022 ceph-mon[31360]: pgmap v16027: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:51.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/763359447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:51.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:52:51.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:50 smithi022 ceph-mon[40250]: pgmap v16027: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:51.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/763359447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:52:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:52:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:52 smithi122 ceph-mon[32255]: pgmap v16028: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:52 smithi022 ceph-mon[31360]: pgmap v16028: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:52 smithi022 ceph-mon[40250]: pgmap v16028: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:54 smithi122 ceph-mon[32255]: pgmap v16029: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:54 smithi022 ceph-mon[31360]: pgmap v16029: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:54 smithi022 ceph-mon[40250]: pgmap v16029: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:52:55.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:52:55.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:52:55.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:52:55.692 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:52:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:52:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:52:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:52:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4038502539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:52:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:52:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:52:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:52:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4038502539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:55 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:52:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:52:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:52:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:52:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4038502539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:52:56.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:55 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:52:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:56 smithi122 ceph-mon[32255]: pgmap v16030: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:56 smithi022 ceph-mon[31360]: pgmap v16030: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:56 smithi022 ceph-mon[40250]: pgmap v16030: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:52:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:52:58 smithi122 ceph-mon[32255]: pgmap v16031: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:52:58 smithi022 ceph-mon[31360]: pgmap v16031: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:52:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:52:58 smithi022 ceph-mon[40250]: pgmap v16031: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:00.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:52:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:53:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:00] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T02:53:00.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:00 smithi022 ceph-mon[31360]: pgmap v16032: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:00 smithi022 ceph-mon[40250]: pgmap v16032: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:01.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:01.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:01.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:00 smithi122 ceph-mon[32255]: pgmap v16032: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2267051010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2267051010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2267051010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:53:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:53:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:02 smithi122 ceph-mon[32255]: pgmap v16033: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:02 smithi022 ceph-mon[31360]: pgmap v16033: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:02 smithi022 ceph-mon[40250]: pgmap v16033: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:04 smithi122 ceph-mon[32255]: pgmap v16034: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:04 smithi022 ceph-mon[31360]: pgmap v16034: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:04 smithi022 ceph-mon[40250]: pgmap v16034: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:06.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:06.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:06.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:06.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:06 smithi122 ceph-mon[32255]: pgmap v16035: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/164874153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:06 smithi022 ceph-mon[31360]: pgmap v16035: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/164874153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:06 smithi022 ceph-mon[40250]: pgmap v16035: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/164874153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:53:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:08 smithi122 ceph-mon[32255]: pgmap v16036: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:08 smithi022 ceph-mon[31360]: pgmap v16036: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:08 smithi022 ceph-mon[40250]: pgmap v16036: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:10.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:53:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:53:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:10] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T02:53:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:10 smithi122 ceph-mon[32255]: pgmap v16037: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:10 smithi022 ceph-mon[31360]: pgmap v16037: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:10 smithi022 ceph-mon[40250]: pgmap v16037: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:11.438 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:11.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:11.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3313139833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3313139833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3313139833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:53:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:53:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:12 smithi122 ceph-mon[32255]: pgmap v16038: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:12 smithi022 ceph-mon[31360]: pgmap v16038: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:12 smithi022 ceph-mon[40250]: pgmap v16038: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:14 smithi122 ceph-mon[32255]: pgmap v16039: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:14 smithi022 ceph-mon[31360]: pgmap v16039: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:14 smithi022 ceph-mon[40250]: pgmap v16039: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:16.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:17.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:17.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:17.180 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:17.180 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:16 smithi022 ceph-mon[31360]: pgmap v16040: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:17.181 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:16 smithi022 ceph-mon[40250]: pgmap v16040: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:16 smithi122 ceph-mon[32255]: pgmap v16040: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:53:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1546815837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1546815837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1546815837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:18 smithi122 ceph-mon[32255]: pgmap v16041: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:18 smithi022 ceph-mon[31360]: pgmap v16041: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:18 smithi022 ceph-mon[40250]: pgmap v16041: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:20.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:53:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:53:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:20] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T02:53:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:20 smithi122 ceph-mon[32255]: pgmap v16042: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:20 smithi022 ceph-mon[31360]: pgmap v16042: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:20 smithi022 ceph-mon[40250]: pgmap v16042: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:21.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:22.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:22.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:22.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:22.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:53:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:53:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:22 smithi122 ceph-mon[32255]: pgmap v16043: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1991505891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:22 smithi022 ceph-mon[31360]: pgmap v16043: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1991505891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:22 smithi022 ceph-mon[40250]: pgmap v16043: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:23.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1991505891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:24 smithi122 ceph-mon[32255]: pgmap v16044: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:24 smithi022 ceph-mon[31360]: pgmap v16044: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:24 smithi022 ceph-mon[40250]: pgmap v16044: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:26 smithi122 ceph-mon[32255]: pgmap v16045: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:26 smithi022 ceph-mon[31360]: pgmap v16045: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:26 smithi022 ceph-mon[40250]: pgmap v16045: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:27.552 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:53:27.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:27.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:27.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:28 smithi022 ceph-mon[31360]: pgmap v16046: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3850590832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:28 smithi022 ceph-mon[40250]: pgmap v16046: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3850590832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:28 smithi122 ceph-mon[32255]: pgmap v16046: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3850590832' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:30.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:53:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:53:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:30] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T02:53:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:30 smithi122 ceph-mon[32255]: pgmap v16047: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:30 smithi022 ceph-mon[31360]: pgmap v16047: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:30 smithi022 ceph-mon[40250]: pgmap v16047: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:53:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:53:32.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:32 smithi122 ceph-mon[32255]: pgmap v16048: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:32 smithi022 ceph-mon[31360]: pgmap v16048: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:32 smithi022 ceph-mon[40250]: pgmap v16048: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:33.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:33.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:33.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1971797520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1971797520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1971797520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:34 smithi122 ceph-mon[32255]: pgmap v16049: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:53:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:53:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:53:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:53:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[31360]: pgmap v16049: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:53:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:53:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:53:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:53:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[40250]: pgmap v16049: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:35.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:53:35.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:53:35.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:53:35.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:53:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:36 smithi122 ceph-mon[32255]: pgmap v16050: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:36 smithi022 ceph-mon[31360]: pgmap v16050: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:36 smithi022 ceph-mon[40250]: pgmap v16050: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:53:38.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:38.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:38.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:38.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:38.672 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:38 smithi022 ceph-mon[40250]: pgmap v16051: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:38.673 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:38 smithi022 ceph-mon[31360]: pgmap v16051: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:38 smithi122 ceph-mon[32255]: pgmap v16051: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2200212912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2200212912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:39.832 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2200212912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:40.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:53:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:53:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:40] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T02:53:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:40 smithi022 ceph-mon[31360]: pgmap v16052: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:40 smithi022 ceph-mon[40250]: pgmap v16052: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:40 smithi122 ceph-mon[32255]: pgmap v16052: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:53:42.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:53:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:42 smithi122 ceph-mon[32255]: pgmap v16053: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:42 smithi022 ceph-mon[31360]: pgmap v16053: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:42 smithi022 ceph-mon[40250]: pgmap v16053: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:44 smithi122 ceph-mon[32255]: pgmap v16054: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/745698535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:44 smithi022 ceph-mon[31360]: pgmap v16054: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/745698535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:44 smithi022 ceph-mon[40250]: pgmap v16054: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/745698535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:46 smithi122 ceph-mon[32255]: pgmap v16055: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:46 smithi022 ceph-mon[31360]: pgmap v16055: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:46 smithi022 ceph-mon[40250]: pgmap v16055: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:53:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:48 smithi022 ceph-mon[40250]: pgmap v16056: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:48 smithi022 ceph-mon[31360]: pgmap v16056: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:48 smithi122 ceph-mon[32255]: pgmap v16056: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:49.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:49.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:49.412 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:49.413 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2526582934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2526582934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:49.843 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2526582934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:50.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:53:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:53:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:50] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T02:53:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:50 smithi022 ceph-mon[31360]: pgmap v16057: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:50 smithi022 ceph-mon[40250]: pgmap v16057: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:50 smithi122 ceph-mon[32255]: pgmap v16057: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:53:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:53:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:53:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:52 smithi122 ceph-mon[32255]: pgmap v16058: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:52 smithi022 ceph-mon[31360]: pgmap v16058: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:52 smithi022 ceph-mon[40250]: pgmap v16058: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:54.414 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:53:54.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:53:54.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:53:54.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:53:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:54 smithi122 ceph-mon[32255]: pgmap v16059: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1674140522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:54 smithi022 ceph-mon[31360]: pgmap v16059: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1674140522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:54 smithi022 ceph-mon[40250]: pgmap v16059: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:53:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1674140522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:53:56.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:53:56.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:53:56.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:55 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:53:56.157 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:53:56.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:53:56.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:55 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:53:56.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:53:56.159 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:53:56.159 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:55 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:53:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:56 smithi122 ceph-mon[32255]: pgmap v16060: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[40250]: pgmap v16060: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[31360]: pgmap v16060: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:53:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:53:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:53:58 smithi022 ceph-mon[31360]: pgmap v16061: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:53:58 smithi022 ceph-mon[40250]: pgmap v16061: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:53:58 smithi122 ceph-mon[32255]: pgmap v16061: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:53:59.785 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:00.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:00.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:00.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:00.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:53:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:54:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:00] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T02:54:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:00 smithi122 ceph-mon[32255]: pgmap v16062: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:01.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/111833769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:00 smithi022 ceph-mon[31360]: pgmap v16062: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/111833769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:00 smithi022 ceph-mon[40250]: pgmap v16062: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/111833769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:54:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:54:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:02 smithi122 ceph-mon[32255]: pgmap v16063: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:02 smithi022 ceph-mon[31360]: pgmap v16063: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:02 smithi022 ceph-mon[40250]: pgmap v16063: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:05.156 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:05.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:04 smithi022 ceph-mon[31360]: pgmap v16064: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:05.157 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:04 smithi022 ceph-mon[40250]: pgmap v16064: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:04 smithi122 ceph-mon[32255]: pgmap v16064: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:05.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:05.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:05.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1708674564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1708674564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1708674564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:06 smithi122 ceph-mon[32255]: pgmap v16065: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:06 smithi022 ceph-mon[31360]: pgmap v16065: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:06 smithi022 ceph-mon[40250]: pgmap v16065: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:54:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:08 smithi022 ceph-mon[31360]: pgmap v16066: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:08 smithi022 ceph-mon[40250]: pgmap v16066: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:08 smithi122 ceph-mon[32255]: pgmap v16066: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:10.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:54:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:54:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:10] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T02:54:10.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:10.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:10.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:10.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:10 smithi122 ceph-mon[32255]: pgmap v16067: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:10 smithi022 ceph-mon[31360]: pgmap v16067: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:10 smithi022 ceph-mon[40250]: pgmap v16067: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3415886166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3415886166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3415886166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:54:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:54:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:12 smithi122 ceph-mon[32255]: pgmap v16068: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:12 smithi022 ceph-mon[31360]: pgmap v16068: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:12 smithi022 ceph-mon[40250]: pgmap v16068: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:14 smithi122 ceph-mon[32255]: pgmap v16069: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:14 smithi022 ceph-mon[31360]: pgmap v16069: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:14 smithi022 ceph-mon[40250]: pgmap v16069: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:15.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:16.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:16.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:16.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:16 smithi122 ceph-mon[32255]: pgmap v16070: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/991329171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:16 smithi022 ceph-mon[31360]: pgmap v16070: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/991329171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:16 smithi022 ceph-mon[40250]: pgmap v16070: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/991329171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:54:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:18 smithi022 ceph-mon[31360]: pgmap v16071: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:18 smithi022 ceph-mon[40250]: pgmap v16071: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:18 smithi122 ceph-mon[32255]: pgmap v16071: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:20.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:54:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:54:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:20] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:54:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:20 smithi122 ceph-mon[32255]: pgmap v16072: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:21.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:21.260 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:20 smithi022 ceph-mon[31360]: pgmap v16072: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:21.260 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:21.261 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:20 smithi022 ceph-mon[40250]: pgmap v16072: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:21.261 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:21.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:21.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:21.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3087429403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3087429403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3087429403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:54:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:54:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:22 smithi122 ceph-mon[32255]: pgmap v16073: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:22 smithi022 ceph-mon[31360]: pgmap v16073: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:22 smithi022 ceph-mon[40250]: pgmap v16073: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:24 smithi122 ceph-mon[32255]: pgmap v16074: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:24 smithi022 ceph-mon[31360]: pgmap v16074: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:24 smithi022 ceph-mon[40250]: pgmap v16074: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:26.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:27.000 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:26 smithi022 ceph-mon[31360]: pgmap v16075: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:27.000 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:26 smithi022 ceph-mon[40250]: pgmap v16075: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:26 smithi122 ceph-mon[32255]: pgmap v16075: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:54:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/225870134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/225870134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/225870134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:28 smithi122 ceph-mon[32255]: pgmap v16076: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:28 smithi022 ceph-mon[31360]: pgmap v16076: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:28 smithi022 ceph-mon[40250]: pgmap v16076: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:30.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:54:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:54:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:30] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:54:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:30 smithi122 ceph-mon[32255]: pgmap v16077: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:30 smithi022 ceph-mon[31360]: pgmap v16077: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:30 smithi022 ceph-mon[40250]: pgmap v16077: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:32.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:32.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:32.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:32.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:54:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:54:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:32 smithi122 ceph-mon[32255]: pgmap v16078: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:33.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3466237413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:33.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:32 smithi022 ceph-mon[40250]: pgmap v16078: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3466237413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:32 smithi022 ceph-mon[31360]: pgmap v16078: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3466237413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:34 smithi122 ceph-mon[32255]: pgmap v16079: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:54:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:54:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:54:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:54:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[31360]: pgmap v16079: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:54:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:54:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:54:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:54:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[40250]: pgmap v16079: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:54:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:54:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:54:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:54:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:36 smithi122 ceph-mon[32255]: pgmap v16080: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:36 smithi022 ceph-mon[31360]: pgmap v16080: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:36 smithi022 ceph-mon[40250]: pgmap v16080: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:37.368 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:37.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:37.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:37.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:37.740 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:54:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/141301208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:38 smithi022 ceph-mon[31360]: pgmap v16081: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/141301208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:38 smithi022 ceph-mon[40250]: pgmap v16081: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/141301208' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:38 smithi122 ceph-mon[32255]: pgmap v16081: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:40.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:54:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:54:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:40] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T02:54:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:40 smithi122 ceph-mon[32255]: pgmap v16082: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:40 smithi022 ceph-mon[31360]: pgmap v16082: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:40 smithi022 ceph-mon[40250]: pgmap v16082: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:54:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:54:42.741 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:42 smithi022 ceph-mon[31360]: pgmap v16083: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:42 smithi022 ceph-mon[40250]: pgmap v16083: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:43.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:43.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:43.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:42 smithi122 ceph-mon[32255]: pgmap v16083: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/553839196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/553839196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/553839196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:44 smithi122 ceph-mon[32255]: pgmap v16084: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:44 smithi022 ceph-mon[31360]: pgmap v16084: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:44 smithi022 ceph-mon[40250]: pgmap v16084: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:46 smithi122 ceph-mon[32255]: pgmap v16085: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:46 smithi022 ceph-mon[31360]: pgmap v16085: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:46 smithi022 ceph-mon[40250]: pgmap v16085: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:54:48.114 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:48.453 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:48 smithi022 ceph-mon[31360]: pgmap v16086: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:48.453 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:48 smithi022 ceph-mon[40250]: pgmap v16086: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:48.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:48.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:48.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:48 smithi122 ceph-mon[32255]: pgmap v16086: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1748503826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1748503826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:49.896 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1748503826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:50.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:54:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:54:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:50] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T02:54:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:50 smithi022 ceph-mon[31360]: pgmap v16087: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:50 smithi022 ceph-mon[40250]: pgmap v16087: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:50 smithi122 ceph-mon[32255]: pgmap v16087: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:54:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:54:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:54:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:52 smithi122 ceph-mon[32255]: pgmap v16088: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:52 smithi022 ceph-mon[31360]: pgmap v16088: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:52 smithi022 ceph-mon[40250]: pgmap v16088: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:53.483 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:53.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:53.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:53.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/485754306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/485754306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/485754306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:54 smithi122 ceph-mon[32255]: pgmap v16089: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:54 smithi022 ceph-mon[31360]: pgmap v16089: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:54 smithi022 ceph-mon[40250]: pgmap v16089: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:54:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:56 smithi122 ceph-mon[32255]: pgmap v16090: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:54:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:54:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:56 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:54:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[31360]: pgmap v16090: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:54:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:54:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:54:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[40250]: pgmap v16090: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:54:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:54:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:56 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:54:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:54:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:54:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:58 smithi122 ceph-mon[32255]: pgmap v16091: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:54:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:58 smithi022 ceph-mon[31360]: pgmap v16091: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:54:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:58 smithi022 ceph-mon[40250]: pgmap v16091: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:54:58.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:54:59.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:54:59.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:54:59.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:54:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:54:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1553707297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:54:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1553707297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:54:59.907 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:54:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1553707297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:00.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:54:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:55:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:00] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T02:55:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:00 smithi022 ceph-mon[31360]: pgmap v16092: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:00 smithi022 ceph-mon[40250]: pgmap v16092: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:00 smithi122 ceph-mon[32255]: pgmap v16092: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:55:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:55:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:02 smithi122 ceph-mon[32255]: pgmap v16093: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:02 smithi022 ceph-mon[31360]: pgmap v16093: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:02 smithi022 ceph-mon[40250]: pgmap v16093: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:04.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:04.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:04.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:04.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:04 smithi122 ceph-mon[32255]: pgmap v16094: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2110668140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:04 smithi022 ceph-mon[31360]: pgmap v16094: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2110668140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:04 smithi022 ceph-mon[40250]: pgmap v16094: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2110668140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:06 smithi122 ceph-mon[32255]: pgmap v16095: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:06 smithi022 ceph-mon[31360]: pgmap v16095: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:06 smithi022 ceph-mon[40250]: pgmap v16095: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:55:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:08 smithi022 ceph-mon[31360]: pgmap v16096: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:08 smithi022 ceph-mon[40250]: pgmap v16096: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:08 smithi122 ceph-mon[32255]: pgmap v16096: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:09.602 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:09.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:09.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:09.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:10.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:55:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:55:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:10] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T02:55:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:10 smithi122 ceph-mon[32255]: pgmap v16097: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:11.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1418596233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:10 smithi022 ceph-mon[31360]: pgmap v16097: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:11.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1418596233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:10 smithi022 ceph-mon[40250]: pgmap v16097: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:11.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1418596233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:55:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:55:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:12 smithi122 ceph-mon[32255]: pgmap v16098: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:12 smithi022 ceph-mon[31360]: pgmap v16098: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:12 smithi022 ceph-mon[40250]: pgmap v16098: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:14 smithi122 ceph-mon[32255]: pgmap v16099: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:14 smithi022 ceph-mon[31360]: pgmap v16099: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:14 smithi022 ceph-mon[40250]: pgmap v16099: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:15.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:15.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:15.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/837307333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/837307333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/837307333' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:16 smithi122 ceph-mon[32255]: pgmap v16100: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:16 smithi022 ceph-mon[31360]: pgmap v16100: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:16 smithi022 ceph-mon[40250]: pgmap v16100: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:55:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:18 smithi022 ceph-mon[31360]: pgmap v16101: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:18 smithi022 ceph-mon[40250]: pgmap v16101: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:18 smithi122 ceph-mon[32255]: pgmap v16101: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:20.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:55:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:55:20.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:20] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:55:20.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:20.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:20.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:20 smithi022 ceph-mon[31360]: pgmap v16102: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3031497207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:20 smithi022 ceph-mon[40250]: pgmap v16102: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3031497207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:20 smithi122 ceph-mon[32255]: pgmap v16102: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:21.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3031497207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:55:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:55:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:22 smithi122 ceph-mon[32255]: pgmap v16103: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:22 smithi022 ceph-mon[31360]: pgmap v16103: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:22 smithi022 ceph-mon[40250]: pgmap v16103: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:24 smithi122 ceph-mon[32255]: pgmap v16104: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:24 smithi022 ceph-mon[31360]: pgmap v16104: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:24 smithi022 ceph-mon[40250]: pgmap v16104: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:25.713 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:26.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:26.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:26.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:26 smithi122 ceph-mon[32255]: pgmap v16105: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4039448307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:26 smithi022 ceph-mon[31360]: pgmap v16105: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4039448307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:26 smithi022 ceph-mon[40250]: pgmap v16105: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4039448307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:55:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:28 smithi022 ceph-mon[31360]: pgmap v16106: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:28.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:28 smithi022 ceph-mon[40250]: pgmap v16106: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:28 smithi122 ceph-mon[32255]: pgmap v16106: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:30.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:30] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T02:55:30.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:55:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:55:31.085 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:30 smithi122 ceph-mon[32255]: pgmap v16107: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:30 smithi022 ceph-mon[31360]: pgmap v16107: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:30 smithi022 ceph-mon[40250]: pgmap v16107: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:31.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:31.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:31.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/901400865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/901400865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/901400865' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:55:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:55:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:32 smithi122 ceph-mon[32255]: pgmap v16108: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:32 smithi022 ceph-mon[31360]: pgmap v16108: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:32 smithi022 ceph-mon[40250]: pgmap v16108: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:34 smithi122 ceph-mon[32255]: pgmap v16109: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:55:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:55:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:55:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:55:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[31360]: pgmap v16109: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:55:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:55:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:55:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:55:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[40250]: pgmap v16109: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:55:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:55:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:55:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:55:36.456 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:36.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:36.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:36.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:36 smithi122 ceph-mon[32255]: pgmap v16110: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1789519310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:36 smithi022 ceph-mon[31360]: pgmap v16110: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1789519310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:36 smithi022 ceph-mon[40250]: pgmap v16110: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1789519310' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:55:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:38 smithi022 ceph-mon[31360]: pgmap v16111: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:38 smithi022 ceph-mon[40250]: pgmap v16111: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:38 smithi122 ceph-mon[32255]: pgmap v16111: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:40] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T02:55:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:55:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:55:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:40 smithi122 ceph-mon[32255]: pgmap v16112: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:41.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:40 smithi022 ceph-mon[31360]: pgmap v16112: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:40 smithi022 ceph-mon[40250]: pgmap v16112: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:41.825 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:42.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:42.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:42.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:55:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:55:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:42 smithi122 ceph-mon[32255]: pgmap v16113: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3064906781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:42 smithi022 ceph-mon[31360]: pgmap v16113: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3064906781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:42 smithi022 ceph-mon[40250]: pgmap v16113: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3064906781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:44 smithi122 ceph-mon[32255]: pgmap v16114: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:44 smithi022 ceph-mon[31360]: pgmap v16114: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:44 smithi022 ceph-mon[40250]: pgmap v16114: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:46 smithi122 ceph-mon[32255]: pgmap v16115: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:47.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:47.193 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:46 smithi022 ceph-mon[40250]: pgmap v16115: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:47.193 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:46 smithi022 ceph-mon[31360]: pgmap v16115: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:55:47.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:47.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:47.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1067816514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:48 smithi022 ceph-mon[40250]: pgmap v16116: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1067816514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:48 smithi022 ceph-mon[31360]: pgmap v16116: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1067816514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:48 smithi122 ceph-mon[32255]: pgmap v16116: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:55:50.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:55:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:55:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:50 smithi122 ceph-mon[32255]: pgmap v16117: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:50 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:50 smithi022 ceph-mon[31360]: pgmap v16117: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:50 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:50 smithi022 ceph-mon[40250]: pgmap v16117: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:50 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:55:52.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:55:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:55:52.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:52.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:52.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:52 smithi122 ceph-mon[32255]: pgmap v16118: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:52 smithi022 ceph-mon[31360]: pgmap v16118: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:52 smithi022 ceph-mon[40250]: pgmap v16118: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/569486140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/569486140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/569486140' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:54 smithi122 ceph-mon[32255]: pgmap v16119: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:54 smithi022 ceph-mon[31360]: pgmap v16119: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:54 smithi022 ceph-mon[40250]: pgmap v16119: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:55:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:56 smithi122 ceph-mon[32255]: pgmap v16120: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:56 smithi022 ceph-mon[31360]: pgmap v16120: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:56 smithi022 ceph-mon[40250]: pgmap v16120: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:55:57.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:55:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:55:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:55:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:55:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:55:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:55:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:55:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:55:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:55:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:55:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:55:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:55:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:55:58.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:55:58.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:55:58.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:55:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:58 smithi122 ceph-mon[32255]: pgmap v16121: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:55:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806667271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:58 smithi022 ceph-mon[31360]: pgmap v16121: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:55:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806667271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:55:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:58 smithi022 ceph-mon[40250]: pgmap v16121: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:55:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:55:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806667271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T02:56:00.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:55:59 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:56:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:00 smithi122 ceph-mon[32255]: pgmap v16122: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:00 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:00 smithi022 ceph-mon[31360]: pgmap v16122: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:00 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:00 smithi022 ceph-mon[40250]: pgmap v16122: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:00 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:56:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:56:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:02 smithi122 ceph-mon[32255]: pgmap v16123: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:02 smithi022 ceph-mon[31360]: pgmap v16123: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:02 smithi022 ceph-mon[40250]: pgmap v16123: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:03.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/578679927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/578679927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/578679927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:04 smithi122 ceph-mon[32255]: pgmap v16124: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:04 smithi022 ceph-mon[31360]: pgmap v16124: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:04 smithi022 ceph-mon[40250]: pgmap v16124: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:06 smithi122 ceph-mon[32255]: pgmap v16125: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:07.309 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:06 smithi022 ceph-mon[31360]: pgmap v16125: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:07.309 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:06 smithi022 ceph-mon[40250]: pgmap v16125: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:56:08.686 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:08 smithi022 ceph-mon[31360]: pgmap v16126: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:08 smithi022 ceph-mon[40250]: pgmap v16126: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:08 smithi122 ceph-mon[32255]: pgmap v16126: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:09.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:09.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:09.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1519489660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1519489660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1519489660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:10] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:56:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:56:09 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:56:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:10 smithi022 ceph-mon[31360]: pgmap v16127: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:10 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:10 smithi022 ceph-mon[40250]: pgmap v16127: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:10 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:10 smithi122 ceph-mon[32255]: pgmap v16127: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:10 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:56:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:56:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:12 smithi122 ceph-mon[32255]: pgmap v16128: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:12 smithi022 ceph-mon[31360]: pgmap v16128: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:12 smithi022 ceph-mon[40250]: pgmap v16128: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:14.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:14.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:14 smithi122 ceph-mon[32255]: pgmap v16129: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1672361448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:14 smithi022 ceph-mon[31360]: pgmap v16129: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1672361448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:14 smithi022 ceph-mon[40250]: pgmap v16129: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1672361448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:16 smithi122 ceph-mon[32255]: pgmap v16130: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:16 smithi022 ceph-mon[31360]: pgmap v16130: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:16 smithi022 ceph-mon[40250]: pgmap v16130: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:56:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:18 smithi022 ceph-mon[31360]: pgmap v16131: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:18 smithi022 ceph-mon[40250]: pgmap v16131: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:18 smithi122 ceph-mon[32255]: pgmap v16131: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:19.429 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:19.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:19.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:19.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2767174212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:20.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:56:19 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:56:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2767174212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2767174212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:20] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T02:56:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:20 smithi122 ceph-mon[32255]: pgmap v16132: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:20 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:20 smithi022 ceph-mon[31360]: pgmap v16132: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:21.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:20 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:20 smithi022 ceph-mon[40250]: pgmap v16132: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:20 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:56:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:56:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:22 smithi122 ceph-mon[32255]: pgmap v16133: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:22 smithi022 ceph-mon[31360]: pgmap v16133: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:22 smithi022 ceph-mon[40250]: pgmap v16133: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:24.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:25.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:25.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:25.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:25.173 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:24 smithi022 ceph-mon[31360]: pgmap v16134: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:25.173 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:24 smithi022 ceph-mon[40250]: pgmap v16134: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:24 smithi122 ceph-mon[32255]: pgmap v16134: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1377658030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1377658030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1377658030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:26 smithi122 ceph-mon[32255]: pgmap v16135: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:26 smithi022 ceph-mon[31360]: pgmap v16135: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:26 smithi022 ceph-mon[40250]: pgmap v16135: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:56:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:28 smithi022 ceph-mon[31360]: pgmap v16136: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:28 smithi022 ceph-mon[40250]: pgmap v16136: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:28 smithi122 ceph-mon[32255]: pgmap v16136: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:30.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:30.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:56:29 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:56:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:30] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T02:56:30.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:30.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:30.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:30 smithi122 ceph-mon[32255]: pgmap v16137: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:30 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/639555891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:30 smithi022 ceph-mon[31360]: pgmap v16137: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:30 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/639555891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:30 smithi022 ceph-mon[40250]: pgmap v16137: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:30 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:31.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/639555891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:56:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:56:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:32 smithi122 ceph-mon[32255]: pgmap v16138: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:32 smithi022 ceph-mon[31360]: pgmap v16138: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:32 smithi022 ceph-mon[40250]: pgmap v16138: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:34 smithi122 ceph-mon[32255]: pgmap v16139: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:56:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:56:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:56:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:56:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[31360]: pgmap v16139: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:56:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:56:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:56:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:56:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[40250]: pgmap v16139: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:56:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:56:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:56:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:56:35.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1927446293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1927446293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1927446293' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:36 smithi122 ceph-mon[32255]: pgmap v16140: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:36 smithi022 ceph-mon[31360]: pgmap v16140: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:36 smithi022 ceph-mon[40250]: pgmap v16140: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:56:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:38 smithi022 ceph-mon[31360]: pgmap v16141: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:38 smithi022 ceph-mon[40250]: pgmap v16141: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:38 smithi122 ceph-mon[32255]: pgmap v16141: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:40] "GET /metrics HTTP/1.1" 200 202472 "" "Prometheus/2.18.1" 2021-12-16T02:56:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:56:39 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:56:40.909 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:40 smithi122 ceph-mon[32255]: pgmap v16142: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:40 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:40 smithi022 ceph-mon[31360]: pgmap v16142: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:40 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:40 smithi022 ceph-mon[40250]: pgmap v16142: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:40 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:41.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:41.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:41.280 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4250947324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4250947324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4250947324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:56:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:56:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:42 smithi122 ceph-mon[32255]: pgmap v16143: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:42 smithi022 ceph-mon[31360]: pgmap v16143: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:42 smithi022 ceph-mon[40250]: pgmap v16143: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:44 smithi122 ceph-mon[32255]: pgmap v16144: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:44 smithi022 ceph-mon[31360]: pgmap v16144: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:44 smithi022 ceph-mon[40250]: pgmap v16144: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:46.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:46.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:46.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:46.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:46 smithi122 ceph-mon[32255]: pgmap v16145: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3793436235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:46 smithi022 ceph-mon[31360]: pgmap v16145: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3793436235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:46 smithi022 ceph-mon[40250]: pgmap v16145: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3793436235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:56:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:48 smithi022 ceph-mon[31360]: pgmap v16146: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:48 smithi022 ceph-mon[40250]: pgmap v16146: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:48 smithi122 ceph-mon[32255]: pgmap v16146: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:50] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:56:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:56:49 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:56:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:50 smithi122 ceph-mon[32255]: pgmap v16147: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:50 smithi022 ceph-mon[31360]: pgmap v16147: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:50 smithi022 ceph-mon[40250]: pgmap v16147: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:51.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:52.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:52.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:52.017 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:56:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:56:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:56:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:52 smithi122 ceph-mon[32255]: pgmap v16148: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2111381198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:52 smithi022 ceph-mon[31360]: pgmap v16148: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2111381198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:52 smithi022 ceph-mon[40250]: pgmap v16148: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2111381198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:54 smithi122 ceph-mon[32255]: pgmap v16149: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:54 smithi022 ceph-mon[31360]: pgmap v16149: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:54 smithi022 ceph-mon[40250]: pgmap v16149: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:56:57.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:56:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:56 smithi122 ceph-mon[32255]: pgmap v16150: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:56 smithi022 ceph-mon[31360]: pgmap v16150: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:56 smithi022 ceph-mon[40250]: pgmap v16150: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:57.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:56:57.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:56:57.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:56:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:56:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1432027127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:56:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:56:58.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:56:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1432027127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:56:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:56:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:56:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1432027127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:56:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:56:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:56:58.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:56:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:58 smithi122 ceph-mon[32255]: pgmap v16151: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:56:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:56:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:58 smithi022 ceph-mon[31360]: pgmap v16151: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:59.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:56:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:56:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:58 smithi022 ceph-mon[40250]: pgmap v16151: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:56:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:56:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:57:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:00] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T02:57:00.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:57:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:57:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:00 smithi122 ceph-mon[32255]: pgmap v16152: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:00 smithi022 ceph-mon[31360]: pgmap v16152: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:00 smithi022 ceph-mon[40250]: pgmap v16152: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:02.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:57:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:57:02.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:02.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:02.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:02 smithi022 ceph-mon[31360]: pgmap v16153: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2398185963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:02 smithi022 ceph-mon[40250]: pgmap v16153: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2398185963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:02 smithi122 ceph-mon[32255]: pgmap v16153: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2398185963' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:04 smithi022 ceph-mon[31360]: pgmap v16154: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:04 smithi022 ceph-mon[40250]: pgmap v16154: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:04 smithi122 ceph-mon[32255]: pgmap v16154: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:06 smithi022 ceph-mon[31360]: pgmap v16155: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:06 smithi022 ceph-mon[40250]: pgmap v16155: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:06 smithi122 ceph-mon[32255]: pgmap v16155: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:07.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:07.761 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:57:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:08 smithi022 ceph-mon[31360]: pgmap v16156: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4184822605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:08 smithi022 ceph-mon[40250]: pgmap v16156: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4184822605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:08 smithi122 ceph-mon[32255]: pgmap v16156: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4184822605' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:10] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T02:57:10.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:57:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:57:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:10 smithi122 ceph-mon[32255]: pgmap v16157: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:10 smithi022 ceph-mon[31360]: pgmap v16157: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:10 smithi022 ceph-mon[40250]: pgmap v16157: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:57:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:57:13.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:12 smithi122 ceph-mon[32255]: pgmap v16158: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:12 smithi022 ceph-mon[31360]: pgmap v16158: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:12 smithi022 ceph-mon[40250]: pgmap v16158: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:13.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:13.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:13.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3670813575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3670813575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3670813575' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:14 smithi122 ceph-mon[32255]: pgmap v16159: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:14 smithi022 ceph-mon[31360]: pgmap v16159: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:14 smithi022 ceph-mon[40250]: pgmap v16159: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:16 smithi122 ceph-mon[32255]: pgmap v16160: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:16 smithi022 ceph-mon[31360]: pgmap v16160: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:16 smithi022 ceph-mon[40250]: pgmap v16160: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:57:18.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:18 smithi022 ceph-mon[31360]: pgmap v16161: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:18 smithi022 ceph-mon[40250]: pgmap v16161: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:18.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:18.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:18.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:18 smithi122 ceph-mon[32255]: pgmap v16161: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3605730332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3605730332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3605730332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:20] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T02:57:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:57:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:57:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:20 smithi022 ceph-mon[31360]: pgmap v16162: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:20 smithi022 ceph-mon[40250]: pgmap v16162: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:20 smithi122 ceph-mon[32255]: pgmap v16162: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:22 smithi122 ceph-mon[32255]: pgmap v16163: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:57:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:57:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:22 smithi022 ceph-mon[31360]: pgmap v16163: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:22 smithi022 ceph-mon[40250]: pgmap v16163: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:23.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:24.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:24.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:24.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:24 smithi122 ceph-mon[32255]: pgmap v16164: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/870823189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:24 smithi022 ceph-mon[31360]: pgmap v16164: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:25.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/870823189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:24 smithi022 ceph-mon[40250]: pgmap v16164: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/870823189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:26 smithi122 ceph-mon[32255]: pgmap v16165: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:26 smithi022 ceph-mon[31360]: pgmap v16165: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:26 smithi022 ceph-mon[40250]: pgmap v16165: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:57:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:28 smithi022 ceph-mon[31360]: pgmap v16166: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:28 smithi022 ceph-mon[40250]: pgmap v16166: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:28 smithi122 ceph-mon[32255]: pgmap v16166: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:29.246 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/456488079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/456488079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/456488079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:30] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T02:57:30.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:57:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:57:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:30 smithi122 ceph-mon[32255]: pgmap v16167: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:30 smithi022 ceph-mon[31360]: pgmap v16167: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:30 smithi022 ceph-mon[40250]: pgmap v16167: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:32.668 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:57:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:57:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:32 smithi122 ceph-mon[32255]: pgmap v16168: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:32 smithi022 ceph-mon[31360]: pgmap v16168: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:32 smithi022 ceph-mon[40250]: pgmap v16168: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:34.617 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:34 smithi122 ceph-mon[32255]: pgmap v16169: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:57:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:57:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:57:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:57:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[31360]: pgmap v16169: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:57:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:57:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:57:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:57:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[40250]: pgmap v16169: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:57:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:57:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:57:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:57:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/370695522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/370695522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/370695522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:36 smithi122 ceph-mon[32255]: pgmap v16170: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:36 smithi022 ceph-mon[31360]: pgmap v16170: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:36 smithi022 ceph-mon[40250]: pgmap v16170: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:57:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:38 smithi022 ceph-mon[31360]: pgmap v16171: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:38 smithi022 ceph-mon[40250]: pgmap v16171: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:38 smithi122 ceph-mon[32255]: pgmap v16171: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:39.991 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:40.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:40.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:40.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:40.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:57:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:57:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:40] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T02:57:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:40 smithi122 ceph-mon[32255]: pgmap v16172: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:41.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3963654938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:40 smithi022 ceph-mon[31360]: pgmap v16172: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3963654938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:40 smithi022 ceph-mon[40250]: pgmap v16172: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3963654938' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:57:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:57:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:42 smithi122 ceph-mon[32255]: pgmap v16173: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:42 smithi022 ceph-mon[31360]: pgmap v16173: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:42 smithi022 ceph-mon[40250]: pgmap v16173: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:44 smithi122 ceph-mon[32255]: pgmap v16174: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:44 smithi022 ceph-mon[31360]: pgmap v16174: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:44 smithi022 ceph-mon[40250]: pgmap v16174: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:45.363 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:45.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:45.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:45.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2091856291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2091856291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2091856291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:46 smithi022 ceph-mon[31360]: pgmap v16175: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:46 smithi022 ceph-mon[40250]: pgmap v16175: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:46 smithi122 ceph-mon[32255]: pgmap v16175: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:57:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:48 smithi022 ceph-mon[31360]: pgmap v16176: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:48 smithi022 ceph-mon[40250]: pgmap v16176: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:48 smithi122 ceph-mon[32255]: pgmap v16176: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:50] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:57:50.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:57:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:57:50.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:50 smithi022 ceph-mon[31360]: pgmap v16177: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:50 smithi022 ceph-mon[40250]: pgmap v16177: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:51.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:51.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:51.101 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:50 smithi122 ceph-mon[32255]: pgmap v16177: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1620337981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:52.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1620337981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:57:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1620337981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:57:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:57:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:52 smithi022 ceph-mon[31360]: pgmap v16178: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:52 smithi022 ceph-mon[40250]: pgmap v16178: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:52 smithi122 ceph-mon[32255]: pgmap v16178: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:54 smithi022 ceph-mon[31360]: pgmap v16179: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:54 smithi022 ceph-mon[40250]: pgmap v16179: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:54 smithi122 ceph-mon[32255]: pgmap v16179: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:57:56.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:57:56.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:57:56.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:57:56.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:57:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:56 smithi022 ceph-mon[31360]: pgmap v16180: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2171990008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:56 smithi022 ceph-mon[40250]: pgmap v16180: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2171990008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:56 smithi122 ceph-mon[32255]: pgmap v16180: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2171990008' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:57:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:57:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[31360]: pgmap v16181: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:57:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:57:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:57:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[40250]: pgmap v16181: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:57:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:57:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:57:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:57:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:58 smithi122 ceph-mon[32255]: pgmap v16181: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:57:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:57:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:57:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:57:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:58:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:00] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:58:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:58:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:58:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:00 smithi122 ceph-mon[32255]: pgmap v16182: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:00 smithi022 ceph-mon[31360]: pgmap v16182: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:00 smithi022 ceph-mon[40250]: pgmap v16182: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:01.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:01.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:01.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/947492455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/947492455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:02.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:02.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:02.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/947492455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:02.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:02.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:58:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:58:03.165 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:02 smithi122 ceph-mon[32255]: pgmap v16183: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:03.166 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:02 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2907M 2021-12-16T02:58:03.166 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:02 smithi022 ceph-mon[31360]: pgmap v16183: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:02 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2907M 2021-12-16T02:58:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:02 smithi022 ceph-mon[40250]: pgmap v16183: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:02 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2907M 2021-12-16T02:58:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:04.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T02:58:04.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:04.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:58:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:04 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3191M 2021-12-16T02:58:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:04 smithi022 ceph-mon[31360]: pgmap v16184: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:04 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3191M 2021-12-16T02:58:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:04 smithi022 ceph-mon[40250]: pgmap v16184: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:04 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3191M 2021-12-16T02:58:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:04 smithi122 ceph-mon[32255]: pgmap v16184: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:06.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:07.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:07.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:07.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:07.223 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:06 smithi022 ceph-mon[31360]: pgmap v16185: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:07.223 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:06 smithi022 ceph-mon[40250]: pgmap v16185: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:06 smithi122 ceph-mon[32255]: pgmap v16185: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:58:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1602143011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1602143011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1602143011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:08 smithi022 ceph-mon[31360]: pgmap v16186: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:08 smithi022 ceph-mon[40250]: pgmap v16186: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:08 smithi122 ceph-mon[32255]: pgmap v16186: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:10.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:58:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:58:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:10] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T02:58:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:10 smithi022 ceph-mon[31360]: pgmap v16187: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:10 smithi022 ceph-mon[40250]: pgmap v16187: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:10 smithi122 ceph-mon[32255]: pgmap v16187: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T02:58:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:12.225 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:12.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:12.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:12.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:12.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:58:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:58:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:12 smithi022 ceph-mon[31360]: pgmap v16188: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2930742249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:12 smithi022 ceph-mon[40250]: pgmap v16188: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2930742249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:12 smithi122 ceph-mon[32255]: pgmap v16188: 132 pgs: 132 active+clean; 582 KiB data, 160 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2930742249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:14 smithi022 ceph-mon[31360]: pgmap v16189: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:14 smithi022 ceph-mon[40250]: pgmap v16189: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:14 smithi122 ceph-mon[32255]: pgmap v16189: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:16 smithi022 ceph-mon[31360]: pgmap v16190: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:16 smithi022 ceph-mon[40250]: pgmap v16190: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:16 smithi122 ceph-mon[32255]: pgmap v16190: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:17.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:58:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:17.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:18 smithi022 ceph-mon[31360]: pgmap v16191: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2278987369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:18 smithi022 ceph-mon[40250]: pgmap v16191: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2278987369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:18 smithi122 ceph-mon[32255]: pgmap v16191: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2278987369' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:20] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:58:20.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:58:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:58:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:20 smithi022 ceph-mon[31360]: pgmap v16192: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:20 smithi022 ceph-mon[40250]: pgmap v16192: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:20 smithi122 ceph-mon[32255]: pgmap v16192: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:58:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:58:22.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:22 smithi022 ceph-mon[31360]: pgmap v16193: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:22 smithi022 ceph-mon[40250]: pgmap v16193: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:23.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:23.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:23.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:22 smithi122 ceph-mon[32255]: pgmap v16193: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T02:58:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3509101895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3509101895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3509101895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:24 smithi022 ceph-mon[31360]: pgmap v16194: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:24 smithi022 ceph-mon[40250]: pgmap v16194: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:24 smithi122 ceph-mon[32255]: pgmap v16194: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T02:58:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:26 smithi022 ceph-mon[31360]: pgmap v16195: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:26 smithi022 ceph-mon[40250]: pgmap v16195: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:26 smithi122 ceph-mon[32255]: pgmap v16195: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:58:28.338 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:28.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:28.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:28.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:28.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:28 smithi022 ceph-mon[40250]: pgmap v16196: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:28.709 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:28 smithi022 ceph-mon[31360]: pgmap v16196: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:28 smithi122 ceph-mon[32255]: pgmap v16196: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2915560590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2915560590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2915560590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:30] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T02:58:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:58:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:58:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:30 smithi022 ceph-mon[31360]: pgmap v16197: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:30 smithi022 ceph-mon[40250]: pgmap v16197: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:30 smithi122 ceph-mon[32255]: pgmap v16197: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:32 smithi122 ceph-mon[32255]: pgmap v16198: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:58:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:58:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:32 smithi022 ceph-mon[31360]: pgmap v16198: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:32 smithi022 ceph-mon[40250]: pgmap v16198: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:33.710 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:34.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:34.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:34.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[31360]: pgmap v16199: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3185941235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:58:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:58:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:58:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:58:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[40250]: pgmap v16199: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3185941235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:58:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:58:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:58:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:58:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:34 smithi122 ceph-mon[32255]: pgmap v16199: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3185941235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:58:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:58:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:58:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:58:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:36 smithi022 ceph-mon[31360]: pgmap v16200: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:36 smithi022 ceph-mon[40250]: pgmap v16200: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:36 smithi122 ceph-mon[32255]: pgmap v16200: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:58:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:38 smithi022 ceph-mon[31360]: pgmap v16201: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:38 smithi022 ceph-mon[40250]: pgmap v16201: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:38 smithi122 ceph-mon[32255]: pgmap v16201: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:39.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:39.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:39.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:39.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3490126403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3490126403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3490126403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:40] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T02:58:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:58:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:58:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:40 smithi022 ceph-mon[31360]: pgmap v16202: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:40 smithi022 ceph-mon[40250]: pgmap v16202: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:40 smithi122 ceph-mon[32255]: pgmap v16202: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:42 smithi122 ceph-mon[32255]: pgmap v16203: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:58:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:58:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:42 smithi022 ceph-mon[31360]: pgmap v16203: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:42 smithi022 ceph-mon[40250]: pgmap v16203: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:44.450 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:44.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:44.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:44.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:44 smithi022 ceph-mon[31360]: pgmap v16204: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/128949730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:44 smithi022 ceph-mon[40250]: pgmap v16204: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/128949730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:44 smithi122 ceph-mon[32255]: pgmap v16204: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/128949730' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:46 smithi022 ceph-mon[31360]: pgmap v16205: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:46 smithi022 ceph-mon[40250]: pgmap v16205: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:46 smithi122 ceph-mon[32255]: pgmap v16205: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:58:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:48 smithi022 ceph-mon[31360]: pgmap v16206: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:48 smithi022 ceph-mon[40250]: pgmap v16206: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:48 smithi122 ceph-mon[32255]: pgmap v16206: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:49.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:50.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:50.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:50.187 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:50.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:58:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:58:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:50] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:58:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:50 smithi022 ceph-mon[31360]: pgmap v16207: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:51.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1495347815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:50 smithi022 ceph-mon[40250]: pgmap v16207: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1495347815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:50 smithi122 ceph-mon[32255]: pgmap v16207: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1495347815' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:58:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:58:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:58:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:52 smithi022 ceph-mon[31360]: pgmap v16208: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:52 smithi022 ceph-mon[40250]: pgmap v16208: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:52 smithi122 ceph-mon[32255]: pgmap v16208: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:55.188 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:58:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:54 smithi022 ceph-mon[40250]: pgmap v16209: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:54 smithi022 ceph-mon[31360]: pgmap v16209: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:54 smithi122 ceph-mon[32255]: pgmap v16209: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:58:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:58:55.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:58:55.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:58:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3416337950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3416337950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3416337950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:58:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:56 smithi022 ceph-mon[31360]: pgmap v16210: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:56 smithi022 ceph-mon[40250]: pgmap v16210: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:56 smithi122 ceph-mon[32255]: pgmap v16210: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:58:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:58:58 smithi022 ceph-mon[31360]: pgmap v16211: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:58:58 smithi022 ceph-mon[40250]: pgmap v16211: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:58:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:58:58 smithi122 ceph-mon[32255]: pgmap v16211: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:00] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T02:59:00.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:59:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:59:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:00.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:00.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:00.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:00 smithi022 ceph-mon[31360]: pgmap v16212: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/500853183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:00 smithi022 ceph-mon[40250]: pgmap v16212: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/500853183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:00 smithi122 ceph-mon[32255]: pgmap v16212: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/500853183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:59:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:59:03.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:02 smithi122 ceph-mon[32255]: pgmap v16213: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:03.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:02 smithi022 ceph-mon[31360]: pgmap v16213: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:03.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:02 smithi022 ceph-mon[40250]: pgmap v16213: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:59:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:59:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:59:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:59:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:59:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:59:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:59:04.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:03 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:59:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T02:59:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T02:59:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T02:59:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:03 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T02:59:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:04 smithi022 ceph-mon[31360]: pgmap v16214: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:04 smithi022 ceph-mon[40250]: pgmap v16214: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:04 smithi122 ceph-mon[32255]: pgmap v16214: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:05.934 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:06.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:06.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:06.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:06 smithi022 ceph-mon[31360]: pgmap v16215: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3996095657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:06 smithi022 ceph-mon[40250]: pgmap v16215: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3996095657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:06 smithi122 ceph-mon[32255]: pgmap v16215: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3996095657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:59:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:08 smithi022 ceph-mon[31360]: pgmap v16216: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:08 smithi022 ceph-mon[40250]: pgmap v16216: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:08 smithi122 ceph-mon[32255]: pgmap v16216: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:10] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:59:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:59:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:59:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:10 smithi022 ceph-mon[31360]: pgmap v16217: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:10 smithi022 ceph-mon[40250]: pgmap v16217: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:11.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:10 smithi122 ceph-mon[32255]: pgmap v16217: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:11.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:11.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:11.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1263704786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1263704786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1263704786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:59:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:59:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:12 smithi022 ceph-mon[31360]: pgmap v16218: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:12 smithi022 ceph-mon[40250]: pgmap v16218: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:12 smithi122 ceph-mon[32255]: pgmap v16218: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:14 smithi022 ceph-mon[31360]: pgmap v16219: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:14 smithi022 ceph-mon[40250]: pgmap v16219: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:14 smithi122 ceph-mon[32255]: pgmap v16219: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:16.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:17.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:17.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:17.051 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:16 smithi022 ceph-mon[31360]: pgmap v16220: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:16 smithi022 ceph-mon[40250]: pgmap v16220: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:16 smithi122 ceph-mon[32255]: pgmap v16220: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:59:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1271462160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1271462160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1271462160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:19.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:18 smithi022 ceph-mon[31360]: pgmap v16221: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:18 smithi022 ceph-mon[40250]: pgmap v16221: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:18 smithi122 ceph-mon[32255]: pgmap v16221: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:20] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:59:20.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:59:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:59:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:20 smithi022 ceph-mon[31360]: pgmap v16222: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:20 smithi022 ceph-mon[40250]: pgmap v16222: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:20 smithi122 ceph-mon[32255]: pgmap v16222: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:22.053 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:22.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:22.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:22.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:59:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:59:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:22 smithi022 ceph-mon[31360]: pgmap v16223: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1824880398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:22 smithi022 ceph-mon[40250]: pgmap v16223: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1824880398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:22 smithi122 ceph-mon[32255]: pgmap v16223: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1824880398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:24 smithi022 ceph-mon[31360]: pgmap v16224: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:24 smithi022 ceph-mon[40250]: pgmap v16224: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:24 smithi122 ceph-mon[32255]: pgmap v16224: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:26 smithi022 ceph-mon[31360]: pgmap v16225: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:26 smithi022 ceph-mon[40250]: pgmap v16225: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:27.422 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:26 smithi122 ceph-mon[32255]: pgmap v16225: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:59:27.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:27.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:27.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/162525097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:28 smithi022 ceph-mon[40250]: pgmap v16226: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/162525097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:28 smithi022 ceph-mon[31360]: pgmap v16226: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/162525097' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:28 smithi122 ceph-mon[32255]: pgmap v16226: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:30] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T02:59:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:59:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:59:31.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:30 smithi022 ceph-mon[31360]: pgmap v16227: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:30 smithi022 ceph-mon[40250]: pgmap v16227: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:30 smithi122 ceph-mon[32255]: pgmap v16227: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:59:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:59:32.796 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:33.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:33.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:33.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:33.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:32 smithi022 ceph-mon[31360]: pgmap v16228: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:33.168 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:32 smithi022 ceph-mon[40250]: pgmap v16228: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:32 smithi122 ceph-mon[32255]: pgmap v16228: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T02:59:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3068669995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3068669995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3068669995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[31360]: pgmap v16229: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:59:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:59:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:59:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:59:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[40250]: pgmap v16229: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:59:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:59:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:59:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:59:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:34 smithi122 ceph-mon[32255]: pgmap v16229: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:59:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:59:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T02:59:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T02:59:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:36 smithi022 ceph-mon[31360]: pgmap v16230: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:36 smithi022 ceph-mon[40250]: pgmap v16230: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:36 smithi122 ceph-mon[32255]: pgmap v16230: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:59:38.168 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:38.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:38 smithi022 ceph-mon[31360]: pgmap v16231: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:38.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:38.537 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:38.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:38 smithi022 ceph-mon[40250]: pgmap v16231: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:38 smithi122 ceph-mon[32255]: pgmap v16231: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:39.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3456181890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3456181890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3456181890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:40] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:59:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:59:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:59:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:40 smithi022 ceph-mon[40250]: pgmap v16232: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:40 smithi022 ceph-mon[31360]: pgmap v16232: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:40 smithi122 ceph-mon[32255]: pgmap v16232: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:42 smithi122 ceph-mon[32255]: pgmap v16233: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:59:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:59:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:42 smithi022 ceph-mon[31360]: pgmap v16233: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:42 smithi022 ceph-mon[40250]: pgmap v16233: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:43.539 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:43.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:43.910 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2554449510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2554449510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2554449510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:44 smithi022 ceph-mon[31360]: pgmap v16234: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:44 smithi022 ceph-mon[40250]: pgmap v16234: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:44 smithi122 ceph-mon[32255]: pgmap v16234: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:46 smithi022 ceph-mon[31360]: pgmap v16235: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:46 smithi022 ceph-mon[40250]: pgmap v16235: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:46 smithi122 ceph-mon[32255]: pgmap v16235: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:59:48.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:48 smithi022 ceph-mon[31360]: pgmap v16236: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:48 smithi022 ceph-mon[40250]: pgmap v16236: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:48.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:48 smithi122 ceph-mon[32255]: pgmap v16236: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:49.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:49.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:49.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1886298839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1886298839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1886298839' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 02:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:50] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T02:59:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 02:59:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T02:59:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:50 smithi022 ceph-mon[31360]: pgmap v16237: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:50 smithi022 ceph-mon[40250]: pgmap v16237: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:50 smithi122 ceph-mon[32255]: pgmap v16237: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T02:59:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:52 smithi122 ceph-mon[32255]: pgmap v16238: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 02:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T02:59:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T02:59:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:52 smithi022 ceph-mon[31360]: pgmap v16238: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:52 smithi022 ceph-mon[40250]: pgmap v16238: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:54.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T02:59:54.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T02:59:54.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T02:59:54.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T02:59:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:54 smithi022 ceph-mon[31360]: pgmap v16239: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1904221005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:54 smithi022 ceph-mon[40250]: pgmap v16239: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1904221005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:54 smithi122 ceph-mon[32255]: pgmap v16239: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T02:59:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1904221005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T02:59:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:56 smithi022 ceph-mon[31360]: pgmap v16240: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:56 smithi022 ceph-mon[40250]: pgmap v16240: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:56 smithi122 ceph-mon[32255]: pgmap v16240: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 02:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:02:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T02:59:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 02:59:58 smithi022 ceph-mon[40250]: pgmap v16241: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 02:59:58 smithi022 ceph-mon[31360]: pgmap v16241: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 02:59:58 smithi122 ceph-mon[32255]: pgmap v16241: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T02:59:59.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:00.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:00.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:00.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:00] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:00:00.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:00:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:00:00.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T03:00:00.227Z caller=compact.go:495 component=tsdb msg="write block" mint=1639612800000 maxt=1639620000000 ulid=01FQ0JXA1J4KQY6P3E4QHBTGQS duration=49.328141ms 2021-12-16T03:00:00.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T03:00:00.237Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=2.868438ms 2021-12-16T03:00:00.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T03:00:00.332Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1639591200000 maxt=1639612800000 ulid=01FQ0JXA3KM5MT4CXEWPB3MNNT sources="[01FPZYA49J2357CZXXSSNE5PMB 01FQ055VHJ7R39RPBMRTBXAVYA 01FQ0C1JSJ5PB1RZDJVDS55TAY]" duration=88.954428ms 2021-12-16T03:00:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:00 smithi022 ceph-mon[31360]: pgmap v16242: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T03:00:01.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2993246491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:00 smithi022 ceph-mon[40250]: pgmap v16242: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T03:00:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2993246491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:00 smithi122 ceph-mon[32255]: pgmap v16242: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T03:00:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2993246491' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:00:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:00:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:02 smithi022 ceph-mon[31360]: pgmap v16243: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:02 smithi022 ceph-mon[40250]: pgmap v16243: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:02 smithi122 ceph-mon[32255]: pgmap v16243: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:03.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:00:03.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:00:03.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:00:03.978 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:00:03.978 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:00:03.978 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:00:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:00:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:00:04.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:00:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:04 smithi022 ceph-mon[31360]: pgmap v16244: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:00:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:04 smithi022 ceph-mon[40250]: pgmap v16244: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:00:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:04 smithi122 ceph-mon[32255]: pgmap v16244: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:00:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3274289155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3274289155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3274289155' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:06 smithi022 ceph-mon[31360]: pgmap v16245: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:06 smithi022 ceph-mon[40250]: pgmap v16245: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:06 smithi122 ceph-mon[32255]: pgmap v16245: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:00:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:08 smithi022 ceph-mon[31360]: pgmap v16246: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:08 smithi022 ceph-mon[40250]: pgmap v16246: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:08 smithi122 ceph-mon[32255]: pgmap v16246: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:10.397 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:10] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T03:00:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:00:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:00:10.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:10 smithi022 ceph-mon[31360]: pgmap v16247: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3271009985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:10 smithi022 ceph-mon[40250]: pgmap v16247: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3271009985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:10 smithi122 ceph-mon[32255]: pgmap v16247: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3271009985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:12.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:00:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:00:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:12 smithi022 ceph-mon[31360]: pgmap v16248: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:12 smithi022 ceph-mon[40250]: pgmap v16248: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:12 smithi122 ceph-mon[32255]: pgmap v16248: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:14 smithi022 ceph-mon[31360]: pgmap v16249: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:14 smithi022 ceph-mon[40250]: pgmap v16249: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:14 smithi122 ceph-mon[32255]: pgmap v16249: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:15.765 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:16.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:16.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:16.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:16 smithi022 ceph-mon[31360]: pgmap v16250: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3716120685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:16 smithi022 ceph-mon[40250]: pgmap v16250: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3716120685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:16 smithi122 ceph-mon[32255]: pgmap v16250: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3716120685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:00:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:18 smithi022 ceph-mon[31360]: pgmap v16251: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:18 smithi022 ceph-mon[40250]: pgmap v16251: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:18 smithi122 ceph-mon[32255]: pgmap v16251: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:20] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:00:20.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:00:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:00:21.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:20 smithi022 ceph-mon[31360]: pgmap v16252: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:20 smithi022 ceph-mon[40250]: pgmap v16252: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:20 smithi122 ceph-mon[32255]: pgmap v16252: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:21.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:21.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:21.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1414890099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1414890099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:22.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1414890099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:00:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:00:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:22 smithi022 ceph-mon[31360]: pgmap v16253: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:22 smithi022 ceph-mon[40250]: pgmap v16253: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:22 smithi122 ceph-mon[32255]: pgmap v16253: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:24 smithi022 ceph-mon[31360]: pgmap v16254: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:24 smithi022 ceph-mon[40250]: pgmap v16254: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:24 smithi122 ceph-mon[32255]: pgmap v16254: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:26.504 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:26.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:26.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:26.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:26 smithi022 ceph-mon[31360]: pgmap v16255: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3029231251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:26 smithi022 ceph-mon[40250]: pgmap v16255: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3029231251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:26 smithi122 ceph-mon[32255]: pgmap v16255: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3029231251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:00:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:28 smithi122 ceph-mon[32255]: pgmap v16256: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:28 smithi022 ceph-mon[31360]: pgmap v16256: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:28 smithi022 ceph-mon[40250]: pgmap v16256: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:30] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:00:30.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:00:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:00:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:30 smithi022 ceph-mon[31360]: pgmap v16257: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:30 smithi022 ceph-mon[40250]: pgmap v16257: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:30 smithi122 ceph-mon[32255]: pgmap v16257: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:31.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:32.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:32.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:32.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:32.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:32.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:00:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:00:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:32 smithi022 ceph-mon[31360]: pgmap v16258: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1424800325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:32 smithi022 ceph-mon[40250]: pgmap v16258: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:33.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1424800325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:32 smithi122 ceph-mon[32255]: pgmap v16258: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1424800325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[31360]: pgmap v16259: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:00:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:00:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:00:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:00:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[40250]: pgmap v16259: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:00:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:00:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:00:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:00:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:34 smithi122 ceph-mon[32255]: pgmap v16259: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:00:37.244 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:37.245 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:36 smithi022 ceph-mon[31360]: pgmap v16260: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:37.245 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:36 smithi022 ceph-mon[40250]: pgmap v16260: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:36 smithi122 ceph-mon[32255]: pgmap v16260: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:00:37.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:37.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:37.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/404058907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:38 smithi122 ceph-mon[32255]: pgmap v16261: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/404058907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:38 smithi022 ceph-mon[31360]: pgmap v16261: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/404058907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:38 smithi022 ceph-mon[40250]: pgmap v16261: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:40] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:00:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:00:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:00:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:40 smithi022 ceph-mon[31360]: pgmap v16262: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:40 smithi022 ceph-mon[40250]: pgmap v16262: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:40 smithi122 ceph-mon[32255]: pgmap v16262: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:42.616 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:00:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:00:42.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:42.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:42.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:42 smithi022 ceph-mon[31360]: pgmap v16263: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3302170905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:42 smithi022 ceph-mon[40250]: pgmap v16263: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3302170905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:42 smithi122 ceph-mon[32255]: pgmap v16263: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3302170905' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:44 smithi022 ceph-mon[31360]: pgmap v16264: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:44 smithi022 ceph-mon[40250]: pgmap v16264: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:44 smithi122 ceph-mon[32255]: pgmap v16264: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:46 smithi022 ceph-mon[31360]: pgmap v16265: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:46 smithi022 ceph-mon[40250]: pgmap v16265: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:46 smithi122 ceph-mon[32255]: pgmap v16265: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:00:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:48.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:48.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:48.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:48 smithi022 ceph-mon[31360]: pgmap v16266: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2268408301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:48 smithi022 ceph-mon[40250]: pgmap v16266: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2268408301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:48 smithi122 ceph-mon[32255]: pgmap v16266: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2268408301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:50.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:00:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:00:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:50] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T03:00:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:50 smithi022 ceph-mon[31360]: pgmap v16267: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:50 smithi022 ceph-mon[40250]: pgmap v16267: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:50 smithi122 ceph-mon[32255]: pgmap v16267: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:00:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:00:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:00:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:52 smithi022 ceph-mon[31360]: pgmap v16268: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:52 smithi022 ceph-mon[40250]: pgmap v16268: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:53.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:52 smithi122 ceph-mon[32255]: pgmap v16268: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:53.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:53.725 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:53.726 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2043559313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2043559313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2043559313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:00:55.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:54 smithi022 ceph-mon[40250]: pgmap v16269: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:55.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:54 smithi022 ceph-mon[31360]: pgmap v16269: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:54 smithi122 ceph-mon[32255]: pgmap v16269: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:00:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:56 smithi022 ceph-mon[31360]: pgmap v16270: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:57.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:56 smithi022 ceph-mon[40250]: pgmap v16270: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:57 smithi122 ceph-mon[32255]: pgmap v16270: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:00:58.727 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:00:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:58 smithi122 ceph-mon[32255]: pgmap v16271: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:58 smithi022 ceph-mon[31360]: pgmap v16271: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:58 smithi022 ceph-mon[40250]: pgmap v16271: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:00:59.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:00:59.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:00:59.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:00:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:00:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2236650831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:00:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2236650831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:00:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2236650831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:00] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T03:01:00.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:01:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:01:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:00 smithi122 ceph-mon[32255]: pgmap v16272: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:00 smithi022 ceph-mon[31360]: pgmap v16272: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:00 smithi022 ceph-mon[40250]: pgmap v16272: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:01.953 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:01.954 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:02 smithi122 ceph-mon[32255]: pgmap v16273: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:01:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:01:03.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:02 smithi022 ceph-mon[31360]: pgmap v16273: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:02 smithi022 ceph-mon[40250]: pgmap v16273: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:04.099 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:04.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:04.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:04.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[31360]: pgmap v16274: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:01:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:01:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:01:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/257999630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:05.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:01:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[40250]: pgmap v16274: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:01:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:01:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:01:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/257999630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:01:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:04 smithi122 ceph-mon[32255]: pgmap v16274: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:01:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:01:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:01:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/257999630' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:01:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:06 smithi022 ceph-mon[31360]: pgmap v16275: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:06 smithi022 ceph-mon[40250]: pgmap v16275: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:06 smithi122 ceph-mon[32255]: pgmap v16275: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:01:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:08 smithi122 ceph-mon[32255]: pgmap v16276: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:08 smithi022 ceph-mon[31360]: pgmap v16276: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:08 smithi022 ceph-mon[40250]: pgmap v16276: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:09.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:09.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:09.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:09.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3000279233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3000279233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3000279233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:10] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:01:10.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:01:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:01:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:10 smithi122 ceph-mon[32255]: pgmap v16277: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:10 smithi022 ceph-mon[31360]: pgmap v16277: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:10 smithi022 ceph-mon[40250]: pgmap v16277: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:01:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:01:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:12 smithi122 ceph-mon[32255]: pgmap v16278: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:12 smithi022 ceph-mon[31360]: pgmap v16278: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:12 smithi022 ceph-mon[40250]: pgmap v16278: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:14.850 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:15.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:15.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:15.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:15.221 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:14 smithi022 ceph-mon[31360]: pgmap v16279: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:15.222 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:14 smithi022 ceph-mon[40250]: pgmap v16279: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:14 smithi122 ceph-mon[32255]: pgmap v16279: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3662676267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3662676267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3662676267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:16 smithi022 ceph-mon[31360]: pgmap v16280: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:16 smithi022 ceph-mon[40250]: pgmap v16280: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:16 smithi122 ceph-mon[32255]: pgmap v16280: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:01:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:18 smithi122 ceph-mon[32255]: pgmap v16281: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:18 smithi022 ceph-mon[31360]: pgmap v16281: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:18 smithi022 ceph-mon[40250]: pgmap v16281: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:20.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:20] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:01:20.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:01:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:01:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:20 smithi022 ceph-mon[31360]: pgmap v16282: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/379761165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:20 smithi022 ceph-mon[40250]: pgmap v16282: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/379761165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:20 smithi122 ceph-mon[32255]: pgmap v16282: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/379761165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:01:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:01:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:22 smithi022 ceph-mon[31360]: pgmap v16283: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:22 smithi022 ceph-mon[40250]: pgmap v16283: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:22 smithi122 ceph-mon[32255]: pgmap v16283: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:25 smithi022 ceph-mon[31360]: pgmap v16284: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:25 smithi022 ceph-mon[40250]: pgmap v16284: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:25 smithi122 ceph-mon[32255]: pgmap v16284: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:25.589 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2909283489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2909283489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2909283489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:27 smithi022 ceph-mon[31360]: pgmap v16285: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:27 smithi022 ceph-mon[40250]: pgmap v16285: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:27 smithi122 ceph-mon[32255]: pgmap v16285: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:01:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:28 smithi122 ceph-mon[32255]: pgmap v16286: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:28 smithi022 ceph-mon[31360]: pgmap v16286: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:28 smithi022 ceph-mon[40250]: pgmap v16286: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:30] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:01:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:01:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:01:30.954 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:30 smithi022 ceph-mon[31360]: pgmap v16287: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:30 smithi022 ceph-mon[40250]: pgmap v16287: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:31.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:31.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:31.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:30 smithi122 ceph-mon[32255]: pgmap v16287: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3224313067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3224313067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3224313067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:01:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:01:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:33 smithi022 ceph-mon[31360]: pgmap v16288: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:33 smithi022 ceph-mon[40250]: pgmap v16288: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:33 smithi122 ceph-mon[32255]: pgmap v16288: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[31360]: pgmap v16289: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:01:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:01:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:01:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[40250]: pgmap v16289: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:01:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:01:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:01:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:35 smithi122 ceph-mon[32255]: pgmap v16289: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:01:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:01:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:01:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:01:36.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:36.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:36.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:37 smithi022 ceph-mon[31360]: pgmap v16290: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2029604795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:37 smithi022 ceph-mon[40250]: pgmap v16290: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2029604795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:37 smithi122 ceph-mon[32255]: pgmap v16290: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2029604795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:01:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:38 smithi122 ceph-mon[32255]: pgmap v16291: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:38 smithi022 ceph-mon[31360]: pgmap v16291: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:38 smithi022 ceph-mon[40250]: pgmap v16291: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:40] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:01:40.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:01:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:01:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:40 smithi022 ceph-mon[31360]: pgmap v16292: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:40 smithi022 ceph-mon[40250]: pgmap v16292: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:40 smithi122 ceph-mon[32255]: pgmap v16292: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:41.695 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:42.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:42.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:42.065 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:01:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:01:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:43 smithi022 ceph-mon[31360]: pgmap v16293: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/534460010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:43 smithi022 ceph-mon[40250]: pgmap v16293: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/534460010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:43 smithi122 ceph-mon[32255]: pgmap v16293: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/534460010' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:45 smithi022 ceph-mon[31360]: pgmap v16294: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:45 smithi022 ceph-mon[40250]: pgmap v16294: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:45 smithi122 ceph-mon[32255]: pgmap v16294: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:47.066 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:47 smithi022 ceph-mon[31360]: pgmap v16295: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:47 smithi022 ceph-mon[40250]: pgmap v16295: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:47.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:47.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:47.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:47 smithi122 ceph-mon[32255]: pgmap v16295: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:01:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2553731739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2553731739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2553731739' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:49 smithi022 ceph-mon[31360]: pgmap v16296: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:49 smithi022 ceph-mon[40250]: pgmap v16296: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:49 smithi122 ceph-mon[32255]: pgmap v16296: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:01:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:01:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:01:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:51 smithi022 ceph-mon[31360]: pgmap v16297: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:51 smithi022 ceph-mon[40250]: pgmap v16297: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:51 smithi122 ceph-mon[32255]: pgmap v16297: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:01:52.434 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:01:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:01:52.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:52.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:52.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:53 smithi022 ceph-mon[31360]: pgmap v16298: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:53.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3192304096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:53 smithi022 ceph-mon[40250]: pgmap v16298: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3192304096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:53 smithi122 ceph-mon[32255]: pgmap v16298: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3192304096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:55 smithi022 ceph-mon[31360]: pgmap v16299: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:55 smithi022 ceph-mon[40250]: pgmap v16299: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:55 smithi122 ceph-mon[32255]: pgmap v16299: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:01:57.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:57 smithi022 ceph-mon[31360]: pgmap v16300: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:57.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:57 smithi022 ceph-mon[40250]: pgmap v16300: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:57 smithi122 ceph-mon[32255]: pgmap v16300: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:01:57.805 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:01:58.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:01:58.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:01:58.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:01:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:58 smithi122 ceph-mon[32255]: pgmap v16301: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:01:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1721161761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:58 smithi022 ceph-mon[31360]: pgmap v16301: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:01:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1721161761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:01:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:58 smithi022 ceph-mon[40250]: pgmap v16301: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:01:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:01:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1721161761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:02:00.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:02:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:02:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:01 smithi022 ceph-mon[31360]: pgmap v16302: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:01.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:01 smithi022 ceph-mon[40250]: pgmap v16302: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:01.766 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:01 smithi122 ceph-mon[32255]: pgmap v16302: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:02:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:02:03.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:03 smithi022 ceph-mon[31360]: pgmap v16303: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:03 smithi022 ceph-mon[40250]: pgmap v16303: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:03 smithi122 ceph-mon[32255]: pgmap v16303: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:03.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:03.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:03.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3981593561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3981593561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3981593561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[31360]: pgmap v16304: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:02:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:02:05.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:02:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[40250]: pgmap v16304: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:02:05.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:02:05.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:05 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:02:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:05 smithi122 ceph-mon[32255]: pgmap v16304: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:02:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:02:05.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:05 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:02:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:02:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:06 smithi122 ceph-mon[32255]: pgmap v16305: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:02:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:06 smithi022 ceph-mon[31360]: pgmap v16305: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:02:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:06 smithi022 ceph-mon[40250]: pgmap v16305: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:02:08.549 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:08.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:08.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:08.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:08.913 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:08 smithi022 ceph-mon[31360]: pgmap v16306: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:08.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:08 smithi022 ceph-mon[40250]: pgmap v16306: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:08 smithi122 ceph-mon[32255]: pgmap v16306: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3077593009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3077593009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3077593009' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:10] "GET /metrics HTTP/1.1" 200 202448 "" "Prometheus/2.18.1" 2021-12-16T03:02:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:02:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:02:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:10 smithi122 ceph-mon[32255]: pgmap v16307: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:10 smithi022 ceph-mon[31360]: pgmap v16307: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:10 smithi022 ceph-mon[40250]: pgmap v16307: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:12.568 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:02:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:02:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:12 smithi122 ceph-mon[32255]: pgmap v16308: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:12 smithi022 ceph-mon[31360]: pgmap v16308: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:12 smithi022 ceph-mon[40250]: pgmap v16308: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:13.914 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:15 smithi022 ceph-mon[31360]: pgmap v16309: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:15.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1508463804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:15 smithi022 ceph-mon[40250]: pgmap v16309: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1508463804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:15 smithi122 ceph-mon[32255]: pgmap v16309: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1508463804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:17 smithi022 ceph-mon[31360]: pgmap v16310: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:17 smithi022 ceph-mon[40250]: pgmap v16310: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:17 smithi122 ceph-mon[32255]: pgmap v16310: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:02:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:18 smithi122 ceph-mon[32255]: pgmap v16311: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:18 smithi022 ceph-mon[31360]: pgmap v16311: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:18 smithi022 ceph-mon[40250]: pgmap v16311: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:19.288 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:19.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:19.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:19.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1206362899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1206362899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1206362899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:20] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:02:20.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:02:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:02:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:20 smithi022 ceph-mon[31360]: pgmap v16312: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:20 smithi022 ceph-mon[40250]: pgmap v16312: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:20 smithi122 ceph-mon[32255]: pgmap v16312: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:02:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:02:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:22 smithi022 ceph-mon[31360]: pgmap v16313: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:22 smithi022 ceph-mon[40250]: pgmap v16313: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:22 smithi122 ceph-mon[32255]: pgmap v16313: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:24.661 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:25.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:25.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:25.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:25 smithi022 ceph-mon[31360]: pgmap v16314: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:25 smithi022 ceph-mon[40250]: pgmap v16314: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:25 smithi122 ceph-mon[32255]: pgmap v16314: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/532270111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/532270111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/532270111' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:27 smithi022 ceph-mon[31360]: pgmap v16315: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:27 smithi022 ceph-mon[40250]: pgmap v16315: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:27 smithi122 ceph-mon[32255]: pgmap v16315: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:02:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:28 smithi122 ceph-mon[32255]: pgmap v16316: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:28 smithi022 ceph-mon[31360]: pgmap v16316: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:28 smithi022 ceph-mon[40250]: pgmap v16316: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:30.034 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:30.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:30] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:02:30.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:02:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:02:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:31 smithi022 ceph-mon[31360]: pgmap v16317: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2987465754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:31 smithi022 ceph-mon[40250]: pgmap v16317: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2987465754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:31 smithi122 ceph-mon[32255]: pgmap v16317: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2987465754' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:02:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:02:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:33 smithi022 ceph-mon[31360]: pgmap v16318: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:33 smithi022 ceph-mon[40250]: pgmap v16318: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:33 smithi122 ceph-mon[32255]: pgmap v16318: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[31360]: pgmap v16319: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:02:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:02:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:02:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:02:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[40250]: pgmap v16319: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:02:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:02:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:02:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:02:35.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:35 smithi122 ceph-mon[32255]: pgmap v16319: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:02:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:02:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:02:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:02:35.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:35.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:35.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2394393720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2394393720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2394393720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:37 smithi022 ceph-mon[31360]: pgmap v16320: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:37 smithi022 ceph-mon[40250]: pgmap v16320: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:37 smithi122 ceph-mon[32255]: pgmap v16320: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:02:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:38 smithi122 ceph-mon[32255]: pgmap v16321: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:38 smithi022 ceph-mon[31360]: pgmap v16321: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:38 smithi022 ceph-mon[40250]: pgmap v16321: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:02:40.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:02:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:02:40.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:41 smithi022 ceph-mon[31360]: pgmap v16322: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:41 smithi022 ceph-mon[40250]: pgmap v16322: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:41.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:41.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:41.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:41 smithi122 ceph-mon[32255]: pgmap v16322: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1200713475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1200713475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1200713475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:02:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:02:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:43 smithi022 ceph-mon[31360]: pgmap v16323: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:43 smithi022 ceph-mon[40250]: pgmap v16323: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:43 smithi122 ceph-mon[32255]: pgmap v16323: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:45 smithi022 ceph-mon[31360]: pgmap v16324: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:45 smithi022 ceph-mon[40250]: pgmap v16324: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:45 smithi122 ceph-mon[32255]: pgmap v16324: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:46.143 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:46.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:46.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:46.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:47 smithi022 ceph-mon[31360]: pgmap v16325: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:47.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4173471246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:47 smithi022 ceph-mon[40250]: pgmap v16325: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4173471246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:47 smithi122 ceph-mon[32255]: pgmap v16325: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4173471246' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:02:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:48 smithi122 ceph-mon[32255]: pgmap v16326: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:48 smithi022 ceph-mon[31360]: pgmap v16326: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:48 smithi022 ceph-mon[40250]: pgmap v16326: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:50] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:02:50.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:02:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:02:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:51 smithi022 ceph-mon[31360]: pgmap v16327: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:51 smithi022 ceph-mon[40250]: pgmap v16327: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:51 smithi122 ceph-mon[32255]: pgmap v16327: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:51.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:51.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:51.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:51.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3058469216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3058469216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:02:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3058469216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:02:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:02:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:53 smithi022 ceph-mon[31360]: pgmap v16328: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:53 smithi022 ceph-mon[40250]: pgmap v16328: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:53 smithi122 ceph-mon[32255]: pgmap v16328: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:55 smithi022 ceph-mon[31360]: pgmap v16329: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:55 smithi022 ceph-mon[40250]: pgmap v16329: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:55 smithi122 ceph-mon[32255]: pgmap v16329: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:02:56.884 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:02:57.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:02:57.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:02:57.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:02:57.256 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:57 smithi022 ceph-mon[31360]: pgmap v16330: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:57.256 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:57 smithi022 ceph-mon[40250]: pgmap v16330: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:57 smithi122 ceph-mon[32255]: pgmap v16330: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:02:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3950492113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3950492113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3950492113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:02:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:02:59 smithi122 ceph-mon[32255]: pgmap v16331: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:02:59 smithi022 ceph-mon[31360]: pgmap v16331: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:02:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:02:59 smithi022 ceph-mon[40250]: pgmap v16331: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:00] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:03:00.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:03:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:03:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:01 smithi122 ceph-mon[32255]: pgmap v16332: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:01 smithi022 ceph-mon[31360]: pgmap v16332: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:01 smithi022 ceph-mon[40250]: pgmap v16332: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:02.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:02.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:02.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:02.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:02.720 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:03:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:03:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:03 smithi122 ceph-mon[32255]: pgmap v16333: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2580053645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:03 smithi022 ceph-mon[31360]: pgmap v16333: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2580053645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:03 smithi022 ceph-mon[40250]: pgmap v16333: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2580053645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:05 smithi122 ceph-mon[32255]: pgmap v16334: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:05.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:05 smithi022 ceph-mon[31360]: pgmap v16334: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:05.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:05 smithi022 ceph-mon[40250]: pgmap v16334: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:03:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:03:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:03:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:03:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:06 smithi122 ceph-mon[32255]: pgmap v16335: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:03:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:03:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:03:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:03:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[31360]: pgmap v16335: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:03:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:03:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:03:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:03:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:06 smithi022 ceph-mon[40250]: pgmap v16335: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:07.625 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:03:07.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:07.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:07.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:08 smithi122 ceph-mon[32255]: pgmap v16336: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3466086490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:08 smithi022 ceph-mon[31360]: pgmap v16336: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3466086490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:08 smithi022 ceph-mon[40250]: pgmap v16336: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3466086490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:10] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:03:10.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:03:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:03:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:11 smithi022 ceph-mon[31360]: pgmap v16337: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:11 smithi022 ceph-mon[40250]: pgmap v16337: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:11 smithi122 ceph-mon[32255]: pgmap v16337: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:03:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:03:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:13 smithi022 ceph-mon[31360]: pgmap v16338: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:13 smithi022 ceph-mon[40250]: pgmap v16338: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:13.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:13.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:13.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:13 smithi122 ceph-mon[32255]: pgmap v16338: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1475225678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1475225678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1475225678' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:15 smithi122 ceph-mon[32255]: pgmap v16339: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:15 smithi022 ceph-mon[31360]: pgmap v16339: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:15 smithi022 ceph-mon[40250]: pgmap v16339: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:17 smithi022 ceph-mon[31360]: pgmap v16340: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:17 smithi022 ceph-mon[40250]: pgmap v16340: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:17 smithi122 ceph-mon[32255]: pgmap v16340: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:03:18.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:18.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:18.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:18.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:18.737 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:18 smithi022 ceph-mon[31360]: pgmap v16341: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:18.738 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:18 smithi022 ceph-mon[40250]: pgmap v16341: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:18 smithi122 ceph-mon[32255]: pgmap v16341: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2265744700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2265744700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:20.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2265744700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:20] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:03:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:03:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:03:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:20 smithi122 ceph-mon[32255]: pgmap v16342: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:20 smithi022 ceph-mon[31360]: pgmap v16342: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:20 smithi022 ceph-mon[40250]: pgmap v16342: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:22.586 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:03:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:03:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:22 smithi122 ceph-mon[32255]: pgmap v16343: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:22 smithi022 ceph-mon[31360]: pgmap v16343: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:22 smithi022 ceph-mon[40250]: pgmap v16343: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:23.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:24.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:24.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:24.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:25 smithi122 ceph-mon[32255]: pgmap v16344: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3854656277' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:25 smithi022 ceph-mon[31360]: pgmap v16344: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3854656277' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:25 smithi022 ceph-mon[40250]: pgmap v16344: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3854656277' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:27 smithi022 ceph-mon[31360]: pgmap v16345: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:27 smithi022 ceph-mon[40250]: pgmap v16345: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:27 smithi122 ceph-mon[32255]: pgmap v16345: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:03:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:28 smithi122 ceph-mon[32255]: pgmap v16346: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:28 smithi022 ceph-mon[31360]: pgmap v16346: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:28 smithi022 ceph-mon[40250]: pgmap v16346: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:29.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:29.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:29.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:29.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/246833805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/246833805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/246833805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:30] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:03:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:03:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:03:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:30 smithi122 ceph-mon[32255]: pgmap v16347: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:30 smithi022 ceph-mon[31360]: pgmap v16347: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:30 smithi022 ceph-mon[40250]: pgmap v16347: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:32.582 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:03:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:03:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:32 smithi122 ceph-mon[32255]: pgmap v16348: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:32 smithi022 ceph-mon[31360]: pgmap v16348: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:32 smithi022 ceph-mon[40250]: pgmap v16348: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:34.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:34.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:34.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:34.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[40250]: pgmap v16349: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:03:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:03:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:03:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:03:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1145989788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[31360]: pgmap v16349: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:03:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:03:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:03:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:03:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1145989788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:35 smithi122 ceph-mon[32255]: pgmap v16349: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:03:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:03:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:03:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:03:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1145989788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:37 smithi022 ceph-mon[31360]: pgmap v16350: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:37 smithi022 ceph-mon[40250]: pgmap v16350: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:37 smithi122 ceph-mon[32255]: pgmap v16350: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:03:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:38 smithi122 ceph-mon[32255]: pgmap v16351: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:38 smithi022 ceph-mon[31360]: pgmap v16351: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:38 smithi022 ceph-mon[40250]: pgmap v16351: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:39.848 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:40.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:40.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:40.217 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:40] "GET /metrics HTTP/1.1" 200 202451 "" "Prometheus/2.18.1" 2021-12-16T03:03:40.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:03:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:03:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:41 smithi122 ceph-mon[32255]: pgmap v16352: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2138661780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:41 smithi022 ceph-mon[31360]: pgmap v16352: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2138661780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:41 smithi022 ceph-mon[40250]: pgmap v16352: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2138661780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:03:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:03:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:43 smithi122 ceph-mon[32255]: pgmap v16353: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:43 smithi022 ceph-mon[31360]: pgmap v16353: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:43 smithi022 ceph-mon[40250]: pgmap v16353: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:45.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:45 smithi122 ceph-mon[32255]: pgmap v16354: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:45 smithi022 ceph-mon[31360]: pgmap v16354: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:45 smithi022 ceph-mon[40250]: pgmap v16354: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:45.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:45.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:45.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/434844290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/434844290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/434844290' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:47 smithi022 ceph-mon[31360]: pgmap v16355: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:47 smithi022 ceph-mon[40250]: pgmap v16355: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:47 smithi122 ceph-mon[32255]: pgmap v16355: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:03:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:48 smithi122 ceph-mon[32255]: pgmap v16356: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:48 smithi022 ceph-mon[31360]: pgmap v16356: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:48 smithi022 ceph-mon[40250]: pgmap v16356: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:50] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T03:03:50.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:03:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:03:50.587 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:50.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:50.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:50.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:51 smithi022 ceph-mon[31360]: pgmap v16357: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3792287287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:51 smithi022 ceph-mon[40250]: pgmap v16357: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3792287287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:51 smithi122 ceph-mon[32255]: pgmap v16357: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3792287287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:03:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:03:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:03:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:53 smithi122 ceph-mon[32255]: pgmap v16358: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:53 smithi022 ceph-mon[31360]: pgmap v16358: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:53 smithi022 ceph-mon[40250]: pgmap v16358: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:55 smithi122 ceph-mon[32255]: pgmap v16359: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:55 smithi022 ceph-mon[31360]: pgmap v16359: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:55 smithi022 ceph-mon[40250]: pgmap v16359: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:03:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:03:56.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:03:56.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:03:56.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:03:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:57 smithi022 ceph-mon[31360]: pgmap v16360: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806927869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:57 smithi022 ceph-mon[40250]: pgmap v16360: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806927869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:57 smithi122 ceph-mon[32255]: pgmap v16360: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806927869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:03:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:03:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:03:58 smithi122 ceph-mon[32255]: pgmap v16361: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:03:58 smithi022 ceph-mon[31360]: pgmap v16361: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:03:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:03:58 smithi022 ceph-mon[40250]: pgmap v16361: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:00] "GET /metrics HTTP/1.1" 200 202450 "" "Prometheus/2.18.1" 2021-12-16T03:04:00.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:04:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:04:01.331 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:01.332 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:01 smithi022 ceph-mon[31360]: pgmap v16362: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:01.332 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:01 smithi022 ceph-mon[40250]: pgmap v16362: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:01 smithi122 ceph-mon[32255]: pgmap v16362: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:01.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:01.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:01.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/274599301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/274599301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/274599301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:04:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:04:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:03 smithi122 ceph-mon[32255]: pgmap v16363: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:03 smithi022 ceph-mon[31360]: pgmap v16363: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:03 smithi022 ceph-mon[40250]: pgmap v16363: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:05 smithi122 ceph-mon[32255]: pgmap v16364: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:05 smithi022 ceph-mon[31360]: pgmap v16364: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:05 smithi022 ceph-mon[40250]: pgmap v16364: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:04:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:04:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:06 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:04:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:06 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:04:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:04:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:04:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:04:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:04:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:04:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:06.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:06 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:06.703 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:07.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:07.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:07.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:07 smithi022 ceph-mon[31360]: pgmap v16365: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:07 smithi022 ceph-mon[40250]: pgmap v16365: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:07.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:07 smithi122 ceph-mon[32255]: pgmap v16365: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:04:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:04:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3919680198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3919680198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3919680198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:09 smithi122 ceph-mon[32255]: pgmap v16366: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:09 smithi022 ceph-mon[31360]: pgmap v16366: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:09 smithi022 ceph-mon[40250]: pgmap v16366: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:10 smithi122 ceph-mon[32255]: pgmap v16367: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:10] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T03:04:10.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:04:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:04:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:10 smithi022 ceph-mon[31360]: pgmap v16367: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:10 smithi022 ceph-mon[40250]: pgmap v16367: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:12.069 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:12 smithi122 ceph-mon[32255]: pgmap v16368: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:04:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:04:12.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:12.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:12.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:12.443 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:12 smithi022 ceph-mon[31360]: pgmap v16368: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:12.443 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:12 smithi022 ceph-mon[40250]: pgmap v16368: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1016729194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1016729194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1016729194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:14 smithi122 ceph-mon[32255]: pgmap v16369: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:14 smithi022 ceph-mon[31360]: pgmap v16369: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:14 smithi022 ceph-mon[40250]: pgmap v16369: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:17 smithi022 ceph-mon[31360]: pgmap v16370: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:17 smithi022 ceph-mon[40250]: pgmap v16370: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:17 smithi122 ceph-mon[32255]: pgmap v16370: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:17.444 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:04:17.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:17.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:17.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2159323561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:18 smithi122 ceph-mon[32255]: pgmap v16371: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2159323561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:18 smithi022 ceph-mon[31360]: pgmap v16371: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2159323561' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:18 smithi022 ceph-mon[40250]: pgmap v16371: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:20] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:04:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:04:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:04:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:21 smithi122 ceph-mon[32255]: pgmap v16372: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:21 smithi022 ceph-mon[31360]: pgmap v16372: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:21 smithi022 ceph-mon[40250]: pgmap v16372: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:04:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:04:22.815 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:23.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:23.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:23.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:23.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:23 smithi022 ceph-mon[31360]: pgmap v16373: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:23.186 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:23 smithi022 ceph-mon[40250]: pgmap v16373: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:23 smithi122 ceph-mon[32255]: pgmap v16373: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/93278791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/93278791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/93278791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:25 smithi122 ceph-mon[32255]: pgmap v16374: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:25 smithi022 ceph-mon[31360]: pgmap v16374: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:25 smithi022 ceph-mon[40250]: pgmap v16374: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:26 smithi122 ceph-mon[32255]: pgmap v16375: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:26 smithi022 ceph-mon[31360]: pgmap v16375: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:26 smithi022 ceph-mon[40250]: pgmap v16375: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:04:28.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:28.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:28.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:28 smithi122 ceph-mon[32255]: pgmap v16376: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2677925769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:28 smithi022 ceph-mon[31360]: pgmap v16376: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2677925769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:28 smithi022 ceph-mon[40250]: pgmap v16376: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2677925769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:30] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:04:30.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:04:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:04:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:31 smithi122 ceph-mon[32255]: pgmap v16377: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:31 smithi022 ceph-mon[31360]: pgmap v16377: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:31 smithi022 ceph-mon[40250]: pgmap v16377: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:04:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:04:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:33 smithi122 ceph-mon[32255]: pgmap v16378: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:33 smithi022 ceph-mon[31360]: pgmap v16378: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:33 smithi022 ceph-mon[40250]: pgmap v16378: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:33.559 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1082785158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1082785158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1082785158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:35 smithi122 ceph-mon[32255]: pgmap v16379: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:04:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:04:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:04:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:04:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[31360]: pgmap v16379: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:04:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:04:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:04:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:04:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[40250]: pgmap v16379: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:04:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:04:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:04:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:04:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:36 smithi122 ceph-mon[32255]: pgmap v16380: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:36 smithi022 ceph-mon[31360]: pgmap v16380: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:36 smithi022 ceph-mon[40250]: pgmap v16380: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:04:38.927 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:38.927 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:38 smithi022 ceph-mon[40250]: pgmap v16381: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:38.928 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:38 smithi022 ceph-mon[31360]: pgmap v16381: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:38 smithi122 ceph-mon[32255]: pgmap v16381: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:39.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:39.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:39.296 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2072788646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2072788646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2072788646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:40] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T03:04:40.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:04:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:04:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:40 smithi122 ceph-mon[32255]: pgmap v16382: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:40 smithi022 ceph-mon[31360]: pgmap v16382: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:40 smithi022 ceph-mon[40250]: pgmap v16382: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:42.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:04:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:04:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:42 smithi122 ceph-mon[32255]: pgmap v16383: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:42 smithi022 ceph-mon[31360]: pgmap v16383: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:42 smithi022 ceph-mon[40250]: pgmap v16383: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:44.298 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:44.667 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:44.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:44.668 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:45 smithi122 ceph-mon[32255]: pgmap v16384: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1815966989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:45 smithi022 ceph-mon[31360]: pgmap v16384: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1815966989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:45 smithi022 ceph-mon[40250]: pgmap v16384: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1815966989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:47 smithi022 ceph-mon[31360]: pgmap v16385: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:47 smithi022 ceph-mon[40250]: pgmap v16385: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:47 smithi122 ceph-mon[32255]: pgmap v16385: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:04:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:48 smithi122 ceph-mon[32255]: pgmap v16386: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:48 smithi022 ceph-mon[31360]: pgmap v16386: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:48 smithi022 ceph-mon[40250]: pgmap v16386: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:49.669 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:50.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:50.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:50.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:50.380 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:50] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:04:50.380 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:50 smithi122 ceph-mon[32255]: pgmap v16387: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:50.380 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2696898848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:50 smithi022 ceph-mon[31360]: pgmap v16387: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2696898848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:50 smithi022 ceph-mon[40250]: pgmap v16387: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2696898848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:50.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:04:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:04:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:04:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:52 smithi122 ceph-mon[32255]: pgmap v16388: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:04:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:04:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:52 smithi022 ceph-mon[31360]: pgmap v16388: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:52 smithi022 ceph-mon[40250]: pgmap v16388: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:55.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:04:55.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:04:55.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:04:55.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:04:55.409 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:55 smithi022 ceph-mon[40250]: pgmap v16389: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:55.409 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:55 smithi022 ceph-mon[31360]: pgmap v16389: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:55 smithi122 ceph-mon[32255]: pgmap v16389: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:04:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/733210029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/733210029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/733210029' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:04:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:57 smithi022 ceph-mon[31360]: pgmap v16390: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:57 smithi022 ceph-mon[40250]: pgmap v16390: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:57 smithi122 ceph-mon[32255]: pgmap v16390: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:04:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:04:58 smithi122 ceph-mon[32255]: pgmap v16391: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:04:58 smithi022 ceph-mon[31360]: pgmap v16391: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:04:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:04:58 smithi022 ceph-mon[40250]: pgmap v16391: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:00.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:00.487 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:00] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:05:00.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:05:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:05:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:00.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:01 smithi122 ceph-mon[32255]: pgmap v16392: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2990591736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:01 smithi022 ceph-mon[31360]: pgmap v16392: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2990591736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:01 smithi022 ceph-mon[40250]: pgmap v16392: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2990591736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:05:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:05:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:03 smithi122 ceph-mon[32255]: pgmap v16393: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:03 smithi022 ceph-mon[31360]: pgmap v16393: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:03 smithi022 ceph-mon[40250]: pgmap v16393: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:04 smithi122 ceph-mon[32255]: pgmap v16394: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:04 smithi022 ceph-mon[31360]: pgmap v16394: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:04 smithi022 ceph-mon[40250]: pgmap v16394: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:05.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:06.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:06.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:06.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:07 smithi122 ceph-mon[32255]: pgmap v16395: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2092143564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:05:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:05:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:05:07.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:05:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[31360]: pgmap v16395: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2092143564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:05:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:05:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:05:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:05:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[40250]: pgmap v16395: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2092143564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:05:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:05:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:05:07.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:05:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:05:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:08 smithi122 ceph-mon[32255]: pgmap v16396: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:08 smithi022 ceph-mon[31360]: pgmap v16396: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:08 smithi022 ceph-mon[40250]: pgmap v16396: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:10.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:05:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:05:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:10] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:05:11.149 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:11 smithi122 ceph-mon[32255]: pgmap v16397: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:11.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:11.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:11.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:11.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:11 smithi022 ceph-mon[31360]: pgmap v16397: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:11.519 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:11 smithi022 ceph-mon[40250]: pgmap v16397: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:05:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:05:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/101730475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/101730475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/101730475' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:13 smithi122 ceph-mon[32255]: pgmap v16398: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:13 smithi022 ceph-mon[31360]: pgmap v16398: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:13 smithi022 ceph-mon[40250]: pgmap v16398: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:14 smithi122 ceph-mon[32255]: pgmap v16399: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:14 smithi022 ceph-mon[31360]: pgmap v16399: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:14 smithi022 ceph-mon[40250]: pgmap v16399: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:16.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:16.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:16.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:16.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:17 smithi022 ceph-mon[31360]: pgmap v16400: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2699030717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:17 smithi022 ceph-mon[40250]: pgmap v16400: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:17.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2699030717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:17 smithi122 ceph-mon[32255]: pgmap v16400: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2699030717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:05:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:18 smithi122 ceph-mon[32255]: pgmap v16401: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:18 smithi022 ceph-mon[31360]: pgmap v16401: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:18 smithi022 ceph-mon[40250]: pgmap v16401: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:20] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:05:20.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:05:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:05:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:21 smithi122 ceph-mon[32255]: pgmap v16402: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:21 smithi022 ceph-mon[31360]: pgmap v16402: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:21 smithi022 ceph-mon[40250]: pgmap v16402: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:21.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:22.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:22.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:22.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:22.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:22.262 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:22 smithi022 ceph-mon[31360]: pgmap v16403: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:22.263 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:22.263 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:22 smithi022 ceph-mon[40250]: pgmap v16403: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:22 smithi122 ceph-mon[32255]: pgmap v16403: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:05:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:05:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3416392891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3416392891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3416392891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:24 smithi122 ceph-mon[32255]: pgmap v16404: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:24 smithi022 ceph-mon[31360]: pgmap v16404: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:24 smithi022 ceph-mon[40250]: pgmap v16404: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:27.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:27 smithi022 ceph-mon[31360]: pgmap v16405: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:27 smithi022 ceph-mon[40250]: pgmap v16405: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:27 smithi122 ceph-mon[32255]: pgmap v16405: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:27.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:27.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:27.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:05:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2271858074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:28 smithi122 ceph-mon[32255]: pgmap v16406: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2271858074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:28 smithi022 ceph-mon[31360]: pgmap v16406: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2271858074' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:28 smithi022 ceph-mon[40250]: pgmap v16406: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:30.419 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:30] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:05:30.420 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:30 smithi122 ceph-mon[32255]: pgmap v16407: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:30 smithi022 ceph-mon[31360]: pgmap v16407: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:30 smithi022 ceph-mon[40250]: pgmap v16407: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:30.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:05:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:05:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:32 smithi122 ceph-mon[32255]: pgmap v16408: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:05:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:05:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:32 smithi022 ceph-mon[31360]: pgmap v16408: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:32 smithi022 ceph-mon[40250]: pgmap v16408: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:32.631 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:33.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:33.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:33.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/788330790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/788330790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/788330790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:34 smithi122 ceph-mon[32255]: pgmap v16409: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:05:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:05:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:05:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:05:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[31360]: pgmap v16409: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:05:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:05:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:05:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:05:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[40250]: pgmap v16409: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:05:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:05:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:05:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:05:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:36 smithi122 ceph-mon[32255]: pgmap v16410: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:36 smithi022 ceph-mon[31360]: pgmap v16410: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:36 smithi022 ceph-mon[40250]: pgmap v16410: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:05:38.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:38.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:38.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:38.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:38 smithi022 ceph-mon[31360]: pgmap v16411: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1130598112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:38 smithi022 ceph-mon[40250]: pgmap v16411: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1130598112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:38 smithi122 ceph-mon[32255]: pgmap v16411: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1130598112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:40.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:40 smithi122 ceph-mon[32255]: pgmap v16412: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:40.429 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:40] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T03:05:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:40 smithi022 ceph-mon[31360]: pgmap v16412: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:40 smithi022 ceph-mon[40250]: pgmap v16412: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:40.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:05:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:05:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:05:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:05:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:42 smithi122 ceph-mon[32255]: pgmap v16413: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:42 smithi022 ceph-mon[31360]: pgmap v16413: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:42 smithi022 ceph-mon[40250]: pgmap v16413: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:43.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:43.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:43.745 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/842292424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/842292424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/842292424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:44 smithi122 ceph-mon[32255]: pgmap v16414: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:44 smithi022 ceph-mon[31360]: pgmap v16414: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:44 smithi022 ceph-mon[40250]: pgmap v16414: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:46 smithi122 ceph-mon[32255]: pgmap v16415: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:46 smithi022 ceph-mon[31360]: pgmap v16415: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:46.710 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:46 smithi022 ceph-mon[40250]: pgmap v16415: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:05:48.747 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:48 smithi122 ceph-mon[32255]: pgmap v16416: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:48 smithi022 ceph-mon[31360]: pgmap v16416: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:48 smithi022 ceph-mon[40250]: pgmap v16416: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3271679166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3271679166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3271679166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:50] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:05:50.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:05:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:05:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:50 smithi122 ceph-mon[32255]: pgmap v16417: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:50 smithi022 ceph-mon[31360]: pgmap v16417: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:50 smithi022 ceph-mon[40250]: pgmap v16417: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:05:52.611 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:05:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:05:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:52 smithi122 ceph-mon[32255]: pgmap v16418: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:52 smithi022 ceph-mon[31360]: pgmap v16418: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:52 smithi022 ceph-mon[40250]: pgmap v16418: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:54.119 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:54 smithi122 ceph-mon[32255]: pgmap v16419: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:54.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:54.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:54.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:05:54.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:54 smithi022 ceph-mon[31360]: pgmap v16419: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:54.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:54 smithi022 ceph-mon[40250]: pgmap v16419: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:05:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/412720183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/412720183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/412720183' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:05:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:56 smithi122 ceph-mon[32255]: pgmap v16420: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:56 smithi022 ceph-mon[31360]: pgmap v16420: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:56 smithi022 ceph-mon[40250]: pgmap v16420: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:05:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:58 smithi122 ceph-mon[32255]: pgmap v16421: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:58 smithi022 ceph-mon[31360]: pgmap v16421: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:58 smithi022 ceph-mon[40250]: pgmap v16421: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:05:59.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:05:59.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:05:59.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:05:59.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:05:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1422465312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:05:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1422465312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:05:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1422465312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:00] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:06:00.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:06:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:06:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:00 smithi122 ceph-mon[32255]: pgmap v16422: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:00 smithi022 ceph-mon[31360]: pgmap v16422: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:00 smithi022 ceph-mon[40250]: pgmap v16422: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:06:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:06:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:02 smithi122 ceph-mon[32255]: pgmap v16423: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:02 smithi022 ceph-mon[31360]: pgmap v16423: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:02 smithi022 ceph-mon[40250]: pgmap v16423: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:04 smithi122 ceph-mon[32255]: pgmap v16424: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:04 smithi022 ceph-mon[31360]: pgmap v16424: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:04 smithi022 ceph-mon[40250]: pgmap v16424: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:04.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:05.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:05.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:05.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:05.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3189418654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3189418654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3189418654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:06 smithi022 ceph-mon[31360]: pgmap v16425: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:06 smithi022 ceph-mon[40250]: pgmap v16425: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:06 smithi122 ceph-mon[32255]: pgmap v16425: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:07.276 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:06:07.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:06:07.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:06:07.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:06:07.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:06:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:06:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:06:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:06:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:06:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:06:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:06:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:08 smithi122 ceph-mon[32255]: pgmap v16426: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:06:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:08 smithi022 ceph-mon[31360]: pgmap v16426: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:06:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:08 smithi022 ceph-mon[40250]: pgmap v16426: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:10.233 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:10 smithi122 ceph-mon[32255]: pgmap v16427: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:10] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:06:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:10 smithi022 ceph-mon[31360]: pgmap v16427: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:10 smithi022 ceph-mon[40250]: pgmap v16427: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:10.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:10.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:10.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:10.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:06:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:06:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3545699371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3545699371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3545699371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:12 smithi122 ceph-mon[32255]: pgmap v16428: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:06:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:06:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:12 smithi022 ceph-mon[31360]: pgmap v16428: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:12 smithi022 ceph-mon[40250]: pgmap v16428: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:14 smithi122 ceph-mon[32255]: pgmap v16429: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:14 smithi022 ceph-mon[31360]: pgmap v16429: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:14 smithi022 ceph-mon[40250]: pgmap v16429: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:15.604 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:15.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:15.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:15.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1715996360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1715996360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1715996360' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:17 smithi022 ceph-mon[31360]: pgmap v16430: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:17.311 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:17 smithi022 ceph-mon[40250]: pgmap v16430: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:17 smithi122 ceph-mon[32255]: pgmap v16430: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:06:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:18 smithi122 ceph-mon[32255]: pgmap v16431: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:18 smithi022 ceph-mon[31360]: pgmap v16431: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:18 smithi022 ceph-mon[40250]: pgmap v16431: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:20 smithi122 ceph-mon[32255]: pgmap v16432: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:20] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:06:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:20 smithi022 ceph-mon[31360]: pgmap v16432: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:20 smithi022 ceph-mon[40250]: pgmap v16432: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:20.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:06:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:06:20.974 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:21.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:21.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:21.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:06:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:06:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2665661430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:22 smithi122 ceph-mon[32255]: pgmap v16433: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2665661430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:22 smithi022 ceph-mon[31360]: pgmap v16433: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2665661430' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:22 smithi022 ceph-mon[40250]: pgmap v16433: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:24 smithi122 ceph-mon[32255]: pgmap v16434: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:24 smithi022 ceph-mon[31360]: pgmap v16434: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:24 smithi022 ceph-mon[40250]: pgmap v16434: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:26.345 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:26 smithi122 ceph-mon[32255]: pgmap v16435: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:26 smithi022 ceph-mon[31360]: pgmap v16435: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:26 smithi022 ceph-mon[40250]: pgmap v16435: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:26.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:26.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:26.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/196212738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:27.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/196212738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/196212738' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:06:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:28 smithi122 ceph-mon[32255]: pgmap v16436: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:28 smithi022 ceph-mon[31360]: pgmap v16436: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:28 smithi022 ceph-mon[40250]: pgmap v16436: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:30 smithi122 ceph-mon[32255]: pgmap v16437: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:30] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:06:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:30 smithi022 ceph-mon[31360]: pgmap v16437: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:30 smithi022 ceph-mon[40250]: pgmap v16437: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:30.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:06:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:06:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:31.715 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:32.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:32.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:32.083 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:32 smithi122 ceph-mon[32255]: pgmap v16438: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/190293153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:06:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:06:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:32 smithi022 ceph-mon[31360]: pgmap v16438: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/190293153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:32 smithi022 ceph-mon[40250]: pgmap v16438: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/190293153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:34 smithi122 ceph-mon[32255]: pgmap v16439: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:34 smithi022 ceph-mon[31360]: pgmap v16439: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:34 smithi022 ceph-mon[40250]: pgmap v16439: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:06:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:06:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:06:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:06:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:06:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:06:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:06:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:06:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:06:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:06:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:06:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:06:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:36 smithi122 ceph-mon[32255]: pgmap v16440: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:36 smithi022 ceph-mon[31360]: pgmap v16440: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:36 smithi022 ceph-mon[40250]: pgmap v16440: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:37.085 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:37.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:37.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:37.455 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/571309030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:37.900 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:06:37.901 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/571309030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/571309030' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:38 smithi122 ceph-mon[32255]: pgmap v16441: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:38 smithi022 ceph-mon[31360]: pgmap v16441: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:38 smithi022 ceph-mon[40250]: pgmap v16441: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:40 smithi122 ceph-mon[32255]: pgmap v16442: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:40] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:06:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:40 smithi022 ceph-mon[31360]: pgmap v16442: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:40 smithi022 ceph-mon[40250]: pgmap v16442: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:40.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:06:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:06:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:42 smithi122 ceph-mon[32255]: pgmap v16443: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:06:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:06:42.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:42.461 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:42 smithi022 ceph-mon[40250]: pgmap v16443: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:42.461 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:42 smithi022 ceph-mon[31360]: pgmap v16443: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:42.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:42.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:42.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2536621789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2536621789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2536621789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:44 smithi122 ceph-mon[32255]: pgmap v16444: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:44 smithi022 ceph-mon[31360]: pgmap v16444: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:44 smithi022 ceph-mon[40250]: pgmap v16444: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:46 smithi122 ceph-mon[32255]: pgmap v16445: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:46 smithi022 ceph-mon[31360]: pgmap v16445: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:46 smithi022 ceph-mon[40250]: pgmap v16445: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:06:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:48.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:48.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:48.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:48 smithi122 ceph-mon[32255]: pgmap v16446: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3037690129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:48 smithi022 ceph-mon[31360]: pgmap v16446: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3037690129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:48 smithi022 ceph-mon[40250]: pgmap v16446: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3037690129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:50 smithi122 ceph-mon[32255]: pgmap v16447: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:50] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:06:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:50 smithi022 ceph-mon[31360]: pgmap v16447: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:50 smithi022 ceph-mon[40250]: pgmap v16447: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:50.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:06:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:06:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:06:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:52 smithi122 ceph-mon[32255]: pgmap v16448: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:06:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:06:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:52 smithi022 ceph-mon[31360]: pgmap v16448: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:52 smithi022 ceph-mon[40250]: pgmap v16448: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:53.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:53.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:53.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4156233109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4156233109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4156233109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:06:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:54 smithi122 ceph-mon[32255]: pgmap v16449: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:54 smithi022 ceph-mon[31360]: pgmap v16449: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:54 smithi022 ceph-mon[40250]: pgmap v16449: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:06:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:56 smithi122 ceph-mon[32255]: pgmap v16450: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:56 smithi022 ceph-mon[31360]: pgmap v16450: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:56 smithi022 ceph-mon[40250]: pgmap v16450: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:06:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:06:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:58 smithi122 ceph-mon[32255]: pgmap v16451: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:58.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:06:58.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:06:58.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:06:58.941 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:58 smithi022 ceph-mon[31360]: pgmap v16451: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:58.941 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:58 smithi022 ceph-mon[40250]: pgmap v16451: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:06:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:06:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1203656669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:06:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1203656669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:06:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1203656669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:00] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:07:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:00 smithi122 ceph-mon[32255]: pgmap v16452: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:00.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:07:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:07:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:00 smithi022 ceph-mon[31360]: pgmap v16452: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:00 smithi022 ceph-mon[40250]: pgmap v16452: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:02.628 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:07:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:07:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:02 smithi122 ceph-mon[32255]: pgmap v16453: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:02 smithi022 ceph-mon[31360]: pgmap v16453: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:02 smithi022 ceph-mon[40250]: pgmap v16453: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:03.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:04 smithi022 ceph-mon[31360]: pgmap v16454: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:04 smithi022 ceph-mon[40250]: pgmap v16454: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:04.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:04.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:04.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:04 smithi122 ceph-mon[32255]: pgmap v16454: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3615027441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3615027441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3615027441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:06 smithi122 ceph-mon[32255]: pgmap v16455: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:06 smithi022 ceph-mon[31360]: pgmap v16455: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:06 smithi022 ceph-mon[40250]: pgmap v16455: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:07:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:07:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:07:07.741 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:07:07.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:07:07.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:07:07.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:07:07.743 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:07:07.743 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:07:07.743 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:07:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:07:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:08 smithi022 ceph-mon[31360]: pgmap v16456: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:07:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:08 smithi022 ceph-mon[40250]: pgmap v16456: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:07:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:08 smithi122 ceph-mon[32255]: pgmap v16456: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:09.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:09.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:09.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:09.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1448245212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1448245212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1448245212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:10] "GET /metrics HTTP/1.1" 200 202446 "" "Prometheus/2.18.1" 2021-12-16T03:07:10.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:07:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:07:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:10 smithi022 ceph-mon[31360]: pgmap v16457: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:10 smithi022 ceph-mon[40250]: pgmap v16457: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:10 smithi122 ceph-mon[32255]: pgmap v16457: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:07:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:07:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:12 smithi022 ceph-mon[31360]: pgmap v16458: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:12 smithi022 ceph-mon[40250]: pgmap v16458: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:12 smithi122 ceph-mon[32255]: pgmap v16458: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:14 smithi122 ceph-mon[32255]: pgmap v16459: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:14 smithi022 ceph-mon[31360]: pgmap v16459: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:14 smithi022 ceph-mon[40250]: pgmap v16459: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:14.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:15.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:15.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:15.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/217163736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/217163736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/217163736' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:16 smithi122 ceph-mon[32255]: pgmap v16460: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:16 smithi022 ceph-mon[31360]: pgmap v16460: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:16 smithi022 ceph-mon[40250]: pgmap v16460: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:07:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:18 smithi122 ceph-mon[32255]: pgmap v16461: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:18 smithi022 ceph-mon[31360]: pgmap v16461: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:18 smithi022 ceph-mon[40250]: pgmap v16461: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:20.056 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:20.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:20.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:20.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:20.426 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:20 smithi022 ceph-mon[40250]: pgmap v16462: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:20.427 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:20 smithi022 ceph-mon[31360]: pgmap v16462: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:20] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:07:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:20 smithi122 ceph-mon[32255]: pgmap v16462: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:20.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:07:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:07:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2949167520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2949167520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2949167520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:22 smithi122 ceph-mon[32255]: pgmap v16463: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:07:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:07:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:22 smithi022 ceph-mon[31360]: pgmap v16463: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:22 smithi022 ceph-mon[40250]: pgmap v16463: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:24 smithi122 ceph-mon[32255]: pgmap v16464: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:24 smithi022 ceph-mon[31360]: pgmap v16464: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:24 smithi022 ceph-mon[40250]: pgmap v16464: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:25.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:25.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3916617665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3916617665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3916617665' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:26 smithi122 ceph-mon[32255]: pgmap v16465: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:26 smithi022 ceph-mon[31360]: pgmap v16465: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:26 smithi022 ceph-mon[40250]: pgmap v16465: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:07:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:28 smithi122 ceph-mon[32255]: pgmap v16466: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:28 smithi022 ceph-mon[31360]: pgmap v16466: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:28 smithi022 ceph-mon[40250]: pgmap v16466: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:30 smithi122 ceph-mon[32255]: pgmap v16467: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:30] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:07:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:30 smithi022 ceph-mon[31360]: pgmap v16467: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:30 smithi022 ceph-mon[40250]: pgmap v16467: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:30.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:30.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:07:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:07:31.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:31.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:31.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/373602966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:32 smithi122 ceph-mon[32255]: pgmap v16468: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:07:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:07:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/373602966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:32 smithi022 ceph-mon[31360]: pgmap v16468: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/373602966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:32 smithi022 ceph-mon[40250]: pgmap v16468: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:34 smithi122 ceph-mon[32255]: pgmap v16469: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:34 smithi022 ceph-mon[31360]: pgmap v16469: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:34 smithi022 ceph-mon[40250]: pgmap v16469: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:07:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:07:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:07:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:07:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:07:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:07:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:07:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:07:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:07:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:07:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:07:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:07:36.168 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:36 smithi122 ceph-mon[32255]: pgmap v16470: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:36 smithi022 ceph-mon[31360]: pgmap v16470: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:36 smithi022 ceph-mon[40250]: pgmap v16470: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:36.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:36.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:36.535 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4037568297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4037568297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4037568297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:07:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:38 smithi122 ceph-mon[32255]: pgmap v16471: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:38 smithi022 ceph-mon[31360]: pgmap v16471: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:38 smithi022 ceph-mon[40250]: pgmap v16471: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:40] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:07:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:40 smithi122 ceph-mon[32255]: pgmap v16472: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:40 smithi022 ceph-mon[31360]: pgmap v16472: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:40 smithi022 ceph-mon[40250]: pgmap v16472: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:40.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:07:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:07:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:41.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:41.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:41.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:41.905 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3783075028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:42 smithi022 ceph-mon[31360]: pgmap v16473: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3783075028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:42 smithi022 ceph-mon[40250]: pgmap v16473: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3783075028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:42 smithi122 ceph-mon[32255]: pgmap v16473: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:07:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:07:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:44 smithi122 ceph-mon[32255]: pgmap v16474: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:44 smithi022 ceph-mon[31360]: pgmap v16474: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:44 smithi022 ceph-mon[40250]: pgmap v16474: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:46 smithi122 ceph-mon[32255]: pgmap v16475: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:46 smithi022 ceph-mon[31360]: pgmap v16475: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:46 smithi022 ceph-mon[40250]: pgmap v16475: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:46.906 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:47.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:47.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:47.277 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3330673849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:07:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3330673849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3330673849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:48 smithi122 ceph-mon[32255]: pgmap v16476: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:48 smithi022 ceph-mon[31360]: pgmap v16476: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:48 smithi022 ceph-mon[40250]: pgmap v16476: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:50 smithi122 ceph-mon[32255]: pgmap v16477: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:07:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:50 smithi022 ceph-mon[31360]: pgmap v16477: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:50 smithi022 ceph-mon[40250]: pgmap v16477: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:50.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:07:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:07:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:07:52.278 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:52 smithi122 ceph-mon[32255]: pgmap v16478: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:07:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:07:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:52 smithi022 ceph-mon[31360]: pgmap v16478: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:52 smithi022 ceph-mon[40250]: pgmap v16478: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3320348884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3320348884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:54.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3320348884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:54 smithi122 ceph-mon[32255]: pgmap v16479: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:54 smithi022 ceph-mon[31360]: pgmap v16479: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:54 smithi022 ceph-mon[40250]: pgmap v16479: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:07:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:56 smithi122 ceph-mon[32255]: pgmap v16480: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:56 smithi022 ceph-mon[31360]: pgmap v16480: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:56 smithi022 ceph-mon[40250]: pgmap v16480: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:57.650 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:07:57.651 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:07:58.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:07:58.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:07:58.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:07:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1544219847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:07:58 smithi122 ceph-mon[32255]: pgmap v16481: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1544219847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:07:58 smithi022 ceph-mon[40250]: pgmap v16481: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:07:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1544219847' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:07:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:07:58 smithi022 ceph-mon[31360]: pgmap v16481: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:00 smithi122 ceph-mon[32255]: pgmap v16482: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:08:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:00 smithi022 ceph-mon[31360]: pgmap v16482: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:00 smithi022 ceph-mon[40250]: pgmap v16482: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:00.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:08:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:08:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:02 smithi122 ceph-mon[32255]: pgmap v16483: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:08:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:08:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:02 smithi022 ceph-mon[31360]: pgmap v16483: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:02 smithi022 ceph-mon[40250]: pgmap v16483: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:03.020 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:03.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:03.391 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:03.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1118146079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1118146079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1118146079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:04 smithi122 ceph-mon[32255]: pgmap v16484: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:04 smithi022 ceph-mon[31360]: pgmap v16484: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:04 smithi022 ceph-mon[40250]: pgmap v16484: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:06 smithi122 ceph-mon[32255]: pgmap v16485: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:06 smithi022 ceph-mon[31360]: pgmap v16485: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:06 smithi022 ceph-mon[40250]: pgmap v16485: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:08:08.393 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:08.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:08.762 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:08.763 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:08.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:08:08.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:08:08.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:08:08.764 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[31360]: pgmap v16486: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:08.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:08:08.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:08:08.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:08:08.765 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:08 smithi022 ceph-mon[40250]: pgmap v16486: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:08:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:08:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:08:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:08 smithi122 ceph-mon[32255]: pgmap v16486: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3186033366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3186033366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3186033366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:10] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:08:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:10 smithi122 ceph-mon[32255]: pgmap v16487: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:10.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:08:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:08:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:10 smithi022 ceph-mon[31360]: pgmap v16487: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:10 smithi022 ceph-mon[40250]: pgmap v16487: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:08:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:08:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:08:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:13 smithi122 ceph-mon[32255]: pgmap v16488: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:13 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[31360]: pgmap v16488: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[40250]: pgmap v16488: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:13 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:13.764 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:14.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:14.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:14.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:14 smithi022 ceph-mon[31360]: pgmap v16489: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2687771745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:14 smithi022 ceph-mon[40250]: pgmap v16489: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2687771745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:14.285 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:14 smithi122 ceph-mon[32255]: pgmap v16489: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:14.285 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2687771745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2892M 2021-12-16T03:08:16.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3185M 2021-12-16T03:08:16.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:16 smithi122 ceph-mon[32255]: pgmap v16490: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2892M 2021-12-16T03:08:16.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3185M 2021-12-16T03:08:16.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[31360]: pgmap v16490: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:16.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2892M 2021-12-16T03:08:16.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.788 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:08:16.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3185M 2021-12-16T03:08:16.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:08:16.789 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:16 smithi022 ceph-mon[40250]: pgmap v16490: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:08:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:18 smithi122 ceph-mon[32255]: pgmap v16491: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:18 smithi022 ceph-mon[31360]: pgmap v16491: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:18 smithi022 ceph-mon[40250]: pgmap v16491: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:19.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:19.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:19.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4147534859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4147534859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4147534859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:20] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:08:20.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:08:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:08:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:20 smithi122 ceph-mon[32255]: pgmap v16492: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:20 smithi022 ceph-mon[31360]: pgmap v16492: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:21.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:20 smithi022 ceph-mon[40250]: pgmap v16492: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:22.645 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:08:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:08:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:22 smithi122 ceph-mon[32255]: pgmap v16493: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:22 smithi022 ceph-mon[31360]: pgmap v16493: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:23.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:22 smithi022 ceph-mon[40250]: pgmap v16493: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:08:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:24 smithi122 ceph-mon[32255]: pgmap v16494: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:24.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:24.515 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:24 smithi022 ceph-mon[40250]: pgmap v16494: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:24.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:24 smithi022 ceph-mon[31360]: pgmap v16494: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:08:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/669842672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/669842672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/669842672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:26 smithi122 ceph-mon[32255]: pgmap v16495: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:26 smithi022 ceph-mon[31360]: pgmap v16495: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:26 smithi022 ceph-mon[40250]: pgmap v16495: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:08:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:28 smithi122 ceph-mon[32255]: pgmap v16496: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:28 smithi022 ceph-mon[31360]: pgmap v16496: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:28 smithi022 ceph-mon[40250]: pgmap v16496: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:29.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:30.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:30.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:30.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:30] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:08:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:30 smithi122 ceph-mon[32255]: pgmap v16497: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:30 smithi022 ceph-mon[31360]: pgmap v16497: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:30 smithi022 ceph-mon[40250]: pgmap v16497: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:30.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:08:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:08:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2610879816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2610879816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2610879816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:32 smithi122 ceph-mon[32255]: pgmap v16498: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:08:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:08:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:32 smithi022 ceph-mon[31360]: pgmap v16498: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:32 smithi022 ceph-mon[40250]: pgmap v16498: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:34 smithi122 ceph-mon[32255]: pgmap v16499: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:34 smithi022 ceph-mon[31360]: pgmap v16499: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:34 smithi022 ceph-mon[40250]: pgmap v16499: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:35.255 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:08:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:08:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:08:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:08:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:08:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:08:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:08:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:08:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:08:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:08:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:08:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:08:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/101896105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:36 smithi122 ceph-mon[32255]: pgmap v16500: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/101896105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:36 smithi022 ceph-mon[31360]: pgmap v16500: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/101896105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:36 smithi022 ceph-mon[40250]: pgmap v16500: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:08:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:38 smithi122 ceph-mon[32255]: pgmap v16501: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:38 smithi022 ceph-mon[31360]: pgmap v16501: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:38 smithi022 ceph-mon[40250]: pgmap v16501: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:40] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T03:08:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:40 smithi122 ceph-mon[32255]: pgmap v16502: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:40 smithi022 ceph-mon[31360]: pgmap v16502: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:40 smithi022 ceph-mon[40250]: pgmap v16502: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:40.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:40.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:08:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:08:40.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:40.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:40.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:41.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3921830877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3921830877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3921830877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:42 smithi122 ceph-mon[32255]: pgmap v16503: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:08:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:08:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:42 smithi022 ceph-mon[31360]: pgmap v16503: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:42 smithi022 ceph-mon[40250]: pgmap v16503: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:44 smithi122 ceph-mon[32255]: pgmap v16504: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:44 smithi022 ceph-mon[31360]: pgmap v16504: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:44 smithi022 ceph-mon[40250]: pgmap v16504: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:45.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:46 smithi022 ceph-mon[31360]: pgmap v16505: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:46 smithi022 ceph-mon[40250]: pgmap v16505: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:46.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:46.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:46.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:46 smithi122 ceph-mon[32255]: pgmap v16505: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3079216566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3079216566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:08:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3079216566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:48 smithi122 ceph-mon[32255]: pgmap v16506: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:48 smithi022 ceph-mon[31360]: pgmap v16506: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:48 smithi022 ceph-mon[40250]: pgmap v16506: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:50 smithi122 ceph-mon[32255]: pgmap v16507: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:50] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:08:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:50 smithi022 ceph-mon[31360]: pgmap v16507: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:50 smithi022 ceph-mon[40250]: pgmap v16507: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:50.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:08:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:08:51.365 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:08:51.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:51.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:51.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2064793609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:52 smithi122 ceph-mon[32255]: pgmap v16508: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:08:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:08:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2064793609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:52 smithi022 ceph-mon[31360]: pgmap v16508: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2064793609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:52 smithi022 ceph-mon[40250]: pgmap v16508: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:54 smithi122 ceph-mon[32255]: pgmap v16509: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:54 smithi022 ceph-mon[31360]: pgmap v16509: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:54 smithi022 ceph-mon[40250]: pgmap v16509: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:08:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:56 smithi122 ceph-mon[32255]: pgmap v16510: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:56 smithi022 ceph-mon[31360]: pgmap v16510: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:56 smithi022 ceph-mon[40250]: pgmap v16510: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:56.735 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:08:57.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:08:57.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:08:57.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:08:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2887165935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2887165935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2887165935' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:08:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:08:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:08:58 smithi122 ceph-mon[32255]: pgmap v16511: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:08:58 smithi022 ceph-mon[31360]: pgmap v16511: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:08:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:08:58 smithi022 ceph-mon[40250]: pgmap v16511: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:00 smithi122 ceph-mon[32255]: pgmap v16512: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:00] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:09:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:00 smithi022 ceph-mon[31360]: pgmap v16512: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:00 smithi022 ceph-mon[40250]: pgmap v16512: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:00.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:09:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:09:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:02.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:02 smithi122 ceph-mon[32255]: pgmap v16513: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:09:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:09:02.475 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:02.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:02.476 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:02.476 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:02 smithi022 ceph-mon[31360]: pgmap v16513: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:02.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:02 smithi022 ceph-mon[40250]: pgmap v16513: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2581814973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2581814973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2581814973' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:04 smithi122 ceph-mon[32255]: pgmap v16514: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:04 smithi022 ceph-mon[31360]: pgmap v16514: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:04 smithi022 ceph-mon[40250]: pgmap v16514: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:06 smithi122 ceph-mon[32255]: pgmap v16515: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:06 smithi022 ceph-mon[31360]: pgmap v16515: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:06 smithi022 ceph-mon[40250]: pgmap v16515: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:07.477 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:09:07.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:07.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:07.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2346628095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:08 smithi122 ceph-mon[32255]: pgmap v16516: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2346628095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:08 smithi022 ceph-mon[31360]: pgmap v16516: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2346628095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:08 smithi022 ceph-mon[40250]: pgmap v16516: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:10 smithi122 ceph-mon[32255]: pgmap v16517: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:10] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:09:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:10 smithi022 ceph-mon[31360]: pgmap v16517: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:10 smithi022 ceph-mon[40250]: pgmap v16517: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:10.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:09:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:09:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:11.659 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:12 smithi122 ceph-mon[32255]: pgmap v16518: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:09:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:09:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:12 smithi022 ceph-mon[31360]: pgmap v16518: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:12 smithi022 ceph-mon[40250]: pgmap v16518: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:12.850 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:13.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:13.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:13.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1719868334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1719868334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1719868334' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:14 smithi122 ceph-mon[32255]: pgmap v16519: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:14 smithi022 ceph-mon[31360]: pgmap v16519: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:14 smithi022 ceph-mon[40250]: pgmap v16519: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:09:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:09:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:09:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:09:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:09:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:09:15.893 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:09:15.894 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:09:15.894 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:09:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:09:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:16 smithi122 ceph-mon[32255]: pgmap v16520: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:09:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:16 smithi022 ceph-mon[31360]: pgmap v16520: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:09:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:16 smithi022 ceph-mon[40250]: pgmap v16520: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:09:18.223 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:18.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:18.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:18.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:18 smithi122 ceph-mon[32255]: pgmap v16521: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1473876508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:18 smithi022 ceph-mon[31360]: pgmap v16521: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1473876508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:18 smithi022 ceph-mon[40250]: pgmap v16521: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1473876508' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:20] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:09:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:20 smithi122 ceph-mon[32255]: pgmap v16522: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:20 smithi022 ceph-mon[31360]: pgmap v16522: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:20 smithi022 ceph-mon[40250]: pgmap v16522: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:20.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:09:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:09:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:22 smithi122 ceph-mon[32255]: pgmap v16523: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:09:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:09:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:22 smithi022 ceph-mon[31360]: pgmap v16523: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:22 smithi022 ceph-mon[40250]: pgmap v16523: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:23.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:23.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:23.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:23.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3151293033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3151293033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3151293033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:25 smithi022 ceph-mon[31360]: pgmap v16524: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:25 smithi022 ceph-mon[40250]: pgmap v16524: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:25 smithi122 ceph-mon[32255]: pgmap v16524: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:26 smithi122 ceph-mon[32255]: pgmap v16525: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:26 smithi022 ceph-mon[31360]: pgmap v16525: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:26 smithi022 ceph-mon[40250]: pgmap v16525: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:09:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:28 smithi122 ceph-mon[32255]: pgmap v16526: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:28.967 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:28.967 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:28 smithi022 ceph-mon[31360]: pgmap v16526: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:28.968 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:28 smithi022 ceph-mon[40250]: pgmap v16526: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:29.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:29.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:29.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2085748797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2085748797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2085748797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:30] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:09:30.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:09:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:09:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:30 smithi122 ceph-mon[32255]: pgmap v16527: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:30 smithi022 ceph-mon[31360]: pgmap v16527: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:30 smithi022 ceph-mon[40250]: pgmap v16527: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:32.661 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:09:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:09:32.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:32 smithi122 ceph-mon[32255]: pgmap v16528: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:32 smithi022 ceph-mon[31360]: pgmap v16528: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:32 smithi022 ceph-mon[40250]: pgmap v16528: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:34.336 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:34 smithi122 ceph-mon[32255]: pgmap v16529: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:34 smithi022 ceph-mon[31360]: pgmap v16529: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:34 smithi022 ceph-mon[40250]: pgmap v16529: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:34.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:34.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:34.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:09:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:09:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:09:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:09:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3035425100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:09:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:09:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:09:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:09:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3035425100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:09:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:09:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:09:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:09:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3035425100' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:36 smithi122 ceph-mon[32255]: pgmap v16530: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:36 smithi022 ceph-mon[31360]: pgmap v16530: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:36 smithi022 ceph-mon[40250]: pgmap v16530: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:09:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:38 smithi122 ceph-mon[32255]: pgmap v16531: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:38 smithi022 ceph-mon[31360]: pgmap v16531: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:38 smithi022 ceph-mon[40250]: pgmap v16531: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:39.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:40.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:40.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:40.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:40] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:09:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2272064026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:40 smithi122 ceph-mon[32255]: pgmap v16532: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2272064026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:40 smithi022 ceph-mon[31360]: pgmap v16532: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2272064026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:40 smithi022 ceph-mon[40250]: pgmap v16532: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:40.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:09:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:09:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:42 smithi122 ceph-mon[32255]: pgmap v16533: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:09:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:09:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:42 smithi022 ceph-mon[31360]: pgmap v16533: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:42 smithi022 ceph-mon[40250]: pgmap v16533: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:44 smithi122 ceph-mon[32255]: pgmap v16534: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:44 smithi022 ceph-mon[31360]: pgmap v16534: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:44 smithi022 ceph-mon[40250]: pgmap v16534: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:45.078 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:45.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:45.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:45.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1896560400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1896560400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1896560400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:46 smithi022 ceph-mon[31360]: pgmap v16535: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:46 smithi022 ceph-mon[40250]: pgmap v16535: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:46 smithi122 ceph-mon[32255]: pgmap v16535: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:09:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:48 smithi122 ceph-mon[32255]: pgmap v16536: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:48 smithi022 ceph-mon[31360]: pgmap v16536: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:48 smithi022 ceph-mon[40250]: pgmap v16536: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:50 smithi122 ceph-mon[32255]: pgmap v16537: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:50] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T03:09:50.445 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:50.446 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:50 smithi022 ceph-mon[31360]: pgmap v16537: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:50.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:50 smithi022 ceph-mon[40250]: pgmap v16537: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:50.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:50.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:50.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:50.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:09:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:09:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2563351921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2563351921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:09:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2563351921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:52 smithi122 ceph-mon[32255]: pgmap v16538: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:09:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:09:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:52 smithi022 ceph-mon[31360]: pgmap v16538: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:52 smithi022 ceph-mon[40250]: pgmap v16538: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:54 smithi122 ceph-mon[32255]: pgmap v16539: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:54 smithi022 ceph-mon[31360]: pgmap v16539: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:54 smithi022 ceph-mon[40250]: pgmap v16539: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:09:55.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:09:56.152 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:56 smithi022 ceph-mon[40250]: pgmap v16540: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:56.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:56 smithi022 ceph-mon[31360]: pgmap v16540: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:56.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:09:56.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:09:56.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:09:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:56 smithi122 ceph-mon[32255]: pgmap v16540: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2030299806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2030299806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:57.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:09:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2030299806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:09:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:09:58 smithi122 ceph-mon[32255]: pgmap v16541: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:09:58 smithi022 ceph-mon[31360]: pgmap v16541: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:09:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:09:58 smithi022 ceph-mon[40250]: pgmap v16541: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T03:10:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:00] "GET /metrics HTTP/1.1" 200 202449 "" "Prometheus/2.18.1" 2021-12-16T03:10:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T03:10:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T03:10:00.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:10:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:10:01.188 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:01 smithi122 ceph-mon[32255]: pgmap v16542: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:01 smithi022 ceph-mon[31360]: pgmap v16542: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:01 smithi022 ceph-mon[40250]: pgmap v16542: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:01.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:01.556 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:01.557 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2347415853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2347415853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2347415853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:10:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:10:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:03 smithi122 ceph-mon[32255]: pgmap v16543: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:03 smithi022 ceph-mon[31360]: pgmap v16543: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:03 smithi022 ceph-mon[40250]: pgmap v16543: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:04 smithi122 ceph-mon[32255]: pgmap v16544: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:04 smithi022 ceph-mon[31360]: pgmap v16544: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:04 smithi022 ceph-mon[40250]: pgmap v16544: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:06 smithi122 ceph-mon[32255]: pgmap v16545: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:06 smithi022 ceph-mon[31360]: pgmap v16545: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:06 smithi022 ceph-mon[40250]: pgmap v16545: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:06.558 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:06.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:06.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:06.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1796707207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1796707207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1796707207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:10:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:08 smithi122 ceph-mon[32255]: pgmap v16546: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:08 smithi022 ceph-mon[31360]: pgmap v16546: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:08 smithi022 ceph-mon[40250]: pgmap v16546: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:10 smithi122 ceph-mon[32255]: pgmap v16547: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:10] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T03:10:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:10 smithi022 ceph-mon[31360]: pgmap v16547: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:10 smithi022 ceph-mon[40250]: pgmap v16547: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:10.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:10:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:10:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:11.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:12 smithi022 ceph-mon[31360]: pgmap v16548: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:12 smithi022 ceph-mon[40250]: pgmap v16548: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:12.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:12.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:12.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:12 smithi122 ceph-mon[32255]: pgmap v16548: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:10:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:10:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1041678577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1041678577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1041678577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:14 smithi122 ceph-mon[32255]: pgmap v16549: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:14 smithi022 ceph-mon[40250]: pgmap v16549: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:14 smithi022 ceph-mon[31360]: pgmap v16549: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:10:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:10:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:15 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:10:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:10:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:10:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:15 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:10:16.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:10:16.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:10:16.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:15 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:10:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:16 smithi022 ceph-mon[31360]: pgmap v16550: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:17.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:16 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:10:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:16 smithi022 ceph-mon[40250]: pgmap v16550: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:16 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:10:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:16 smithi122 ceph-mon[32255]: pgmap v16550: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:16 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:10:17.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:17.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:17.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:17.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:10:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1854622968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:18 smithi122 ceph-mon[32255]: pgmap v16551: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1854622968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:18 smithi022 ceph-mon[31360]: pgmap v16551: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1854622968' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:18 smithi022 ceph-mon[40250]: pgmap v16551: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:20] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:10:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:20 smithi122 ceph-mon[32255]: pgmap v16552: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:20 smithi022 ceph-mon[31360]: pgmap v16552: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:20 smithi022 ceph-mon[40250]: pgmap v16552: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:20.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:10:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:10:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:22 smithi122 ceph-mon[32255]: pgmap v16553: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:10:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:10:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:22 smithi022 ceph-mon[31360]: pgmap v16553: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:22 smithi022 ceph-mon[40250]: pgmap v16553: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:22.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:23.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:23.042 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:23.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3210894549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3210894549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3210894549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:24 smithi122 ceph-mon[32255]: pgmap v16554: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:24 smithi022 ceph-mon[31360]: pgmap v16554: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:24 smithi022 ceph-mon[40250]: pgmap v16554: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:26 smithi122 ceph-mon[32255]: pgmap v16555: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:26 smithi022 ceph-mon[31360]: pgmap v16555: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:26 smithi022 ceph-mon[40250]: pgmap v16555: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:10:28.045 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:28.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:28.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:28.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:28.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:28 smithi022 ceph-mon[40250]: pgmap v16556: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:28.663 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/101211247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:28 smithi122 ceph-mon[32255]: pgmap v16556: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/101211247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:28 smithi022 ceph-mon[31360]: pgmap v16556: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:29.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/101211247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:30] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:10:30.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:10:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:10:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:31 smithi122 ceph-mon[32255]: pgmap v16557: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:31 smithi022 ceph-mon[31360]: pgmap v16557: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:31 smithi022 ceph-mon[40250]: pgmap v16557: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:10:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:10:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:33 smithi022 ceph-mon[31360]: pgmap v16558: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:33 smithi022 ceph-mon[40250]: pgmap v16558: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:33.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:33.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:33.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:33 smithi122 ceph-mon[32255]: pgmap v16558: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3464222663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:10:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:10:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:10:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:10:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3464222663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:10:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:10:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:10:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:10:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3464222663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:10:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:10:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:10:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:10:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:35 smithi122 ceph-mon[32255]: pgmap v16559: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:35 smithi022 ceph-mon[31360]: pgmap v16559: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:35 smithi022 ceph-mon[40250]: pgmap v16559: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:37.668 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:10:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:37 smithi122 ceph-mon[32255]: pgmap v16560: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:38.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:37 smithi022 ceph-mon[40250]: pgmap v16560: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:38.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:37 smithi022 ceph-mon[31360]: pgmap v16560: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:38.782 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:39.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:39.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:39.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:39 smithi122 ceph-mon[32255]: pgmap v16561: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3131752785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:39 smithi022 ceph-mon[31360]: pgmap v16561: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3131752785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:39 smithi022 ceph-mon[40250]: pgmap v16561: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3131752785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:40] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:10:41.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:10:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:10:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:41 smithi122 ceph-mon[32255]: pgmap v16562: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:41 smithi022 ceph-mon[31360]: pgmap v16562: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:41 smithi022 ceph-mon[40250]: pgmap v16562: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:10:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:10:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:43 smithi122 ceph-mon[32255]: pgmap v16563: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:43 smithi022 ceph-mon[31360]: pgmap v16563: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:43 smithi022 ceph-mon[40250]: pgmap v16563: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:44.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:44.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:44.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/624329398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/624329398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/624329398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:45 smithi122 ceph-mon[32255]: pgmap v16564: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:45 smithi022 ceph-mon[31360]: pgmap v16564: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:45 smithi022 ceph-mon[40250]: pgmap v16564: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:47.686 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:10:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:47 smithi022 ceph-mon[31360]: pgmap v16565: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:47 smithi022 ceph-mon[40250]: pgmap v16565: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:47 smithi122 ceph-mon[32255]: pgmap v16565: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:49.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:49 smithi022 ceph-mon[31360]: pgmap v16566: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:49 smithi022 ceph-mon[40250]: pgmap v16566: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:49.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:49.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:49.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:49 smithi122 ceph-mon[32255]: pgmap v16566: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:10:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:10:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/811133197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/811133197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/811133197' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:51.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:10:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:10:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:51 smithi022 ceph-mon[31360]: pgmap v16567: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:51 smithi022 ceph-mon[40250]: pgmap v16567: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:51 smithi122 ceph-mon[32255]: pgmap v16567: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:10:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:10:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:10:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:53 smithi022 ceph-mon[31360]: pgmap v16568: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:53 smithi022 ceph-mon[40250]: pgmap v16568: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:53 smithi122 ceph-mon[32255]: pgmap v16568: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:54.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:10:55.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:10:55.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:10:55.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:10:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:55 smithi022 ceph-mon[31360]: pgmap v16569: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3389000797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:55 smithi022 ceph-mon[40250]: pgmap v16569: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3389000797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:55 smithi122 ceph-mon[32255]: pgmap v16569: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:10:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3389000797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:10:57.703 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:10:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:57 smithi022 ceph-mon[31360]: pgmap v16570: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:57 smithi022 ceph-mon[40250]: pgmap v16570: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:10:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:57 smithi122 ceph-mon[32255]: pgmap v16570: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:10:59 smithi022 ceph-mon[31360]: pgmap v16571: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:10:59 smithi022 ceph-mon[40250]: pgmap v16571: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:10:59 smithi122 ceph-mon[32255]: pgmap v16571: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:11:00.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:00.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:00.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3796785031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3796785031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3796785031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:01.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:11:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:11:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:01 smithi022 ceph-mon[31360]: pgmap v16572: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:01 smithi022 ceph-mon[40250]: pgmap v16572: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:01 smithi122 ceph-mon[32255]: pgmap v16572: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:11:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:11:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:03 smithi022 ceph-mon[31360]: pgmap v16573: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:03 smithi022 ceph-mon[40250]: pgmap v16573: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:03 smithi122 ceph-mon[32255]: pgmap v16573: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:05.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:06.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:06.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:06.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:06.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:05 smithi022 ceph-mon[31360]: pgmap v16574: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:06.006 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:05 smithi022 ceph-mon[40250]: pgmap v16574: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:05 smithi122 ceph-mon[32255]: pgmap v16574: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4031312316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4031312316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4031312316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:07.722 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:11:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:07 smithi022 ceph-mon[31360]: pgmap v16575: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:07 smithi022 ceph-mon[40250]: pgmap v16575: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:07 smithi122 ceph-mon[32255]: pgmap v16575: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:09 smithi022 ceph-mon[31360]: pgmap v16576: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:09 smithi022 ceph-mon[40250]: pgmap v16576: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:09 smithi122 ceph-mon[32255]: pgmap v16576: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:10] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:11:11.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:11.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:11:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:11:11.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:11.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:11.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:11 smithi022 ceph-mon[31360]: pgmap v16577: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/82048784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:11 smithi022 ceph-mon[40250]: pgmap v16577: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/82048784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:11 smithi122 ceph-mon[32255]: pgmap v16577: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/82048784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:11:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:11:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:13 smithi022 ceph-mon[31360]: pgmap v16578: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:13 smithi022 ceph-mon[40250]: pgmap v16578: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:13 smithi122 ceph-mon[32255]: pgmap v16578: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:15 smithi022 ceph-mon[31360]: pgmap v16579: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:15 smithi022 ceph-mon[40250]: pgmap v16579: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:15 smithi122 ceph-mon[32255]: pgmap v16579: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:16.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:11:16.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:11:16.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:16 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:11:16.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:16.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:16.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:11:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:11:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:16 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:11:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:11:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:11:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:16 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:11:17.739 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:11:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:17 smithi022 ceph-mon[31360]: pgmap v16580: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2723757545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:11:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:17 smithi022 ceph-mon[40250]: pgmap v16580: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2723757545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:11:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:17 smithi122 ceph-mon[32255]: pgmap v16580: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2723757545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:11:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:19 smithi022 ceph-mon[31360]: pgmap v16581: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:19 smithi022 ceph-mon[40250]: pgmap v16581: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:19 smithi122 ceph-mon[32255]: pgmap v16581: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:20] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:11:21.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:11:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:11:21.768 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:21 smithi022 ceph-mon[31360]: pgmap v16582: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:21 smithi022 ceph-mon[40250]: pgmap v16582: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:22.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:22.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:22.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:21 smithi122 ceph-mon[32255]: pgmap v16582: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:11:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:11:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1269065458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1269065458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1269065458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:23 smithi022 ceph-mon[31360]: pgmap v16583: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:23 smithi022 ceph-mon[40250]: pgmap v16583: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:23 smithi122 ceph-mon[32255]: pgmap v16583: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:25 smithi022 ceph-mon[31360]: pgmap v16584: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:25 smithi022 ceph-mon[40250]: pgmap v16584: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:25 smithi122 ceph-mon[32255]: pgmap v16584: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:27.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:27.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:27.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:27.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:27.757 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:11:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:27 smithi022 ceph-mon[31360]: pgmap v16585: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1705878274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:27 smithi022 ceph-mon[40250]: pgmap v16585: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:28.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1705878274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:27 smithi122 ceph-mon[32255]: pgmap v16585: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1705878274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:29 smithi022 ceph-mon[31360]: pgmap v16586: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:30.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:29 smithi022 ceph-mon[40250]: pgmap v16586: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:29 smithi122 ceph-mon[32255]: pgmap v16586: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:30] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:11:31.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:11:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:11:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:31 smithi022 ceph-mon[31360]: pgmap v16587: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:31 smithi022 ceph-mon[40250]: pgmap v16587: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:31 smithi122 ceph-mon[32255]: pgmap v16587: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:32.505 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:11:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:11:32.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:32.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:32.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:33 smithi022 ceph-mon[31360]: pgmap v16588: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/38131716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:33 smithi022 ceph-mon[40250]: pgmap v16588: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/38131716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:33 smithi122 ceph-mon[32255]: pgmap v16588: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/38131716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:11:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:11:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:11:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:11:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:11:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:11:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:11:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:11:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:11:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:11:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:11:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:11:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:35 smithi022 ceph-mon[31360]: pgmap v16589: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:35 smithi022 ceph-mon[40250]: pgmap v16589: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:35 smithi122 ceph-mon[32255]: pgmap v16589: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:37.774 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:11:37.884 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:37 smithi022 ceph-mon[31360]: pgmap v16590: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:37 smithi022 ceph-mon[40250]: pgmap v16590: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:37 smithi122 ceph-mon[32255]: pgmap v16590: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:38.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:38.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:38.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1313836249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1313836249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1313836249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:39 smithi122 ceph-mon[32255]: pgmap v16591: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:39 smithi022 ceph-mon[31360]: pgmap v16591: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:39 smithi022 ceph-mon[40250]: pgmap v16591: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:11:41.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:11:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:11:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:41 smithi122 ceph-mon[32255]: pgmap v16592: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:41 smithi022 ceph-mon[31360]: pgmap v16592: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:41 smithi022 ceph-mon[40250]: pgmap v16592: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:11:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:11:43.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:43.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:43.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:43.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:43 smithi022 ceph-mon[31360]: pgmap v16593: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/573359447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:43 smithi022 ceph-mon[40250]: pgmap v16593: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/573359447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:43 smithi122 ceph-mon[32255]: pgmap v16593: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/573359447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:45 smithi122 ceph-mon[32255]: pgmap v16594: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:45 smithi022 ceph-mon[31360]: pgmap v16594: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:45 smithi022 ceph-mon[40250]: pgmap v16594: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:11:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:47 smithi122 ceph-mon[32255]: pgmap v16595: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:47 smithi022 ceph-mon[31360]: pgmap v16595: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:47 smithi022 ceph-mon[40250]: pgmap v16595: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:48.624 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:48.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:48.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:48.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:49 smithi122 ceph-mon[32255]: pgmap v16596: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2233116921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:49 smithi022 ceph-mon[31360]: pgmap v16596: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2233116921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:49 smithi022 ceph-mon[40250]: pgmap v16596: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2233116921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:50] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T03:11:51.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:11:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:11:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:51 smithi122 ceph-mon[32255]: pgmap v16597: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:51 smithi022 ceph-mon[31360]: pgmap v16597: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:51 smithi022 ceph-mon[40250]: pgmap v16597: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:11:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:11:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:11:54.000 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:53 smithi122 ceph-mon[32255]: pgmap v16598: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:53 smithi022 ceph-mon[31360]: pgmap v16598: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:53 smithi022 ceph-mon[40250]: pgmap v16598: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:54.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:54.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:54.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:11:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1375238125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1375238125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1375238125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:11:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:55 smithi122 ceph-mon[32255]: pgmap v16599: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:55 smithi022 ceph-mon[31360]: pgmap v16599: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:55 smithi022 ceph-mon[40250]: pgmap v16599: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:11:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:11:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:57 smithi122 ceph-mon[32255]: pgmap v16600: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:57 smithi022 ceph-mon[31360]: pgmap v16600: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:57 smithi022 ceph-mon[40250]: pgmap v16600: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:11:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:11:59.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:11:59.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:11:59.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:59 smithi022 ceph-mon[31360]: pgmap v16601: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:11:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1318886583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:59 smithi022 ceph-mon[40250]: pgmap v16601: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:11:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1318886583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:59 smithi122 ceph-mon[32255]: pgmap v16601: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:11:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1318886583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:00] "GET /metrics HTTP/1.1" 200 202456 "" "Prometheus/2.18.1" 2021-12-16T03:12:01.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:12:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:12:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:01 smithi122 ceph-mon[32255]: pgmap v16602: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:01 smithi022 ceph-mon[31360]: pgmap v16602: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:01 smithi022 ceph-mon[40250]: pgmap v16602: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:12:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:12:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:03 smithi122 ceph-mon[32255]: pgmap v16603: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:03 smithi022 ceph-mon[31360]: pgmap v16603: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:03 smithi022 ceph-mon[40250]: pgmap v16603: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:04.740 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:05.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:05 smithi122 ceph-mon[32255]: pgmap v16604: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/733486115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:05 smithi022 ceph-mon[40250]: pgmap v16604: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/733486115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:05 smithi022 ceph-mon[31360]: pgmap v16604: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/733486115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:12:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:07 smithi122 ceph-mon[32255]: pgmap v16605: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:07 smithi022 ceph-mon[31360]: pgmap v16605: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:07 smithi022 ceph-mon[40250]: pgmap v16605: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:10.114 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:10.115 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:09 smithi022 ceph-mon[31360]: pgmap v16606: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:10.115 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:09 smithi022 ceph-mon[40250]: pgmap v16606: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:09 smithi122 ceph-mon[32255]: pgmap v16606: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:10] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:12:10.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:10.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:10.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2249801725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:11.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:12:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:12:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2249801725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2249801725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:11 smithi122 ceph-mon[32255]: pgmap v16607: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:11 smithi022 ceph-mon[31360]: pgmap v16607: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:11 smithi022 ceph-mon[40250]: pgmap v16607: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:12:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:12:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:13 smithi122 ceph-mon[32255]: pgmap v16608: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:13 smithi022 ceph-mon[31360]: pgmap v16608: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:13 smithi022 ceph-mon[40250]: pgmap v16608: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:15.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:15.854 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:16.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:16.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:15 smithi122 ceph-mon[32255]: pgmap v16609: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:15 smithi022 ceph-mon[31360]: pgmap v16609: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:15 smithi022 ceph-mon[40250]: pgmap v16609: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3514585503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3514585503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3514585503' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:12:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:17 smithi122 ceph-mon[32255]: pgmap v16610: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:12:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:12:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:12:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:12:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[31360]: pgmap v16610: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:12:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:12:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:12:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:12:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[40250]: pgmap v16610: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:12:18.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:12:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:12:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:12:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:19 smithi122 ceph-mon[32255]: pgmap v16611: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:19 smithi022 ceph-mon[31360]: pgmap v16611: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:19 smithi022 ceph-mon[40250]: pgmap v16611: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:20] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:12:20.856 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:21.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:12:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:12:21.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:21.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:21.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:21 smithi122 ceph-mon[32255]: pgmap v16612: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1041416539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:21 smithi022 ceph-mon[31360]: pgmap v16612: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1041416539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:21 smithi022 ceph-mon[40250]: pgmap v16612: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1041416539' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:12:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:12:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:23 smithi122 ceph-mon[32255]: pgmap v16613: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:23 smithi022 ceph-mon[31360]: pgmap v16613: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:23 smithi022 ceph-mon[40250]: pgmap v16613: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:25 smithi122 ceph-mon[32255]: pgmap v16614: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:26.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:26.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:25 smithi022 ceph-mon[31360]: pgmap v16614: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:26.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:25 smithi022 ceph-mon[40250]: pgmap v16614: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:26.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:26.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2259021196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2259021196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2259021196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:12:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:27 smithi122 ceph-mon[32255]: pgmap v16615: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:27 smithi022 ceph-mon[31360]: pgmap v16615: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:27 smithi022 ceph-mon[40250]: pgmap v16615: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:29 smithi122 ceph-mon[32255]: pgmap v16616: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:29 smithi022 ceph-mon[31360]: pgmap v16616: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:29 smithi022 ceph-mon[40250]: pgmap v16616: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:30] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:12:31.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:12:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:12:31.598 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:31.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:31.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:31.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:31 smithi122 ceph-mon[32255]: pgmap v16617: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:31 smithi022 ceph-mon[31360]: pgmap v16617: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:31 smithi022 ceph-mon[40250]: pgmap v16617: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:12:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:12:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2364114617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2364114617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2364114617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:33 smithi122 ceph-mon[32255]: pgmap v16618: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:33 smithi022 ceph-mon[31360]: pgmap v16618: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:33 smithi022 ceph-mon[40250]: pgmap v16618: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:12:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:12:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:12:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:12:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:12:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:12:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:12:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:12:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:12:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:12:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:12:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:12:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:35 smithi122 ceph-mon[32255]: pgmap v16619: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:35 smithi022 ceph-mon[31360]: pgmap v16619: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:35 smithi022 ceph-mon[40250]: pgmap v16619: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:36.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:37.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:37.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:37.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:12:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:37 smithi122 ceph-mon[32255]: pgmap v16620: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/320684608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:37 smithi022 ceph-mon[31360]: pgmap v16620: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/320684608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:37 smithi022 ceph-mon[40250]: pgmap v16620: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/320684608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:39 smithi122 ceph-mon[32255]: pgmap v16621: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:39 smithi022 ceph-mon[31360]: pgmap v16621: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:39 smithi022 ceph-mon[40250]: pgmap v16621: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:40] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:12:41.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:12:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:12:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:41 smithi122 ceph-mon[32255]: pgmap v16622: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:41 smithi022 ceph-mon[31360]: pgmap v16622: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:41 smithi022 ceph-mon[40250]: pgmap v16622: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:42.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:12:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:12:42.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:42.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:42.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:43 smithi022 ceph-mon[31360]: pgmap v16623: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/913821569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:43 smithi022 ceph-mon[40250]: pgmap v16623: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/913821569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:43 smithi122 ceph-mon[32255]: pgmap v16623: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/913821569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:45 smithi022 ceph-mon[31360]: pgmap v16624: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:45 smithi022 ceph-mon[40250]: pgmap v16624: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:45 smithi122 ceph-mon[32255]: pgmap v16624: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:47.692 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:12:47.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:47 smithi022 ceph-mon[31360]: pgmap v16625: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:47 smithi022 ceph-mon[40250]: pgmap v16625: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:48.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:48.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:48.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:47 smithi122 ceph-mon[32255]: pgmap v16625: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2596054718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2596054718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2596054718' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:49 smithi022 ceph-mon[31360]: pgmap v16626: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:49 smithi022 ceph-mon[40250]: pgmap v16626: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:49 smithi122 ceph-mon[32255]: pgmap v16626: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:50] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T03:12:51.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:12:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:12:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:51 smithi022 ceph-mon[31360]: pgmap v16627: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:51 smithi022 ceph-mon[40250]: pgmap v16627: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:51 smithi122 ceph-mon[32255]: pgmap v16627: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:12:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:12:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:12:53.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:53.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:53.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:53.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:12:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:53 smithi022 ceph-mon[31360]: pgmap v16628: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1594647848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:53 smithi022 ceph-mon[40250]: pgmap v16628: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1594647848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:53 smithi122 ceph-mon[32255]: pgmap v16628: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1594647848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:12:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:55 smithi022 ceph-mon[31360]: pgmap v16629: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:55 smithi022 ceph-mon[40250]: pgmap v16629: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:55 smithi122 ceph-mon[32255]: pgmap v16629: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:12:57.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:12:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:57 smithi022 ceph-mon[31360]: pgmap v16630: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:57 smithi022 ceph-mon[40250]: pgmap v16630: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:57 smithi122 ceph-mon[32255]: pgmap v16630: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:12:58.453 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:12:58.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:12:58.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:12:58.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:59 smithi022 ceph-mon[31360]: pgmap v16631: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:00.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:12:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2287849791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:59 smithi022 ceph-mon[40250]: pgmap v16631: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:12:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2287849791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:59 smithi122 ceph-mon[32255]: pgmap v16631: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:12:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2287849791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:00] "GET /metrics HTTP/1.1" 200 202452 "" "Prometheus/2.18.1" 2021-12-16T03:13:01.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:13:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:13:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:01 smithi022 ceph-mon[31360]: pgmap v16632: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:01 smithi022 ceph-mon[40250]: pgmap v16632: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:01 smithi122 ceph-mon[32255]: pgmap v16632: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:13:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:13:03.826 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:03 smithi022 ceph-mon[31360]: pgmap v16633: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:04.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:03 smithi022 ceph-mon[40250]: pgmap v16633: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:03 smithi122 ceph-mon[32255]: pgmap v16633: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:04.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:04.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:04.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1618356806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:05.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1618356806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1618356806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:05 smithi022 ceph-mon[31360]: pgmap v16634: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:05 smithi022 ceph-mon[40250]: pgmap v16634: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:05 smithi122 ceph-mon[32255]: pgmap v16634: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:07.726 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:13:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:07 smithi022 ceph-mon[31360]: pgmap v16635: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:07 smithi022 ceph-mon[40250]: pgmap v16635: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:07 smithi122 ceph-mon[32255]: pgmap v16635: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:09.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:09.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:09.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:09.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:09 smithi022 ceph-mon[31360]: pgmap v16636: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:10.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1127082653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:09 smithi022 ceph-mon[40250]: pgmap v16636: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1127082653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:09 smithi122 ceph-mon[32255]: pgmap v16636: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1127082653' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:10] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:13:11.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:13:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:13:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:11 smithi022 ceph-mon[31360]: pgmap v16637: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:11 smithi022 ceph-mon[40250]: pgmap v16637: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:11 smithi122 ceph-mon[32255]: pgmap v16637: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:13:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:13:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:13 smithi022 ceph-mon[31360]: pgmap v16638: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:13 smithi022 ceph-mon[40250]: pgmap v16638: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:13 smithi122 ceph-mon[32255]: pgmap v16638: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:14.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:15 smithi022 ceph-mon[31360]: pgmap v16639: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1944998275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:15 smithi022 ceph-mon[40250]: pgmap v16639: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1944998275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:15 smithi122 ceph-mon[32255]: pgmap v16639: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1944998275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:17.739 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:13:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[31360]: pgmap v16640: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:13:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:13:18.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:13:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[40250]: pgmap v16640: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:13:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:13:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:13:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:17 smithi122 ceph-mon[32255]: pgmap v16640: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:13:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:13:18.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:13:19.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:18 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:13:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:18 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:13:19.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:18 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:13:19.936 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:19 smithi122 ceph-mon[32255]: pgmap v16641: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:19 smithi022 ceph-mon[31360]: pgmap v16641: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:19 smithi022 ceph-mon[40250]: pgmap v16641: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:20.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:20.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:20.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:20] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:13:21.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:13:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:13:21.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1526149781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1526149781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1526149781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:21 smithi122 ceph-mon[32255]: pgmap v16642: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:21 smithi022 ceph-mon[31360]: pgmap v16642: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:21 smithi022 ceph-mon[40250]: pgmap v16642: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:13:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:13:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:23 smithi122 ceph-mon[32255]: pgmap v16643: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:23 smithi022 ceph-mon[31360]: pgmap v16643: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:23 smithi022 ceph-mon[40250]: pgmap v16643: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:25.307 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:25.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:25.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:25.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:25 smithi022 ceph-mon[31360]: pgmap v16644: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4051263700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:25 smithi022 ceph-mon[40250]: pgmap v16644: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4051263700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:25 smithi122 ceph-mon[32255]: pgmap v16644: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4051263700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:13:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:27 smithi122 ceph-mon[32255]: pgmap v16645: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:27 smithi022 ceph-mon[31360]: pgmap v16645: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:27 smithi022 ceph-mon[40250]: pgmap v16645: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:29 smithi122 ceph-mon[32255]: pgmap v16646: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:29 smithi022 ceph-mon[31360]: pgmap v16646: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:29 smithi022 ceph-mon[40250]: pgmap v16646: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:30] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:13:30.679 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:31.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:31.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:31.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:31.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:13:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:13:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:31 smithi122 ceph-mon[32255]: pgmap v16647: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:32.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/391546050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:31 smithi022 ceph-mon[31360]: pgmap v16647: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/391546050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:31 smithi022 ceph-mon[40250]: pgmap v16647: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/391546050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:13:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:13:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:33 smithi122 ceph-mon[32255]: pgmap v16648: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:33 smithi022 ceph-mon[31360]: pgmap v16648: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:33 smithi022 ceph-mon[40250]: pgmap v16648: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:13:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:13:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:13:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:13:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:13:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:13:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:13:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:13:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:13:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:13:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:13:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:13:36.050 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:35 smithi122 ceph-mon[32255]: pgmap v16649: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:35 smithi022 ceph-mon[31360]: pgmap v16649: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:35 smithi022 ceph-mon[40250]: pgmap v16649: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:36.417 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1662994313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1662994313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1662994313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:13:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:37 smithi122 ceph-mon[32255]: pgmap v16650: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:37 smithi022 ceph-mon[31360]: pgmap v16650: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:37 smithi022 ceph-mon[40250]: pgmap v16650: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:39 smithi122 ceph-mon[32255]: pgmap v16651: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:39 smithi022 ceph-mon[31360]: pgmap v16651: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:39 smithi022 ceph-mon[40250]: pgmap v16651: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:40] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:13:41.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:13:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:13:41.419 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:41.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:41.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:41.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:41 smithi122 ceph-mon[32255]: pgmap v16652: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4069294037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:41 smithi022 ceph-mon[31360]: pgmap v16652: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4069294037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:41 smithi022 ceph-mon[40250]: pgmap v16652: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4069294037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:13:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:13:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:43 smithi122 ceph-mon[32255]: pgmap v16653: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:43 smithi022 ceph-mon[31360]: pgmap v16653: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:43 smithi022 ceph-mon[40250]: pgmap v16653: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:45 smithi122 ceph-mon[32255]: pgmap v16654: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:45 smithi022 ceph-mon[31360]: pgmap v16654: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:45 smithi022 ceph-mon[40250]: pgmap v16654: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:46.783 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:47.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:47.153 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:47.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:13:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:47 smithi122 ceph-mon[32255]: pgmap v16655: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1833935018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:47 smithi022 ceph-mon[31360]: pgmap v16655: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1833935018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:47 smithi022 ceph-mon[40250]: pgmap v16655: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1833935018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:49 smithi122 ceph-mon[32255]: pgmap v16656: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:49 smithi022 ceph-mon[31360]: pgmap v16656: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:49 smithi022 ceph-mon[40250]: pgmap v16656: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:50] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:13:51.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:13:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:13:52.155 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:52.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:51 smithi022 ceph-mon[31360]: pgmap v16657: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:52.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:52.156 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:51 smithi022 ceph-mon[40250]: pgmap v16657: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:52.157 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:51 smithi122 ceph-mon[32255]: pgmap v16657: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:13:52.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:52.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:52.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:13:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:13:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4243000018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4243000018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4243000018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:53 smithi122 ceph-mon[32255]: pgmap v16658: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:53 smithi022 ceph-mon[31360]: pgmap v16658: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:53 smithi022 ceph-mon[40250]: pgmap v16658: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:55 smithi122 ceph-mon[32255]: pgmap v16659: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:55 smithi022 ceph-mon[31360]: pgmap v16659: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:55 smithi022 ceph-mon[40250]: pgmap v16659: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:13:57.526 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:13:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:13:57.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:13:57.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:13:57.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:13:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:57 smithi122 ceph-mon[32255]: pgmap v16660: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:57 smithi022 ceph-mon[31360]: pgmap v16660: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:57 smithi022 ceph-mon[40250]: pgmap v16660: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:13:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1787167757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1787167757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:13:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1787167757' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:13:59 smithi122 ceph-mon[32255]: pgmap v16661: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:13:59 smithi022 ceph-mon[31360]: pgmap v16661: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:13:59 smithi022 ceph-mon[40250]: pgmap v16661: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:00] "GET /metrics HTTP/1.1" 200 202454 "" "Prometheus/2.18.1" 2021-12-16T03:14:01.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:14:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:14:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:01 smithi122 ceph-mon[32255]: pgmap v16662: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:01 smithi022 ceph-mon[31360]: pgmap v16662: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:01 smithi022 ceph-mon[40250]: pgmap v16662: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:14:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:14:02.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:03.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:03.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:03 smithi122 ceph-mon[32255]: pgmap v16663: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1514561775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:03 smithi022 ceph-mon[31360]: pgmap v16663: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1514561775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:03 smithi022 ceph-mon[40250]: pgmap v16663: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1514561775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:05 smithi122 ceph-mon[32255]: pgmap v16664: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:05 smithi022 ceph-mon[31360]: pgmap v16664: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:05 smithi022 ceph-mon[40250]: pgmap v16664: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:14:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:07 smithi122 ceph-mon[32255]: pgmap v16665: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:08.270 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:08.271 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:07 smithi022 ceph-mon[31360]: pgmap v16665: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:08.272 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:07 smithi022 ceph-mon[40250]: pgmap v16665: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:08.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:08.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:08.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:09.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1420924940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1420924940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1420924940' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:09 smithi122 ceph-mon[32255]: pgmap v16666: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:09 smithi022 ceph-mon[31360]: pgmap v16666: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:09 smithi022 ceph-mon[40250]: pgmap v16666: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:10] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:14:11.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:14:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:14:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:11 smithi122 ceph-mon[32255]: pgmap v16667: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:11 smithi022 ceph-mon[31360]: pgmap v16667: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:11 smithi022 ceph-mon[40250]: pgmap v16667: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:14:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:14:13.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:14.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:14.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:14.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:14.010 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:13 smithi022 ceph-mon[31360]: pgmap v16668: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:14.011 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:13 smithi022 ceph-mon[40250]: pgmap v16668: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:13 smithi122 ceph-mon[32255]: pgmap v16668: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/341498603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/341498603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/341498603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:15 smithi122 ceph-mon[32255]: pgmap v16669: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:15 smithi022 ceph-mon[31360]: pgmap v16669: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:15 smithi022 ceph-mon[40250]: pgmap v16669: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:14:18.153 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:17 smithi122 ceph-mon[32255]: pgmap v16670: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:18.154 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:14:18.154 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:14:18.154 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:17 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:14:18.182 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[31360]: pgmap v16670: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:18.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:14:18.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:14:18.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:14:18.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[40250]: pgmap v16670: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:18.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:14:18.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:14:18.184 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:17 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:14:19.011 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[40250]: pgmap v16671: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[31360]: pgmap v16671: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:19.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:19.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:19 smithi122 ceph-mon[32255]: pgmap v16671: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:19.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:14:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:20] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:14:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/939059556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/939059556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/939059556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:21.170 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:14:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:14:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:21 smithi122 ceph-mon[32255]: pgmap v16672: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:21 smithi022 ceph-mon[31360]: pgmap v16672: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:21 smithi022 ceph-mon[40250]: pgmap v16672: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:14:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:14:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:23 smithi022 ceph-mon[31360]: pgmap v16673: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:23 smithi022 ceph-mon[40250]: pgmap v16673: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:23 smithi122 ceph-mon[32255]: pgmap v16673: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:24.378 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:24.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:24.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:24.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:25 smithi022 ceph-mon[31360]: pgmap v16674: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3095019807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:25 smithi022 ceph-mon[40250]: pgmap v16674: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3095019807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:25 smithi122 ceph-mon[32255]: pgmap v16674: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3095019807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:27.716 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:14:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:27 smithi022 ceph-mon[31360]: pgmap v16675: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:27 smithi022 ceph-mon[40250]: pgmap v16675: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:27 smithi122 ceph-mon[32255]: pgmap v16675: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:29.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:29 smithi022 ceph-mon[40250]: pgmap v16676: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:29 smithi022 ceph-mon[31360]: pgmap v16676: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:30.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:29 smithi122 ceph-mon[32255]: pgmap v16676: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:30] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:14:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/623559962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/623559962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:31.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/623559962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:14:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:14:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:31 smithi022 ceph-mon[31360]: pgmap v16677: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:31 smithi022 ceph-mon[40250]: pgmap v16677: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:31 smithi122 ceph-mon[32255]: pgmap v16677: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:14:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:14:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:33 smithi022 ceph-mon[31360]: pgmap v16678: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:34.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:33 smithi022 ceph-mon[40250]: pgmap v16678: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:33 smithi122 ceph-mon[32255]: pgmap v16678: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:14:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:14:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:14:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:14:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:14:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:14:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:14:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:14:35.119 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:14:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:14:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:14:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:14:35.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:35 smithi022 ceph-mon[31360]: pgmap v16679: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508930821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:35 smithi022 ceph-mon[40250]: pgmap v16679: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508930821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:35 smithi122 ceph-mon[32255]: pgmap v16679: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508930821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:37.736 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:14:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:37 smithi022 ceph-mon[31360]: pgmap v16680: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:37 smithi022 ceph-mon[40250]: pgmap v16680: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:37 smithi122 ceph-mon[32255]: pgmap v16680: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:39 smithi022 ceph-mon[31360]: pgmap v16681: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:39 smithi022 ceph-mon[40250]: pgmap v16681: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:39 smithi122 ceph-mon[32255]: pgmap v16681: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:40] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:14:40.492 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:40.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:40.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:40.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:41.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:14:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:14:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:41 smithi022 ceph-mon[31360]: pgmap v16682: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2460304601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:41 smithi022 ceph-mon[40250]: pgmap v16682: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2460304601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:41 smithi122 ceph-mon[32255]: pgmap v16682: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2460304601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:14:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:14:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:43 smithi022 ceph-mon[31360]: pgmap v16683: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:43 smithi022 ceph-mon[40250]: pgmap v16683: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:43 smithi122 ceph-mon[32255]: pgmap v16683: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:45.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:45 smithi022 ceph-mon[31360]: pgmap v16684: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:45 smithi022 ceph-mon[40250]: pgmap v16684: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:45 smithi122 ceph-mon[32255]: pgmap v16684: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:46.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:46.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:46.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2995026524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2995026524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:47.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2995026524' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:47.753 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:14:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:47 smithi022 ceph-mon[31360]: pgmap v16685: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:47 smithi022 ceph-mon[40250]: pgmap v16685: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:47 smithi122 ceph-mon[32255]: pgmap v16685: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:50.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:49 smithi022 ceph-mon[31360]: pgmap v16686: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:49 smithi022 ceph-mon[40250]: pgmap v16686: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:49 smithi122 ceph-mon[32255]: pgmap v16686: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:50] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:14:51.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:14:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:14:51.233 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:51.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:51.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:51.603 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:51 smithi022 ceph-mon[31360]: pgmap v16687: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:52.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/7959577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:51 smithi022 ceph-mon[40250]: pgmap v16687: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:52.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/7959577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:51 smithi122 ceph-mon[32255]: pgmap v16687: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:14:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/7959577' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:14:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:14:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:53 smithi022 ceph-mon[31360]: pgmap v16688: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:54.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:53 smithi022 ceph-mon[40250]: pgmap v16688: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:53 smithi122 ceph-mon[32255]: pgmap v16688: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:55 smithi022 ceph-mon[31360]: pgmap v16689: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:55 smithi022 ceph-mon[40250]: pgmap v16689: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:55 smithi122 ceph-mon[32255]: pgmap v16689: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:14:56.604 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:14:56.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:14:56.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:14:56.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:14:57.766 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:14:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:57 smithi022 ceph-mon[31360]: pgmap v16690: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3733975003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:57 smithi022 ceph-mon[40250]: pgmap v16690: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3733975003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:14:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:57 smithi122 ceph-mon[32255]: pgmap v16690: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:14:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3733975003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:14:59 smithi022 ceph-mon[31360]: pgmap v16691: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:14:59 smithi022 ceph-mon[40250]: pgmap v16691: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:14:59 smithi122 ceph-mon[32255]: pgmap v16691: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:00] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:15:01.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:15:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:15:01.981 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:01 smithi022 ceph-mon[31360]: pgmap v16692: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:01 smithi022 ceph-mon[40250]: pgmap v16692: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:01 smithi122 ceph-mon[32255]: pgmap v16692: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:02.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:02.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:02.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:15:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:15:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1731967660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1731967660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1731967660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:03 smithi122 ceph-mon[32255]: pgmap v16693: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:03 smithi022 ceph-mon[31360]: pgmap v16693: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:03 smithi022 ceph-mon[40250]: pgmap v16693: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:05 smithi122 ceph-mon[32255]: pgmap v16694: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:05 smithi022 ceph-mon[31360]: pgmap v16694: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:05 smithi022 ceph-mon[40250]: pgmap v16694: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:07.349 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:07.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:07.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:07.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:07.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:15:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:07 smithi022 ceph-mon[31360]: pgmap v16695: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:08.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3141272900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:07 smithi022 ceph-mon[40250]: pgmap v16695: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:08.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3141272900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:07 smithi122 ceph-mon[32255]: pgmap v16695: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3141272900' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:09 smithi122 ceph-mon[32255]: pgmap v16696: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:10.295 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:09 smithi022 ceph-mon[31360]: pgmap v16696: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:10.295 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:09 smithi022 ceph-mon[40250]: pgmap v16696: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:10] "GET /metrics HTTP/1.1" 200 202455 "" "Prometheus/2.18.1" 2021-12-16T03:15:11.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:15:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:15:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:11 smithi122 ceph-mon[32255]: pgmap v16697: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:11 smithi022 ceph-mon[31360]: pgmap v16697: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:11 smithi022 ceph-mon[40250]: pgmap v16697: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:15:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:15:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:13.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:13.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:13.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:13 smithi122 ceph-mon[32255]: pgmap v16698: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2632804681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:13 smithi022 ceph-mon[31360]: pgmap v16698: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2632804681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:13 smithi022 ceph-mon[40250]: pgmap v16698: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2632804681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:15 smithi122 ceph-mon[32255]: pgmap v16699: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:15 smithi022 ceph-mon[31360]: pgmap v16699: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:15 smithi022 ceph-mon[40250]: pgmap v16699: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:15:18.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:18.094 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:17 smithi022 ceph-mon[31360]: pgmap v16700: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:18.095 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:17 smithi022 ceph-mon[40250]: pgmap v16700: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:17 smithi122 ceph-mon[32255]: pgmap v16700: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3180660637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:15:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:15:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:18 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:15:19.068 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3180660637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:19.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:15:19.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:15:19.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:15:19.069 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3180660637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:19.070 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:15:19.070 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:15:19.070 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:18 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:15:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:20] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:15:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:20 smithi122 ceph-mon[32255]: pgmap v16701: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:15:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:20 smithi022 ceph-mon[31360]: pgmap v16701: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:15:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:20 smithi022 ceph-mon[40250]: pgmap v16701: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:15:21.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:15:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:15:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:22 smithi122 ceph-mon[32255]: pgmap v16702: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:15:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:15:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:22 smithi022 ceph-mon[31360]: pgmap v16702: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:22 smithi022 ceph-mon[40250]: pgmap v16702: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:23.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:23.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:23.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:23.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:24 smithi022 ceph-mon[31360]: pgmap v16703: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/702241711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:24 smithi022 ceph-mon[40250]: pgmap v16703: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:24.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/702241711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:24 smithi122 ceph-mon[32255]: pgmap v16703: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/702241711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:26 smithi122 ceph-mon[32255]: pgmap v16704: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:26 smithi022 ceph-mon[31360]: pgmap v16704: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:26 smithi022 ceph-mon[40250]: pgmap v16704: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:15:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:28 smithi122 ceph-mon[32255]: pgmap v16705: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:28 smithi022 ceph-mon[31360]: pgmap v16705: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:28 smithi022 ceph-mon[40250]: pgmap v16705: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:28.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:29.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:29.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:29.211 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:30] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:15:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:30 smithi122 ceph-mon[32255]: pgmap v16706: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4222642103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:30 smithi022 ceph-mon[31360]: pgmap v16706: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4222642103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:30 smithi022 ceph-mon[40250]: pgmap v16706: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4222642103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:15:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:15:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:15:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:15:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:32 smithi122 ceph-mon[32255]: pgmap v16707: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:32 smithi022 ceph-mon[31360]: pgmap v16707: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:32 smithi022 ceph-mon[40250]: pgmap v16707: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:34.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:34 smithi122 ceph-mon[32255]: pgmap v16708: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:34 smithi022 ceph-mon[31360]: pgmap v16708: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:34 smithi022 ceph-mon[40250]: pgmap v16708: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:34.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:34.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:34.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:15:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:15:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:15:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:15:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3591478423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:15:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:15:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:15:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:15:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3591478423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:15:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:15:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:15:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:15:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3591478423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:36 smithi122 ceph-mon[32255]: pgmap v16709: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:36 smithi022 ceph-mon[31360]: pgmap v16709: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:36 smithi022 ceph-mon[40250]: pgmap v16709: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:15:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:38 smithi122 ceph-mon[32255]: pgmap v16710: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:38 smithi022 ceph-mon[31360]: pgmap v16710: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:38 smithi022 ceph-mon[40250]: pgmap v16710: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:39.583 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:39.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:39.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:39.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:40 smithi022 ceph-mon[31360]: pgmap v16711: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/394516287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:40 smithi022 ceph-mon[40250]: pgmap v16711: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/394516287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:15:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:40 smithi122 ceph-mon[32255]: pgmap v16711: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/394516287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:41.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:15:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:15:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:15:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:15:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:42 smithi122 ceph-mon[32255]: pgmap v16712: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:42 smithi022 ceph-mon[31360]: pgmap v16712: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:42 smithi022 ceph-mon[40250]: pgmap v16712: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:44 smithi122 ceph-mon[32255]: pgmap v16713: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:44 smithi022 ceph-mon[31360]: pgmap v16713: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:44 smithi022 ceph-mon[40250]: pgmap v16713: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:44.955 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:45.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:45.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:45.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:46 smithi122 ceph-mon[32255]: pgmap v16714: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1024365637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:46 smithi022 ceph-mon[31360]: pgmap v16714: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1024365637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:46 smithi022 ceph-mon[40250]: pgmap v16714: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1024365637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:15:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:48 smithi122 ceph-mon[32255]: pgmap v16715: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:48 smithi022 ceph-mon[31360]: pgmap v16715: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:48 smithi022 ceph-mon[40250]: pgmap v16715: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:50.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:50 smithi122 ceph-mon[32255]: pgmap v16716: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:50] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:15:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:50 smithi022 ceph-mon[31360]: pgmap v16716: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:50 smithi022 ceph-mon[40250]: pgmap v16716: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:50.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:50.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:50.691 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:15:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:15:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/324698648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/324698648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/324698648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:52 smithi122 ceph-mon[32255]: pgmap v16717: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:15:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:15:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:52 smithi022 ceph-mon[31360]: pgmap v16717: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:52 smithi022 ceph-mon[40250]: pgmap v16717: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:15:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:54 smithi122 ceph-mon[32255]: pgmap v16718: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:54 smithi022 ceph-mon[31360]: pgmap v16718: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:54 smithi022 ceph-mon[40250]: pgmap v16718: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:55.693 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:15:56.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:15:56.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:15:56.061 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:15:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:56 smithi122 ceph-mon[32255]: pgmap v16719: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1878099037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:56 smithi022 ceph-mon[31360]: pgmap v16719: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1878099037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:56 smithi022 ceph-mon[40250]: pgmap v16719: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:15:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1878099037' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:15:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:15:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:15:58 smithi122 ceph-mon[32255]: pgmap v16720: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:15:58 smithi022 ceph-mon[31360]: pgmap v16720: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:15:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:15:58 smithi022 ceph-mon[40250]: pgmap v16720: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:00] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:16:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:00 smithi122 ceph-mon[32255]: pgmap v16721: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:00 smithi022 ceph-mon[31360]: pgmap v16721: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:00 smithi022 ceph-mon[40250]: pgmap v16721: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:01.062 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:01.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:01.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:01.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:01.528 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:16:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:16:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:16:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:16:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:02 smithi122 ceph-mon[32255]: pgmap v16722: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1641219440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:02 smithi022 ceph-mon[31360]: pgmap v16722: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1641219440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:02 smithi022 ceph-mon[40250]: pgmap v16722: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1641219440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:04 smithi122 ceph-mon[32255]: pgmap v16723: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:04 smithi022 ceph-mon[31360]: pgmap v16723: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:04 smithi022 ceph-mon[40250]: pgmap v16723: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:06.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:06.432 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:06 smithi022 ceph-mon[31360]: pgmap v16724: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:06.433 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:06 smithi022 ceph-mon[40250]: pgmap v16724: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:06 smithi122 ceph-mon[32255]: pgmap v16724: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:06.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:06.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:06.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/287339163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/287339163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:16:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/287339163' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:08 smithi122 ceph-mon[32255]: pgmap v16725: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:08 smithi022 ceph-mon[31360]: pgmap v16725: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:08 smithi022 ceph-mon[40250]: pgmap v16725: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:10 smithi122 ceph-mon[32255]: pgmap v16726: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:10] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:16:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:10 smithi022 ceph-mon[31360]: pgmap v16726: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:10 smithi022 ceph-mon[40250]: pgmap v16726: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:16:10 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:16:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:11 smithi122 ceph-mon[32255]: pgmap v16727: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:11 smithi022 ceph-mon[31360]: pgmap v16727: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:11 smithi022 ceph-mon[40250]: pgmap v16727: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:11.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:12.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:12.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:12.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2859856706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:16:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:16:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2859856706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2859856706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:13 smithi022 ceph-mon[31360]: pgmap v16728: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:14.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:13 smithi022 ceph-mon[40250]: pgmap v16728: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:13 smithi122 ceph-mon[32255]: pgmap v16728: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:15 smithi022 ceph-mon[31360]: pgmap v16729: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:15 smithi022 ceph-mon[40250]: pgmap v16729: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:15 smithi122 ceph-mon[32255]: pgmap v16729: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:17.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:16:17.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:17.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:17.539 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:17 smithi022 ceph-mon[31360]: pgmap v16730: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:18.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1702088723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:18.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:17 smithi022 ceph-mon[40250]: pgmap v16730: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:18.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1702088723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:17 smithi122 ceph-mon[32255]: pgmap v16730: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1702088723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:19 smithi122 ceph-mon[32255]: pgmap v16731: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:16:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:16:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:16:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:16:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[31360]: pgmap v16731: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:16:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:16:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:16:20.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:16:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[40250]: pgmap v16731: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:20.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:16:20.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:16:20.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:16:20.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:16:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:20] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:16:21.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:16:20 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:16:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:21 smithi022 ceph-mon[31360]: pgmap v16732: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:21 smithi022 ceph-mon[40250]: pgmap v16732: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:21 smithi122 ceph-mon[32255]: pgmap v16732: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:22.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:16:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:16:22.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:22.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:22.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:23 smithi022 ceph-mon[31360]: pgmap v16733: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3094628024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:23 smithi022 ceph-mon[40250]: pgmap v16733: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3094628024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:23 smithi122 ceph-mon[32255]: pgmap v16733: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:24.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3094628024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:25 smithi022 ceph-mon[31360]: pgmap v16734: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:26.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:25 smithi022 ceph-mon[40250]: pgmap v16734: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:25 smithi122 ceph-mon[32255]: pgmap v16734: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:27.753 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:16:27.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:28.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:27 smithi022 ceph-mon[31360]: pgmap v16735: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:27 smithi022 ceph-mon[40250]: pgmap v16735: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:27 smithi122 ceph-mon[32255]: pgmap v16735: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:28.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:28.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:28.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/766178672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/766178672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/766178672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:29 smithi022 ceph-mon[31360]: pgmap v16736: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:29 smithi022 ceph-mon[40250]: pgmap v16736: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:29 smithi122 ceph-mon[32255]: pgmap v16736: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:30] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:16:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:16:30 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:16:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:31 smithi022 ceph-mon[31360]: pgmap v16737: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:31 smithi022 ceph-mon[40250]: pgmap v16737: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:31 smithi122 ceph-mon[32255]: pgmap v16737: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:16:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:16:33.285 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:33.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:33.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:33 smithi022 ceph-mon[31360]: pgmap v16738: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:34.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2685799326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:33 smithi022 ceph-mon[40250]: pgmap v16738: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2685799326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:33 smithi122 ceph-mon[32255]: pgmap v16738: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2685799326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:16:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:16:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:16:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:16:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:16:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:16:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:16:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:16:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:16:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:16:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:16:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:16:36.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:35 smithi022 ceph-mon[40250]: pgmap v16739: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:35 smithi022 ceph-mon[31360]: pgmap v16739: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:35 smithi122 ceph-mon[32255]: pgmap v16739: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:37.775 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:16:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:37 smithi022 ceph-mon[31360]: pgmap v16740: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:37 smithi022 ceph-mon[40250]: pgmap v16740: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:37 smithi122 ceph-mon[32255]: pgmap v16740: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:38.658 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:39.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:39.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:39.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:39 smithi022 ceph-mon[31360]: pgmap v16741: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:40.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1708900806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:39 smithi022 ceph-mon[40250]: pgmap v16741: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1708900806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:39 smithi122 ceph-mon[32255]: pgmap v16741: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1708900806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:16:41.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:16:40 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:16:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:41 smithi122 ceph-mon[32255]: pgmap v16742: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:41 smithi022 ceph-mon[31360]: pgmap v16742: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:41 smithi022 ceph-mon[40250]: pgmap v16742: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:42.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:16:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:16:44.031 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:43 smithi122 ceph-mon[32255]: pgmap v16743: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:43 smithi022 ceph-mon[31360]: pgmap v16743: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:43 smithi022 ceph-mon[40250]: pgmap v16743: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:44.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:44.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:44.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:45.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3059093313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:45.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3059093313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:45.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3059093313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:45 smithi122 ceph-mon[32255]: pgmap v16744: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:45 smithi022 ceph-mon[31360]: pgmap v16744: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:45 smithi022 ceph-mon[40250]: pgmap v16744: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:16:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:47 smithi122 ceph-mon[32255]: pgmap v16745: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:47 smithi022 ceph-mon[31360]: pgmap v16745: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:47 smithi022 ceph-mon[40250]: pgmap v16745: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:49.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:49.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:49.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:49.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:49 smithi022 ceph-mon[31360]: pgmap v16746: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:50.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1507257139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:49 smithi022 ceph-mon[40250]: pgmap v16746: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1507257139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:49 smithi122 ceph-mon[32255]: pgmap v16746: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1507257139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:50] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:16:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:16:50 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:16:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:51 smithi122 ceph-mon[32255]: pgmap v16747: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:51 smithi022 ceph-mon[31360]: pgmap v16747: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:51 smithi022 ceph-mon[40250]: pgmap v16747: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:16:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:16:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:16:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:53 smithi122 ceph-mon[32255]: pgmap v16748: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:53 smithi022 ceph-mon[31360]: pgmap v16748: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:53 smithi022 ceph-mon[40250]: pgmap v16748: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:54.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:16:55.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:16:55.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:16:55.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:16:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:55 smithi122 ceph-mon[32255]: pgmap v16749: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2285947249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:55 smithi022 ceph-mon[31360]: pgmap v16749: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:56.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2285947249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:55 smithi022 ceph-mon[40250]: pgmap v16749: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:16:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2285947249' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:16:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:16:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:57 smithi122 ceph-mon[32255]: pgmap v16750: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:57 smithi022 ceph-mon[31360]: pgmap v16750: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:16:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:57 smithi022 ceph-mon[40250]: pgmap v16750: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:00.147 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:00.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:16:59 smithi022 ceph-mon[31360]: pgmap v16751: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:00.148 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:16:59 smithi022 ceph-mon[40250]: pgmap v16751: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:16:59 smithi122 ceph-mon[32255]: pgmap v16751: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:00] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:17:00.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:00.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:00.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:01.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:17:00 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:17:01.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2737780944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2737780944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2737780944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:01 smithi122 ceph-mon[32255]: pgmap v16752: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:01 smithi022 ceph-mon[31360]: pgmap v16752: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:01 smithi022 ceph-mon[40250]: pgmap v16752: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:17:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:17:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:03 smithi122 ceph-mon[32255]: pgmap v16753: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:03 smithi022 ceph-mon[31360]: pgmap v16753: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:03 smithi022 ceph-mon[40250]: pgmap v16753: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:05.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:05.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:05.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:05.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:05 smithi122 ceph-mon[32255]: pgmap v16754: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:05 smithi022 ceph-mon[31360]: pgmap v16754: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:05 smithi022 ceph-mon[40250]: pgmap v16754: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2116802498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2116802498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:07.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2116802498' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:17:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:07 smithi122 ceph-mon[32255]: pgmap v16755: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:07 smithi022 ceph-mon[31360]: pgmap v16755: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:07 smithi022 ceph-mon[40250]: pgmap v16755: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:09 smithi122 ceph-mon[32255]: pgmap v16756: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:09 smithi022 ceph-mon[31360]: pgmap v16756: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:09 smithi022 ceph-mon[40250]: pgmap v16756: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:10] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:17:10.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:11.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:11.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:11.264 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:17:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:17:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:11 smithi122 ceph-mon[32255]: pgmap v16757: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2200773965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:11 smithi022 ceph-mon[31360]: pgmap v16757: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2200773965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:11 smithi022 ceph-mon[40250]: pgmap v16757: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2200773965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:17:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:17:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:13 smithi122 ceph-mon[32255]: pgmap v16758: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:13 smithi022 ceph-mon[31360]: pgmap v16758: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:13 smithi022 ceph-mon[40250]: pgmap v16758: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:15 smithi122 ceph-mon[32255]: pgmap v16759: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:16.266 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:16.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:15 smithi022 ceph-mon[31360]: pgmap v16759: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:16.267 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:15 smithi022 ceph-mon[40250]: pgmap v16759: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:16.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:16.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:16.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3870172941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3870172941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:17.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3870172941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:17:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:17 smithi122 ceph-mon[32255]: pgmap v16760: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:17 smithi022 ceph-mon[31360]: pgmap v16760: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:17 smithi022 ceph-mon[40250]: pgmap v16760: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:19 smithi122 ceph-mon[32255]: pgmap v16761: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:17:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:17:20.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:17:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[31360]: pgmap v16761: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:17:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:17:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:17:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[40250]: pgmap v16761: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:20.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:17:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:17:20.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:17:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:20] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:17:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:17:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:17:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:17:21.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:17:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:17:21.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:22.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:22.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:22.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:22 smithi022 ceph-mon[31360]: pgmap v16762: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3327108231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:22 smithi022 ceph-mon[40250]: pgmap v16762: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3327108231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:22 smithi122 ceph-mon[32255]: pgmap v16762: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3327108231' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:17:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:17:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:24 smithi022 ceph-mon[31360]: pgmap v16763: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:24 smithi022 ceph-mon[40250]: pgmap v16763: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:24 smithi122 ceph-mon[32255]: pgmap v16763: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:26 smithi022 ceph-mon[31360]: pgmap v16764: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:26 smithi022 ceph-mon[40250]: pgmap v16764: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:26 smithi122 ceph-mon[32255]: pgmap v16764: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:27.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:27.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:27.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:17:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:28 smithi022 ceph-mon[31360]: pgmap v16765: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1093571554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:28 smithi022 ceph-mon[40250]: pgmap v16765: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1093571554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:28 smithi122 ceph-mon[32255]: pgmap v16765: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1093571554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:30 smithi022 ceph-mon[31360]: pgmap v16766: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:30 smithi022 ceph-mon[40250]: pgmap v16766: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:30] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:17:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:30 smithi122 ceph-mon[32255]: pgmap v16766: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:17:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:17:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:32 smithi022 ceph-mon[31360]: pgmap v16767: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:32 smithi022 ceph-mon[40250]: pgmap v16767: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:32 smithi122 ceph-mon[32255]: pgmap v16767: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:32.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:17:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:17:32.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:32.746 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:32.747 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3552358439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3552358439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3552358439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:34 smithi122 ceph-mon[32255]: pgmap v16768: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:34 smithi022 ceph-mon[31360]: pgmap v16768: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:34 smithi022 ceph-mon[40250]: pgmap v16768: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:17:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:17:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:17:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:17:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:17:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:17:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:17:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:17:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:17:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:17:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:17:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:17:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:36 smithi122 ceph-mon[32255]: pgmap v16769: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:36 smithi022 ceph-mon[31360]: pgmap v16769: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:36 smithi022 ceph-mon[40250]: pgmap v16769: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:37.748 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:37.749 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:17:38.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:38.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:38.116 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:38 smithi122 ceph-mon[32255]: pgmap v16770: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:38 smithi022 ceph-mon[31360]: pgmap v16770: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:38 smithi022 ceph-mon[40250]: pgmap v16770: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2949481680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2949481680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2949481680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:17:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:40 smithi122 ceph-mon[32255]: pgmap v16771: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:40 smithi022 ceph-mon[31360]: pgmap v16771: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:40 smithi022 ceph-mon[40250]: pgmap v16771: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:41.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:17:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:17:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:42 smithi122 ceph-mon[32255]: pgmap v16772: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:42 smithi022 ceph-mon[31360]: pgmap v16772: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:42 smithi022 ceph-mon[40250]: pgmap v16772: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:17:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:17:43.118 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:43.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:43.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:43.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:44 smithi122 ceph-mon[32255]: pgmap v16773: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2492285075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:44 smithi022 ceph-mon[31360]: pgmap v16773: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2492285075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:44 smithi022 ceph-mon[40250]: pgmap v16773: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2492285075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:46 smithi122 ceph-mon[32255]: pgmap v16774: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:46 smithi022 ceph-mon[31360]: pgmap v16774: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:46 smithi022 ceph-mon[40250]: pgmap v16774: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:17:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:48 smithi122 ceph-mon[32255]: pgmap v16775: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:48.496 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:48 smithi022 ceph-mon[31360]: pgmap v16775: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:48.496 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:48 smithi022 ceph-mon[40250]: pgmap v16775: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:48.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:48.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:48.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/733894080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/733894080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/733894080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:50 smithi122 ceph-mon[32255]: pgmap v16776: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:50] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T03:17:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:50 smithi022 ceph-mon[31360]: pgmap v16776: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:50 smithi022 ceph-mon[40250]: pgmap v16776: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:17:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:17:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:17:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:52 smithi122 ceph-mon[32255]: pgmap v16777: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:17:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:17:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:52 smithi022 ceph-mon[31360]: pgmap v16777: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:52 smithi022 ceph-mon[40250]: pgmap v16777: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:17:53.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:54.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:54.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:54.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:17:54.235 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:54 smithi022 ceph-mon[31360]: pgmap v16778: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:54.236 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:54 smithi022 ceph-mon[40250]: pgmap v16778: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:54 smithi122 ceph-mon[32255]: pgmap v16778: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:17:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3702050570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3702050570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3702050570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:17:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:56 smithi122 ceph-mon[32255]: pgmap v16779: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T03:17:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:56 smithi022 ceph-mon[31360]: pgmap v16779: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T03:17:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:56 smithi022 ceph-mon[40250]: pgmap v16779: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T03:17:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:17:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:17:58 smithi122 ceph-mon[32255]: pgmap v16780: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T03:17:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:17:58 smithi022 ceph-mon[31360]: pgmap v16780: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T03:17:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:17:58 smithi022 ceph-mon[40250]: pgmap v16780: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T03:17:59.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:17:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:17:59.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:17:59.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:00] "GET /metrics HTTP/1.1" 200 202453 "" "Prometheus/2.18.1" 2021-12-16T03:18:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:00 smithi122 ceph-mon[32255]: pgmap v16781: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T03:18:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3480619006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:00 smithi022 ceph-mon[31360]: pgmap v16781: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T03:18:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3480619006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:00 smithi022 ceph-mon[40250]: pgmap v16781: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T03:18:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3480619006' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:01.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:18:01.685 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[33927]: 2021-12-16T03:18:01.304+0000 7f1127243700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.685 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[33927]: 2021-12-16T03:18:01.397+0000 7f1127243700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[63782]: 2021-12-16T03:18:01.304+0000 7f7f3047b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[63782]: 2021-12-16T03:18:01.397+0000 7f7f3047b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.686 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[66158]: 2021-12-16T03:18:01.304+0000 7f2279993700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.687 INFO:journalctl@ceph.osd.5.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[66158]: 2021-12-16T03:18:01.397+0000 7f2279993700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.687 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[61490]: 2021-12-16T03:18:01.304+0000 7fc767d4a700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.687 INFO:journalctl@ceph.osd.6.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[61490]: 2021-12-16T03:18:01.397+0000 7fc767d4a700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.688 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[58825]: 2021-12-16T03:18:01.304+0000 7fe414dee700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.688 INFO:journalctl@ceph.osd.7.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[58825]: 2021-12-16T03:18:01.397+0000 7fe414dee700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[32203]: 2021-12-16T03:18:01.304+0000 7ff247768700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[32203]: 2021-12-16T03:18:01.304+0000 7ff247768700 -1 mon.b@2(peon) e3 *** Got Signal Hangup *** 2021-12-16T03:18:01.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[32203]: 2021-12-16T03:18:01.397+0000 7ff247768700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:18:01.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:01 smithi122 conmon[32203]: 2021-12-16T03:18:01.397+0000 7ff247768700 -1 mon.b@2(peon) e3 *** Got Signal Hangup *** 2021-12-16T03:18:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:02 smithi122 ceph-mon[32255]: pgmap v16782: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T03:18:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:18:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:18:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:02 smithi022 ceph-mon[31360]: pgmap v16782: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T03:18:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:02 smithi022 ceph-mon[40250]: pgmap v16782: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T03:18:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:04 smithi122 ceph-mon[32255]: pgmap v16783: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T03:18:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:04 smithi022 ceph-mon[31360]: pgmap v16783: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T03:18:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:04 smithi022 ceph-mon[40250]: pgmap v16783: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T03:18:04.611 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:04.981 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:04.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:04.982 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1981724546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1981724546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1981724546' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:06 smithi122 ceph-mon[32255]: pgmap v16784: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T03:18:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:06 smithi022 ceph-mon[31360]: pgmap v16784: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T03:18:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:06 smithi022 ceph-mon[40250]: pgmap v16784: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T03:18:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:18:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:08 smithi122 ceph-mon[32255]: pgmap v16785: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:18:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:08 smithi022 ceph-mon[31360]: pgmap v16785: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:18:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:08 smithi022 ceph-mon[40250]: pgmap v16785: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:18:09.984 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:10 smithi022 ceph-mon[31360]: pgmap v16786: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:18:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:10 smithi022 ceph-mon[40250]: pgmap v16786: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:18:10.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:10.353 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:10.354 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:10 smithi122 ceph-mon[32255]: pgmap v16786: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:18:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:10] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:18:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:18:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:18:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2247819786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2247819786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2247819786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:12 smithi122 ceph-mon[32255]: pgmap v16787: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:18:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:18:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:12 smithi022 ceph-mon[31360]: pgmap v16787: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:12 smithi022 ceph-mon[40250]: pgmap v16787: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:14 smithi122 ceph-mon[32255]: pgmap v16788: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:14 smithi022 ceph-mon[31360]: pgmap v16788: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:14 smithi022 ceph-mon[40250]: pgmap v16788: 132 pgs: 132 active+clean; 582 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:15.352 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:15.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:15.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:15.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:16 smithi122 ceph-mon[32255]: pgmap v16789: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1453390095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:16 smithi022 ceph-mon[31360]: pgmap v16789: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1453390095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:16 smithi022 ceph-mon[40250]: pgmap v16789: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1453390095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:18:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:18 smithi122 ceph-mon[32255]: pgmap v16790: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:18 smithi022 ceph-mon[31360]: pgmap v16790: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:18 smithi022 ceph-mon[40250]: pgmap v16790: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:20] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:18:20.544 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:20 smithi122 ceph-mon[32255]: pgmap v16791: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:20.545 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:18:20.545 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:18:20.545 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:18:20.546 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[31360]: pgmap v16791: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:20.546 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:18:20.546 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:18:20.547 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:18:20.547 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[40250]: pgmap v16791: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:20.547 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:18:20.548 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:18:20.548 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:18:20.722 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:21.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:21.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:21.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1965730607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:21.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:18:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:18:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1965730607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1965730607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:18:22.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:18:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:22 smithi122 ceph-mon[32255]: pgmap v16792: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:22 smithi022 ceph-mon[40250]: pgmap v16792: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:22 smithi022 ceph-mon[31360]: pgmap v16792: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: pgmap v16793: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:24 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: pgmap v16793: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: pgmap v16793: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:18:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:24.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:24 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2878M 2021-12-16T03:18:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2878M 2021-12-16T03:18:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2878M 2021-12-16T03:18:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:25.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:26.095 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:26 smithi122 ceph-mon[32255]: pgmap v16794: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:26 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi122 to 3269M 2021-12-16T03:18:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:26.462 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[40250]: pgmap v16794: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:26.463 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:26.463 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:26.463 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi122 to 3269M 2021-12-16T03:18:26.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:26.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:26.464 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[31360]: pgmap v16794: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:18:26.464 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:26.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:18:26.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi122 to 3269M 2021-12-16T03:18:26.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:26.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:18:26.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:26.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:26.466 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3634619604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3634619604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3634619604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:18:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:28 smithi122 ceph-mon[32255]: pgmap v16795: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:28 smithi022 ceph-mon[31360]: pgmap v16795: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:28 smithi022 ceph-mon[40250]: pgmap v16795: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:30 smithi122 ceph-mon[32255]: pgmap v16796: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:30] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:18:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:30 smithi022 ceph-mon[31360]: pgmap v16796: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:30 smithi022 ceph-mon[40250]: pgmap v16796: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:18:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:18:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:31.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:31.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:31.833 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:18:32.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:18:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:32 smithi122 ceph-mon[32255]: pgmap v16797: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/399627424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:32 smithi022 ceph-mon[31360]: pgmap v16797: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/399627424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:32 smithi022 ceph-mon[40250]: pgmap v16797: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/399627424' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:34 smithi122 ceph-mon[32255]: pgmap v16798: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:34 smithi022 ceph-mon[31360]: pgmap v16798: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:34 smithi022 ceph-mon[40250]: pgmap v16798: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:18:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:18:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:18:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:18:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:18:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:18:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:18:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:18:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:18:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:18:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:18:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:18:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:36 smithi122 ceph-mon[32255]: pgmap v16799: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:36 smithi022 ceph-mon[31360]: pgmap v16799: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:36 smithi022 ceph-mon[40250]: pgmap v16799: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:36.834 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:37.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:37.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:37.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:18:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:38 smithi122 ceph-mon[32255]: pgmap v16800: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/932449278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:38 smithi022 ceph-mon[31360]: pgmap v16800: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/932449278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:38 smithi022 ceph-mon[40250]: pgmap v16800: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/932449278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:40] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:18:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:40 smithi122 ceph-mon[32255]: pgmap v16801: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:40 smithi022 ceph-mon[31360]: pgmap v16801: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:40 smithi022 ceph-mon[40250]: pgmap v16801: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:41.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:18:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:18:42.207 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:42 smithi122 ceph-mon[32255]: pgmap v16802: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:18:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:18:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:42 smithi022 ceph-mon[31360]: pgmap v16802: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:42 smithi022 ceph-mon[40250]: pgmap v16802: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:42.576 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:42.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:42.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3447059016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3447059016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3447059016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:44 smithi122 ceph-mon[32255]: pgmap v16803: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:44 smithi022 ceph-mon[31360]: pgmap v16803: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:44 smithi022 ceph-mon[40250]: pgmap v16803: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:46 smithi122 ceph-mon[32255]: pgmap v16804: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:46 smithi022 ceph-mon[31360]: pgmap v16804: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:46 smithi022 ceph-mon[40250]: pgmap v16804: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:47.578 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:18:47.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:47.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:47.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:48 smithi022 ceph-mon[31360]: pgmap v16805: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/792638263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:48 smithi022 ceph-mon[40250]: pgmap v16805: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/792638263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:48 smithi122 ceph-mon[32255]: pgmap v16805: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/792638263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:50 smithi122 ceph-mon[32255]: pgmap v16806: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:50] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:18:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:50 smithi022 ceph-mon[31360]: pgmap v16806: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:50 smithi022 ceph-mon[40250]: pgmap v16806: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:18:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:18:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:52 smithi122 ceph-mon[32255]: pgmap v16807: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:18:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:18:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:52 smithi022 ceph-mon[31360]: pgmap v16807: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:52 smithi022 ceph-mon[40250]: pgmap v16807: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:18:52.951 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:54 smithi122 ceph-mon[32255]: pgmap v16808: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1016678463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:54 smithi022 ceph-mon[31360]: pgmap v16808: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1016678463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:54 smithi022 ceph-mon[40250]: pgmap v16808: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1016678463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:56 smithi122 ceph-mon[32255]: pgmap v16809: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:56 smithi022 ceph-mon[31360]: pgmap v16809: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:56 smithi022 ceph-mon[40250]: pgmap v16809: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:18:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:57 smithi022 ceph-mon[31360]: pgmap v16810: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:18:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:57 smithi022 ceph-mon[40250]: pgmap v16810: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:57 smithi122 ceph-mon[32255]: pgmap v16810: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:18:58.324 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:18:58.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:18:58.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:18:58.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:18:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3302508869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3302508869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:18:59.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3302508869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:18:59 smithi022 ceph-mon[31360]: pgmap v16811: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:18:59 smithi022 ceph-mon[40250]: pgmap v16811: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:18:59 smithi122 ceph-mon[32255]: pgmap v16811: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:00] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:19:01.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:19:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:19:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:01 smithi022 ceph-mon[31360]: pgmap v16812: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:01 smithi022 ceph-mon[40250]: pgmap v16812: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:01 smithi122 ceph-mon[32255]: pgmap v16812: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:19:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:19:03.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:03 smithi022 ceph-mon[31360]: pgmap v16813: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:03 smithi022 ceph-mon[40250]: pgmap v16813: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:04.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:04.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:04.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:03 smithi122 ceph-mon[32255]: pgmap v16813: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:05.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/583717755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/583717755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/583717755' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:05 smithi122 ceph-mon[32255]: pgmap v16814: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:05 smithi022 ceph-mon[31360]: pgmap v16814: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:05 smithi022 ceph-mon[40250]: pgmap v16814: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:19:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:07 smithi122 ceph-mon[32255]: pgmap v16815: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:07 smithi022 ceph-mon[31360]: pgmap v16815: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:07 smithi022 ceph-mon[40250]: pgmap v16815: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:09.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:09 smithi122 ceph-mon[32255]: pgmap v16816: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/716927103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:09 smithi022 ceph-mon[31360]: pgmap v16816: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/716927103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:09 smithi022 ceph-mon[40250]: pgmap v16816: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/716927103' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:10] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:19:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:19:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:19:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:11 smithi122 ceph-mon[32255]: pgmap v16817: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:11 smithi022 ceph-mon[31360]: pgmap v16817: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:11 smithi022 ceph-mon[40250]: pgmap v16817: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:19:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:19:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:13 smithi122 ceph-mon[32255]: pgmap v16818: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:13 smithi022 ceph-mon[31360]: pgmap v16818: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:13 smithi022 ceph-mon[40250]: pgmap v16818: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:14.425 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:14.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:14.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:14.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:15.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1411437090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1411437090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1411437090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:15 smithi122 ceph-mon[32255]: pgmap v16819: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:15 smithi022 ceph-mon[31360]: pgmap v16819: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:15 smithi022 ceph-mon[40250]: pgmap v16819: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:19:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:17 smithi122 ceph-mon[32255]: pgmap v16820: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:17 smithi022 ceph-mon[31360]: pgmap v16820: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:17 smithi022 ceph-mon[40250]: pgmap v16820: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:19.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:20.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:20.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:20.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:20.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:19 smithi022 ceph-mon[31360]: pgmap v16821: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:20.164 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:19 smithi022 ceph-mon[40250]: pgmap v16821: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:20.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:19 smithi122 ceph-mon[32255]: pgmap v16821: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:20] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:19:21.113 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/63298177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/63298177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/63298177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:21.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:19:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:19:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:21 smithi122 ceph-mon[32255]: pgmap v16822: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:21 smithi022 ceph-mon[31360]: pgmap v16822: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:21 smithi022 ceph-mon[40250]: pgmap v16822: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:19:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:19:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:23 smithi122 ceph-mon[32255]: pgmap v16823: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:23 smithi022 ceph-mon[31360]: pgmap v16823: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:23 smithi022 ceph-mon[40250]: pgmap v16823: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:25.164 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:25.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:25.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:25.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:25 smithi122 ceph-mon[32255]: pgmap v16824: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:19:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:19:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:19:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4031618568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:25 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:19:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[31360]: pgmap v16824: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:19:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4031618568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:19:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[40250]: pgmap v16824: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:19:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:19:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:19:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4031618568' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:25 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:19:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:19:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:27 smithi122 ceph-mon[32255]: pgmap v16825: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:27 smithi022 ceph-mon[31360]: pgmap v16825: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:27 smithi022 ceph-mon[40250]: pgmap v16825: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:29 smithi122 ceph-mon[32255]: pgmap v16826: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:29 smithi022 ceph-mon[31360]: pgmap v16826: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:29 smithi022 ceph-mon[40250]: pgmap v16826: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:30] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:19:30.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:30.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:30.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:30.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:19:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:19:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:31 smithi122 ceph-mon[32255]: pgmap v16827: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1558109818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:31 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:31 smithi022 ceph-mon[31360]: pgmap v16827: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1558109818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:32.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:31 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:31 smithi022 ceph-mon[40250]: pgmap v16827: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1558109818' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:31 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:19:32.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:19:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:33 smithi122 ceph-mon[32255]: pgmap v16828: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:33 smithi022 ceph-mon[31360]: pgmap v16828: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:33 smithi022 ceph-mon[40250]: pgmap v16828: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:19:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:19:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:19:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:19:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:19:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:19:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:19:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:19:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:19:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:19:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:19:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:19:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:35 smithi122 ceph-mon[32255]: pgmap v16829: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:35 smithi022 ceph-mon[31360]: pgmap v16829: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:35 smithi022 ceph-mon[40250]: pgmap v16829: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:36.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:36.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:36.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:37.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2427150656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2427150656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2427150656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:19:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:37 smithi122 ceph-mon[32255]: pgmap v16830: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:37 smithi022 ceph-mon[31360]: pgmap v16830: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:37 smithi022 ceph-mon[40250]: pgmap v16830: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:39 smithi122 ceph-mon[32255]: pgmap v16831: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:39 smithi022 ceph-mon[31360]: pgmap v16831: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:39 smithi022 ceph-mon[40250]: pgmap v16831: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:40] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:19:41.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:41.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:19:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:19:41.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:41.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:41.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:41 smithi022 ceph-mon[31360]: pgmap v16832: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:41 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:42.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2302552021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:41 smithi022 ceph-mon[40250]: pgmap v16832: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:41 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:42.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2302552021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:41 smithi122 ceph-mon[32255]: pgmap v16832: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:41 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:42.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2302552021' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:19:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:19:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:43 smithi122 ceph-mon[32255]: pgmap v16833: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:43 smithi022 ceph-mon[31360]: pgmap v16833: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:43 smithi022 ceph-mon[40250]: pgmap v16833: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:45 smithi122 ceph-mon[32255]: pgmap v16834: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:45 smithi022 ceph-mon[31360]: pgmap v16834: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:45 smithi022 ceph-mon[40250]: pgmap v16834: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:46.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:47.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:47.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:47.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:19:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:47 smithi122 ceph-mon[32255]: pgmap v16835: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2463945959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:47 smithi022 ceph-mon[31360]: pgmap v16835: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2463945959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:47 smithi022 ceph-mon[40250]: pgmap v16835: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:48.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2463945959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:49 smithi122 ceph-mon[32255]: pgmap v16836: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:49 smithi022 ceph-mon[31360]: pgmap v16836: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:49 smithi022 ceph-mon[40250]: pgmap v16836: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:50] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:19:51.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:19:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:19:52.027 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:51 smithi122 ceph-mon[32255]: pgmap v16837: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:51 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:51 smithi022 ceph-mon[31360]: pgmap v16837: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:51 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:51 smithi022 ceph-mon[40250]: pgmap v16837: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:51 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:19:52.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:52.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:52.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:19:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:19:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/447280054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/447280054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/447280054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:53 smithi122 ceph-mon[32255]: pgmap v16838: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:53 smithi022 ceph-mon[31360]: pgmap v16838: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:53 smithi022 ceph-mon[40250]: pgmap v16838: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:55 smithi122 ceph-mon[32255]: pgmap v16839: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:55 smithi022 ceph-mon[31360]: pgmap v16839: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:55 smithi022 ceph-mon[40250]: pgmap v16839: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:19:57.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:19:57.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:19:57.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:19:57.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:19:57.768 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:19:58.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:57 smithi022 ceph-mon[31360]: pgmap v16840: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:58.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1389881436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:57 smithi022 ceph-mon[40250]: pgmap v16840: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:58.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1389881436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:19:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:57 smithi122 ceph-mon[32255]: pgmap v16840: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:19:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1389881436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:19:59 smithi122 ceph-mon[32255]: pgmap v16841: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:19:59 smithi022 ceph-mon[31360]: pgmap v16841: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:19:59 smithi022 ceph-mon[40250]: pgmap v16841: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:00] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:20:01.152 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T03:20:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T03:20:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T03:20:01.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:20:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:20:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:01 smithi122 ceph-mon[32255]: pgmap v16842: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:01 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:01 smithi022 ceph-mon[31360]: pgmap v16842: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:01 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:01 smithi022 ceph-mon[40250]: pgmap v16842: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:01 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:20:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:20:02.769 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:03.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:03.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:03.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:03 smithi122 ceph-mon[32255]: pgmap v16843: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:04.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1582247349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:04.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:03 smithi022 ceph-mon[40250]: pgmap v16843: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1582247349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:03 smithi022 ceph-mon[31360]: pgmap v16843: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1582247349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:05 smithi122 ceph-mon[32255]: pgmap v16844: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:06.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:05 smithi022 ceph-mon[40250]: pgmap v16844: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:05 smithi022 ceph-mon[31360]: pgmap v16844: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:20:08.144 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:08.144 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:07 smithi022 ceph-mon[31360]: pgmap v16845: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:08.145 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:07 smithi022 ceph-mon[40250]: pgmap v16845: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:07 smithi122 ceph-mon[32255]: pgmap v16845: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:08.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:08.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:08.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:09.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/459706977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/459706977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/459706977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:09 smithi122 ceph-mon[32255]: pgmap v16846: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:09 smithi022 ceph-mon[31360]: pgmap v16846: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:09 smithi022 ceph-mon[40250]: pgmap v16846: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:10] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:20:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:20:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:20:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:11 smithi122 ceph-mon[32255]: pgmap v16847: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:11 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:11 smithi022 ceph-mon[31360]: pgmap v16847: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:11 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:11 smithi022 ceph-mon[40250]: pgmap v16847: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:12.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:11 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:20:12.318Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:20:13.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:13.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:13 smithi122 ceph-mon[32255]: pgmap v16848: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2345593957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:13 smithi022 ceph-mon[31360]: pgmap v16848: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2345593957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:13 smithi022 ceph-mon[40250]: pgmap v16848: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2345593957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:15 smithi122 ceph-mon[32255]: pgmap v16849: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:15 smithi022 ceph-mon[31360]: pgmap v16849: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:15 smithi022 ceph-mon[40250]: pgmap v16849: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:20:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:17 smithi122 ceph-mon[32255]: pgmap v16850: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:17 smithi022 ceph-mon[31360]: pgmap v16850: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:17 smithi022 ceph-mon[40250]: pgmap v16850: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:18.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:19.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:19.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:19.253 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:19 smithi122 ceph-mon[32255]: pgmap v16851: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/450263787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:19 smithi022 ceph-mon[31360]: pgmap v16851: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/450263787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:19 smithi022 ceph-mon[40250]: pgmap v16851: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/450263787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:20] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:20:21.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:20:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:20:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:21 smithi122 ceph-mon[32255]: pgmap v16852: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:21 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:21 smithi022 ceph-mon[31360]: pgmap v16852: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:21 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:21 smithi022 ceph-mon[40250]: pgmap v16852: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:21 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:20:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:20:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:23 smithi122 ceph-mon[32255]: pgmap v16853: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:24.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:24.255 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:23 smithi022 ceph-mon[31360]: pgmap v16853: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:24.255 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:23 smithi022 ceph-mon[40250]: pgmap v16853: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:24.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:24.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:24.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:24.908 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4118058068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:24.909 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4118058068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:25.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4118058068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:25 smithi122 ceph-mon[32255]: pgmap v16854: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:20:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:20:26.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:25 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:20:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[31360]: pgmap v16854: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:20:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:20:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:20:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[40250]: pgmap v16854: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:20:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:20:26.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:25 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:20:27.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:20:27.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:20:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:20:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:20:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:28 smithi122 ceph-mon[32255]: pgmap v16855: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:28 smithi022 ceph-mon[31360]: pgmap v16855: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:28 smithi022 ceph-mon[40250]: pgmap v16855: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:29.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:29.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:29.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:29.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:30 smithi022 ceph-mon[31360]: pgmap v16856: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2816920510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:30 smithi022 ceph-mon[40250]: pgmap v16856: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2816920510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:30] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:20:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:30 smithi122 ceph-mon[32255]: pgmap v16856: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2816920510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:20:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:20:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:32 smithi122 ceph-mon[32255]: pgmap v16857: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:20:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:20:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:32 smithi022 ceph-mon[31360]: pgmap v16857: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:32 smithi022 ceph-mon[40250]: pgmap v16857: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:34 smithi122 ceph-mon[32255]: pgmap v16858: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:34 smithi022 ceph-mon[31360]: pgmap v16858: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:34 smithi022 ceph-mon[40250]: pgmap v16858: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:34.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:20:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:20:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:20:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:20:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:20:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:20:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:20:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:20:35.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:35.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:35.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:20:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:20:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:20:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:20:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:36 smithi122 ceph-mon[32255]: pgmap v16859: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2646903821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:36 smithi022 ceph-mon[31360]: pgmap v16859: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2646903821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:36 smithi022 ceph-mon[40250]: pgmap v16859: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2646903821' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:20:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:38 smithi122 ceph-mon[32255]: pgmap v16860: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:38 smithi022 ceph-mon[31360]: pgmap v16860: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:38 smithi022 ceph-mon[40250]: pgmap v16860: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:40.366 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:40.367 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:40 smithi022 ceph-mon[31360]: pgmap v16861: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:40.367 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:40 smithi022 ceph-mon[40250]: pgmap v16861: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:40] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:20:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:40 smithi122 ceph-mon[32255]: pgmap v16861: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:40.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:40.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:40.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4005287855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:41.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:20:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:20:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4005287855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4005287855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:20:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:20:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:42 smithi122 ceph-mon[32255]: pgmap v16862: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:42 smithi022 ceph-mon[31360]: pgmap v16862: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:42 smithi022 ceph-mon[40250]: pgmap v16862: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:44 smithi122 ceph-mon[32255]: pgmap v16863: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:44 smithi022 ceph-mon[31360]: pgmap v16863: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:44 smithi022 ceph-mon[40250]: pgmap v16863: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:45.735 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:46.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:46.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:46.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:46 smithi122 ceph-mon[32255]: pgmap v16864: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1912607462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:46 smithi022 ceph-mon[31360]: pgmap v16864: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1912607462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:46 smithi022 ceph-mon[40250]: pgmap v16864: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1912607462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:20:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:48 smithi122 ceph-mon[32255]: pgmap v16865: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:48 smithi022 ceph-mon[31360]: pgmap v16865: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:48 smithi022 ceph-mon[40250]: pgmap v16865: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:50] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:20:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:50 smithi122 ceph-mon[32255]: pgmap v16866: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:50 smithi022 ceph-mon[31360]: pgmap v16866: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:50 smithi022 ceph-mon[40250]: pgmap v16866: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:51.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:51.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:51.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:51.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:20:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:20:52.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:20:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:20:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:52 smithi122 ceph-mon[32255]: pgmap v16867: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1812290385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:52 smithi022 ceph-mon[31360]: pgmap v16867: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1812290385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:52 smithi022 ceph-mon[40250]: pgmap v16867: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1812290385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:20:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:54 smithi122 ceph-mon[32255]: pgmap v16868: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:54 smithi022 ceph-mon[31360]: pgmap v16868: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:54 smithi022 ceph-mon[40250]: pgmap v16868: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:20:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:56 smithi122 ceph-mon[32255]: pgmap v16869: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:56.476 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:20:56.476 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:56 smithi022 ceph-mon[31360]: pgmap v16869: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:56.477 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:56 smithi022 ceph-mon[40250]: pgmap v16869: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:20:56.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:20:56.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:20:56.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:20:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/548135212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/548135212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/548135212' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:20:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:20:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:20:58 smithi122 ceph-mon[32255]: pgmap v16870: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:20:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:20:58 smithi022 ceph-mon[31360]: pgmap v16870: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:20:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:20:58 smithi022 ceph-mon[40250]: pgmap v16870: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:21:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:00 smithi122 ceph-mon[32255]: pgmap v16871: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:00] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:21:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:00 smithi022 ceph-mon[31360]: pgmap v16871: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:00 smithi022 ceph-mon[40250]: pgmap v16871: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:21:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:21:01.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:02.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:02.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:02.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:02 smithi122 ceph-mon[32255]: pgmap v16872: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:21:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:21:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:21:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:02 smithi022 ceph-mon[31360]: pgmap v16872: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:21:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:02 smithi022 ceph-mon[40250]: pgmap v16872: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:21:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1918298882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1918298882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1918298882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:04 smithi122 ceph-mon[32255]: pgmap v16873: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:21:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:04 smithi022 ceph-mon[31360]: pgmap v16873: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:21:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:04 smithi022 ceph-mon[40250]: pgmap v16873: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:21:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:06 smithi122 ceph-mon[32255]: pgmap v16874: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:06 smithi022 ceph-mon[31360]: pgmap v16874: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:06 smithi022 ceph-mon[40250]: pgmap v16874: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:07.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:21:07.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:07.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:07.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:08 smithi122 ceph-mon[32255]: pgmap v16875: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/836826779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:08 smithi022 ceph-mon[31360]: pgmap v16875: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/836826779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:08 smithi022 ceph-mon[40250]: pgmap v16875: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/836826779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:10] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:21:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:10 smithi122 ceph-mon[32255]: pgmap v16876: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:10 smithi022 ceph-mon[31360]: pgmap v16876: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:10 smithi022 ceph-mon[40250]: pgmap v16876: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:21:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:21:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:12 smithi122 ceph-mon[32255]: pgmap v16877: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:21:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:21:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:12 smithi022 ceph-mon[31360]: pgmap v16877: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:12 smithi022 ceph-mon[40250]: pgmap v16877: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:12.584 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:12.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:12.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:12.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1645138786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1645138786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1645138786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:14 smithi122 ceph-mon[32255]: pgmap v16878: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:14 smithi022 ceph-mon[31360]: pgmap v16878: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:14 smithi022 ceph-mon[40250]: pgmap v16878: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:16 smithi022 ceph-mon[31360]: pgmap v16879: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:16 smithi022 ceph-mon[40250]: pgmap v16879: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:16 smithi122 ceph-mon[32255]: pgmap v16879: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:21:17.959 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:18 smithi022 ceph-mon[31360]: pgmap v16880: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:18 smithi022 ceph-mon[40250]: pgmap v16880: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:18.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:18.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:18.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:18 smithi122 ceph-mon[32255]: pgmap v16880: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3632990330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3632990330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3632990330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:20] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:21:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:20 smithi122 ceph-mon[32255]: pgmap v16881: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:20 smithi022 ceph-mon[31360]: pgmap v16881: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:20 smithi022 ceph-mon[40250]: pgmap v16881: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:21 smithi022 ceph-mon[31360]: pgmap v16882: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:21 smithi022 ceph-mon[40250]: pgmap v16882: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:21 smithi122 ceph-mon[32255]: pgmap v16882: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:21.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:21:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:21:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:21:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:21:23.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:23.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:23.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:23.695 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:23 smithi022 ceph-mon[31360]: pgmap v16883: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3171201238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:23 smithi022 ceph-mon[40250]: pgmap v16883: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3171201238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:23 smithi122 ceph-mon[32255]: pgmap v16883: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3171201238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:26.118 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:25 smithi122 ceph-mon[32255]: pgmap v16884: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:26.119 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:25 smithi022 ceph-mon[31360]: pgmap v16884: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:26.120 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:25 smithi022 ceph-mon[40250]: pgmap v16884: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:26.807 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:21:26.807 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:21:26.808 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:21:26.808 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:21:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:21:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:21:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:21:27.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:26 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:21:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:21:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:21:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:21:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:26 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:21:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:21:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:27 smithi122 ceph-mon[32255]: pgmap v16885: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:27 smithi022 ceph-mon[31360]: pgmap v16885: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:27 smithi022 ceph-mon[40250]: pgmap v16885: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:28.698 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:29.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:29.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:29.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:29 smithi122 ceph-mon[32255]: pgmap v16886: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:30.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1776579042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:29 smithi022 ceph-mon[31360]: pgmap v16886: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:30.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1776579042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:29 smithi022 ceph-mon[40250]: pgmap v16886: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1776579042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:30] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:21:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:21:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:21:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:31 smithi122 ceph-mon[32255]: pgmap v16887: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:31 smithi022 ceph-mon[31360]: pgmap v16887: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:31 smithi022 ceph-mon[40250]: pgmap v16887: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:21:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:21:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:33 smithi122 ceph-mon[32255]: pgmap v16888: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:33 smithi022 ceph-mon[40250]: pgmap v16888: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:33 smithi022 ceph-mon[31360]: pgmap v16888: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:34.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:34.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:34.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:21:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:21:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:21:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:21:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/187835569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:21:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:21:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:21:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:21:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/187835569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:21:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:21:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:21:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:21:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/187835569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:35 smithi122 ceph-mon[32255]: pgmap v16889: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:35 smithi022 ceph-mon[31360]: pgmap v16889: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:35 smithi022 ceph-mon[40250]: pgmap v16889: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:21:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:37 smithi122 ceph-mon[32255]: pgmap v16890: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:37 smithi022 ceph-mon[31360]: pgmap v16890: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:37 smithi022 ceph-mon[40250]: pgmap v16890: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:39.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:39.809 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:39.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:39.810 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:39 smithi122 ceph-mon[32255]: pgmap v16891: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3056999899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:39 smithi022 ceph-mon[31360]: pgmap v16891: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3056999899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:39 smithi022 ceph-mon[40250]: pgmap v16891: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3056999899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:40] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:21:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:21:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:21:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:41 smithi122 ceph-mon[32255]: pgmap v16892: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:41 smithi022 ceph-mon[31360]: pgmap v16892: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:41 smithi022 ceph-mon[40250]: pgmap v16892: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:21:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:21:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:43 smithi122 ceph-mon[32255]: pgmap v16893: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:43 smithi022 ceph-mon[31360]: pgmap v16893: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:43 smithi022 ceph-mon[40250]: pgmap v16893: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:44.811 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:45.179 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:45.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:45.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:45 smithi122 ceph-mon[32255]: pgmap v16894: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2766798951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:45 smithi022 ceph-mon[40250]: pgmap v16894: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:46.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2766798951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:45 smithi022 ceph-mon[31360]: pgmap v16894: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2766798951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:21:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:47 smithi122 ceph-mon[32255]: pgmap v16895: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:47 smithi022 ceph-mon[31360]: pgmap v16895: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:47 smithi022 ceph-mon[40250]: pgmap v16895: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:49 smithi122 ceph-mon[32255]: pgmap v16896: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:50.181 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:50.182 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:49 smithi022 ceph-mon[31360]: pgmap v16896: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:50.182 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:49 smithi022 ceph-mon[40250]: pgmap v16896: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:50] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:21:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:50.548 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2037975980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2037975980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2037975980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:21:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:21:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:51 smithi122 ceph-mon[32255]: pgmap v16897: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:51 smithi022 ceph-mon[31360]: pgmap v16897: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:51 smithi022 ceph-mon[40250]: pgmap v16897: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:21:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:21:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:21:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:53 smithi122 ceph-mon[32255]: pgmap v16898: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:53 smithi022 ceph-mon[31360]: pgmap v16898: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:53 smithi022 ceph-mon[40250]: pgmap v16898: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:55.549 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:21:55.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:21:55.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:21:55.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:21:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:55 smithi122 ceph-mon[32255]: pgmap v16899: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:55 smithi022 ceph-mon[31360]: pgmap v16899: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:55 smithi022 ceph-mon[40250]: pgmap v16899: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:21:57.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3816719429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3816719429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:57.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3816719429' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:21:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:21:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:57 smithi122 ceph-mon[32255]: pgmap v16900: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:57 smithi022 ceph-mon[31360]: pgmap v16900: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:21:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:57 smithi022 ceph-mon[40250]: pgmap v16900: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:00.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:21:59 smithi122 ceph-mon[32255]: pgmap v16901: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:21:59 smithi022 ceph-mon[31360]: pgmap v16901: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:21:59 smithi022 ceph-mon[40250]: pgmap v16901: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:00] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:22:00.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:01.294 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:01.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:01.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:22:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:22:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:01 smithi122 ceph-mon[32255]: pgmap v16902: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/75704175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:01 smithi022 ceph-mon[31360]: pgmap v16902: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/75704175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:01 smithi022 ceph-mon[40250]: pgmap v16902: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/75704175' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:22:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:22:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:04.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:03 smithi122 ceph-mon[32255]: pgmap v16903: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:03 smithi022 ceph-mon[31360]: pgmap v16903: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:03 smithi022 ceph-mon[40250]: pgmap v16903: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:06.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:05 smithi122 ceph-mon[32255]: pgmap v16904: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:05 smithi022 ceph-mon[31360]: pgmap v16904: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:05 smithi022 ceph-mon[40250]: pgmap v16904: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:06.296 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:06.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:06.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:06.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2054466868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2054466868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:07.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2054466868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:22:08.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:07 smithi122 ceph-mon[32255]: pgmap v16905: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:07 smithi022 ceph-mon[31360]: pgmap v16905: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:07 smithi022 ceph-mon[40250]: pgmap v16905: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:09 smithi122 ceph-mon[32255]: pgmap v16906: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:09 smithi022 ceph-mon[31360]: pgmap v16906: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:09 smithi022 ceph-mon[40250]: pgmap v16906: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:10] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:22:11.672 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:22:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:22:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:11 smithi022 ceph-mon[31360]: pgmap v16907: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:11 smithi022 ceph-mon[40250]: pgmap v16907: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:12.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:12.040 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:12.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:11 smithi122 ceph-mon[32255]: pgmap v16907: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:22:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:22:13.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:13.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3026265082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3026265082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3026265082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:13 smithi122 ceph-mon[32255]: pgmap v16908: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:13 smithi022 ceph-mon[31360]: pgmap v16908: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:13 smithi022 ceph-mon[40250]: pgmap v16908: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:15 smithi122 ceph-mon[32255]: pgmap v16909: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:15 smithi022 ceph-mon[31360]: pgmap v16909: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:15 smithi022 ceph-mon[40250]: pgmap v16909: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:17.043 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:17.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:17.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:17.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:22:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:17 smithi122 ceph-mon[32255]: pgmap v16910: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3908416170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:18.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:17 smithi022 ceph-mon[40250]: pgmap v16910: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3908416170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:17 smithi022 ceph-mon[31360]: pgmap v16910: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3908416170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:19 smithi122 ceph-mon[32255]: pgmap v16911: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:19 smithi022 ceph-mon[31360]: pgmap v16911: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:19 smithi022 ceph-mon[40250]: pgmap v16911: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:20] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:22:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:22:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:22:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:21 smithi122 ceph-mon[32255]: pgmap v16912: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:21 smithi022 ceph-mon[31360]: pgmap v16912: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:21 smithi022 ceph-mon[40250]: pgmap v16912: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:22.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:22:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:22:22.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:22.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:22.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3409551397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3409551397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3409551397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:23 smithi122 ceph-mon[32255]: pgmap v16913: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:23 smithi022 ceph-mon[31360]: pgmap v16913: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:23 smithi022 ceph-mon[40250]: pgmap v16913: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:25 smithi122 ceph-mon[32255]: pgmap v16914: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:25 smithi022 ceph-mon[31360]: pgmap v16914: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:25 smithi022 ceph-mon[40250]: pgmap v16914: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:26.928 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:22:26.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:22:26.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:26 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:22:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:22:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:22:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:26 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:22:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:22:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:22:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:26 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:22:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:22:27.781 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:28.149 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:28.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:28.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:28.150 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:28 smithi022 ceph-mon[40250]: pgmap v16915: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:28.151 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:22:28.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:28 smithi022 ceph-mon[31360]: pgmap v16915: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:28.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:22:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:28 smithi122 ceph-mon[32255]: pgmap v16915: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:22:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1674916054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1674916054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1674916054' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:30 smithi022 ceph-mon[31360]: pgmap v16916: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:30 smithi022 ceph-mon[40250]: pgmap v16916: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:30] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:22:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:30 smithi122 ceph-mon[32255]: pgmap v16916: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:22:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:22:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:32 smithi022 ceph-mon[31360]: pgmap v16917: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:32 smithi022 ceph-mon[40250]: pgmap v16917: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:32 smithi122 ceph-mon[32255]: pgmap v16917: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:22:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:22:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:33.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:33.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:33.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:33.522 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:33.522 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:34 smithi122 ceph-mon[32255]: pgmap v16918: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/377133870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:34 smithi022 ceph-mon[31360]: pgmap v16918: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/377133870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:34 smithi022 ceph-mon[40250]: pgmap v16918: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/377133870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:22:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:22:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:22:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:22:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:22:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:22:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:22:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:22:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:22:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:22:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:22:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:22:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:36 smithi122 ceph-mon[32255]: pgmap v16919: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:36 smithi022 ceph-mon[31360]: pgmap v16919: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:36 smithi022 ceph-mon[40250]: pgmap v16919: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:22:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:38 smithi122 ceph-mon[32255]: pgmap v16920: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:38.523 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:38.524 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:38 smithi022 ceph-mon[31360]: pgmap v16920: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:38.524 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:38 smithi022 ceph-mon[40250]: pgmap v16920: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:38.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:38.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:38.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3867915599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:39.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3867915599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3867915599' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:40 smithi122 ceph-mon[32255]: pgmap v16921: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:40] "GET /metrics HTTP/1.1" 200 202474 "" "Prometheus/2.18.1" 2021-12-16T03:22:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:40 smithi022 ceph-mon[31360]: pgmap v16921: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:40 smithi022 ceph-mon[40250]: pgmap v16921: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:22:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:22:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:42 smithi122 ceph-mon[32255]: pgmap v16922: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:42 smithi022 ceph-mon[31360]: pgmap v16922: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:42 smithi022 ceph-mon[40250]: pgmap v16922: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:22:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:22:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:43.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:44.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:44.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:44.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:44.258 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:44 smithi022 ceph-mon[31360]: pgmap v16923: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:44.259 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:44 smithi022 ceph-mon[40250]: pgmap v16923: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:44 smithi122 ceph-mon[32255]: pgmap v16923: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/602570837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/602570837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/602570837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:46 smithi122 ceph-mon[32255]: pgmap v16924: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:46 smithi022 ceph-mon[31360]: pgmap v16924: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:46 smithi022 ceph-mon[40250]: pgmap v16924: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:22:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:48 smithi122 ceph-mon[32255]: pgmap v16925: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:48 smithi022 ceph-mon[31360]: pgmap v16925: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:48 smithi022 ceph-mon[40250]: pgmap v16925: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:49.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:49.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:49.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:49.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:50] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:22:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:50 smithi122 ceph-mon[32255]: pgmap v16926: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1749489789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:50 smithi022 ceph-mon[31360]: pgmap v16926: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1749489789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:50 smithi022 ceph-mon[40250]: pgmap v16926: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1749489789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:22:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:22:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:52 smithi122 ceph-mon[32255]: pgmap v16927: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:22:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:22:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:52 smithi022 ceph-mon[31360]: pgmap v16927: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:52 smithi022 ceph-mon[40250]: pgmap v16927: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:22:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:54 smithi122 ceph-mon[32255]: pgmap v16928: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:54 smithi022 ceph-mon[31360]: pgmap v16928: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:54 smithi022 ceph-mon[40250]: pgmap v16928: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:54.629 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:22:54.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:22:54.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:22:54.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:22:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/910389945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/910389945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/910389945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:22:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:56 smithi122 ceph-mon[32255]: pgmap v16929: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:56 smithi022 ceph-mon[31360]: pgmap v16929: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:56 smithi022 ceph-mon[40250]: pgmap v16929: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:22:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:22:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:22:58 smithi122 ceph-mon[32255]: pgmap v16930: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:22:58 smithi022 ceph-mon[40250]: pgmap v16930: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:22:58 smithi022 ceph-mon[31360]: pgmap v16930: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:22:59.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:00 smithi022 ceph-mon[31360]: pgmap v16931: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:00 smithi022 ceph-mon[40250]: pgmap v16931: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:00.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:00.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:00.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:00] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:23:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:00 smithi122 ceph-mon[32255]: pgmap v16931: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:01.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:23:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:23:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1120949517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1120949517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1120949517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:02 smithi122 ceph-mon[32255]: pgmap v16932: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:23:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:23:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:02 smithi022 ceph-mon[31360]: pgmap v16932: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:02 smithi022 ceph-mon[40250]: pgmap v16932: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:04 smithi122 ceph-mon[32255]: pgmap v16933: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:04 smithi022 ceph-mon[31360]: pgmap v16933: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:04 smithi022 ceph-mon[40250]: pgmap v16933: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:05.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:05.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:05.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:05.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:06 smithi122 ceph-mon[32255]: pgmap v16934: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/939595941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:06 smithi022 ceph-mon[31360]: pgmap v16934: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/939595941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:06 smithi022 ceph-mon[40250]: pgmap v16934: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/939595941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:23:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:08 smithi122 ceph-mon[32255]: pgmap v16935: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:08 smithi022 ceph-mon[31360]: pgmap v16935: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:08 smithi022 ceph-mon[40250]: pgmap v16935: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:10] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:23:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:10 smithi122 ceph-mon[32255]: pgmap v16936: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:10 smithi022 ceph-mon[31360]: pgmap v16936: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:10 smithi022 ceph-mon[40250]: pgmap v16936: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:10.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:11.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:11.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:11.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/926367363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:11.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:23:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:23:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/926367363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/926367363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:12 smithi122 ceph-mon[32255]: pgmap v16937: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:23:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:23:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:12 smithi022 ceph-mon[31360]: pgmap v16937: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:12.630 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:12 smithi022 ceph-mon[40250]: pgmap v16937: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:14 smithi122 ceph-mon[32255]: pgmap v16938: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:14 smithi022 ceph-mon[31360]: pgmap v16938: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:14 smithi022 ceph-mon[40250]: pgmap v16938: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:16.114 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:16 smithi122 ceph-mon[32255]: pgmap v16939: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:16.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:16.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:16.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:16.478 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:16 smithi022 ceph-mon[31360]: pgmap v16939: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:16.479 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:16 smithi022 ceph-mon[40250]: pgmap v16939: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/437475397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/437475397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/437475397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:23:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:18 smithi122 ceph-mon[32255]: pgmap v16940: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:18 smithi022 ceph-mon[31360]: pgmap v16940: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:18 smithi022 ceph-mon[40250]: pgmap v16940: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:20] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:23:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:20 smithi122 ceph-mon[32255]: pgmap v16941: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:20 smithi022 ceph-mon[31360]: pgmap v16941: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:20 smithi022 ceph-mon[40250]: pgmap v16941: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:21.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:23:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:23:21.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:21.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:21.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:22 smithi122 ceph-mon[32255]: pgmap v16942: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1811694636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:23:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:23:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:22 smithi022 ceph-mon[31360]: pgmap v16942: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1811694636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:22 smithi022 ceph-mon[40250]: pgmap v16942: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1811694636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:24 smithi122 ceph-mon[32255]: pgmap v16943: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:24 smithi022 ceph-mon[31360]: pgmap v16943: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:24 smithi022 ceph-mon[40250]: pgmap v16943: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:26 smithi122 ceph-mon[32255]: pgmap v16944: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:26 smithi022 ceph-mon[31360]: pgmap v16944: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:26 smithi022 ceph-mon[40250]: pgmap v16944: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:26.845 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:27.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:27.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:27.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:23:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:23:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:23:27.462 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:23:27.463 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:23:27.463 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:23:27.463 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:23:27.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:23:27.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:23:27.464 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:23:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:28 smithi122 ceph-mon[32255]: pgmap v16945: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2993555114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:23:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:28 smithi022 ceph-mon[31360]: pgmap v16945: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2993555114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:23:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:28 smithi022 ceph-mon[40250]: pgmap v16945: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2993555114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:28.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:23:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:30] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:23:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:30 smithi122 ceph-mon[32255]: pgmap v16946: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:30 smithi022 ceph-mon[31360]: pgmap v16946: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:30 smithi022 ceph-mon[40250]: pgmap v16946: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:23:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:23:32.220 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:32 smithi122 ceph-mon[32255]: pgmap v16947: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:23:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:23:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:32 smithi022 ceph-mon[31360]: pgmap v16947: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:32 smithi022 ceph-mon[40250]: pgmap v16947: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:32.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:32.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:32.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1248846959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1248846959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1248846959' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:34 smithi122 ceph-mon[32255]: pgmap v16948: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:34 smithi022 ceph-mon[31360]: pgmap v16948: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:34 smithi022 ceph-mon[40250]: pgmap v16948: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:23:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:23:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:23:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:23:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:23:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:23:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:23:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:23:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:23:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:23:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:23:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:23:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:36 smithi122 ceph-mon[32255]: pgmap v16949: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:36 smithi022 ceph-mon[31360]: pgmap v16949: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:36 smithi022 ceph-mon[40250]: pgmap v16949: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:37.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:23:37.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:37.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:37.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:38 smithi022 ceph-mon[31360]: pgmap v16950: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3235381974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:38 smithi022 ceph-mon[40250]: pgmap v16950: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3235381974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:38 smithi122 ceph-mon[32255]: pgmap v16950: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3235381974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:40] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:23:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:40 smithi122 ceph-mon[32255]: pgmap v16951: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:40 smithi022 ceph-mon[31360]: pgmap v16951: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:40 smithi022 ceph-mon[40250]: pgmap v16951: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:23:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:23:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:42 smithi122 ceph-mon[32255]: pgmap v16952: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:23:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:23:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:42 smithi022 ceph-mon[31360]: pgmap v16952: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:42 smithi022 ceph-mon[40250]: pgmap v16952: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:42.961 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:43.330 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:43.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:43.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:43.336 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:44 smithi122 ceph-mon[32255]: pgmap v16953: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3569048570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:44 smithi022 ceph-mon[31360]: pgmap v16953: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3569048570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:44 smithi022 ceph-mon[40250]: pgmap v16953: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3569048570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:46 smithi022 ceph-mon[31360]: pgmap v16954: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:46 smithi022 ceph-mon[40250]: pgmap v16954: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:46 smithi122 ceph-mon[32255]: pgmap v16954: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:23:48.338 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:48 smithi022 ceph-mon[31360]: pgmap v16955: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:48 smithi022 ceph-mon[40250]: pgmap v16955: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:48 smithi122 ceph-mon[32255]: pgmap v16955: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:48.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:48.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:48.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3550755631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3550755631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3550755631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:50 smithi122 ceph-mon[32255]: pgmap v16956: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:50] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:23:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:50 smithi022 ceph-mon[40250]: pgmap v16956: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:50 smithi022 ceph-mon[31360]: pgmap v16956: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:23:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:23:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:52 smithi022 ceph-mon[31360]: pgmap v16957: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:52 smithi022 ceph-mon[40250]: pgmap v16957: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:52 smithi122 ceph-mon[32255]: pgmap v16957: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:23:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:23:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:23:53.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:54.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:54.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:54.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:23:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:54 smithi022 ceph-mon[40250]: pgmap v16958: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1586637623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:54 smithi022 ceph-mon[31360]: pgmap v16958: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1586637623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:54 smithi122 ceph-mon[32255]: pgmap v16958: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1586637623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:23:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:56 smithi022 ceph-mon[40250]: pgmap v16959: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:56 smithi022 ceph-mon[31360]: pgmap v16959: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:56 smithi122 ceph-mon[32255]: pgmap v16959: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:23:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:23:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:23:58 smithi022 ceph-mon[40250]: pgmap v16960: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:23:58 smithi022 ceph-mon[31360]: pgmap v16960: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:23:58 smithi122 ceph-mon[32255]: pgmap v16960: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:23:59.077 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:23:59.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:23:59.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:23:59.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:00] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:24:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:00 smithi122 ceph-mon[32255]: pgmap v16961: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/965522300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:00 smithi022 ceph-mon[31360]: pgmap v16961: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/965522300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:00 smithi022 ceph-mon[40250]: pgmap v16961: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/965522300' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:24:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:24:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:02 smithi022 ceph-mon[31360]: pgmap v16962: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:02 smithi022 ceph-mon[40250]: pgmap v16962: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:02 smithi122 ceph-mon[32255]: pgmap v16962: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:24:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:24:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:03 smithi022 ceph-mon[31360]: pgmap v16963: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:03 smithi022 ceph-mon[40250]: pgmap v16963: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:03 smithi122 ceph-mon[32255]: pgmap v16963: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:04.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:04.818 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:04.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:04.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:05 smithi122 ceph-mon[32255]: pgmap v16964: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3927853325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:05 smithi022 ceph-mon[31360]: pgmap v16964: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3927853325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:05 smithi022 ceph-mon[40250]: pgmap v16964: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3927853325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:07 smithi122 ceph-mon[32255]: pgmap v16965: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:07 smithi022 ceph-mon[31360]: pgmap v16965: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:24:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:07 smithi022 ceph-mon[40250]: pgmap v16965: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:09.820 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:10.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:09 smithi122 ceph-mon[32255]: pgmap v16966: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:10.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:10.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:10.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:10.191 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:09 smithi022 ceph-mon[31360]: pgmap v16966: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:10.192 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:09 smithi022 ceph-mon[40250]: pgmap v16966: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:10] "GET /metrics HTTP/1.1" 200 202475 "" "Prometheus/2.18.1" 2021-12-16T03:24:11.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3810448318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3810448318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3810448318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:24:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:24:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:11 smithi122 ceph-mon[32255]: pgmap v16967: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:11 smithi022 ceph-mon[31360]: pgmap v16967: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:11 smithi022 ceph-mon[40250]: pgmap v16967: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:24:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:24:13.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:13 smithi122 ceph-mon[32255]: pgmap v16968: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:14.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:13 smithi022 ceph-mon[31360]: pgmap v16968: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:13 smithi022 ceph-mon[40250]: pgmap v16968: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:15.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:15.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:15.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:15.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:15 smithi122 ceph-mon[32255]: pgmap v16969: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/817016038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:15 smithi022 ceph-mon[31360]: pgmap v16969: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/817016038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:15 smithi022 ceph-mon[40250]: pgmap v16969: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/817016038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:24:18.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:17 smithi122 ceph-mon[32255]: pgmap v16970: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:17 smithi022 ceph-mon[31360]: pgmap v16970: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:17 smithi022 ceph-mon[40250]: pgmap v16970: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:20.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:19 smithi122 ceph-mon[32255]: pgmap v16971: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:19 smithi022 ceph-mon[31360]: pgmap v16971: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:20.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:19 smithi022 ceph-mon[40250]: pgmap v16971: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:20] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:24:20.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:20.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:20.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:20.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:24:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:24:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:21 smithi122 ceph-mon[32255]: pgmap v16972: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1623033463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:21 smithi022 ceph-mon[31360]: pgmap v16972: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1623033463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:21 smithi022 ceph-mon[40250]: pgmap v16972: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:22.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1623033463' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:24:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:24:23.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:24.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:23 smithi122 ceph-mon[32255]: pgmap v16973: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:24.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:23 smithi022 ceph-mon[31360]: pgmap v16973: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:23 smithi022 ceph-mon[40250]: pgmap v16973: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:25.930 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:25 smithi122 ceph-mon[32255]: pgmap v16974: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:25 smithi022 ceph-mon[31360]: pgmap v16974: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:25 smithi022 ceph-mon[40250]: pgmap v16974: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:26.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:26.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:26.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:27.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2089201764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2089201764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2089201764' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:24:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:27 smithi122 ceph-mon[32255]: pgmap v16975: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:28.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:24:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:24:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:27 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:24:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[31360]: pgmap v16975: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:24:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:24:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:24:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[40250]: pgmap v16975: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:24:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:24:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:27 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:24:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:24:30.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:29 smithi122 ceph-mon[32255]: pgmap v16976: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:29 smithi022 ceph-mon[31360]: pgmap v16976: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:29 smithi022 ceph-mon[40250]: pgmap v16976: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:30] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:24:31.301 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:31.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:31.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:31.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:24:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:24:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:31 smithi022 ceph-mon[31360]: pgmap v16977: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:32.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3806235962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:32.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:31 smithi022 ceph-mon[40250]: pgmap v16977: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3806235962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:31 smithi122 ceph-mon[32255]: pgmap v16977: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3806235962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:24:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:24:33.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:33 smithi122 ceph-mon[32255]: pgmap v16978: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:33 smithi022 ceph-mon[31360]: pgmap v16978: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:33 smithi022 ceph-mon[40250]: pgmap v16978: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:24:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:24:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:24:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:24:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:24:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:24:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:24:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:24:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:24:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:24:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:24:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:24:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:35 smithi122 ceph-mon[32255]: pgmap v16979: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:35 smithi022 ceph-mon[31360]: pgmap v16979: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:35 smithi022 ceph-mon[40250]: pgmap v16979: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:36.667 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:37.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:37.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:37.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:24:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:37 smithi122 ceph-mon[32255]: pgmap v16980: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/591049662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:37 smithi022 ceph-mon[31360]: pgmap v16980: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/591049662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:37 smithi022 ceph-mon[40250]: pgmap v16980: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/591049662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:40.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:39 smithi122 ceph-mon[32255]: pgmap v16981: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:39 smithi022 ceph-mon[31360]: pgmap v16981: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:39 smithi022 ceph-mon[40250]: pgmap v16981: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:40] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:24:41.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:24:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:24:42.038 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:41 smithi122 ceph-mon[32255]: pgmap v16982: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:41 smithi022 ceph-mon[31360]: pgmap v16982: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:41 smithi022 ceph-mon[40250]: pgmap v16982: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:42.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:42.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:42.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:24:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:24:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3527362068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3527362068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3527362068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:43 smithi122 ceph-mon[32255]: pgmap v16983: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:43 smithi022 ceph-mon[31360]: pgmap v16983: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:43 smithi022 ceph-mon[40250]: pgmap v16983: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:45 smithi122 ceph-mon[32255]: pgmap v16984: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:45 smithi022 ceph-mon[31360]: pgmap v16984: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:45 smithi022 ceph-mon[40250]: pgmap v16984: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:47.407 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:47.774 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:47.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:47.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:47.775 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:24:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:47 smithi022 ceph-mon[31360]: pgmap v16985: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:48.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3820326770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:47 smithi022 ceph-mon[40250]: pgmap v16985: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3820326770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:47 smithi122 ceph-mon[32255]: pgmap v16985: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3820326770' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:49 smithi122 ceph-mon[32255]: pgmap v16986: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:49 smithi022 ceph-mon[31360]: pgmap v16986: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:49 smithi022 ceph-mon[40250]: pgmap v16986: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:50] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:24:51.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:24:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:24:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:51 smithi122 ceph-mon[32255]: pgmap v16987: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:51 smithi022 ceph-mon[31360]: pgmap v16987: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:51 smithi022 ceph-mon[40250]: pgmap v16987: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:24:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:24:52.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:53.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:53.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:53.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:53.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:24:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:53 smithi122 ceph-mon[32255]: pgmap v16988: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/320854533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:53 smithi022 ceph-mon[31360]: pgmap v16988: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/320854533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:53 smithi022 ceph-mon[40250]: pgmap v16988: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/320854533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:55 smithi122 ceph-mon[32255]: pgmap v16989: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:55 smithi022 ceph-mon[31360]: pgmap v16989: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:55 smithi022 ceph-mon[40250]: pgmap v16989: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:24:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:24:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:24:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:57 smithi122 ceph-mon[32255]: pgmap v16990: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:57 smithi022 ceph-mon[31360]: pgmap v16990: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:57 smithi022 ceph-mon[40250]: pgmap v16990: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:24:58.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:24:58.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:24:58.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:24:59.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3920783855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:59.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3920783855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:24:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3920783855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:24:59 smithi022 ceph-mon[31360]: pgmap v16991: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:24:59 smithi022 ceph-mon[40250]: pgmap v16991: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:24:59 smithi122 ceph-mon[32255]: pgmap v16991: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:00] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:25:01.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:25:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:25:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:01 smithi022 ceph-mon[31360]: pgmap v16992: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:01 smithi022 ceph-mon[40250]: pgmap v16992: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:01 smithi122 ceph-mon[32255]: pgmap v16992: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:25:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:25:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:03.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:03.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:03.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:03.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:03 smithi022 ceph-mon[31360]: pgmap v16993: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:04.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1743490995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:03 smithi022 ceph-mon[40250]: pgmap v16993: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:04.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1743490995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:03 smithi122 ceph-mon[32255]: pgmap v16993: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1743490995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:05 smithi022 ceph-mon[31360]: pgmap v16994: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:05 smithi022 ceph-mon[40250]: pgmap v16994: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:05 smithi122 ceph-mon[32255]: pgmap v16994: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:25:08.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:07 smithi022 ceph-mon[31360]: pgmap v16995: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:07 smithi022 ceph-mon[40250]: pgmap v16995: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:07 smithi122 ceph-mon[32255]: pgmap v16995: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:09.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:09.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:09.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:09 smithi022 ceph-mon[31360]: pgmap v16996: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2776905165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:09 smithi022 ceph-mon[40250]: pgmap v16996: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2776905165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:10] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:25:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:09 smithi122 ceph-mon[32255]: pgmap v16996: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2776905165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:11.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:25:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:25:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:11 smithi022 ceph-mon[31360]: pgmap v16997: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:11 smithi022 ceph-mon[40250]: pgmap v16997: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:11 smithi122 ceph-mon[32255]: pgmap v16997: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:25:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:25:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:14.260 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:13 smithi022 ceph-mon[40250]: pgmap v16998: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:14.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:13 smithi022 ceph-mon[31360]: pgmap v16998: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:13 smithi122 ceph-mon[32255]: pgmap v16998: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:14.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:14.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:14.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:15.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1257095564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:15.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1257095564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1257095564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:15 smithi022 ceph-mon[31360]: pgmap v16999: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:15 smithi022 ceph-mon[40250]: pgmap v16999: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:15 smithi122 ceph-mon[32255]: pgmap v16999: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:25:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:17 smithi022 ceph-mon[31360]: pgmap v17000: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:17 smithi022 ceph-mon[40250]: pgmap v17000: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:17 smithi122 ceph-mon[32255]: pgmap v17000: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:19.631 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:19.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:19.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:19.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:19 smithi022 ceph-mon[31360]: pgmap v17001: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:19 smithi022 ceph-mon[40250]: pgmap v17001: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:19 smithi122 ceph-mon[32255]: pgmap v17001: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:20] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:25:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3654740725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3654740725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:21.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3654740725' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:21.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:25:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:25:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:21 smithi022 ceph-mon[31360]: pgmap v17002: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:21 smithi022 ceph-mon[40250]: pgmap v17002: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:21 smithi122 ceph-mon[32255]: pgmap v17002: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:25:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:25:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:23 smithi022 ceph-mon[31360]: pgmap v17003: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:23 smithi022 ceph-mon[40250]: pgmap v17003: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:23 smithi122 ceph-mon[32255]: pgmap v17003: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:24.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:25 smithi022 ceph-mon[31360]: pgmap v17004: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/835831589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:25 smithi022 ceph-mon[40250]: pgmap v17004: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/835831589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:25 smithi122 ceph-mon[32255]: pgmap v17004: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/835831589' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:25:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:27 smithi022 ceph-mon[31360]: pgmap v17005: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:27 smithi022 ceph-mon[40250]: pgmap v17005: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:28.362 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:27 smithi122 ceph-mon[32255]: pgmap v17005: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:29.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:25:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:25:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:25:29.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:25:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:25:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:25:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:25:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:25:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:25:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:25:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:25:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:25:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:30 smithi022 ceph-mon[31360]: pgmap v17006: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:30 smithi022 ceph-mon[40250]: pgmap v17006: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:30.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:30] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:25:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:30 smithi122 ceph-mon[32255]: pgmap v17006: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:30.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:30.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:30.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:31.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2987882098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:31.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2987882098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:31.431 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2987882098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:31.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:25:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:25:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:32 smithi022 ceph-mon[31360]: pgmap v17007: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:32 smithi022 ceph-mon[40250]: pgmap v17007: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:32 smithi122 ceph-mon[32255]: pgmap v17007: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:25:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:25:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:34 smithi022 ceph-mon[31360]: pgmap v17008: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:34 smithi022 ceph-mon[40250]: pgmap v17008: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:34 smithi122 ceph-mon[32255]: pgmap v17008: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:25:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:25:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:25:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:25:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:25:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:25:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:25:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:25:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:25:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:25:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:25:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:25:35.739 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:36.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:36.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:36.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:36 smithi022 ceph-mon[31360]: pgmap v17009: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:36 smithi022 ceph-mon[40250]: pgmap v17009: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:36 smithi122 ceph-mon[32255]: pgmap v17009: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3570956189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3570956189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3570956189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:25:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:38 smithi022 ceph-mon[31360]: pgmap v17010: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:38 smithi022 ceph-mon[40250]: pgmap v17010: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:38 smithi122 ceph-mon[32255]: pgmap v17010: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:40 smithi022 ceph-mon[31360]: pgmap v17011: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:40 smithi022 ceph-mon[40250]: pgmap v17011: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:40] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:25:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:40 smithi122 ceph-mon[32255]: pgmap v17011: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:41.111 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:41.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:25:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:25:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:42 smithi022 ceph-mon[31360]: pgmap v17012: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:42.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3540478797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:42 smithi022 ceph-mon[40250]: pgmap v17012: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3540478797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:42 smithi122 ceph-mon[32255]: pgmap v17012: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3540478797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:25:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:25:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:44 smithi122 ceph-mon[32255]: pgmap v17013: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:44 smithi022 ceph-mon[31360]: pgmap v17013: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:44 smithi022 ceph-mon[40250]: pgmap v17013: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:46 smithi122 ceph-mon[32255]: pgmap v17014: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:46.482 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:46 smithi022 ceph-mon[31360]: pgmap v17014: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:46.482 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:46 smithi022 ceph-mon[40250]: pgmap v17014: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:46.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:46.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:46.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:47.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/288746933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:47.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/288746933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/288746933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:25:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:48 smithi122 ceph-mon[32255]: pgmap v17015: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:48 smithi022 ceph-mon[31360]: pgmap v17015: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:48 smithi022 ceph-mon[40250]: pgmap v17015: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:50 smithi122 ceph-mon[32255]: pgmap v17016: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:50] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:25:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:50 smithi022 ceph-mon[31360]: pgmap v17016: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:50 smithi022 ceph-mon[40250]: pgmap v17016: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:51.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:25:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:25:52.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:52.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:52.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:52.215 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:52 smithi022 ceph-mon[31360]: pgmap v17017: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:52.216 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:52 smithi022 ceph-mon[40250]: pgmap v17017: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:52 smithi122 ceph-mon[32255]: pgmap v17017: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:25:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:25:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:53.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3509391582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3509391582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:25:53.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3509391582' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:54 smithi122 ceph-mon[32255]: pgmap v17018: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:54 smithi022 ceph-mon[31360]: pgmap v17018: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:54 smithi022 ceph-mon[40250]: pgmap v17018: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:56 smithi122 ceph-mon[32255]: pgmap v17019: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:56 smithi022 ceph-mon[31360]: pgmap v17019: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:56 smithi022 ceph-mon[40250]: pgmap v17019: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:25:57.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:25:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:25:57.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:25:57.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:25:57.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:25:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:58 smithi122 ceph-mon[32255]: pgmap v17020: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:25:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2762175667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:58 smithi022 ceph-mon[31360]: pgmap v17020: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:25:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2762175667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:25:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:58 smithi022 ceph-mon[40250]: pgmap v17020: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:25:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:25:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2762175667' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:00] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:26:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:00 smithi122 ceph-mon[32255]: pgmap v17021: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:00 smithi022 ceph-mon[31360]: pgmap v17021: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:00 smithi022 ceph-mon[40250]: pgmap v17021: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:01.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:26:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:26:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:02 smithi122 ceph-mon[32255]: pgmap v17022: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:26:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:26:02.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:02 smithi022 ceph-mon[31360]: pgmap v17022: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:02.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:02 smithi022 ceph-mon[40250]: pgmap v17022: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:02.586 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:02.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:02.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:02.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2344704545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2344704545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2344704545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:04 smithi122 ceph-mon[32255]: pgmap v17023: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:04 smithi022 ceph-mon[31360]: pgmap v17023: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:04 smithi022 ceph-mon[40250]: pgmap v17023: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:06 smithi122 ceph-mon[32255]: pgmap v17024: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:06 smithi022 ceph-mon[31360]: pgmap v17024: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:06 smithi022 ceph-mon[40250]: pgmap v17024: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:26:07.955 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:08 smithi022 ceph-mon[31360]: pgmap v17025: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:08 smithi022 ceph-mon[40250]: pgmap v17025: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:08.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:08.325 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:08.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:08 smithi122 ceph-mon[32255]: pgmap v17025: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1594047467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1594047467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1594047467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:10] "GET /metrics HTTP/1.1" 200 202473 "" "Prometheus/2.18.1" 2021-12-16T03:26:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:10 smithi122 ceph-mon[32255]: pgmap v17026: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:10 smithi022 ceph-mon[31360]: pgmap v17026: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:10 smithi022 ceph-mon[40250]: pgmap v17026: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:26:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:26:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:12 smithi122 ceph-mon[32255]: pgmap v17027: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:26:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:26:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:12 smithi022 ceph-mon[31360]: pgmap v17027: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:12 smithi022 ceph-mon[40250]: pgmap v17027: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:13.327 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:13.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:13.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:13.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:14 smithi122 ceph-mon[32255]: pgmap v17028: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2658825203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:14 smithi022 ceph-mon[31360]: pgmap v17028: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2658825203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:14 smithi022 ceph-mon[40250]: pgmap v17028: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2658825203' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:16 smithi122 ceph-mon[32255]: pgmap v17029: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:16 smithi022 ceph-mon[31360]: pgmap v17029: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:16.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:16 smithi022 ceph-mon[40250]: pgmap v17029: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:26:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:18 smithi122 ceph-mon[32255]: pgmap v17030: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:18 smithi022 ceph-mon[31360]: pgmap v17030: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:18 smithi022 ceph-mon[40250]: pgmap v17030: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:18.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:19.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:19.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:19.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/440960901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/440960901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:19.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/440960901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:20 smithi122 ceph-mon[32255]: pgmap v17031: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:20] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:26:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:20 smithi022 ceph-mon[31360]: pgmap v17031: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:20 smithi022 ceph-mon[40250]: pgmap v17031: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:26:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:26:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:26:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:26:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:22 smithi122 ceph-mon[32255]: pgmap v17032: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:22 smithi022 ceph-mon[31360]: pgmap v17032: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:22 smithi022 ceph-mon[40250]: pgmap v17032: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:24.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:24 smithi122 ceph-mon[32255]: pgmap v17033: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:24.442 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:24.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:24.443 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:24.443 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:24 smithi022 ceph-mon[31360]: pgmap v17033: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:24.444 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:24 smithi022 ceph-mon[40250]: pgmap v17033: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3838271910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3838271910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3838271910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:26 smithi122 ceph-mon[32255]: pgmap v17034: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:26 smithi022 ceph-mon[31360]: pgmap v17034: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:26 smithi022 ceph-mon[40250]: pgmap v17034: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:26:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:28 smithi122 ceph-mon[32255]: pgmap v17035: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:28 smithi022 ceph-mon[31360]: pgmap v17035: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:28 smithi022 ceph-mon[40250]: pgmap v17035: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:29.129 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:26:29.130 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:26:29.130 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:26:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:26:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:26:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:26:29.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:26:29.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:26:29.445 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:26:29.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:29.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:29.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:30] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:26:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:30 smithi122 ceph-mon[32255]: pgmap v17036: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:26:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/64151985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:30 smithi022 ceph-mon[31360]: pgmap v17036: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:26:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/64151985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:30 smithi022 ceph-mon[40250]: pgmap v17036: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:26:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/64151985' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:26:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:26:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:32 smithi122 ceph-mon[32255]: pgmap v17037: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:26:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:26:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:32 smithi022 ceph-mon[31360]: pgmap v17037: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:32 smithi022 ceph-mon[40250]: pgmap v17037: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:33.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:33.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:34 smithi122 ceph-mon[32255]: pgmap v17038: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:34 smithi022 ceph-mon[31360]: pgmap v17038: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:34 smithi022 ceph-mon[40250]: pgmap v17038: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:34.803 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:35.144 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:26:35.145 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:26:35.145 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:26:35.145 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:26:35.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:35.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:35.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:26:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:26:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:26:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:26:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:26:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:26:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:26:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:26:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:36 smithi122 ceph-mon[32255]: pgmap v17039: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/704611778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:36 smithi022 ceph-mon[31360]: pgmap v17039: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/704611778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:36 smithi022 ceph-mon[40250]: pgmap v17039: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/704611778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:26:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:38 smithi122 ceph-mon[32255]: pgmap v17040: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:38 smithi022 ceph-mon[31360]: pgmap v17040: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:38 smithi022 ceph-mon[40250]: pgmap v17040: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T03:26:40.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:40] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:26:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:40 smithi122 ceph-mon[32255]: pgmap v17041: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:40 smithi022 ceph-mon[31360]: pgmap v17041: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:40 smithi022 ceph-mon[40250]: pgmap v17041: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T03:26:40.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:40.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:40.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1514733137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1514733137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1514733137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:41.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:26:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:26:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:42 smithi122 ceph-mon[32255]: pgmap v17042: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:26:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:26:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:42 smithi022 ceph-mon[31360]: pgmap v17042: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:42 smithi022 ceph-mon[40250]: pgmap v17042: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:44 smithi122 ceph-mon[32255]: pgmap v17043: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:44 smithi022 ceph-mon[31360]: pgmap v17043: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:44 smithi022 ceph-mon[40250]: pgmap v17043: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:45.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:45.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:45.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:45.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:46 smithi122 ceph-mon[32255]: pgmap v17044: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1397407044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:46 smithi022 ceph-mon[31360]: pgmap v17044: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1397407044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:46 smithi022 ceph-mon[40250]: pgmap v17044: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1397407044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:26:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:48 smithi122 ceph-mon[32255]: pgmap v17045: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:48 smithi022 ceph-mon[31360]: pgmap v17045: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:48.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:48 smithi022 ceph-mon[40250]: pgmap v17045: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:50] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:26:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:50 smithi122 ceph-mon[32255]: pgmap v17046: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:50 smithi022 ceph-mon[31360]: pgmap v17046: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:50 smithi022 ceph-mon[40250]: pgmap v17046: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:50.916 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:51.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:51.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:51.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:26:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:26:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:52 smithi122 ceph-mon[32255]: pgmap v17047: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1338833126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:26:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:26:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:52 smithi022 ceph-mon[31360]: pgmap v17047: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1338833126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:52 smithi022 ceph-mon[40250]: pgmap v17047: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1338833126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:26:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:54 smithi122 ceph-mon[32255]: pgmap v17048: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:54 smithi022 ceph-mon[31360]: pgmap v17048: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:54 smithi022 ceph-mon[40250]: pgmap v17048: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:56.290 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:26:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:56 smithi022 ceph-mon[31360]: pgmap v17049: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:56 smithi022 ceph-mon[40250]: pgmap v17049: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:56.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:26:56.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:26:56.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:26:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:56 smithi122 ceph-mon[32255]: pgmap v17049: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:26:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/538301592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:26:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/538301592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/538301592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:26:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:26:58 smithi022 ceph-mon[31360]: pgmap v17050: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:26:58 smithi022 ceph-mon[40250]: pgmap v17050: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:26:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:26:58 smithi122 ceph-mon[32255]: pgmap v17050: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:00] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:27:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:00 smithi122 ceph-mon[32255]: pgmap v17051: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:00 smithi022 ceph-mon[31360]: pgmap v17051: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:00 smithi022 ceph-mon[40250]: pgmap v17051: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:01.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:01.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:27:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:27:02.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:02.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:02.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:02 smithi022 ceph-mon[31360]: pgmap v17052: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3094640570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:02 smithi022 ceph-mon[40250]: pgmap v17052: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3094640570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:02 smithi122 ceph-mon[32255]: pgmap v17052: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3094640570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:27:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:27:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:04 smithi022 ceph-mon[31360]: pgmap v17053: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:04.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:04 smithi022 ceph-mon[40250]: pgmap v17053: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:04 smithi122 ceph-mon[32255]: pgmap v17053: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:06 smithi022 ceph-mon[31360]: pgmap v17054: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:06 smithi022 ceph-mon[40250]: pgmap v17054: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:06 smithi122 ceph-mon[32255]: pgmap v17054: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:07.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:07.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:07.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:07.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:27:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:08 smithi022 ceph-mon[31360]: pgmap v17055: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1770329319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:08 smithi022 ceph-mon[40250]: pgmap v17055: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1770329319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:08 smithi122 ceph-mon[32255]: pgmap v17055: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1770329319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:10 smithi122 ceph-mon[32255]: pgmap v17056: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:10] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:27:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:10 smithi022 ceph-mon[31360]: pgmap v17056: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:10 smithi022 ceph-mon[40250]: pgmap v17056: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:27:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:27:12.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:12 smithi022 ceph-mon[31360]: pgmap v17057: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:12 smithi022 ceph-mon[40250]: pgmap v17057: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:12 smithi122 ceph-mon[32255]: pgmap v17057: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:27:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:27:12.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:12.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:12.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3510156046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3510156046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3510156046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:14 smithi022 ceph-mon[31360]: pgmap v17058: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:14 smithi022 ceph-mon[40250]: pgmap v17058: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:14 smithi122 ceph-mon[32255]: pgmap v17058: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:16 smithi022 ceph-mon[31360]: pgmap v17059: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:16 smithi022 ceph-mon[40250]: pgmap v17059: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:16 smithi122 ceph-mon[32255]: pgmap v17059: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:17.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:17.767 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:27:18.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:18.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:18.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:18 smithi022 ceph-mon[31360]: pgmap v17060: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1525075046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:18 smithi022 ceph-mon[40250]: pgmap v17060: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1525075046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:18 smithi122 ceph-mon[32255]: pgmap v17060: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:18.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1525075046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:20 smithi122 ceph-mon[32255]: pgmap v17061: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:20] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:27:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:20 smithi022 ceph-mon[31360]: pgmap v17061: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:20 smithi022 ceph-mon[40250]: pgmap v17061: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:27:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:27:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:22 smithi022 ceph-mon[31360]: pgmap v17062: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:22 smithi022 ceph-mon[40250]: pgmap v17062: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:22 smithi122 ceph-mon[32255]: pgmap v17062: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:27:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:27:23.140 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:23.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:23.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:23.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:23.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:23.517 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:24 smithi022 ceph-mon[31360]: pgmap v17063: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/479317120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:24 smithi022 ceph-mon[40250]: pgmap v17063: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/479317120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:24 smithi122 ceph-mon[32255]: pgmap v17063: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/479317120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:26 smithi022 ceph-mon[31360]: pgmap v17064: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:26 smithi022 ceph-mon[40250]: pgmap v17064: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:26 smithi122 ceph-mon[32255]: pgmap v17064: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:27:28.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:28.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:28 smithi022 ceph-mon[31360]: pgmap v17065: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:28.519 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:28 smithi022 ceph-mon[40250]: pgmap v17065: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:28 smithi122 ceph-mon[32255]: pgmap v17065: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:28.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:28.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:29 smithi022 ceph-mon[31360]: pgmap v17066: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187727473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:29 smithi022 ceph-mon[40250]: pgmap v17066: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187727473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:29 smithi122 ceph-mon[32255]: pgmap v17066: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187727473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:30] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:27:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:27:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:27:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:27:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:27:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:27:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:27:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:27:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:27:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:27:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:27:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:27:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:27:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:31 smithi022 ceph-mon[31360]: pgmap v17067: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:31 smithi022 ceph-mon[40250]: pgmap v17067: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:31.536 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:31 smithi122 ceph-mon[32255]: pgmap v17067: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:27:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:27:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:27:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:27:33.889 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:34.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:33 smithi122 ceph-mon[32255]: pgmap v17068: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:34.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:34.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:34.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:34.261 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:33 smithi022 ceph-mon[40250]: pgmap v17068: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:34.261 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:33 smithi022 ceph-mon[31360]: pgmap v17068: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1863585182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:27:35.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:27:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:27:35.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:27:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1863585182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:27:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:27:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:27:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:27:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1863585182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:27:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:27:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:27:35.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:27:36.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:35 smithi122 ceph-mon[32255]: pgmap v17069: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:35 smithi022 ceph-mon[31360]: pgmap v17069: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:36.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:35 smithi022 ceph-mon[40250]: pgmap v17069: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:27:38.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:37 smithi122 ceph-mon[32255]: pgmap v17070: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:37 smithi022 ceph-mon[31360]: pgmap v17070: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:37 smithi022 ceph-mon[40250]: pgmap v17070: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:39.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:39.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:39.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:39.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:39 smithi122 ceph-mon[32255]: pgmap v17071: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:40.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/194848038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:39 smithi022 ceph-mon[31360]: pgmap v17071: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:40.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/194848038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:39 smithi022 ceph-mon[40250]: pgmap v17071: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:40.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/194848038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:40] "GET /metrics HTTP/1.1" 200 202472 "" "Prometheus/2.18.1" 2021-12-16T03:27:41.895 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:27:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:27:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:41 smithi122 ceph-mon[32255]: pgmap v17072: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:41 smithi022 ceph-mon[31360]: pgmap v17072: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:41 smithi022 ceph-mon[40250]: pgmap v17072: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:27:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:27:43.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:44.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:43 smithi122 ceph-mon[32255]: pgmap v17073: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:43 smithi022 ceph-mon[31360]: pgmap v17073: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:44.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:43 smithi022 ceph-mon[40250]: pgmap v17073: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:44.630 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:45.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:45.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:45.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:45 smithi122 ceph-mon[32255]: pgmap v17074: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:46.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2225504109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:45 smithi022 ceph-mon[31360]: pgmap v17074: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:46.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2225504109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:45 smithi022 ceph-mon[40250]: pgmap v17074: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2225504109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:27:48.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:47 smithi122 ceph-mon[32255]: pgmap v17075: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:47 smithi022 ceph-mon[31360]: pgmap v17075: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:47 smithi022 ceph-mon[40250]: pgmap v17075: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:50.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:49 smithi122 ceph-mon[32255]: pgmap v17076: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:49 smithi022 ceph-mon[31360]: pgmap v17076: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:49 smithi022 ceph-mon[40250]: pgmap v17076: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:50.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:50.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:50.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:50] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:27:51.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/268963880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/268963880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/268963880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:51.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:27:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:27:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:51 smithi122 ceph-mon[32255]: pgmap v17077: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:51 smithi022 ceph-mon[31360]: pgmap v17077: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:51 smithi022 ceph-mon[40250]: pgmap v17077: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:27:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:27:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:27:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:53 smithi122 ceph-mon[32255]: pgmap v17078: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:53 smithi022 ceph-mon[31360]: pgmap v17078: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:53 smithi022 ceph-mon[40250]: pgmap v17078: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:55.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:27:55.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:27:55.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:27:55.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:27:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:55 smithi022 ceph-mon[31360]: pgmap v17079: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2734267070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:55 smithi022 ceph-mon[40250]: pgmap v17079: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2734267070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:55 smithi122 ceph-mon[32255]: pgmap v17079: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:27:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2734267070' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:27:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:27:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:57 smithi122 ceph-mon[32255]: pgmap v17080: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:57 smithi022 ceph-mon[31360]: pgmap v17080: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:27:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:57 smithi022 ceph-mon[40250]: pgmap v17080: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:00.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:00] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:28:00.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:27:59 smithi122 ceph-mon[32255]: pgmap v17081: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:27:59 smithi022 ceph-mon[31360]: pgmap v17081: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:27:59 smithi022 ceph-mon[40250]: pgmap v17081: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:00.743 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:01.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:01.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:01.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:01.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:28:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:28:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:01 smithi122 ceph-mon[32255]: pgmap v17082: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/915133861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:01 smithi022 ceph-mon[31360]: pgmap v17082: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/915133861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:01 smithi022 ceph-mon[40250]: pgmap v17082: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/915133861' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:28:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:28:03.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:03 smithi022 ceph-mon[31360]: pgmap v17083: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:03 smithi022 ceph-mon[40250]: pgmap v17083: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:03 smithi122 ceph-mon[32255]: pgmap v17083: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:06.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:05 smithi022 ceph-mon[31360]: pgmap v17084: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:05 smithi022 ceph-mon[40250]: pgmap v17084: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:05 smithi122 ceph-mon[32255]: pgmap v17084: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:06.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:06.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:06.671 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2513334151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2513334151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2513334151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:28:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:07 smithi022 ceph-mon[31360]: pgmap v17085: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:07 smithi022 ceph-mon[40250]: pgmap v17085: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:07 smithi122 ceph-mon[32255]: pgmap v17085: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:09 smithi022 ceph-mon[31360]: pgmap v17086: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:09 smithi022 ceph-mon[40250]: pgmap v17086: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:10] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:28:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:09 smithi122 ceph-mon[32255]: pgmap v17086: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:28:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:11.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:11.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:11.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:28:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:28:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:11 smithi022 ceph-mon[31360]: pgmap v17087: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3998823657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:11 smithi022 ceph-mon[40250]: pgmap v17087: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3998823657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:11 smithi122 ceph-mon[32255]: pgmap v17087: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3998823657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:28:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:28:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:13 smithi022 ceph-mon[31360]: pgmap v17088: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:13 smithi022 ceph-mon[40250]: pgmap v17088: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:13 smithi122 ceph-mon[32255]: pgmap v17088: 132 pgs: 132 active+clean; 582 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:15 smithi022 ceph-mon[31360]: pgmap v17089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:15 smithi022 ceph-mon[40250]: pgmap v17089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:15 smithi122 ceph-mon[32255]: pgmap v17089: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:16.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:17.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:17.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:17.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:28:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:17 smithi022 ceph-mon[31360]: pgmap v17090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:18.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2339430656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:17 smithi022 ceph-mon[40250]: pgmap v17090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2339430656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:17 smithi122 ceph-mon[32255]: pgmap v17090: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2339430656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:19 smithi022 ceph-mon[31360]: pgmap v17091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:19 smithi022 ceph-mon[40250]: pgmap v17091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:20] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:28:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:19 smithi122 ceph-mon[32255]: pgmap v17091: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:28:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:28:22.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:22.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:21 smithi022 ceph-mon[31360]: pgmap v17092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:22.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:21 smithi022 ceph-mon[40250]: pgmap v17092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:21 smithi122 ceph-mon[32255]: pgmap v17092: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:22.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:22.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:22.641 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:28:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:28:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:23.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3009519580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3009519580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3009519580' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:23 smithi022 ceph-mon[31360]: pgmap v17093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:23 smithi022 ceph-mon[40250]: pgmap v17093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:23 smithi122 ceph-mon[32255]: pgmap v17093: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:28:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:25 smithi022 ceph-mon[31360]: pgmap v17094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:25 smithi022 ceph-mon[40250]: pgmap v17094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:25 smithi122 ceph-mon[32255]: pgmap v17094: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:28:27.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:27.643 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:28:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:27 smithi022 ceph-mon[31360]: pgmap v17095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:27 smithi022 ceph-mon[40250]: pgmap v17095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:27 smithi122 ceph-mon[32255]: pgmap v17095: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:29.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/869892545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:29.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/869892545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/869892545' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[31360]: pgmap v17096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:28:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:28:30.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:28:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[40250]: pgmap v17096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:28:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:28:30.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:28:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:29 smithi122 ceph-mon[32255]: pgmap v17096: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:30.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:28:30.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:28:30.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:28:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:30] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:28:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:28:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:28:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:31 smithi022 ceph-mon[31360]: pgmap v17097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:32.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:31 smithi022 ceph-mon[40250]: pgmap v17097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:32.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:31 smithi122 ceph-mon[32255]: pgmap v17097: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:28:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:28:33.016 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:33.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:33.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:33.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:34.336 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: pgmap v17098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:34.337 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:34.337 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3642117220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:34.338 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.338 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.339 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.339 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.340 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.340 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.340 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.341 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.341 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: pgmap v17098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3642117220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: pgmap v17098: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3642117220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:28:34.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:35 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 2863M 2021-12-16T03:28:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:28:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:28:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:28:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:28:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 2863M 2021-12-16T03:28:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:28:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:28:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:28:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:28:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 2863M 2021-12-16T03:28:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:28:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:28:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:28:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:28:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:28:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:36 smithi122 ceph-mon[32255]: pgmap v17099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:36 smithi022 ceph-mon[31360]: pgmap v17099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:36 smithi022 ceph-mon[40250]: pgmap v17099: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:28:38.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:38.402 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:38 smithi022 ceph-mon[40250]: pgmap v17100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:38.402 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:38 smithi022 ceph-mon[31360]: pgmap v17100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:38 smithi122 ceph-mon[32255]: pgmap v17100: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:38.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:38.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:38.773 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1125483695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1125483695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1125483695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:40 smithi122 ceph-mon[32255]: pgmap v17101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:40] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:28:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:40 smithi022 ceph-mon[31360]: pgmap v17101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:40 smithi022 ceph-mon[40250]: pgmap v17101: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:41.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:28:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:28:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:42 smithi122 ceph-mon[32255]: pgmap v17102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:28:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:28:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:42 smithi022 ceph-mon[40250]: pgmap v17102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:42 smithi022 ceph-mon[31360]: pgmap v17102: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:43.774 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:44.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:44.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:44.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:44 smithi122 ceph-mon[32255]: pgmap v17103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:44 smithi022 ceph-mon[40250]: pgmap v17103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:44 smithi022 ceph-mon[31360]: pgmap v17103: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3276572842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3276572842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3276572842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:46 smithi122 ceph-mon[32255]: pgmap v17104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:46 smithi022 ceph-mon[31360]: pgmap v17104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:46 smithi022 ceph-mon[40250]: pgmap v17104: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:28:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:48 smithi122 ceph-mon[32255]: pgmap v17105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:48 smithi022 ceph-mon[31360]: pgmap v17105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:48 smithi022 ceph-mon[40250]: pgmap v17105: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:49.143 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:49.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:49.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:49.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:50] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:28:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:50 smithi122 ceph-mon[32255]: pgmap v17106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/665033555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:50 smithi022 ceph-mon[31360]: pgmap v17106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/665033555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:50 smithi022 ceph-mon[40250]: pgmap v17106: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/665033555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:28:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:28:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:52 smithi122 ceph-mon[32255]: pgmap v17107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:28:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:28:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:52 smithi022 ceph-mon[31360]: pgmap v17107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:52 smithi022 ceph-mon[40250]: pgmap v17107: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:28:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:54 smithi122 ceph-mon[32255]: pgmap v17108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:54.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:28:54.512 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:54 smithi022 ceph-mon[40250]: pgmap v17108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:54.512 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:54 smithi022 ceph-mon[31360]: pgmap v17108: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:54.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:28:54.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:28:54.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:28:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/716824139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/716824139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/716824139' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:28:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:56 smithi122 ceph-mon[32255]: pgmap v17109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:56 smithi022 ceph-mon[31360]: pgmap v17109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:56 smithi022 ceph-mon[40250]: pgmap v17109: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:28:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:28:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:28:58 smithi122 ceph-mon[32255]: pgmap v17110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:28:58 smithi022 ceph-mon[31360]: pgmap v17110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:28:58 smithi022 ceph-mon[40250]: pgmap v17110: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:28:59.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:00.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:00.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:00.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:00] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:29:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:00 smithi122 ceph-mon[32255]: pgmap v17111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:00 smithi022 ceph-mon[31360]: pgmap v17111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:00 smithi022 ceph-mon[40250]: pgmap v17111: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3564352239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3564352239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3564352239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:01.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:29:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:29:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:02 smithi122 ceph-mon[32255]: pgmap v17112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:29:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:29:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:02 smithi022 ceph-mon[31360]: pgmap v17112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:02 smithi022 ceph-mon[40250]: pgmap v17112: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:04 smithi122 ceph-mon[32255]: pgmap v17113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:04 smithi022 ceph-mon[31360]: pgmap v17113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:04 smithi022 ceph-mon[40250]: pgmap v17113: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:05.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:05.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:05.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:05.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:06 smithi122 ceph-mon[32255]: pgmap v17114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3140015993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:06 smithi022 ceph-mon[31360]: pgmap v17114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3140015993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:06 smithi022 ceph-mon[40250]: pgmap v17114: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3140015993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:29:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:08 smithi122 ceph-mon[32255]: pgmap v17115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:08 smithi022 ceph-mon[31360]: pgmap v17115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:08 smithi022 ceph-mon[40250]: pgmap v17115: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:10] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:29:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:10 smithi122 ceph-mon[32255]: pgmap v17116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:10 smithi022 ceph-mon[31360]: pgmap v17116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:10 smithi022 ceph-mon[40250]: pgmap v17116: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:10.623 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:10.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:10.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:10.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3087766796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3087766796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3087766796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:11.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:29:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:29:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:12 smithi122 ceph-mon[32255]: pgmap v17117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:29:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:29:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:12 smithi022 ceph-mon[31360]: pgmap v17117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:12 smithi022 ceph-mon[40250]: pgmap v17117: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:14 smithi122 ceph-mon[32255]: pgmap v17118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:14 smithi022 ceph-mon[31360]: pgmap v17118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:14 smithi022 ceph-mon[40250]: pgmap v17118: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:15.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:16 smithi022 ceph-mon[31360]: pgmap v17119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:16 smithi022 ceph-mon[40250]: pgmap v17119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:16.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:16.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:16.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:16 smithi122 ceph-mon[32255]: pgmap v17119: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2013075400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2013075400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:29:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2013075400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:18 smithi122 ceph-mon[32255]: pgmap v17120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:18 smithi022 ceph-mon[31360]: pgmap v17120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:18 smithi022 ceph-mon[40250]: pgmap v17120: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:20 smithi122 ceph-mon[32255]: pgmap v17121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:20] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:29:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:20 smithi022 ceph-mon[31360]: pgmap v17121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:20 smithi022 ceph-mon[40250]: pgmap v17121: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:21.363 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:21.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:21.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:21.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:21.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:29:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:29:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:29:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:29:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:22 smithi122 ceph-mon[32255]: pgmap v17122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1748762939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:22 smithi022 ceph-mon[31360]: pgmap v17122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1748762939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:22 smithi022 ceph-mon[40250]: pgmap v17122: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1748762939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:24 smithi122 ceph-mon[32255]: pgmap v17123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:24 smithi022 ceph-mon[31360]: pgmap v17123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:24 smithi022 ceph-mon[40250]: pgmap v17123: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:26 smithi022 ceph-mon[31360]: pgmap v17124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:26 smithi022 ceph-mon[40250]: pgmap v17124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:26 smithi122 ceph-mon[32255]: pgmap v17124: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:26.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:27.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:27.110 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:27.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3839229484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:27.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3839229484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3839229484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:29:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:28 smithi022 ceph-mon[31360]: pgmap v17125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:28 smithi022 ceph-mon[40250]: pgmap v17125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:28 smithi122 ceph-mon[32255]: pgmap v17125: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:30] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:29:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:30 smithi122 ceph-mon[32255]: pgmap v17126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:30 smithi022 ceph-mon[31360]: pgmap v17126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:30 smithi022 ceph-mon[40250]: pgmap v17126: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:31.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:29:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:29:32.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:32.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:32.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:32.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:32.483 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:32 smithi022 ceph-mon[31360]: pgmap v17127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:32 smithi022 ceph-mon[40250]: pgmap v17127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:32 smithi122 ceph-mon[32255]: pgmap v17127: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:29:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:29:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/380716625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/380716625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/380716625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:34 smithi022 ceph-mon[31360]: pgmap v17128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:34 smithi022 ceph-mon[40250]: pgmap v17128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:34 smithi122 ceph-mon[32255]: pgmap v17128: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:35.213 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:29:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:29:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:29:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:29:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:29:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:29:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:29:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:29:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:29:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:29:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:29:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:29:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:29:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:29:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:29:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:29:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:29:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:29:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:29:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:29:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:29:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:36 smithi022 ceph-mon[31360]: pgmap v17129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:29:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:36 smithi022 ceph-mon[40250]: pgmap v17129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:29:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:36 smithi122 ceph-mon[32255]: pgmap v17129: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:29:37.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:29:37.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:37.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:37.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:38 smithi022 ceph-mon[31360]: pgmap v17130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2332157288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:38 smithi022 ceph-mon[40250]: pgmap v17130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2332157288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:38 smithi122 ceph-mon[32255]: pgmap v17130: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2332157288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:40] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:29:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:40 smithi122 ceph-mon[32255]: pgmap v17131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:40 smithi022 ceph-mon[31360]: pgmap v17131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:40 smithi022 ceph-mon[40250]: pgmap v17131: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:41.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:29:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:29:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:42 smithi022 ceph-mon[31360]: pgmap v17132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:42 smithi022 ceph-mon[40250]: pgmap v17132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:42 smithi122 ceph-mon[32255]: pgmap v17132: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:29:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:29:42.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:43.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:43.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:43.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2527164654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2527164654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2527164654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:44 smithi022 ceph-mon[31360]: pgmap v17133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:44 smithi022 ceph-mon[40250]: pgmap v17133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:44 smithi122 ceph-mon[32255]: pgmap v17133: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:45 smithi022 ceph-mon[31360]: pgmap v17134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:45 smithi022 ceph-mon[40250]: pgmap v17134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:45 smithi122 ceph-mon[32255]: pgmap v17134: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:47 smithi022 ceph-mon[31360]: pgmap v17135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:29:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:47 smithi022 ceph-mon[40250]: pgmap v17135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:47 smithi122 ceph-mon[32255]: pgmap v17135: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:48.223 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:48.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:48.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:48.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:49.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2787153715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:49.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2787153715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2787153715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:50.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:49 smithi122 ceph-mon[32255]: pgmap v17136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:49 smithi022 ceph-mon[31360]: pgmap v17136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:49 smithi022 ceph-mon[40250]: pgmap v17136: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:50] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:29:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:51 smithi122 ceph-mon[32255]: pgmap v17137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:51.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:29:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:29:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:51 smithi022 ceph-mon[31360]: pgmap v17137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:51 smithi022 ceph-mon[40250]: pgmap v17137: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:29:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:29:53.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:29:53.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:53.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:53.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:53.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:29:54.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:53 smithi122 ceph-mon[32255]: pgmap v17138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:53 smithi022 ceph-mon[31360]: pgmap v17138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:53 smithi022 ceph-mon[40250]: pgmap v17138: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:55.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3891902771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:55.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3891902771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:55.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3891902771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:29:56.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:55 smithi122 ceph-mon[32255]: pgmap v17139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:55 smithi022 ceph-mon[31360]: pgmap v17139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:55 smithi022 ceph-mon[40250]: pgmap v17139: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:29:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:29:58.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:57 smithi122 ceph-mon[32255]: pgmap v17140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:57 smithi022 ceph-mon[31360]: pgmap v17140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:57 smithi022 ceph-mon[40250]: pgmap v17140: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:29:58.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:29:59.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:29:59.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:29:59.335 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:00.185 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:00] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:30:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:59 smithi122 ceph-mon[32255]: pgmap v17141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:00.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:29:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2617852159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:59 smithi022 ceph-mon[31360]: pgmap v17141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:29:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2617852159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:59 smithi022 ceph-mon[40250]: pgmap v17141: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:29:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2617852159' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T03:30:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T03:30:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T03:30:01.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:30:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:30:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:01 smithi022 ceph-mon[31360]: pgmap v17142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:01 smithi022 ceph-mon[40250]: pgmap v17142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:01 smithi122 ceph-mon[32255]: pgmap v17142: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:30:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:30:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:02 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:02 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:02 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:03 smithi022 ceph-mon[31360]: pgmap v17143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:03 smithi022 ceph-mon[40250]: pgmap v17143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:04.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:03 smithi122 ceph-mon[32255]: pgmap v17143: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:04.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:04.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:04.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1658820798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1658820798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1658820798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:05 smithi022 ceph-mon[31360]: pgmap v17144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:05 smithi022 ceph-mon[40250]: pgmap v17144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:05 smithi122 ceph-mon[32255]: pgmap v17144: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:30:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:07 smithi022 ceph-mon[31360]: pgmap v17145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:07 smithi022 ceph-mon[40250]: pgmap v17145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:07 smithi122 ceph-mon[32255]: pgmap v17145: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:09.705 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:10.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:10.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:10.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:09 smithi022 ceph-mon[31360]: pgmap v17146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:10.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:09 smithi022 ceph-mon[40250]: pgmap v17146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:10] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:30:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:09 smithi122 ceph-mon[32255]: pgmap v17146: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:11.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1495886313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:11.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1495886313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1495886313' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:11.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:30:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:30:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:11 smithi022 ceph-mon[31360]: pgmap v17147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:11 smithi022 ceph-mon[40250]: pgmap v17147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:11 smithi122 ceph-mon[32255]: pgmap v17147: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:30:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:30:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:13 smithi022 ceph-mon[31360]: pgmap v17148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:13 smithi022 ceph-mon[40250]: pgmap v17148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:13 smithi122 ceph-mon[32255]: pgmap v17148: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:15.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:15.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:15.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:15.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:15 smithi022 ceph-mon[31360]: pgmap v17149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:16.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/573169769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:15 smithi022 ceph-mon[40250]: pgmap v17149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:16.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/573169769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:15 smithi122 ceph-mon[32255]: pgmap v17149: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/573169769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:30:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:17 smithi022 ceph-mon[31360]: pgmap v17150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:17 smithi022 ceph-mon[40250]: pgmap v17150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:17 smithi122 ceph-mon[32255]: pgmap v17150: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:19 smithi022 ceph-mon[31360]: pgmap v17151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:19 smithi022 ceph-mon[40250]: pgmap v17151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:19 smithi122 ceph-mon[32255]: pgmap v17151: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:20] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:30:20.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:21.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3435105456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:21.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3435105456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3435105456' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:21.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:30:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:30:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:21 smithi022 ceph-mon[31360]: pgmap v17152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:21 smithi022 ceph-mon[40250]: pgmap v17152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:21 smithi122 ceph-mon[32255]: pgmap v17152: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:30:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:30:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:22 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:22 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:22 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:23 smithi022 ceph-mon[31360]: pgmap v17153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:23 smithi022 ceph-mon[40250]: pgmap v17153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:23 smithi122 ceph-mon[32255]: pgmap v17153: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:25.819 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:26.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:26.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:26.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:26.190 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:25 smithi022 ceph-mon[31360]: pgmap v17154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:26.190 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:25 smithi022 ceph-mon[40250]: pgmap v17154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:25 smithi122 ceph-mon[32255]: pgmap v17154: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1861541086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:27.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1861541086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1861541086' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:30:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:28 smithi022 ceph-mon[31360]: pgmap v17155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:28 smithi022 ceph-mon[40250]: pgmap v17155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:28 smithi122 ceph-mon[32255]: pgmap v17155: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:30 smithi022 ceph-mon[31360]: pgmap v17156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:30 smithi022 ceph-mon[40250]: pgmap v17156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:30] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:30:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:30 smithi122 ceph-mon[32255]: pgmap v17156: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:31.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:31.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:31.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:32.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:30:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:30:32.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:32 smithi022 ceph-mon[31360]: pgmap v17157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:32.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/42469186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:32.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:32 smithi022 ceph-mon[40250]: pgmap v17157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:32.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/42469186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:32 smithi122 ceph-mon[32255]: pgmap v17157: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/42469186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:30:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:30:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:34 smithi022 ceph-mon[31360]: pgmap v17158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:34 smithi022 ceph-mon[40250]: pgmap v17158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:34 smithi122 ceph-mon[32255]: pgmap v17158: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:30:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:30:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:30:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:30:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:30:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:30:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:30:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:30:35.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:30:35.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:30:35.292 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:30:35.292 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:30:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[31360]: pgmap v17159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:30:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:30:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:30:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:30:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[40250]: pgmap v17159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:30:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:30:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:30:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:30:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:36 smithi122 ceph-mon[32255]: pgmap v17159: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:30:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:30:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:30:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:30:36.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:36.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:36.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:36.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3248750353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:37.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3248750353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3248750353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:30:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:38 smithi022 ceph-mon[31360]: pgmap v17160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:38.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:38 smithi022 ceph-mon[40250]: pgmap v17160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:38 smithi122 ceph-mon[32255]: pgmap v17160: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:40 smithi122 ceph-mon[32255]: pgmap v17161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:40] "GET /metrics HTTP/1.1" 200 202460 "" "Prometheus/2.18.1" 2021-12-16T03:30:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:40 smithi022 ceph-mon[31360]: pgmap v17161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:40 smithi022 ceph-mon[40250]: pgmap v17161: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:41.932 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:42.031 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:30:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:30:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:42 smithi022 ceph-mon[31360]: pgmap v17162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:42 smithi022 ceph-mon[40250]: pgmap v17162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:42.297 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:42.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:42.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:42 smithi122 ceph-mon[32255]: pgmap v17162: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:30:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:30:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1421096723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1421096723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1421096723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:44 smithi122 ceph-mon[32255]: pgmap v17163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:44 smithi022 ceph-mon[31360]: pgmap v17163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:44 smithi022 ceph-mon[40250]: pgmap v17163: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:46 smithi122 ceph-mon[32255]: pgmap v17164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:46 smithi022 ceph-mon[31360]: pgmap v17164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:46 smithi022 ceph-mon[40250]: pgmap v17164: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:47.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:47.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:47.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:47.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:47.670 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:30:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:48 smithi122 ceph-mon[32255]: pgmap v17165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1994642091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:48 smithi022 ceph-mon[31360]: pgmap v17165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1994642091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:48 smithi022 ceph-mon[40250]: pgmap v17165: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1994642091' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:50] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:30:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:50 smithi122 ceph-mon[32255]: pgmap v17166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:50 smithi022 ceph-mon[31360]: pgmap v17166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:50 smithi022 ceph-mon[40250]: pgmap v17166: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:52.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:30:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:30:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:52 smithi122 ceph-mon[32255]: pgmap v17167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:52 smithi022 ceph-mon[31360]: pgmap v17167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:52 smithi022 ceph-mon[40250]: pgmap v17167: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:30:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:30:53.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:53.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:53.041 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4285487675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4285487675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:30:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4285487675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:54 smithi122 ceph-mon[32255]: pgmap v17168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:54 smithi022 ceph-mon[31360]: pgmap v17168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:54 smithi022 ceph-mon[40250]: pgmap v17168: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:56 smithi122 ceph-mon[32255]: pgmap v17169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:56 smithi022 ceph-mon[31360]: pgmap v17169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:56 smithi022 ceph-mon[40250]: pgmap v17169: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:30:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:30:58.043 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:30:58.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:30:58.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:30:58.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:30:58.406 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:58 smithi022 ceph-mon[40250]: pgmap v17170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:58.406 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:58 smithi022 ceph-mon[31360]: pgmap v17170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:58 smithi122 ceph-mon[32255]: pgmap v17170: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:30:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:30:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2053511534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:30:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2053511534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:30:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:30:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2053511534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:00] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:31:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:00 smithi122 ceph-mon[32255]: pgmap v17171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:00 smithi022 ceph-mon[31360]: pgmap v17171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:00 smithi022 ceph-mon[40250]: pgmap v17171: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:02.067 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:31:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:31:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:02 smithi122 ceph-mon[32255]: pgmap v17172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:31:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:31:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:02 smithi022 ceph-mon[31360]: pgmap v17172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:02 smithi022 ceph-mon[40250]: pgmap v17172: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:03.407 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:03.407 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:03.408 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:03.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:03.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:03.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:04 smithi122 ceph-mon[32255]: pgmap v17173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1503795162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:04 smithi022 ceph-mon[31360]: pgmap v17173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1503795162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:04 smithi022 ceph-mon[40250]: pgmap v17173: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1503795162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:06 smithi122 ceph-mon[32255]: pgmap v17174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:06 smithi022 ceph-mon[31360]: pgmap v17174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:06 smithi022 ceph-mon[40250]: pgmap v17174: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:31:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:08 smithi122 ceph-mon[32255]: pgmap v17175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:08 smithi022 ceph-mon[31360]: pgmap v17175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:08 smithi022 ceph-mon[40250]: pgmap v17175: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:08.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:09.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:09.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:09.148 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:10] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:31:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:10 smithi122 ceph-mon[32255]: pgmap v17176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1087271124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:10 smithi022 ceph-mon[31360]: pgmap v17176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1087271124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:10 smithi022 ceph-mon[40250]: pgmap v17176: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1087271124' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:12.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:31:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:31:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:12 smithi122 ceph-mon[32255]: pgmap v17177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:31:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:31:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:12 smithi022 ceph-mon[31360]: pgmap v17177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:12 smithi022 ceph-mon[40250]: pgmap v17177: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:14.149 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:14 smithi122 ceph-mon[32255]: pgmap v17178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:14.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:14.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:14 smithi022 ceph-mon[31360]: pgmap v17178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:14.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:14 smithi022 ceph-mon[40250]: pgmap v17178: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2349729895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2349729895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2349729895' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:16 smithi122 ceph-mon[32255]: pgmap v17179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:16 smithi022 ceph-mon[31360]: pgmap v17179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:16 smithi022 ceph-mon[40250]: pgmap v17179: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:31:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:18 smithi122 ceph-mon[32255]: pgmap v17180: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:18 smithi022 ceph-mon[40250]: pgmap v17180: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:18 smithi022 ceph-mon[31360]: pgmap v17180: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:19.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:19.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:19.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:19.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:20 smithi022 ceph-mon[31360]: pgmap v17181: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:20.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2058404642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:20 smithi022 ceph-mon[40250]: pgmap v17181: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2058404642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:20] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:31:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:20 smithi122 ceph-mon[32255]: pgmap v17181: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2058404642' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:22.105 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:31:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:31:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:22 smithi122 ceph-mon[32255]: pgmap v17182: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:31:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:31:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:22 smithi022 ceph-mon[31360]: pgmap v17182: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:22 smithi022 ceph-mon[40250]: pgmap v17182: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:24 smithi122 ceph-mon[32255]: pgmap v17183: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:24 smithi022 ceph-mon[31360]: pgmap v17183: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:24 smithi022 ceph-mon[40250]: pgmap v17183: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:25.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:25.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:25.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:26 smithi122 ceph-mon[32255]: pgmap v17184: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2523496216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:26 smithi022 ceph-mon[31360]: pgmap v17184: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2523496216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:26 smithi022 ceph-mon[40250]: pgmap v17184: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2523496216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:31:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:28 smithi122 ceph-mon[32255]: pgmap v17185: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:28 smithi022 ceph-mon[40250]: pgmap v17185: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:28 smithi022 ceph-mon[31360]: pgmap v17185: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:30.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:30] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:31:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:30 smithi122 ceph-mon[32255]: pgmap v17186: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:30 smithi022 ceph-mon[40250]: pgmap v17186: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:30 smithi022 ceph-mon[31360]: pgmap v17186: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:30.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:30.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:30.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2801510744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2801510744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2801510744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:32.126 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:31:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:31:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:32 smithi122 ceph-mon[32255]: pgmap v17187: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:32.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:31:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:31:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:32 smithi022 ceph-mon[31360]: pgmap v17187: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:32 smithi022 ceph-mon[40250]: pgmap v17187: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:34 smithi122 ceph-mon[32255]: pgmap v17188: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:34 smithi022 ceph-mon[31360]: pgmap v17188: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:34 smithi022 ceph-mon[40250]: pgmap v17188: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:31:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:31:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:31:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:31:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:31:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:31:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:31:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:31:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:31:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:31:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:31:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:31:35.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:36.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:36.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:36.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:36.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[31360]: pgmap v17189: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:31:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:31:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:31:36.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/834694189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[40250]: pgmap v17189: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:31:36.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:31:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:31:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/834694189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:36 smithi122 ceph-mon[32255]: pgmap v17189: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:31:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:31:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:31:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/834694189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:31:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:31:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:31:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:31:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:38 smithi022 ceph-mon[31360]: pgmap v17190: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:38 smithi022 ceph-mon[40250]: pgmap v17190: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:38 smithi122 ceph-mon[32255]: pgmap v17190: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:40] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:31:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:40 smithi122 ceph-mon[32255]: pgmap v17191: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:40 smithi022 ceph-mon[31360]: pgmap v17191: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:40 smithi022 ceph-mon[40250]: pgmap v17191: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:41.029 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:41.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:41.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:41.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:31:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:31:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:42 smithi022 ceph-mon[31360]: pgmap v17192: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1155784695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:42 smithi022 ceph-mon[40250]: pgmap v17192: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1155784695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:42 smithi122 ceph-mon[32255]: pgmap v17192: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1155784695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:31:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:31:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:44 smithi022 ceph-mon[31360]: pgmap v17193: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:44 smithi022 ceph-mon[40250]: pgmap v17193: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:44 smithi122 ceph-mon[32255]: pgmap v17193: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:46.401 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:46 smithi022 ceph-mon[31360]: pgmap v17194: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:46 smithi022 ceph-mon[40250]: pgmap v17194: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:46 smithi122 ceph-mon[32255]: pgmap v17194: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2427033051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:31:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2427033051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2427033051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:48 smithi022 ceph-mon[31360]: pgmap v17195: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:48 smithi022 ceph-mon[40250]: pgmap v17195: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:48 smithi122 ceph-mon[32255]: pgmap v17195: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:50 smithi122 ceph-mon[32255]: pgmap v17196: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:50] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:31:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:50 smithi022 ceph-mon[31360]: pgmap v17196: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:50 smithi022 ceph-mon[40250]: pgmap v17196: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:51.769 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:52.136 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:52.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:52.137 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:31:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:31:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:52 smithi022 ceph-mon[31360]: pgmap v17197: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2493818342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:52 smithi022 ceph-mon[40250]: pgmap v17197: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2493818342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:52 smithi122 ceph-mon[32255]: pgmap v17197: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2493818342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:31:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:31:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:31:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:54 smithi022 ceph-mon[31360]: pgmap v17198: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:54 smithi022 ceph-mon[40250]: pgmap v17198: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:54 smithi122 ceph-mon[32255]: pgmap v17198: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:56 smithi022 ceph-mon[31360]: pgmap v17199: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:56 smithi022 ceph-mon[40250]: pgmap v17199: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:56 smithi122 ceph-mon[32255]: pgmap v17199: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:31:57.138 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:31:57.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:31:57.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:31:57.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:31:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:31:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:58 smithi022 ceph-mon[40250]: pgmap v17200: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:31:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2782314999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:58 smithi022 ceph-mon[31360]: pgmap v17200: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:31:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2782314999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:31:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:58 smithi122 ceph-mon[32255]: pgmap v17200: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:31:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:31:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2782314999' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:00] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:32:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:00 smithi122 ceph-mon[32255]: pgmap v17201: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:00 smithi022 ceph-mon[31360]: pgmap v17201: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:00 smithi022 ceph-mon[40250]: pgmap v17201: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:32:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:32:02.236 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[31781]: 2021-12-16T03:32:02.079+0000 7f774003c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.237 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[31781]: 2021-12-16T03:32:02.175+0000 7f774003c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.237 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[73926]: 2021-12-16T03:32:02.079+0000 7f8e3f3e7700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.238 INFO:journalctl@ceph.osd.1.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[73926]: 2021-12-16T03:32:02.175+0000 7f8e3f3e7700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.238 INFO:journalctl@ceph.rgw.foo.a.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[77089]: 2021-12-16T03:32:02.079+0000 7fa98237b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.239 INFO:journalctl@ceph.rgw.foo.a.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[77089]: 2021-12-16T03:32:02.175+0000 7fa98237b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.239 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[71584]: 2021-12-16T03:32:02.079+0000 7f25a6289700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.239 INFO:journalctl@ceph.osd.0.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[71584]: 2021-12-16T03:32:02.175+0000 7f25a6289700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[31336]: 2021-12-16T03:32:02.079+0000 7f7bd9b74700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[31336]: 2021-12-16T03:32:02.079+0000 7f7bd9b74700 -1 mon.a@0(leader) e3 *** Got Signal Hangup *** 2021-12-16T03:32:02.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[31336]: 2021-12-16T03:32:02.175+0000 7f7bd9b74700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.240 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[31336]: 2021-12-16T03:32:02.175+0000 7f7bd9b74700 -1 mon.a@0(leader) e3 *** Got Signal Hangup *** 2021-12-16T03:32:02.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[40225]: 2021-12-16T03:32:02.079+0000 7f6c6c248700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[40225]: 2021-12-16T03:32:02.079+0000 7f6c6c248700 -1 mon.c@1(peon) e3 *** Got Signal Hangup *** 2021-12-16T03:32:02.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[40225]: 2021-12-16T03:32:02.175+0000 7f6c6c248700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.242 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[40225]: 2021-12-16T03:32:02.175+0000 7f6c6c248700 -1 mon.c@1(peon) e3 *** Got Signal Hangup *** 2021-12-16T03:32:02.242 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[69154]: 2021-12-16T03:32:02.079+0000 7f4f4102a700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.242 INFO:journalctl@ceph.osd.2.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[69154]: 2021-12-16T03:32:02.175+0000 7f4f4102a700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.243 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[66422]: 2021-12-16T03:32:02.079+0000 7fe669d2d700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.243 INFO:journalctl@ceph.osd.3.smithi022.stdout:Dec 16 03:32:02 smithi022 conmon[66422]: 2021-12-16T03:32:02.175+0000 7fe669d2d700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2021-12-16T03:32:02.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:02 smithi022 ceph-mon[31360]: pgmap v17202: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:02.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:02 smithi022 ceph-mon[40250]: pgmap v17202: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:02.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:02 smithi122 ceph-mon[32255]: pgmap v17202: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:32:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:32:02.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:02.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:02.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:03.241 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:03.242 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4202255518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4202255518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4202255518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:04 smithi022 ceph-mon[31360]: pgmap v17203: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:04 smithi022 ceph-mon[40250]: pgmap v17203: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:04 smithi122 ceph-mon[32255]: pgmap v17203: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:06 smithi022 ceph-mon[31360]: pgmap v17204: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:06.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:06 smithi022 ceph-mon[40250]: pgmap v17204: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:06 smithi122 ceph-mon[32255]: pgmap v17204: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:32:07.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:08.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:08.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:08.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:08 smithi022 ceph-mon[31360]: pgmap v17205: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:08.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2508943538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:08 smithi022 ceph-mon[40250]: pgmap v17205: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2508943538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:08 smithi122 ceph-mon[32255]: pgmap v17205: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:32:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2508943538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:10 smithi122 ceph-mon[32255]: pgmap v17206: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:10] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:32:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:10 smithi022 ceph-mon[31360]: pgmap v17206: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:10 smithi022 ceph-mon[40250]: pgmap v17206: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:32:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:32:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:12 smithi022 ceph-mon[40250]: pgmap v17207: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:12 smithi022 ceph-mon[31360]: pgmap v17207: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:12 smithi122 ceph-mon[32255]: pgmap v17207: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:32:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:32:13.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:13.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:13.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:13.615 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:14 smithi022 ceph-mon[31360]: pgmap v17208: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1021188331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:14 smithi022 ceph-mon[40250]: pgmap v17208: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1021188331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:14 smithi122 ceph-mon[32255]: pgmap v17208: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1021188331' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:16 smithi022 ceph-mon[31360]: pgmap v17209: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:16 smithi022 ceph-mon[40250]: pgmap v17209: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:16 smithi122 ceph-mon[32255]: pgmap v17209: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:32:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:18 smithi022 ceph-mon[31360]: pgmap v17210: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:18 smithi022 ceph-mon[40250]: pgmap v17210: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:18.616 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:18 smithi122 ceph-mon[32255]: pgmap v17210: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:18.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:18.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:19.268 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4161287403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4161287403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4161287403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:20 smithi122 ceph-mon[32255]: pgmap v17211: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:20] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:32:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:20 smithi022 ceph-mon[31360]: pgmap v17211: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:20.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:20 smithi022 ceph-mon[40250]: pgmap v17211: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:22.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:32:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:32:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:22 smithi022 ceph-mon[31360]: pgmap v17212: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:22 smithi022 ceph-mon[40250]: pgmap v17212: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:22 smithi122 ceph-mon[32255]: pgmap v17212: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:32:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:32:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:23.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:24.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:24.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:24.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:24 smithi022 ceph-mon[31360]: pgmap v17213: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:24 smithi022 ceph-mon[40250]: pgmap v17213: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:24 smithi122 ceph-mon[32255]: pgmap v17213: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2271310687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2271310687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2271310687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:26 smithi122 ceph-mon[32255]: pgmap v17214: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:26 smithi022 ceph-mon[31360]: pgmap v17214: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:26 smithi022 ceph-mon[40250]: pgmap v17214: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:32:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:28 smithi122 ceph-mon[32255]: pgmap v17215: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:28.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:28 smithi022 ceph-mon[40250]: pgmap v17215: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:28 smithi022 ceph-mon[31360]: pgmap v17215: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:29 smithi122 ceph-mon[32255]: pgmap v17216: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:29.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:29.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:29.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:29.707 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:29 smithi022 ceph-mon[40250]: pgmap v17216: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:29.707 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:29 smithi022 ceph-mon[31360]: pgmap v17216: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2380980218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:30] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:32:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2380980218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2380980218' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:31 smithi122 ceph-mon[32255]: pgmap v17217: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:31.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:31 smithi022 ceph-mon[40250]: pgmap v17217: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:31 smithi022 ceph-mon[31360]: pgmap v17217: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:32:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:32:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:32 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:32:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:32:32.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:32 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:32 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:34.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:33 smithi022 ceph-mon[40250]: pgmap v17218: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:33 smithi022 ceph-mon[31360]: pgmap v17218: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:33 smithi122 ceph-mon[32255]: pgmap v17218: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:34.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:32:35.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:32:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:32:35.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:32:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:32:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:32:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:32:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:32:35.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:35.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:35.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:32:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:32:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:32:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:32:36.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:35 smithi022 ceph-mon[31360]: pgmap v17219: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:36.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3074136888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:36.229 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:35 smithi022 ceph-mon[40250]: pgmap v17219: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:36.229 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3074136888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:36.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:35 smithi122 ceph-mon[32255]: pgmap v17219: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:36.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3074136888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:36.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:32:36.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:32:36.956 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:32:36.957 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:32:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:32:37.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:32:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:32:37.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:32:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:32:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:32:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:36 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:32:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:32:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:32:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:37 smithi022 ceph-mon[31360]: pgmap v17220: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:38.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:37 smithi022 ceph-mon[40250]: pgmap v17220: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:37 smithi122 ceph-mon[32255]: pgmap v17220: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:40.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:40.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:39 smithi022 ceph-mon[31360]: pgmap v17221: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:40.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:39 smithi022 ceph-mon[40250]: pgmap v17221: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:40] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:32:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:39 smithi122 ceph-mon[32255]: pgmap v17221: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:41.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4178283153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:41.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4178283153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4178283153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:42.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:41 smithi122 ceph-mon[32255]: pgmap v17222: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:32:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:32:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:41 smithi022 ceph-mon[31360]: pgmap v17222: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:41 smithi022 ceph-mon[40250]: pgmap v17222: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:32:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:32:43.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:44.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:43 smithi022 ceph-mon[31360]: pgmap v17223: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:44.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:43 smithi022 ceph-mon[40250]: pgmap v17223: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:43 smithi122 ceph-mon[32255]: pgmap v17223: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:45.446 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:45.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:45.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:45.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:45 smithi022 ceph-mon[31360]: pgmap v17224: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:46.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/342606044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:45 smithi022 ceph-mon[40250]: pgmap v17224: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:46.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/342606044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:45 smithi122 ceph-mon[32255]: pgmap v17224: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/342606044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:32:48.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:47 smithi022 ceph-mon[31360]: pgmap v17225: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:47 smithi022 ceph-mon[40250]: pgmap v17225: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:47 smithi122 ceph-mon[32255]: pgmap v17225: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:49 smithi022 ceph-mon[31360]: pgmap v17226: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:49 smithi022 ceph-mon[40250]: pgmap v17226: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:49 smithi122 ceph-mon[32255]: pgmap v17226: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:50] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:32:50.803 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:51.164 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:51.165 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:51.166 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:32:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:32:52.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:51 smithi122 ceph-mon[32255]: pgmap v17227: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:52.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1326597028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:51 smithi022 ceph-mon[31360]: pgmap v17227: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1326597028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:51 smithi022 ceph-mon[40250]: pgmap v17227: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:52.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1326597028' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:32:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:32:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:32:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:53 smithi022 ceph-mon[31360]: pgmap v17228: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:53 smithi022 ceph-mon[40250]: pgmap v17228: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:53 smithi122 ceph-mon[32255]: pgmap v17228: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:56.166 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:32:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:55 smithi022 ceph-mon[31360]: pgmap v17229: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:55 smithi022 ceph-mon[40250]: pgmap v17229: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:55 smithi122 ceph-mon[32255]: pgmap v17229: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:32:56.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:32:56.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:32:56.530 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:32:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2167454560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2167454560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2167454560' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:32:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:32:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:57 smithi022 ceph-mon[31360]: pgmap v17230: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:58.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:57 smithi022 ceph-mon[40250]: pgmap v17230: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:32:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:57 smithi122 ceph-mon[32255]: pgmap v17230: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:32:59 smithi022 ceph-mon[31360]: pgmap v17231: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:32:59 smithi022 ceph-mon[40250]: pgmap v17231: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:00] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:33:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:32:59 smithi122 ceph-mon[32255]: pgmap v17231: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:01.531 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:01.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:01.896 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:01.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:33:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:33:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:02 smithi122 ceph-mon[32255]: pgmap v17232: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4286538927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:01 smithi022 ceph-mon[31360]: pgmap v17232: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4286538927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:02 smithi022 ceph-mon[40250]: pgmap v17232: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4286538927' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:33:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:33:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:03.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:04 smithi022 ceph-mon[31360]: pgmap v17233: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:04 smithi022 ceph-mon[40250]: pgmap v17233: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:04 smithi122 ceph-mon[32255]: pgmap v17233: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:06 smithi022 ceph-mon[31360]: pgmap v17234: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:06 smithi022 ceph-mon[40250]: pgmap v17234: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:06 smithi122 ceph-mon[32255]: pgmap v17234: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:06.898 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:07.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:07.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:07.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:33:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:08 smithi022 ceph-mon[31360]: pgmap v17235: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2872995142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:08 smithi022 ceph-mon[40250]: pgmap v17235: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2872995142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:08 smithi122 ceph-mon[32255]: pgmap v17235: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2872995142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:10 smithi022 ceph-mon[31360]: pgmap v17236: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:10 smithi022 ceph-mon[40250]: pgmap v17236: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:10 smithi122 ceph-mon[32255]: pgmap v17236: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:10] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:33:12.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:12 smithi122 ceph-mon[32255]: pgmap v17237: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:12.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:33:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:33:12.261 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:12.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:12 smithi022 ceph-mon[31360]: pgmap v17237: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:12 smithi022 ceph-mon[40250]: pgmap v17237: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:12.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:12.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:12.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:33:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:33:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2788171311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2788171311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2788171311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:14 smithi022 ceph-mon[31360]: pgmap v17238: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:14 smithi022 ceph-mon[40250]: pgmap v17238: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:14 smithi122 ceph-mon[32255]: pgmap v17238: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:16.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:16 smithi022 ceph-mon[31360]: pgmap v17239: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:16.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:16 smithi022 ceph-mon[40250]: pgmap v17239: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:16 smithi122 ceph-mon[32255]: pgmap v17239: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:17.624 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:33:18.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:18.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:18.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:18 smithi022 ceph-mon[31360]: pgmap v17240: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2615512769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:18 smithi022 ceph-mon[40250]: pgmap v17240: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2615512769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:18 smithi122 ceph-mon[32255]: pgmap v17240: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2615512769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:20 smithi022 ceph-mon[31360]: pgmap v17241: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:20 smithi022 ceph-mon[40250]: pgmap v17241: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:20 smithi122 ceph-mon[32255]: pgmap v17241: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:20] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:33:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:22 smithi122 ceph-mon[32255]: pgmap v17242: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:22.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:33:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:33:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:22 smithi022 ceph-mon[31360]: pgmap v17242: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:22 smithi022 ceph-mon[40250]: pgmap v17242: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:33:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:33:23.010 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:23.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:24 smithi122 ceph-mon[32255]: pgmap v17243: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1524439464' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:24 smithi022 ceph-mon[31360]: pgmap v17243: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1524439464' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:24 smithi022 ceph-mon[40250]: pgmap v17243: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1524439464' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:26 smithi122 ceph-mon[32255]: pgmap v17244: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:26 smithi022 ceph-mon[31360]: pgmap v17244: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:26 smithi022 ceph-mon[40250]: pgmap v17244: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:33:28.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:28.376 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:28 smithi022 ceph-mon[40250]: pgmap v17245: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:28.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:28 smithi022 ceph-mon[31360]: pgmap v17245: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:28 smithi122 ceph-mon[32255]: pgmap v17245: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:28.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:28.733 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:28.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1039407076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:29.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1039407076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1039407076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:30] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:33:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:30 smithi122 ceph-mon[32255]: pgmap v17246: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:30 smithi022 ceph-mon[40250]: pgmap v17246: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:30 smithi022 ceph-mon[31360]: pgmap v17246: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:32 smithi122 ceph-mon[32255]: pgmap v17247: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:33:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:33:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:32 smithi022 ceph-mon[31360]: pgmap v17247: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:32 smithi022 ceph-mon[40250]: pgmap v17247: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:33:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:33:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:33.735 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:34.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:34.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:34.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:34 smithi122 ceph-mon[32255]: pgmap v17248: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:34 smithi022 ceph-mon[31360]: pgmap v17248: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:34 smithi022 ceph-mon[40250]: pgmap v17248: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1842373795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:33:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:33:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:33:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:33:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1842373795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:33:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:33:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:33:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:33:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1842373795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:33:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:33:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:33:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:33:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:36 smithi122 ceph-mon[32255]: pgmap v17249: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:36 smithi022 ceph-mon[31360]: pgmap v17249: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:36 smithi022 ceph-mon[40250]: pgmap v17249: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:33:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:33:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:33:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:33:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:33:37.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:33:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:33:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:33:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:33:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:33:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:38 smithi122 ceph-mon[32255]: pgmap v17250: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:33:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:38 smithi022 ceph-mon[31360]: pgmap v17250: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:33:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:38 smithi022 ceph-mon[40250]: pgmap v17250: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:33:39.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:39.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:39.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:39.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:40] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:33:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:40 smithi122 ceph-mon[32255]: pgmap v17251: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1704835761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:40 smithi022 ceph-mon[40250]: pgmap v17251: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1704835761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:40 smithi022 ceph-mon[31360]: pgmap v17251: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1704835761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:42.124 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:33:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:33:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:42 smithi122 ceph-mon[32255]: pgmap v17252: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:42.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:33:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:33:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:42 smithi022 ceph-mon[31360]: pgmap v17252: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:42 smithi022 ceph-mon[40250]: pgmap v17252: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:43.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:44 smithi122 ceph-mon[32255]: pgmap v17253: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:44.454 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:44.454 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:44 smithi022 ceph-mon[31360]: pgmap v17253: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:44.454 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:44 smithi022 ceph-mon[40250]: pgmap v17253: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:44.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:44.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:44.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2991493902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2991493902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2991493902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:46 smithi122 ceph-mon[32255]: pgmap v17254: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:46 smithi022 ceph-mon[31360]: pgmap v17254: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:46 smithi022 ceph-mon[40250]: pgmap v17254: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:33:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:48 smithi122 ceph-mon[32255]: pgmap v17255: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:48 smithi022 ceph-mon[31360]: pgmap v17255: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:48 smithi022 ceph-mon[40250]: pgmap v17255: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:49.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:50.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:50.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:50.186 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:50 smithi122 ceph-mon[32255]: pgmap v17256: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:50] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:33:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:50 smithi022 ceph-mon[31360]: pgmap v17256: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:50 smithi022 ceph-mon[40250]: pgmap v17256: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3772337251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3772337251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3772337251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:52.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:33:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:33:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:52 smithi122 ceph-mon[32255]: pgmap v17257: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:33:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:33:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:52 smithi022 ceph-mon[40250]: pgmap v17257: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:52 smithi022 ceph-mon[31360]: pgmap v17257: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:33:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:54 smithi122 ceph-mon[32255]: pgmap v17258: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:54 smithi022 ceph-mon[40250]: pgmap v17258: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:54 smithi022 ceph-mon[31360]: pgmap v17258: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:55.187 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:33:55.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:33:55.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:33:55.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:33:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:56 smithi122 ceph-mon[32255]: pgmap v17259: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1840411067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:56 smithi022 ceph-mon[31360]: pgmap v17259: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1840411067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:56 smithi022 ceph-mon[40250]: pgmap v17259: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:33:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1840411067' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:33:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:33:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:33:58 smithi122 ceph-mon[32255]: pgmap v17260: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:33:58 smithi022 ceph-mon[31360]: pgmap v17260: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:33:58.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:33:58 smithi022 ceph-mon[40250]: pgmap v17260: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:00 smithi122 ceph-mon[32255]: pgmap v17261: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:00] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:34:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:00 smithi022 ceph-mon[31360]: pgmap v17261: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:00 smithi022 ceph-mon[40250]: pgmap v17261: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:00.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:00.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:00.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:00.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:01.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2274651404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:01.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2274651404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2274651404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:02.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:34:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:34:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:02 smithi122 ceph-mon[32255]: pgmap v17262: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:34:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:34:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:02 smithi022 ceph-mon[31360]: pgmap v17262: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:02.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:02 smithi022 ceph-mon[40250]: pgmap v17262: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:04 smithi122 ceph-mon[32255]: pgmap v17263: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:04 smithi022 ceph-mon[31360]: pgmap v17263: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:04 smithi022 ceph-mon[40250]: pgmap v17263: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:05.922 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:06.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:06.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:06.295 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:06 smithi122 ceph-mon[32255]: pgmap v17264: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:06 smithi022 ceph-mon[31360]: pgmap v17264: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:06 smithi022 ceph-mon[40250]: pgmap v17264: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1613542206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1613542206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:34:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1613542206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:08 smithi122 ceph-mon[32255]: pgmap v17265: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:08 smithi022 ceph-mon[31360]: pgmap v17265: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:08.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:08 smithi022 ceph-mon[40250]: pgmap v17265: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:10] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:34:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:10 smithi122 ceph-mon[32255]: pgmap v17266: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:10 smithi022 ceph-mon[31360]: pgmap v17266: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:10 smithi022 ceph-mon[40250]: pgmap v17266: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:11.297 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:11.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:11.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:11.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:34:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:34:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:12 smithi022 ceph-mon[31360]: pgmap v17267: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/724960345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:12 smithi022 ceph-mon[40250]: pgmap v17267: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/724960345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:34:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:34:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:12 smithi122 ceph-mon[32255]: pgmap v17267: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/724960345' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:14 smithi022 ceph-mon[31360]: pgmap v17268: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:14.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:14 smithi022 ceph-mon[40250]: pgmap v17268: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:14 smithi122 ceph-mon[32255]: pgmap v17268: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:16 smithi022 ceph-mon[31360]: pgmap v17269: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:16 smithi022 ceph-mon[40250]: pgmap v17269: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:16.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:16 smithi122 ceph-mon[32255]: pgmap v17269: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:17.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1476604637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:17.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1476604637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1476604637' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:34:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:18 smithi022 ceph-mon[31360]: pgmap v17270: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:18 smithi022 ceph-mon[40250]: pgmap v17270: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:18 smithi122 ceph-mon[32255]: pgmap v17270: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:20 smithi122 ceph-mon[32255]: pgmap v17271: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:20] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:34:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:20 smithi022 ceph-mon[31360]: pgmap v17271: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:20 smithi022 ceph-mon[40250]: pgmap v17271: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:22.040 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:22.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:34:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:34:22.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:22.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:22.410 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:22.411 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:22 smithi022 ceph-mon[31360]: pgmap v17272: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:22.411 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:22 smithi022 ceph-mon[40250]: pgmap v17272: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:22 smithi122 ceph-mon[32255]: pgmap v17272: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:34:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:34:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2906910042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2906910042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2906910042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:24 smithi022 ceph-mon[31360]: pgmap v17273: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:24 smithi022 ceph-mon[40250]: pgmap v17273: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:24 smithi122 ceph-mon[32255]: pgmap v17273: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:26 smithi022 ceph-mon[31360]: pgmap v17274: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:26 smithi022 ceph-mon[40250]: pgmap v17274: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:26 smithi122 ceph-mon[32255]: pgmap v17274: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:27.412 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:34:27.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:27.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:27.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:28 smithi022 ceph-mon[31360]: pgmap v17275: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1648260601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:28 smithi022 ceph-mon[40250]: pgmap v17275: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1648260601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:28 smithi122 ceph-mon[32255]: pgmap v17275: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1648260601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:30] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:34:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:30 smithi122 ceph-mon[32255]: pgmap v17276: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:30 smithi022 ceph-mon[31360]: pgmap v17276: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:30 smithi022 ceph-mon[40250]: pgmap v17276: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:34:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:34:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:32 smithi022 ceph-mon[31360]: pgmap v17277: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:32 smithi022 ceph-mon[40250]: pgmap v17277: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:32 smithi122 ceph-mon[32255]: pgmap v17277: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:34:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:34:32.783 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:33.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:33.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:33.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1972943933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1972943933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1972943933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:34 smithi022 ceph-mon[31360]: pgmap v17278: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:34 smithi022 ceph-mon[40250]: pgmap v17278: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:34 smithi122 ceph-mon[32255]: pgmap v17278: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:34:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:34:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:34:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:34:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:34:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:34:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:34:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:34:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:34:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:34:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:34:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:34:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:36 smithi022 ceph-mon[31360]: pgmap v17279: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:36 smithi022 ceph-mon[40250]: pgmap v17279: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:36 smithi122 ceph-mon[32255]: pgmap v17279: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:34:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:34:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:34:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:34:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:34:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:34:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:34:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:34:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:34:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:34:38.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:38.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:38.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:38.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:38.515 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[40250]: pgmap v17280: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:38.515 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[31360]: pgmap v17280: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:38.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:38 smithi122 ceph-mon[32255]: pgmap v17280: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:34:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1629130251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:39.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1629130251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1629130251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:40] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:34:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:40 smithi122 ceph-mon[32255]: pgmap v17281: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:40 smithi022 ceph-mon[31360]: pgmap v17281: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:40 smithi022 ceph-mon[40250]: pgmap v17281: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:34:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:34:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:42 smithi022 ceph-mon[31360]: pgmap v17282: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:42 smithi022 ceph-mon[40250]: pgmap v17282: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:34:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:34:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:42 smithi122 ceph-mon[32255]: pgmap v17282: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:43.516 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:43.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:43.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:43.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:43.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:43.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:44 smithi022 ceph-mon[31360]: pgmap v17283: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/407104118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:44 smithi022 ceph-mon[40250]: pgmap v17283: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/407104118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:44 smithi122 ceph-mon[32255]: pgmap v17283: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/407104118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:46 smithi022 ceph-mon[31360]: pgmap v17284: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:46 smithi022 ceph-mon[40250]: pgmap v17284: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:46 smithi122 ceph-mon[32255]: pgmap v17284: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:34:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:48 smithi022 ceph-mon[31360]: pgmap v17285: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:48 smithi022 ceph-mon[40250]: pgmap v17285: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:48 smithi122 ceph-mon[32255]: pgmap v17285: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:48.885 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:49.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:49.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:49.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/477235727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/477235727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/477235727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:50] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:34:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:50 smithi122 ceph-mon[32255]: pgmap v17286: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:50 smithi022 ceph-mon[31360]: pgmap v17286: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:50 smithi022 ceph-mon[40250]: pgmap v17286: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:52.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:34:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:34:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:52 smithi022 ceph-mon[31360]: pgmap v17287: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:52 smithi022 ceph-mon[40250]: pgmap v17287: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:52 smithi122 ceph-mon[32255]: pgmap v17287: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:34:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:34:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:34:54.261 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:34:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:54 smithi022 ceph-mon[31360]: pgmap v17288: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:54 smithi022 ceph-mon[40250]: pgmap v17288: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:54.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:34:54.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:34:54.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:34:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:54 smithi122 ceph-mon[32255]: pgmap v17288: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2232238490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2232238490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2232238490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:34:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:56 smithi122 ceph-mon[32255]: pgmap v17289: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:56 smithi022 ceph-mon[31360]: pgmap v17289: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:56 smithi022 ceph-mon[40250]: pgmap v17289: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:34:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:34:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:34:58 smithi122 ceph-mon[32255]: pgmap v17290: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:34:58 smithi022 ceph-mon[31360]: pgmap v17290: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:34:58 smithi022 ceph-mon[40250]: pgmap v17290: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:34:59.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:00] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:35:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:00 smithi122 ceph-mon[32255]: pgmap v17291: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/312315983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:00 smithi022 ceph-mon[31360]: pgmap v17291: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/312315983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:00 smithi022 ceph-mon[40250]: pgmap v17291: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/312315983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:02.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:35:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:35:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:02 smithi122 ceph-mon[32255]: pgmap v17292: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:35:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:35:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:02 smithi022 ceph-mon[31360]: pgmap v17292: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:02 smithi022 ceph-mon[40250]: pgmap v17292: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:04 smithi122 ceph-mon[32255]: pgmap v17293: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:04 smithi022 ceph-mon[31360]: pgmap v17293: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:04 smithi022 ceph-mon[40250]: pgmap v17293: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:05.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:05.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:05.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:06 smithi122 ceph-mon[32255]: pgmap v17294: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2143431886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:06 smithi022 ceph-mon[31360]: pgmap v17294: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2143431886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:06 smithi022 ceph-mon[40250]: pgmap v17294: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2143431886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:35:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:08 smithi122 ceph-mon[32255]: pgmap v17295: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:08 smithi022 ceph-mon[31360]: pgmap v17295: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:08 smithi022 ceph-mon[40250]: pgmap v17295: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:09 smithi122 ceph-mon[32255]: pgmap v17296: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:09 smithi022 ceph-mon[31360]: pgmap v17296: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:09 smithi022 ceph-mon[40250]: pgmap v17296: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:10.376 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:10] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:35:10.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:10.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:10.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:11.909 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:11 smithi122 ceph-mon[32255]: pgmap v17297: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:11.909 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2305460903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:11 smithi022 ceph-mon[31360]: pgmap v17297: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2305460903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:11 smithi022 ceph-mon[40250]: pgmap v17297: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2305460903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:12.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:35:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:35:12.606 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:35:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:35:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:12 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:12 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:12 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:13 smithi022 ceph-mon[31360]: pgmap v17298: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:13 smithi022 ceph-mon[40250]: pgmap v17298: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:13 smithi122 ceph-mon[32255]: pgmap v17298: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:15.751 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:15 smithi022 ceph-mon[40250]: pgmap v17299: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:15 smithi022 ceph-mon[31360]: pgmap v17299: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:16.121 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:16.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:16.122 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:15 smithi122 ceph-mon[32255]: pgmap v17299: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:17.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1086969101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:17.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1086969101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1086969101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:35:18.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:17 smithi022 ceph-mon[31360]: pgmap v17300: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:18.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:17 smithi022 ceph-mon[40250]: pgmap v17300: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:17 smithi122 ceph-mon[32255]: pgmap v17300: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:20.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:19 smithi022 ceph-mon[31360]: pgmap v17301: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:20.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:19 smithi022 ceph-mon[40250]: pgmap v17301: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:19 smithi122 ceph-mon[32255]: pgmap v17301: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:20] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:35:21.124 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:21.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:21.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:21.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:22.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:35:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:35:22.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:21 smithi122 ceph-mon[32255]: pgmap v17302: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:22.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3341611913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:22.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:21 smithi022 ceph-mon[31360]: pgmap v17302: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:22.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3341611913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:21 smithi022 ceph-mon[40250]: pgmap v17302: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:22.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3341611913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:35:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:35:23.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:23.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:24.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:24 smithi022 ceph-mon[31360]: pgmap v17303: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:24.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:24 smithi022 ceph-mon[40250]: pgmap v17303: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:24 smithi122 ceph-mon[32255]: pgmap v17303: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:26 smithi022 ceph-mon[31360]: pgmap v17304: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:26 smithi022 ceph-mon[40250]: pgmap v17304: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:26 smithi122 ceph-mon[32255]: pgmap v17304: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:26.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:26.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:26.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:26.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:27.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3680249170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:27.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3680249170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3680249170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:35:28.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:28 smithi022 ceph-mon[31360]: pgmap v17305: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:28.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:28 smithi022 ceph-mon[40250]: pgmap v17305: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:28 smithi122 ceph-mon[32255]: pgmap v17305: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:30 smithi022 ceph-mon[31360]: pgmap v17306: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:30 smithi022 ceph-mon[40250]: pgmap v17306: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:30] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:35:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:30 smithi122 ceph-mon[32255]: pgmap v17306: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:31.866 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:32.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:32 smithi122 ceph-mon[32255]: pgmap v17307: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:32.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:35:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:35:32.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:32.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:32.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:32.237 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:32 smithi022 ceph-mon[31360]: pgmap v17307: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:32.237 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:32 smithi022 ceph-mon[40250]: pgmap v17307: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:35:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:35:33.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:33.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2396814920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:33.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2396814920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2396814920' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:34 smithi022 ceph-mon[31360]: pgmap v17308: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:34 smithi022 ceph-mon[40250]: pgmap v17308: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:34 smithi122 ceph-mon[32255]: pgmap v17308: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:35.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:35:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:35:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:35:35.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:35:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:35:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:35:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:35:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:35:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:35:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:35:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:35:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:35:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:36 smithi122 ceph-mon[32255]: pgmap v17309: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:36 smithi022 ceph-mon[31360]: pgmap v17309: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:36 smithi022 ceph-mon[40250]: pgmap v17309: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:35:37.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:37.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:37.606 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:38.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[31360]: pgmap v17310: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1814552299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:35:38.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:35:38.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:35:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[40250]: pgmap v17310: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:38.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1814552299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:35:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:35:38.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:38 smithi122 ceph-mon[32255]: pgmap v17310: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1814552299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:35:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:35:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:35:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:35:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:39 smithi022 ceph-mon[31360]: pgmap v17311: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:35:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:39 smithi022 ceph-mon[40250]: pgmap v17311: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:35:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:39 smithi122 ceph-mon[32255]: pgmap v17311: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:40] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:35:41.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:41 smithi122 ceph-mon[32255]: pgmap v17312: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:41 smithi022 ceph-mon[31360]: pgmap v17312: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:41 smithi022 ceph-mon[40250]: pgmap v17312: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:42.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:35:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:35:42.608 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:42.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:35:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:35:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:42 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:42.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:42.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:42.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:42.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:42 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:42.977 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:42 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:43 smithi122 ceph-mon[32255]: pgmap v17313: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:43.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1108755434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:43 smithi022 ceph-mon[31360]: pgmap v17313: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:44.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1108755434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:43 smithi022 ceph-mon[40250]: pgmap v17313: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1108755434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:45 smithi122 ceph-mon[32255]: pgmap v17314: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:45 smithi022 ceph-mon[31360]: pgmap v17314: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:45 smithi022 ceph-mon[40250]: pgmap v17314: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:47 smithi022 ceph-mon[31360]: pgmap v17315: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:47.999 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:35:48.000 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:47 smithi022 ceph-mon[40250]: pgmap v17315: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:48.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:48.001 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:47 smithi122 ceph-mon[32255]: pgmap v17315: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:48.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:48.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:48.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:49.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2221435953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:49.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2221435953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2221435953' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:50.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:49 smithi022 ceph-mon[31360]: pgmap v17316: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:49 smithi022 ceph-mon[40250]: pgmap v17316: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:49 smithi122 ceph-mon[32255]: pgmap v17316: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:50] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:35:52.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:51 smithi022 ceph-mon[31360]: pgmap v17317: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:51 smithi022 ceph-mon[40250]: pgmap v17317: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:51 smithi122 ceph-mon[32255]: pgmap v17317: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:52.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:35:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:35:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:35:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:35:53.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:53.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:53.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:35:53.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:53.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:53.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:35:54.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:53 smithi022 ceph-mon[31360]: pgmap v17318: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:54.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1286548979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:53 smithi022 ceph-mon[40250]: pgmap v17318: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:54.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1286548979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:53 smithi122 ceph-mon[32255]: pgmap v17318: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1286548979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:35:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:56 smithi022 ceph-mon[31360]: pgmap v17319: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:56 smithi022 ceph-mon[40250]: pgmap v17319: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:56 smithi122 ceph-mon[32255]: pgmap v17319: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:35:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:35:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:35:58 smithi022 ceph-mon[31360]: pgmap v17320: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:58.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:35:58 smithi022 ceph-mon[40250]: pgmap v17320: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:35:58 smithi122 ceph-mon[32255]: pgmap v17320: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:35:58.719 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:35:59.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:35:59.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:35:59.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:00 smithi022 ceph-mon[31360]: pgmap v17321: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:00.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1282079198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:00 smithi022 ceph-mon[40250]: pgmap v17321: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:00.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1282079198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:00] "GET /metrics HTTP/1.1" 200 202457 "" "Prometheus/2.18.1" 2021-12-16T03:36:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:00 smithi122 ceph-mon[32255]: pgmap v17321: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1282079198' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:02.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:02 smithi022 ceph-mon[31360]: pgmap v17322: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:02.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:02 smithi022 ceph-mon[40250]: pgmap v17322: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:36:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:36:02.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:02 smithi122 ceph-mon[32255]: pgmap v17322: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:36:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:36:03.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:03.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:04.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:04.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:04 smithi022 ceph-mon[31360]: pgmap v17323: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:04.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:04 smithi022 ceph-mon[40250]: pgmap v17323: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:04 smithi122 ceph-mon[32255]: pgmap v17323: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:04.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:04.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:04.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:05.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1993181687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:05.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1993181687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1993181687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:06.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:06 smithi022 ceph-mon[31360]: pgmap v17324: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:06.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:06 smithi022 ceph-mon[40250]: pgmap v17324: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:06 smithi122 ceph-mon[32255]: pgmap v17324: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:36:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:08 smithi022 ceph-mon[31360]: pgmap v17325: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:08 smithi022 ceph-mon[40250]: pgmap v17325: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:08 smithi122 ceph-mon[32255]: pgmap v17325: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:09.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:09.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:09.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:09.831 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:10.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:10 smithi022 ceph-mon[31360]: pgmap v17326: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/825667281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:10 smithi022 ceph-mon[40250]: pgmap v17326: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:10.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/825667281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:10] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:36:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:10 smithi122 ceph-mon[32255]: pgmap v17326: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/825667281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:12.316 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:36:11 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:36:12.317 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:36:12.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:36:12.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:12 smithi122 ceph-mon[32255]: pgmap v17327: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:12 smithi022 ceph-mon[31360]: pgmap v17327: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:12 smithi022 ceph-mon[40250]: pgmap v17327: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:14 smithi122 ceph-mon[32255]: pgmap v17328: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:14 smithi022 ceph-mon[31360]: pgmap v17328: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:14 smithi022 ceph-mon[40250]: pgmap v17328: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:14.833 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:15.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:15.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:16 smithi122 ceph-mon[32255]: pgmap v17329: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3545738827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:16 smithi022 ceph-mon[31360]: pgmap v17329: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3545738827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:16 smithi022 ceph-mon[40250]: pgmap v17329: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3545738827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:36:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:18 smithi122 ceph-mon[32255]: pgmap v17330: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:18 smithi022 ceph-mon[31360]: pgmap v17330: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:18 smithi022 ceph-mon[40250]: pgmap v17330: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:20.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:20 smithi122 ceph-mon[32255]: pgmap v17331: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:20] "GET /metrics HTTP/1.1" 200 202472 "" "Prometheus/2.18.1" 2021-12-16T03:36:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:20 smithi022 ceph-mon[31360]: pgmap v17331: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:20 smithi022 ceph-mon[40250]: pgmap v17331: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:20.575 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4053305363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4053305363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4053305363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:22 smithi122 ceph-mon[32255]: pgmap v17332: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:22.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:36:21 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:36:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:22 smithi022 ceph-mon[31360]: pgmap v17332: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:22 smithi022 ceph-mon[40250]: pgmap v17332: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:36:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:36:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:24 smithi122 ceph-mon[32255]: pgmap v17333: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:24 smithi022 ceph-mon[31360]: pgmap v17333: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:24 smithi022 ceph-mon[40250]: pgmap v17333: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:25.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:25.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:25.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:25.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:26.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:26 smithi022 ceph-mon[31360]: pgmap v17334: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:26.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2334168358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:26 smithi022 ceph-mon[40250]: pgmap v17334: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:26.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2334168358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:26 smithi122 ceph-mon[32255]: pgmap v17334: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2334168358' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:36:28.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:28 smithi122 ceph-mon[32255]: pgmap v17335: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:28 smithi022 ceph-mon[31360]: pgmap v17335: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:28 smithi022 ceph-mon[40250]: pgmap v17335: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:30] "GET /metrics HTTP/1.1" 200 202472 "" "Prometheus/2.18.1" 2021-12-16T03:36:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:30 smithi122 ceph-mon[32255]: pgmap v17336: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:30 smithi022 ceph-mon[31360]: pgmap v17336: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:30 smithi022 ceph-mon[40250]: pgmap v17336: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:30.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:31.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:31.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:31.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:32 smithi122 ceph-mon[32255]: pgmap v17337: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/813257263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:32.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:36:31 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:36:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:32 smithi022 ceph-mon[31360]: pgmap v17337: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/813257263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:32 smithi022 ceph-mon[40250]: pgmap v17337: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/813257263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:36:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:36:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:34 smithi122 ceph-mon[32255]: pgmap v17338: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:34 smithi022 ceph-mon[31360]: pgmap v17338: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:34 smithi022 ceph-mon[40250]: pgmap v17338: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:36:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:36:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:36:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:36:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:36:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:36:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:36:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:36:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:36:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:36:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:36:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:36:36.320 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:36 smithi122 ceph-mon[32255]: pgmap v17339: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:36 smithi022 ceph-mon[31360]: pgmap v17339: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:36 smithi022 ceph-mon[40250]: pgmap v17339: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:36.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:36.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:36.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:37.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4117182005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4117182005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4117182005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:36:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:38 smithi122 ceph-mon[32255]: pgmap v17340: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:38.480 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:38 smithi022 ceph-mon[31360]: pgmap v17340: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:38.481 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:38 smithi022 ceph-mon[40250]: pgmap v17340: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:36:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:36:39.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:36:39.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:36:39.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:36:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:36:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:36:39.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:36:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:36:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:36:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:36:39.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:36:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:40 smithi122 ceph-mon[32255]: pgmap v17341: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:40] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:36:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:40 smithi022 ceph-mon[31360]: pgmap v17341: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:40 smithi022 ceph-mon[40250]: pgmap v17341: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:41.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:42.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:42.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:42 smithi122 ceph-mon[32255]: pgmap v17342: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1232773724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:42.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:36:41 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:36:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:42 smithi022 ceph-mon[31360]: pgmap v17342: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1232773724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:42 smithi022 ceph-mon[40250]: pgmap v17342: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1232773724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:36:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:36:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:44 smithi122 ceph-mon[32255]: pgmap v17343: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:44 smithi022 ceph-mon[31360]: pgmap v17343: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:44 smithi022 ceph-mon[40250]: pgmap v17343: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:46 smithi122 ceph-mon[32255]: pgmap v17344: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:46 smithi022 ceph-mon[31360]: pgmap v17344: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:46 smithi022 ceph-mon[40250]: pgmap v17344: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:47.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:47.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:47.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:47.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:36:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:48 smithi122 ceph-mon[32255]: pgmap v17345: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3792077417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:48 smithi022 ceph-mon[31360]: pgmap v17345: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3792077417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:48 smithi022 ceph-mon[40250]: pgmap v17345: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3792077417' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:50 smithi122 ceph-mon[32255]: pgmap v17346: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:50] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:36:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:50 smithi022 ceph-mon[31360]: pgmap v17346: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:50 smithi022 ceph-mon[40250]: pgmap v17346: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:52 smithi122 ceph-mon[32255]: pgmap v17347: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:52.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:36:51 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:36:52.435 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:52.436 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:52 smithi022 ceph-mon[31360]: pgmap v17347: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:52.436 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:52 smithi022 ceph-mon[40250]: pgmap v17347: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:36:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:36:52.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:52.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:52.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:53.110 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:53.110 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2261398112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2261398112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:36:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2261398112' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:54 smithi122 ceph-mon[32255]: pgmap v17348: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:54 smithi022 ceph-mon[31360]: pgmap v17348: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:54 smithi022 ceph-mon[40250]: pgmap v17348: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:56 smithi122 ceph-mon[32255]: pgmap v17349: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:56 smithi022 ceph-mon[31360]: pgmap v17349: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:56 smithi022 ceph-mon[40250]: pgmap v17349: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:36:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:36:57.804 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:36:58.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:36:58.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:36:58.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:36:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:58 smithi122 ceph-mon[32255]: pgmap v17350: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:58 smithi022 ceph-mon[31360]: pgmap v17350: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:58 smithi022 ceph-mon[40250]: pgmap v17350: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:36:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:36:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3288390836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:36:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3288390836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:36:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:36:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3288390836' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:00] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:37:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:00 smithi122 ceph-mon[32255]: pgmap v17351: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:00 smithi022 ceph-mon[31360]: pgmap v17351: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:00 smithi022 ceph-mon[40250]: pgmap v17351: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:02 smithi122 ceph-mon[32255]: pgmap v17352: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:37:01 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:37:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:02 smithi022 ceph-mon[31360]: pgmap v17352: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:02 smithi022 ceph-mon[40250]: pgmap v17352: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:37:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:37:03.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:03.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:03.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:03.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:04 smithi122 ceph-mon[32255]: pgmap v17353: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1146269190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:04 smithi022 ceph-mon[31360]: pgmap v17353: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1146269190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:04 smithi022 ceph-mon[40250]: pgmap v17353: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1146269190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:06 smithi122 ceph-mon[32255]: pgmap v17354: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:06 smithi022 ceph-mon[31360]: pgmap v17354: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:06 smithi022 ceph-mon[40250]: pgmap v17354: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:37:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:08 smithi122 ceph-mon[32255]: pgmap v17355: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:08 smithi022 ceph-mon[31360]: pgmap v17355: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:08 smithi022 ceph-mon[40250]: pgmap v17355: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:08.551 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:08.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:08.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:08.920 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:09.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1812700003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:09.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1812700003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1812700003' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:10] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:37:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:10 smithi122 ceph-mon[32255]: pgmap v17356: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:10 smithi022 ceph-mon[31360]: pgmap v17356: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:10 smithi022 ceph-mon[40250]: pgmap v17356: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:12.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:12 smithi122 ceph-mon[32255]: pgmap v17357: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:12.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:37:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:37:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:12 smithi022 ceph-mon[31360]: pgmap v17357: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:12 smithi022 ceph-mon[40250]: pgmap v17357: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:37:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:37:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:13.921 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:14.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:14 smithi022 ceph-mon[31360]: pgmap v17358: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:14.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:14 smithi022 ceph-mon[40250]: pgmap v17358: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:14.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:14.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:14.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:14 smithi122 ceph-mon[32255]: pgmap v17358: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3126005768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3126005768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3126005768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:16 smithi122 ceph-mon[32255]: pgmap v17359: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:16 smithi022 ceph-mon[31360]: pgmap v17359: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:16 smithi022 ceph-mon[40250]: pgmap v17359: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:37:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:18 smithi122 ceph-mon[32255]: pgmap v17360: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:18 smithi022 ceph-mon[31360]: pgmap v17360: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:18 smithi022 ceph-mon[40250]: pgmap v17360: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:19.294 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:19.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:19.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:19.663 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:20] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:37:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:20 smithi122 ceph-mon[32255]: pgmap v17361: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3202887076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:20 smithi022 ceph-mon[31360]: pgmap v17361: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3202887076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:20 smithi022 ceph-mon[40250]: pgmap v17361: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3202887076' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:22.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:22 smithi122 ceph-mon[32255]: pgmap v17362: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:22.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:37:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:37:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:22 smithi022 ceph-mon[31360]: pgmap v17362: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:22 smithi022 ceph-mon[40250]: pgmap v17362: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:37:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:37:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:24 smithi122 ceph-mon[32255]: pgmap v17363: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:24 smithi022 ceph-mon[31360]: pgmap v17363: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:24 smithi022 ceph-mon[40250]: pgmap v17363: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:24.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:25.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:25.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:25.034 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3940166849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:25.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3940166849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3940166849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:26 smithi122 ceph-mon[32255]: pgmap v17364: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:26 smithi022 ceph-mon[31360]: pgmap v17364: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:26 smithi022 ceph-mon[40250]: pgmap v17364: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:37:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:28 smithi122 ceph-mon[32255]: pgmap v17365: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:28 smithi022 ceph-mon[31360]: pgmap v17365: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:28 smithi022 ceph-mon[40250]: pgmap v17365: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:30.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:30.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:30.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:30.409 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:30 smithi022 ceph-mon[31360]: pgmap v17366: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:30.409 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:30 smithi022 ceph-mon[40250]: pgmap v17366: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:30] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:37:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:30 smithi122 ceph-mon[32255]: pgmap v17366: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1478608662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1478608662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1478608662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:32.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:32 smithi122 ceph-mon[32255]: pgmap v17367: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:32.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:37:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:37:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:32 smithi022 ceph-mon[31360]: pgmap v17367: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:32 smithi022 ceph-mon[40250]: pgmap v17367: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:37:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:37:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:34 smithi022 ceph-mon[31360]: pgmap v17368: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:34 smithi022 ceph-mon[40250]: pgmap v17368: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:34 smithi122 ceph-mon[32255]: pgmap v17368: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:35.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:37:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:37:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:37:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:37:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:37:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:37:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:37:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:37:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:37:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:37:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:37:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:36 smithi022 ceph-mon[31360]: pgmap v17369: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/721132434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:36 smithi022 ceph-mon[40250]: pgmap v17369: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/721132434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:36 smithi122 ceph-mon[32255]: pgmap v17369: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/721132434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:37:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:38 smithi022 ceph-mon[31360]: pgmap v17370: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:38 smithi022 ceph-mon[40250]: pgmap v17370: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:38 smithi122 ceph-mon[32255]: pgmap v17370: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:39.209 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:37:39.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:37:39.210 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:37:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:37:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:37:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:37:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:37:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:37:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:37:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:40] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:37:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:40 smithi122 ceph-mon[32255]: pgmap v17371: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:37:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:40 smithi022 ceph-mon[31360]: pgmap v17371: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:37:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:40 smithi022 ceph-mon[40250]: pgmap v17371: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:37:40.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:41.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1571783033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1571783033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1571783033' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:42.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:42 smithi122 ceph-mon[32255]: pgmap v17372: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:42.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:37:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:37:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:42 smithi022 ceph-mon[31360]: pgmap v17372: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:42 smithi022 ceph-mon[40250]: pgmap v17372: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:37:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:37:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:44 smithi022 ceph-mon[31360]: pgmap v17373: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:44 smithi022 ceph-mon[40250]: pgmap v17373: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:44 smithi122 ceph-mon[32255]: pgmap v17373: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:46.156 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:46.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:46.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:46.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:46.525 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:46 smithi022 ceph-mon[31360]: pgmap v17374: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:46.526 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:46 smithi022 ceph-mon[40250]: pgmap v17374: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:46 smithi122 ceph-mon[32255]: pgmap v17374: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:37:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2361777513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2361777513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2361777513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:48 smithi022 ceph-mon[31360]: pgmap v17375: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:48 smithi022 ceph-mon[40250]: pgmap v17375: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:48 smithi122 ceph-mon[32255]: pgmap v17375: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:50 smithi122 ceph-mon[32255]: pgmap v17376: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:50] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:37:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:50 smithi022 ceph-mon[31360]: pgmap v17376: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:50 smithi022 ceph-mon[40250]: pgmap v17376: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:51.527 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:52.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:37:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:37:52.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:52 smithi122 ceph-mon[32255]: pgmap v17377: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:52.315 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3209089979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:52 smithi022 ceph-mon[40250]: pgmap v17377: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3209089979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:52 smithi022 ceph-mon[31360]: pgmap v17377: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3209089979' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:37:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:37:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:37:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:54 smithi022 ceph-mon[31360]: pgmap v17378: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:54 smithi022 ceph-mon[40250]: pgmap v17378: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:54 smithi122 ceph-mon[32255]: pgmap v17378: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:56 smithi022 ceph-mon[31360]: pgmap v17379: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:56 smithi022 ceph-mon[40250]: pgmap v17379: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:56 smithi122 ceph-mon[32255]: pgmap v17379: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:37:56.901 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:37:57.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:37:57.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:37:57.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:37:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:37:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:58 smithi022 ceph-mon[31360]: pgmap v17380: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:37:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3998226471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:58 smithi022 ceph-mon[40250]: pgmap v17380: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:37:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3998226471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:37:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:58 smithi122 ceph-mon[32255]: pgmap v17380: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:37:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:37:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3998226471' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:00] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:38:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:00 smithi122 ceph-mon[32255]: pgmap v17381: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:00 smithi022 ceph-mon[31360]: pgmap v17381: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:00 smithi022 ceph-mon[40250]: pgmap v17381: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:02.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:02.314 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:02 smithi122 ceph-mon[32255]: pgmap v17382: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:38:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:38:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:02 smithi022 ceph-mon[31360]: pgmap v17382: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:02 smithi022 ceph-mon[40250]: pgmap v17382: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:02.649 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:02.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:02.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:38:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:38:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2955324323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2955324323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2955324323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:04 smithi022 ceph-mon[31360]: pgmap v17383: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:04 smithi022 ceph-mon[40250]: pgmap v17383: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:04 smithi122 ceph-mon[32255]: pgmap v17383: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:06 smithi022 ceph-mon[31360]: pgmap v17384: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:06 smithi022 ceph-mon[40250]: pgmap v17384: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:06 smithi122 ceph-mon[32255]: pgmap v17384: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:07.651 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:07.652 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:38:08.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:08.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:08.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:08 smithi022 ceph-mon[31360]: pgmap v17385: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/163717969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:08 smithi022 ceph-mon[40250]: pgmap v17385: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/163717969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:08 smithi122 ceph-mon[32255]: pgmap v17385: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/163717969' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:10] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:38:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:10 smithi122 ceph-mon[32255]: pgmap v17386: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:10 smithi022 ceph-mon[31360]: pgmap v17386: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:10 smithi022 ceph-mon[40250]: pgmap v17386: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:38:12.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:12 smithi122 ceph-mon[32255]: pgmap v17387: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:12.318 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:38:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:38:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:12 smithi022 ceph-mon[31360]: pgmap v17387: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:12 smithi022 ceph-mon[40250]: pgmap v17387: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:38:12.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:38:13.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:13.276 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:13.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:13.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:13.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:13.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:14 smithi022 ceph-mon[31360]: pgmap v17388: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1116463386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:14 smithi022 ceph-mon[40250]: pgmap v17388: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1116463386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:14 smithi122 ceph-mon[32255]: pgmap v17388: 132 pgs: 132 active+clean; 582 KiB data, 159 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1116463386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:16 smithi122 ceph-mon[32255]: pgmap v17389: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:16 smithi022 ceph-mon[31360]: pgmap v17389: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:16 smithi022 ceph-mon[40250]: pgmap v17389: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:38:18.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:18 smithi122 ceph-mon[32255]: pgmap v17390: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:18.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:18.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:18.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:18.771 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:18 smithi022 ceph-mon[31360]: pgmap v17390: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:18.771 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:18 smithi022 ceph-mon[40250]: pgmap v17390: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2867098849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2867098849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2867098849' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:20 smithi122 ceph-mon[32255]: pgmap v17391: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:20] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:38:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:20 smithi022 ceph-mon[31360]: pgmap v17391: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:20 smithi022 ceph-mon[40250]: pgmap v17391: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:22.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:38:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:38:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:22 smithi122 ceph-mon[32255]: pgmap v17392: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:22.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:38:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:38:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:22 smithi022 ceph-mon[31360]: pgmap v17392: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:22 smithi022 ceph-mon[40250]: pgmap v17392: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:23.772 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:23.772 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:23.773 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:24.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:24.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:24.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:24 smithi022 ceph-mon[31360]: pgmap v17393: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2906422442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:24 smithi022 ceph-mon[40250]: pgmap v17393: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2906422442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:24 smithi122 ceph-mon[32255]: pgmap v17393: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:38:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2906422442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:26 smithi122 ceph-mon[32255]: pgmap v17394: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:26 smithi022 ceph-mon[31360]: pgmap v17394: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:26 smithi022 ceph-mon[40250]: pgmap v17394: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:38:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:38:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:28 smithi122 ceph-mon[32255]: pgmap v17395: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:28 smithi022 ceph-mon[31360]: pgmap v17395: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:28 smithi022 ceph-mon[40250]: pgmap v17395: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:29.144 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:29.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:29.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:29.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:30] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:38:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:30 smithi122 ceph-mon[32255]: pgmap v17396: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4056956918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:30 smithi022 ceph-mon[31360]: pgmap v17396: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4056956918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:30 smithi022 ceph-mon[40250]: pgmap v17396: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4056956918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:32 smithi122 ceph-mon[32255]: pgmap v17397: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:32.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:38:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:38:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:38:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:38:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:32 smithi022 ceph-mon[31360]: pgmap v17397: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:32 smithi022 ceph-mon[40250]: pgmap v17397: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:34.517 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:34 smithi122 ceph-mon[32255]: pgmap v17398: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:34 smithi022 ceph-mon[31360]: pgmap v17398: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:34 smithi022 ceph-mon[40250]: pgmap v17398: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:34.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:34.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:34.884 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:38:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:38:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:38:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:38:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4180643632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:38:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:38:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:38:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:38:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4180643632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:38:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:38:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:38:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:38:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4180643632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:36 smithi122 ceph-mon[32255]: pgmap v17399: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:36 smithi022 ceph-mon[31360]: pgmap v17399: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:36 smithi022 ceph-mon[40250]: pgmap v17399: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:38:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:38 smithi122 ceph-mon[32255]: pgmap v17400: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:38 smithi022 ceph-mon[31360]: pgmap v17400: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:38 smithi022 ceph-mon[40250]: pgmap v17400: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:39.886 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:40.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:40.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:40.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:40] "GET /metrics HTTP/1.1" 200 202464 "" "Prometheus/2.18.1" 2021-12-16T03:38:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:40 smithi122 ceph-mon[32255]: pgmap v17401: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:38:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:38:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:38:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3831264131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[31360]: pgmap v17401: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:38:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:38:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:38:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3831264131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[40250]: pgmap v17401: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:38:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:38:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:38:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3831264131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:42 smithi122 ceph-mon[32255]: pgmap v17402: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:42.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:38:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:38:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:38:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:38:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:42 smithi022 ceph-mon[31360]: pgmap v17402: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:42 smithi022 ceph-mon[40250]: pgmap v17402: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:43.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:43 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:43.730 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:43.731 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:43 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:43.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:43.731 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:43 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:44 smithi122 ceph-mon[32255]: pgmap v17403: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:44 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:44 smithi022 ceph-mon[31360]: pgmap v17403: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:44 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:44 smithi022 ceph-mon[40250]: pgmap v17403: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:44 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:45.258 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:45.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:45.505 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:45 smithi022 ceph-mon[31360]: pgmap v17404: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:45.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:45.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:45 smithi022 ceph-mon[40250]: pgmap v17404: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:45.671 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:45.672 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:45 smithi122 ceph-mon[32255]: pgmap v17404: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:46.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2437281517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2437281517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:46.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:46.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:46.866 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2437281517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:46.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:46.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.869 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.869 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2021-12-16T03:38:46.869 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:38:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:48 smithi122 ceph-mon[32255]: Adjusting osd_memory_target on smithi022 to 3270M 2021-12-16T03:38:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:48 smithi122 ceph-mon[32255]: pgmap v17405: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[31360]: Adjusting osd_memory_target on smithi022 to 3270M 2021-12-16T03:38:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[31360]: pgmap v17405: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[40250]: Adjusting osd_memory_target on smithi022 to 3270M 2021-12-16T03:38:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[40250]: pgmap v17405: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:38:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:50] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:38:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:50 smithi122 ceph-mon[32255]: pgmap v17406: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:50 smithi022 ceph-mon[31360]: pgmap v17406: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:50 smithi022 ceph-mon[40250]: pgmap v17406: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:50.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:50.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:50.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:50.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:51.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2143662186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:51.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2143662186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2143662186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:52 smithi122 ceph-mon[32255]: pgmap v17407: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:52.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:38:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:38:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:38:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:38:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:52 smithi022 ceph-mon[31360]: pgmap v17407: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:52 smithi022 ceph-mon[40250]: pgmap v17407: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:38:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:54 smithi122 ceph-mon[32255]: pgmap v17408: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:54 smithi022 ceph-mon[31360]: pgmap v17408: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:54 smithi022 ceph-mon[40250]: pgmap v17408: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:38:56.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:56 smithi022 ceph-mon[31360]: pgmap v17409: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:56.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:56 smithi022 ceph-mon[40250]: pgmap v17409: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:56.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:38:56.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:38:56.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:38:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:56 smithi122 ceph-mon[32255]: pgmap v17409: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:38:57.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2703954078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:57.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2703954078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2703954078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:38:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:38:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:38:58 smithi122 ceph-mon[32255]: pgmap v17410: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:38:58 smithi022 ceph-mon[31360]: pgmap v17410: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:38:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:38:58 smithi022 ceph-mon[40250]: pgmap v17410: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:00] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:00 smithi122 ceph-mon[32255]: pgmap v17411: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:00 smithi022 ceph-mon[31360]: pgmap v17411: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:00 smithi022 ceph-mon[40250]: pgmap v17411: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:01.366 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:01.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:01.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:01.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:02 smithi122 ceph-mon[32255]: pgmap v17412: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1459501814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:02.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:39:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:39:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:39:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:39:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:02 smithi022 ceph-mon[31360]: pgmap v17412: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1459501814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:02 smithi022 ceph-mon[40250]: pgmap v17412: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1459501814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:04 smithi122 ceph-mon[32255]: pgmap v17413: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:04 smithi022 ceph-mon[31360]: pgmap v17413: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:04 smithi022 ceph-mon[40250]: pgmap v17413: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:06 smithi122 ceph-mon[32255]: pgmap v17414: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:06 smithi022 ceph-mon[31360]: pgmap v17414: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:06 smithi022 ceph-mon[40250]: pgmap v17414: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:06.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2474067131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:07.438 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2474067131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:07.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2474067131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:39:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:08 smithi122 ceph-mon[32255]: pgmap v17415: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:08 smithi022 ceph-mon[31360]: pgmap v17415: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:08 smithi022 ceph-mon[40250]: pgmap v17415: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:10 smithi122 ceph-mon[32255]: pgmap v17416: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:10] "GET /metrics HTTP/1.1" 200 202459 "" "Prometheus/2.18.1" 2021-12-16T03:39:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:10 smithi022 ceph-mon[31360]: pgmap v17416: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:10 smithi022 ceph-mon[40250]: pgmap v17416: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:12.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:39:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:39:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:12 smithi122 ceph-mon[32255]: pgmap v17417: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:39:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:39:12.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:12.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:12.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:12.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:12 smithi022 ceph-mon[31360]: pgmap v17417: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:12.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:12 smithi022 ceph-mon[40250]: pgmap v17417: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/146643981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/146643981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/146643981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:14 smithi122 ceph-mon[32255]: pgmap v17418: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:14 smithi022 ceph-mon[31360]: pgmap v17418: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:14 smithi022 ceph-mon[40250]: pgmap v17418: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:16 smithi122 ceph-mon[32255]: pgmap v17419: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:16 smithi022 ceph-mon[31360]: pgmap v17419: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:16 smithi022 ceph-mon[40250]: pgmap v17419: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:17.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:39:17.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:17.860 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:17.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:18 smithi122 ceph-mon[32255]: pgmap v17420: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/468221364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:18 smithi022 ceph-mon[31360]: pgmap v17420: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/468221364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:18 smithi022 ceph-mon[40250]: pgmap v17420: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/468221364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:20 smithi122 ceph-mon[32255]: pgmap v17421: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:20] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:39:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:20 smithi022 ceph-mon[31360]: pgmap v17421: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:20 smithi022 ceph-mon[40250]: pgmap v17421: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:22 smithi122 ceph-mon[32255]: pgmap v17422: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:22.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:39:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:39:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:39:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:39:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:22 smithi022 ceph-mon[31360]: pgmap v17422: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:22 smithi022 ceph-mon[40250]: pgmap v17422: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:22.862 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:23.138 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:23.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:23.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:23.235 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:23.236 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:24 smithi122 ceph-mon[32255]: pgmap v17423: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3949335627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:24 smithi022 ceph-mon[31360]: pgmap v17423: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3949335627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:24 smithi022 ceph-mon[40250]: pgmap v17423: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3949335627' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:26 smithi122 ceph-mon[32255]: pgmap v17424: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:26 smithi022 ceph-mon[31360]: pgmap v17424: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:26 smithi022 ceph-mon[40250]: pgmap v17424: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:39:28.237 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:28 smithi122 ceph-mon[32255]: pgmap v17425: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:28 smithi022 ceph-mon[40250]: pgmap v17425: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:28 smithi022 ceph-mon[31360]: pgmap v17425: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2078330199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2078330199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2078330199' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:30] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:39:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:30 smithi122 ceph-mon[32255]: pgmap v17426: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:30 smithi022 ceph-mon[31360]: pgmap v17426: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:30 smithi022 ceph-mon[40250]: pgmap v17426: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:32 smithi122 ceph-mon[32255]: pgmap v17427: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:39:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:39:32.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:39:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:39:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:32 smithi022 ceph-mon[31360]: pgmap v17427: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:32 smithi022 ceph-mon[40250]: pgmap v17427: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:33.610 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:33.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:33.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:33.978 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:34 smithi022 ceph-mon[31360]: pgmap v17428: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/961501595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:34 smithi022 ceph-mon[40250]: pgmap v17428: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/961501595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:34 smithi122 ceph-mon[32255]: pgmap v17428: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/961501595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:39:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:39:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:39:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:39:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:39:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:39:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:39:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:39:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:39:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:39:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:39:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:39:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:36 smithi122 ceph-mon[32255]: pgmap v17429: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:36 smithi022 ceph-mon[31360]: pgmap v17429: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:36 smithi022 ceph-mon[40250]: pgmap v17429: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:39:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:38 smithi122 ceph-mon[32255]: pgmap v17430: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:38 smithi022 ceph-mon[31360]: pgmap v17430: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:38 smithi022 ceph-mon[40250]: pgmap v17430: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:38.980 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:40] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:39:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:40 smithi122 ceph-mon[32255]: pgmap v17431: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/628897910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:40 smithi022 ceph-mon[31360]: pgmap v17431: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/628897910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:40 smithi022 ceph-mon[40250]: pgmap v17431: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/628897910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:42 smithi122 ceph-mon[32255]: pgmap v17432: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:42.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:39:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:39:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:39:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:39:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:42 smithi022 ceph-mon[31360]: pgmap v17432: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:42 smithi022 ceph-mon[40250]: pgmap v17432: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:44 smithi122 ceph-mon[32255]: pgmap v17433: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:44 smithi022 ceph-mon[40250]: pgmap v17433: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:44 smithi022 ceph-mon[31360]: pgmap v17433: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:44.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:44.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:44.723 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/318046987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/318046987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/318046987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:46 smithi022 ceph-mon[31360]: pgmap v17434: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:46 smithi022 ceph-mon[40250]: pgmap v17434: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:46 smithi122 ceph-mon[32255]: pgmap v17434: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:39:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:39:47.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:47 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:39:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:39:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:39:47.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:47 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:39:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:39:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:39:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:47 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:39:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:39:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:48 smithi022 ceph-mon[31360]: pgmap v17435: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:39:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:48 smithi022 ceph-mon[40250]: pgmap v17435: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:39:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:48 smithi122 ceph-mon[32255]: pgmap v17435: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:39:49.727 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:50 smithi122 ceph-mon[32255]: pgmap v17436: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4012639603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:39:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:50 smithi022 ceph-mon[31360]: pgmap v17436: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4012639603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:50 smithi022 ceph-mon[40250]: pgmap v17436: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4012639603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:52.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:39:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:39:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:39:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:39:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:52 smithi122 ceph-mon[32255]: pgmap v17437: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:52 smithi022 ceph-mon[31360]: pgmap v17437: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:52 smithi022 ceph-mon[40250]: pgmap v17437: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:39:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:54 smithi022 ceph-mon[31360]: pgmap v17438: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:54 smithi022 ceph-mon[40250]: pgmap v17438: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:54 smithi122 ceph-mon[32255]: pgmap v17438: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:55.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:39:55.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:39:55.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:39:55.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:39:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:56 smithi022 ceph-mon[31360]: pgmap v17439: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2697159616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:56 smithi022 ceph-mon[40250]: pgmap v17439: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2697159616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:56 smithi122 ceph-mon[32255]: pgmap v17439: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:39:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2697159616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:39:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:39:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:39:58 smithi022 ceph-mon[31360]: pgmap v17440: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:39:58 smithi022 ceph-mon[40250]: pgmap v17440: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:39:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:39:58 smithi122 ceph-mon[32255]: pgmap v17440: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:40:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:00 smithi122 ceph-mon[32255]: pgmap v17441: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:00.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:00 smithi022 ceph-mon[31360]: pgmap v17441: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:00.471 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T03:40:00.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:00 smithi022 ceph-mon[40250]: pgmap v17441: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:00.472 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T03:40:00.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:00.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:00.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:01.218 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3928140536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3928140536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3928140536' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:02 smithi122 ceph-mon[32255]: pgmap v17442: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:02.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:40:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:40:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:40:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:40:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:02 smithi022 ceph-mon[31360]: pgmap v17442: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:02 smithi022 ceph-mon[40250]: pgmap v17442: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:04 smithi022 ceph-mon[31360]: pgmap v17443: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:04 smithi022 ceph-mon[40250]: pgmap v17443: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:04 smithi122 ceph-mon[32255]: pgmap v17443: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:05.838 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:06.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:06.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:06.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:06 smithi022 ceph-mon[31360]: pgmap v17444: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2147177353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:06 smithi022 ceph-mon[40250]: pgmap v17444: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2147177353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:06 smithi122 ceph-mon[32255]: pgmap v17444: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2147177353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:40:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:08 smithi022 ceph-mon[31360]: pgmap v17445: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:08 smithi022 ceph-mon[40250]: pgmap v17445: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:08 smithi122 ceph-mon[32255]: pgmap v17445: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:10 smithi122 ceph-mon[32255]: pgmap v17446: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:10] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:40:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:10 smithi022 ceph-mon[31360]: pgmap v17446: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:10 smithi022 ceph-mon[40250]: pgmap v17446: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:11.209 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:11.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:11.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:11.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:12.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:40:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:40:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:12 smithi122 ceph-mon[32255]: pgmap v17447: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/171175053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:40:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:40:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:12 smithi022 ceph-mon[31360]: pgmap v17447: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/171175053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:12 smithi022 ceph-mon[40250]: pgmap v17447: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/171175053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:14 smithi022 ceph-mon[31360]: pgmap v17448: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:14 smithi022 ceph-mon[40250]: pgmap v17448: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:14 smithi122 ceph-mon[32255]: pgmap v17448: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:16 smithi022 ceph-mon[31360]: pgmap v17449: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:16 smithi022 ceph-mon[40250]: pgmap v17449: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:16.581 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:16 smithi122 ceph-mon[32255]: pgmap v17449: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/19795742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:40:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/19795742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/19795742' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:18 smithi022 ceph-mon[31360]: pgmap v17450: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:18 smithi022 ceph-mon[40250]: pgmap v17450: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:18 smithi122 ceph-mon[32255]: pgmap v17450: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:20 smithi122 ceph-mon[32255]: pgmap v17451: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:20] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:40:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:20 smithi022 ceph-mon[31360]: pgmap v17451: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:20 smithi022 ceph-mon[40250]: pgmap v17451: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:22.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:22.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:22.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:22.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:40:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:40:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:40:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:40:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:22 smithi122 ceph-mon[32255]: pgmap v17452: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:22 smithi022 ceph-mon[31360]: pgmap v17452: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:22 smithi022 ceph-mon[40250]: pgmap v17452: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/209341228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/209341228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/209341228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:24 smithi022 ceph-mon[31360]: pgmap v17453: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:24 smithi022 ceph-mon[40250]: pgmap v17453: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:24 smithi122 ceph-mon[32255]: pgmap v17453: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:26 smithi022 ceph-mon[31360]: pgmap v17454: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:26 smithi022 ceph-mon[40250]: pgmap v17454: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:26 smithi122 ceph-mon[32255]: pgmap v17454: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:27.330 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:27.698 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:27.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:27.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:27.699 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:40:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:28 smithi022 ceph-mon[31360]: pgmap v17455: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2908573332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:28 smithi022 ceph-mon[40250]: pgmap v17455: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2908573332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:28 smithi122 ceph-mon[32255]: pgmap v17455: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:28.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2908573332' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:30] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:40:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:30 smithi122 ceph-mon[32255]: pgmap v17456: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:30 smithi022 ceph-mon[31360]: pgmap v17456: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:30 smithi022 ceph-mon[40250]: pgmap v17456: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:32 smithi122 ceph-mon[32255]: pgmap v17457: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:32.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:40:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:40:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:40:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:40:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:32 smithi022 ceph-mon[31360]: pgmap v17457: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:32 smithi022 ceph-mon[40250]: pgmap v17457: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:32.700 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:33.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:33.075 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:33.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1516868686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1516868686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1516868686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:34 smithi122 ceph-mon[32255]: pgmap v17458: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:34 smithi022 ceph-mon[31360]: pgmap v17458: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:34 smithi022 ceph-mon[40250]: pgmap v17458: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:40:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:40:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:40:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:40:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:40:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:40:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:40:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:40:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:40:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:40:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:40:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:40:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:36 smithi122 ceph-mon[32255]: pgmap v17459: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:36 smithi022 ceph-mon[31360]: pgmap v17459: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:36 smithi022 ceph-mon[40250]: pgmap v17459: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:40:38.076 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:38.451 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:38 smithi022 ceph-mon[31360]: pgmap v17460: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:38.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:38 smithi022 ceph-mon[40250]: pgmap v17460: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:38 smithi122 ceph-mon[32255]: pgmap v17460: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4176143094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4176143094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4176143094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:40 smithi122 ceph-mon[32255]: pgmap v17461: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:40] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:40:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:40 smithi022 ceph-mon[31360]: pgmap v17461: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:40 smithi022 ceph-mon[40250]: pgmap v17461: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:42.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:40:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:40:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:42 smithi122 ceph-mon[32255]: pgmap v17462: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:40:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:40:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:42 smithi022 ceph-mon[31360]: pgmap v17462: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:42 smithi022 ceph-mon[40250]: pgmap v17462: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:43.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:43.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:44 smithi122 ceph-mon[32255]: pgmap v17463: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1820329632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:44 smithi022 ceph-mon[31360]: pgmap v17463: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1820329632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:44 smithi022 ceph-mon[40250]: pgmap v17463: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1820329632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:46 smithi122 ceph-mon[32255]: pgmap v17464: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:46 smithi022 ceph-mon[31360]: pgmap v17464: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:46 smithi022 ceph-mon[40250]: pgmap v17464: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:40:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:48 smithi122 ceph-mon[32255]: pgmap v17465: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:40:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:40:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:40:48.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:40:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[31360]: pgmap v17465: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:40:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:40:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:40:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:40:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[40250]: pgmap v17465: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:40:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:40:48.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:40:48.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:40:48.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:49.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:49.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:49.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/849484617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/849484617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/849484617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:50 smithi122 ceph-mon[32255]: pgmap v17466: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:50] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:40:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:50 smithi022 ceph-mon[31360]: pgmap v17466: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:50 smithi022 ceph-mon[40250]: pgmap v17466: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:52 smithi122 ceph-mon[32255]: pgmap v17467: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:52.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:40:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:40:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:40:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:40:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:52 smithi022 ceph-mon[31360]: pgmap v17467: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:52 smithi022 ceph-mon[40250]: pgmap v17467: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:40:54.195 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:54 smithi022 ceph-mon[31360]: pgmap v17468: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:54 smithi022 ceph-mon[40250]: pgmap v17468: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:54.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:54.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:54.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:40:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:54 smithi122 ceph-mon[32255]: pgmap v17468: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3515454289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3515454289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3515454289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:40:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:56 smithi122 ceph-mon[32255]: pgmap v17469: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:56 smithi022 ceph-mon[31360]: pgmap v17469: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:56 smithi022 ceph-mon[40250]: pgmap v17469: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:40:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:40:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:40:58 smithi122 ceph-mon[32255]: pgmap v17470: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:40:58 smithi022 ceph-mon[31360]: pgmap v17470: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:40:58 smithi022 ceph-mon[40250]: pgmap v17470: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:40:59.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:40:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:40:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:40:59.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:00] "GET /metrics HTTP/1.1" 200 202461 "" "Prometheus/2.18.1" 2021-12-16T03:41:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:00 smithi122 ceph-mon[32255]: pgmap v17471: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3656947294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:00 smithi022 ceph-mon[31360]: pgmap v17471: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3656947294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:00 smithi022 ceph-mon[40250]: pgmap v17471: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3656947294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:41:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:41:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:02 smithi122 ceph-mon[32255]: pgmap v17472: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:41:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:41:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:02 smithi022 ceph-mon[31360]: pgmap v17472: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:02 smithi022 ceph-mon[40250]: pgmap v17472: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:04 smithi122 ceph-mon[32255]: pgmap v17473: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:04 smithi022 ceph-mon[31360]: pgmap v17473: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:04 smithi022 ceph-mon[40250]: pgmap v17473: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:04.943 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:05.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:05.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:05.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2630033418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2630033418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2630033418' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:06 smithi122 ceph-mon[32255]: pgmap v17474: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:06 smithi022 ceph-mon[31360]: pgmap v17474: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:06 smithi022 ceph-mon[40250]: pgmap v17474: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:07 smithi122 ceph-mon[32255]: pgmap v17475: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:07 smithi022 ceph-mon[31360]: pgmap v17475: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:41:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:07 smithi022 ceph-mon[40250]: pgmap v17475: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:10.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:10.317 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:10 smithi022 ceph-mon[31360]: pgmap v17476: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:10.318 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:10 smithi022 ceph-mon[40250]: pgmap v17476: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:10 smithi122 ceph-mon[32255]: pgmap v17476: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:10] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:41:10.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:10.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:10.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:11.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2775763217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2775763217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2775763217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:12 smithi122 ceph-mon[32255]: pgmap v17477: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:12.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:41:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:41:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:41:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:41:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:12 smithi022 ceph-mon[31360]: pgmap v17477: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:12 smithi022 ceph-mon[40250]: pgmap v17477: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:14 smithi122 ceph-mon[32255]: pgmap v17478: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:14 smithi022 ceph-mon[31360]: pgmap v17478: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:14 smithi022 ceph-mon[40250]: pgmap v17478: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:15.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:16.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:16.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:16.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:16 smithi122 ceph-mon[32255]: pgmap v17479: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3772552956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:16 smithi022 ceph-mon[31360]: pgmap v17479: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3772552956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:16 smithi022 ceph-mon[40250]: pgmap v17479: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3772552956' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:41:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:18 smithi122 ceph-mon[32255]: pgmap v17480: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:18 smithi022 ceph-mon[31360]: pgmap v17480: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:18 smithi022 ceph-mon[40250]: pgmap v17480: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:20] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:41:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:20 smithi122 ceph-mon[32255]: pgmap v17481: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:20 smithi022 ceph-mon[31360]: pgmap v17481: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:20 smithi022 ceph-mon[40250]: pgmap v17481: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:21.060 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:22 smithi122 ceph-mon[32255]: pgmap v17482: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/534345453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:22.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:41:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:41:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:41:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:41:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:22 smithi022 ceph-mon[31360]: pgmap v17482: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/534345453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:22 smithi022 ceph-mon[40250]: pgmap v17482: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/534345453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:24 smithi122 ceph-mon[32255]: pgmap v17483: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:24 smithi022 ceph-mon[31360]: pgmap v17483: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:24 smithi022 ceph-mon[40250]: pgmap v17483: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:26.430 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:26.431 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:26 smithi022 ceph-mon[31360]: pgmap v17484: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:26.431 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:26 smithi022 ceph-mon[40250]: pgmap v17484: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:26.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:26 smithi122 ceph-mon[32255]: pgmap v17484: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:26.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:26.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:26.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:27.092 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3949558233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:27.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3949558233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3949558233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:41:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:28 smithi122 ceph-mon[32255]: pgmap v17485: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:28 smithi022 ceph-mon[31360]: pgmap v17485: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:28 smithi022 ceph-mon[40250]: pgmap v17485: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:30] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:41:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:30 smithi122 ceph-mon[32255]: pgmap v17486: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:30 smithi022 ceph-mon[31360]: pgmap v17486: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:30 smithi022 ceph-mon[40250]: pgmap v17486: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:31.801 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:32.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:32.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:32.174 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:32 smithi122 ceph-mon[32255]: pgmap v17487: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:32.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:41:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:41:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:41:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:41:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:32 smithi022 ceph-mon[31360]: pgmap v17487: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:32 smithi022 ceph-mon[40250]: pgmap v17487: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/452166476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/452166476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/452166476' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:34 smithi122 ceph-mon[32255]: pgmap v17488: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:34 smithi022 ceph-mon[31360]: pgmap v17488: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:34 smithi022 ceph-mon[40250]: pgmap v17488: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:41:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:41:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:41:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:41:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:41:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:41:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:41:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:41:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:41:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:41:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:41:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:41:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:36 smithi122 ceph-mon[32255]: pgmap v17489: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:36 smithi022 ceph-mon[31360]: pgmap v17489: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:36 smithi022 ceph-mon[40250]: pgmap v17489: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:37.176 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:41:37.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:37.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:37.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:38 smithi122 ceph-mon[32255]: pgmap v17490: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3213217689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:38 smithi022 ceph-mon[31360]: pgmap v17490: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3213217689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:38 smithi022 ceph-mon[40250]: pgmap v17490: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3213217689' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:40] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:41:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:40 smithi122 ceph-mon[32255]: pgmap v17491: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:40 smithi022 ceph-mon[40250]: pgmap v17491: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:40 smithi022 ceph-mon[31360]: pgmap v17491: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:42 smithi122 ceph-mon[32255]: pgmap v17492: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:42.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:41:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:41:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:41:42.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:41:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:42 smithi022 ceph-mon[31360]: pgmap v17492: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:42 smithi022 ceph-mon[40250]: pgmap v17492: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:42.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:42.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:42.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:42.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:43.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:43.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3633921482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:43.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3633921482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3633921482' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:44 smithi122 ceph-mon[32255]: pgmap v17493: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:44 smithi022 ceph-mon[31360]: pgmap v17493: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:44 smithi022 ceph-mon[40250]: pgmap v17493: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:46 smithi122 ceph-mon[32255]: pgmap v17494: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:46 smithi022 ceph-mon[31360]: pgmap v17494: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:46.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:46 smithi022 ceph-mon[40250]: pgmap v17494: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:41:47.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[40250]: pgmap v17495: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:48.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:41:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:41:48.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:41:48.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[31360]: pgmap v17495: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:41:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:41:48.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:48 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:41:48.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:48.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:48.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:48 smithi122 ceph-mon[32255]: pgmap v17495: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:41:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:41:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:48 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:41:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3228167970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:49.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:41:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3228167970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:41:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3228167970' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:41:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:50 smithi122 ceph-mon[32255]: pgmap v17496: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:50] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:41:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:50 smithi022 ceph-mon[31360]: pgmap v17496: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:50 smithi022 ceph-mon[40250]: pgmap v17496: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:52 smithi122 ceph-mon[32255]: pgmap v17497: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:52.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:41:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:41:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:41:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:41:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:52 smithi022 ceph-mon[31360]: pgmap v17497: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:52 smithi022 ceph-mon[40250]: pgmap v17497: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:53.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:41:53.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:53.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:53.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:54 smithi122 ceph-mon[32255]: pgmap v17498: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3683665621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:54 smithi022 ceph-mon[31360]: pgmap v17498: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3683665621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:54 smithi022 ceph-mon[40250]: pgmap v17498: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3683665621' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:56 smithi122 ceph-mon[32255]: pgmap v17499: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:56 smithi022 ceph-mon[31360]: pgmap v17499: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:56 smithi022 ceph-mon[40250]: pgmap v17499: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:41:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:41:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:58 smithi122 ceph-mon[32255]: pgmap v17500: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:58 smithi022 ceph-mon[31360]: pgmap v17500: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:58 smithi022 ceph-mon[40250]: pgmap v17500: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:41:58.680 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:41:59.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:41:59.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:41:59.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:41:59.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:41:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2372429782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:41:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2372429782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:41:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:41:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2372429782' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:00 smithi122 ceph-mon[32255]: pgmap v17501: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:00] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:42:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:00 smithi022 ceph-mon[31360]: pgmap v17501: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:00 smithi022 ceph-mon[40250]: pgmap v17501: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:02 smithi122 ceph-mon[32255]: pgmap v17502: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:02.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:42:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:42:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:42:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:42:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:02 smithi022 ceph-mon[31360]: pgmap v17502: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:02 smithi022 ceph-mon[40250]: pgmap v17502: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:04.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:04.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:04.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:04.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:04.426 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:04 smithi022 ceph-mon[40250]: pgmap v17503: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:04.427 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:04 smithi022 ceph-mon[31360]: pgmap v17503: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:04.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:04 smithi122 ceph-mon[32255]: pgmap v17503: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:05.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/435561474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/435561474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/435561474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:06 smithi122 ceph-mon[32255]: pgmap v17504: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:06 smithi022 ceph-mon[31360]: pgmap v17504: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:06 smithi022 ceph-mon[40250]: pgmap v17504: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:42:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:08 smithi122 ceph-mon[32255]: pgmap v17505: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:08 smithi022 ceph-mon[31360]: pgmap v17505: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:08 smithi022 ceph-mon[40250]: pgmap v17505: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:09.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:09.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:09.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:10] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:42:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:10 smithi122 ceph-mon[32255]: pgmap v17506: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1678853317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:10 smithi022 ceph-mon[31360]: pgmap v17506: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1678853317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:10 smithi022 ceph-mon[40250]: pgmap v17506: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1678853317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:12.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:42:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:42:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:12 smithi122 ceph-mon[32255]: pgmap v17507: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:42:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:42:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:12 smithi022 ceph-mon[31360]: pgmap v17507: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:12 smithi022 ceph-mon[40250]: pgmap v17507: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:14 smithi122 ceph-mon[32255]: pgmap v17508: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:14 smithi022 ceph-mon[31360]: pgmap v17508: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:14 smithi022 ceph-mon[40250]: pgmap v17508: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:14.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:15.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:15.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:15.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3916306462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3916306462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3916306462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:16 smithi022 ceph-mon[31360]: pgmap v17509: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:16 smithi022 ceph-mon[40250]: pgmap v17509: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:16 smithi122 ceph-mon[32255]: pgmap v17509: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:42:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:18 smithi022 ceph-mon[31360]: pgmap v17510: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:18 smithi022 ceph-mon[40250]: pgmap v17510: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:18 smithi122 ceph-mon[32255]: pgmap v17510: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:20.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:20] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:42:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:20 smithi122 ceph-mon[32255]: pgmap v17511: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:20 smithi022 ceph-mon[31360]: pgmap v17511: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:20 smithi022 ceph-mon[40250]: pgmap v17511: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:20.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:20.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:20.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2992300574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2992300574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2992300574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:22 smithi022 ceph-mon[31360]: pgmap v17512: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:22 smithi022 ceph-mon[40250]: pgmap v17512: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:22 smithi122 ceph-mon[32255]: pgmap v17512: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:22.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:42:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:42:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:42:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:42:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:24 smithi022 ceph-mon[31360]: pgmap v17513: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:24 smithi022 ceph-mon[40250]: pgmap v17513: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:24 smithi122 ceph-mon[32255]: pgmap v17513: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:25.545 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:25.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:25.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:25.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:26 smithi022 ceph-mon[31360]: pgmap v17514: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/976672869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:26 smithi022 ceph-mon[40250]: pgmap v17514: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/976672869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:26 smithi122 ceph-mon[32255]: pgmap v17514: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/976672869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:42:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:28 smithi022 ceph-mon[31360]: pgmap v17515: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:28 smithi022 ceph-mon[40250]: pgmap v17515: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:28 smithi122 ceph-mon[32255]: pgmap v17515: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:30 smithi122 ceph-mon[32255]: pgmap v17516: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:30] "GET /metrics HTTP/1.1" 200 202466 "" "Prometheus/2.18.1" 2021-12-16T03:42:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:30 smithi022 ceph-mon[31360]: pgmap v17516: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:30 smithi022 ceph-mon[40250]: pgmap v17516: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:30.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:31.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:31.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:31.290 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:32 smithi022 ceph-mon[31360]: pgmap v17517: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1305951604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:32 smithi022 ceph-mon[40250]: pgmap v17517: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1305951604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:42:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:42:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:42:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:42:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:32 smithi122 ceph-mon[32255]: pgmap v17517: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1305951604' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:34 smithi022 ceph-mon[31360]: pgmap v17518: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:34 smithi022 ceph-mon[40250]: pgmap v17518: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:34 smithi122 ceph-mon[32255]: pgmap v17518: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:42:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:42:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:42:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:42:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:42:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:42:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:42:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:42:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:42:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:42:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:42:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:42:36.291 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:36 smithi022 ceph-mon[31360]: pgmap v17519: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:36 smithi022 ceph-mon[40250]: pgmap v17519: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:36.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:36.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:36.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:36 smithi122 ceph-mon[32255]: pgmap v17519: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2789892886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:42:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2789892886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2789892886' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:38 smithi022 ceph-mon[31360]: pgmap v17520: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:38 smithi022 ceph-mon[40250]: pgmap v17520: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:38 smithi122 ceph-mon[32255]: pgmap v17520: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:40 smithi122 ceph-mon[32255]: pgmap v17521: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:40] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:42:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:40 smithi022 ceph-mon[31360]: pgmap v17521: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:40 smithi022 ceph-mon[40250]: pgmap v17521: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:41.662 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:42.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:42.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:42.033 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:42 smithi022 ceph-mon[31360]: pgmap v17522: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2772344629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:42 smithi022 ceph-mon[40250]: pgmap v17522: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2772344629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:42 smithi122 ceph-mon[32255]: pgmap v17522: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2772344629' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:42:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:42:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:42:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:42:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:44 smithi022 ceph-mon[31360]: pgmap v17523: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:44 smithi022 ceph-mon[40250]: pgmap v17523: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:44 smithi122 ceph-mon[32255]: pgmap v17523: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:46 smithi022 ceph-mon[31360]: pgmap v17524: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:46 smithi022 ceph-mon[40250]: pgmap v17524: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:46 smithi122 ceph-mon[32255]: pgmap v17524: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:47.034 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:47.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:47.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:47.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:42:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:48 smithi022 ceph-mon[31360]: pgmap v17525: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4046756901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:48 smithi022 ceph-mon[40250]: pgmap v17525: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4046756901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:48.545 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:48 smithi122 ceph-mon[32255]: pgmap v17525: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:48.545 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4046756901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:42:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:42:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:42:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:42:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:42:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:42:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:42:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:49 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:42:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:42:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:42:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:42:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:49 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:42:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:50] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:42:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:50 smithi122 ceph-mon[32255]: pgmap v17526: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:50 smithi022 ceph-mon[31360]: pgmap v17526: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:50 smithi022 ceph-mon[40250]: pgmap v17526: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:52.409 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:52 smithi022 ceph-mon[31360]: pgmap v17527: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:52.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:52 smithi022 ceph-mon[40250]: pgmap v17527: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:52 smithi122 ceph-mon[32255]: pgmap v17527: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:42:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:42:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:42:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:42:52.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:52.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:52.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/941723624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/941723624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:42:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/941723624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:54 smithi022 ceph-mon[31360]: pgmap v17528: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:54 smithi022 ceph-mon[40250]: pgmap v17528: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:54 smithi122 ceph-mon[32255]: pgmap v17528: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:56 smithi022 ceph-mon[31360]: pgmap v17529: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:56 smithi022 ceph-mon[40250]: pgmap v17529: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:56 smithi122 ceph-mon[32255]: pgmap v17529: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:42:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:42:57.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:42:58.150 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:42:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:42:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:42:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:58 smithi022 ceph-mon[31360]: pgmap v17530: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:42:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4123465977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:58 smithi022 ceph-mon[40250]: pgmap v17530: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:42:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4123465977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:42:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:58 smithi122 ceph-mon[32255]: pgmap v17530: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:42:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:42:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4123465977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:00 smithi122 ceph-mon[32255]: pgmap v17531: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:00] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:43:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:00 smithi022 ceph-mon[31360]: pgmap v17531: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:00 smithi022 ceph-mon[40250]: pgmap v17531: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:02 smithi022 ceph-mon[31360]: pgmap v17532: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:02 smithi022 ceph-mon[40250]: pgmap v17532: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:02 smithi122 ceph-mon[32255]: pgmap v17532: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:43:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:43:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:43:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:43:03.152 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:03.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:03.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:03.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:04 smithi122 ceph-mon[32255]: pgmap v17533: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3644787338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:04 smithi022 ceph-mon[31360]: pgmap v17533: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3644787338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:04 smithi022 ceph-mon[40250]: pgmap v17533: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3644787338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:06 smithi122 ceph-mon[32255]: pgmap v17534: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:06 smithi022 ceph-mon[31360]: pgmap v17534: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:06 smithi022 ceph-mon[40250]: pgmap v17534: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:43:08.529 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:08 smithi122 ceph-mon[32255]: pgmap v17535: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:08 smithi022 ceph-mon[31360]: pgmap v17535: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:08 smithi022 ceph-mon[40250]: pgmap v17535: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:08.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4138118553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4138118553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4138118553' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:10] "GET /metrics HTTP/1.1" 200 202465 "" "Prometheus/2.18.1" 2021-12-16T03:43:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:10 smithi122 ceph-mon[32255]: pgmap v17536: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:10 smithi022 ceph-mon[31360]: pgmap v17536: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:10 smithi022 ceph-mon[40250]: pgmap v17536: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:12 smithi122 ceph-mon[32255]: pgmap v17537: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:43:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:43:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:43:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:43:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:12 smithi022 ceph-mon[31360]: pgmap v17537: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:12 smithi022 ceph-mon[40250]: pgmap v17537: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:13.902 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:14 smithi022 ceph-mon[31360]: pgmap v17538: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/342402483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:14 smithi022 ceph-mon[40250]: pgmap v17538: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/342402483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:14 smithi122 ceph-mon[32255]: pgmap v17538: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:43:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/342402483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:16 smithi122 ceph-mon[32255]: pgmap v17539: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:16 smithi022 ceph-mon[31360]: pgmap v17539: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:16 smithi022 ceph-mon[40250]: pgmap v17539: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:43:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:18 smithi122 ceph-mon[32255]: pgmap v17540: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:18 smithi022 ceph-mon[31360]: pgmap v17540: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:18 smithi022 ceph-mon[40250]: pgmap v17540: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:19.272 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:19.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:19.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:19.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:20] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:43:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:20 smithi122 ceph-mon[32255]: pgmap v17541: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/311813997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:20 smithi022 ceph-mon[31360]: pgmap v17541: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/311813997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:20 smithi022 ceph-mon[40250]: pgmap v17541: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/311813997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:22 smithi122 ceph-mon[32255]: pgmap v17542: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:22.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:43:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:43:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:43:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:43:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:22 smithi022 ceph-mon[31360]: pgmap v17542: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:22 smithi022 ceph-mon[40250]: pgmap v17542: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:24.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:24.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:24 smithi022 ceph-mon[31360]: pgmap v17543: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:24.650 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:24 smithi022 ceph-mon[40250]: pgmap v17543: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:24 smithi122 ceph-mon[32255]: pgmap v17543: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:25.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:25.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:25.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3651058068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3651058068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3651058068' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:26 smithi122 ceph-mon[32255]: pgmap v17544: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:26 smithi022 ceph-mon[31360]: pgmap v17544: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:26 smithi022 ceph-mon[40250]: pgmap v17544: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:43:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:28 smithi122 ceph-mon[32255]: pgmap v17545: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:28 smithi022 ceph-mon[31360]: pgmap v17545: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:28 smithi022 ceph-mon[40250]: pgmap v17545: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:30.023 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:30.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:30.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:30.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:30] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:43:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:30 smithi122 ceph-mon[32255]: pgmap v17546: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:30 smithi022 ceph-mon[31360]: pgmap v17546: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:30 smithi022 ceph-mon[40250]: pgmap v17546: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1191819106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1191819106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1191819106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:32 smithi122 ceph-mon[32255]: pgmap v17547: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:43:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:43:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:43:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:43:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:32 smithi022 ceph-mon[31360]: pgmap v17547: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:32 smithi022 ceph-mon[40250]: pgmap v17547: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:34 smithi122 ceph-mon[32255]: pgmap v17548: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:34 smithi022 ceph-mon[31360]: pgmap v17548: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:34 smithi022 ceph-mon[40250]: pgmap v17548: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:35.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:43:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:43:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:43:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:43:35.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:35.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:35.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:35.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:43:35.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:43:35.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:43:35.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:43:35.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:43:35.763 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:43:35.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:43:35.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:43:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:36 smithi122 ceph-mon[32255]: pgmap v17549: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1263211877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:36 smithi022 ceph-mon[31360]: pgmap v17549: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1263211877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:36 smithi022 ceph-mon[40250]: pgmap v17549: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1263211877' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:43:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:38 smithi122 ceph-mon[32255]: pgmap v17550: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:38 smithi022 ceph-mon[31360]: pgmap v17550: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:38 smithi022 ceph-mon[40250]: pgmap v17550: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:40] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:43:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:40 smithi122 ceph-mon[32255]: pgmap v17551: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:40.763 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:40.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:40 smithi022 ceph-mon[31360]: pgmap v17551: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:40.764 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:40 smithi022 ceph-mon[40250]: pgmap v17551: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:41.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:41.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:41.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/580512120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/580512120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/580512120' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:42.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:43:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:43:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:42 smithi122 ceph-mon[32255]: pgmap v17552: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:43:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:43:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:42 smithi022 ceph-mon[31360]: pgmap v17552: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:42 smithi022 ceph-mon[40250]: pgmap v17552: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:44 smithi122 ceph-mon[32255]: pgmap v17553: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:44 smithi022 ceph-mon[31360]: pgmap v17553: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:44 smithi022 ceph-mon[40250]: pgmap v17553: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:46.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:46.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:46.510 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:46.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:46.511 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:46 smithi022 ceph-mon[31360]: pgmap v17554: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:46.512 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:46 smithi022 ceph-mon[40250]: pgmap v17554: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:46 smithi122 ceph-mon[32255]: pgmap v17554: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4214247964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:43:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4214247964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:47.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4214247964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:48 smithi122 ceph-mon[32255]: pgmap v17555: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:48 smithi022 ceph-mon[31360]: pgmap v17555: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:48 smithi022 ceph-mon[40250]: pgmap v17555: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:49 smithi122 ceph-mon[32255]: pgmap v17556: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:43:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:43:49.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:49 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:43:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[31360]: pgmap v17556: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:43:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:43:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:43:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[40250]: pgmap v17556: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:43:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:43:49.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:49 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:43:50.431 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:50] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:43:50.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:43:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:43:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:43:51.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:51 smithi022 ceph-mon[31360]: pgmap v17557: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:51 smithi022 ceph-mon[40250]: pgmap v17557: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:51.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:51.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:51.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:51 smithi122 ceph-mon[32255]: pgmap v17557: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:43:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:43:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1535695093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:52 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:43:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:43:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1535695093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:52 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1535695093' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:52 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:43:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:54 smithi122 ceph-mon[32255]: pgmap v17558: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:54 smithi022 ceph-mon[31360]: pgmap v17558: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:54 smithi022 ceph-mon[40250]: pgmap v17558: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:56 smithi122 ceph-mon[32255]: pgmap v17559: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:56 smithi022 ceph-mon[31360]: pgmap v17559: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:56 smithi022 ceph-mon[40250]: pgmap v17559: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:43:57.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:43:57.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:43:57.259 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:43:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:43:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:58 smithi122 ceph-mon[32255]: pgmap v17560: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:43:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2038091531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:58 smithi022 ceph-mon[31360]: pgmap v17560: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:43:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2038091531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:43:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:58 smithi022 ceph-mon[40250]: pgmap v17560: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:43:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:43:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2038091531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:00 smithi122 ceph-mon[32255]: pgmap v17561: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:00] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:44:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:00 smithi022 ceph-mon[31360]: pgmap v17561: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:00 smithi022 ceph-mon[40250]: pgmap v17561: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:02.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:02 smithi122 ceph-mon[32255]: pgmap v17562: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:02.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:44:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:44:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:44:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:44:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:02 smithi022 ceph-mon[31360]: pgmap v17562: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:02 smithi022 ceph-mon[40250]: pgmap v17562: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:02.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:02.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:02.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3234929072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3234929072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3234929072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:04 smithi122 ceph-mon[32255]: pgmap v17563: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:04 smithi022 ceph-mon[31360]: pgmap v17563: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:04 smithi022 ceph-mon[40250]: pgmap v17563: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:06 smithi122 ceph-mon[32255]: pgmap v17564: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:06 smithi022 ceph-mon[31360]: pgmap v17564: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:06 smithi022 ceph-mon[40250]: pgmap v17564: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:07.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:44:08.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:08.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:08.007 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:08 smithi022 ceph-mon[31360]: pgmap v17565: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:08.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3924500656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:08 smithi022 ceph-mon[40250]: pgmap v17565: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:08.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3924500656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:08 smithi122 ceph-mon[32255]: pgmap v17565: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3924500656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:10] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:44:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:10 smithi122 ceph-mon[32255]: pgmap v17566: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:10 smithi022 ceph-mon[31360]: pgmap v17566: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:10 smithi022 ceph-mon[40250]: pgmap v17566: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:12.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:44:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:44:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:12 smithi122 ceph-mon[32255]: pgmap v17567: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:44:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:44:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:12 smithi022 ceph-mon[31360]: pgmap v17567: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:12 smithi022 ceph-mon[40250]: pgmap v17567: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:13 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:13.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:13 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:13.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:13.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:13.390 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:13 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:14 smithi122 ceph-mon[32255]: pgmap v17568: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2636670483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:14 smithi022 ceph-mon[31360]: pgmap v17568: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2636670483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:14 smithi022 ceph-mon[40250]: pgmap v17568: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2636670483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:16 smithi122 ceph-mon[32255]: pgmap v17569: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:16 smithi022 ceph-mon[31360]: pgmap v17569: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:16 smithi022 ceph-mon[40250]: pgmap v17569: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:44:18.391 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:18.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:18 smithi022 ceph-mon[31360]: pgmap v17570: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:18.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:18 smithi022 ceph-mon[40250]: pgmap v17570: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:18 smithi122 ceph-mon[32255]: pgmap v17570: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/442099822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/442099822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/442099822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:20 smithi122 ceph-mon[32255]: pgmap v17571: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:20] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:44:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:20 smithi022 ceph-mon[31360]: pgmap v17571: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:20 smithi022 ceph-mon[40250]: pgmap v17571: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:22 smithi122 ceph-mon[32255]: pgmap v17572: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:22.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:44:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:44:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:44:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:44:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:22 smithi022 ceph-mon[31360]: pgmap v17572: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:22 smithi022 ceph-mon[40250]: pgmap v17572: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:23 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:23 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:23 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:23.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:24.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:24.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:24.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:24 smithi122 ceph-mon[32255]: pgmap v17573: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3212345672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:24 smithi022 ceph-mon[31360]: pgmap v17573: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3212345672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:24 smithi022 ceph-mon[40250]: pgmap v17573: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3212345672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:26 smithi122 ceph-mon[32255]: pgmap v17574: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:26 smithi022 ceph-mon[31360]: pgmap v17574: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:26 smithi022 ceph-mon[40250]: pgmap v17574: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:44:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:28 smithi122 ceph-mon[32255]: pgmap v17575: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:28 smithi022 ceph-mon[31360]: pgmap v17575: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:28 smithi022 ceph-mon[40250]: pgmap v17575: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:29.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:29.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:29.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:29.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:30] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:44:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:30 smithi122 ceph-mon[32255]: pgmap v17576: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/396535436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:30 smithi022 ceph-mon[31360]: pgmap v17576: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/396535436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:30 smithi022 ceph-mon[40250]: pgmap v17576: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/396535436' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:32 smithi122 ceph-mon[32255]: pgmap v17577: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:32.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:44:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:44:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:44:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:44:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:32 smithi022 ceph-mon[31360]: pgmap v17577: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:32 smithi022 ceph-mon[40250]: pgmap v17577: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:33 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:33 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:33 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:34.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:34 smithi122 ceph-mon[32255]: pgmap v17578: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:34.509 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:34.510 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:34 smithi022 ceph-mon[31360]: pgmap v17578: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:34.511 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:34 smithi022 ceph-mon[40250]: pgmap v17578: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:34.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:34.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:34.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:44:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:44:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:44:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:44:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/37299317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:44:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:44:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:44:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:44:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/37299317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:44:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:44:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:44:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:44:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/37299317' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:36 smithi122 ceph-mon[32255]: pgmap v17579: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:36 smithi022 ceph-mon[31360]: pgmap v17579: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:36 smithi022 ceph-mon[40250]: pgmap v17579: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:44:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:38 smithi122 ceph-mon[32255]: pgmap v17580: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:38 smithi022 ceph-mon[31360]: pgmap v17580: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:38 smithi022 ceph-mon[40250]: pgmap v17580: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:39.881 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:40.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:40.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:40.252 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:40 smithi122 ceph-mon[32255]: pgmap v17581: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:40] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:44:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:40 smithi022 ceph-mon[31360]: pgmap v17581: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:40.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:40 smithi022 ceph-mon[40250]: pgmap v17581: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1534283259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1534283259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1534283259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:42 smithi122 ceph-mon[32255]: pgmap v17582: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:42.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:44:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:44:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:44:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:44:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:42 smithi022 ceph-mon[31360]: pgmap v17582: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:42 smithi022 ceph-mon[40250]: pgmap v17582: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:43 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:43 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:43 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:44 smithi122 ceph-mon[32255]: pgmap v17583: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:44 smithi022 ceph-mon[31360]: pgmap v17583: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:44 smithi022 ceph-mon[40250]: pgmap v17583: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:45.254 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:45.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:45.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:45.625 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:46 smithi022 ceph-mon[31360]: pgmap v17584: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4234517663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:46 smithi022 ceph-mon[40250]: pgmap v17584: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4234517663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:46 smithi122 ceph-mon[32255]: pgmap v17584: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4234517663' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:44:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:48 smithi022 ceph-mon[31360]: pgmap v17585: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:48 smithi022 ceph-mon[40250]: pgmap v17585: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:48 smithi122 ceph-mon[32255]: pgmap v17585: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:50] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:44:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:50 smithi122 ceph-mon[32255]: pgmap v17586: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:44:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:44:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:44:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[40250]: pgmap v17586: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:44:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:44:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:44:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[31360]: pgmap v17586: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:44:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:44:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:44:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:50.627 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:50.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:50.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:50.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1904866085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1904866085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:44:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1904866085' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:52 smithi122 ceph-mon[32255]: pgmap v17587: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:52.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:44:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:44:52.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:44:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:44:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:52 smithi022 ceph-mon[31360]: pgmap v17587: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:52 smithi022 ceph-mon[40250]: pgmap v17587: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:53 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:53 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:53 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:44:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:54 smithi122 ceph-mon[32255]: pgmap v17588: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:54 smithi022 ceph-mon[31360]: pgmap v17588: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:54 smithi022 ceph-mon[40250]: pgmap v17588: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:55.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:44:56.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:44:56.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:44:56.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:44:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:56 smithi122 ceph-mon[32255]: pgmap v17589: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:56 smithi022 ceph-mon[31360]: pgmap v17589: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:56 smithi022 ceph-mon[40250]: pgmap v17589: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:44:57.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1037186520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1037186520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:44:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1037186520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:44:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:44:58 smithi122 ceph-mon[32255]: pgmap v17590: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:44:58 smithi022 ceph-mon[31360]: pgmap v17590: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:44:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:44:58 smithi022 ceph-mon[40250]: pgmap v17590: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:00] "GET /metrics HTTP/1.1" 200 202471 "" "Prometheus/2.18.1" 2021-12-16T03:45:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:00 smithi122 ceph-mon[32255]: pgmap v17591: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:00 smithi022 ceph-mon[31360]: pgmap v17591: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:00 smithi022 ceph-mon[40250]: pgmap v17591: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:01.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:01.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:01.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:01.732 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:45:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:45:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:45:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:45:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:02 smithi122 ceph-mon[32255]: pgmap v17592: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/975680833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:02 smithi022 ceph-mon[31360]: pgmap v17592: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/975680833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:02 smithi022 ceph-mon[40250]: pgmap v17592: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/975680833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:03 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:03 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:03 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:04.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:04 smithi122 ceph-mon[32255]: pgmap v17593: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:04 smithi022 ceph-mon[31360]: pgmap v17593: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:04 smithi022 ceph-mon[40250]: pgmap v17593: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:06 smithi122 ceph-mon[32255]: pgmap v17594: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:06.733 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:06.734 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:06 smithi022 ceph-mon[31360]: pgmap v17594: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:06.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:06 smithi022 ceph-mon[40250]: pgmap v17594: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:07.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:07.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:07.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:07.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/762951521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:07.392 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/762951521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/762951521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:45:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:08 smithi122 ceph-mon[32255]: pgmap v17595: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:08.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:08 smithi022 ceph-mon[40250]: pgmap v17595: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:08 smithi022 ceph-mon[31360]: pgmap v17595: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:10 smithi122 ceph-mon[32255]: pgmap v17596: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:10] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:45:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:10 smithi022 ceph-mon[40250]: pgmap v17596: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:10 smithi022 ceph-mon[31360]: pgmap v17596: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:12.105 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:12.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:12.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:12.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:12.479 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:12 smithi022 ceph-mon[31360]: pgmap v17597: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:12.479 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:12 smithi022 ceph-mon[40250]: pgmap v17597: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:12 smithi122 ceph-mon[32255]: pgmap v17597: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:45:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:45:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:45:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:45:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3448722944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3448722944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3448722944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:14 smithi122 ceph-mon[32255]: pgmap v17598: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:14 smithi022 ceph-mon[31360]: pgmap v17598: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:14 smithi022 ceph-mon[40250]: pgmap v17598: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:16.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:16 smithi122 ceph-mon[32255]: pgmap v17599: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:16 smithi022 ceph-mon[31360]: pgmap v17599: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:16 smithi022 ceph-mon[40250]: pgmap v17599: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:17.480 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:45:17.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:17.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:17.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:18 smithi122 ceph-mon[32255]: pgmap v17600: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2051871365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:18 smithi022 ceph-mon[31360]: pgmap v17600: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2051871365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:18 smithi022 ceph-mon[40250]: pgmap v17600: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2051871365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:20 smithi122 ceph-mon[32255]: pgmap v17601: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:20] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:45:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:20 smithi022 ceph-mon[31360]: pgmap v17601: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:20 smithi022 ceph-mon[40250]: pgmap v17601: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:22 smithi122 ceph-mon[32255]: pgmap v17602: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:22.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:45:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:45:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:45:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:45:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:22 smithi022 ceph-mon[31360]: pgmap v17602: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:22 smithi022 ceph-mon[40250]: pgmap v17602: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:22.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:23.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:23.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:23.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:24 smithi122 ceph-mon[32255]: pgmap v17603: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1737889365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:24 smithi022 ceph-mon[31360]: pgmap v17603: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1737889365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:24 smithi022 ceph-mon[40250]: pgmap v17603: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1737889365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:26 smithi122 ceph-mon[32255]: pgmap v17604: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:26 smithi022 ceph-mon[31360]: pgmap v17604: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:26 smithi022 ceph-mon[40250]: pgmap v17604: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:45:28.230 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:28 smithi122 ceph-mon[32255]: pgmap v17605: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:28 smithi022 ceph-mon[31360]: pgmap v17605: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:28 smithi022 ceph-mon[40250]: pgmap v17605: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:28.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:28.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:28.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3517807660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3517807660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3517807660' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:30 smithi122 ceph-mon[32255]: pgmap v17606: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:30] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:45:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:30 smithi022 ceph-mon[31360]: pgmap v17606: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:30 smithi022 ceph-mon[40250]: pgmap v17606: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:32.408 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:32 smithi122 ceph-mon[32255]: pgmap v17607: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:32.409 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:45:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:45:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:32 smithi022 ceph-mon[31360]: pgmap v17607: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:32 smithi022 ceph-mon[40250]: pgmap v17607: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:45:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:45:33.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:33.972 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:33.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:33.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:34.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:34 smithi022 ceph-mon[31360]: pgmap v17608: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:34.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1151052098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:34.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:34 smithi022 ceph-mon[40250]: pgmap v17608: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:34.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1151052098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:34 smithi122 ceph-mon[32255]: pgmap v17608: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1151052098' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:45:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:45:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:45:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:45:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:45:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:45:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:45:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:45:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:45:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:45:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:45:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:45:36.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:36 smithi122 ceph-mon[32255]: pgmap v17609: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:36 smithi022 ceph-mon[31360]: pgmap v17609: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:36 smithi022 ceph-mon[40250]: pgmap v17609: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:45:38.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:38 smithi122 ceph-mon[32255]: pgmap v17610: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:38 smithi022 ceph-mon[31360]: pgmap v17610: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:38 smithi022 ceph-mon[40250]: pgmap v17610: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:38.974 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:39.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:39.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:39.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:40] "GET /metrics HTTP/1.1" 200 202469 "" "Prometheus/2.18.1" 2021-12-16T03:45:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:40 smithi122 ceph-mon[32255]: pgmap v17611: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3526271675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:40 smithi022 ceph-mon[31360]: pgmap v17611: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3526271675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:40 smithi022 ceph-mon[40250]: pgmap v17611: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3526271675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:42.409 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:42 smithi122 ceph-mon[32255]: pgmap v17612: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:42.410 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:45:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:45:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:42 smithi022 ceph-mon[31360]: pgmap v17612: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:42.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:42 smithi022 ceph-mon[40250]: pgmap v17612: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:42.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:45:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:45:44.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:44.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:44 smithi122 ceph-mon[32255]: pgmap v17613: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:44 smithi022 ceph-mon[31360]: pgmap v17613: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:44 smithi022 ceph-mon[40250]: pgmap v17613: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:44.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:44.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:44.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3307713682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3307713682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3307713682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:46.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:46 smithi122 ceph-mon[32255]: pgmap v17614: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:46 smithi022 ceph-mon[31360]: pgmap v17614: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:46 smithi022 ceph-mon[40250]: pgmap v17614: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:45:48.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:48 smithi122 ceph-mon[32255]: pgmap v17615: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:48 smithi022 ceph-mon[31360]: pgmap v17615: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:48 smithi022 ceph-mon[40250]: pgmap v17615: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:49.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:50.089 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:50.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:50.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:50.354 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:50 smithi022 ceph-mon[31360]: pgmap v17616: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:50.354 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/628509171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:50.355 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:50 smithi022 ceph-mon[40250]: pgmap v17616: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:50.355 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/628509171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:50 smithi122 ceph-mon[32255]: pgmap v17616: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/628509171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:50] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:45:51.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:45:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:45:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:45:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:45:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:45:51.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:45:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:45:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:45:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:45:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:45:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:45:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:45:52.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:45:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:45:52.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:52 smithi122 ceph-mon[32255]: pgmap v17617: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:52.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:45:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:45:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:52 smithi022 ceph-mon[31360]: pgmap v17617: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:52 smithi022 ceph-mon[40250]: pgmap v17617: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:54 smithi122 ceph-mon[32255]: pgmap v17618: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:54 smithi022 ceph-mon[31360]: pgmap v17618: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:45:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:54 smithi022 ceph-mon[40250]: pgmap v17618: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:55.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:45:55.456 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:45:55.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:45:55.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:45:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:56 smithi122 ceph-mon[32255]: pgmap v17619: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1425256398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:56 smithi022 ceph-mon[31360]: pgmap v17619: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1425256398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:56 smithi022 ceph-mon[40250]: pgmap v17619: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:45:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1425256398' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:45:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:45:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:45:58 smithi122 ceph-mon[32255]: pgmap v17620: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:45:58 smithi022 ceph-mon[31360]: pgmap v17620: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:45:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:45:58 smithi022 ceph-mon[40250]: pgmap v17620: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:00 smithi122 ceph-mon[32255]: pgmap v17621: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:00] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:46:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:00.458 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:00 smithi022 ceph-mon[31360]: pgmap v17621: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:00.459 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:00 smithi022 ceph-mon[40250]: pgmap v17621: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:00.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:00.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:00.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:01.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3856508385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3856508385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3856508385' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:02 smithi122 ceph-mon[32255]: pgmap v17622: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:02.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:46:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:46:02.440 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:46:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:46:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:02 smithi022 ceph-mon[31360]: pgmap v17622: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:02 smithi022 ceph-mon[40250]: pgmap v17622: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:04 smithi122 ceph-mon[32255]: pgmap v17623: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:04 smithi022 ceph-mon[31360]: pgmap v17623: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:04 smithi022 ceph-mon[40250]: pgmap v17623: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:05.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:06.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:06.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:06.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:06 smithi122 ceph-mon[32255]: pgmap v17624: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:06 smithi022 ceph-mon[31360]: pgmap v17624: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:06 smithi022 ceph-mon[40250]: pgmap v17624: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4095667394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:46:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4095667394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4095667394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:08 smithi022 ceph-mon[31360]: pgmap v17625: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:08 smithi022 ceph-mon[40250]: pgmap v17625: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:08 smithi122 ceph-mon[32255]: pgmap v17625: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:10 smithi122 ceph-mon[32255]: pgmap v17626: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:10] "GET /metrics HTTP/1.1" 200 202470 "" "Prometheus/2.18.1" 2021-12-16T03:46:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:10 smithi022 ceph-mon[31360]: pgmap v17626: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:10 smithi022 ceph-mon[40250]: pgmap v17626: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:11.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:11.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:11.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:11.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:12 smithi022 ceph-mon[31360]: pgmap v17627: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/24465824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:12 smithi022 ceph-mon[40250]: pgmap v17627: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/24465824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:46:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:46:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:12 smithi122 ceph-mon[32255]: pgmap v17627: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/24465824' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:46:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:46:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:14 smithi022 ceph-mon[31360]: pgmap v17628: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:14 smithi022 ceph-mon[40250]: pgmap v17628: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:14 smithi122 ceph-mon[32255]: pgmap v17628: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:16 smithi022 ceph-mon[31360]: pgmap v17629: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:16 smithi022 ceph-mon[40250]: pgmap v17629: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:16.572 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:16 smithi122 ceph-mon[32255]: pgmap v17629: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:16.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:16.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:16.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1440108816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:46:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1440108816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1440108816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:18 smithi022 ceph-mon[31360]: pgmap v17630: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:18 smithi022 ceph-mon[40250]: pgmap v17630: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:18 smithi122 ceph-mon[32255]: pgmap v17630: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:20 smithi122 ceph-mon[32255]: pgmap v17631: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:20] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:46:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:20 smithi022 ceph-mon[31360]: pgmap v17631: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:20 smithi022 ceph-mon[40250]: pgmap v17631: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:21.947 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:22.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:22.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:22.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:22 smithi022 ceph-mon[31360]: pgmap v17632: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:22 smithi022 ceph-mon[40250]: pgmap v17632: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:22.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:46:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:46:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:46:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:46:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:22 smithi122 ceph-mon[32255]: pgmap v17632: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2428738038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2428738038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2428738038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:24 smithi022 ceph-mon[31360]: pgmap v17633: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:24 smithi022 ceph-mon[40250]: pgmap v17633: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:24 smithi122 ceph-mon[32255]: pgmap v17633: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:26 smithi022 ceph-mon[31360]: pgmap v17634: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:26 smithi022 ceph-mon[40250]: pgmap v17634: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:26 smithi122 ceph-mon[32255]: pgmap v17634: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:27.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:27.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:27.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:27.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:27.681 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:46:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:28 smithi022 ceph-mon[31360]: pgmap v17635: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2782430044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:28 smithi022 ceph-mon[40250]: pgmap v17635: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2782430044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:28 smithi122 ceph-mon[32255]: pgmap v17635: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2782430044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:30 smithi122 ceph-mon[32255]: pgmap v17636: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:30] "GET /metrics HTTP/1.1" 200 202463 "" "Prometheus/2.18.1" 2021-12-16T03:46:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:30 smithi022 ceph-mon[31360]: pgmap v17636: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:30 smithi022 ceph-mon[40250]: pgmap v17636: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:32 smithi022 ceph-mon[31360]: pgmap v17637: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:32 smithi022 ceph-mon[40250]: pgmap v17637: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:32.682 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:32.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:32 smithi122 ceph-mon[32255]: pgmap v17637: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:46:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:46:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:46:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:46:33.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:33.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:33.055 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2748366214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2748366214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2748366214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:34 smithi022 ceph-mon[31360]: pgmap v17638: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:34 smithi022 ceph-mon[40250]: pgmap v17638: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:34.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:34 smithi122 ceph-mon[32255]: pgmap v17638: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:46:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:46:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:46:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:46:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:46:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:46:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:46:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:46:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:46:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:46:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:46:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:46:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:36 smithi022 ceph-mon[31360]: pgmap v17639: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:36 smithi022 ceph-mon[40250]: pgmap v17639: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:36 smithi122 ceph-mon[32255]: pgmap v17639: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:46:38.057 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:38.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:38.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:38.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:38.428 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:38 smithi022 ceph-mon[31360]: pgmap v17640: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:38.428 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:38 smithi022 ceph-mon[40250]: pgmap v17640: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:38 smithi122 ceph-mon[32255]: pgmap v17640: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:39.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/418445114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/418445114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/418445114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:40 smithi122 ceph-mon[32255]: pgmap v17641: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:40] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:46:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:40 smithi022 ceph-mon[31360]: pgmap v17641: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:40 smithi022 ceph-mon[40250]: pgmap v17641: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:42 smithi022 ceph-mon[31360]: pgmap v17642: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:42 smithi022 ceph-mon[40250]: pgmap v17642: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:42.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:46:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:46:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:42 smithi122 ceph-mon[32255]: pgmap v17642: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:46:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:46:43.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:43.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:43.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:43.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:44 smithi022 ceph-mon[31360]: pgmap v17643: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3067823745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:44 smithi022 ceph-mon[40250]: pgmap v17643: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:44.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3067823745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:44 smithi122 ceph-mon[32255]: pgmap v17643: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3067823745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:46 smithi022 ceph-mon[31360]: pgmap v17644: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:46 smithi022 ceph-mon[40250]: pgmap v17644: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:46 smithi122 ceph-mon[32255]: pgmap v17644: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:46:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:48 smithi022 ceph-mon[31360]: pgmap v17645: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:48 smithi022 ceph-mon[40250]: pgmap v17645: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:48 smithi122 ceph-mon[32255]: pgmap v17645: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:48.799 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:49.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:49.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:49.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1839418352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:49.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1839418352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1839418352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:50] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:46:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:50 smithi122 ceph-mon[32255]: pgmap v17646: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:50 smithi022 ceph-mon[31360]: pgmap v17646: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:50 smithi022 ceph-mon[40250]: pgmap v17646: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:46:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:46:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:46:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:46:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:46:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:46:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:46:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:46:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:46:51.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:46:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:52 smithi022 ceph-mon[31360]: pgmap v17647: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:52 smithi022 ceph-mon[40250]: pgmap v17647: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:52 smithi122 ceph-mon[32255]: pgmap v17647: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:52.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:46:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:46:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:46:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:46:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:54.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:54 smithi022 ceph-mon[40250]: pgmap v17648: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:54 smithi022 ceph-mon[31360]: pgmap v17648: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:54.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:54.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:54.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:46:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:46:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:54 smithi122 ceph-mon[32255]: pgmap v17648: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2552371800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2552371800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2552371800' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:46:56.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:56 smithi122 ceph-mon[32255]: pgmap v17649: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:56 smithi022 ceph-mon[31360]: pgmap v17649: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:56 smithi022 ceph-mon[40250]: pgmap v17649: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:46:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:46:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:46:58 smithi122 ceph-mon[32255]: pgmap v17650: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:46:58 smithi022 ceph-mon[31360]: pgmap v17650: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:46:58 smithi022 ceph-mon[40250]: pgmap v17650: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:46:59.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:46:59.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:46:59.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:46:59.916 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:00] "GET /metrics HTTP/1.1" 200 202458 "" "Prometheus/2.18.1" 2021-12-16T03:47:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:00 smithi122 ceph-mon[32255]: pgmap v17651: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/19423590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:00 smithi022 ceph-mon[31360]: pgmap v17651: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/19423590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:00 smithi022 ceph-mon[40250]: pgmap v17651: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/19423590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:02 smithi122 ceph-mon[32255]: pgmap v17652: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:02.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:47:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:47:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:47:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:47:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:02 smithi022 ceph-mon[31360]: pgmap v17652: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:02 smithi022 ceph-mon[40250]: pgmap v17652: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:04 smithi122 ceph-mon[32255]: pgmap v17653: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:04 smithi022 ceph-mon[31360]: pgmap v17653: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:04 smithi022 ceph-mon[40250]: pgmap v17653: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:04.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:05.287 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4173971517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4173971517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4173971517' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:06 smithi122 ceph-mon[32255]: pgmap v17654: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:06 smithi022 ceph-mon[31360]: pgmap v17654: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:06 smithi022 ceph-mon[40250]: pgmap v17654: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:47:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:08 smithi122 ceph-mon[32255]: pgmap v17655: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:08 smithi022 ceph-mon[31360]: pgmap v17655: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:08 smithi022 ceph-mon[40250]: pgmap v17655: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:10.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:10] "GET /metrics HTTP/1.1" 200 202468 "" "Prometheus/2.18.1" 2021-12-16T03:47:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:10 smithi122 ceph-mon[32255]: pgmap v17656: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:10.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:10.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:10.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:10.660 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:10 smithi022 ceph-mon[31360]: pgmap v17656: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:10.660 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:10 smithi022 ceph-mon[40250]: pgmap v17656: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3491068178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3491068178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3491068178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:12 smithi122 ceph-mon[32255]: pgmap v17657: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:47:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:47:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:47:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:47:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:12 smithi022 ceph-mon[31360]: pgmap v17657: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:12 smithi022 ceph-mon[40250]: pgmap v17657: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:14 smithi122 ceph-mon[32255]: pgmap v17658: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:14 smithi022 ceph-mon[31360]: pgmap v17658: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:14 smithi022 ceph-mon[40250]: pgmap v17658: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:16.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:16.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:16.032 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:16 smithi122 ceph-mon[32255]: pgmap v17659: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1591220228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:16 smithi022 ceph-mon[31360]: pgmap v17659: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1591220228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:16 smithi022 ceph-mon[40250]: pgmap v17659: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1591220228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:47:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:18 smithi122 ceph-mon[32255]: pgmap v17660: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:18 smithi022 ceph-mon[31360]: pgmap v17660: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:18 smithi022 ceph-mon[40250]: pgmap v17660: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:20] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:47:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:20 smithi122 ceph-mon[32255]: pgmap v17661: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:20 smithi022 ceph-mon[31360]: pgmap v17661: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:20 smithi022 ceph-mon[40250]: pgmap v17661: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:21.034 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:21.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:21.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:21.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:47:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:47:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:22 smithi122 ceph-mon[32255]: pgmap v17662: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1330202147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:22.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:47:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:47:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:22 smithi022 ceph-mon[31360]: pgmap v17662: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1330202147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:22 smithi022 ceph-mon[40250]: pgmap v17662: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1330202147' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:24 smithi122 ceph-mon[32255]: pgmap v17663: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:24 smithi022 ceph-mon[31360]: pgmap v17663: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:24 smithi022 ceph-mon[40250]: pgmap v17663: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:26.406 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:26 smithi122 ceph-mon[32255]: pgmap v17664: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:26 smithi022 ceph-mon[40250]: pgmap v17664: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:26 smithi022 ceph-mon[31360]: pgmap v17664: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:26.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:26.779 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:26.780 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:27.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2383417283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2383417283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2383417283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:47:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:28 smithi122 ceph-mon[32255]: pgmap v17665: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:28 smithi022 ceph-mon[31360]: pgmap v17665: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:28 smithi022 ceph-mon[40250]: pgmap v17665: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:30 smithi122 ceph-mon[32255]: pgmap v17666: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:30] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:47:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:30 smithi022 ceph-mon[31360]: pgmap v17666: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:30 smithi022 ceph-mon[40250]: pgmap v17666: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:31.780 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:32.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:32.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:32.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:32 smithi022 ceph-mon[31360]: pgmap v17667: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4076270670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:32 smithi022 ceph-mon[40250]: pgmap v17667: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4076270670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:32.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:47:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:47:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:32 smithi122 ceph-mon[32255]: pgmap v17667: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4076270670' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:47:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:47:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:34 smithi122 ceph-mon[32255]: pgmap v17668: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:34 smithi022 ceph-mon[31360]: pgmap v17668: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:34 smithi022 ceph-mon[40250]: pgmap v17668: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:47:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:47:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:47:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:47:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:47:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:47:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:47:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:47:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:47:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:47:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:47:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:47:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:36 smithi122 ceph-mon[32255]: pgmap v17669: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:36 smithi022 ceph-mon[31360]: pgmap v17669: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:36 smithi022 ceph-mon[40250]: pgmap v17669: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:37.158 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:37.527 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:38 smithi122 ceph-mon[32255]: pgmap v17670: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/729329737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:38 smithi022 ceph-mon[31360]: pgmap v17670: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/729329737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:38 smithi022 ceph-mon[40250]: pgmap v17670: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/729329737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:40 smithi122 ceph-mon[32255]: pgmap v17671: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:40] "GET /metrics HTTP/1.1" 200 202462 "" "Prometheus/2.18.1" 2021-12-16T03:47:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:40 smithi022 ceph-mon[31360]: pgmap v17671: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:40 smithi022 ceph-mon[40250]: pgmap v17671: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:42.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:42 smithi122 ceph-mon[32255]: pgmap v17672: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:42.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:47:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:47:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:47:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:47:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:42 smithi022 ceph-mon[31360]: pgmap v17672: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:42 smithi022 ceph-mon[40250]: pgmap v17672: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4073610792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4073610792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4073610792' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:44 smithi122 ceph-mon[32255]: pgmap v17673: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:44 smithi022 ceph-mon[31360]: pgmap v17673: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:44 smithi022 ceph-mon[40250]: pgmap v17673: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:46 smithi122 ceph-mon[32255]: pgmap v17674: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:46 smithi022 ceph-mon[31360]: pgmap v17674: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:46 smithi022 ceph-mon[40250]: pgmap v17674: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:47:47.900 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:48.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:48.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:48.270 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:48 smithi022 ceph-mon[31360]: pgmap v17675: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/43720574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:48 smithi022 ceph-mon[40250]: pgmap v17675: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/43720574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:48 smithi122 ceph-mon[32255]: pgmap v17675: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/43720574' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:50 smithi122 ceph-mon[32255]: pgmap v17676: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:50] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:47:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:50 smithi022 ceph-mon[31360]: pgmap v17676: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:50 smithi022 ceph-mon[40250]: pgmap v17676: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:47:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:47:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:47:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:47:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:47:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:47:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:47:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:47:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:47:52.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:47:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:47:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:52 smithi122 ceph-mon[32255]: pgmap v17677: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:47:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:47:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:47:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:52 smithi022 ceph-mon[40250]: pgmap v17677: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:47:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:52 smithi022 ceph-mon[31360]: pgmap v17677: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:47:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:53.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:53.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:53.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:54 smithi122 ceph-mon[32255]: pgmap v17678: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2557682880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:54 smithi022 ceph-mon[31360]: pgmap v17678: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2557682880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:47:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:54 smithi022 ceph-mon[40250]: pgmap v17678: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2557682880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:56 smithi122 ceph-mon[32255]: pgmap v17679: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:56 smithi022 ceph-mon[31360]: pgmap v17679: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:56 smithi022 ceph-mon[40250]: pgmap v17679: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:47:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:47:58.642 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:47:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:58 smithi122 ceph-mon[32255]: pgmap v17680: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:58 smithi022 ceph-mon[31360]: pgmap v17680: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:58 smithi022 ceph-mon[40250]: pgmap v17680: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:47:59.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:47:59.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:47:59.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:47:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:47:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3341582058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:47:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3341582058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:47:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:47:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3341582058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:00 smithi122 ceph-mon[32255]: pgmap v17681: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:00] "GET /metrics HTTP/1.1" 200 202467 "" "Prometheus/2.18.1" 2021-12-16T03:48:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:00 smithi022 ceph-mon[31360]: pgmap v17681: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:00 smithi022 ceph-mon[40250]: pgmap v17681: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:48:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:48:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:02 smithi122 ceph-mon[32255]: pgmap v17682: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:48:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:48:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:02 smithi022 ceph-mon[31360]: pgmap v17682: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:02 smithi022 ceph-mon[40250]: pgmap v17682: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:04.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:04.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:04.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:04.380 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:04 smithi122 ceph-mon[32255]: pgmap v17683: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1940500846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:04 smithi022 ceph-mon[31360]: pgmap v17683: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1940500846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:04 smithi022 ceph-mon[40250]: pgmap v17683: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1940500846' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:06 smithi122 ceph-mon[32255]: pgmap v17684: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:06 smithi022 ceph-mon[31360]: pgmap v17684: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:06 smithi022 ceph-mon[40250]: pgmap v17684: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:48:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:08 smithi122 ceph-mon[32255]: pgmap v17685: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:08 smithi022 ceph-mon[31360]: pgmap v17685: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:08 smithi022 ceph-mon[40250]: pgmap v17685: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:09.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:09.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:09.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:09.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:09.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:09 smithi022 ceph-mon[31360]: pgmap v17686: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:09.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:09 smithi022 ceph-mon[40250]: pgmap v17686: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:09 smithi122 ceph-mon[32255]: pgmap v17686: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:48:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:10] "GET /metrics HTTP/1.1" 200 202480 "" "Prometheus/2.18.1" 2021-12-16T03:48:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1713428487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1713428487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1713428487' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:11 smithi022 ceph-mon[31360]: pgmap v17687: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:11 smithi022 ceph-mon[40250]: pgmap v17687: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:11 smithi122 ceph-mon[32255]: pgmap v17687: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:12.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:48:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:48:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:48:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:48:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:14 smithi122 ceph-mon[32255]: pgmap v17688: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:14 smithi022 ceph-mon[31360]: pgmap v17688: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:14 smithi022 ceph-mon[40250]: pgmap v17688: 132 pgs: 132 active+clean; 582 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:14.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:15.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:15.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:15.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1419183034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1419183034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1419183034' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:16 smithi122 ceph-mon[32255]: pgmap v17689: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:16 smithi022 ceph-mon[31360]: pgmap v17689: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:16 smithi022 ceph-mon[40250]: pgmap v17689: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:48:18.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:18 smithi122 ceph-mon[32255]: pgmap v17690: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:18 smithi022 ceph-mon[31360]: pgmap v17690: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:18 smithi022 ceph-mon[40250]: pgmap v17690: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:20.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:20] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T03:48:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:20 smithi122 ceph-mon[32255]: pgmap v17691: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:20.497 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:20.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:20.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:20.499 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:20 smithi022 ceph-mon[31360]: pgmap v17691: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:20.499 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:20 smithi022 ceph-mon[40250]: pgmap v17691: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/175433248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/175433248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/175433248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:22.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:22 smithi122 ceph-mon[32255]: pgmap v17692: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:22.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:48:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:48:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:22 smithi022 ceph-mon[31360]: pgmap v17692: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:22.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:22 smithi022 ceph-mon[40250]: pgmap v17692: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:22.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:48:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:48:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:24.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:24 smithi122 ceph-mon[32255]: pgmap v17693: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:24 smithi022 ceph-mon[31360]: pgmap v17693: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:24 smithi022 ceph-mon[40250]: pgmap v17693: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:48:25.499 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:25.867 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:25.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:25.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:26 smithi122 ceph-mon[32255]: pgmap v17694: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1401297023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:26 smithi022 ceph-mon[31360]: pgmap v17694: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1401297023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:26 smithi022 ceph-mon[40250]: pgmap v17694: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:48:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1401297023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:48:28.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:28 smithi122 ceph-mon[32255]: pgmap v17695: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:28 smithi022 ceph-mon[31360]: pgmap v17695: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:28 smithi022 ceph-mon[40250]: pgmap v17695: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:30 smithi122 ceph-mon[32255]: pgmap v17696: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:30] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T03:48:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:30 smithi022 ceph-mon[31360]: pgmap v17696: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:30.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:30 smithi022 ceph-mon[40250]: pgmap v17696: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:30.869 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:31.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:31.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:31.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:32 smithi122 ceph-mon[32255]: pgmap v17697: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:32.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/733189215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:48:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:48:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:32 smithi022 ceph-mon[31360]: pgmap v17697: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/733189215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:32 smithi022 ceph-mon[40250]: pgmap v17697: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/733189215' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:32.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:48:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:48:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:34.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:34 smithi122 ceph-mon[32255]: pgmap v17698: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:34 smithi022 ceph-mon[31360]: pgmap v17698: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:34 smithi022 ceph-mon[40250]: pgmap v17698: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:48:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:48:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:48:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:48:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:48:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:48:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:48:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:48:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:48:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:48:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:48:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:48:36.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:36 smithi022 ceph-mon[31360]: pgmap v17699: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:36 smithi022 ceph-mon[40250]: pgmap v17699: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:36.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:36.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:36.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:36 smithi122 ceph-mon[32255]: pgmap v17699: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:37.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:48:37.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4010484042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:37.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4010484042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4010484042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:38 smithi022 ceph-mon[31360]: pgmap v17700: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:38 smithi022 ceph-mon[40250]: pgmap v17700: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:38 smithi122 ceph-mon[32255]: pgmap v17700: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:40 smithi122 ceph-mon[32255]: pgmap v17701: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:40] "GET /metrics HTTP/1.1" 200 202489 "" "Prometheus/2.18.1" 2021-12-16T03:48:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:40 smithi022 ceph-mon[31360]: pgmap v17701: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:40 smithi022 ceph-mon[40250]: pgmap v17701: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:41.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:41.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:41.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:41.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:42 smithi022 ceph-mon[31360]: pgmap v17702: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:42.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4284482563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:42 smithi022 ceph-mon[40250]: pgmap v17702: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:42.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4284482563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:42.542 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:42 smithi122 ceph-mon[32255]: pgmap v17702: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:42.543 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4284482563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:42.543 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:48:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:48:42.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:48:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:48:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:44 smithi022 ceph-mon[31360]: pgmap v17703: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:44 smithi022 ceph-mon[40250]: pgmap v17703: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:44 smithi122 ceph-mon[32255]: pgmap v17703: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:46 smithi022 ceph-mon[31360]: pgmap v17704: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:46 smithi022 ceph-mon[40250]: pgmap v17704: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:46 smithi122 ceph-mon[32255]: pgmap v17704: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:46.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:47.362 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:47.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:47.363 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:48:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:48 smithi022 ceph-mon[31360]: pgmap v17705: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3209073459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:48 smithi022 ceph-mon[40250]: pgmap v17705: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3209073459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:48 smithi122 ceph-mon[32255]: pgmap v17705: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3209073459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:50 smithi122 ceph-mon[32255]: pgmap v17706: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:50] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:48:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:50 smithi022 ceph-mon[31360]: pgmap v17706: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:50 smithi022 ceph-mon[40250]: pgmap v17706: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:52.364 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[31360]: pgmap v17707: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:48:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:48:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:48:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[40250]: pgmap v17707: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:48:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:48:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:48:52.551 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:48:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:48:52.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:52 smithi122 ceph-mon[32255]: pgmap v17707: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:52.552 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:48:52.552 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:48:52.552 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:48:52.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:52.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:52.731 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:52.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:48:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:48:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2282798251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:53.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2282798251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:53.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2282798251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:54 smithi022 ceph-mon[31360]: pgmap v17708: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:54 smithi022 ceph-mon[40250]: pgmap v17708: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:54.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:48:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:54 smithi122 ceph-mon[32255]: pgmap v17708: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:56 smithi122 ceph-mon[32255]: pgmap v17709: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:56 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:56 smithi022 ceph-mon[31360]: pgmap v17709: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:56 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:56 smithi022 ceph-mon[40250]: pgmap v17709: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:48:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:56.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:56 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:57.733 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:48:57.734 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:48:57.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:57.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:57 smithi022 ceph-mon[31360]: pgmap v17710: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:57.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:57.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:57 smithi022 ceph-mon[40250]: pgmap v17710: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:48:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:57 smithi122 ceph-mon[32255]: pgmap v17710: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:48:58.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:48:58.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:48:58.104 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:48:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2014736469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2014736469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2014736469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:48:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:48:59 smithi122 ceph-mon[32255]: pgmap v17711: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:48:59 smithi022 ceph-mon[31360]: pgmap v17711: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:48:59 smithi022 ceph-mon[40250]: pgmap v17711: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:00.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:00] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:49:02.184 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:01 smithi122 ceph-mon[32255]: pgmap v17712: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:02.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:01 smithi022 ceph-mon[31360]: pgmap v17712: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:01 smithi022 ceph-mon[40250]: pgmap v17712: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:02.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:49:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:49:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:49:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:49:03.106 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:03.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:03.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:03.479 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:04 smithi122 ceph-mon[32255]: pgmap v17713: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:04.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2876356314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:04 smithi022 ceph-mon[31360]: pgmap v17713: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2876356314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:04 smithi022 ceph-mon[40250]: pgmap v17713: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2876356314' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:06.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:06 smithi122 ceph-mon[32255]: pgmap v17714: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:06 smithi022 ceph-mon[31360]: pgmap v17714: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:06 smithi022 ceph-mon[40250]: pgmap v17714: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:49:08.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:08 smithi122 ceph-mon[32255]: pgmap v17715: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:08.481 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:08.481 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:08 smithi022 ceph-mon[31360]: pgmap v17715: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:08.482 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:08 smithi022 ceph-mon[40250]: pgmap v17715: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:08.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:08.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:08.853 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:09.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4290383148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4290383148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:09.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4290383148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:10] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T03:49:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:10 smithi122 ceph-mon[32255]: pgmap v17716: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:10 smithi022 ceph-mon[31360]: pgmap v17716: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:10 smithi022 ceph-mon[40250]: pgmap v17716: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:12.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:49:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:49:12.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:12 smithi122 ceph-mon[32255]: pgmap v17717: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:12 smithi022 ceph-mon[31360]: pgmap v17717: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:12.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:12 smithi022 ceph-mon[40250]: pgmap v17717: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:12.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:49:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:49:13.855 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:14.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:14.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:14.226 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:14.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:14 smithi122 ceph-mon[32255]: pgmap v17718: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:14 smithi022 ceph-mon[31360]: pgmap v17718: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:14 smithi022 ceph-mon[40250]: pgmap v17718: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1345766394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1345766394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1345766394' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:16 smithi122 ceph-mon[32255]: pgmap v17719: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:16 smithi022 ceph-mon[31360]: pgmap v17719: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:16 smithi022 ceph-mon[40250]: pgmap v17719: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:49:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:18 smithi022 ceph-mon[31360]: pgmap v17720: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:18 smithi022 ceph-mon[40250]: pgmap v17720: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:18 smithi122 ceph-mon[32255]: pgmap v17720: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:19.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:19.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:19.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:19.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:20] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:49:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:20 smithi122 ceph-mon[32255]: pgmap v17721: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3110898113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:20 smithi022 ceph-mon[31360]: pgmap v17721: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3110898113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:20 smithi022 ceph-mon[40250]: pgmap v17721: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:20.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3110898113' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:22 smithi022 ceph-mon[31360]: pgmap v17722: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:22 smithi022 ceph-mon[40250]: pgmap v17722: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:22.579 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:22 smithi122 ceph-mon[32255]: pgmap v17722: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:22.580 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:49:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:49:22.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:49:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:49:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:24 smithi022 ceph-mon[31360]: pgmap v17723: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:24 smithi022 ceph-mon[40250]: pgmap v17723: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:24.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:24 smithi122 ceph-mon[32255]: pgmap v17723: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:25.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1713860978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:25.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1713860978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1713860978' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:26 smithi022 ceph-mon[31360]: pgmap v17724: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:26 smithi022 ceph-mon[40250]: pgmap v17724: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:26 smithi122 ceph-mon[32255]: pgmap v17724: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:49:28.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:28 smithi022 ceph-mon[40250]: pgmap v17725: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:28.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:28 smithi022 ceph-mon[31360]: pgmap v17725: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:28 smithi122 ceph-mon[32255]: pgmap v17725: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:29.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:30.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:30 smithi022 ceph-mon[31360]: pgmap v17726: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:30.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:30 smithi022 ceph-mon[40250]: pgmap v17726: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:30.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:30.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:30.343 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:30 smithi122 ceph-mon[32255]: pgmap v17726: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:30] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:49:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2077670538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2077670538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2077670538' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:32 smithi022 ceph-mon[31360]: pgmap v17727: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:32.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:32 smithi022 ceph-mon[40250]: pgmap v17727: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:32.590 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:32 smithi122 ceph-mon[32255]: pgmap v17727: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:32.590 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:49:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:49:32.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:49:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:49:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:34 smithi022 ceph-mon[31360]: pgmap v17728: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:34 smithi022 ceph-mon[40250]: pgmap v17728: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:34 smithi122 ceph-mon[32255]: pgmap v17728: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:35.344 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:49:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:49:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:49:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:49:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:49:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:49:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:49:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:49:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:49:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:49:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:49:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:49:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:36 smithi022 ceph-mon[31360]: pgmap v17729: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4246667887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:36 smithi022 ceph-mon[40250]: pgmap v17729: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4246667887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:36 smithi122 ceph-mon[32255]: pgmap v17729: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4246667887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:49:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:38 smithi022 ceph-mon[31360]: pgmap v17730: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:38 smithi022 ceph-mon[40250]: pgmap v17730: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:38 smithi122 ceph-mon[32255]: pgmap v17730: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:40] "GET /metrics HTTP/1.1" 200 202481 "" "Prometheus/2.18.1" 2021-12-16T03:49:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:40 smithi122 ceph-mon[32255]: pgmap v17731: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:40 smithi022 ceph-mon[31360]: pgmap v17731: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:40 smithi022 ceph-mon[40250]: pgmap v17731: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:40.716 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4114026276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4114026276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4114026276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:42 smithi022 ceph-mon[31360]: pgmap v17732: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:42 smithi022 ceph-mon[40250]: pgmap v17732: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:42.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:42 smithi122 ceph-mon[32255]: pgmap v17732: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:42.591 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:49:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:49:42.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:49:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:49:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:44 smithi022 ceph-mon[31360]: pgmap v17733: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:44 smithi022 ceph-mon[40250]: pgmap v17733: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:44 smithi122 ceph-mon[32255]: pgmap v17733: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:46.092 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:46.461 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:46 smithi022 ceph-mon[40250]: pgmap v17734: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:46.462 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:46 smithi022 ceph-mon[31360]: pgmap v17734: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:46 smithi122 ceph-mon[32255]: pgmap v17734: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4035737962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:49:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4035737962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4035737962' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:48 smithi022 ceph-mon[31360]: pgmap v17735: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:48 smithi022 ceph-mon[40250]: pgmap v17735: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:48 smithi122 ceph-mon[32255]: pgmap v17735: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:50 smithi122 ceph-mon[32255]: pgmap v17736: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:50] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:49:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:50 smithi022 ceph-mon[31360]: pgmap v17736: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:50.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:50 smithi022 ceph-mon[40250]: pgmap v17736: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:51.463 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:51.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:51.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:51.832 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:52 smithi022 ceph-mon[31360]: pgmap v17737: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2015180297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:52 smithi022 ceph-mon[40250]: pgmap v17737: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2015180297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:52 smithi122 ceph-mon[32255]: pgmap v17737: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2015180297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:52.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:49:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:49:52.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:49:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:49:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:54 smithi022 ceph-mon[31360]: pgmap v17738: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:54 smithi022 ceph-mon[40250]: pgmap v17738: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:49:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:54 smithi122 ceph-mon[32255]: pgmap v17738: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:56 smithi022 ceph-mon[31360]: pgmap v17739: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:56 smithi022 ceph-mon[40250]: pgmap v17739: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:56.658 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:56 smithi122 ceph-mon[32255]: pgmap v17739: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:49:56.834 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:49:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:49:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:49:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:49:57.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:49:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/169675338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:57.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:49:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:49:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:49:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:49:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/169675338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:49:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:49:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:49:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:57 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:49:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/169675338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:49:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:49:58.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:49:58 smithi022 ceph-mon[31360]: pgmap v17740: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:49:58 smithi022 ceph-mon[40250]: pgmap v17740: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:49:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:49:58 smithi122 ceph-mon[32255]: pgmap v17740: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:00] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:50:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:00 smithi122 ceph-mon[32255]: pgmap v17741: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T03:50:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:00 smithi022 ceph-mon[31360]: pgmap v17741: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T03:50:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:00 smithi022 ceph-mon[40250]: pgmap v17741: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:00.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T03:50:02.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:02 smithi022 ceph-mon[31360]: pgmap v17742: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:02 smithi022 ceph-mon[40250]: pgmap v17742: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:02.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:02.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:02.573 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:02.605 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:02 smithi122 ceph-mon[32255]: pgmap v17742: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:02.605 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:50:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:50:02.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:50:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:50:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2265060711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2265060711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2265060711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:04 smithi022 ceph-mon[31360]: pgmap v17743: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:04 smithi022 ceph-mon[40250]: pgmap v17743: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:04 smithi122 ceph-mon[32255]: pgmap v17743: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:06 smithi122 ceph-mon[32255]: pgmap v17744: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:06 smithi022 ceph-mon[31360]: pgmap v17744: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:06 smithi022 ceph-mon[40250]: pgmap v17744: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:07.574 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:50:07.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:07.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:07.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:08 smithi122 ceph-mon[32255]: pgmap v17745: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3752959420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:08 smithi022 ceph-mon[31360]: pgmap v17745: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3752959420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:08 smithi022 ceph-mon[40250]: pgmap v17745: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3752959420' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:10] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:50:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:10 smithi122 ceph-mon[32255]: pgmap v17746: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:10 smithi022 ceph-mon[31360]: pgmap v17746: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:10 smithi022 ceph-mon[40250]: pgmap v17746: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:12.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:12 smithi122 ceph-mon[32255]: pgmap v17747: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:12.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:50:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:50:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:12 smithi022 ceph-mon[31360]: pgmap v17747: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:12 smithi022 ceph-mon[40250]: pgmap v17747: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:12.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:50:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:50:12.945 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:14 smithi122 ceph-mon[32255]: pgmap v17748: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/470123189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:14 smithi022 ceph-mon[31360]: pgmap v17748: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/470123189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:14 smithi022 ceph-mon[40250]: pgmap v17748: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/470123189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:16 smithi122 ceph-mon[32255]: pgmap v17749: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:16 smithi022 ceph-mon[31360]: pgmap v17749: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:16 smithi022 ceph-mon[40250]: pgmap v17749: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:50:18.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:18 smithi122 ceph-mon[32255]: pgmap v17750: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:18.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:18.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:18.693 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:18.694 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:18 smithi022 ceph-mon[31360]: pgmap v17750: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:18.694 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:18 smithi022 ceph-mon[40250]: pgmap v17750: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2442787357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2442787357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2442787357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:20 smithi122 ceph-mon[32255]: pgmap v17751: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:20] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T03:50:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:20 smithi022 ceph-mon[31360]: pgmap v17751: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:20 smithi022 ceph-mon[40250]: pgmap v17751: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:22.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:22 smithi122 ceph-mon[32255]: pgmap v17752: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:22.624 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:50:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:50:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:22 smithi022 ceph-mon[31360]: pgmap v17752: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:22 smithi022 ceph-mon[40250]: pgmap v17752: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:22.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:50:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:50:23.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:24.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:24.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:24.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:24 smithi022 ceph-mon[31360]: pgmap v17753: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/741458573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:24 smithi022 ceph-mon[40250]: pgmap v17753: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/741458573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:24 smithi122 ceph-mon[32255]: pgmap v17753: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/741458573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:26 smithi122 ceph-mon[32255]: pgmap v17754: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:26 smithi022 ceph-mon[31360]: pgmap v17754: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:26 smithi022 ceph-mon[40250]: pgmap v17754: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:50:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:28 smithi122 ceph-mon[32255]: pgmap v17755: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:28 smithi022 ceph-mon[31360]: pgmap v17755: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:28 smithi022 ceph-mon[40250]: pgmap v17755: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:29.069 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:29.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:29.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:29.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:30] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T03:50:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:30 smithi122 ceph-mon[32255]: pgmap v17756: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2464561912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:30 smithi022 ceph-mon[31360]: pgmap v17756: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2464561912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:30 smithi022 ceph-mon[40250]: pgmap v17756: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2464561912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:32.635 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:32 smithi122 ceph-mon[32255]: pgmap v17757: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:32.635 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:50:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:50:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:32 smithi022 ceph-mon[31360]: pgmap v17757: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:32 smithi022 ceph-mon[40250]: pgmap v17757: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:32.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:50:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:50:34.442 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:34 smithi122 ceph-mon[32255]: pgmap v17758: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:34.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:34 smithi022 ceph-mon[40250]: pgmap v17758: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:34 smithi022 ceph-mon[31360]: pgmap v17758: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:50:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:50:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:50:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:50:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1786028897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:35.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:50:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:50:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:50:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:50:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1786028897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:50:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:50:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:50:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:50:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1786028897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:36 smithi122 ceph-mon[32255]: pgmap v17759: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:36 smithi022 ceph-mon[31360]: pgmap v17759: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:36 smithi022 ceph-mon[40250]: pgmap v17759: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:50:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:38 smithi122 ceph-mon[32255]: pgmap v17760: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:38 smithi022 ceph-mon[31360]: pgmap v17760: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:38 smithi022 ceph-mon[40250]: pgmap v17760: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:39.814 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:40.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:40.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:40.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:40] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:50:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:40 smithi122 ceph-mon[32255]: pgmap v17761: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/57970741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:40 smithi022 ceph-mon[31360]: pgmap v17761: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/57970741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:40 smithi022 ceph-mon[40250]: pgmap v17761: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/57970741' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:42.645 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:42 smithi122 ceph-mon[32255]: pgmap v17762: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:42.645 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:50:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:50:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:42 smithi022 ceph-mon[31360]: pgmap v17762: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:42 smithi022 ceph-mon[40250]: pgmap v17762: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:42.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:50:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:50:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:44 smithi122 ceph-mon[32255]: pgmap v17763: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:44 smithi022 ceph-mon[31360]: pgmap v17763: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:44 smithi022 ceph-mon[40250]: pgmap v17763: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:45.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:45.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:45.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:46 smithi122 ceph-mon[32255]: pgmap v17764: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/199296048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:46 smithi022 ceph-mon[31360]: pgmap v17764: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/199296048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:46 smithi022 ceph-mon[40250]: pgmap v17764: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/199296048' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:50:48.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:48 smithi122 ceph-mon[32255]: pgmap v17765: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:48 smithi022 ceph-mon[40250]: pgmap v17765: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:48 smithi022 ceph-mon[31360]: pgmap v17765: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:50] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T03:50:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:50 smithi122 ceph-mon[32255]: pgmap v17766: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:50.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:50 smithi022 ceph-mon[31360]: pgmap v17766: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:50 smithi022 ceph-mon[40250]: pgmap v17766: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:50.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:50.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:50.928 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/182411746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/182411746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/182411746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:52.649 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:52 smithi122 ceph-mon[32255]: pgmap v17767: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:52.650 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:50:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:50:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:52 smithi022 ceph-mon[31360]: pgmap v17767: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:52 smithi022 ceph-mon[40250]: pgmap v17767: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:52.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:50:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:50:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:54 smithi122 ceph-mon[32255]: pgmap v17768: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:54 smithi022 ceph-mon[31360]: pgmap v17768: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:50:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:54 smithi022 ceph-mon[40250]: pgmap v17768: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:55.930 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:50:56.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:50:56.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:50:56.299 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:50:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:56 smithi122 ceph-mon[32255]: pgmap v17769: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3703280717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:56 smithi022 ceph-mon[31360]: pgmap v17769: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3703280717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:56 smithi022 ceph-mon[40250]: pgmap v17769: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:50:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3703280717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:50:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:50:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:50:57.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:57 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:50:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:50:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:50:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:50:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:57 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:50:57.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:50:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:50:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:57 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:50:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:58 smithi122 ceph-mon[32255]: pgmap v17770: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:50:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:50:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:58 smithi022 ceph-mon[31360]: pgmap v17770: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:50:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:50:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:58 smithi022 ceph-mon[40250]: pgmap v17770: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:50:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:50:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:51:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:00] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T03:51:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:00 smithi122 ceph-mon[32255]: pgmap v17771: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:00 smithi022 ceph-mon[31360]: pgmap v17771: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:00 smithi022 ceph-mon[40250]: pgmap v17771: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:01.301 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:01.672 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:01.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:01.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:02.655 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:51:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:51:02.656 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:02 smithi122 ceph-mon[32255]: pgmap v17772: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:02.657 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2680628719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:02 smithi022 ceph-mon[31360]: pgmap v17772: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2680628719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:02 smithi022 ceph-mon[40250]: pgmap v17772: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2680628719' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:02.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:51:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:51:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:04 smithi122 ceph-mon[32255]: pgmap v17773: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:04 smithi022 ceph-mon[31360]: pgmap v17773: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:04 smithi022 ceph-mon[40250]: pgmap v17773: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:06.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:06.675 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:06 smithi022 ceph-mon[31360]: pgmap v17774: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:06.676 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:06 smithi022 ceph-mon[40250]: pgmap v17774: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:06 smithi122 ceph-mon[32255]: pgmap v17774: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:07.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:07.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:07.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3877001530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3877001530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:51:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3877001530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:08 smithi122 ceph-mon[32255]: pgmap v17775: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:08 smithi022 ceph-mon[31360]: pgmap v17775: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:08 smithi022 ceph-mon[40250]: pgmap v17775: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:10] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T03:51:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:10 smithi122 ceph-mon[32255]: pgmap v17776: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:10 smithi022 ceph-mon[31360]: pgmap v17776: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:10 smithi022 ceph-mon[40250]: pgmap v17776: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:12.047 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:12.420 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:12.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:12.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:12.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:12 smithi122 ceph-mon[32255]: pgmap v17777: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:12.667 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:51:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:51:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:12 smithi022 ceph-mon[31360]: pgmap v17777: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:12 smithi022 ceph-mon[40250]: pgmap v17777: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:12.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:51:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:51:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/978459882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:13.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/978459882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:13.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/978459882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:14 smithi122 ceph-mon[32255]: pgmap v17778: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:14 smithi022 ceph-mon[31360]: pgmap v17778: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:14 smithi022 ceph-mon[40250]: pgmap v17778: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:16 smithi122 ceph-mon[32255]: pgmap v17779: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:16 smithi022 ceph-mon[31360]: pgmap v17779: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:16 smithi022 ceph-mon[40250]: pgmap v17779: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:17.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:51:17.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:17.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:17.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:18 smithi122 ceph-mon[32255]: pgmap v17780: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/21370220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:18 smithi022 ceph-mon[31360]: pgmap v17780: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/21370220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:18 smithi022 ceph-mon[40250]: pgmap v17780: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/21370220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:20 smithi122 ceph-mon[32255]: pgmap v17781: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:20] "GET /metrics HTTP/1.1" 200 202475 "" "Prometheus/2.18.1" 2021-12-16T03:51:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:20 smithi022 ceph-mon[31360]: pgmap v17781: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:20 smithi022 ceph-mon[40250]: pgmap v17781: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:22 smithi122 ceph-mon[32255]: pgmap v17782: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:22.672 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:51:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:51:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:22 smithi022 ceph-mon[31360]: pgmap v17782: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:22 smithi022 ceph-mon[40250]: pgmap v17782: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:22.791 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:22.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:51:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:51:23.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:23.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:23.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3317070435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3317070435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3317070435' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:24 smithi122 ceph-mon[32255]: pgmap v17783: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:24 smithi022 ceph-mon[31360]: pgmap v17783: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:24 smithi022 ceph-mon[40250]: pgmap v17783: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:26 smithi122 ceph-mon[32255]: pgmap v17784: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:26 smithi022 ceph-mon[31360]: pgmap v17784: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:26.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:26 smithi022 ceph-mon[40250]: pgmap v17784: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:51:28.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:28.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:28.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:28.536 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:28 smithi122 ceph-mon[32255]: pgmap v17785: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:28 smithi022 ceph-mon[31360]: pgmap v17785: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:28 smithi022 ceph-mon[40250]: pgmap v17785: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1932243648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1932243648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1932243648' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:30.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:30] "GET /metrics HTTP/1.1" 200 202475 "" "Prometheus/2.18.1" 2021-12-16T03:51:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:30 smithi122 ceph-mon[32255]: pgmap v17786: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:30 smithi022 ceph-mon[31360]: pgmap v17786: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:30 smithi022 ceph-mon[40250]: pgmap v17786: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:32.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:32 smithi122 ceph-mon[32255]: pgmap v17787: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:32.680 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:51:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:51:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:32 smithi022 ceph-mon[31360]: pgmap v17787: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:32 smithi022 ceph-mon[40250]: pgmap v17787: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:32.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:51:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:51:33.537 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:33.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:33.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:33.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:34 smithi122 ceph-mon[32255]: pgmap v17788: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1592566707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:51:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:51:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:51:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:51:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[31360]: pgmap v17788: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1592566707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:51:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:51:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:51:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:51:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[40250]: pgmap v17788: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1592566707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:51:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:51:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:51:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:51:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:36 smithi122 ceph-mon[32255]: pgmap v17789: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:36 smithi022 ceph-mon[31360]: pgmap v17789: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:36 smithi022 ceph-mon[40250]: pgmap v17789: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:51:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:38 smithi022 ceph-mon[31360]: pgmap v17790: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:38 smithi022 ceph-mon[40250]: pgmap v17790: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:38.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:38 smithi122 ceph-mon[32255]: pgmap v17790: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:39.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:39.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:39.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3746375625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3746375625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3746375625' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:40] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T03:51:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:40 smithi022 ceph-mon[31360]: pgmap v17791: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:40 smithi022 ceph-mon[40250]: pgmap v17791: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:40 smithi122 ceph-mon[32255]: pgmap v17791: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:41 smithi022 ceph-mon[31360]: pgmap v17792: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:41 smithi022 ceph-mon[40250]: pgmap v17792: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:41 smithi122 ceph-mon[32255]: pgmap v17792: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:42.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:51:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:51:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:51:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:51:44.281 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:44 smithi022 ceph-mon[31360]: pgmap v17793: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:44 smithi022 ceph-mon[40250]: pgmap v17793: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:44.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:44.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:44.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:44 smithi122 ceph-mon[32255]: pgmap v17793: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3830020254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3830020254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3830020254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:46 smithi022 ceph-mon[31360]: pgmap v17794: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:46 smithi022 ceph-mon[40250]: pgmap v17794: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:46 smithi122 ceph-mon[32255]: pgmap v17794: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:51:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:48 smithi022 ceph-mon[31360]: pgmap v17795: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:48 smithi022 ceph-mon[40250]: pgmap v17795: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:48 smithi122 ceph-mon[32255]: pgmap v17795: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:49.652 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:50.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:50.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:50.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:50.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:50 smithi022 ceph-mon[40250]: pgmap v17796: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:50.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1455971990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:50 smithi022 ceph-mon[31360]: pgmap v17796: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:50.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1455971990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:50] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T03:51:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:50 smithi122 ceph-mon[32255]: pgmap v17796: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1455971990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:52 smithi022 ceph-mon[31360]: pgmap v17797: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:52 smithi022 ceph-mon[40250]: pgmap v17797: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:52 smithi122 ceph-mon[32255]: pgmap v17797: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:51:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:51:53.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:51:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:51:54.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:54 smithi022 ceph-mon[31360]: pgmap v17798: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:54.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:54 smithi022 ceph-mon[40250]: pgmap v17798: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:51:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:54 smithi122 ceph-mon[32255]: pgmap v17798: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:55.021 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:51:55.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:51:55.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:51:55.394 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:51:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:56 smithi022 ceph-mon[31360]: pgmap v17799: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3407883752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:56 smithi022 ceph-mon[40250]: pgmap v17799: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3407883752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:56 smithi122 ceph-mon[32255]: pgmap v17799: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:51:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3407883752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:51:57.647 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:51:58.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[31360]: pgmap v17800: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:51:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:51:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:51:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:51:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[40250]: pgmap v17800: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:51:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:51:58.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:51:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:51:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:51:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:58 smithi122 ceph-mon[32255]: pgmap v17800: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:51:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:51:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:51:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:51:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:51:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:52:00.395 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:00 smithi122 ceph-mon[32255]: pgmap v17801: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:00] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T03:52:00.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:00 smithi022 ceph-mon[31360]: pgmap v17801: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:00.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:00 smithi022 ceph-mon[40250]: pgmap v17801: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:00.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:00.837 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:00.838 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/386533266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/386533266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/386533266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:02 smithi022 ceph-mon[31360]: pgmap v17802: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:02 smithi022 ceph-mon[40250]: pgmap v17802: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:02 smithi122 ceph-mon[32255]: pgmap v17802: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:52:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:52:03.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:52:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:52:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:04 smithi022 ceph-mon[31360]: pgmap v17803: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:04 smithi022 ceph-mon[40250]: pgmap v17803: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:04 smithi122 ceph-mon[32255]: pgmap v17803: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:05.777 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:06.142 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:06.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:06.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:06 smithi022 ceph-mon[31360]: pgmap v17804: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1328992691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:06 smithi022 ceph-mon[40250]: pgmap v17804: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1328992691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:06 smithi122 ceph-mon[32255]: pgmap v17804: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1328992691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:52:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:08 smithi022 ceph-mon[31360]: pgmap v17805: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:08 smithi022 ceph-mon[40250]: pgmap v17805: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:08 smithi122 ceph-mon[32255]: pgmap v17805: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:10] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:52:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:10 smithi122 ceph-mon[32255]: pgmap v17806: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:10 smithi022 ceph-mon[31360]: pgmap v17806: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:10 smithi022 ceph-mon[40250]: pgmap v17806: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:11.144 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:11.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:11.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:11.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:12 smithi022 ceph-mon[31360]: pgmap v17807: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1925000981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:12 smithi022 ceph-mon[40250]: pgmap v17807: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1925000981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:52:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:52:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:12 smithi122 ceph-mon[32255]: pgmap v17807: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1925000981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:13.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:52:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:52:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:14 smithi022 ceph-mon[31360]: pgmap v17808: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:14 smithi022 ceph-mon[40250]: pgmap v17808: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:14 smithi122 ceph-mon[32255]: pgmap v17808: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:16.517 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:16 smithi022 ceph-mon[31360]: pgmap v17809: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:16.518 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:16 smithi022 ceph-mon[40250]: pgmap v17809: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:16.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:16 smithi122 ceph-mon[32255]: pgmap v17809: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:16.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:16.886 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:16.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:17.250 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2141802993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:17.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:52:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2141802993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2141802993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:18 smithi022 ceph-mon[31360]: pgmap v17810: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:18 smithi022 ceph-mon[40250]: pgmap v17810: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:18 smithi122 ceph-mon[32255]: pgmap v17810: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:20] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:52:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:20 smithi122 ceph-mon[32255]: pgmap v17811: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:20 smithi022 ceph-mon[31360]: pgmap v17811: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:20 smithi022 ceph-mon[40250]: pgmap v17811: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:21.888 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:22.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:22.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:22.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:22 smithi022 ceph-mon[31360]: pgmap v17812: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3489158779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:22 smithi022 ceph-mon[40250]: pgmap v17812: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3489158779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:22 smithi122 ceph-mon[32255]: pgmap v17812: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3489158779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:52:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:52:23.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:52:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:52:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:24 smithi022 ceph-mon[31360]: pgmap v17813: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:24 smithi022 ceph-mon[40250]: pgmap v17813: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:24 smithi122 ceph-mon[32255]: pgmap v17813: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:26 smithi022 ceph-mon[31360]: pgmap v17814: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:26 smithi022 ceph-mon[40250]: pgmap v17814: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:26.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:26 smithi122 ceph-mon[32255]: pgmap v17814: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:27.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:52:27.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:27.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:27.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:28 smithi022 ceph-mon[31360]: pgmap v17815: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1208473698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:28 smithi022 ceph-mon[40250]: pgmap v17815: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1208473698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:28 smithi122 ceph-mon[32255]: pgmap v17815: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T03:52:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1208473698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:30] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:52:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:30 smithi122 ceph-mon[32255]: pgmap v17816: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:30 smithi022 ceph-mon[31360]: pgmap v17816: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:30 smithi022 ceph-mon[40250]: pgmap v17816: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:32 smithi022 ceph-mon[31360]: pgmap v17817: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:32 smithi022 ceph-mon[40250]: pgmap v17817: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:32.631 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:32 smithi122 ceph-mon[32255]: pgmap v17817: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:52:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:52:33.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:33.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:33.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:33.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:52:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:52:33.271 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2625306769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:33.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2625306769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2625306769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:34.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:34 smithi022 ceph-mon[40250]: pgmap v17818: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:34 smithi022 ceph-mon[31360]: pgmap v17818: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:34 smithi122 ceph-mon[32255]: pgmap v17818: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:52:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:52:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:52:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:52:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:52:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:52:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:52:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:52:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:52:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:52:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:52:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:52:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:36 smithi122 ceph-mon[32255]: pgmap v17819: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:36 smithi022 ceph-mon[31360]: pgmap v17819: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:36 smithi022 ceph-mon[40250]: pgmap v17819: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:52:38.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:38.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:38.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:38.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:38 smithi122 ceph-mon[32255]: pgmap v17820: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:38 smithi022 ceph-mon[31360]: pgmap v17820: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:38 smithi022 ceph-mon[40250]: pgmap v17820: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:39.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2963622248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2963622248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2963622248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:40 smithi122 ceph-mon[32255]: pgmap v17821: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:40] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T03:52:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:40 smithi022 ceph-mon[31360]: pgmap v17821: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:40 smithi022 ceph-mon[40250]: pgmap v17821: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:42 smithi122 ceph-mon[32255]: pgmap v17822: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:52:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:52:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:42 smithi022 ceph-mon[31360]: pgmap v17822: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:42 smithi022 ceph-mon[40250]: pgmap v17822: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:43.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:52:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:52:43.380 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:43.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:43.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:43.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:44 smithi122 ceph-mon[32255]: pgmap v17823: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2169345891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:44 smithi022 ceph-mon[31360]: pgmap v17823: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2169345891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:44 smithi022 ceph-mon[40250]: pgmap v17823: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2169345891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:46 smithi122 ceph-mon[32255]: pgmap v17824: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:46 smithi022 ceph-mon[31360]: pgmap v17824: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:46 smithi022 ceph-mon[40250]: pgmap v17824: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:52:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:48 smithi122 ceph-mon[32255]: pgmap v17825: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:48.749 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:48.750 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:48 smithi022 ceph-mon[31360]: pgmap v17825: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:48.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:48 smithi022 ceph-mon[40250]: pgmap v17825: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2113822007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2113822007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2113822007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:50 smithi122 ceph-mon[32255]: pgmap v17826: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:50] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T03:52:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:50 smithi022 ceph-mon[31360]: pgmap v17826: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:50 smithi022 ceph-mon[40250]: pgmap v17826: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:52 smithi122 ceph-mon[32255]: pgmap v17827: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:52:52.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:52:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:52 smithi022 ceph-mon[31360]: pgmap v17827: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:52 smithi022 ceph-mon[40250]: pgmap v17827: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:53.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:52:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:52:54.122 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:54.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:54.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:54.498 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:54.499 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:54.499 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:54 smithi022 ceph-mon[31360]: pgmap v17828: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:54.499 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:54.500 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:54 smithi022 ceph-mon[40250]: pgmap v17828: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:52:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:54 smithi122 ceph-mon[32255]: pgmap v17828: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2004209364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2004209364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2004209364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:52:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:56 smithi122 ceph-mon[32255]: pgmap v17829: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:56 smithi022 ceph-mon[31360]: pgmap v17829: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:56 smithi022 ceph-mon[40250]: pgmap v17829: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:52:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:58 smithi122 ceph-mon[32255]: pgmap v17830: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:52:58.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:52:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:58 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:52:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[31360]: pgmap v17830: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:52:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:52:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:52:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[40250]: pgmap v17830: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:52:58.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:52:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:52:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:58 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:52:59.500 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:52:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:52:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:52:59 smithi022 ceph-mon[31360]: pgmap v17831: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:52:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:52:59 smithi022 ceph-mon[40250]: pgmap v17831: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:52:59.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:52:59.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:52:59.868 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:52:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:52:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:52:59 smithi122 ceph-mon[32255]: pgmap v17831: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:00] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T03:53:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3154069921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3154069921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3154069921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:01 smithi022 ceph-mon[31360]: pgmap v17832: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:01 smithi022 ceph-mon[40250]: pgmap v17832: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:01 smithi122 ceph-mon[32255]: pgmap v17832: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:53:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:53:03.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:53:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:53:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:04 smithi022 ceph-mon[31360]: pgmap v17833: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:04 smithi022 ceph-mon[40250]: pgmap v17833: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:04 smithi122 ceph-mon[32255]: pgmap v17833: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:04.870 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:05.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:05.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:05.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:06 smithi022 ceph-mon[31360]: pgmap v17834: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:06.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1345853576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:06 smithi022 ceph-mon[40250]: pgmap v17834: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:06.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1345853576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:06 smithi122 ceph-mon[32255]: pgmap v17834: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1345853576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:53:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:08 smithi022 ceph-mon[31360]: pgmap v17835: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:08 smithi022 ceph-mon[40250]: pgmap v17835: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:08 smithi122 ceph-mon[32255]: pgmap v17835: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:10 smithi122 ceph-mon[32255]: pgmap v17836: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:10] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T03:53:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:10 smithi022 ceph-mon[31360]: pgmap v17836: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:10.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:10 smithi022 ceph-mon[40250]: pgmap v17836: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:10.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:10.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:10.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3752593721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3752593721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3752593721' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:12.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:12 smithi022 ceph-mon[31360]: pgmap v17837: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:12.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:12 smithi022 ceph-mon[40250]: pgmap v17837: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:12 smithi122 ceph-mon[32255]: pgmap v17837: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:53:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:53:13.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:53:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:53:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:14 smithi022 ceph-mon[31360]: pgmap v17838: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:14.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:14 smithi022 ceph-mon[40250]: pgmap v17838: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:14 smithi122 ceph-mon[32255]: pgmap v17838: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:15.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:15.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:15.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:15.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:16 smithi022 ceph-mon[31360]: pgmap v17839: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3874593727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:16 smithi022 ceph-mon[40250]: pgmap v17839: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3874593727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:16 smithi122 ceph-mon[32255]: pgmap v17839: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3874593727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:53:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:18 smithi022 ceph-mon[31360]: pgmap v17840: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:18 smithi022 ceph-mon[40250]: pgmap v17840: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:18 smithi122 ceph-mon[32255]: pgmap v17840: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:20] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:53:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:20 smithi122 ceph-mon[32255]: pgmap v17841: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:20 smithi022 ceph-mon[31360]: pgmap v17841: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:20 smithi022 ceph-mon[40250]: pgmap v17841: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:20.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:21.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:21.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:21.366 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:22 smithi022 ceph-mon[31360]: pgmap v17842: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/411758948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:22 smithi022 ceph-mon[40250]: pgmap v17842: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/411758948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:53:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:53:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:22 smithi122 ceph-mon[32255]: pgmap v17842: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:22.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/411758948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:23.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:53:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:53:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:24 smithi022 ceph-mon[31360]: pgmap v17843: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:24 smithi022 ceph-mon[40250]: pgmap v17843: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:24 smithi122 ceph-mon[32255]: pgmap v17843: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:26.368 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:26 smithi022 ceph-mon[31360]: pgmap v17844: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:26 smithi022 ceph-mon[40250]: pgmap v17844: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:26 smithi122 ceph-mon[32255]: pgmap v17844: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:26.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:26.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:26.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3547549361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:53:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3547549361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3547549361' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:28.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:28 smithi022 ceph-mon[31360]: pgmap v17845: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:28.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:28 smithi022 ceph-mon[40250]: pgmap v17845: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:28 smithi122 ceph-mon[32255]: pgmap v17845: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:30] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:53:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:30 smithi122 ceph-mon[32255]: pgmap v17846: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:30 smithi022 ceph-mon[31360]: pgmap v17846: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:30 smithi022 ceph-mon[40250]: pgmap v17846: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:31.743 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:32.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:32.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:32.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:32.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:32 smithi022 ceph-mon[31360]: pgmap v17847: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:32.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3434335618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:32 smithi022 ceph-mon[40250]: pgmap v17847: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:32.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3434335618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:53:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:53:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:32 smithi122 ceph-mon[32255]: pgmap v17847: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3434335618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:33.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:53:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:53:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:34 smithi022 ceph-mon[31360]: pgmap v17848: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:34 smithi022 ceph-mon[40250]: pgmap v17848: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:34 smithi122 ceph-mon[32255]: pgmap v17848: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:53:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:53:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:53:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:53:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:53:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:53:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:53:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:53:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:53:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:53:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:53:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:53:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:36 smithi022 ceph-mon[31360]: pgmap v17849: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:36 smithi022 ceph-mon[40250]: pgmap v17849: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:36 smithi122 ceph-mon[32255]: pgmap v17849: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:37.114 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:37.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:37.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:37.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:53:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:38 smithi022 ceph-mon[31360]: pgmap v17850: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1770426615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:38 smithi022 ceph-mon[40250]: pgmap v17850: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1770426615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:38 smithi122 ceph-mon[32255]: pgmap v17850: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1770426615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:40] "GET /metrics HTTP/1.1" 200 202488 "" "Prometheus/2.18.1" 2021-12-16T03:53:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:40 smithi122 ceph-mon[32255]: pgmap v17851: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:40 smithi022 ceph-mon[31360]: pgmap v17851: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:40 smithi022 ceph-mon[40250]: pgmap v17851: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:42.486 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:42 smithi122 ceph-mon[32255]: pgmap v17852: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:53:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:53:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:42 smithi022 ceph-mon[31360]: pgmap v17852: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:42 smithi022 ceph-mon[40250]: pgmap v17852: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:42.858 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:42.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:42.859 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:53:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:53:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3457430928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3457430928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3457430928' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:44 smithi122 ceph-mon[32255]: pgmap v17853: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:44 smithi022 ceph-mon[31360]: pgmap v17853: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:44 smithi022 ceph-mon[40250]: pgmap v17853: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:46 smithi122 ceph-mon[32255]: pgmap v17854: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:46 smithi022 ceph-mon[31360]: pgmap v17854: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:46 smithi022 ceph-mon[40250]: pgmap v17854: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:53:47.861 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:48.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:48.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:48.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:48 smithi022 ceph-mon[31360]: pgmap v17855: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/80857118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:48 smithi022 ceph-mon[40250]: pgmap v17855: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/80857118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:48 smithi122 ceph-mon[32255]: pgmap v17855: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/80857118' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:50] "GET /metrics HTTP/1.1" 200 202485 "" "Prometheus/2.18.1" 2021-12-16T03:53:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:50 smithi122 ceph-mon[32255]: pgmap v17856: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:50 smithi022 ceph-mon[31360]: pgmap v17856: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:50 smithi022 ceph-mon[40250]: pgmap v17856: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:52 smithi122 ceph-mon[32255]: pgmap v17857: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:53:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:53:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:52 smithi022 ceph-mon[31360]: pgmap v17857: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:52 smithi022 ceph-mon[40250]: pgmap v17857: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:53.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:53:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:53:53.234 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:53.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:53.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:53.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:54 smithi122 ceph-mon[32255]: pgmap v17858: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2880414717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:54 smithi022 ceph-mon[31360]: pgmap v17858: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2880414717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:53:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:54 smithi022 ceph-mon[40250]: pgmap v17858: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2880414717' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:56 smithi122 ceph-mon[32255]: pgmap v17859: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:56 smithi022 ceph-mon[31360]: pgmap v17859: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:56 smithi022 ceph-mon[40250]: pgmap v17859: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:53:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:53:58.570 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:58 smithi122 ceph-mon[32255]: pgmap v17860: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:58.571 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:58 smithi022 ceph-mon[31360]: pgmap v17860: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:58.572 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:58 smithi022 ceph-mon[40250]: pgmap v17860: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:53:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:53:58.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:53:58.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:53:58.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:53:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:53:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:53:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:53:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3634116771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:53:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:53:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:53:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:53:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:53:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3634116771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:53:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:53:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:53:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:53:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3634116771' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:53:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:53:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:54:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:00 smithi122 ceph-mon[32255]: pgmap v17861: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:00] "GET /metrics HTTP/1.1" 200 202485 "" "Prometheus/2.18.1" 2021-12-16T03:54:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:00 smithi022 ceph-mon[31360]: pgmap v17861: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:00 smithi022 ceph-mon[40250]: pgmap v17861: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:02 smithi122 ceph-mon[32255]: pgmap v17862: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:54:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:54:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:02 smithi022 ceph-mon[31360]: pgmap v17862: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:02 smithi022 ceph-mon[40250]: pgmap v17862: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:03.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:54:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:54:03.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:04.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:04.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:04.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:04 smithi122 ceph-mon[32255]: pgmap v17863: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:04 smithi022 ceph-mon[31360]: pgmap v17863: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:04 smithi022 ceph-mon[40250]: pgmap v17863: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3998466056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3998466056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3998466056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:06 smithi122 ceph-mon[32255]: pgmap v17864: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:06 smithi022 ceph-mon[31360]: pgmap v17864: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:06 smithi022 ceph-mon[40250]: pgmap v17864: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:54:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:08 smithi122 ceph-mon[32255]: pgmap v17865: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:08 smithi022 ceph-mon[31360]: pgmap v17865: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:08 smithi022 ceph-mon[40250]: pgmap v17865: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:09.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:09.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:09.721 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:10] "GET /metrics HTTP/1.1" 200 202488 "" "Prometheus/2.18.1" 2021-12-16T03:54:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:10 smithi122 ceph-mon[32255]: pgmap v17866: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2163848386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:10 smithi022 ceph-mon[31360]: pgmap v17866: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2163848386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:10 smithi022 ceph-mon[40250]: pgmap v17866: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2163848386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:12 smithi122 ceph-mon[32255]: pgmap v17867: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:54:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:54:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:12 smithi022 ceph-mon[31360]: pgmap v17867: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:12 smithi022 ceph-mon[40250]: pgmap v17867: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:13.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:54:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:54:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:14 smithi122 ceph-mon[32255]: pgmap v17868: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:14.722 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:14.723 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:14.723 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:14 smithi022 ceph-mon[31360]: pgmap v17868: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:14.724 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:14.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:14 smithi022 ceph-mon[40250]: pgmap v17868: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:15.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:15.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:15.093 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:15.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/427702196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:15.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/427702196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/427702196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:16 smithi122 ceph-mon[32255]: pgmap v17869: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:16 smithi022 ceph-mon[31360]: pgmap v17869: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:16 smithi022 ceph-mon[40250]: pgmap v17869: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:54:18.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:18 smithi122 ceph-mon[32255]: pgmap v17870: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:18 smithi022 ceph-mon[31360]: pgmap v17870: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:18 smithi022 ceph-mon[40250]: pgmap v17870: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:20.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:20] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:54:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:20 smithi122 ceph-mon[32255]: pgmap v17871: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:20.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:20.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:20.465 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:20 smithi022 ceph-mon[31360]: pgmap v17871: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:20 smithi022 ceph-mon[40250]: pgmap v17871: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/905679692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/905679692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/905679692' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:22 smithi122 ceph-mon[32255]: pgmap v17872: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:54:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:54:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:22 smithi022 ceph-mon[31360]: pgmap v17872: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:22 smithi022 ceph-mon[40250]: pgmap v17872: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:23.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:54:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:54:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:24 smithi122 ceph-mon[32255]: pgmap v17873: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:24 smithi022 ceph-mon[31360]: pgmap v17873: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:24 smithi022 ceph-mon[40250]: pgmap v17873: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:25.467 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:25.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:25.834 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:25.835 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:26 smithi122 ceph-mon[32255]: pgmap v17874: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1789245531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:26 smithi022 ceph-mon[31360]: pgmap v17874: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1789245531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:26 smithi022 ceph-mon[40250]: pgmap v17874: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1789245531' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:54:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:28 smithi122 ceph-mon[32255]: pgmap v17875: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:28 smithi022 ceph-mon[31360]: pgmap v17875: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:28 smithi022 ceph-mon[40250]: pgmap v17875: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:30 smithi122 ceph-mon[32255]: pgmap v17876: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:30] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:54:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:30 smithi022 ceph-mon[31360]: pgmap v17876: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:30 smithi022 ceph-mon[40250]: pgmap v17876: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:30.836 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:31.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:31.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:31.205 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/961464510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/961464510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/961464510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:32 smithi122 ceph-mon[32255]: pgmap v17877: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:54:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:54:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:32 smithi022 ceph-mon[31360]: pgmap v17877: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:32 smithi022 ceph-mon[40250]: pgmap v17877: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:33.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:54:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:54:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:34 smithi122 ceph-mon[32255]: pgmap v17878: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:34 smithi022 ceph-mon[31360]: pgmap v17878: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:34 smithi022 ceph-mon[40250]: pgmap v17878: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:54:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:54:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:54:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:54:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:54:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:54:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:54:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:54:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:54:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:54:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:54:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:54:36.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:36 smithi022 ceph-mon[31360]: pgmap v17879: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:36 smithi022 ceph-mon[40250]: pgmap v17879: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:36.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:36.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:36.579 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:36.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:36 smithi122 ceph-mon[32255]: pgmap v17879: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3548393775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3548393775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:54:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3548393775' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:38 smithi122 ceph-mon[32255]: pgmap v17880: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:38 smithi022 ceph-mon[31360]: pgmap v17880: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:38 smithi022 ceph-mon[40250]: pgmap v17880: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:40 smithi122 ceph-mon[32255]: pgmap v17881: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:40] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:54:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:40 smithi022 ceph-mon[31360]: pgmap v17881: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:40 smithi022 ceph-mon[40250]: pgmap v17881: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:41.580 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:41.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:41.948 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:41.949 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:42 smithi122 ceph-mon[32255]: pgmap v17882: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1947288243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:54:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:54:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:42 smithi022 ceph-mon[31360]: pgmap v17882: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1947288243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:42 smithi022 ceph-mon[40250]: pgmap v17882: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1947288243' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:54:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:54:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:44 smithi122 ceph-mon[32255]: pgmap v17883: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:44 smithi022 ceph-mon[31360]: pgmap v17883: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:44 smithi022 ceph-mon[40250]: pgmap v17883: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:46 smithi122 ceph-mon[32255]: pgmap v17884: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:46 smithi022 ceph-mon[31360]: pgmap v17884: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:46 smithi022 ceph-mon[40250]: pgmap v17884: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:46.950 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:47.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:47.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:47.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4150563990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4150563990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:54:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4150563990' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:48 smithi122 ceph-mon[32255]: pgmap v17885: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:48 smithi022 ceph-mon[31360]: pgmap v17885: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:48 smithi022 ceph-mon[40250]: pgmap v17885: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:50 smithi122 ceph-mon[32255]: pgmap v17886: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:50] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T03:54:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:50 smithi022 ceph-mon[31360]: pgmap v17886: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:50 smithi022 ceph-mon[40250]: pgmap v17886: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:52 smithi122 ceph-mon[32255]: pgmap v17887: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:54:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:54:52.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:52.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:52.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:52.698 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:52 smithi022 ceph-mon[31360]: pgmap v17887: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:52.698 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:52 smithi022 ceph-mon[40250]: pgmap v17887: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:53.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:54:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:54:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/870942319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/870942319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/870942319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:54 smithi122 ceph-mon[32255]: pgmap v17888: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:54 smithi022 ceph-mon[31360]: pgmap v17888: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:54:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:54 smithi022 ceph-mon[40250]: pgmap v17888: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:56 smithi122 ceph-mon[32255]: pgmap v17889: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:56 smithi022 ceph-mon[31360]: pgmap v17889: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:56 smithi022 ceph-mon[40250]: pgmap v17889: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:54:57.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:54:57.699 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:54:58.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:54:58.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:54:58.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:54:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:58 smithi122 ceph-mon[32255]: pgmap v17890: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3841455943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:58 smithi022 ceph-mon[31360]: pgmap v17890: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3841455943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:58 smithi022 ceph-mon[40250]: pgmap v17890: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:54:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3841455943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:54:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:54:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:54:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:59 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:54:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:54:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:54:59 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:54:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:54:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:54:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:54:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:54:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:54:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:54:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:54:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:54:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:54:59.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:54:59 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:55:00.425 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:00] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T03:55:00.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:00 smithi122 ceph-mon[32255]: pgmap v17891: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:00.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:55:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:00 smithi022 ceph-mon[31360]: pgmap v17891: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:55:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:00 smithi022 ceph-mon[40250]: pgmap v17891: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:55:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:02 smithi122 ceph-mon[32255]: pgmap v17892: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:55:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:55:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:02 smithi022 ceph-mon[31360]: pgmap v17892: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:02 smithi022 ceph-mon[40250]: pgmap v17892: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:03.071 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:03.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:55:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:55:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:04 smithi122 ceph-mon[32255]: pgmap v17893: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/298605879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:04 smithi022 ceph-mon[31360]: pgmap v17893: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/298605879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:04 smithi022 ceph-mon[40250]: pgmap v17893: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/298605879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:06 smithi122 ceph-mon[32255]: pgmap v17894: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:06 smithi022 ceph-mon[31360]: pgmap v17894: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:06 smithi022 ceph-mon[40250]: pgmap v17894: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:55:08.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:08 smithi022 ceph-mon[31360]: pgmap v17895: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:08 smithi022 ceph-mon[40250]: pgmap v17895: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:08.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:08.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:08.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:08 smithi122 ceph-mon[32255]: pgmap v17895: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1427291569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1427291569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1427291569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:10] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:55:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:10 smithi022 ceph-mon[31360]: pgmap v17896: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:10 smithi022 ceph-mon[40250]: pgmap v17896: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:10 smithi122 ceph-mon[32255]: pgmap v17896: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:12 smithi122 ceph-mon[32255]: pgmap v17897: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:55:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:55:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:12 smithi022 ceph-mon[31360]: pgmap v17897: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:12 smithi022 ceph-mon[40250]: pgmap v17897: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:13.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:55:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:55:13.821 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:14.191 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:14.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:14.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:14 smithi022 ceph-mon[31360]: pgmap v17898: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1606968842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:14 smithi022 ceph-mon[40250]: pgmap v17898: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1606968842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:14 smithi122 ceph-mon[32255]: pgmap v17898: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1606968842' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:16 smithi022 ceph-mon[31360]: pgmap v17899: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:16 smithi022 ceph-mon[40250]: pgmap v17899: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:16 smithi122 ceph-mon[32255]: pgmap v17899: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:55:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:18 smithi022 ceph-mon[31360]: pgmap v17900: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:18 smithi022 ceph-mon[40250]: pgmap v17900: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:18 smithi122 ceph-mon[32255]: pgmap v17900: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:19.193 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:19.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:19.566 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:19.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:20] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:55:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:20 smithi022 ceph-mon[31360]: pgmap v17901: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2921041805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:20 smithi022 ceph-mon[40250]: pgmap v17901: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2921041805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:20 smithi122 ceph-mon[32255]: pgmap v17901: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2921041805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:22 smithi122 ceph-mon[32255]: pgmap v17902: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:55:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:55:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:22 smithi022 ceph-mon[31360]: pgmap v17902: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:22 smithi022 ceph-mon[40250]: pgmap v17902: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:23.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:55:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:55:24.568 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:24 smithi022 ceph-mon[31360]: pgmap v17903: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:24 smithi022 ceph-mon[40250]: pgmap v17903: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:24 smithi122 ceph-mon[32255]: pgmap v17903: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:24.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:24.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:24.937 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3354599946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:25.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3354599946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3354599946' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:26 smithi022 ceph-mon[31360]: pgmap v17904: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:26 smithi022 ceph-mon[40250]: pgmap v17904: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:26 smithi122 ceph-mon[32255]: pgmap v17904: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:55:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:28 smithi022 ceph-mon[31360]: pgmap v17905: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:28 smithi022 ceph-mon[40250]: pgmap v17905: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:28 smithi122 ceph-mon[32255]: pgmap v17905: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:29.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:30.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:30] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:55:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:30 smithi022 ceph-mon[31360]: pgmap v17906: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1076091227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:30 smithi022 ceph-mon[40250]: pgmap v17906: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1076091227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:30 smithi122 ceph-mon[32255]: pgmap v17906: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1076091227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:31 smithi022 ceph-mon[31360]: pgmap v17907: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:31 smithi022 ceph-mon[40250]: pgmap v17907: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:31 smithi122 ceph-mon[32255]: pgmap v17907: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:55:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:55:33.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:55:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:55:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:34 smithi022 ceph-mon[31360]: pgmap v17908: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:34 smithi022 ceph-mon[40250]: pgmap v17908: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:34 smithi122 ceph-mon[32255]: pgmap v17908: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:35.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:55:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:55:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:55:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:55:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:55:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:55:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:55:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:55:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:55:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:55:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:55:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:55:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:36 smithi022 ceph-mon[31360]: pgmap v17909: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2798211645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:36 smithi022 ceph-mon[40250]: pgmap v17909: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:36.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2798211645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:36 smithi122 ceph-mon[32255]: pgmap v17909: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2798211645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:55:38.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:38 smithi022 ceph-mon[31360]: pgmap v17910: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:38.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:38 smithi022 ceph-mon[40250]: pgmap v17910: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:38 smithi122 ceph-mon[32255]: pgmap v17910: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:40 smithi122 ceph-mon[32255]: pgmap v17911: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:40] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:55:40.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:40 smithi022 ceph-mon[31360]: pgmap v17911: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:40.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:40 smithi022 ceph-mon[40250]: pgmap v17911: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:41.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:41.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:41.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2776380671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2776380671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2776380671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:42 smithi022 ceph-mon[31360]: pgmap v17912: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:42 smithi022 ceph-mon[40250]: pgmap v17912: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:42 smithi122 ceph-mon[32255]: pgmap v17912: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:55:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:55:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:55:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:55:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:44.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:44 smithi022 ceph-mon[31360]: pgmap v17913: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:44.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:44 smithi022 ceph-mon[40250]: pgmap v17913: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:44 smithi122 ceph-mon[32255]: pgmap v17913: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:46.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:46.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:46.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:46.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:46.425 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:46 smithi022 ceph-mon[40250]: pgmap v17914: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:46.425 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:46 smithi022 ceph-mon[31360]: pgmap v17914: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:46 smithi122 ceph-mon[32255]: pgmap v17914: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1788345080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:55:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1788345080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1788345080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:48 smithi122 ceph-mon[32255]: pgmap v17915: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:48 smithi022 ceph-mon[31360]: pgmap v17915: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:48 smithi022 ceph-mon[40250]: pgmap v17915: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:50 smithi122 ceph-mon[32255]: pgmap v17916: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:50] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:55:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:50 smithi022 ceph-mon[31360]: pgmap v17916: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:50 smithi022 ceph-mon[40250]: pgmap v17916: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:51.426 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:51.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:51.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:51.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:55:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:55:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:52 smithi122 ceph-mon[32255]: pgmap v17917: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1191847326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:52 smithi022 ceph-mon[31360]: pgmap v17917: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1191847326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:52 smithi022 ceph-mon[40250]: pgmap v17917: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1191847326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:53.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:55:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:55:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:54 smithi122 ceph-mon[32255]: pgmap v17918: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:54 smithi022 ceph-mon[31360]: pgmap v17918: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:55:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:54 smithi022 ceph-mon[40250]: pgmap v17918: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:56 smithi122 ceph-mon[32255]: pgmap v17919: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:56 smithi022 ceph-mon[31360]: pgmap v17919: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:56 smithi022 ceph-mon[40250]: pgmap v17919: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:55:56.796 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:55:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:55:57.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:55:57.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:55:57.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3298149442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:55:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3298149442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3298149442' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:55:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:55:58 smithi122 ceph-mon[32255]: pgmap v17920: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:55:58 smithi022 ceph-mon[31360]: pgmap v17920: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:55:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:55:58 smithi022 ceph-mon[40250]: pgmap v17920: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:00] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:56:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:00 smithi122 ceph-mon[32255]: pgmap v17921: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:56:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:56:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:56:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[31360]: pgmap v17921: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:56:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:56:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:56:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[40250]: pgmap v17921: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:56:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:56:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:56:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:56:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:56:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:56:02.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:02 smithi022 ceph-mon[31360]: pgmap v17922: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:02 smithi022 ceph-mon[40250]: pgmap v17922: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:02.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:02.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:02.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:02 smithi122 ceph-mon[32255]: pgmap v17922: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:56:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:56:03.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:56:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:56:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2612865525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2612865525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2612865525' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:04 smithi122 ceph-mon[32255]: pgmap v17923: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:04 smithi022 ceph-mon[31360]: pgmap v17923: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:04 smithi022 ceph-mon[40250]: pgmap v17923: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:06 smithi122 ceph-mon[32255]: pgmap v17924: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:06 smithi022 ceph-mon[31360]: pgmap v17924: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:06 smithi022 ceph-mon[40250]: pgmap v17924: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:07.544 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:56:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:08 smithi122 ceph-mon[32255]: pgmap v17925: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3565869952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:08 smithi022 ceph-mon[31360]: pgmap v17925: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3565869952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:08 smithi022 ceph-mon[40250]: pgmap v17925: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3565869952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:10 smithi122 ceph-mon[32255]: pgmap v17926: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:10] "GET /metrics HTTP/1.1" 200 202488 "" "Prometheus/2.18.1" 2021-12-16T03:56:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:10 smithi022 ceph-mon[31360]: pgmap v17926: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:10 smithi022 ceph-mon[40250]: pgmap v17926: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:12 smithi122 ceph-mon[32255]: pgmap v17927: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:56:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:56:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:12 smithi022 ceph-mon[31360]: pgmap v17927: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:12 smithi022 ceph-mon[40250]: pgmap v17927: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:12.914 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:13.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:56:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:56:13.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:13.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:13.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:14.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:14 smithi022 ceph-mon[31360]: pgmap v17928: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2086030506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:14 smithi022 ceph-mon[40250]: pgmap v17928: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2086030506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:14 smithi122 ceph-mon[32255]: pgmap v17928: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:14.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2086030506' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:16.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:16 smithi022 ceph-mon[31360]: pgmap v17929: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:16 smithi022 ceph-mon[40250]: pgmap v17929: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:16 smithi122 ceph-mon[32255]: pgmap v17929: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:56:18.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:18 smithi022 ceph-mon[31360]: pgmap v17930: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:18 smithi022 ceph-mon[40250]: pgmap v17930: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:18.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:18.650 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:18.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:18 smithi122 ceph-mon[32255]: pgmap v17930: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1151900299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1151900299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1151900299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:20 smithi122 ceph-mon[32255]: pgmap v17931: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:20] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:56:20.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:20 smithi022 ceph-mon[31360]: pgmap v17931: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:20 smithi022 ceph-mon[40250]: pgmap v17931: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:22 smithi022 ceph-mon[31360]: pgmap v17932: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:22 smithi022 ceph-mon[40250]: pgmap v17932: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:22 smithi122 ceph-mon[32255]: pgmap v17932: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:56:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:56:23.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:56:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:56:23.652 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:24.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:24.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:24.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:24 smithi122 ceph-mon[32255]: pgmap v17933: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1167065647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:24 smithi022 ceph-mon[31360]: pgmap v17933: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1167065647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:24 smithi022 ceph-mon[40250]: pgmap v17933: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1167065647' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:26 smithi122 ceph-mon[32255]: pgmap v17934: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:26 smithi022 ceph-mon[31360]: pgmap v17934: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:26 smithi022 ceph-mon[40250]: pgmap v17934: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:56:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:28 smithi122 ceph-mon[32255]: pgmap v17935: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:28 smithi022 ceph-mon[31360]: pgmap v17935: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:28 smithi022 ceph-mon[40250]: pgmap v17935: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:30 smithi122 ceph-mon[32255]: pgmap v17936: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/78819286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:30] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:56:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:30 smithi022 ceph-mon[31360]: pgmap v17936: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/78819286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:30 smithi022 ceph-mon[40250]: pgmap v17936: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:30.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/78819286' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:32 smithi122 ceph-mon[32255]: pgmap v17937: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:56:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:56:32.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:32 smithi022 ceph-mon[31360]: pgmap v17937: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:32.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:32 smithi022 ceph-mon[40250]: pgmap v17937: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:33.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:56:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:56:34.397 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:34 smithi122 ceph-mon[32255]: pgmap v17938: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:34.766 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:34.766 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:34 smithi022 ceph-mon[31360]: pgmap v17938: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:34.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:34.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:34 smithi022 ceph-mon[40250]: pgmap v17938: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:56:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:56:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2788854281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:56:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:56:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:56:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:56:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2788854281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:56:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:56:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:56:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:56:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2788854281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:36 smithi122 ceph-mon[32255]: pgmap v17939: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:36 smithi022 ceph-mon[31360]: pgmap v17939: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:36 smithi022 ceph-mon[40250]: pgmap v17939: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:56:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:38 smithi122 ceph-mon[32255]: pgmap v17940: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:38 smithi022 ceph-mon[31360]: pgmap v17940: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:38.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:38 smithi022 ceph-mon[40250]: pgmap v17940: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:39.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:40.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:40.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:40.135 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:40 smithi122 ceph-mon[32255]: pgmap v17941: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/536197368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:40] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T03:56:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:40 smithi022 ceph-mon[31360]: pgmap v17941: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:40.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/536197368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:40 smithi022 ceph-mon[40250]: pgmap v17941: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/536197368' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:42 smithi122 ceph-mon[32255]: pgmap v17942: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:56:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:56:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:42 smithi022 ceph-mon[31360]: pgmap v17942: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:42 smithi022 ceph-mon[40250]: pgmap v17942: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:43.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:56:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:56:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:44 smithi122 ceph-mon[32255]: pgmap v17943: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:44 smithi022 ceph-mon[31360]: pgmap v17943: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:44 smithi022 ceph-mon[40250]: pgmap v17943: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:45.136 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:45.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:45.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:45.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:46 smithi122 ceph-mon[32255]: pgmap v17944: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/797893934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:46 smithi022 ceph-mon[31360]: pgmap v17944: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/797893934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:46 smithi022 ceph-mon[40250]: pgmap v17944: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:46.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/797893934' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:56:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:48 smithi122 ceph-mon[32255]: pgmap v17945: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:48 smithi022 ceph-mon[31360]: pgmap v17945: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:48 smithi022 ceph-mon[40250]: pgmap v17945: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:50 smithi122 ceph-mon[32255]: pgmap v17946: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:50] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:56:50.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:50 smithi022 ceph-mon[31360]: pgmap v17946: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:50 smithi022 ceph-mon[40250]: pgmap v17946: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4147659569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4147659569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4147659569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:52 smithi122 ceph-mon[32255]: pgmap v17947: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:56:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:56:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:52 smithi022 ceph-mon[31360]: pgmap v17947: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:52 smithi022 ceph-mon[40250]: pgmap v17947: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:53.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:56:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:56:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:54 smithi122 ceph-mon[32255]: pgmap v17948: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:54 smithi022 ceph-mon[31360]: pgmap v17948: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:56:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:54 smithi022 ceph-mon[40250]: pgmap v17948: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:55.880 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:56:56.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:56:56.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:56:56.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:56:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:56 smithi022 ceph-mon[31360]: pgmap v17949: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2504436991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:56 smithi022 ceph-mon[40250]: pgmap v17949: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2504436991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:56 smithi122 ceph-mon[32255]: pgmap v17949: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:56:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2504436991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:56:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:56:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:56:58 smithi122 ceph-mon[32255]: pgmap v17950: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:56:58 smithi022 ceph-mon[31360]: pgmap v17950: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:56:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:56:58 smithi022 ceph-mon[40250]: pgmap v17950: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:00] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:57:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:00 smithi122 ceph-mon[32255]: pgmap v17951: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:57:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:57:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:57:00.765 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[31360]: pgmap v17951: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:00.765 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:57:00.766 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:57:00.766 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:57:00.766 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[40250]: pgmap v17951: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:00.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:57:00.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:57:00.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:57:01.252 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:01.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:01.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:01.624 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:57:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:01 smithi022 ceph-mon[31360]: pgmap v17952: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:02.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3266229707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:57:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:01 smithi022 ceph-mon[40250]: pgmap v17952: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:02.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3266229707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:57:02.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:01 smithi122 ceph-mon[32255]: pgmap v17952: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:02.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3266229707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:57:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:57:03.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:57:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:57:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:04 smithi022 ceph-mon[31360]: pgmap v17953: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:04.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:04 smithi022 ceph-mon[40250]: pgmap v17953: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:04 smithi122 ceph-mon[32255]: pgmap v17953: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:06 smithi022 ceph-mon[31360]: pgmap v17954: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:06 smithi022 ceph-mon[40250]: pgmap v17954: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:06.626 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:06 smithi122 ceph-mon[32255]: pgmap v17954: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:06.997 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:06.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:06.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:07.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1052362071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:07.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1052362071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:57:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1052362071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:08 smithi122 ceph-mon[32255]: pgmap v17955: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:08 smithi022 ceph-mon[31360]: pgmap v17955: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:08 smithi022 ceph-mon[40250]: pgmap v17955: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:10 smithi122 ceph-mon[32255]: pgmap v17956: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:10] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:57:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:10 smithi022 ceph-mon[31360]: pgmap v17956: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:10 smithi022 ceph-mon[40250]: pgmap v17956: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:11.999 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:12.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:12.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:12.371 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:12 smithi122 ceph-mon[32255]: pgmap v17957: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:57:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:57:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:12 smithi022 ceph-mon[31360]: pgmap v17957: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:12 smithi022 ceph-mon[40250]: pgmap v17957: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:13.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:57:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:57:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/864610894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/864610894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/864610894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:14 smithi122 ceph-mon[32255]: pgmap v17958: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:14 smithi022 ceph-mon[31360]: pgmap v17958: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:14 smithi022 ceph-mon[40250]: pgmap v17958: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:16 smithi122 ceph-mon[32255]: pgmap v17959: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:16 smithi022 ceph-mon[31360]: pgmap v17959: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:16 smithi022 ceph-mon[40250]: pgmap v17959: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:17.373 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:17.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:17.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:17.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:17.741 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:57:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:18 smithi122 ceph-mon[32255]: pgmap v17960: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/33203298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:18 smithi022 ceph-mon[31360]: pgmap v17960: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/33203298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:18 smithi022 ceph-mon[40250]: pgmap v17960: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/33203298' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:20 smithi122 ceph-mon[32255]: pgmap v17961: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:20] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:57:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:20 smithi022 ceph-mon[31360]: pgmap v17961: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:20 smithi022 ceph-mon[40250]: pgmap v17961: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:22 smithi122 ceph-mon[32255]: pgmap v17962: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:57:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:57:22.741 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:22.742 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:22 smithi022 ceph-mon[31360]: pgmap v17962: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:22.742 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:22 smithi022 ceph-mon[40250]: pgmap v17962: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:23.117 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:23.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:23.118 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:23.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:57:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:57:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2755214432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2755214432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2755214432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:24 smithi122 ceph-mon[32255]: pgmap v17963: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:24.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:24 smithi022 ceph-mon[31360]: pgmap v17963: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:24.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:24.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:24 smithi022 ceph-mon[40250]: pgmap v17963: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:26 smithi122 ceph-mon[32255]: pgmap v17964: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:26 smithi022 ceph-mon[31360]: pgmap v17964: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:26 smithi022 ceph-mon[40250]: pgmap v17964: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:57:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:28.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:28.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:28.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:28.493 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:28 smithi022 ceph-mon[40250]: pgmap v17965: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:28.494 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:28 smithi022 ceph-mon[31360]: pgmap v17965: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:28 smithi122 ceph-mon[32255]: pgmap v17965: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1586150893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:29.811 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1586150893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:29.812 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1586150893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:30 smithi122 ceph-mon[32255]: pgmap v17966: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:30] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T03:57:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:30 smithi022 ceph-mon[31360]: pgmap v17966: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:30 smithi022 ceph-mon[40250]: pgmap v17966: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:32 smithi122 ceph-mon[32255]: pgmap v17967: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:57:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:57:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:32 smithi022 ceph-mon[31360]: pgmap v17967: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:32 smithi022 ceph-mon[40250]: pgmap v17967: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:57:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:57:33.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:33.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:33.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:33.866 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:34 smithi122 ceph-mon[32255]: pgmap v17968: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3456394356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:34 smithi022 ceph-mon[31360]: pgmap v17968: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3456394356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:34 smithi022 ceph-mon[40250]: pgmap v17968: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3456394356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:57:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:57:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:57:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:57:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:57:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:57:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:57:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:57:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:57:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:57:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:57:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:57:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:36 smithi122 ceph-mon[32255]: pgmap v17969: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:36 smithi022 ceph-mon[31360]: pgmap v17969: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:36 smithi022 ceph-mon[40250]: pgmap v17969: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:57:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:38 smithi122 ceph-mon[32255]: pgmap v17970: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:38 smithi022 ceph-mon[31360]: pgmap v17970: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:38 smithi022 ceph-mon[40250]: pgmap v17970: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:38.868 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:39.239 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:39.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:39.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3693094659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3693094659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3693094659' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:40] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T03:57:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:40 smithi122 ceph-mon[32255]: pgmap v17971: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:40 smithi022 ceph-mon[31360]: pgmap v17971: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:40 smithi022 ceph-mon[40250]: pgmap v17971: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:42 smithi122 ceph-mon[32255]: pgmap v17972: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:57:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:57:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:42 smithi022 ceph-mon[31360]: pgmap v17972: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:42 smithi022 ceph-mon[40250]: pgmap v17972: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:57:42 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:57:44.241 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:44.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:44 smithi022 ceph-mon[40250]: pgmap v17973: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:44.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:44 smithi022 ceph-mon[31360]: pgmap v17973: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:44 smithi122 ceph-mon[32255]: pgmap v17973: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/190587664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:45.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/190587664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:45.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/190587664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:46 smithi122 ceph-mon[32255]: pgmap v17974: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:46 smithi022 ceph-mon[31360]: pgmap v17974: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:46 smithi022 ceph-mon[40250]: pgmap v17974: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:57:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:48 smithi122 ceph-mon[32255]: pgmap v17975: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:48 smithi022 ceph-mon[31360]: pgmap v17975: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:48 smithi022 ceph-mon[40250]: pgmap v17975: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:49.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:49.983 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:49.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:49.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:50] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:57:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:50 smithi122 ceph-mon[32255]: pgmap v17976: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1040570987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:50 smithi022 ceph-mon[31360]: pgmap v17976: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1040570987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:50 smithi022 ceph-mon[40250]: pgmap v17976: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1040570987' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:52 smithi122 ceph-mon[32255]: pgmap v17977: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:57:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:57:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:52 smithi022 ceph-mon[31360]: pgmap v17977: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:52 smithi022 ceph-mon[40250]: pgmap v17977: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:57:52 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:57:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:54 smithi122 ceph-mon[32255]: pgmap v17978: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:54.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:54 smithi022 ceph-mon[40250]: pgmap v17978: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:57:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:54 smithi022 ceph-mon[31360]: pgmap v17978: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:54.985 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:57:55.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:57:55.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:57:55.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:57:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3372229272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3372229272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3372229272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:57:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:56 smithi122 ceph-mon[32255]: pgmap v17979: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:56 smithi022 ceph-mon[31360]: pgmap v17979: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:56 smithi022 ceph-mon[40250]: pgmap v17979: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:57:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:57:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:57:58 smithi122 ceph-mon[32255]: pgmap v17980: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:57:58 smithi022 ceph-mon[31360]: pgmap v17980: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:57:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:57:58 smithi022 ceph-mon[40250]: pgmap v17980: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:00.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:00 smithi122 ceph-mon[32255]: pgmap v17981: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:00] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T03:58:00.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:00.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:00.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:00.735 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:00 smithi022 ceph-mon[40250]: pgmap v17981: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:00.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:00 smithi022 ceph-mon[31360]: pgmap v17981: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3855390449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:58:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:58:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:58:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:58:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3855390449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:58:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:58:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:58:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:58:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3855390449' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:58:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:58:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:58:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:58:02.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:02 smithi122 ceph-mon[32255]: pgmap v17982: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:02.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:58:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:58:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:02 smithi022 ceph-mon[31360]: pgmap v17982: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:02 smithi022 ceph-mon[40250]: pgmap v17982: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:58:02 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:58:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:04 smithi122 ceph-mon[32255]: pgmap v17983: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:04 smithi022 ceph-mon[31360]: pgmap v17983: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:04 smithi022 ceph-mon[40250]: pgmap v17983: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:05.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:06.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:06.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:06.105 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:06 smithi122 ceph-mon[32255]: pgmap v17984: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3131108233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:06 smithi022 ceph-mon[31360]: pgmap v17984: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3131108233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:06 smithi022 ceph-mon[40250]: pgmap v17984: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3131108233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:58:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:08 smithi122 ceph-mon[32255]: pgmap v17985: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:08 smithi022 ceph-mon[31360]: pgmap v17985: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:08 smithi022 ceph-mon[40250]: pgmap v17985: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:10] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T03:58:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:10 smithi122 ceph-mon[32255]: pgmap v17986: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:10 smithi022 ceph-mon[40250]: pgmap v17986: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:10 smithi022 ceph-mon[31360]: pgmap v17986: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T03:58:11.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:11.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:11.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:11.474 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:12 smithi122 ceph-mon[32255]: pgmap v17987: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1013971347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:58:12.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:12 smithi022 ceph-mon[31360]: pgmap v17987: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1013971347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:12 smithi022 ceph-mon[40250]: pgmap v17987: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1013971347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:58:12 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:58:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:14 smithi122 ceph-mon[32255]: pgmap v17988: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:14 smithi022 ceph-mon[31360]: pgmap v17988: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:14 smithi022 ceph-mon[40250]: pgmap v17988: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:16.476 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:16 smithi122 ceph-mon[32255]: pgmap v17989: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:16 smithi022 ceph-mon[31360]: pgmap v17989: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:16 smithi022 ceph-mon[40250]: pgmap v17989: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:16.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:16.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:16.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3495662584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3495662584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:58:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3495662584' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:18 smithi122 ceph-mon[32255]: pgmap v17990: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:18 smithi022 ceph-mon[31360]: pgmap v17990: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:18 smithi022 ceph-mon[40250]: pgmap v17990: 132 pgs: 132 active+clean; 582 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:20 smithi122 ceph-mon[32255]: pgmap v17991: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:20] "GET /metrics HTTP/1.1" 200 202478 "" "Prometheus/2.18.1" 2021-12-16T03:58:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:20 smithi022 ceph-mon[31360]: pgmap v17991: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:20 smithi022 ceph-mon[40250]: pgmap v17991: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:21.844 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:22.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:22.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:22.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:22 smithi022 ceph-mon[31360]: pgmap v17992: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2529743635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:22 smithi022 ceph-mon[40250]: pgmap v17992: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2529743635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:22 smithi122 ceph-mon[32255]: pgmap v17992: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2529743635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:58:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:58:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:58:22 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:58:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:24 smithi122 ceph-mon[32255]: pgmap v17993: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:24.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:24 smithi022 ceph-mon[31360]: pgmap v17993: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:24 smithi022 ceph-mon[40250]: pgmap v17993: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T03:58:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:26 smithi122 ceph-mon[32255]: pgmap v17994: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:26 smithi022 ceph-mon[31360]: pgmap v17994: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:26 smithi022 ceph-mon[40250]: pgmap v17994: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T03:58:27.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:58:27.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:27.586 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:27.587 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:28 smithi122 ceph-mon[32255]: pgmap v17995: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4227539080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:28 smithi022 ceph-mon[31360]: pgmap v17995: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4227539080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:28 smithi022 ceph-mon[40250]: pgmap v17995: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4227539080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:30 smithi122 ceph-mon[32255]: pgmap v17996: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:30] "GET /metrics HTTP/1.1" 200 202478 "" "Prometheus/2.18.1" 2021-12-16T03:58:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:30 smithi022 ceph-mon[31360]: pgmap v17996: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:30 smithi022 ceph-mon[40250]: pgmap v17996: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:32.588 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:32 smithi122 ceph-mon[32255]: pgmap v17997: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:58:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:58:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:32 smithi022 ceph-mon[31360]: pgmap v17997: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:32 smithi022 ceph-mon[40250]: pgmap v17997: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:32.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:32.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:32.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:33.425 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:58:32 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:58:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2357190366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2357190366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2357190366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:34 smithi122 ceph-mon[32255]: pgmap v17998: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:34 smithi022 ceph-mon[31360]: pgmap v17998: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:34 smithi022 ceph-mon[40250]: pgmap v17998: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:58:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:58:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:58:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:58:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:58:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:58:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:58:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:58:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:58:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:58:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:58:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:58:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:36 smithi122 ceph-mon[32255]: pgmap v17999: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:36 smithi022 ceph-mon[31360]: pgmap v17999: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:36 smithi022 ceph-mon[40250]: pgmap v17999: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:58:37.959 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:38.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:38.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:38.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:38 smithi022 ceph-mon[31360]: pgmap v18000: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1540311863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:38 smithi022 ceph-mon[40250]: pgmap v18000: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1540311863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:38 smithi122 ceph-mon[32255]: pgmap v18000: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1540311863' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:40] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T03:58:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:40 smithi022 ceph-mon[31360]: pgmap v18001: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:40 smithi022 ceph-mon[40250]: pgmap v18001: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:40 smithi122 ceph-mon[32255]: pgmap v18001: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:42 smithi122 ceph-mon[32255]: pgmap v18002: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:58:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:58:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:42 smithi022 ceph-mon[31360]: pgmap v18002: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:42 smithi022 ceph-mon[40250]: pgmap v18002: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:43.332 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:58:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:58:43.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:43.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:43.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:44 smithi022 ceph-mon[31360]: pgmap v18003: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3309587752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:44 smithi022 ceph-mon[40250]: pgmap v18003: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3309587752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:44 smithi122 ceph-mon[32255]: pgmap v18003: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3309587752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:46 smithi022 ceph-mon[31360]: pgmap v18004: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:46 smithi022 ceph-mon[40250]: pgmap v18004: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:46 smithi122 ceph-mon[32255]: pgmap v18004: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:58:48.706 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:48.707 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:48 smithi022 ceph-mon[31360]: pgmap v18005: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:48.708 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:48 smithi022 ceph-mon[40250]: pgmap v18005: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:48 smithi122 ceph-mon[32255]: pgmap v18005: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:49.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:49.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:49.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/25554695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/25554695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/25554695' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:50] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T03:58:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:50 smithi022 ceph-mon[31360]: pgmap v18006: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:50 smithi022 ceph-mon[40250]: pgmap v18006: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:50 smithi122 ceph-mon[32255]: pgmap v18006: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:52 smithi122 ceph-mon[32255]: pgmap v18007: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:58:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:58:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:52 smithi022 ceph-mon[31360]: pgmap v18007: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:52 smithi022 ceph-mon[40250]: pgmap v18007: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:58:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:58:54.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:54.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:54.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:58:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:54 smithi022 ceph-mon[31360]: pgmap v18008: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2618145929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:54 smithi022 ceph-mon[40250]: pgmap v18008: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2618145929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:54 smithi122 ceph-mon[32255]: pgmap v18008: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:58:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2618145929' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:58:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:56 smithi022 ceph-mon[31360]: pgmap v18009: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:56 smithi022 ceph-mon[40250]: pgmap v18009: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:56 smithi122 ceph-mon[32255]: pgmap v18009: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:58:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:58:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:58:58 smithi022 ceph-mon[31360]: pgmap v18010: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:58:58 smithi022 ceph-mon[40250]: pgmap v18010: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:58:58 smithi122 ceph-mon[32255]: pgmap v18010: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:58:59.455 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:58:59.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:58:59.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:58:59.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:00] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T03:59:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:00 smithi022 ceph-mon[31360]: pgmap v18011: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3222333896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:00 smithi022 ceph-mon[40250]: pgmap v18011: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:00.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3222333896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:00 smithi122 ceph-mon[32255]: pgmap v18011: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3222333896' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:59:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:59:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:59:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:59:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:59:01.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:59:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T03:59:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T03:59:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T03:59:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:02 smithi122 ceph-mon[32255]: pgmap v18012: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:59:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:59:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:02 smithi022 ceph-mon[31360]: pgmap v18012: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:02 smithi022 ceph-mon[40250]: pgmap v18012: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:59:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:59:04.724 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:04 smithi022 ceph-mon[31360]: pgmap v18013: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:04.725 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:04.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:04 smithi022 ceph-mon[40250]: pgmap v18013: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:04.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:04.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:04 smithi122 ceph-mon[32255]: pgmap v18013: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:05.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:05.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:05.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:05 smithi022 ceph-mon[40250]: pgmap v18014: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3402338242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:05 smithi022 ceph-mon[31360]: pgmap v18014: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3402338242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:06.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:06.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:05 smithi122 ceph-mon[32255]: pgmap v18014: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:06.045 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3402338242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:07.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:07.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:07.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T03:59:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:59:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:08 smithi022 ceph-mon[31360]: pgmap v18015: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:08 smithi022 ceph-mon[40250]: pgmap v18015: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:08 smithi122 ceph-mon[32255]: pgmap v18015: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:10] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:59:10.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:10.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:10.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:10 smithi022 ceph-mon[31360]: pgmap v18016: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:10 smithi022 ceph-mon[40250]: pgmap v18016: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:10 smithi122 ceph-mon[32255]: pgmap v18016: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/327996586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/327996586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/327996586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:12 smithi122 ceph-mon[32255]: pgmap v18017: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:59:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:59:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:12 smithi022 ceph-mon[31360]: pgmap v18017: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:12 smithi022 ceph-mon[40250]: pgmap v18017: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:59:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:59:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:14 smithi022 ceph-mon[31360]: pgmap v18018: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:14 smithi022 ceph-mon[40250]: pgmap v18018: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:14 smithi122 ceph-mon[32255]: pgmap v18018: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:15.572 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:15.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:15.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:15.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:16 smithi022 ceph-mon[31360]: pgmap v18019: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3552079687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:16 smithi022 ceph-mon[40250]: pgmap v18019: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3552079687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:16 smithi122 ceph-mon[32255]: pgmap v18019: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3552079687' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:59:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:18 smithi022 ceph-mon[31360]: pgmap v18020: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:18 smithi022 ceph-mon[40250]: pgmap v18020: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:18 smithi122 ceph-mon[32255]: pgmap v18020: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:20] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T03:59:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:20 smithi022 ceph-mon[31360]: pgmap v18021: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:20 smithi022 ceph-mon[40250]: pgmap v18021: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:20 smithi122 ceph-mon[32255]: pgmap v18021: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:20.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:21.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:21.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:21.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:21.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:21 smithi022 ceph-mon[31360]: pgmap v18022: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3222098796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:21 smithi022 ceph-mon[40250]: pgmap v18022: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:21.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3222098796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:21 smithi122 ceph-mon[32255]: pgmap v18022: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:21.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3222098796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:59:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:59:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:59:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:59:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:24 smithi122 ceph-mon[32255]: pgmap v18023: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:24 smithi022 ceph-mon[31360]: pgmap v18023: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:24 smithi022 ceph-mon[40250]: pgmap v18023: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:26.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:26.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:26.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:26.684 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:26.685 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:26 smithi022 ceph-mon[31360]: pgmap v18024: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:26.685 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:26 smithi022 ceph-mon[40250]: pgmap v18024: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:26 smithi122 ceph-mon[32255]: pgmap v18024: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/766252228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/766252228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:59:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/766252228' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:28 smithi122 ceph-mon[32255]: pgmap v18025: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:28 smithi022 ceph-mon[31360]: pgmap v18025: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:28 smithi022 ceph-mon[40250]: pgmap v18025: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:30 smithi122 ceph-mon[32255]: pgmap v18026: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:30] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T03:59:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:30 smithi022 ceph-mon[31360]: pgmap v18026: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:30 smithi022 ceph-mon[40250]: pgmap v18026: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:31.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:32.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:32.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:32.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:59:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:59:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:32 smithi122 ceph-mon[32255]: pgmap v18027: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/269928311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:32 smithi022 ceph-mon[31360]: pgmap v18027: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/269928311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:32 smithi022 ceph-mon[40250]: pgmap v18027: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/269928311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:59:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:59:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:34 smithi122 ceph-mon[32255]: pgmap v18028: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:34 smithi022 ceph-mon[31360]: pgmap v18028: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:34 smithi022 ceph-mon[40250]: pgmap v18028: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:59:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:59:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:59:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:59:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:59:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:59:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:59:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:59:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:59:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:59:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T03:59:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T03:59:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:36 smithi122 ceph-mon[32255]: pgmap v18029: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:36 smithi022 ceph-mon[31360]: pgmap v18029: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:36 smithi022 ceph-mon[40250]: pgmap v18029: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:37.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:37.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:37.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:37.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:59:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:38 smithi122 ceph-mon[32255]: pgmap v18030: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/112774357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:38 smithi022 ceph-mon[31360]: pgmap v18030: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/112774357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:38 smithi022 ceph-mon[40250]: pgmap v18030: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/112774357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:40 smithi122 ceph-mon[32255]: pgmap v18031: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:40] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T03:59:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:40 smithi022 ceph-mon[31360]: pgmap v18031: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:40 smithi022 ceph-mon[40250]: pgmap v18031: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:42.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:42 smithi122 ceph-mon[32255]: pgmap v18032: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:59:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:59:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:42 smithi022 ceph-mon[31360]: pgmap v18032: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:42.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:42 smithi022 ceph-mon[40250]: pgmap v18032: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:42.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:42.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:42.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:43.339 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:59:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:59:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2759630833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2759630833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2759630833' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:44 smithi122 ceph-mon[32255]: pgmap v18033: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:44 smithi022 ceph-mon[31360]: pgmap v18033: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:44 smithi022 ceph-mon[40250]: pgmap v18033: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:46 smithi122 ceph-mon[32255]: pgmap v18034: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:46 smithi022 ceph-mon[31360]: pgmap v18034: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:46 smithi022 ceph-mon[40250]: pgmap v18034: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:59:47.803 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:48.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:48.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:48.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:48 smithi022 ceph-mon[31360]: pgmap v18035: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:48.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2057702523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:48 smithi022 ceph-mon[40250]: pgmap v18035: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:48.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2057702523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:48 smithi122 ceph-mon[32255]: pgmap v18035: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2057702523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:50 smithi122 ceph-mon[32255]: pgmap v18036: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 03:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:50] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T03:59:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:50 smithi022 ceph-mon[31360]: pgmap v18036: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:50 smithi022 ceph-mon[40250]: pgmap v18036: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:52 smithi122 ceph-mon[32255]: pgmap v18037: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 03:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T03:59:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T03:59:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:52 smithi022 ceph-mon[31360]: pgmap v18037: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:52 smithi022 ceph-mon[40250]: pgmap v18037: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:53.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 03:59:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T03:59:53.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:53.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:53.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:54 smithi122 ceph-mon[32255]: pgmap v18038: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2533423919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:54 smithi022 ceph-mon[31360]: pgmap v18038: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2533423919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:54 smithi022 ceph-mon[40250]: pgmap v18038: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T03:59:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2533423919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:56 smithi122 ceph-mon[32255]: pgmap v18039: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:56 smithi022 ceph-mon[31360]: pgmap v18039: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:56 smithi022 ceph-mon[40250]: pgmap v18039: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T03:59:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 03:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:03:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T03:59:58.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T03:59:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:58 smithi122 ceph-mon[32255]: pgmap v18040: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:58 smithi022 ceph-mon[31360]: pgmap v18040: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:58 smithi022 ceph-mon[40250]: pgmap v18040: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T03:59:58.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T03:59:58.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T03:59:58.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T03:59:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 03:59:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2774830671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 03:59:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2774830671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T03:59:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 03:59:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2774830671' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:00] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:00:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:00 smithi122 ceph-mon[32255]: pgmap v18041: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T04:00:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:00 smithi022 ceph-mon[31360]: pgmap v18041: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T04:00:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:00 smithi022 ceph-mon[40250]: pgmap v18041: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T04:00:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:02 smithi122 ceph-mon[32255]: pgmap v18042: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:00:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:00:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:02 smithi022 ceph-mon[31360]: pgmap v18042: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:02 smithi022 ceph-mon[40250]: pgmap v18042: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:03.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:00:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:00:03.920 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:04.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:04.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:04.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:04 smithi122 ceph-mon[32255]: pgmap v18043: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2783191993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:04 smithi022 ceph-mon[31360]: pgmap v18043: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2783191993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:04 smithi022 ceph-mon[40250]: pgmap v18043: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2783191993' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:06 smithi122 ceph-mon[32255]: pgmap v18044: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:06 smithi022 ceph-mon[31360]: pgmap v18044: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:06 smithi022 ceph-mon[40250]: pgmap v18044: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:00:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:00:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:00:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:07 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:00:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:00:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:00:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:00:07.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:00:07.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:00:07.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:00:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:00:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:00:07.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:07 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:00:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:08 smithi122 ceph-mon[32255]: pgmap v18045: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:08 smithi022 ceph-mon[31360]: pgmap v18045: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:08 smithi022 ceph-mon[40250]: pgmap v18045: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:09.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:09.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:09.664 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:09.665 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:10] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T04:00:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:10 smithi122 ceph-mon[32255]: pgmap v18046: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2631543989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:10 smithi022 ceph-mon[31360]: pgmap v18046: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2631543989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:10 smithi022 ceph-mon[40250]: pgmap v18046: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2631543989' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:12 smithi122 ceph-mon[32255]: pgmap v18047: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:00:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:00:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:12 smithi022 ceph-mon[31360]: pgmap v18047: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:12 smithi022 ceph-mon[40250]: pgmap v18047: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:00:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:00:14.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:14 smithi022 ceph-mon[40250]: pgmap v18048: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:14.667 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:14.667 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:14 smithi022 ceph-mon[31360]: pgmap v18048: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:14.668 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:14.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:14.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:14 smithi122 ceph-mon[32255]: pgmap v18048: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:15.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:15.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:15.038 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:15.389 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3525037189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3525037189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3525037189' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:16 smithi122 ceph-mon[32255]: pgmap v18049: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:16 smithi022 ceph-mon[31360]: pgmap v18049: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:16 smithi022 ceph-mon[40250]: pgmap v18049: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:00:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:18 smithi122 ceph-mon[32255]: pgmap v18050: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:18 smithi022 ceph-mon[31360]: pgmap v18050: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:18 smithi022 ceph-mon[40250]: pgmap v18050: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:20.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:20.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:20.407 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:20] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T04:00:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:20 smithi122 ceph-mon[32255]: pgmap v18051: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2440718404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:20 smithi022 ceph-mon[31360]: pgmap v18051: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2440718404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:20 smithi022 ceph-mon[40250]: pgmap v18051: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2440718404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:22 smithi122 ceph-mon[32255]: pgmap v18052: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:00:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:00:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:22 smithi022 ceph-mon[31360]: pgmap v18052: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:22 smithi022 ceph-mon[40250]: pgmap v18052: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:00:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:00:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:24 smithi122 ceph-mon[32255]: pgmap v18053: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:24 smithi022 ceph-mon[31360]: pgmap v18053: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:24 smithi022 ceph-mon[40250]: pgmap v18053: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:25.408 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:25.776 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:26 smithi122 ceph-mon[32255]: pgmap v18054: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3570152758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:26 smithi022 ceph-mon[31360]: pgmap v18054: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3570152758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:26 smithi022 ceph-mon[40250]: pgmap v18054: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3570152758' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:00:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:28 smithi122 ceph-mon[32255]: pgmap v18055: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:28 smithi022 ceph-mon[31360]: pgmap v18055: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:28 smithi022 ceph-mon[40250]: pgmap v18055: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:30 smithi122 ceph-mon[32255]: pgmap v18056: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:30] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T04:00:30.778 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:30 smithi022 ceph-mon[31360]: pgmap v18056: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:30 smithi022 ceph-mon[40250]: pgmap v18056: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:31.146 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:31.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:31.147 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2263125570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2263125570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2263125570' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:32 smithi122 ceph-mon[32255]: pgmap v18057: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:00:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:00:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:32 smithi022 ceph-mon[31360]: pgmap v18057: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:32 smithi022 ceph-mon[40250]: pgmap v18057: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:00:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:00:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:34 smithi122 ceph-mon[32255]: pgmap v18058: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:34 smithi022 ceph-mon[31360]: pgmap v18058: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:34 smithi022 ceph-mon[40250]: pgmap v18058: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:00:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:00:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:00:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:00:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:00:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:00:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:00:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:00:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:00:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:00:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:00:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:00:36.148 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:36.521 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:36.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:36.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:36 smithi122 ceph-mon[32255]: pgmap v18059: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:36 smithi022 ceph-mon[31360]: pgmap v18059: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:36 smithi022 ceph-mon[40250]: pgmap v18059: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3805504186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3805504186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:00:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3805504186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:38 smithi122 ceph-mon[32255]: pgmap v18060: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:38 smithi022 ceph-mon[31360]: pgmap v18060: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:38 smithi022 ceph-mon[40250]: pgmap v18060: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:40.433 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:40] "GET /metrics HTTP/1.1" 200 202485 "" "Prometheus/2.18.1" 2021-12-16T04:00:40.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:40 smithi122 ceph-mon[32255]: pgmap v18061: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:40 smithi022 ceph-mon[31360]: pgmap v18061: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:40 smithi022 ceph-mon[40250]: pgmap v18061: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:41.523 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:41.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:41.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:41.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:42 smithi122 ceph-mon[32255]: pgmap v18062: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3459311595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:00:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:00:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:42 smithi022 ceph-mon[31360]: pgmap v18062: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3459311595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:42 smithi022 ceph-mon[40250]: pgmap v18062: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3459311595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:00:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:00:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:44 smithi022 ceph-mon[31360]: pgmap v18063: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:44 smithi022 ceph-mon[40250]: pgmap v18063: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:44 smithi122 ceph-mon[32255]: pgmap v18063: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:46 smithi022 ceph-mon[31360]: pgmap v18064: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:46 smithi022 ceph-mon[40250]: pgmap v18064: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:46.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:46 smithi122 ceph-mon[32255]: pgmap v18064: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:47.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:47.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:47.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1060576236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:00:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1060576236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1060576236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:48 smithi022 ceph-mon[31360]: pgmap v18065: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:48 smithi022 ceph-mon[40250]: pgmap v18065: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:48 smithi122 ceph-mon[32255]: pgmap v18065: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:50] "GET /metrics HTTP/1.1" 200 202480 "" "Prometheus/2.18.1" 2021-12-16T04:00:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:50 smithi022 ceph-mon[31360]: pgmap v18066: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:50 smithi022 ceph-mon[40250]: pgmap v18066: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:50 smithi122 ceph-mon[32255]: pgmap v18066: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:52.268 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:52.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:52 smithi022 ceph-mon[31360]: pgmap v18067: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:52.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:52 smithi022 ceph-mon[40250]: pgmap v18067: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:52.639 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:52.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:52.640 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:52 smithi122 ceph-mon[32255]: pgmap v18067: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:00:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:00:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:00:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:00:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2108858323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2108858323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:53.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2108858323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:54 smithi022 ceph-mon[31360]: pgmap v18068: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:54 smithi022 ceph-mon[40250]: pgmap v18068: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:54 smithi122 ceph-mon[32255]: pgmap v18068: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:00:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:56 smithi022 ceph-mon[31360]: pgmap v18069: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:56 smithi022 ceph-mon[40250]: pgmap v18069: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:56 smithi122 ceph-mon[32255]: pgmap v18069: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:00:57.641 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:00:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:00:58.009 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:00:58.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:00:58.010 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:00:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:58 smithi022 ceph-mon[40250]: pgmap v18070: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:00:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3558656227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:58 smithi022 ceph-mon[31360]: pgmap v18070: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:00:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3558656227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:00:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:58 smithi122 ceph-mon[32255]: pgmap v18070: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:00:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:00:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3558656227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:00] "GET /metrics HTTP/1.1" 200 202480 "" "Prometheus/2.18.1" 2021-12-16T04:01:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:00 smithi022 ceph-mon[31360]: pgmap v18071: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:00 smithi022 ceph-mon[40250]: pgmap v18071: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:00 smithi122 ceph-mon[32255]: pgmap v18071: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:01:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:01:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:02 smithi122 ceph-mon[32255]: pgmap v18072: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:02 smithi022 ceph-mon[31360]: pgmap v18072: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:02 smithi022 ceph-mon[40250]: pgmap v18072: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:03.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:03.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:03.386 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:03.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:01:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:01:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:04 smithi122 ceph-mon[32255]: pgmap v18073: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/635951364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:04 smithi022 ceph-mon[31360]: pgmap v18073: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/635951364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:04 smithi022 ceph-mon[40250]: pgmap v18073: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/635951364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:06 smithi122 ceph-mon[32255]: pgmap v18074: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:06 smithi022 ceph-mon[31360]: pgmap v18074: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:06 smithi022 ceph-mon[40250]: pgmap v18074: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:01:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:01:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:07 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:01:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:01:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:01:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:01:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:07 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:01:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:01:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:01:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:07 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:01:08.388 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:08.758 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:08.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:08.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:08.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:08 smithi022 ceph-mon[31360]: pgmap v18075: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:08.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:01:08.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:08 smithi022 ceph-mon[40250]: pgmap v18075: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:08.760 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:01:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:08 smithi122 ceph-mon[32255]: pgmap v18075: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:01:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1282845529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1282845529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1282845529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:10] "GET /metrics HTTP/1.1" 200 202481 "" "Prometheus/2.18.1" 2021-12-16T04:01:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:10 smithi022 ceph-mon[31360]: pgmap v18076: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:10 smithi022 ceph-mon[40250]: pgmap v18076: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:10 smithi122 ceph-mon[32255]: pgmap v18076: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:01:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:01:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:12 smithi122 ceph-mon[32255]: pgmap v18077: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:12 smithi022 ceph-mon[31360]: pgmap v18077: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:12 smithi022 ceph-mon[40250]: pgmap v18077: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:01:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:01:13.760 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:14.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:14.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:14.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:14 smithi022 ceph-mon[31360]: pgmap v18078: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2148960810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:14 smithi022 ceph-mon[40250]: pgmap v18078: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2148960810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:14 smithi122 ceph-mon[32255]: pgmap v18078: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2148960810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:16 smithi022 ceph-mon[40250]: pgmap v18079: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:16 smithi022 ceph-mon[31360]: pgmap v18079: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:16 smithi122 ceph-mon[32255]: pgmap v18079: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:01:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:18 smithi022 ceph-mon[31360]: pgmap v18080: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:18 smithi022 ceph-mon[40250]: pgmap v18080: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:18 smithi122 ceph-mon[32255]: pgmap v18080: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:19.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:19.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:19.503 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:20] "GET /metrics HTTP/1.1" 200 202483 "" "Prometheus/2.18.1" 2021-12-16T04:01:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:20 smithi022 ceph-mon[31360]: pgmap v18081: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4126462701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:20 smithi022 ceph-mon[40250]: pgmap v18081: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4126462701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:20 smithi122 ceph-mon[32255]: pgmap v18081: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4126462701' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:22 smithi122 ceph-mon[32255]: pgmap v18082: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:01:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:01:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:22 smithi022 ceph-mon[31360]: pgmap v18082: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:22 smithi022 ceph-mon[40250]: pgmap v18082: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:01:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:01:24.504 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:24 smithi022 ceph-mon[31360]: pgmap v18083: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:24 smithi022 ceph-mon[40250]: pgmap v18083: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:24 smithi122 ceph-mon[32255]: pgmap v18083: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/899135903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/899135903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/899135903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:26 smithi022 ceph-mon[31360]: pgmap v18084: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:26 smithi022 ceph-mon[40250]: pgmap v18084: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:26 smithi122 ceph-mon[32255]: pgmap v18084: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:01:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:28 smithi022 ceph-mon[31360]: pgmap v18085: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:28 smithi022 ceph-mon[40250]: pgmap v18085: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:28 smithi122 ceph-mon[32255]: pgmap v18085: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:29.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:30.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:30.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:30.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:30] "GET /metrics HTTP/1.1" 200 202483 "" "Prometheus/2.18.1" 2021-12-16T04:01:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:30 smithi022 ceph-mon[31360]: pgmap v18086: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/142010882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:30 smithi022 ceph-mon[40250]: pgmap v18086: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/142010882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:30 smithi122 ceph-mon[32255]: pgmap v18086: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/142010882' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:32 smithi122 ceph-mon[32255]: pgmap v18087: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:01:32.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:01:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:32 smithi022 ceph-mon[31360]: pgmap v18087: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:32 smithi022 ceph-mon[40250]: pgmap v18087: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:01:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:01:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[31360]: pgmap v18088: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:01:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:01:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:01:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:01:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[40250]: pgmap v18088: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:01:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:01:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:01:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:01:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:34 smithi122 ceph-mon[32255]: pgmap v18088: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:01:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:01:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:01:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:01:35.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:36 smithi022 ceph-mon[31360]: pgmap v18089: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1825903088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:36 smithi022 ceph-mon[40250]: pgmap v18089: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1825903088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:36 smithi122 ceph-mon[32255]: pgmap v18089: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1825903088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:01:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:38 smithi022 ceph-mon[31360]: pgmap v18090: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:38 smithi022 ceph-mon[40250]: pgmap v18090: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:38 smithi122 ceph-mon[32255]: pgmap v18090: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:40] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:01:40.624 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:40 smithi022 ceph-mon[31360]: pgmap v18091: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:40 smithi022 ceph-mon[40250]: pgmap v18091: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:40 smithi122 ceph-mon[32255]: pgmap v18091: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:40.992 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:40.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:40.993 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/928899964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/928899964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/928899964' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:42 smithi122 ceph-mon[32255]: pgmap v18092: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:01:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:01:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:42 smithi022 ceph-mon[31360]: pgmap v18092: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:42 smithi022 ceph-mon[40250]: pgmap v18092: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:01:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:01:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:44 smithi022 ceph-mon[31360]: pgmap v18093: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:44 smithi022 ceph-mon[40250]: pgmap v18093: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:44 smithi122 ceph-mon[32255]: pgmap v18093: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:45.994 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:46.364 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:46.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:46.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:46 smithi022 ceph-mon[31360]: pgmap v18094: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1929171616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:46 smithi022 ceph-mon[40250]: pgmap v18094: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1929171616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:46 smithi122 ceph-mon[32255]: pgmap v18094: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1929171616' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:01:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:48 smithi022 ceph-mon[31360]: pgmap v18095: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:48 smithi022 ceph-mon[40250]: pgmap v18095: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:48 smithi122 ceph-mon[32255]: pgmap v18095: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:50] "GET /metrics HTTP/1.1" 200 202474 "" "Prometheus/2.18.1" 2021-12-16T04:01:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:50 smithi022 ceph-mon[31360]: pgmap v18096: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:50 smithi022 ceph-mon[40250]: pgmap v18096: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:50 smithi122 ceph-mon[32255]: pgmap v18096: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:51.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:51.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:51.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:01:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:01:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:52 smithi122 ceph-mon[32255]: pgmap v18097: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2540543822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:52 smithi022 ceph-mon[31360]: pgmap v18097: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2540543822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:52 smithi022 ceph-mon[40250]: pgmap v18097: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2540543822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:01:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:01:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:54 smithi022 ceph-mon[31360]: pgmap v18098: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:54 smithi022 ceph-mon[40250]: pgmap v18098: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:54 smithi122 ceph-mon[32255]: pgmap v18098: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:01:56.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:01:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:56 smithi022 ceph-mon[31360]: pgmap v18099: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:56 smithi022 ceph-mon[40250]: pgmap v18099: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:56 smithi122 ceph-mon[32255]: pgmap v18099: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:01:57.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:01:57.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:01:57.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:01:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2582231806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:01:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2582231806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2582231806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:01:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:58 smithi122 ceph-mon[32255]: pgmap v18100: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:58 smithi022 ceph-mon[31360]: pgmap v18100: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:58 smithi022 ceph-mon[40250]: pgmap v18100: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:01:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:01:59 smithi122 ceph-mon[32255]: pgmap v18101: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:00.186 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:01:59 smithi022 ceph-mon[31360]: pgmap v18101: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:00.187 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:01:59 smithi022 ceph-mon[40250]: pgmap v18101: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:00] "GET /metrics HTTP/1.1" 200 202474 "" "Prometheus/2.18.1" 2021-12-16T04:02:02.114 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:02.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:02 smithi122 ceph-mon[32255]: pgmap v18102: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:02.435 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:02:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:02:02.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:02.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:02.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:02.483 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:02 smithi022 ceph-mon[31360]: pgmap v18102: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:02.484 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:02 smithi022 ceph-mon[40250]: pgmap v18102: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:02:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:02:03.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3740374950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3740374950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3740374950' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:04 smithi122 ceph-mon[32255]: pgmap v18103: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:04 smithi022 ceph-mon[31360]: pgmap v18103: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:04 smithi022 ceph-mon[40250]: pgmap v18103: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:06 smithi122 ceph-mon[32255]: pgmap v18104: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:06 smithi022 ceph-mon[31360]: pgmap v18104: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:06 smithi022 ceph-mon[40250]: pgmap v18104: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:07.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:02:07.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:07.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:07.872 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:08 smithi122 ceph-mon[32255]: pgmap v18105: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:02:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:02:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:02:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3305106359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:02:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[31360]: pgmap v18105: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:02:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:02:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:02:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3305106359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:02:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[40250]: pgmap v18105: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:02:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:02:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:02:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3305106359' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:02:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:10] "GET /metrics HTTP/1.1" 200 202476 "" "Prometheus/2.18.1" 2021-12-16T04:02:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:10 smithi122 ceph-mon[32255]: pgmap v18106: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:10 smithi022 ceph-mon[31360]: pgmap v18106: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:10 smithi022 ceph-mon[40250]: pgmap v18106: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:02:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:02:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:12 smithi122 ceph-mon[32255]: pgmap v18107: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:12 smithi022 ceph-mon[31360]: pgmap v18107: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:12 smithi022 ceph-mon[40250]: pgmap v18107: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:12.874 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:13.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:13.245 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:13.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:13.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1650127474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:02:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:02:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1650127474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:13.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1650127474' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:14 smithi122 ceph-mon[32255]: pgmap v18108: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:14 smithi022 ceph-mon[31360]: pgmap v18108: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:14 smithi022 ceph-mon[40250]: pgmap v18108: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:16 smithi122 ceph-mon[32255]: pgmap v18109: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:16 smithi022 ceph-mon[31360]: pgmap v18109: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:16.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:16 smithi022 ceph-mon[40250]: pgmap v18109: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:02:18.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:18 smithi022 ceph-mon[31360]: pgmap v18110: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:18.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:18 smithi022 ceph-mon[40250]: pgmap v18110: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:18.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:18.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:18.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:18 smithi122 ceph-mon[32255]: pgmap v18110: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2812011325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2812011325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2812011325' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:20 smithi122 ceph-mon[32255]: pgmap v18111: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:20] "GET /metrics HTTP/1.1" 200 202477 "" "Prometheus/2.18.1" 2021-12-16T04:02:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:20 smithi022 ceph-mon[31360]: pgmap v18111: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:20 smithi022 ceph-mon[40250]: pgmap v18111: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:22 smithi122 ceph-mon[32255]: pgmap v18112: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:02:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:02:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:22 smithi022 ceph-mon[40250]: pgmap v18112: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:22 smithi022 ceph-mon[31360]: pgmap v18112: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:23.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:02:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:02:23.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:23.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:23.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:23.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:24 smithi122 ceph-mon[32255]: pgmap v18113: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2944922857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:24 smithi022 ceph-mon[31360]: pgmap v18113: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2944922857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:24 smithi022 ceph-mon[40250]: pgmap v18113: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2944922857' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:26 smithi122 ceph-mon[32255]: pgmap v18114: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:26 smithi022 ceph-mon[31360]: pgmap v18114: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:26 smithi022 ceph-mon[40250]: pgmap v18114: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:02:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:28 smithi122 ceph-mon[32255]: pgmap v18115: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:28 smithi022 ceph-mon[31360]: pgmap v18115: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:28 smithi022 ceph-mon[40250]: pgmap v18115: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:28.989 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:29.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:29.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:29.359 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2576865785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2576865785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2576865785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:30] "GET /metrics HTTP/1.1" 200 202477 "" "Prometheus/2.18.1" 2021-12-16T04:02:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:30 smithi122 ceph-mon[32255]: pgmap v18116: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:30 smithi022 ceph-mon[31360]: pgmap v18116: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:30 smithi022 ceph-mon[40250]: pgmap v18116: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:32 smithi122 ceph-mon[32255]: pgmap v18117: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:02:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:02:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:32 smithi022 ceph-mon[31360]: pgmap v18117: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:32 smithi022 ceph-mon[40250]: pgmap v18117: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:33.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:02:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:02:34.361 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:34 smithi122 ceph-mon[32255]: pgmap v18118: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:34.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:34.734 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:34.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:34.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:34 smithi022 ceph-mon[31360]: pgmap v18118: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:34.735 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:34.736 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:34 smithi022 ceph-mon[40250]: pgmap v18118: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:34.736 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:02:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:02:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:02:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:02:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1736934031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1736934031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:02:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:02:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:02:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:02:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1736934031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:36 smithi122 ceph-mon[32255]: pgmap v18119: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:36 smithi022 ceph-mon[31360]: pgmap v18119: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:36 smithi022 ceph-mon[40250]: pgmap v18119: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:02:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:38 smithi122 ceph-mon[32255]: pgmap v18120: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:38 smithi022 ceph-mon[31360]: pgmap v18120: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:38 smithi022 ceph-mon[40250]: pgmap v18120: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:02:39.736 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:40.106 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:40.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:40.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:40 smithi122 ceph-mon[32255]: pgmap v18121: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2779003781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:40] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T04:02:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:40 smithi022 ceph-mon[31360]: pgmap v18121: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2779003781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:40 smithi022 ceph-mon[40250]: pgmap v18121: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2779003781' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:42 smithi122 ceph-mon[32255]: pgmap v18122: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:02:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:02:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:42 smithi022 ceph-mon[31360]: pgmap v18122: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:42 smithi022 ceph-mon[40250]: pgmap v18122: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:43.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:02:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:02:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:44 smithi122 ceph-mon[32255]: pgmap v18123: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:44 smithi022 ceph-mon[31360]: pgmap v18123: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:44 smithi022 ceph-mon[40250]: pgmap v18123: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:45.108 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:45.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:45.477 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:45.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:46 smithi122 ceph-mon[32255]: pgmap v18124: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1167118809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:46 smithi022 ceph-mon[31360]: pgmap v18124: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1167118809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:46 smithi022 ceph-mon[40250]: pgmap v18124: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1167118809' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:02:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:48 smithi122 ceph-mon[32255]: pgmap v18125: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:48 smithi022 ceph-mon[31360]: pgmap v18125: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:48 smithi022 ceph-mon[40250]: pgmap v18125: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:50 smithi122 ceph-mon[32255]: pgmap v18126: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:50] "GET /metrics HTTP/1.1" 200 202489 "" "Prometheus/2.18.1" 2021-12-16T04:02:50.479 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:50 smithi022 ceph-mon[40250]: pgmap v18126: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:50 smithi022 ceph-mon[31360]: pgmap v18126: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:50.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/834749554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/834749554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/834749554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:52 smithi122 ceph-mon[32255]: pgmap v18127: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:02:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:02:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:52 smithi022 ceph-mon[31360]: pgmap v18127: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:52 smithi022 ceph-mon[40250]: pgmap v18127: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:53.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:02:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:02:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:54 smithi122 ceph-mon[32255]: pgmap v18128: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:54 smithi022 ceph-mon[31360]: pgmap v18128: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:54 smithi022 ceph-mon[40250]: pgmap v18128: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:02:55.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:02:56.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:02:56.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:02:56.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:02:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:56 smithi022 ceph-mon[31360]: pgmap v18129: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2722476452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:56 smithi022 ceph-mon[40250]: pgmap v18129: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2722476452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:56 smithi122 ceph-mon[32255]: pgmap v18129: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:02:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2722476452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:02:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:02:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:02:58 smithi122 ceph-mon[32255]: pgmap v18130: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:02:58 smithi022 ceph-mon[31360]: pgmap v18130: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:02:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:02:58 smithi022 ceph-mon[40250]: pgmap v18130: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:00] "GET /metrics HTTP/1.1" 200 202489 "" "Prometheus/2.18.1" 2021-12-16T04:03:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:00 smithi022 ceph-mon[31360]: pgmap v18131: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:00 smithi022 ceph-mon[40250]: pgmap v18131: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:00 smithi122 ceph-mon[32255]: pgmap v18131: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:01.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:01.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:01.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:01.592 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:03:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:03:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:02 smithi122 ceph-mon[32255]: pgmap v18132: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1746183181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:02 smithi022 ceph-mon[31360]: pgmap v18132: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1746183181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:02 smithi022 ceph-mon[40250]: pgmap v18132: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1746183181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:03.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:03:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:03:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:04 smithi022 ceph-mon[31360]: pgmap v18133: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:04 smithi022 ceph-mon[40250]: pgmap v18133: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:04 smithi122 ceph-mon[32255]: pgmap v18133: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:06.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:06 smithi022 ceph-mon[31360]: pgmap v18134: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:06 smithi022 ceph-mon[40250]: pgmap v18134: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:06 smithi122 ceph-mon[32255]: pgmap v18134: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:06.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:06.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:06.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1397207014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:03:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1397207014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1397207014' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:08 smithi122 ceph-mon[32255]: pgmap v18135: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:03:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:03:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:08 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:03:08.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:03:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[31360]: pgmap v18135: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:03:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:03:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:03:08.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:03:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[40250]: pgmap v18135: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:03:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:03:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:03:08.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:03:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:10] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T04:03:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:10 smithi022 ceph-mon[31360]: pgmap v18136: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:10 smithi022 ceph-mon[40250]: pgmap v18136: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:10 smithi122 ceph-mon[32255]: pgmap v18136: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:11.965 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:12.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:12.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:12.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:12 smithi122 ceph-mon[32255]: pgmap v18137: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3536746236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:03:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:03:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:12 smithi022 ceph-mon[31360]: pgmap v18137: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3536746236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:12 smithi022 ceph-mon[40250]: pgmap v18137: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3536746236' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:13.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:03:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:03:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:14 smithi022 ceph-mon[31360]: pgmap v18138: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:14 smithi022 ceph-mon[40250]: pgmap v18138: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:14 smithi122 ceph-mon[32255]: pgmap v18138: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:16 smithi022 ceph-mon[31360]: pgmap v18139: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:16 smithi022 ceph-mon[40250]: pgmap v18139: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:16 smithi122 ceph-mon[32255]: pgmap v18139: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:17.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:17.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:17.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:17.707 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:03:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:18 smithi022 ceph-mon[31360]: pgmap v18140: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3987336051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:18 smithi022 ceph-mon[40250]: pgmap v18140: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3987336051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:18 smithi122 ceph-mon[32255]: pgmap v18140: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3987336051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:20] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:03:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:20 smithi022 ceph-mon[31360]: pgmap v18141: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:20 smithi022 ceph-mon[40250]: pgmap v18141: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:20 smithi122 ceph-mon[32255]: pgmap v18141: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:22 smithi122 ceph-mon[32255]: pgmap v18142: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:03:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:03:22.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:22 smithi022 ceph-mon[31360]: pgmap v18142: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:22.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:22 smithi022 ceph-mon[40250]: pgmap v18142: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:23.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:23.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:23.074 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:23.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:03:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:03:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1764544772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1764544772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1764544772' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:24 smithi022 ceph-mon[31360]: pgmap v18143: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:24 smithi022 ceph-mon[40250]: pgmap v18143: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:24 smithi122 ceph-mon[32255]: pgmap v18143: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:26 smithi022 ceph-mon[31360]: pgmap v18144: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:26 smithi022 ceph-mon[40250]: pgmap v18144: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:26 smithi122 ceph-mon[32255]: pgmap v18144: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:03:28.076 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:28.447 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:28.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:28.448 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:28 smithi022 ceph-mon[31360]: pgmap v18145: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4200635472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:28 smithi022 ceph-mon[40250]: pgmap v18145: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:28.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4200635472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:28 smithi122 ceph-mon[32255]: pgmap v18145: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4200635472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:30] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:03:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:30 smithi022 ceph-mon[31360]: pgmap v18146: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:30 smithi022 ceph-mon[40250]: pgmap v18146: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:30 smithi122 ceph-mon[32255]: pgmap v18146: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:32 smithi122 ceph-mon[32255]: pgmap v18147: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:03:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:03:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:32 smithi022 ceph-mon[31360]: pgmap v18147: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:32 smithi022 ceph-mon[40250]: pgmap v18147: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:33.449 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:03:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:03:33.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:33.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:33.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[31360]: pgmap v18148: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1666336167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:03:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:03:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:03:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:03:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[40250]: pgmap v18148: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1666336167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:03:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:03:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:03:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:03:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:34 smithi122 ceph-mon[32255]: pgmap v18148: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1666336167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:03:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:03:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:03:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:03:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:36 smithi022 ceph-mon[31360]: pgmap v18149: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:36 smithi022 ceph-mon[40250]: pgmap v18149: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:36 smithi122 ceph-mon[32255]: pgmap v18149: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:03:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:38 smithi022 ceph-mon[31360]: pgmap v18150: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:38 smithi022 ceph-mon[40250]: pgmap v18150: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:38 smithi122 ceph-mon[32255]: pgmap v18150: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:39.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:39.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:39.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2792241051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2792241051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2792241051' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:40] "GET /metrics HTTP/1.1" 200 202481 "" "Prometheus/2.18.1" 2021-12-16T04:03:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:40 smithi022 ceph-mon[31360]: pgmap v18151: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:40 smithi022 ceph-mon[40250]: pgmap v18151: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:40 smithi122 ceph-mon[32255]: pgmap v18151: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:42 smithi122 ceph-mon[32255]: pgmap v18152: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:03:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:03:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:42 smithi022 ceph-mon[31360]: pgmap v18152: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:42 smithi022 ceph-mon[40250]: pgmap v18152: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:03:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:03:44.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:44.570 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:44.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:44.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:44 smithi022 ceph-mon[31360]: pgmap v18153: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:44 smithi022 ceph-mon[40250]: pgmap v18153: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:44 smithi122 ceph-mon[32255]: pgmap v18153: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3155319131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:45.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3155319131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3155319131' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:46 smithi022 ceph-mon[31360]: pgmap v18154: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:46 smithi022 ceph-mon[40250]: pgmap v18154: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:46 smithi122 ceph-mon[32255]: pgmap v18154: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:03:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:48 smithi022 ceph-mon[31360]: pgmap v18155: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:48 smithi022 ceph-mon[40250]: pgmap v18155: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:48 smithi122 ceph-mon[32255]: pgmap v18155: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:49.572 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:49.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:49.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:49.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:50] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T04:03:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:50 smithi022 ceph-mon[31360]: pgmap v18156: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:50.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/133384939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:50 smithi022 ceph-mon[40250]: pgmap v18156: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/133384939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:50 smithi122 ceph-mon[32255]: pgmap v18156: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/133384939' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:52 smithi122 ceph-mon[32255]: pgmap v18157: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:03:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:03:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:52 smithi022 ceph-mon[31360]: pgmap v18157: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:52 smithi022 ceph-mon[40250]: pgmap v18157: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:53.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:03:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:03:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:54 smithi022 ceph-mon[31360]: pgmap v18158: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:54 smithi022 ceph-mon[40250]: pgmap v18158: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:54 smithi122 ceph-mon[32255]: pgmap v18158: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:03:54.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:03:55.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:03:55.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:03:55.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:03:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3629930397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3629930397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3629930397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:03:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:56 smithi022 ceph-mon[31360]: pgmap v18159: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:56 smithi022 ceph-mon[40250]: pgmap v18159: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:56 smithi122 ceph-mon[32255]: pgmap v18159: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:03:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:03:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:03:58 smithi122 ceph-mon[32255]: pgmap v18160: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:03:58 smithi022 ceph-mon[31360]: pgmap v18160: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:03:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:03:58 smithi022 ceph-mon[40250]: pgmap v18160: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:00.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:00] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T04:04:00.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:00.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:00.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:00.690 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:00 smithi022 ceph-mon[31360]: pgmap v18161: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:00.690 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:00 smithi022 ceph-mon[40250]: pgmap v18161: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:00 smithi122 ceph-mon[32255]: pgmap v18161: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:01.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2821793686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:02.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2821793686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:02.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2821793686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:02 smithi122 ceph-mon[32255]: pgmap v18162: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:04:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:04:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:02 smithi022 ceph-mon[31360]: pgmap v18162: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:03.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:02 smithi022 ceph-mon[40250]: pgmap v18162: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:04:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:04:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:04 smithi122 ceph-mon[32255]: pgmap v18163: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:04 smithi022 ceph-mon[31360]: pgmap v18163: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:04 smithi022 ceph-mon[40250]: pgmap v18163: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:05.690 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:05 smithi122 ceph-mon[32255]: pgmap v18164: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:05 smithi022 ceph-mon[31360]: pgmap v18164: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:05 smithi022 ceph-mon[40250]: pgmap v18164: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:06.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:06.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:06.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/665941837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/665941837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:07.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/665941837' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:07 smithi022 ceph-mon[31360]: pgmap v18165: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:04:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:04:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:07 smithi022 ceph-mon[40250]: pgmap v18165: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:07 smithi122 ceph-mon[32255]: pgmap v18165: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:04:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:04:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:04:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:04:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:04:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:04:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:04:09.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:04:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:04:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:04:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:04:09.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:04:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:10 smithi122 ceph-mon[32255]: pgmap v18166: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:04:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:10] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T04:04:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:10 smithi022 ceph-mon[31360]: pgmap v18166: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:10 smithi022 ceph-mon[40250]: pgmap v18166: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:11.062 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:11.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:11.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:11.434 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:12 smithi122 ceph-mon[32255]: pgmap v18167: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:12.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/341939384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:04:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:04:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:04:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:12 smithi022 ceph-mon[31360]: pgmap v18167: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/341939384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:12 smithi022 ceph-mon[40250]: pgmap v18167: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/341939384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:04:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:04:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:14 smithi122 ceph-mon[32255]: pgmap v18168: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:14 smithi022 ceph-mon[31360]: pgmap v18168: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:14 smithi022 ceph-mon[40250]: pgmap v18168: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:16.435 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:16 smithi122 ceph-mon[32255]: pgmap v18169: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:16 smithi022 ceph-mon[31360]: pgmap v18169: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:16 smithi022 ceph-mon[40250]: pgmap v18169: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:16.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:16.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:16.807 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2798016862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2798016862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:04:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:04:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2798016862' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:18 smithi122 ceph-mon[32255]: pgmap v18170: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:18 smithi022 ceph-mon[31360]: pgmap v18170: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:18 smithi022 ceph-mon[40250]: pgmap v18170: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:04:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:20] "GET /metrics HTTP/1.1" 200 202485 "" "Prometheus/2.18.1" 2021-12-16T04:04:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:20 smithi122 ceph-mon[32255]: pgmap v18171: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:20 smithi022 ceph-mon[31360]: pgmap v18171: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:20 smithi022 ceph-mon[40250]: pgmap v18171: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:21.808 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:22.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:22.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:22.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:22 smithi022 ceph-mon[31360]: pgmap v18172: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:22.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3928791181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:22 smithi022 ceph-mon[40250]: pgmap v18172: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3928791181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:22 smithi122 ceph-mon[32255]: pgmap v18172: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3928791181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:04:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:04:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:04:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:04:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:04:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:24 smithi122 ceph-mon[32255]: pgmap v18173: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:24 smithi022 ceph-mon[31360]: pgmap v18173: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:24 smithi022 ceph-mon[40250]: pgmap v18173: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:26 smithi122 ceph-mon[32255]: pgmap v18174: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:26 smithi022 ceph-mon[31360]: pgmap v18174: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:26 smithi022 ceph-mon[40250]: pgmap v18174: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:27.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:04:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:28 smithi122 ceph-mon[32255]: pgmap v18175: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3407055161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:28 smithi022 ceph-mon[31360]: pgmap v18175: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3407055161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:28 smithi022 ceph-mon[40250]: pgmap v18175: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3407055161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:30.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:30 smithi122 ceph-mon[32255]: pgmap v18176: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:04:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:30] "GET /metrics HTTP/1.1" 200 202485 "" "Prometheus/2.18.1" 2021-12-16T04:04:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:30 smithi022 ceph-mon[31360]: pgmap v18176: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:30 smithi022 ceph-mon[40250]: pgmap v18176: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:32.552 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:32 smithi122 ceph-mon[32255]: pgmap v18177: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:04:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:04:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:04:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:32 smithi022 ceph-mon[31360]: pgmap v18177: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:32 smithi022 ceph-mon[40250]: pgmap v18177: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:32.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:32.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:32.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1709768427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:33.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:04:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:04:33.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1709768427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1709768427' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:34 smithi122 ceph-mon[32255]: pgmap v18178: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:34 smithi022 ceph-mon[31360]: pgmap v18178: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:34 smithi022 ceph-mon[40250]: pgmap v18178: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:04:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:04:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:04:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:04:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:04:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:04:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:04:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:04:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:04:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:04:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:04:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:04:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:36 smithi122 ceph-mon[32255]: pgmap v18179: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:36 smithi022 ceph-mon[31360]: pgmap v18179: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:36 smithi022 ceph-mon[40250]: pgmap v18179: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:04:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:04:37.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:38.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:38.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:38.293 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:38 smithi122 ceph-mon[32255]: pgmap v18180: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3779529254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:38 smithi022 ceph-mon[40250]: pgmap v18180: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:38.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3779529254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:38.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:38 smithi022 ceph-mon[31360]: pgmap v18180: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:38.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3779529254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:40.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:40 smithi122 ceph-mon[32255]: pgmap v18181: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:04:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:40] "GET /metrics HTTP/1.1" 200 202482 "" "Prometheus/2.18.1" 2021-12-16T04:04:40.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:40 smithi022 ceph-mon[40250]: pgmap v18181: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:40.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:40 smithi022 ceph-mon[31360]: pgmap v18181: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:42 smithi122 ceph-mon[32255]: pgmap v18182: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:04:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:04:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:04:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:42 smithi022 ceph-mon[31360]: pgmap v18182: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:42 smithi022 ceph-mon[40250]: pgmap v18182: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:43.295 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:04:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:04:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:44 smithi122 ceph-mon[32255]: pgmap v18183: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:44.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1900559356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:44 smithi022 ceph-mon[31360]: pgmap v18183: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1900559356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:44 smithi022 ceph-mon[40250]: pgmap v18183: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1900559356' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:46 smithi122 ceph-mon[32255]: pgmap v18184: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:46 smithi022 ceph-mon[31360]: pgmap v18184: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:46 smithi022 ceph-mon[40250]: pgmap v18184: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:04:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:04:48.668 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:48 smithi122 ceph-mon[32255]: pgmap v18185: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:48 smithi022 ceph-mon[31360]: pgmap v18185: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:48 smithi022 ceph-mon[40250]: pgmap v18185: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:49.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:49.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:49.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:49.432 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/374204995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/374204995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:49.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/374204995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:50.433 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:04:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:50] "GET /metrics HTTP/1.1" 200 202485 "" "Prometheus/2.18.1" 2021-12-16T04:04:50.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:50 smithi122 ceph-mon[32255]: pgmap v18186: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:50 smithi022 ceph-mon[31360]: pgmap v18186: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:50 smithi022 ceph-mon[40250]: pgmap v18186: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:52 smithi122 ceph-mon[32255]: pgmap v18187: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:04:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:04:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:04:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:52 smithi022 ceph-mon[31360]: pgmap v18187: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:52 smithi022 ceph-mon[40250]: pgmap v18187: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:53.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:04:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:04:54.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:04:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:54 smithi022 ceph-mon[31360]: pgmap v18188: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/513482165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:54 smithi022 ceph-mon[40250]: pgmap v18188: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/513482165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:54 smithi122 ceph-mon[32255]: pgmap v18188: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:04:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/513482165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:04:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:56 smithi022 ceph-mon[31360]: pgmap v18189: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:56 smithi022 ceph-mon[40250]: pgmap v18189: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:56 smithi122 ceph-mon[32255]: pgmap v18189: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:04:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:04:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:04:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:04:58 smithi022 ceph-mon[31360]: pgmap v18190: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:04:58 smithi022 ceph-mon[40250]: pgmap v18190: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:04:58 smithi122 ceph-mon[32255]: pgmap v18190: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:04:59.410 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:04:59.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:04:59.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:04:59.783 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:05:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:00] "GET /metrics HTTP/1.1" 200 202485 "" "Prometheus/2.18.1" 2021-12-16T04:05:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:00 smithi022 ceph-mon[31360]: pgmap v18191: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2093876859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:00 smithi022 ceph-mon[40250]: pgmap v18191: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2093876859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:00 smithi122 ceph-mon[32255]: pgmap v18191: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2093876859' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:02.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:02 smithi122 ceph-mon[32255]: pgmap v18192: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:05:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:05:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:05:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:02 smithi022 ceph-mon[31360]: pgmap v18192: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:02 smithi022 ceph-mon[40250]: pgmap v18192: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:05:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:05:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:04 smithi022 ceph-mon[31360]: pgmap v18193: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:04 smithi022 ceph-mon[40250]: pgmap v18193: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:04.784 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:04 smithi122 ceph-mon[32255]: pgmap v18193: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:05.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:05.157 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:05.158 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:05.455 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1136415365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:05.456 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1136415365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1136415365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:06 smithi022 ceph-mon[31360]: pgmap v18194: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:06 smithi022 ceph-mon[40250]: pgmap v18194: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:06 smithi122 ceph-mon[32255]: pgmap v18194: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:05:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:05:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:08 smithi022 ceph-mon[31360]: pgmap v18195: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:08 smithi022 ceph-mon[40250]: pgmap v18195: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:08.885 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:08 smithi122 ceph-mon[32255]: pgmap v18195: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:05:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:05:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:05:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:05:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:05:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:05:09.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:09.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:09.669 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:05:09.669 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:05:09.669 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:05:09.669 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:09.670 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:05:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:10] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T04:05:10.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:10.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:10.527 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:10 smithi022 ceph-mon[31360]: pgmap v18196: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:10 smithi022 ceph-mon[40250]: pgmap v18196: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:10 smithi122 ceph-mon[32255]: pgmap v18196: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:05:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2477660974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2477660974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2477660974' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:12 smithi122 ceph-mon[32255]: pgmap v18197: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:05:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:05:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:05:12.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:12 smithi022 ceph-mon[31360]: pgmap v18197: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:12 smithi022 ceph-mon[40250]: pgmap v18197: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:05:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:05:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:14 smithi022 ceph-mon[31360]: pgmap v18198: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:14 smithi022 ceph-mon[40250]: pgmap v18198: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:14 smithi122 ceph-mon[32255]: pgmap v18198: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:15.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:15.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:16 smithi022 ceph-mon[31360]: pgmap v18199: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2982128075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:16 smithi022 ceph-mon[40250]: pgmap v18199: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2982128075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:16 smithi122 ceph-mon[32255]: pgmap v18199: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2982128075' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:05:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:05:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:18 smithi022 ceph-mon[31360]: pgmap v18200: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:18 smithi022 ceph-mon[40250]: pgmap v18200: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:18 smithi122 ceph-mon[32255]: pgmap v18200: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:05:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:20] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:05:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:20 smithi022 ceph-mon[31360]: pgmap v18201: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:20 smithi022 ceph-mon[40250]: pgmap v18201: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:20.901 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:20 smithi122 ceph-mon[32255]: pgmap v18201: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:21.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:21.274 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:21.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1205874473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:21.647 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1205874473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1205874473' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:22 smithi122 ceph-mon[32255]: pgmap v18202: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:05:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:05:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:05:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:22 smithi022 ceph-mon[31360]: pgmap v18202: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:22 smithi022 ceph-mon[40250]: pgmap v18202: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:05:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:05:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:24 smithi022 ceph-mon[31360]: pgmap v18203: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:24 smithi022 ceph-mon[40250]: pgmap v18203: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:24 smithi122 ceph-mon[32255]: pgmap v18203: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:26.276 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:26 smithi022 ceph-mon[31360]: pgmap v18204: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:26 smithi022 ceph-mon[40250]: pgmap v18204: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:26.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:26 smithi122 ceph-mon[32255]: pgmap v18204: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/609139822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:05:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:05:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/609139822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/609139822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:28 smithi022 ceph-mon[31360]: pgmap v18205: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:28 smithi022 ceph-mon[40250]: pgmap v18205: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:28 smithi122 ceph-mon[32255]: pgmap v18205: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:05:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:30] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:05:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:30 smithi022 ceph-mon[31360]: pgmap v18206: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:30 smithi022 ceph-mon[40250]: pgmap v18206: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:30 smithi122 ceph-mon[32255]: pgmap v18206: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:31.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:32.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:32.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:32.016 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:32 smithi122 ceph-mon[32255]: pgmap v18207: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2186117709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:05:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:05:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:05:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:32 smithi022 ceph-mon[31360]: pgmap v18207: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2186117709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:32 smithi022 ceph-mon[40250]: pgmap v18207: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2186117709' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:05:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:05:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[31360]: pgmap v18208: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:05:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:05:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:05:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:05:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[40250]: pgmap v18208: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:34.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:05:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:05:34.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:05:34.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:05:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:34 smithi122 ceph-mon[32255]: pgmap v18208: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:05:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:05:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:05:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:05:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:36 smithi022 ceph-mon[31360]: pgmap v18209: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:36 smithi022 ceph-mon[40250]: pgmap v18209: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:36 smithi122 ceph-mon[32255]: pgmap v18209: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:37.018 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:37.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:37.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:37.387 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2834384778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:05:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:05:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2834384778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2834384778' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:38 smithi022 ceph-mon[31360]: pgmap v18210: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:38 smithi022 ceph-mon[40250]: pgmap v18210: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:38 smithi122 ceph-mon[32255]: pgmap v18210: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:05:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:40] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:05:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:40 smithi022 ceph-mon[31360]: pgmap v18211: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:40 smithi022 ceph-mon[40250]: pgmap v18211: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:40 smithi122 ceph-mon[32255]: pgmap v18211: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:42.389 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:42 smithi122 ceph-mon[32255]: pgmap v18212: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:05:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:05:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:05:42.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:42.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:42.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:42.761 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:42 smithi022 ceph-mon[40250]: pgmap v18212: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:42.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:42 smithi022 ceph-mon[31360]: pgmap v18212: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:05:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:05:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4214145633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4214145633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4214145633' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:44 smithi122 ceph-mon[32255]: pgmap v18213: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:44 smithi022 ceph-mon[31360]: pgmap v18213: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:44 smithi022 ceph-mon[40250]: pgmap v18213: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:46 smithi122 ceph-mon[32255]: pgmap v18214: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:46 smithi022 ceph-mon[31360]: pgmap v18214: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:46 smithi022 ceph-mon[40250]: pgmap v18214: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:47.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:47.762 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:05:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:05:48.129 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:48.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:48.130 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:48 smithi122 ceph-mon[32255]: pgmap v18215: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2201822720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:48 smithi022 ceph-mon[31360]: pgmap v18215: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2201822720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:48 smithi022 ceph-mon[40250]: pgmap v18215: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2201822720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:05:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:50] "GET /metrics HTTP/1.1" 200 202483 "" "Prometheus/2.18.1" 2021-12-16T04:05:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:50 smithi122 ceph-mon[32255]: pgmap v18216: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:50 smithi022 ceph-mon[31360]: pgmap v18216: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:51.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:50 smithi022 ceph-mon[40250]: pgmap v18216: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:52 smithi122 ceph-mon[32255]: pgmap v18217: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:05:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:05:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:05:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:52 smithi022 ceph-mon[31360]: pgmap v18217: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:52 smithi022 ceph-mon[40250]: pgmap v18217: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:53.131 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:53.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:53.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:53.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:53.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:05:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:05:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:54 smithi122 ceph-mon[32255]: pgmap v18218: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:54.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2885454898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:54 smithi022 ceph-mon[31360]: pgmap v18218: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2885454898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:54 smithi022 ceph-mon[40250]: pgmap v18218: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:05:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2885454898' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:56 smithi122 ceph-mon[32255]: pgmap v18219: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:56 smithi022 ceph-mon[31360]: pgmap v18219: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:56 smithi022 ceph-mon[40250]: pgmap v18219: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:05:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:05:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:05:58.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:05:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:58 smithi122 ceph-mon[32255]: pgmap v18220: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:58.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:58 smithi022 ceph-mon[31360]: pgmap v18220: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:58 smithi022 ceph-mon[40250]: pgmap v18220: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:05:58.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:05:58.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:05:58.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:05:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:05:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/312705144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:05:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/312705144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:05:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:05:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/312705144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:06:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:00] "GET /metrics HTTP/1.1" 200 202483 "" "Prometheus/2.18.1" 2021-12-16T04:06:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:00 smithi122 ceph-mon[32255]: pgmap v18221: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:00 smithi022 ceph-mon[31360]: pgmap v18221: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:00 smithi022 ceph-mon[40250]: pgmap v18221: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:02 smithi122 ceph-mon[32255]: pgmap v18222: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:06:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:06:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:06:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:02 smithi022 ceph-mon[31360]: pgmap v18222: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:02 smithi022 ceph-mon[40250]: pgmap v18222: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:06:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:06:03.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:04.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:04.246 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:04.247 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:04.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:04 smithi022 ceph-mon[31360]: pgmap v18223: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2613944005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:04 smithi022 ceph-mon[40250]: pgmap v18223: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2613944005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:04 smithi122 ceph-mon[32255]: pgmap v18223: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2613944005' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:06 smithi122 ceph-mon[32255]: pgmap v18224: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:06 smithi022 ceph-mon[31360]: pgmap v18224: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:06 smithi022 ceph-mon[40250]: pgmap v18224: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:06:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:06:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:08 smithi122 ceph-mon[32255]: pgmap v18225: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:08 smithi022 ceph-mon[31360]: pgmap v18225: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:08 smithi022 ceph-mon[40250]: pgmap v18225: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:09.248 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:09.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:09.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:09.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:10.415 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:06:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:10] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:06:10.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:10 smithi122 ceph-mon[32255]: pgmap v18226: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:10.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3581601540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:06:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:06:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:06:10.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:06:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[31360]: pgmap v18226: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3581601540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:06:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:06:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:06:10.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:06:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[40250]: pgmap v18226: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3581601540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:06:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:06:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:06:10.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:06:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:12 smithi122 ceph-mon[32255]: pgmap v18227: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:06:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:06:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:06:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:12 smithi022 ceph-mon[31360]: pgmap v18227: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:12 smithi022 ceph-mon[40250]: pgmap v18227: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:06:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:06:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:14 smithi122 ceph-mon[32255]: pgmap v18228: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:14 smithi022 ceph-mon[31360]: pgmap v18228: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:14 smithi022 ceph-mon[40250]: pgmap v18228: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:14.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:14.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:14.987 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1812234450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1812234450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1812234450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:16 smithi122 ceph-mon[32255]: pgmap v18229: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:16 smithi022 ceph-mon[31360]: pgmap v18229: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:16 smithi022 ceph-mon[40250]: pgmap v18229: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:06:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:06:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:18 smithi122 ceph-mon[32255]: pgmap v18230: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:18 smithi022 ceph-mon[31360]: pgmap v18230: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:18 smithi022 ceph-mon[40250]: pgmap v18230: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:19.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:20.431 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:06:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:20] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T04:06:20.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:20 smithi122 ceph-mon[32255]: pgmap v18231: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:20.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/401046681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:20 smithi022 ceph-mon[31360]: pgmap v18231: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/401046681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:20 smithi022 ceph-mon[40250]: pgmap v18231: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/401046681' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:22 smithi122 ceph-mon[32255]: pgmap v18232: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:06:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:06:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:06:22.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:22 smithi022 ceph-mon[31360]: pgmap v18232: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:22.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:22 smithi022 ceph-mon[40250]: pgmap v18232: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:06:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:06:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:24 smithi022 ceph-mon[31360]: pgmap v18233: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:24 smithi022 ceph-mon[40250]: pgmap v18233: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:24 smithi122 ceph-mon[32255]: pgmap v18233: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:25.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:25.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:25.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:25.730 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:26 smithi022 ceph-mon[31360]: pgmap v18234: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2074396104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:26 smithi022 ceph-mon[40250]: pgmap v18234: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2074396104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:26 smithi122 ceph-mon[32255]: pgmap v18234: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2074396104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:06:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:06:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:28 smithi022 ceph-mon[31360]: pgmap v18235: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:28 smithi022 ceph-mon[40250]: pgmap v18235: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:28 smithi122 ceph-mon[32255]: pgmap v18235: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:06:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:30] "GET /metrics HTTP/1.1" 200 202487 "" "Prometheus/2.18.1" 2021-12-16T04:06:30.732 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:30.733 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:30 smithi022 ceph-mon[31360]: pgmap v18236: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:30.734 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:30 smithi022 ceph-mon[40250]: pgmap v18236: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:30 smithi122 ceph-mon[32255]: pgmap v18236: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:31.098 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:31.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:31.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:31.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2761227186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2761227186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2761227186' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:32.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:32 smithi122 ceph-mon[32255]: pgmap v18237: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:06:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:06:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:06:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:32 smithi022 ceph-mon[31360]: pgmap v18237: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:32 smithi022 ceph-mon[40250]: pgmap v18237: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:06:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:06:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:34 smithi022 ceph-mon[31360]: pgmap v18238: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:34 smithi022 ceph-mon[40250]: pgmap v18238: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:34 smithi122 ceph-mon[32255]: pgmap v18238: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:06:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:06:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:06:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:06:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:06:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:06:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:06:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:06:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:06:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:06:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:06:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:06:36.100 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:36.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:36.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:36 smithi022 ceph-mon[31360]: pgmap v18239: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:36 smithi022 ceph-mon[40250]: pgmap v18239: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:36 smithi122 ceph-mon[32255]: pgmap v18239: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:06:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:06:37.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/852600400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/852600400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/852600400' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:38 smithi022 ceph-mon[31360]: pgmap v18240: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:38 smithi022 ceph-mon[40250]: pgmap v18240: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:38 smithi122 ceph-mon[32255]: pgmap v18240: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:06:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:40] "GET /metrics HTTP/1.1" 200 202479 "" "Prometheus/2.18.1" 2021-12-16T04:06:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:40 smithi022 ceph-mon[31360]: pgmap v18241: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:40 smithi022 ceph-mon[40250]: pgmap v18241: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:40 smithi122 ceph-mon[32255]: pgmap v18241: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:41.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:06:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:06:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:06:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:42 smithi122 ceph-mon[32255]: pgmap v18242: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1337311684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:42.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:42 smithi022 ceph-mon[40250]: pgmap v18242: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1337311684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:42 smithi022 ceph-mon[31360]: pgmap v18242: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:42.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1337311684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:06:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:06:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:44 smithi022 ceph-mon[31360]: pgmap v18243: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:44.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:44 smithi022 ceph-mon[40250]: pgmap v18243: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:44.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:44 smithi122 ceph-mon[32255]: pgmap v18243: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:46 smithi022 ceph-mon[31360]: pgmap v18244: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:46 smithi022 ceph-mon[40250]: pgmap v18244: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:46.844 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:46 smithi122 ceph-mon[32255]: pgmap v18244: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:47.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:47.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:47.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:47.482 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:06:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:06:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2623579161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2623579161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2623579161' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:48 smithi022 ceph-mon[31360]: pgmap v18245: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:48.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:48 smithi022 ceph-mon[40250]: pgmap v18245: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:48 smithi122 ceph-mon[32255]: pgmap v18245: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:06:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:50] "GET /metrics HTTP/1.1" 200 202481 "" "Prometheus/2.18.1" 2021-12-16T04:06:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:50 smithi022 ceph-mon[31360]: pgmap v18246: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:50 smithi022 ceph-mon[40250]: pgmap v18246: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:50 smithi122 ceph-mon[32255]: pgmap v18246: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:52.216 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:52.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:52.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:52.585 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:06:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:06:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:06:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:52 smithi122 ceph-mon[32255]: pgmap v18247: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:52 smithi022 ceph-mon[31360]: pgmap v18247: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:52 smithi022 ceph-mon[40250]: pgmap v18247: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4202063848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:53.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:06:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:06:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4202063848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4202063848' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:54 smithi022 ceph-mon[31360]: pgmap v18248: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:54.801 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:54.801 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:54 smithi022 ceph-mon[40250]: pgmap v18248: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:54.802 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:54 smithi122 ceph-mon[32255]: pgmap v18248: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:06:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:56 smithi022 ceph-mon[31360]: pgmap v18249: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:56 smithi022 ceph-mon[40250]: pgmap v18249: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:56 smithi122 ceph-mon[32255]: pgmap v18249: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:06:57.586 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:06:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:06:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:06:57.954 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:06:57.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:06:57.955 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:06:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:58 smithi022 ceph-mon[31360]: pgmap v18250: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:06:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2437463158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:58 smithi022 ceph-mon[40250]: pgmap v18250: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:06:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2437463158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:06:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:58 smithi122 ceph-mon[32255]: pgmap v18250: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:06:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:06:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2437463158' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:07:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:00] "GET /metrics HTTP/1.1" 200 202481 "" "Prometheus/2.18.1" 2021-12-16T04:07:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:00 smithi022 ceph-mon[31360]: pgmap v18251: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:00 smithi022 ceph-mon[40250]: pgmap v18251: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:00 smithi122 ceph-mon[32255]: pgmap v18251: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:02 smithi122 ceph-mon[32255]: pgmap v18252: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:07:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:07:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:07:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:02 smithi022 ceph-mon[31360]: pgmap v18252: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:02 smithi022 ceph-mon[40250]: pgmap v18252: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:02.956 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:03.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:03.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:03.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1503294278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:07:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:07:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1503294278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1503294278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:04 smithi022 ceph-mon[31360]: pgmap v18253: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:04 smithi022 ceph-mon[40250]: pgmap v18253: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:04 smithi122 ceph-mon[32255]: pgmap v18253: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:06 smithi022 ceph-mon[31360]: pgmap v18254: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:06 smithi022 ceph-mon[40250]: pgmap v18254: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:06 smithi122 ceph-mon[32255]: pgmap v18254: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:07:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:07:08.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:08.703 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:08.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:08.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:08.705 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:08 smithi022 ceph-mon[31360]: pgmap v18255: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:08.705 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:08 smithi022 ceph-mon[40250]: pgmap v18255: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:08 smithi122 ceph-mon[32255]: pgmap v18255: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1995525261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1995525261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1995525261' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:07:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:10] "GET /metrics HTTP/1.1" 200 202479 "" "Prometheus/2.18.1" 2021-12-16T04:07:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[31360]: pgmap v18256: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:07:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:07:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:07:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[40250]: pgmap v18256: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:07:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:07:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:07:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:10 smithi122 ceph-mon[32255]: pgmap v18256: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:07:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:07:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:07:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:07:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:11 smithi122 ceph-mon[32255]: pgmap v18257: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:07:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:11 smithi022 ceph-mon[31360]: pgmap v18257: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:07:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:11 smithi022 ceph-mon[40250]: pgmap v18257: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:07:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:07:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:07:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:07:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:07:13.706 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:14.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:14.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:14.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:14 smithi122 ceph-mon[32255]: pgmap v18258: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:14.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4202592066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:14 smithi022 ceph-mon[31360]: pgmap v18258: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4202592066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:14 smithi022 ceph-mon[40250]: pgmap v18258: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4202592066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:16 smithi122 ceph-mon[32255]: pgmap v18259: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:16 smithi022 ceph-mon[31360]: pgmap v18259: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:16 smithi022 ceph-mon[40250]: pgmap v18259: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:07:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:07:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:18 smithi122 ceph-mon[32255]: pgmap v18260: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:18 smithi022 ceph-mon[31360]: pgmap v18260: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:18 smithi022 ceph-mon[40250]: pgmap v18260: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:19.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:19.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:19.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:19.460 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:07:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:20] "GET /metrics HTTP/1.1" 200 202481 "" "Prometheus/2.18.1" 2021-12-16T04:07:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:20 smithi122 ceph-mon[32255]: pgmap v18261: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3996304634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:20 smithi022 ceph-mon[31360]: pgmap v18261: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3996304634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:20 smithi022 ceph-mon[40250]: pgmap v18261: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3996304634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:22 smithi122 ceph-mon[32255]: pgmap v18262: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:07:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:07:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:07:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:22 smithi022 ceph-mon[31360]: pgmap v18262: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:22 smithi022 ceph-mon[40250]: pgmap v18262: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:07:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:07:24.457 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:24 smithi122 ceph-mon[32255]: pgmap v18263: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:24 smithi022 ceph-mon[31360]: pgmap v18263: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:24 smithi022 ceph-mon[40250]: pgmap v18263: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1100270796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1100270796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:25.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1100270796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:26 smithi122 ceph-mon[32255]: pgmap v18264: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:26 smithi022 ceph-mon[31360]: pgmap v18264: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:26 smithi022 ceph-mon[40250]: pgmap v18264: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:07:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:07:28.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:28 smithi122 ceph-mon[32255]: pgmap v18265: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:28 smithi022 ceph-mon[31360]: pgmap v18265: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:28 smithi022 ceph-mon[40250]: pgmap v18265: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:29.832 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:30.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:30.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:30.204 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:07:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:30] "GET /metrics HTTP/1.1" 200 202481 "" "Prometheus/2.18.1" 2021-12-16T04:07:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:30 smithi022 ceph-mon[31360]: pgmap v18266: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:30.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/927619364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:30 smithi022 ceph-mon[40250]: pgmap v18266: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:30.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/927619364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:30 smithi122 ceph-mon[32255]: pgmap v18266: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:30.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/927619364' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:32 smithi122 ceph-mon[32255]: pgmap v18267: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:07:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:07:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:07:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:32 smithi022 ceph-mon[31360]: pgmap v18267: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:32 smithi022 ceph-mon[40250]: pgmap v18267: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:07:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:07:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:34 smithi022 ceph-mon[31360]: pgmap v18268: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:34 smithi022 ceph-mon[40250]: pgmap v18268: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:34 smithi122 ceph-mon[32255]: pgmap v18268: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:35.206 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:07:35.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:07:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:07:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:07:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:07:35.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:07:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:07:35.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:07:35.577 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:35.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:35.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:07:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:07:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:07:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:07:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:36 smithi022 ceph-mon[31360]: pgmap v18269: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1586151656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:36 smithi022 ceph-mon[40250]: pgmap v18269: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1586151656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:36 smithi122 ceph-mon[32255]: pgmap v18269: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1586151656' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:07:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:07:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:38 smithi022 ceph-mon[31360]: pgmap v18270: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:38 smithi022 ceph-mon[40250]: pgmap v18270: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:38 smithi122 ceph-mon[32255]: pgmap v18270: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:07:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:40] "GET /metrics HTTP/1.1" 200 202486 "" "Prometheus/2.18.1" 2021-12-16T04:07:40.579 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:40 smithi022 ceph-mon[31360]: pgmap v18271: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:40 smithi022 ceph-mon[40250]: pgmap v18271: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:40 smithi122 ceph-mon[32255]: pgmap v18271: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:40.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:40.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:40.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2083650521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2083650521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2083650521' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:42 smithi122 ceph-mon[32255]: pgmap v18272: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:07:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:07:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:07:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:42 smithi022 ceph-mon[31360]: pgmap v18272: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:42 smithi022 ceph-mon[40250]: pgmap v18272: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:43.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:07:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:07:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:44 smithi022 ceph-mon[31360]: pgmap v18273: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:44 smithi022 ceph-mon[40250]: pgmap v18273: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:44 smithi122 ceph-mon[32255]: pgmap v18273: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:45.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:46.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:46.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:46.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:46 smithi022 ceph-mon[31360]: pgmap v18274: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3497810618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:46 smithi022 ceph-mon[40250]: pgmap v18274: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3497810618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:46 smithi122 ceph-mon[32255]: pgmap v18274: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3497810618' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:07:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:07:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:48 smithi022 ceph-mon[31360]: pgmap v18275: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:48 smithi022 ceph-mon[40250]: pgmap v18275: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:48 smithi122 ceph-mon[32255]: pgmap v18275: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:07:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:50] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T04:07:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:50 smithi022 ceph-mon[31360]: pgmap v18276: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:50.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:50 smithi022 ceph-mon[40250]: pgmap v18276: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:50 smithi122 ceph-mon[32255]: pgmap v18276: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:51.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:52 smithi122 ceph-mon[32255]: pgmap v18277: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1263536578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:07:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:07:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:07:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:52 smithi022 ceph-mon[31360]: pgmap v18277: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:52.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1263536578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:52 smithi022 ceph-mon[40250]: pgmap v18277: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:52.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1263536578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:53.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:07:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:07:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:54 smithi022 ceph-mon[31360]: pgmap v18278: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:54 smithi022 ceph-mon[40250]: pgmap v18278: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:54 smithi122 ceph-mon[32255]: pgmap v18278: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:07:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:07:56.694 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:07:56.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:56 smithi022 ceph-mon[40250]: pgmap v18279: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:56 smithi022 ceph-mon[31360]: pgmap v18279: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:56 smithi122 ceph-mon[32255]: pgmap v18279: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:57.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:07:57.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:07:57.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:07:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3327993136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:07:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:07:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3327993136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:57.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3327993136' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:07:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:07:58 smithi022 ceph-mon[31360]: pgmap v18280: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:07:58 smithi022 ceph-mon[40250]: pgmap v18280: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:07:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:07:58 smithi122 ceph-mon[32255]: pgmap v18280: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:08:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:00] "GET /metrics HTTP/1.1" 200 202484 "" "Prometheus/2.18.1" 2021-12-16T04:08:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:00 smithi022 ceph-mon[31360]: pgmap v18281: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:00 smithi022 ceph-mon[40250]: pgmap v18281: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:00 smithi122 ceph-mon[32255]: pgmap v18281: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:08:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:08:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:08:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:02 smithi122 ceph-mon[32255]: pgmap v18282: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3131308685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:02 smithi022 ceph-mon[31360]: pgmap v18282: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3131308685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:02 smithi022 ceph-mon[40250]: pgmap v18282: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3131308685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:03.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:08:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:08:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:04 smithi022 ceph-mon[31360]: pgmap v18283: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:04 smithi022 ceph-mon[40250]: pgmap v18283: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:04 smithi122 ceph-mon[32255]: pgmap v18283: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:06 smithi022 ceph-mon[31360]: pgmap v18284: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:06 smithi022 ceph-mon[40250]: pgmap v18284: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:06 smithi122 ceph-mon[32255]: pgmap v18284: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:07.435 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:08:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:08:07.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:07.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:07.806 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:08 smithi022 ceph-mon[31360]: pgmap v18285: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1945824238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:08 smithi022 ceph-mon[40250]: pgmap v18285: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1945824238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:08 smithi122 ceph-mon[32255]: pgmap v18285: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1945824238' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:08:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:10] "GET /metrics HTTP/1.1" 200 202483 "" "Prometheus/2.18.1" 2021-12-16T04:08:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:10 smithi022 ceph-mon[31360]: pgmap v18286: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:10 smithi022 ceph-mon[40250]: pgmap v18286: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:10 smithi122 ceph-mon[32255]: pgmap v18286: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:08:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:08:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:08:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:08:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:08:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:08:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:08:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:08:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:08:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:08:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:08:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:08:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:08:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:12 smithi122 ceph-mon[32255]: pgmap v18287: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:08:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:08:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:08:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:12 smithi022 ceph-mon[31360]: pgmap v18287: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:12 smithi022 ceph-mon[40250]: pgmap v18287: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:12.808 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:13.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:08:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:08:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3830909669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3830909669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:13.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3830909669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:14 smithi022 ceph-mon[31360]: pgmap v18288: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2021-12-16T04:08:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:14 smithi022 ceph-mon[40250]: pgmap v18288: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2021-12-16T04:08:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:14 smithi122 ceph-mon[32255]: pgmap v18288: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2021-12-16T04:08:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:16 smithi022 ceph-mon[31360]: pgmap v18289: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-16T04:08:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:16 smithi022 ceph-mon[40250]: pgmap v18289: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-16T04:08:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:16 smithi122 ceph-mon[32255]: pgmap v18289: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-16T04:08:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:08:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:18 smithi022 ceph-mon[31360]: pgmap v18290: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-16T04:08:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:18 smithi022 ceph-mon[40250]: pgmap v18290: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-16T04:08:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:18 smithi122 ceph-mon[32255]: pgmap v18290: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2021-12-16T04:08:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/452808726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/452808726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/452808726' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:08:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:20] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:08:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:20 smithi022 ceph-mon[31360]: pgmap v18291: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:08:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:20 smithi022 ceph-mon[40250]: pgmap v18291: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:08:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:20 smithi122 ceph-mon[32255]: pgmap v18291: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:08:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:22 smithi122 ceph-mon[32255]: pgmap v18292: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:08:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:08:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:08:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:08:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:22 smithi022 ceph-mon[31360]: pgmap v18292: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:08:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:22 smithi022 ceph-mon[40250]: pgmap v18292: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:08:23.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:08:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:08:23.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:24 smithi022 ceph-mon[40250]: pgmap v18293: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:08:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1312601318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:24 smithi022 ceph-mon[31360]: pgmap v18293: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:08:24.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1312601318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:24 smithi122 ceph-mon[32255]: pgmap v18293: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:08:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1312601318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:26 smithi022 ceph-mon[31360]: pgmap v18294: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T04:08:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:26 smithi022 ceph-mon[40250]: pgmap v18294: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T04:08:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:26 smithi122 ceph-mon[32255]: pgmap v18294: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2021-12-16T04:08:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:08:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:08:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:28 smithi122 ceph-mon[32255]: pgmap v18295: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 8 op/s 2021-12-16T04:08:28.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:28.976 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:28 smithi022 ceph-mon[31360]: pgmap v18295: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 8 op/s 2021-12-16T04:08:28.976 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:28 smithi022 ceph-mon[40250]: pgmap v18295: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 8 op/s 2021-12-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:29.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3315818116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3315818116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:29.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3315818116' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:08:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:30] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:08:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:30 smithi122 ceph-mon[32255]: pgmap v18296: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 0 B/s wr, 8 op/s 2021-12-16T04:08:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:30 smithi022 ceph-mon[31360]: pgmap v18296: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 0 B/s wr, 8 op/s 2021-12-16T04:08:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:30 smithi022 ceph-mon[40250]: pgmap v18296: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 0 B/s wr, 8 op/s 2021-12-16T04:08:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:32 smithi122 ceph-mon[32255]: pgmap v18297: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:08:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:08:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:08:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:32 smithi022 ceph-mon[31360]: pgmap v18297: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:32 smithi022 ceph-mon[40250]: pgmap v18297: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:33.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:08:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:08:34.351 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:34.722 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:34.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:34.749 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:34.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[40250]: pgmap v18298: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:34.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:34.750 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:08:34.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:08:34.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:08:34.751 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:08:34.752 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[31360]: pgmap v18298: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:34.752 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:34.752 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:08:34.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:08:34.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:08:34.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:08:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:34 smithi122 ceph-mon[32255]: pgmap v18298: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:08:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:08:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:08:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:08:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3959592031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3959592031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3959592031' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:36 smithi122 ceph-mon[32255]: pgmap v18299: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:36 smithi022 ceph-mon[31360]: pgmap v18299: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:36 smithi022 ceph-mon[40250]: pgmap v18299: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:08:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:08:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:38 smithi122 ceph-mon[32255]: pgmap v18300: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:38 smithi022 ceph-mon[31360]: pgmap v18300: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:38 smithi022 ceph-mon[40250]: pgmap v18300: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:40.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:40.095 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:40.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:08:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:40] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:08:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:40 smithi122 ceph-mon[32255]: pgmap v18301: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3630254675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:40 smithi022 ceph-mon[31360]: pgmap v18301: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3630254675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:40 smithi022 ceph-mon[40250]: pgmap v18301: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:41.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3630254675' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:42 smithi122 ceph-mon[32255]: pgmap v18302: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:08:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:08:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:08:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:42 smithi022 ceph-mon[31360]: pgmap v18302: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:43.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:42 smithi022 ceph-mon[40250]: pgmap v18302: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:08:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:08:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:44 smithi122 ceph-mon[32255]: pgmap v18303: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:44 smithi022 ceph-mon[31360]: pgmap v18303: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:44 smithi022 ceph-mon[40250]: pgmap v18303: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:45.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:45.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:45.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:45.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/411981316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/411981316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/411981316' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:46 smithi122 ceph-mon[32255]: pgmap v18304: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:46 smithi022 ceph-mon[31360]: pgmap v18304: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:46 smithi022 ceph-mon[40250]: pgmap v18304: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:08:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:08:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:48 smithi122 ceph-mon[32255]: pgmap v18305: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:48 smithi022 ceph-mon[31360]: pgmap v18305: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:48 smithi022 ceph-mon[40250]: pgmap v18305: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:08:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:50] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:08:50.472 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:50 smithi022 ceph-mon[31360]: pgmap v18306: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:50 smithi022 ceph-mon[40250]: pgmap v18306: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:50.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:50.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:50.842 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:50 smithi122 ceph-mon[32255]: pgmap v18306: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2196242811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2196242811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:52.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2196242811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:52.581 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:08:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:08:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:08:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:52 smithi122 ceph-mon[32255]: pgmap v18307: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:52 smithi022 ceph-mon[31360]: pgmap v18307: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:52 smithi022 ceph-mon[40250]: pgmap v18307: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:53.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:08:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:08:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:54 smithi122 ceph-mon[32255]: pgmap v18308: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:54 smithi022 ceph-mon[31360]: pgmap v18308: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:54 smithi022 ceph-mon[40250]: pgmap v18308: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:08:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:08:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:08:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:08:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:08:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:56 smithi122 ceph-mon[32255]: pgmap v18309: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2349371686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:57.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:56 smithi022 ceph-mon[31360]: pgmap v18309: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2349371686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:57.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:56 smithi022 ceph-mon[40250]: pgmap v18309: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:57.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2349371686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:08:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:08:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:08:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:58 smithi122 ceph-mon[32255]: pgmap v18310: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:58 smithi022 ceph-mon[31360]: pgmap v18310: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:58 smithi022 ceph-mon[40250]: pgmap v18310: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:08:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:08:59 smithi122 ceph-mon[32255]: pgmap v18311: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:00.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:08:59 smithi022 ceph-mon[31360]: pgmap v18311: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:00.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:08:59 smithi022 ceph-mon[40250]: pgmap v18311: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:09:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:00] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:09:01.217 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:01.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:01.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:01.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:02 smithi022 ceph-mon[31360]: pgmap v18312: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4028316459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:02 smithi022 ceph-mon[40250]: pgmap v18312: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4028316459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:02 smithi122 ceph-mon[32255]: pgmap v18312: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4028316459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:09:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:09:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:09:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:09:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:09:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:04 smithi122 ceph-mon[32255]: pgmap v18313: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:04 smithi022 ceph-mon[31360]: pgmap v18313: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:04 smithi022 ceph-mon[40250]: pgmap v18313: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:06.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:06 smithi022 ceph-mon[31360]: pgmap v18314: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:06.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:06 smithi022 ceph-mon[40250]: pgmap v18314: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:06 smithi122 ceph-mon[32255]: pgmap v18314: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:06.959 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:06.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:06.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/149578145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:09:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:09:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/149578145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:07.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/149578145' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:08 smithi022 ceph-mon[31360]: pgmap v18315: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:08 smithi022 ceph-mon[40250]: pgmap v18315: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:08 smithi122 ceph-mon[32255]: pgmap v18315: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:09:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:10] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:09:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:10 smithi022 ceph-mon[31360]: pgmap v18316: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:10 smithi022 ceph-mon[40250]: pgmap v18316: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:10 smithi122 ceph-mon[32255]: pgmap v18316: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:11.450 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:09:11.450 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:09:11.451 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:11 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:09:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:09:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:09:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:11 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:09:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:09:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:09:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:11 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:09:11.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:12.331 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:12.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:12.332 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:12 smithi122 ceph-mon[32255]: pgmap v18317: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/738321148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:09:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:09:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:09:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:12 smithi022 ceph-mon[31360]: pgmap v18317: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/738321148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:12 smithi022 ceph-mon[40250]: pgmap v18317: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/738321148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:09:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:09:14.767 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:14 smithi022 ceph-mon[31360]: pgmap v18318: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:14.768 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:14.769 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:14 smithi022 ceph-mon[40250]: pgmap v18318: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:14.769 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:14 smithi122 ceph-mon[32255]: pgmap v18318: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[31360]: pgmap v18319: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[40250]: pgmap v18319: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:15 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:15 smithi122 ceph-mon[32255]: pgmap v18319: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:16.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:16.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:15 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:17.333 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:17.334 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:17 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:17.334 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:17 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:17.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:17 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:17.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:17.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:17.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:17.697 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:09:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:09:18.219 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:18 smithi022 ceph-mon[31360]: pgmap v18320: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/318598603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:18 smithi022 ceph-mon[40250]: pgmap v18320: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/318598603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:18.572 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:18 smithi122 ceph-mon[32255]: pgmap v18320: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:18.572 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/318598603' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:19.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:09:20.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:20 smithi122 ceph-mon[32255]: pgmap v18321: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:09:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:20] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:09:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:20 smithi022 ceph-mon[31360]: pgmap v18321: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:20 smithi022 ceph-mon[40250]: pgmap v18321: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:22 smithi122 ceph-mon[32255]: pgmap v18322: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:09:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:09:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:09:22.698 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:22 smithi022 ceph-mon[31360]: pgmap v18322: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:22.699 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:22 smithi022 ceph-mon[40250]: pgmap v18322: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:22.699 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:23.068 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:23.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:23.070 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1275031416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:09:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:09:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1275031416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:23.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1275031416' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:24 smithi022 ceph-mon[31360]: pgmap v18323: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:24 smithi022 ceph-mon[40250]: pgmap v18323: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:24 smithi122 ceph-mon[32255]: pgmap v18323: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:26 smithi022 ceph-mon[31360]: pgmap v18324: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:26 smithi022 ceph-mon[40250]: pgmap v18324: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:26 smithi122 ceph-mon[32255]: pgmap v18324: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:09:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:09:28.070 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:28.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:28.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:28.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:28 smithi022 ceph-mon[31360]: pgmap v18325: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1418782852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:28 smithi022 ceph-mon[40250]: pgmap v18325: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1418782852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:28 smithi122 ceph-mon[32255]: pgmap v18325: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1418782852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:09:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:30] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:09:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:30 smithi022 ceph-mon[31360]: pgmap v18326: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:30 smithi022 ceph-mon[40250]: pgmap v18326: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:30 smithi122 ceph-mon[32255]: pgmap v18326: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:32 smithi122 ceph-mon[32255]: pgmap v18327: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:09:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:09:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:09:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:32 smithi022 ceph-mon[31360]: pgmap v18327: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:32 smithi022 ceph-mon[40250]: pgmap v18327: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:33.814 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:33.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:33.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:09:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:09:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:34 smithi022 ceph-mon[31360]: pgmap v18328: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3483206804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:34.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:34 smithi022 ceph-mon[40250]: pgmap v18328: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3483206804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:34 smithi122 ceph-mon[32255]: pgmap v18328: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3483206804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:09:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:09:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:09:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:09:35.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:09:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:09:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:09:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:09:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:09:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:09:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:09:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:09:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:36 smithi022 ceph-mon[31360]: pgmap v18329: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:36 smithi022 ceph-mon[40250]: pgmap v18329: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:36 smithi122 ceph-mon[32255]: pgmap v18329: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:09:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:09:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:38 smithi022 ceph-mon[31360]: pgmap v18330: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:38 smithi022 ceph-mon[40250]: pgmap v18330: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:38 smithi122 ceph-mon[32255]: pgmap v18330: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:39.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:39.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:39.189 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:39.465 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2068438716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:39.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2068438716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2068438716' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:09:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:40] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:09:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:40 smithi022 ceph-mon[31360]: pgmap v18331: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:40 smithi022 ceph-mon[40250]: pgmap v18331: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:40 smithi122 ceph-mon[32255]: pgmap v18331: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:42 smithi122 ceph-mon[32255]: pgmap v18332: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:09:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:09:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:09:42.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:42 smithi022 ceph-mon[31360]: pgmap v18332: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:42.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:42 smithi022 ceph-mon[40250]: pgmap v18332: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:09:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:09:44.190 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:44.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:44.561 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:44.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:44 smithi022 ceph-mon[31360]: pgmap v18333: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:44.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:44 smithi022 ceph-mon[40250]: pgmap v18333: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:44.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:44 smithi122 ceph-mon[32255]: pgmap v18333: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1071394088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:45.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1071394088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1071394088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:46 smithi022 ceph-mon[31360]: pgmap v18334: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:46 smithi022 ceph-mon[40250]: pgmap v18334: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:46 smithi122 ceph-mon[32255]: pgmap v18334: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:09:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:09:48.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:48 smithi022 ceph-mon[31360]: pgmap v18335: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:48 smithi022 ceph-mon[40250]: pgmap v18335: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:48 smithi122 ceph-mon[32255]: pgmap v18335: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:49.563 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:49.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:49.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:49.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:09:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:50] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:09:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:50 smithi022 ceph-mon[31360]: pgmap v18336: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:50.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/338692134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:50 smithi022 ceph-mon[40250]: pgmap v18336: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:50.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/338692134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:50 smithi122 ceph-mon[32255]: pgmap v18336: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/338692134' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:52 smithi122 ceph-mon[32255]: pgmap v18337: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:09:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:09:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:09:52.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:52 smithi022 ceph-mon[40250]: pgmap v18337: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:52.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:52 smithi022 ceph-mon[31360]: pgmap v18337: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:09:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:09:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:54 smithi022 ceph-mon[31360]: pgmap v18338: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:54 smithi022 ceph-mon[40250]: pgmap v18338: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:54 smithi122 ceph-mon[32255]: pgmap v18338: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:09:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:09:54.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:09:55.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:09:55.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:09:55.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:09:55.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1238108376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1238108376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1238108376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:09:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:56 smithi022 ceph-mon[31360]: pgmap v18339: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:56 smithi022 ceph-mon[40250]: pgmap v18339: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:56 smithi122 ceph-mon[32255]: pgmap v18339: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:09:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:09:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:09:58 smithi022 ceph-mon[31360]: pgmap v18340: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:58.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:09:58 smithi022 ceph-mon[40250]: pgmap v18340: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:09:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:09:58 smithi122 ceph-mon[32255]: pgmap v18340: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:00.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:10:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:00] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:10:00.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:00.679 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:00.680 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:00.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:00 smithi022 ceph-mon[31360]: pgmap v18341: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:00.680 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T04:10:00.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:00 smithi022 ceph-mon[40250]: pgmap v18341: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:00.681 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T04:10:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:00 smithi122 ceph-mon[32255]: pgmap v18341: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T04:10:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3697827301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3697827301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:01.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3697827301' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:02 smithi122 ceph-mon[32255]: pgmap v18342: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:10:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:10:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:10:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:02 smithi022 ceph-mon[31360]: pgmap v18342: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:02 smithi022 ceph-mon[40250]: pgmap v18342: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:10:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:10:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:04 smithi022 ceph-mon[31360]: pgmap v18343: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:04.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:04 smithi022 ceph-mon[40250]: pgmap v18343: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:04 smithi122 ceph-mon[32255]: pgmap v18343: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:05.681 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:06.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:06.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:06.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:06 smithi022 ceph-mon[31360]: pgmap v18344: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2655457899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:06 smithi022 ceph-mon[40250]: pgmap v18344: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2655457899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:06 smithi122 ceph-mon[32255]: pgmap v18344: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2655457899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:10:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:10:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:08 smithi022 ceph-mon[31360]: pgmap v18345: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:08 smithi022 ceph-mon[40250]: pgmap v18345: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:08 smithi122 ceph-mon[32255]: pgmap v18345: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:10:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:10] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:10:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:10 smithi022 ceph-mon[31360]: pgmap v18346: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:10 smithi022 ceph-mon[40250]: pgmap v18346: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:10 smithi122 ceph-mon[32255]: pgmap v18346: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:11.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:11.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:11.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:11.423 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1275051072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:11.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1275051072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1275051072' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:10:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:10:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:10:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:12 smithi122 ceph-mon[32255]: pgmap v18347: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:12 smithi022 ceph-mon[31360]: pgmap v18347: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:12 smithi022 ceph-mon[40250]: pgmap v18347: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:10:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:10:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:14 smithi022 ceph-mon[31360]: pgmap v18348: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:14 smithi022 ceph-mon[40250]: pgmap v18348: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:14.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:14 smithi122 ceph-mon[32255]: pgmap v18348: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:16.426 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:16 smithi022 ceph-mon[31360]: pgmap v18349: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:16 smithi022 ceph-mon[40250]: pgmap v18349: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:16.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:16.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:16.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:16 smithi122 ceph-mon[32255]: pgmap v18349: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3712957765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:10:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:10:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3712957765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3712957765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:18.822 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:18 smithi122 ceph-mon[32255]: pgmap v18350: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:18.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:18 smithi022 ceph-mon[31360]: pgmap v18350: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:18.832 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:18 smithi022 ceph-mon[40250]: pgmap v18350: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:10:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:10:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:10:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:19 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:10:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:10:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:10:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:10:20.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:10:20.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:10:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:10:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:10:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:19 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:10:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:10:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:20] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T04:10:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:20 smithi122 ceph-mon[32255]: pgmap v18351: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:20 smithi022 ceph-mon[31360]: pgmap v18351: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:20 smithi022 ceph-mon[40250]: pgmap v18351: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:21.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:22.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:22.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:22.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:22 smithi122 ceph-mon[32255]: pgmap v18352: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4241514141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:10:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:10:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:10:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:22 smithi022 ceph-mon[31360]: pgmap v18352: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:23.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4241514141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:22 smithi022 ceph-mon[40250]: pgmap v18352: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4241514141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:23.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:10:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:10:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:24 smithi122 ceph-mon[32255]: pgmap v18353: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:24 smithi022 ceph-mon[31360]: pgmap v18353: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:25.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:24 smithi022 ceph-mon[40250]: pgmap v18353: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:25.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:26 smithi122 ceph-mon[32255]: pgmap v18354: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:26 smithi022 ceph-mon[31360]: pgmap v18354: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:26 smithi022 ceph-mon[40250]: pgmap v18354: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:27.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:10:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:10:27.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:27.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:27.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/351588991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:28.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/351588991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:28.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/351588991' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:28 smithi122 ceph-mon[32255]: pgmap v18355: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:28 smithi022 ceph-mon[31360]: pgmap v18355: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:28 smithi022 ceph-mon[40250]: pgmap v18355: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:10:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:30] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T04:10:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:30 smithi122 ceph-mon[32255]: pgmap v18356: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:30 smithi022 ceph-mon[31360]: pgmap v18356: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:30 smithi022 ceph-mon[40250]: pgmap v18356: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:32.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:32 smithi122 ceph-mon[32255]: pgmap v18357: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:10:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:10:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:10:32.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:32.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:32.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:32.915 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:32 smithi022 ceph-mon[40250]: pgmap v18357: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:32.915 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:32 smithi022 ceph-mon[31360]: pgmap v18357: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3543038352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:10:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:10:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3543038352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3543038352' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:34 smithi122 ceph-mon[32255]: pgmap v18358: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:10:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:10:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:10:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:10:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[31360]: pgmap v18358: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:10:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:10:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:10:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:10:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[40250]: pgmap v18358: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:10:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:10:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:10:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:10:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:36 smithi122 ceph-mon[32255]: pgmap v18359: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:36 smithi022 ceph-mon[31360]: pgmap v18359: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:37.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:36 smithi022 ceph-mon[40250]: pgmap v18359: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:10:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:10:37.915 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:38.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:38.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:38.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:38 smithi122 ceph-mon[32255]: pgmap v18360: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/564238694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:38 smithi022 ceph-mon[31360]: pgmap v18360: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/564238694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:38 smithi022 ceph-mon[40250]: pgmap v18360: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/564238694' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:10:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:40] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T04:10:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:40 smithi122 ceph-mon[32255]: pgmap v18361: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:40 smithi022 ceph-mon[31360]: pgmap v18361: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:40 smithi022 ceph-mon[40250]: pgmap v18361: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:42.578 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:10:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:10:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:10:42.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:42 smithi122 ceph-mon[32255]: pgmap v18362: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:42 smithi022 ceph-mon[31360]: pgmap v18362: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:42 smithi022 ceph-mon[40250]: pgmap v18362: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:43.287 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:10:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:10:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:44 smithi122 ceph-mon[32255]: pgmap v18363: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1785472668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:44 smithi022 ceph-mon[31360]: pgmap v18363: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1785472668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:44 smithi022 ceph-mon[40250]: pgmap v18363: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1785472668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:46 smithi122 ceph-mon[32255]: pgmap v18364: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:46 smithi022 ceph-mon[31360]: pgmap v18364: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:46 smithi022 ceph-mon[40250]: pgmap v18364: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:10:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:10:48.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:48 smithi122 ceph-mon[32255]: pgmap v18365: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:48 smithi022 ceph-mon[31360]: pgmap v18365: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:49.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:48 smithi022 ceph-mon[40250]: pgmap v18365: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:49.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:49.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:49.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/512406583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:50.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/512406583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:50.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/512406583' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:10:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:50] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:10:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:50 smithi122 ceph-mon[32255]: pgmap v18366: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:50 smithi022 ceph-mon[31360]: pgmap v18366: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:50 smithi022 ceph-mon[40250]: pgmap v18366: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:52.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:10:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:10:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:10:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:52 smithi122 ceph-mon[32255]: pgmap v18367: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:52 smithi022 ceph-mon[31360]: pgmap v18367: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:52 smithi022 ceph-mon[40250]: pgmap v18367: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:10:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:10:54.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:54.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:54.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:10:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:54 smithi022 ceph-mon[31360]: pgmap v18368: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1213098655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:54 smithi022 ceph-mon[40250]: pgmap v18368: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1213098655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:54 smithi122 ceph-mon[32255]: pgmap v18368: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:10:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:10:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1213098655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:10:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:56 smithi122 ceph-mon[32255]: pgmap v18369: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:56 smithi022 ceph-mon[31360]: pgmap v18369: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:56 smithi022 ceph-mon[40250]: pgmap v18369: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:10:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:10:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:10:58 smithi122 ceph-mon[32255]: pgmap v18370: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:10:58 smithi022 ceph-mon[31360]: pgmap v18370: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:59.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:10:58 smithi022 ceph-mon[40250]: pgmap v18370: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:10:59.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:10:59.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:10:59.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:10:59.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:11:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:00] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:11:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:00 smithi122 ceph-mon[32255]: pgmap v18371: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/526562632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:01.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:00 smithi022 ceph-mon[31360]: pgmap v18371: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/526562632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:00 smithi022 ceph-mon[40250]: pgmap v18371: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/526562632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:02.609 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:11:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:11:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:11:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:02 smithi122 ceph-mon[32255]: pgmap v18372: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:03.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:02 smithi022 ceph-mon[31360]: pgmap v18372: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:03.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:02 smithi022 ceph-mon[40250]: pgmap v18372: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:11:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:11:04.774 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:04.774 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:04 smithi022 ceph-mon[31360]: pgmap v18373: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:04.775 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:04 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:04.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:04 smithi022 ceph-mon[40250]: pgmap v18373: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:04.775 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:04 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:04 smithi122 ceph-mon[32255]: pgmap v18373: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:04 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:05.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:05.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:05.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2711723274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2711723274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2711723274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:06 smithi022 ceph-mon[31360]: pgmap v18374: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:06 smithi022 ceph-mon[40250]: pgmap v18374: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:06 smithi122 ceph-mon[32255]: pgmap v18374: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:11:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:11:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:08 smithi022 ceph-mon[31360]: pgmap v18375: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:08 smithi022 ceph-mon[40250]: pgmap v18375: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:08 smithi122 ceph-mon[32255]: pgmap v18375: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:10.145 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:11:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:10] "GET /metrics HTTP/1.1" 200 202501 "" "Prometheus/2.18.1" 2021-12-16T04:11:10.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:10.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:10.514 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:10 smithi022 ceph-mon[31360]: pgmap v18376: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:10.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:10 smithi022 ceph-mon[40250]: pgmap v18376: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:10 smithi122 ceph-mon[32255]: pgmap v18376: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:11.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3900339519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:11.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3900339519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3900339519' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:11:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:11:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:11:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:12 smithi122 ceph-mon[32255]: pgmap v18377: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:12 smithi022 ceph-mon[31360]: pgmap v18377: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:12 smithi022 ceph-mon[40250]: pgmap v18377: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:11:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:11:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:14 smithi022 ceph-mon[31360]: pgmap v18378: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:14.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:14 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:14 smithi022 ceph-mon[40250]: pgmap v18378: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:14 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:14 smithi122 ceph-mon[32255]: pgmap v18378: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:14 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:15.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:15.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:15.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:16 smithi022 ceph-mon[31360]: pgmap v18379: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1658010193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:16 smithi022 ceph-mon[40250]: pgmap v18379: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1658010193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:16 smithi122 ceph-mon[32255]: pgmap v18379: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1658010193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:11:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:11:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:18 smithi022 ceph-mon[31360]: pgmap v18380: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:18 smithi022 ceph-mon[40250]: pgmap v18380: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:18 smithi122 ceph-mon[32255]: pgmap v18380: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:11:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:11:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:19 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:11:19.723 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:11:19.724 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:11:19.724 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:19 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:11:19.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:11:19.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:11:19.725 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:19 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:11:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:11:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:20] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:11:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:20 smithi022 ceph-mon[31360]: pgmap v18381: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:11:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:20 smithi022 ceph-mon[40250]: pgmap v18381: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:20.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:11:20.884 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:20 smithi122 ceph-mon[32255]: pgmap v18381: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:11:21.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:21.254 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:21.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:21.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2415923543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2415923543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2415923543' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:22 smithi122 ceph-mon[32255]: pgmap v18382: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:11:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:11:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:11:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:22 smithi022 ceph-mon[31360]: pgmap v18382: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:22 smithi022 ceph-mon[40250]: pgmap v18382: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:23.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:11:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:11:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:24 smithi022 ceph-mon[31360]: pgmap v18383: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:24 smithi022 ceph-mon[40250]: pgmap v18383: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:24 smithi122 ceph-mon[32255]: pgmap v18383: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:26.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:26 smithi022 ceph-mon[31360]: pgmap v18384: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:26.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:26 smithi022 ceph-mon[40250]: pgmap v18384: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:26 smithi122 ceph-mon[32255]: pgmap v18384: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:27.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2410990707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:11:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:11:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2410990707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:27.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2410990707' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:28.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:28 smithi022 ceph-mon[31360]: pgmap v18385: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:28 smithi022 ceph-mon[40250]: pgmap v18385: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:28 smithi122 ceph-mon[32255]: pgmap v18385: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:11:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:30] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:11:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:30 smithi022 ceph-mon[31360]: pgmap v18386: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:30.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:30 smithi022 ceph-mon[40250]: pgmap v18386: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:30 smithi122 ceph-mon[32255]: pgmap v18386: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:31.630 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:31.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:31.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:31.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:32 smithi122 ceph-mon[32255]: pgmap v18387: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2818233289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:11:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:11:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:11:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:32 smithi022 ceph-mon[31360]: pgmap v18387: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2818233289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:32 smithi022 ceph-mon[40250]: pgmap v18387: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2818233289' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:11:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:11:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[31360]: pgmap v18388: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:34.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:11:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:11:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:11:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[40250]: pgmap v18388: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:11:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:11:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:11:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:11:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:34 smithi122 ceph-mon[32255]: pgmap v18388: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:11:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:11:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:11:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:11:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:36 smithi022 ceph-mon[31360]: pgmap v18389: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:36 smithi022 ceph-mon[40250]: pgmap v18389: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:36 smithi122 ceph-mon[32255]: pgmap v18389: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:36.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:37.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:37.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:37.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2340927237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:11:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:11:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2340927237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2340927237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:38 smithi022 ceph-mon[31360]: pgmap v18390: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:38 smithi022 ceph-mon[40250]: pgmap v18390: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:38 smithi122 ceph-mon[32255]: pgmap v18390: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:11:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:40] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T04:11:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:40 smithi122 ceph-mon[32255]: pgmap v18391: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:41.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:40 smithi022 ceph-mon[31360]: pgmap v18391: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:41.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:40 smithi022 ceph-mon[40250]: pgmap v18391: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:42.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:42 smithi122 ceph-mon[32255]: pgmap v18392: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:11:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:11:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:11:42.737 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:42.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:42.738 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:42.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:42 smithi022 ceph-mon[31360]: pgmap v18392: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:42.739 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:42 smithi022 ceph-mon[40250]: pgmap v18392: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:11:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:11:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1300474386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:44.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1300474386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:44.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1300474386' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:44 smithi122 ceph-mon[32255]: pgmap v18393: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:44 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:44 smithi022 ceph-mon[31360]: pgmap v18393: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:44 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:44 smithi022 ceph-mon[40250]: pgmap v18393: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:44 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:46 smithi122 ceph-mon[32255]: pgmap v18394: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:46 smithi022 ceph-mon[31360]: pgmap v18394: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:47.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:46 smithi022 ceph-mon[40250]: pgmap v18394: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:47.739 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:47.740 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:11:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:11:48.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:48.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:48.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:48 smithi122 ceph-mon[32255]: pgmap v18395: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/692706094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:48 smithi022 ceph-mon[31360]: pgmap v18395: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/692706094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:48 smithi022 ceph-mon[40250]: pgmap v18395: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/692706094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:11:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:50] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:11:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:50 smithi122 ceph-mon[32255]: pgmap v18396: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:50 smithi022 ceph-mon[31360]: pgmap v18396: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:50 smithi022 ceph-mon[40250]: pgmap v18396: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:52 smithi122 ceph-mon[32255]: pgmap v18397: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:11:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:11:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:11:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:52 smithi022 ceph-mon[31360]: pgmap v18397: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:53.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:52 smithi022 ceph-mon[40250]: pgmap v18397: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:53.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:53.484 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:53.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:53.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:11:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:11:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:54 smithi022 ceph-mon[31360]: pgmap v18398: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:54.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1117392444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:54 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:54 smithi022 ceph-mon[40250]: pgmap v18398: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:54.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1117392444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:54 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:54 smithi122 ceph-mon[32255]: pgmap v18398: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:11:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1117392444' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:54 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:11:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:56 smithi022 ceph-mon[31360]: pgmap v18399: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:56 smithi022 ceph-mon[40250]: pgmap v18399: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:56 smithi122 ceph-mon[32255]: pgmap v18399: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:11:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:11:58.486 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:11:58.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:58 smithi022 ceph-mon[31360]: pgmap v18400: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:58.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:58 smithi022 ceph-mon[40250]: pgmap v18400: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:58.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:11:58.946 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:11:58.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:11:58.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:58 smithi122 ceph-mon[32255]: pgmap v18400: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:11:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:11:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/925640363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:11:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/925640363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:11:59.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:11:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/925640363' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:12:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:00] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:12:00.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:00 smithi022 ceph-mon[31360]: pgmap v18401: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:00 smithi022 ceph-mon[40250]: pgmap v18401: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:00 smithi122 ceph-mon[32255]: pgmap v18401: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:02 smithi122 ceph-mon[32255]: pgmap v18402: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:12:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:12:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:12:02.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:02 smithi022 ceph-mon[31360]: pgmap v18402: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:02 smithi022 ceph-mon[40250]: pgmap v18402: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:03.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:12:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:12:04.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:04.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:04.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:04.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:04 smithi022 ceph-mon[31360]: pgmap v18403: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:04.491 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2194020691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:04.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:04 smithi022 ceph-mon[40250]: pgmap v18403: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:04.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2194020691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:04 smithi122 ceph-mon[32255]: pgmap v18403: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2194020691' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:05.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:06 smithi022 ceph-mon[31360]: pgmap v18404: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:06 smithi022 ceph-mon[40250]: pgmap v18404: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:06 smithi122 ceph-mon[32255]: pgmap v18404: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:12:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:12:08.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:08 smithi022 ceph-mon[31360]: pgmap v18405: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:08.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:08 smithi022 ceph-mon[40250]: pgmap v18405: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:08 smithi122 ceph-mon[32255]: pgmap v18405: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:09.231 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:09.598 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:09.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:09.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:12:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:10] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T04:12:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:10 smithi022 ceph-mon[31360]: pgmap v18406: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:10.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4243054518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:10 smithi022 ceph-mon[40250]: pgmap v18406: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:10.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4243054518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:10 smithi122 ceph-mon[32255]: pgmap v18406: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4243054518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:12.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:12 smithi122 ceph-mon[32255]: pgmap v18407: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:12:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:12:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:12:12.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:12 smithi022 ceph-mon[31360]: pgmap v18407: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:12.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:12 smithi022 ceph-mon[40250]: pgmap v18407: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:12:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:12:14.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:14.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:14 smithi022 ceph-mon[31360]: pgmap v18408: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:14.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:14 smithi022 ceph-mon[40250]: pgmap v18408: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:14 smithi122 ceph-mon[32255]: pgmap v18408: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:14.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:14.979 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:14.980 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:15.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/551310760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:15.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/551310760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/551310760' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:16.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:16 smithi022 ceph-mon[31360]: pgmap v18409: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:16.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:16 smithi022 ceph-mon[40250]: pgmap v18409: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:16 smithi122 ceph-mon[32255]: pgmap v18409: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:17.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:12:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:12:18.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:18 smithi022 ceph-mon[31360]: pgmap v18410: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:18.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:18 smithi022 ceph-mon[40250]: pgmap v18410: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:18 smithi122 ceph-mon[32255]: pgmap v18410: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:19.973 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:20.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:20.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:20.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:12:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:20] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:12:20.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[31360]: pgmap v18411: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:12:20.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3295211873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[40250]: pgmap v18411: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:12:20.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:12:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:12:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:12:20.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3295211873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:20 smithi122 ceph-mon[32255]: pgmap v18411: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:12:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:12:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:12:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:20 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:12:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3295211873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:22.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:22 smithi122 ceph-mon[32255]: pgmap v18412: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:12:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:12:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:12:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:22 smithi022 ceph-mon[31360]: pgmap v18412: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:22 smithi022 ceph-mon[40250]: pgmap v18412: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:23.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:12:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:12:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:24 smithi022 ceph-mon[31360]: pgmap v18413: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:24.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:24 smithi022 ceph-mon[40250]: pgmap v18413: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:24 smithi122 ceph-mon[32255]: pgmap v18413: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:25.334 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:25.707 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:25.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:25.708 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:25.709 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:25.709 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:26 smithi122 ceph-mon[32255]: pgmap v18414: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1762939828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:26 smithi022 ceph-mon[31360]: pgmap v18414: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1762939828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:26 smithi022 ceph-mon[40250]: pgmap v18414: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1762939828' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:12:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:12:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:28 smithi122 ceph-mon[32255]: pgmap v18415: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:28 smithi022 ceph-mon[31360]: pgmap v18415: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:29.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:28 smithi022 ceph-mon[40250]: pgmap v18415: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:12:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:30] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:12:30.709 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:30.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:30 smithi122 ceph-mon[32255]: pgmap v18416: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:31.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:30 smithi022 ceph-mon[40250]: pgmap v18416: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:31.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:30 smithi022 ceph-mon[31360]: pgmap v18416: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:31.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:31.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:31.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2953395910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2953395910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:32.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2953395910' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:32.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:32 smithi122 ceph-mon[32255]: pgmap v18417: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:12:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:12:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:12:33.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:32 smithi022 ceph-mon[31360]: pgmap v18417: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:33.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:32 smithi022 ceph-mon[40250]: pgmap v18417: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:12:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:12:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:34 smithi122 ceph-mon[32255]: pgmap v18418: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:12:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:12:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:12:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:12:35.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[31360]: pgmap v18418: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:12:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:12:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:12:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:12:35.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[40250]: pgmap v18418: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:12:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:12:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:12:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:12:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:36.080 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:36 smithi022 ceph-mon[31360]: pgmap v18419: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2479364588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:36 smithi022 ceph-mon[40250]: pgmap v18419: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2479364588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:36 smithi122 ceph-mon[32255]: pgmap v18419: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2479364588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:12:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:12:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:38 smithi122 ceph-mon[32255]: pgmap v18420: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:39.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:38 smithi022 ceph-mon[31360]: pgmap v18420: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:38 smithi022 ceph-mon[40250]: pgmap v18420: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:12:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:40] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:12:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:40 smithi122 ceph-mon[32255]: pgmap v18421: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:41.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:40 smithi022 ceph-mon[31360]: pgmap v18421: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:41.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:40 smithi022 ceph-mon[40250]: pgmap v18421: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:41.452 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:41.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:41.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:41.823 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:42 smithi122 ceph-mon[32255]: pgmap v18422: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2388879715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:12:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:12:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:12:43.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:42 smithi022 ceph-mon[31360]: pgmap v18422: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2388879715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:42 smithi022 ceph-mon[40250]: pgmap v18422: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:43.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2388879715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:43.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:12:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:12:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:44 smithi122 ceph-mon[32255]: pgmap v18423: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:44 smithi022 ceph-mon[31360]: pgmap v18423: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:45.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:44 smithi022 ceph-mon[40250]: pgmap v18423: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:46.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:46.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:46.825 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:46 smithi022 ceph-mon[31360]: pgmap v18424: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:46.825 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:46 smithi022 ceph-mon[40250]: pgmap v18424: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:46 smithi122 ceph-mon[32255]: pgmap v18424: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:47.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:47.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:47.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:47.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:12:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:12:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3100598066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3100598066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:48.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3100598066' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:48 smithi122 ceph-mon[32255]: pgmap v18425: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:49.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:48 smithi022 ceph-mon[31360]: pgmap v18425: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:48 smithi022 ceph-mon[40250]: pgmap v18425: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:12:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:50] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:12:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:50 smithi122 ceph-mon[32255]: pgmap v18426: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:50 smithi022 ceph-mon[31360]: pgmap v18426: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:50 smithi022 ceph-mon[40250]: pgmap v18426: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:52.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:52.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:52.578 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:12:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:12:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:12:52.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:52 smithi122 ceph-mon[32255]: pgmap v18427: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3406002827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:52 smithi022 ceph-mon[31360]: pgmap v18427: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:53.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3406002827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:52 smithi022 ceph-mon[40250]: pgmap v18427: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3406002827' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:53.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:12:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:12:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:54 smithi122 ceph-mon[32255]: pgmap v18428: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:54 smithi022 ceph-mon[31360]: pgmap v18428: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:54 smithi022 ceph-mon[40250]: pgmap v18428: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:12:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:56.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:12:56.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:56 smithi122 ceph-mon[32255]: pgmap v18429: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:57.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:56 smithi022 ceph-mon[31360]: pgmap v18429: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:57.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:56 smithi022 ceph-mon[40250]: pgmap v18429: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:57.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:12:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:12:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:12:57.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:12:57.933 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:12:57.934 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:12:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:58 smithi122 ceph-mon[32255]: pgmap v18430: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:58.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:12:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3799304326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:58 smithi022 ceph-mon[31360]: pgmap v18430: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:12:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3799304326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:12:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:58 smithi022 ceph-mon[40250]: pgmap v18430: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:12:59.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:12:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3799304326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:13:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:00] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:13:00.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:00 smithi122 ceph-mon[32255]: pgmap v18431: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:00 smithi022 ceph-mon[31360]: pgmap v18431: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:00 smithi022 ceph-mon[40250]: pgmap v18431: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:02.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:13:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:13:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:13:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:02 smithi122 ceph-mon[32255]: pgmap v18432: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:02.938 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:02 smithi022 ceph-mon[40250]: pgmap v18432: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:02.938 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:02 smithi022 ceph-mon[31360]: pgmap v18432: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:03.307 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:03.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:13:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:13:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1628929564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:04.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1628929564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:04.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1628929564' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:04 smithi122 ceph-mon[32255]: pgmap v18433: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:05.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:04 smithi022 ceph-mon[31360]: pgmap v18433: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:05.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:04 smithi022 ceph-mon[40250]: pgmap v18433: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:06.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:06.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:06 smithi122 ceph-mon[32255]: pgmap v18434: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:07.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:06 smithi022 ceph-mon[31360]: pgmap v18434: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:07.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:06 smithi022 ceph-mon[40250]: pgmap v18434: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:07.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:13:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:13:08.309 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:08.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:08.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:08.681 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:08 smithi122 ceph-mon[32255]: pgmap v18435: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:08 smithi022 ceph-mon[31360]: pgmap v18435: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:08 smithi022 ceph-mon[40250]: pgmap v18435: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:09.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1218281700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:10.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1218281700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:10.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1218281700' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:13:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:10] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:13:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:10 smithi122 ceph-mon[32255]: pgmap v18436: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:10 smithi022 ceph-mon[31360]: pgmap v18436: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:10 smithi022 ceph-mon[40250]: pgmap v18436: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:12.618 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:13:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:13:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:13:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:12 smithi122 ceph-mon[32255]: pgmap v18437: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:12 smithi022 ceph-mon[31360]: pgmap v18437: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:12 smithi022 ceph-mon[40250]: pgmap v18437: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:13.683 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:13:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:13:14.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:14.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:14.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:14 smithi122 ceph-mon[32255]: pgmap v18438: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2990853853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:15.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:14 smithi022 ceph-mon[31360]: pgmap v18438: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2990853853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:14 smithi022 ceph-mon[40250]: pgmap v18438: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2990853853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:16 smithi122 ceph-mon[32255]: pgmap v18439: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:17.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:16 smithi022 ceph-mon[31360]: pgmap v18439: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:17.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:16 smithi022 ceph-mon[40250]: pgmap v18439: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:13:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:13:18.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:18 smithi122 ceph-mon[32255]: pgmap v18440: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:19.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:18 smithi022 ceph-mon[31360]: pgmap v18440: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:19.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:18 smithi022 ceph-mon[40250]: pgmap v18440: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:19.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:19.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:19.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:19.427 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1919545657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1919545657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1919545657' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:13:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:20] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:13:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:20 smithi122 ceph-mon[32255]: pgmap v18441: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:20.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:13:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:13:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:20 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:13:21.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[31360]: pgmap v18441: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:13:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:13:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:13:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[40250]: pgmap v18441: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:13:21.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:13:21.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:20 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:13:21.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:13:21.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:21 smithi122 ceph-mon[32255]: pgmap v18442: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:13:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:21 smithi022 ceph-mon[31360]: pgmap v18442: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:13:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:21 smithi022 ceph-mon[40250]: pgmap v18442: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:13:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:13:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:13:23.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:13:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:13:24.428 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:24.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:24 smithi022 ceph-mon[31360]: pgmap v18443: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:24.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:24 smithi022 ceph-mon[40250]: pgmap v18443: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:24.798 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:24.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:24.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:24 smithi122 ceph-mon[32255]: pgmap v18443: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:25.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:25.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3644545379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:25.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:25.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3644545379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3644545379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:26.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:26 smithi022 ceph-mon[31360]: pgmap v18444: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:26.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:26 smithi022 ceph-mon[40250]: pgmap v18444: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:26.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:26 smithi122 ceph-mon[32255]: pgmap v18444: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:13:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:13:28.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:28 smithi022 ceph-mon[31360]: pgmap v18445: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:28.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:28 smithi022 ceph-mon[40250]: pgmap v18445: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:28.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:28 smithi122 ceph-mon[32255]: pgmap v18445: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:29.800 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:30.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:30.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:30.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:13:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:30] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:13:30.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:30 smithi022 ceph-mon[31360]: pgmap v18446: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:30.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/806108501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:30 smithi022 ceph-mon[40250]: pgmap v18446: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:30.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/806108501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:30 smithi122 ceph-mon[32255]: pgmap v18446: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/806108501' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:13:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:13:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:13:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:32 smithi122 ceph-mon[32255]: pgmap v18447: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:32.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:32 smithi022 ceph-mon[31360]: pgmap v18447: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:32 smithi022 ceph-mon[40250]: pgmap v18447: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:33.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:13:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[31360]: pgmap v18448: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:13:34.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:13:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:13:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[40250]: pgmap v18448: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:34.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:13:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:13:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:13:34.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:13:34.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:34 smithi122 ceph-mon[32255]: pgmap v18448: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:13:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:13:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:13:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:13:35.171 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:35.520 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:35.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:35.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:35.543 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:36 smithi022 ceph-mon[31360]: pgmap v18449: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:36.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1059201658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:36 smithi022 ceph-mon[40250]: pgmap v18449: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:36.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1059201658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:36 smithi122 ceph-mon[32255]: pgmap v18449: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1059201658' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:37.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:13:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:13:38.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:38 smithi022 ceph-mon[31360]: pgmap v18450: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:38 smithi022 ceph-mon[40250]: pgmap v18450: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:38 smithi122 ceph-mon[32255]: pgmap v18450: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:13:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:40] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T04:13:40.544 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:40 smithi022 ceph-mon[31360]: pgmap v18451: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:40 smithi022 ceph-mon[40250]: pgmap v18451: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:40.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:40.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:40.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:40.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:40 smithi122 ceph-mon[32255]: pgmap v18451: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2046318307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2046318307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:42.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2046318307' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:42.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:42 smithi122 ceph-mon[32255]: pgmap v18452: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:13:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:13:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:13:43.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:42 smithi022 ceph-mon[31360]: pgmap v18452: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:42 smithi022 ceph-mon[40250]: pgmap v18452: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:13:44.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:13:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:13:44.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:44 smithi122 ceph-mon[32255]: pgmap v18453: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:45.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:44 smithi022 ceph-mon[31360]: pgmap v18453: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:45.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:44 smithi022 ceph-mon[40250]: pgmap v18453: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:45.915 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:45.915 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:45.916 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:45.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:46.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:46.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:46.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:46.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:46 smithi122 ceph-mon[32255]: pgmap v18454: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3149784392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:46 smithi022 ceph-mon[31360]: pgmap v18454: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3149784392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:46 smithi022 ceph-mon[40250]: pgmap v18454: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3149784392' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:47.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:13:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:13:48.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:48 smithi122 ceph-mon[32255]: pgmap v18455: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:48 smithi022 ceph-mon[31360]: pgmap v18455: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:49.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:48 smithi022 ceph-mon[40250]: pgmap v18455: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:13:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:50] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:13:50.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:50 smithi122 ceph-mon[32255]: pgmap v18456: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:51.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:50 smithi022 ceph-mon[31360]: pgmap v18456: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:51.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:50 smithi022 ceph-mon[40250]: pgmap v18456: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:51.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:51.660 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:51.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:51.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:13:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:13:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:13:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:52 smithi122 ceph-mon[32255]: pgmap v18457: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3934125513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:53.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:52 smithi022 ceph-mon[40250]: pgmap v18457: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:53.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3934125513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:52 smithi022 ceph-mon[31360]: pgmap v18457: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:53.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3934125513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:54.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:13:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:13:54.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:54 smithi122 ceph-mon[32255]: pgmap v18458: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:55.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:54 smithi022 ceph-mon[31360]: pgmap v18458: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:55.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:54 smithi022 ceph-mon[40250]: pgmap v18458: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:13:55.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:56.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:13:56.662 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:13:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:56 smithi122 ceph-mon[32255]: pgmap v18459: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:57.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:56 smithi022 ceph-mon[40250]: pgmap v18459: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:56 smithi022 ceph-mon[31360]: pgmap v18459: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:57.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:13:57.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:13:57.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:13:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1417681055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:13:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:13:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1417681055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:57.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1417681055' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:13:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:13:58 smithi122 ceph-mon[32255]: pgmap v18460: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:13:58 smithi022 ceph-mon[31360]: pgmap v18460: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:13:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:13:58 smithi022 ceph-mon[40250]: pgmap v18460: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:14:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:00] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:14:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:00 smithi122 ceph-mon[32255]: pgmap v18461: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:00 smithi022 ceph-mon[31360]: pgmap v18461: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:01.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:00 smithi022 ceph-mon[40250]: pgmap v18461: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:02.031 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:02.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:02.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:02.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:02.570 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:14:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:14:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:14:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:02 smithi022 ceph-mon[31360]: pgmap v18462: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:02.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2100951530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:02.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:02 smithi022 ceph-mon[40250]: pgmap v18462: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:02.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2100951530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:02.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:02 smithi122 ceph-mon[32255]: pgmap v18462: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:02.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2100951530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:04.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:14:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:14:04.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:04 smithi122 ceph-mon[32255]: pgmap v18463: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:04 smithi022 ceph-mon[31360]: pgmap v18463: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:04 smithi022 ceph-mon[40250]: pgmap v18463: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:05.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:06.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:06.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:06 smithi122 ceph-mon[32255]: pgmap v18464: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:07.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:06 smithi022 ceph-mon[31360]: pgmap v18464: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:06 smithi022 ceph-mon[40250]: pgmap v18464: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:07.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:07.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:07.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:07.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:07.772 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:14:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:14:08.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:08 smithi122 ceph-mon[32255]: pgmap v18465: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/930678977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:08 smithi022 ceph-mon[31360]: pgmap v18465: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/930678977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:09.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:08 smithi022 ceph-mon[40250]: pgmap v18465: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/930678977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:14:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:10] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:14:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:10 smithi122 ceph-mon[32255]: pgmap v18466: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:10 smithi022 ceph-mon[31360]: pgmap v18466: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:10 smithi022 ceph-mon[40250]: pgmap v18466: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:12.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:14:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:14:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:14:12.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:12 smithi122 ceph-mon[32255]: pgmap v18467: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:12 smithi022 ceph-mon[31360]: pgmap v18467: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:12 smithi022 ceph-mon[40250]: pgmap v18467: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:13.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:13.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:13.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:13.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:14:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:14:13.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/991783807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:14.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/991783807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:14.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/991783807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:14 smithi122 ceph-mon[32255]: pgmap v18468: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:14 smithi022 ceph-mon[31360]: pgmap v18468: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:15.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:14 smithi022 ceph-mon[40250]: pgmap v18468: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:16 smithi122 ceph-mon[32255]: pgmap v18469: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:16 smithi022 ceph-mon[31360]: pgmap v18469: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:16 smithi022 ceph-mon[40250]: pgmap v18469: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:14:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:14:18.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:18.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:18.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:18.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:18 smithi022 ceph-mon[31360]: pgmap v18470: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:18.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1985795367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:18 smithi022 ceph-mon[40250]: pgmap v18470: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:18.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1985795367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:18 smithi122 ceph-mon[32255]: pgmap v18470: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1985795367' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:14:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:20] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:14:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:20 smithi122 ceph-mon[32255]: pgmap v18471: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:20 smithi022 ceph-mon[31360]: pgmap v18471: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:20 smithi022 ceph-mon[40250]: pgmap v18471: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:14:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:14:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:14:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:14:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:14:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:14:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:14:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:14:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:14:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:14:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:14:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:14:22.607 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:14:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:14:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:14:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:22 smithi122 ceph-mon[32255]: pgmap v18472: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:23.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:22 smithi022 ceph-mon[31360]: pgmap v18472: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:23.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:22 smithi022 ceph-mon[40250]: pgmap v18472: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:23.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:23.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:23.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:23.890 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:24.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:14:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:14:24.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:24 smithi122 ceph-mon[32255]: pgmap v18473: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2393228376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:24 smithi022 ceph-mon[31360]: pgmap v18473: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2393228376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:24 smithi022 ceph-mon[40250]: pgmap v18473: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2393228376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:26.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:26 smithi122 ceph-mon[32255]: pgmap v18474: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:26 smithi022 ceph-mon[31360]: pgmap v18474: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:27.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:26 smithi022 ceph-mon[40250]: pgmap v18474: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:14:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:14:28.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:28.892 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:28 smithi022 ceph-mon[31360]: pgmap v18475: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:28.892 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:28 smithi022 ceph-mon[40250]: pgmap v18475: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:28 smithi122 ceph-mon[32255]: pgmap v18475: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:29.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:29.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:29.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:29.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2417191011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:30.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2417191011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:30.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2417191011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:14:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:30] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:14:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:30 smithi122 ceph-mon[32255]: pgmap v18476: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:30 smithi022 ceph-mon[31360]: pgmap v18476: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:31.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:30 smithi022 ceph-mon[40250]: pgmap v18476: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:32.626 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:14:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:14:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:14:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:32 smithi122 ceph-mon[32255]: pgmap v18477: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:32 smithi022 ceph-mon[31360]: pgmap v18477: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:33.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:32 smithi022 ceph-mon[40250]: pgmap v18477: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:34.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:14:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:14:34.263 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:34.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:34.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:34.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:34 smithi122 ceph-mon[32255]: pgmap v18478: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:14:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:14:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:14:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:14:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/552221157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[31360]: pgmap v18478: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:14:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:14:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:14:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:14:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/552221157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[40250]: pgmap v18478: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:14:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:14:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:14:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:14:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/552221157' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:36 smithi122 ceph-mon[32255]: pgmap v18479: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:36 smithi022 ceph-mon[31360]: pgmap v18479: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:36 smithi022 ceph-mon[40250]: pgmap v18479: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:14:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:14:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:38 smithi122 ceph-mon[32255]: pgmap v18480: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:38 smithi022 ceph-mon[31360]: pgmap v18480: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:39.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:38 smithi022 ceph-mon[40250]: pgmap v18480: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:39.633 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:40.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:40.000 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:40.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:14:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:40] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:14:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:40 smithi122 ceph-mon[32255]: pgmap v18481: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3839628040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:40 smithi022 ceph-mon[31360]: pgmap v18481: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3839628040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:41.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:40 smithi022 ceph-mon[40250]: pgmap v18481: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:41.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3839628040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:42.642 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:14:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:14:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:14:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:42 smithi122 ceph-mon[32255]: pgmap v18482: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:42 smithi022 ceph-mon[31360]: pgmap v18482: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:42 smithi022 ceph-mon[40250]: pgmap v18482: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:14:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:14:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:44 smithi122 ceph-mon[32255]: pgmap v18483: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:45.002 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:45.003 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:44 smithi022 ceph-mon[31360]: pgmap v18483: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:45.003 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:44 smithi022 ceph-mon[40250]: pgmap v18483: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:45.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:45.375 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:45.376 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4080223844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:46.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4080223844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4080223844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:46 smithi122 ceph-mon[32255]: pgmap v18484: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:46 smithi022 ceph-mon[31360]: pgmap v18484: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:46 smithi022 ceph-mon[40250]: pgmap v18484: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:47.659 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:14:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:14:47.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:47 smithi122 ceph-mon[32255]: pgmap v18485: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:48.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:47 smithi022 ceph-mon[31360]: pgmap v18485: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:47 smithi022 ceph-mon[40250]: pgmap v18485: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:50.377 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:14:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:50] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:14:50.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:50.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:50.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:50.749 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:50 smithi022 ceph-mon[31360]: pgmap v18486: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:50.749 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:50 smithi022 ceph-mon[40250]: pgmap v18486: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:50 smithi122 ceph-mon[32255]: pgmap v18486: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/361925803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:51.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/361925803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:51.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/361925803' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:52.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:52 smithi122 ceph-mon[32255]: pgmap v18487: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:14:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:14:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:14:52.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:52 smithi022 ceph-mon[31360]: pgmap v18487: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:52.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:52 smithi022 ceph-mon[40250]: pgmap v18487: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:54.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:14:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:14:54.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:54 smithi022 ceph-mon[31360]: pgmap v18488: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:54.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:54 smithi022 ceph-mon[40250]: pgmap v18488: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:54.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:54 smithi122 ceph-mon[32255]: pgmap v18488: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:14:55.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:14:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:14:56.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:14:56.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:14:56.120 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:14:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:56 smithi122 ceph-mon[32255]: pgmap v18489: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1317697478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:56 smithi022 ceph-mon[31360]: pgmap v18489: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:57.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1317697478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:56 smithi022 ceph-mon[40250]: pgmap v18489: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1317697478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:14:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:14:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:14:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:14:58 smithi122 ceph-mon[32255]: pgmap v18490: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:59.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:14:58 smithi022 ceph-mon[31360]: pgmap v18490: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:14:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:14:58 smithi022 ceph-mon[40250]: pgmap v18490: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:15:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:00] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:15:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:00 smithi122 ceph-mon[32255]: pgmap v18491: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:01.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:00 smithi022 ceph-mon[31360]: pgmap v18491: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:01.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:00 smithi022 ceph-mon[40250]: pgmap v18491: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:01.122 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:01.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2184501213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2184501213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:01.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2184501213' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:02.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:02 smithi122 ceph-mon[32255]: pgmap v18492: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:15:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:15:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:15:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:02 smithi022 ceph-mon[31360]: pgmap v18492: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:03.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:02 smithi022 ceph-mon[40250]: pgmap v18492: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:04.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:15:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:15:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:04 smithi122 ceph-mon[32255]: pgmap v18493: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:05.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:04 smithi022 ceph-mon[31360]: pgmap v18493: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:04 smithi022 ceph-mon[40250]: pgmap v18493: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:06.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:06.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:06.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:06 smithi022 ceph-mon[31360]: pgmap v18494: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:06.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:06 smithi022 ceph-mon[40250]: pgmap v18494: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:06.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:06.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:06.861 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:06 smithi122 ceph-mon[32255]: pgmap v18494: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1889132829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:15:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:15:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1889132829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:07.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1889132829' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:08 smithi122 ceph-mon[32255]: pgmap v18495: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:09.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:08 smithi022 ceph-mon[31360]: pgmap v18495: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:08 smithi022 ceph-mon[40250]: pgmap v18495: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:15:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:10] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T04:15:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:10 smithi122 ceph-mon[32255]: pgmap v18496: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:11.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:10 smithi022 ceph-mon[31360]: pgmap v18496: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:11.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:10 smithi022 ceph-mon[40250]: pgmap v18496: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:11.863 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:12.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:12.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:12.232 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:12.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:15:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:15:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:15:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:12 smithi122 ceph-mon[32255]: pgmap v18497: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:12.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1907590259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:13.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:12 smithi022 ceph-mon[31360]: pgmap v18497: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1907590259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:12 smithi022 ceph-mon[40250]: pgmap v18497: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1907590259' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:14.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:15:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:15:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:14 smithi122 ceph-mon[32255]: pgmap v18498: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:15.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:14 smithi022 ceph-mon[31360]: pgmap v18498: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:14 smithi022 ceph-mon[40250]: pgmap v18498: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:16 smithi122 ceph-mon[32255]: pgmap v18499: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:17.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:16 smithi022 ceph-mon[31360]: pgmap v18499: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:17.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:16 smithi022 ceph-mon[40250]: pgmap v18499: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:17.233 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:15:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:15:17.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:17.600 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:17.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:18 smithi122 ceph-mon[32255]: pgmap v18500: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1286888615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:19.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:18 smithi022 ceph-mon[31360]: pgmap v18500: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1286888615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:18 smithi022 ceph-mon[40250]: pgmap v18500: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1286888615' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:15:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:20] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:15:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:20 smithi122 ceph-mon[32255]: pgmap v18501: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:20 smithi022 ceph-mon[31360]: pgmap v18501: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:20 smithi022 ceph-mon[40250]: pgmap v18501: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:15:21.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:15:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:21 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:15:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:21 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:21.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:15:21.980 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:15:21.980 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:15:21.980 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:21.980 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:21.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:15:21.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:15:21.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:15:21.981 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:21.982 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:21 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:22.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:15:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:15:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:15:22.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:22 smithi122 ceph-mon[32255]: pgmap v18502: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:22.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:22.973 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:22.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:22.974 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:22 smithi022 ceph-mon[31360]: pgmap v18502: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:22.974 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:22.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:22 smithi022 ceph-mon[40250]: pgmap v18502: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:22.975 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:15:23.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:15:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:15:23.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1197090350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:24.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1197090350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:24.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1197090350' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:24 smithi122 ceph-mon[32255]: pgmap v18503: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:25.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:24 smithi022 ceph-mon[31360]: pgmap v18503: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:24 smithi022 ceph-mon[40250]: pgmap v18503: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:25.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:26 smithi122 ceph-mon[32255]: pgmap v18504: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:27.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:26 smithi022 ceph-mon[31360]: pgmap v18504: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:26 smithi022 ceph-mon[40250]: pgmap v18504: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:15:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:15:27.975 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:28.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:28.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:28.344 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:28 smithi122 ceph-mon[32255]: pgmap v18505: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4066568214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:28 smithi022 ceph-mon[31360]: pgmap v18505: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4066568214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:28 smithi022 ceph-mon[40250]: pgmap v18505: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:29.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4066568214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:15:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:30] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:15:30.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:30 smithi122 ceph-mon[32255]: pgmap v18506: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:31.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:30 smithi022 ceph-mon[31360]: pgmap v18506: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:30 smithi022 ceph-mon[40250]: pgmap v18506: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:32.611 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:15:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:15:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:15:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:32 smithi122 ceph-mon[32255]: pgmap v18507: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:33.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:32 smithi022 ceph-mon[31360]: pgmap v18507: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:33.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:32 smithi022 ceph-mon[40250]: pgmap v18507: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:33.346 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:33.718 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:33.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:33.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:34.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:15:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:15:34.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:34 smithi122 ceph-mon[32255]: pgmap v18508: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2519744106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:15:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:15:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:15:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:15:35.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[31360]: pgmap v18508: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2519744106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:15:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:15:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:15:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:15:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[40250]: pgmap v18508: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2519744106' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:15:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:15:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:15:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:15:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:36 smithi122 ceph-mon[32255]: pgmap v18509: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:37.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:36 smithi022 ceph-mon[31360]: pgmap v18509: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:36 smithi022 ceph-mon[40250]: pgmap v18509: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:15:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:15:38.721 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:38 smithi122 ceph-mon[32255]: pgmap v18510: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:39.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:38 smithi022 ceph-mon[31360]: pgmap v18510: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:39.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:38 smithi022 ceph-mon[40250]: pgmap v18510: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:39.087 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:39.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:39.088 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:39.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1529316554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:40.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1529316554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:40.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1529316554' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:15:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:40] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:15:40.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:40 smithi122 ceph-mon[32255]: pgmap v18511: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:41.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:40 smithi022 ceph-mon[31360]: pgmap v18511: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:41.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:40 smithi022 ceph-mon[40250]: pgmap v18511: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:42.632 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:15:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:15:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:15:42.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:42 smithi122 ceph-mon[32255]: pgmap v18512: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:43.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:42 smithi022 ceph-mon[31360]: pgmap v18512: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:43.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:42 smithi022 ceph-mon[40250]: pgmap v18512: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:44.089 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:44.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:15:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:15:44.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:44.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:44.457 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:44.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:44 smithi022 ceph-mon[31360]: pgmap v18513: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:44.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/607435278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:44.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:44 smithi022 ceph-mon[40250]: pgmap v18513: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:44.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/607435278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:44 smithi122 ceph-mon[32255]: pgmap v18513: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/607435278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:46.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:46 smithi122 ceph-mon[32255]: pgmap v18514: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:47.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:46 smithi022 ceph-mon[31360]: pgmap v18514: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:46 smithi022 ceph-mon[40250]: pgmap v18514: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:15:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:15:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:48 smithi122 ceph-mon[32255]: pgmap v18515: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:49.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:48 smithi022 ceph-mon[31360]: pgmap v18515: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:49.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:48 smithi022 ceph-mon[40250]: pgmap v18515: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:49.459 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:49.827 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:15:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:50] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:15:50.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:50 smithi122 ceph-mon[32255]: pgmap v18516: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/599739514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:50 smithi022 ceph-mon[40250]: pgmap v18516: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/599739514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:50 smithi022 ceph-mon[31360]: pgmap v18516: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:51.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/599739514' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:52.649 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:15:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:15:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:15:52.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:52 smithi122 ceph-mon[32255]: pgmap v18517: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:53.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:52 smithi022 ceph-mon[31360]: pgmap v18517: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:53.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:52 smithi022 ceph-mon[40250]: pgmap v18517: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:54.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:15:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:15:54.829 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:15:54.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:54 smithi122 ceph-mon[32255]: pgmap v18518: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:55.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:54 smithi022 ceph-mon[31360]: pgmap v18518: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:55.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:54 smithi022 ceph-mon[40250]: pgmap v18518: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:15:55.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:15:55.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:15:55.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:15:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1096003838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1096003838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:15:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1096003838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:15:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:56 smithi122 ceph-mon[32255]: pgmap v18519: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:56 smithi022 ceph-mon[31360]: pgmap v18519: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:56 smithi022 ceph-mon[40250]: pgmap v18519: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:15:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:15:58.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:58 smithi122 ceph-mon[32255]: pgmap v18520: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:58 smithi022 ceph-mon[31360]: pgmap v18520: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:59.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:58 smithi022 ceph-mon[40250]: pgmap v18520: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:15:59.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:15:59 smithi122 ceph-mon[32255]: pgmap v18521: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:00.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:15:59 smithi022 ceph-mon[31360]: pgmap v18521: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:00.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:15:59 smithi022 ceph-mon[40250]: pgmap v18521: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:00.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:16:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:00] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:16:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:00.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1072946099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1072946099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1072946099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:01 smithi122 ceph-mon[32255]: pgmap v18522: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:02.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:01 smithi022 ceph-mon[31360]: pgmap v18522: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:02.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:01 smithi022 ceph-mon[40250]: pgmap v18522: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:02.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:16:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:16:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:16:04.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:16:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:16:04.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:04 smithi122 ceph-mon[32255]: pgmap v18523: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:04 smithi022 ceph-mon[31360]: pgmap v18523: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:05.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:04 smithi022 ceph-mon[40250]: pgmap v18523: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:05.569 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:05.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:05.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:05.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:05.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:05.941 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:05.941 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:06.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:06 smithi122 ceph-mon[32255]: pgmap v18524: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1521503735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:06 smithi022 ceph-mon[31360]: pgmap v18524: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1521503735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:07.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:06 smithi022 ceph-mon[40250]: pgmap v18524: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1521503735' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:16:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:16:08.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:08 smithi122 ceph-mon[32255]: pgmap v18525: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:08 smithi022 ceph-mon[31360]: pgmap v18525: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:09.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:08 smithi022 ceph-mon[40250]: pgmap v18525: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:16:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:10] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:16:10.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:10 smithi122 ceph-mon[32255]: pgmap v18526: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:10.942 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:10.942 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:10 smithi022 ceph-mon[40250]: pgmap v18526: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:10.943 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:10 smithi022 ceph-mon[31360]: pgmap v18526: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:11.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:11.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:11.315 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:11.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/487043943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/487043943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:11.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/487043943' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:12.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:12 smithi122 ceph-mon[32255]: pgmap v18527: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:16:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:16:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:16:13.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:12 smithi022 ceph-mon[31360]: pgmap v18527: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:13.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:12 smithi022 ceph-mon[40250]: pgmap v18527: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:14.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:16:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:16:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:14 smithi122 ceph-mon[32255]: pgmap v18528: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:14 smithi022 ceph-mon[31360]: pgmap v18528: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:15.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:14 smithi022 ceph-mon[40250]: pgmap v18528: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:15.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:16.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:16.317 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:16.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:16.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:16.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:16.688 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:16 smithi022 ceph-mon[31360]: pgmap v18529: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:16.688 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:16 smithi022 ceph-mon[40250]: pgmap v18529: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:16.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:16 smithi122 ceph-mon[32255]: pgmap v18529: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:17.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1251055921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:16:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:16:17.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1251055921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:17.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1251055921' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:18.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:18 smithi122 ceph-mon[32255]: pgmap v18530: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:18 smithi022 ceph-mon[31360]: pgmap v18530: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:19.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:18 smithi022 ceph-mon[40250]: pgmap v18530: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:16:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:20] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:16:20.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:20 smithi122 ceph-mon[32255]: pgmap v18531: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:20 smithi022 ceph-mon[31360]: pgmap v18531: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:21.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:20 smithi022 ceph-mon[40250]: pgmap v18531: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:21.688 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:22.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:22.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:22.059 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:22.573 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:16:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:16:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:16:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[31360]: pgmap v18532: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:16:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:16:22.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:16:22.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1193206190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:22.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:16:22.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[40250]: pgmap v18532: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:22.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:16:22.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:16:22.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:16:22.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1193206190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:22.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:22 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:16:22.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:22 smithi122 ceph-mon[32255]: pgmap v18532: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:16:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:16:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:16:22.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1193206190' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:22.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:22 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:16:24.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:16:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:16:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:24 smithi122 ceph-mon[32255]: pgmap v18533: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:24 smithi022 ceph-mon[31360]: pgmap v18533: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:25.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:24 smithi022 ceph-mon[40250]: pgmap v18533: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:26.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:26.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:26.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:26 smithi122 ceph-mon[32255]: pgmap v18534: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:27.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:26 smithi022 ceph-mon[31360]: pgmap v18534: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:27.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:26 smithi022 ceph-mon[40250]: pgmap v18534: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:27.061 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:27.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:27.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:27.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:27.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3831299591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:16:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:16:27.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3831299591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:27.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3831299591' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:28.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:28 smithi122 ceph-mon[32255]: pgmap v18535: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:28 smithi022 ceph-mon[31360]: pgmap v18535: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:28 smithi022 ceph-mon[40250]: pgmap v18535: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:16:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:30] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:16:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:30 smithi122 ceph-mon[32255]: pgmap v18536: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:30 smithi022 ceph-mon[31360]: pgmap v18536: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:31.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:30 smithi022 ceph-mon[40250]: pgmap v18536: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:32.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:16:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:16:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:16:32.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:32 smithi022 ceph-mon[31360]: pgmap v18537: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:32.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:32 smithi022 ceph-mon[40250]: pgmap v18537: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:32.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:32.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:32.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:32.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:32 smithi122 ceph-mon[32255]: pgmap v18537: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:33.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/55171680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:33.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:16:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:16:34.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/55171680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:34.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/55171680' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:34 smithi122 ceph-mon[32255]: pgmap v18538: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:16:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:16:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:16:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:16:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[31360]: pgmap v18538: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:16:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:16:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:16:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:16:35.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[40250]: pgmap v18538: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:16:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:16:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:16:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:16:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:36.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:36 smithi122 ceph-mon[32255]: pgmap v18539: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:37.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:36 smithi022 ceph-mon[31360]: pgmap v18539: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:37.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:36 smithi022 ceph-mon[40250]: pgmap v18539: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:16:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:16:37.806 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:38.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:38.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:38.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:38.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:38 smithi122 ceph-mon[32255]: pgmap v18540: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3768964023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:38 smithi022 ceph-mon[31360]: pgmap v18540: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3768964023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:38 smithi022 ceph-mon[40250]: pgmap v18540: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3768964023' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:16:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:40] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:16:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:40 smithi122 ceph-mon[32255]: pgmap v18541: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:40 smithi022 ceph-mon[31360]: pgmap v18541: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:41.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:40 smithi022 ceph-mon[40250]: pgmap v18541: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:42.617 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:16:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:16:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:16:42.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:42 smithi122 ceph-mon[32255]: pgmap v18542: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:42 smithi022 ceph-mon[31360]: pgmap v18542: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:43.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:42 smithi022 ceph-mon[40250]: pgmap v18542: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:43.175 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:43.540 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:43.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:43.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:43.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2314219768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:43.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:16:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:16:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2314219768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:44.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2314219768' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:44.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:44 smithi122 ceph-mon[32255]: pgmap v18543: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:44 smithi022 ceph-mon[31360]: pgmap v18543: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:45.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:44 smithi022 ceph-mon[40250]: pgmap v18543: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:45.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:46.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:46 smithi122 ceph-mon[32255]: pgmap v18544: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:46 smithi022 ceph-mon[31360]: pgmap v18544: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:47.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:46 smithi022 ceph-mon[40250]: pgmap v18544: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:16:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:16:48.542 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:48.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:48.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:48.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:48.912 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:48 smithi022 ceph-mon[31360]: pgmap v18545: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:48.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:48 smithi022 ceph-mon[40250]: pgmap v18545: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:48.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:48 smithi122 ceph-mon[32255]: pgmap v18545: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:49.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3492313617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:50.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3492313617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:50.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3492313617' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:16:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:50] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:16:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:50 smithi122 ceph-mon[32255]: pgmap v18546: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:50 smithi022 ceph-mon[31360]: pgmap v18546: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:51.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:50 smithi022 ceph-mon[40250]: pgmap v18546: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:52.639 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:16:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:16:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:16:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:52 smithi122 ceph-mon[32255]: pgmap v18547: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:53.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:52 smithi022 ceph-mon[31360]: pgmap v18547: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:53.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:52 smithi022 ceph-mon[40250]: pgmap v18547: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:53.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:54.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:16:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:16:54.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:54.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:54.281 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:54 smithi122 ceph-mon[32255]: pgmap v18548: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3522715237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:55.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:54 smithi022 ceph-mon[31360]: pgmap v18548: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:55.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3522715237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:55.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:54 smithi022 ceph-mon[40250]: pgmap v18548: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:16:55.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3522715237' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:56.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:16:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:56 smithi122 ceph-mon[32255]: pgmap v18549: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:56 smithi022 ceph-mon[31360]: pgmap v18549: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:56 smithi022 ceph-mon[40250]: pgmap v18549: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:16:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:16:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:58 smithi122 ceph-mon[32255]: pgmap v18550: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:59.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:58 smithi022 ceph-mon[31360]: pgmap v18550: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:59.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:58 smithi022 ceph-mon[40250]: pgmap v18550: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:16:59.283 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:16:59.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:16:59.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:16:59.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:16:59.657 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:16:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/438350127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:59.658 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:16:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/438350127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:16:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:16:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/438350127' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:17:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:00] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:17:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:00 smithi122 ceph-mon[32255]: pgmap v18551: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:01.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:00 smithi022 ceph-mon[31360]: pgmap v18551: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:00 smithi022 ceph-mon[40250]: pgmap v18551: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:02.660 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:17:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:17:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:17:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:02 smithi122 ceph-mon[32255]: pgmap v18552: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:03.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:02 smithi022 ceph-mon[31360]: pgmap v18552: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:03.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:02 smithi022 ceph-mon[40250]: pgmap v18552: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:04.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:17:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:17:04.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:04 smithi122 ceph-mon[32255]: pgmap v18553: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:05.027 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:04 smithi022 ceph-mon[31360]: pgmap v18553: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:05.027 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:04 smithi022 ceph-mon[40250]: pgmap v18553: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/978350357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:06.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/978350357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/978350357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:06 smithi122 ceph-mon[32255]: pgmap v18554: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:07.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:06 smithi022 ceph-mon[31360]: pgmap v18554: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:06 smithi022 ceph-mon[40250]: pgmap v18554: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:17:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:17:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:08 smithi122 ceph-mon[32255]: pgmap v18555: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:09.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:08 smithi022 ceph-mon[31360]: pgmap v18555: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:09.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:08 smithi022 ceph-mon[40250]: pgmap v18555: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:10.028 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:10.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:10.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:10.395 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:17:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:10] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:17:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:10 smithi122 ceph-mon[32255]: pgmap v18556: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/491741125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:10 smithi022 ceph-mon[31360]: pgmap v18556: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:11.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/491741125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:10 smithi022 ceph-mon[40250]: pgmap v18556: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/491741125' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:11.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:11 smithi122 ceph-mon[32255]: pgmap v18557: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:12.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:11 smithi022 ceph-mon[31360]: pgmap v18557: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:11 smithi022 ceph-mon[40250]: pgmap v18557: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:17:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:17:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:17:14.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:17:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:17:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:14 smithi122 ceph-mon[32255]: pgmap v18558: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:14 smithi022 ceph-mon[31360]: pgmap v18558: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:14 smithi022 ceph-mon[40250]: pgmap v18558: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:15.397 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:15.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:15.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:15.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:15.770 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:15.771 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:16 smithi122 ceph-mon[32255]: pgmap v18559: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2517788233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:17.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:16 smithi022 ceph-mon[31360]: pgmap v18559: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:17.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2517788233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:16 smithi022 ceph-mon[40250]: pgmap v18559: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2517788233' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:17:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:17:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:18 smithi122 ceph-mon[32255]: pgmap v18560: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:19.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:18 smithi022 ceph-mon[31360]: pgmap v18560: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:18 smithi022 ceph-mon[40250]: pgmap v18560: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:17:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:20] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:17:20.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:20.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:20 smithi122 ceph-mon[32255]: pgmap v18561: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:21.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:20 smithi022 ceph-mon[31360]: pgmap v18561: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:20 smithi022 ceph-mon[40250]: pgmap v18561: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:21.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:21.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:21.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:21.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1888252516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:22.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1888252516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:22.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1888252516' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:22.568 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:17:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:17:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:17:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[31360]: pgmap v18562: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:22.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:17:22.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:17:22.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:17:22.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[40250]: pgmap v18562: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:22.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:17:22.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:17:22.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:22 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:17:22.865 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:22 smithi122 ceph-mon[32255]: pgmap v18562: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:22.866 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:17:22.866 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:17:22.866 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:22 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:17:24.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:17:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:17:24.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:17:24.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:23 smithi122 ceph-mon[32255]: pgmap v18563: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:17:24.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:23 smithi022 ceph-mon[31360]: pgmap v18563: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:17:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:23 smithi022 ceph-mon[40250]: pgmap v18563: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:25.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:25.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:25.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:26.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:25 smithi122 ceph-mon[32255]: pgmap v18564: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:26.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:25 smithi022 ceph-mon[31360]: pgmap v18564: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:26.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:25 smithi022 ceph-mon[40250]: pgmap v18564: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:26.512 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:26.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:26.513 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:27.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/435394746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:27.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/435394746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:27.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/435394746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:17:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:17:28.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:27 smithi122 ceph-mon[32255]: pgmap v18565: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:28.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:27 smithi022 ceph-mon[31360]: pgmap v18565: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:28.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:27 smithi022 ceph-mon[40250]: pgmap v18565: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:17:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:30] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:17:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:30 smithi122 ceph-mon[32255]: pgmap v18566: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:31.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:30 smithi022 ceph-mon[31360]: pgmap v18566: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:31.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:30 smithi022 ceph-mon[40250]: pgmap v18566: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:31.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:31.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:31.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:31.889 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:32.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:32 smithi122 ceph-mon[32255]: pgmap v18567: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4235699002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:32.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:17:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:17:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:17:33.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:32 smithi022 ceph-mon[31360]: pgmap v18567: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4235699002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:32 smithi022 ceph-mon[40250]: pgmap v18567: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4235699002' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:34.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:17:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:17:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:34 smithi122 ceph-mon[32255]: pgmap v18568: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:34.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:17:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:17:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:17:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:17:35.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[31360]: pgmap v18568: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:17:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:17:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:17:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:17:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[40250]: pgmap v18568: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:17:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:17:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:17:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:17:35.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:36.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:36.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:36.891 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:36.891 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:36 smithi022 ceph-mon[40250]: pgmap v18569: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:36.892 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:36 smithi022 ceph-mon[31360]: pgmap v18569: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:36.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:36 smithi122 ceph-mon[32255]: pgmap v18569: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:37.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:37.263 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:37.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:17:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:17:37.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/308653468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:38.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/308653468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/308653468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:38.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:38 smithi122 ceph-mon[32255]: pgmap v18570: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:39.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:38 smithi022 ceph-mon[31360]: pgmap v18570: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:39.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:38 smithi022 ceph-mon[40250]: pgmap v18570: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:17:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:40] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:17:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:40 smithi122 ceph-mon[32255]: pgmap v18571: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:41.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:40 smithi022 ceph-mon[31360]: pgmap v18571: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:41.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:40 smithi022 ceph-mon[40250]: pgmap v18571: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:42.264 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:42.584 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:17:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:17:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:17:42.632 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:42.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:42.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:42.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:42 smithi122 ceph-mon[32255]: pgmap v18572: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:43.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:42 smithi022 ceph-mon[31360]: pgmap v18572: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:43.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:42 smithi022 ceph-mon[40250]: pgmap v18572: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:43.870 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3628836196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:44.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3628836196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:44.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3628836196' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:44.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:17:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:17:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:44 smithi122 ceph-mon[32255]: pgmap v18573: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:45.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:44 smithi022 ceph-mon[31360]: pgmap v18573: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:45.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:44 smithi022 ceph-mon[40250]: pgmap v18573: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:46.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:46.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:46 smithi122 ceph-mon[32255]: pgmap v18574: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:47.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:46 smithi022 ceph-mon[31360]: pgmap v18574: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:47.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:46 smithi022 ceph-mon[40250]: pgmap v18574: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:47.634 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:17:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:17:48.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:48.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:48.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:48 smithi122 ceph-mon[32255]: pgmap v18575: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:48.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1606146452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:49.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:48 smithi022 ceph-mon[31360]: pgmap v18575: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1606146452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:48 smithi022 ceph-mon[40250]: pgmap v18575: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1606146452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:17:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:50] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:17:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:50 smithi122 ceph-mon[32255]: pgmap v18576: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:51.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:50 smithi022 ceph-mon[31360]: pgmap v18576: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:51.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:50 smithi022 ceph-mon[40250]: pgmap v18576: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:17:52.603 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:17:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:17:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:17:52.604 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:52 smithi122 ceph-mon[32255]: pgmap v18577: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:53.003 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:53.004 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:52 smithi022 ceph-mon[40250]: pgmap v18577: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:53.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:52 smithi022 ceph-mon[31360]: pgmap v18577: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:17:53.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:53.377 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:53.378 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:53.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3220767838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:53.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3220767838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:53.881 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3220767838' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:17:54.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:17:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:17:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:54 smithi122 ceph-mon[32255]: pgmap v18578: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:55.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:54 smithi022 ceph-mon[31360]: pgmap v18578: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:55.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:54 smithi022 ceph-mon[40250]: pgmap v18578: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:56.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:17:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:56 smithi122 ceph-mon[32255]: pgmap v18579: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:56 smithi022 ceph-mon[31360]: pgmap v18579: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:56 smithi022 ceph-mon[40250]: pgmap v18579: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:17:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:17:58.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:17:58.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:17:58.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:17:58.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:17:58.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:58 smithi022 ceph-mon[31360]: pgmap v18580: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:58.754 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:58 smithi022 ceph-mon[40250]: pgmap v18580: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:58 smithi122 ceph-mon[32255]: pgmap v18580: 132 pgs: 132 active+clean; 582 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2021-12-16T04:17:59.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:17:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/107914402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:00.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:17:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/107914402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:00.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:17:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/107914402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:18:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:00] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:18:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:00 smithi122 ceph-mon[32255]: pgmap v18581: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T04:18:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:00 smithi022 ceph-mon[31360]: pgmap v18581: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T04:18:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:00 smithi022 ceph-mon[40250]: pgmap v18581: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T04:18:02.626 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:18:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:18:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:18:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:02 smithi122 ceph-mon[32255]: pgmap v18582: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T04:18:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:02 smithi022 ceph-mon[31360]: pgmap v18582: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T04:18:03.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:02 smithi022 ceph-mon[40250]: pgmap v18582: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-12-16T04:18:03.754 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:04.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:04.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:04.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:04.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:18:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:18:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:04 smithi122 ceph-mon[32255]: pgmap v18583: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T04:18:04.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4162399563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:04 smithi022 ceph-mon[31360]: pgmap v18583: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T04:18:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4162399563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:04 smithi022 ceph-mon[40250]: pgmap v18583: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2021-12-16T04:18:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4162399563' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:05.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:06.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:06 smithi122 ceph-mon[32255]: pgmap v18584: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:06 smithi022 ceph-mon[31360]: pgmap v18584: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:06 smithi022 ceph-mon[40250]: pgmap v18584: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:18:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:18:08.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:08 smithi122 ceph-mon[32255]: pgmap v18585: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:09.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:08 smithi022 ceph-mon[31360]: pgmap v18585: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:09.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:08 smithi022 ceph-mon[40250]: pgmap v18585: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:09.125 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:09.492 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:09.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:09.493 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:09.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1446393526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:09.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1446393526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:09.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1446393526' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:18:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:10] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:18:10.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:10 smithi122 ceph-mon[32255]: pgmap v18586: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:10 smithi022 ceph-mon[31360]: pgmap v18586: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:10 smithi022 ceph-mon[40250]: pgmap v18586: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:18:12.648 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:18:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:18:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:18:12.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:12 smithi122 ceph-mon[32255]: pgmap v18587: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:13.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:12 smithi022 ceph-mon[31360]: pgmap v18587: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:12 smithi022 ceph-mon[40250]: pgmap v18587: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:14.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:18:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:18:14.494 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:14.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:14 smithi022 ceph-mon[31360]: pgmap v18588: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:14.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:14 smithi022 ceph-mon[40250]: pgmap v18588: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:14.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:14.862 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:14.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:14 smithi122 ceph-mon[32255]: pgmap v18588: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:15.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:15.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3901820320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:16.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:16.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3901820320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:16.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3901820320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:16 smithi122 ceph-mon[32255]: pgmap v18589: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:17.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:16 smithi022 ceph-mon[31360]: pgmap v18589: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:16 smithi022 ceph-mon[40250]: pgmap v18589: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:18:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:18:18.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:18 smithi122 ceph-mon[32255]: pgmap v18590: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:19.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:18 smithi022 ceph-mon[31360]: pgmap v18590: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:19.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:18 smithi022 ceph-mon[40250]: pgmap v18590: 132 pgs: 132 active+clean; 582 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:19.864 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:20.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:20.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:20.234 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:18:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:20] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:18:20.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:20 smithi122 ceph-mon[32255]: pgmap v18591: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:20.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/493871533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:21.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:20 smithi022 ceph-mon[31360]: pgmap v18591: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:21.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/493871533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:21.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:20 smithi022 ceph-mon[40250]: pgmap v18591: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:21.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/493871533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:22.663 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:18:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:18:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:18:22.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:22 smithi122 ceph-mon[32255]: pgmap v18592: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:22.938 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:22 smithi022 ceph-mon[31360]: pgmap v18592: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:22.939 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:22 smithi022 ceph-mon[40250]: pgmap v18592: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:23.667 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:18:23.668 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:18:23.668 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:18:23.668 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:18:23.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:18:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:18:23.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:18:23.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:18:23.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:18:23.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:23 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:18:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:18:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:18:24.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:18:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:23 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:18:24.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:24 smithi122 ceph-mon[32255]: pgmap v18593: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:25.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:24 smithi022 ceph-mon[31360]: pgmap v18593: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:25.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:24 smithi022 ceph-mon[40250]: pgmap v18593: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:25.236 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:25.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/570723965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/570723965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:26.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:26.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/570723965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:26.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:26 smithi122 ceph-mon[32255]: pgmap v18594: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:27.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:26 smithi022 ceph-mon[31360]: pgmap v18594: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:27.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:26 smithi022 ceph-mon[40250]: pgmap v18594: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:18:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:18:28.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:28 smithi122 ceph-mon[32255]: pgmap v18595: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:29.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:28 smithi022 ceph-mon[31360]: pgmap v18595: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:29.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:28 smithi022 ceph-mon[40250]: pgmap v18595: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:18:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:18:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:30] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:18:30.606 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:30.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:30 smithi122 ceph-mon[32255]: pgmap v18596: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:30.976 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:30.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:30.977 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:30.978 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:30 smithi022 ceph-mon[31360]: pgmap v18596: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:30.978 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:30 smithi022 ceph-mon[40250]: pgmap v18596: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:18:31.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2312872753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:32.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2312872753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:32.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2312872753' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:18:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:18:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:18:32.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:32 smithi122 ceph-mon[32255]: pgmap v18597: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:32 smithi022 ceph-mon[31360]: pgmap v18597: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:33.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:32 smithi022 ceph-mon[40250]: pgmap v18597: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:34.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:18:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:18:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[31360]: pgmap v18598: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:18:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:18:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:18:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:18:35.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[40250]: pgmap v18598: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:18:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:18:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:18:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:18:35.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:34 smithi122 ceph-mon[32255]: pgmap v18598: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:35.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:18:35.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:18:35.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:18:35.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:18:35.978 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:35.979 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:35.979 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:36.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:36.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:36.348 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:36.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:36 smithi022 ceph-mon[31360]: pgmap v18599: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/641477686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:36 smithi022 ceph-mon[40250]: pgmap v18599: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:37.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/641477686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:37.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:36 smithi122 ceph-mon[32255]: pgmap v18599: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:37.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/641477686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:37.698 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:18:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:18:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:37 smithi022 ceph-mon[31360]: pgmap v18600: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:38.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:37 smithi022 ceph-mon[40250]: pgmap v18600: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:38.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:37 smithi122 ceph-mon[32255]: pgmap v18600: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:18:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:40] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:18:40.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:40 smithi122 ceph-mon[32255]: pgmap v18601: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:41.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:40 smithi022 ceph-mon[31360]: pgmap v18601: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:41.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:40 smithi022 ceph-mon[40250]: pgmap v18601: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:41.350 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:41.716 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:41.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:41.717 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:42.574 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:18:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:18:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:18:42.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:42 smithi122 ceph-mon[32255]: pgmap v18602: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:42.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3436594380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:43.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:42 smithi022 ceph-mon[31360]: pgmap v18602: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:43.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3436594380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:43.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:42 smithi022 ceph-mon[40250]: pgmap v18602: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:43.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3436594380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:44.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:18:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:18:44.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:44 smithi122 ceph-mon[32255]: pgmap v18603: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:45.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:44 smithi022 ceph-mon[31360]: pgmap v18603: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:45.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:44 smithi022 ceph-mon[40250]: pgmap v18603: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:46.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:46.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:46.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:46 smithi122 ceph-mon[32255]: pgmap v18604: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:47.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:46 smithi022 ceph-mon[31360]: pgmap v18604: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:47.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:46 smithi022 ceph-mon[40250]: pgmap v18604: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:47.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:47.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:47.092 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:47.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3254924885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:47.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:18:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:18:47.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3254924885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3254924885' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:48 smithi122 ceph-mon[32255]: pgmap v18605: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:48 smithi022 ceph-mon[31360]: pgmap v18605: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:48 smithi022 ceph-mon[40250]: pgmap v18605: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:18:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:50] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:18:50.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:50 smithi122 ceph-mon[32255]: pgmap v18606: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:51.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:50 smithi022 ceph-mon[31360]: pgmap v18606: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:51.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:50 smithi022 ceph-mon[40250]: pgmap v18606: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:52.090 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:52.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:52.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:52.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:52.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:18:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:18:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:18:52.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:52 smithi022 ceph-mon[31360]: pgmap v18607: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:52.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1430689191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:52.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:52 smithi022 ceph-mon[40250]: pgmap v18607: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:52.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1430689191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:52.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:52 smithi122 ceph-mon[32255]: pgmap v18607: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:52.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1430689191' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:54.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:18:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:18:54.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:54 smithi122 ceph-mon[32255]: pgmap v18608: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:55.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:54 smithi022 ceph-mon[31360]: pgmap v18608: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:55.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:54 smithi022 ceph-mon[40250]: pgmap v18608: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:18:55.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:56.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:56.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:18:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:56 smithi122 ceph-mon[32255]: pgmap v18609: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:57.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:56 smithi022 ceph-mon[31360]: pgmap v18609: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:56 smithi022 ceph-mon[40250]: pgmap v18609: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:57.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:18:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:18:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:18:57.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:18:57.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:18:57.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:18:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:58 smithi122 ceph-mon[32255]: pgmap v18610: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:58.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:18:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1452064257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:59.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:58 smithi022 ceph-mon[31360]: pgmap v18610: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:59.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:18:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1452064257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:18:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:58 smithi022 ceph-mon[40250]: pgmap v18610: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:18:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:18:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1452064257' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:19:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:00] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:19:00.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:00 smithi122 ceph-mon[32255]: pgmap v18611: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:00 smithi022 ceph-mon[31360]: pgmap v18611: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:01.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:00 smithi022 ceph-mon[40250]: pgmap v18611: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:02.611 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:19:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:19:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:19:02.831 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:02 smithi122 ceph-mon[32255]: pgmap v18612: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:02 smithi022 ceph-mon[31360]: pgmap v18612: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:03.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:02 smithi022 ceph-mon[40250]: pgmap v18612: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:03.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:03.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:03.200 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:03.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1103219702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:04.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1103219702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1103219702' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:04.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:19:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:19:04.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:04 smithi122 ceph-mon[32255]: pgmap v18613: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:04 smithi022 ceph-mon[31360]: pgmap v18613: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:04 smithi022 ceph-mon[40250]: pgmap v18613: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:06.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:06.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:06 smithi122 ceph-mon[32255]: pgmap v18614: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:06 smithi022 ceph-mon[31360]: pgmap v18614: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:07.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:06 smithi022 ceph-mon[40250]: pgmap v18614: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:19:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:19:08.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:08.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:08.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:08.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:08.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:08 smithi122 ceph-mon[32255]: pgmap v18615: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:08.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1159607949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:09.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:08 smithi022 ceph-mon[31360]: pgmap v18615: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:09.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1159607949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:08 smithi022 ceph-mon[40250]: pgmap v18615: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1159607949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:10.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:19:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:10] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:19:10.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:10 smithi122 ceph-mon[32255]: pgmap v18616: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:10 smithi022 ceph-mon[31360]: pgmap v18616: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:11.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:10 smithi022 ceph-mon[40250]: pgmap v18616: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:12.631 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:19:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:19:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:19:12.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:12 smithi122 ceph-mon[32255]: pgmap v18617: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:13.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:12 smithi022 ceph-mon[31360]: pgmap v18617: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:13.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:12 smithi022 ceph-mon[40250]: pgmap v18617: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:13.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:14.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:19:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:19:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:14 smithi122 ceph-mon[32255]: pgmap v18618: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2449924250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:15.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:14 smithi022 ceph-mon[31360]: pgmap v18618: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:15.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2449924250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:15.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:14 smithi022 ceph-mon[40250]: pgmap v18618: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:15.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2449924250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:15.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:16.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:16.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:16 smithi122 ceph-mon[32255]: pgmap v18619: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:17.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:16 smithi022 ceph-mon[31360]: pgmap v18619: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:17.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:16 smithi022 ceph-mon[40250]: pgmap v18619: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:19:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:19:18.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:18 smithi122 ceph-mon[32255]: pgmap v18620: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:18.949 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:18 smithi022 ceph-mon[40250]: pgmap v18620: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:18.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:18 smithi022 ceph-mon[31360]: pgmap v18620: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:18.950 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:19.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:19.323 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:19.324 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:19.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1929609180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:20.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1929609180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:20.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1929609180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:19:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:20] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:19:20.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:20 smithi122 ceph-mon[32255]: pgmap v18621: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:21.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:20 smithi022 ceph-mon[31360]: pgmap v18621: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:21.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:20 smithi022 ceph-mon[40250]: pgmap v18621: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:22.652 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:19:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:19:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:19:22.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:22 smithi122 ceph-mon[32255]: pgmap v18622: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:23.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:22 smithi022 ceph-mon[31360]: pgmap v18622: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:23.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:22 smithi022 ceph-mon[40250]: pgmap v18622: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:23.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:19:23.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:19:23.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:23 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:19:24.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:19:24.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:19:24.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:23 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:19:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:19:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:19:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:23 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:19:24.325 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:24.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:19:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:19:24.659 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:24 smithi022 ceph-mon[31360]: pgmap v18623: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:24.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:24.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:24.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:24 smithi122 ceph-mon[32255]: pgmap v18623: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:25.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:24 smithi022 ceph-mon[40250]: pgmap v18623: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:25.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1731806052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1731806052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:26.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:26.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1731806052' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:26.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:26 smithi122 ceph-mon[32255]: pgmap v18624: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:27.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:26 smithi022 ceph-mon[31360]: pgmap v18624: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:27.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:26 smithi022 ceph-mon[40250]: pgmap v18624: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:27.531 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:19:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:19:28.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:28.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:27 smithi122 ceph-mon[32255]: pgmap v18625: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:28.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:27 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:28.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:28.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:27 smithi022 ceph-mon[31360]: pgmap v18625: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:28.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:27 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:28.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:28.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:27 smithi022 ceph-mon[40250]: pgmap v18625: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:28.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:27 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:29.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:29.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:19:29.706 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:30.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:30.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:30.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:19:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:30] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:19:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:30 smithi122 ceph-mon[32255]: pgmap v18626: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:30.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3841686415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:30.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:30 smithi022 ceph-mon[31360]: pgmap v18626: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:30.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3841686415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:30.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:30 smithi022 ceph-mon[40250]: pgmap v18626: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:30.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3841686415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:32.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:19:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:19:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:19:32.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:32 smithi122 ceph-mon[32255]: pgmap v18627: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:32 smithi022 ceph-mon[31360]: pgmap v18627: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:32 smithi022 ceph-mon[40250]: pgmap v18627: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:34.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:19:33 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:19:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:34 smithi122 ceph-mon[32255]: pgmap v18628: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:19:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:19:34.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:19:34.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:19:35.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[31360]: pgmap v18628: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:19:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:19:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:19:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:19:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[40250]: pgmap v18628: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:19:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:19:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:19:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:19:35.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:35.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:35.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:35.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:35.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2076211297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:35.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2076211297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:35.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:35.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2076211297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:36.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:36 smithi122 ceph-mon[32255]: pgmap v18629: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:36 smithi022 ceph-mon[31360]: pgmap v18629: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:37.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:36 smithi022 ceph-mon[40250]: pgmap v18629: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:19:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:19:38.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:38 smithi122 ceph-mon[32255]: pgmap v18630: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:39.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:38 smithi022 ceph-mon[31360]: pgmap v18630: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:39.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:38 smithi022 ceph-mon[40250]: pgmap v18630: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:19:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:40] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:19:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:40.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:40 smithi022 ceph-mon[31360]: pgmap v18631: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:40.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:40 smithi022 ceph-mon[40250]: pgmap v18631: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:40.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:40.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:40.819 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:40.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:40 smithi122 ceph-mon[32255]: pgmap v18631: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:41.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/523671478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:42.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/523671478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:42.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/523671478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:42.600 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:19:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:19:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:19:42.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:42 smithi122 ceph-mon[32255]: pgmap v18632: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:43.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:42 smithi022 ceph-mon[31360]: pgmap v18632: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:43.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:42 smithi022 ceph-mon[40250]: pgmap v18632: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:44.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:19:43 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:19:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:44 smithi122 ceph-mon[32255]: pgmap v18633: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:45.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:44 smithi022 ceph-mon[31360]: pgmap v18633: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:45.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:44 smithi022 ceph-mon[40250]: pgmap v18633: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:45.820 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:45.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:46.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:46.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:46.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:46 smithi122 ceph-mon[32255]: pgmap v18634: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:46.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2305519567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:47.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:46 smithi022 ceph-mon[31360]: pgmap v18634: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:47.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2305519567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:47.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:46 smithi022 ceph-mon[40250]: pgmap v18634: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:47.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2305519567' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:19:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:19:48.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:48 smithi122 ceph-mon[32255]: pgmap v18635: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:48 smithi022 ceph-mon[31360]: pgmap v18635: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:48 smithi022 ceph-mon[40250]: pgmap v18635: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:19:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:50] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:19:50.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:50 smithi122 ceph-mon[32255]: pgmap v18636: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:51.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:50 smithi022 ceph-mon[31360]: pgmap v18636: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:51.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:50 smithi022 ceph-mon[40250]: pgmap v18636: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:51.190 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:51.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:51.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:51.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:51.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3819481636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:52.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3819481636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:52.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3819481636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:52.620 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:19:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:19:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:19:52.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:52 smithi122 ceph-mon[32255]: pgmap v18637: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:52 smithi022 ceph-mon[31360]: pgmap v18637: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:53.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:52 smithi022 ceph-mon[40250]: pgmap v18637: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:54.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:19:53 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:19:54.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:54 smithi122 ceph-mon[32255]: pgmap v18638: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:55.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:54 smithi022 ceph-mon[31360]: pgmap v18638: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:55.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:54 smithi022 ceph-mon[40250]: pgmap v18638: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:19:55.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:56.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:19:56.560 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:19:56.929 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:19:56.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:19:56.930 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:19:56.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:56 smithi022 ceph-mon[31360]: pgmap v18639: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:56.931 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:56 smithi022 ceph-mon[40250]: pgmap v18639: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:56.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:56 smithi122 ceph-mon[32255]: pgmap v18639: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:57.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1919574579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:19:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:19:57.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1919574579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:57.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1919574579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:19:58.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:19:58 smithi122 ceph-mon[32255]: pgmap v18640: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:59.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:19:58 smithi022 ceph-mon[31360]: pgmap v18640: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:19:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:19:58 smithi022 ceph-mon[40250]: pgmap v18640: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:20:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:00] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:20:00.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:00 smithi122 ceph-mon[32255]: pgmap v18641: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:00.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T04:20:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:00 smithi022 ceph-mon[31360]: pgmap v18641: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:01.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T04:20:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:00 smithi022 ceph-mon[40250]: pgmap v18641: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:01.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T04:20:01.931 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:02.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:02.300 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:02.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:02.641 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:20:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:20:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:20:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:02 smithi122 ceph-mon[32255]: pgmap v18642: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1290095749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:02 smithi022 ceph-mon[31360]: pgmap v18642: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1290095749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:03.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:02 smithi022 ceph-mon[40250]: pgmap v18642: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:03.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1290095749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:04.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:20:03 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:20:04.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:04 smithi122 ceph-mon[32255]: pgmap v18643: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:04 smithi022 ceph-mon[31360]: pgmap v18643: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:04 smithi022 ceph-mon[40250]: pgmap v18643: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:06.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:06.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:06 smithi122 ceph-mon[32255]: pgmap v18644: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:06 smithi022 ceph-mon[31360]: pgmap v18644: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:06 smithi022 ceph-mon[40250]: pgmap v18644: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:07.302 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:07.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:07.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:07.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:07.675 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:20:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:20:08.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:08 smithi122 ceph-mon[32255]: pgmap v18645: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:08.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1811798807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:09.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:08 smithi022 ceph-mon[31360]: pgmap v18645: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:09.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1811798807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:08 smithi022 ceph-mon[40250]: pgmap v18645: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1811798807' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:10.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:20:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:10] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:20:10.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:10 smithi122 ceph-mon[32255]: pgmap v18646: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:10 smithi022 ceph-mon[31360]: pgmap v18646: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:11.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:10 smithi022 ceph-mon[40250]: pgmap v18646: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:12.658 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:20:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:20:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:20:12.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:12.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:12 smithi122 ceph-mon[32255]: pgmap v18647: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:13.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:12 smithi022 ceph-mon[31360]: pgmap v18647: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:13.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:12 smithi022 ceph-mon[40250]: pgmap v18647: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:13.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:13.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:13.043 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:13.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2379640908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:14.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2379640908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:14.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2379640908' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:14.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:20:13 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:20:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:14 smithi122 ceph-mon[32255]: pgmap v18648: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:15.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:14 smithi022 ceph-mon[31360]: pgmap v18648: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:15.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:14 smithi022 ceph-mon[40250]: pgmap v18648: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:15.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:16.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:16.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:16 smithi122 ceph-mon[32255]: pgmap v18649: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:17.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:16 smithi022 ceph-mon[31360]: pgmap v18649: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:17.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:16 smithi022 ceph-mon[40250]: pgmap v18649: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:20:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:20:18.046 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:18.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:18.415 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:18.416 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:18.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:18 smithi122 ceph-mon[32255]: pgmap v18650: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:18.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3417544383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:19.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:18 smithi022 ceph-mon[31360]: pgmap v18650: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:19.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3417544383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:19.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:18 smithi022 ceph-mon[40250]: pgmap v18650: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:19.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3417544383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:20:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:20] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:20:20.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:20 smithi122 ceph-mon[32255]: pgmap v18651: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:21.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:20 smithi022 ceph-mon[31360]: pgmap v18651: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:21.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:20 smithi022 ceph-mon[40250]: pgmap v18651: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:22.681 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:20:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:20:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:20:22.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:22 smithi122 ceph-mon[32255]: pgmap v18652: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:23.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:22 smithi022 ceph-mon[31360]: pgmap v18652: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:23.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:22 smithi022 ceph-mon[40250]: pgmap v18652: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:23.418 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:23.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:23.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:23.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:24.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:20:23 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:20:24.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:24 smithi122 ceph-mon[32255]: pgmap v18653: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:24.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/21428170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:25.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:24 smithi022 ceph-mon[31360]: pgmap v18653: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:25.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/21428170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:25.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:24 smithi022 ceph-mon[40250]: pgmap v18653: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:25.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/21428170' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:26.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:26.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:27.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:26 smithi022 ceph-mon[31360]: pgmap v18654: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:27.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:26 smithi022 ceph-mon[40250]: pgmap v18654: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:27.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:26 smithi122 ceph-mon[32255]: pgmap v18654: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:20:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:20:28.789 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:28.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:28 smithi122 ceph-mon[32255]: pgmap v18655: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:28.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:20:28.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:20:28.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:20:28.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:28 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:20:29.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[31360]: pgmap v18655: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:29.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:20:29.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:20:29.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:20:29.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:20:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[40250]: pgmap v18655: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:20:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:20:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:20:29.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:28 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:20:29.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:29.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:29.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:30.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4031944874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:30.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4031944874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:30.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4031944874' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:20:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:30] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:20:31.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:30 smithi022 ceph-mon[31360]: pgmap v18656: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:31.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:30 smithi022 ceph-mon[40250]: pgmap v18656: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:31.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:30 smithi122 ceph-mon[32255]: pgmap v18656: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:32.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:20:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:20:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:20:33.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:32 smithi022 ceph-mon[31360]: pgmap v18657: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:33.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:32 smithi022 ceph-mon[40250]: pgmap v18657: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:32 smithi122 ceph-mon[32255]: pgmap v18657: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:34.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:34.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:20:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:20:34.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:34.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:34.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[31360]: pgmap v18658: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1892106321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:20:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:20:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:20:34.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:20:34.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[40250]: pgmap v18658: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:34.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1892106321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:20:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:20:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:20:34.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:20:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:34 smithi122 ceph-mon[32255]: pgmap v18658: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1892106321' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:20:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:20:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:20:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:20:36.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:36.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:36.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:37.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:36 smithi022 ceph-mon[31360]: pgmap v18659: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:37.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:36 smithi022 ceph-mon[40250]: pgmap v18659: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:37.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:36 smithi122 ceph-mon[32255]: pgmap v18659: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:37.715 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:20:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:20:37.715 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:37 smithi022 ceph-mon[40250]: pgmap v18660: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:37 smithi022 ceph-mon[31360]: pgmap v18660: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:37 smithi122 ceph-mon[32255]: pgmap v18660: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:39.518 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:39.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:39.887 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:39.888 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:20:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:40] "GET /metrics HTTP/1.1" 200 202490 "" "Prometheus/2.18.1" 2021-12-16T04:20:40.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:40 smithi122 ceph-mon[32255]: pgmap v18661: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:40.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/605130355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:41.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:40 smithi022 ceph-mon[31360]: pgmap v18661: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:41.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/605130355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:41.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:40 smithi022 ceph-mon[40250]: pgmap v18661: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:41.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/605130355' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:42.601 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:20:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:20:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:20:42.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:42 smithi122 ceph-mon[32255]: pgmap v18662: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:43.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:42 smithi022 ceph-mon[31360]: pgmap v18662: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:43.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:42 smithi022 ceph-mon[40250]: pgmap v18662: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:44.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:20:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:20:44.889 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:44.889 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:44 smithi022 ceph-mon[40250]: pgmap v18663: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:44.890 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:44 smithi022 ceph-mon[31360]: pgmap v18663: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:44 smithi122 ceph-mon[32255]: pgmap v18663: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:45.260 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:45.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:45.261 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:45.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:45.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/605751173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:46.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:46.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/605751173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/605751173' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:46.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:46 smithi122 ceph-mon[32255]: pgmap v18664: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:47.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:46 smithi022 ceph-mon[31360]: pgmap v18664: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:47.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:46 smithi022 ceph-mon[40250]: pgmap v18664: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:20:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:20:48.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:48 smithi122 ceph-mon[32255]: pgmap v18665: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:48 smithi022 ceph-mon[31360]: pgmap v18665: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:48 smithi022 ceph-mon[40250]: pgmap v18665: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:20:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:50] "GET /metrics HTTP/1.1" 200 202489 "" "Prometheus/2.18.1" 2021-12-16T04:20:50.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:50.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:50.631 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:50.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:50 smithi122 ceph-mon[32255]: pgmap v18666: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:50 smithi022 ceph-mon[31360]: pgmap v18666: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:50 smithi022 ceph-mon[40250]: pgmap v18666: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:51.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/871788805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:52.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/871788805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:52.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/871788805' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:52.622 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:20:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:20:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:20:52.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:52 smithi122 ceph-mon[32255]: pgmap v18667: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:53.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:52 smithi022 ceph-mon[31360]: pgmap v18667: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:53.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:52 smithi022 ceph-mon[40250]: pgmap v18667: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:54.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:20:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:20:54.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:54 smithi122 ceph-mon[32255]: pgmap v18668: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:55.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:54 smithi022 ceph-mon[31360]: pgmap v18668: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:55.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:54 smithi022 ceph-mon[40250]: pgmap v18668: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:20:55.632 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:20:55.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:56.000 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:56.001 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:20:56.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:20:56.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:20:56.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:20:56.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:56 smithi122 ceph-mon[32255]: pgmap v18669: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:56.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1797295720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:57.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:56 smithi022 ceph-mon[31360]: pgmap v18669: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1797295720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:57.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:56 smithi022 ceph-mon[40250]: pgmap v18669: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:57.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1797295720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:20:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:20:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:20:58.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:20:58 smithi122 ceph-mon[32255]: pgmap v18670: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:59.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:20:58 smithi022 ceph-mon[31360]: pgmap v18670: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:20:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:20:58 smithi022 ceph-mon[40250]: pgmap v18670: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:00.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:21:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:00] "GET /metrics HTTP/1.1" 200 202489 "" "Prometheus/2.18.1" 2021-12-16T04:21:00.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:00 smithi122 ceph-mon[32255]: pgmap v18671: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:01.200 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:01.201 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:00 smithi022 ceph-mon[31360]: pgmap v18671: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:01.201 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:00 smithi022 ceph-mon[40250]: pgmap v18671: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:01.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:01.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:01.374 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2797309965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:02.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2797309965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:02.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2797309965' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:02.646 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:21:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:21:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:21:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:02 smithi122 ceph-mon[32255]: pgmap v18672: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:02 smithi022 ceph-mon[31360]: pgmap v18672: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:03.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:02 smithi022 ceph-mon[40250]: pgmap v18672: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:04.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:21:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:21:04.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:04 smithi122 ceph-mon[32255]: pgmap v18673: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:05.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:04 smithi022 ceph-mon[31360]: pgmap v18673: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:04 smithi022 ceph-mon[40250]: pgmap v18673: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:06.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:06.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:06.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:06.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:06.750 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:06.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:06 smithi122 ceph-mon[32255]: pgmap v18674: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:07.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:06 smithi022 ceph-mon[31360]: pgmap v18674: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:07.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:06 smithi022 ceph-mon[40250]: pgmap v18674: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:07.659 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:21:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:21:07.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1786755493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:08.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1786755493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:08.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1786755493' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:08.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:08 smithi122 ceph-mon[32255]: pgmap v18675: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:09.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:08 smithi022 ceph-mon[31360]: pgmap v18675: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:08 smithi022 ceph-mon[40250]: pgmap v18675: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:10.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:21:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:10] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:21:10.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:10 smithi122 ceph-mon[32255]: pgmap v18676: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:11.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:10 smithi022 ceph-mon[31360]: pgmap v18676: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:11.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:10 smithi022 ceph-mon[40250]: pgmap v18676: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:12.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:12.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:12.123 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:12.666 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:21:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:21:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:21:12.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:12 smithi122 ceph-mon[32255]: pgmap v18677: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:12.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3404396510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:13.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:12 smithi022 ceph-mon[40250]: pgmap v18677: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:13.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3404396510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:12 smithi022 ceph-mon[31360]: pgmap v18677: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3404396510' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:14.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:21:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:21:14.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:14 smithi122 ceph-mon[32255]: pgmap v18678: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:15.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:14 smithi022 ceph-mon[40250]: pgmap v18678: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:15.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:14 smithi022 ceph-mon[31360]: pgmap v18678: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:15.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:16.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:16.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:16.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:16 smithi122 ceph-mon[32255]: pgmap v18679: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:17.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:16 smithi022 ceph-mon[31360]: pgmap v18679: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:17.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:16 smithi022 ceph-mon[40250]: pgmap v18679: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:17.125 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:17.494 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:17.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:17.495 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:17.679 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:21:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:21:17.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/32652952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:18.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/32652952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:18.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/32652952' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:18.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:18 smithi122 ceph-mon[32255]: pgmap v18680: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:19.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:18 smithi022 ceph-mon[31360]: pgmap v18680: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:19.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:18 smithi022 ceph-mon[40250]: pgmap v18680: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:21:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:20] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:21:20.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:20 smithi122 ceph-mon[32255]: pgmap v18681: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:21.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:20 smithi022 ceph-mon[31360]: pgmap v18681: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:21.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:20 smithi022 ceph-mon[40250]: pgmap v18681: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:21:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:21:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:21:22.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:22 smithi022 ceph-mon[31360]: pgmap v18682: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:22.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:22 smithi022 ceph-mon[40250]: pgmap v18682: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:22.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:22.869 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:22.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:22.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:22 smithi122 ceph-mon[32255]: pgmap v18682: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:24.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2596306955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:24.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2596306955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:24.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2596306955' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:24.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:21:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:21:25.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:24 smithi022 ceph-mon[31360]: pgmap v18683: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:25.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:24 smithi022 ceph-mon[40250]: pgmap v18683: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:25.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:24 smithi122 ceph-mon[32255]: pgmap v18683: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:26.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:26.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:26.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:27.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:26 smithi022 ceph-mon[31360]: pgmap v18684: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:27.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:26 smithi022 ceph-mon[40250]: pgmap v18684: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:27.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:26 smithi122 ceph-mon[32255]: pgmap v18684: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:27.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:21:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:21:27.871 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:28.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:28.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:28.241 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:29.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[40250]: pgmap v18685: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:29.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/274481040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:29.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:21:29.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:21:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:21:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[31360]: pgmap v18685: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/274481040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:29.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:21:29.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:21:29.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:28 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:21:29.129 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:28 smithi122 ceph-mon[32255]: pgmap v18685: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:29.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/274481040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:29.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:21:29.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:21:29.130 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:28 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:21:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:21:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:30] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:21:30.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:21:30.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:30 smithi122 ceph-mon[32255]: pgmap v18686: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:30.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:21:30.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:30 smithi022 ceph-mon[31360]: pgmap v18686: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:30.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:21:30.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:30 smithi022 ceph-mon[40250]: pgmap v18686: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:32 smithi122 ceph-mon[32255]: pgmap v18687: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:21:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:21:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:21:32.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:32 smithi022 ceph-mon[31360]: pgmap v18687: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:32.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:32 smithi022 ceph-mon[40250]: pgmap v18687: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:33.243 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:33.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:33.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:33.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:34.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:21:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:21:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:34 smithi022 ceph-mon[31360]: pgmap v18688: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:34.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2198125784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:34 smithi022 ceph-mon[40250]: pgmap v18688: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:34.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2198125784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:34.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:34 smithi122 ceph-mon[32255]: pgmap v18688: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:34.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2198125784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:35.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:35.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:21:35.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:21:35.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:21:35.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:21:35.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:35.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:21:35.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:21:35.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:21:35.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:21:35.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:35.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:21:35.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:21:35.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:21:35.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:21:36.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:36 smithi022 ceph-mon[31360]: pgmap v18689: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:36.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:36 smithi022 ceph-mon[40250]: pgmap v18689: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:36.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:36 smithi122 ceph-mon[32255]: pgmap v18689: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:37.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:21:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:21:38.615 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:38.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:38 smithi022 ceph-mon[31360]: pgmap v18690: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:38.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:38 smithi022 ceph-mon[40250]: pgmap v18690: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:38.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:38 smithi122 ceph-mon[32255]: pgmap v18690: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:38.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:38.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:38.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:39.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4216228318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:40.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4216228318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:40.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4216228318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:21:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:40] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:21:40.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:40 smithi122 ceph-mon[32255]: pgmap v18691: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:41.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:40 smithi022 ceph-mon[31360]: pgmap v18691: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:41.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:40 smithi022 ceph-mon[40250]: pgmap v18691: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:42.622 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:21:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:21:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:21:42.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:42 smithi122 ceph-mon[32255]: pgmap v18692: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:43.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:42 smithi022 ceph-mon[31360]: pgmap v18692: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:43.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:42 smithi022 ceph-mon[40250]: pgmap v18692: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:43.990 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:44.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:44.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:44.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:44.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:21:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:21:44.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:44 smithi122 ceph-mon[32255]: pgmap v18693: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:44.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2625210489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:44 smithi022 ceph-mon[31360]: pgmap v18693: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2625210489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:45.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:44 smithi022 ceph-mon[40250]: pgmap v18693: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:45.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2625210489' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:45.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:46.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:46.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:46.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:46 smithi122 ceph-mon[32255]: pgmap v18694: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:47.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:46 smithi022 ceph-mon[31360]: pgmap v18694: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:47.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:46 smithi022 ceph-mon[40250]: pgmap v18694: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:47.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:21:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:21:48.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:48 smithi122 ceph-mon[32255]: pgmap v18695: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:49.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:48 smithi022 ceph-mon[31360]: pgmap v18695: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:49.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:48 smithi022 ceph-mon[40250]: pgmap v18695: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:49.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:49.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:49.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:49.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:21:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:50] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:21:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:50 smithi122 ceph-mon[32255]: pgmap v18696: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/386240597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:50 smithi022 ceph-mon[31360]: pgmap v18696: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/386240597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:50 smithi022 ceph-mon[40250]: pgmap v18696: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:21:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/386240597' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:52.639 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:21:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:21:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:21:52.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:52 smithi122 ceph-mon[32255]: pgmap v18697: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:53.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:52 smithi022 ceph-mon[31360]: pgmap v18697: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:53.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:52 smithi022 ceph-mon[40250]: pgmap v18697: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:21:54.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:21:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:21:54.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:21:54.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:54 smithi122 ceph-mon[32255]: pgmap v18698: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:21:55.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:54 smithi022 ceph-mon[31360]: pgmap v18698: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:21:55.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:54 smithi022 ceph-mon[40250]: pgmap v18698: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:21:55.096 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:21:55.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:21:55.097 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:21:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2654858239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2654858239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:56.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:21:56.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2654858239' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:21:56.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:56 smithi122 ceph-mon[32255]: pgmap v18699: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:21:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:56 smithi022 ceph-mon[31360]: pgmap v18699: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:21:57.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:56 smithi022 ceph-mon[40250]: pgmap v18699: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:21:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:21:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:21:58.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:21:58 smithi122 ceph-mon[32255]: pgmap v18700: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:21:59.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:21:58 smithi022 ceph-mon[31360]: pgmap v18700: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:21:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:21:58 smithi022 ceph-mon[40250]: pgmap v18700: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:22:00.098 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:22:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:00] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:22:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:00.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:00.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:00.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:00 smithi022 ceph-mon[31360]: pgmap v18701: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:22:00.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/161042432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:00.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:00 smithi022 ceph-mon[40250]: pgmap v18701: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:22:00.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/161042432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:00.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:00 smithi122 ceph-mon[32255]: pgmap v18701: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:22:00.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/161042432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:02.656 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:22:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:22:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:22:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:02 smithi122 ceph-mon[32255]: pgmap v18702: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:22:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:02 smithi022 ceph-mon[31360]: pgmap v18702: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:22:03.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:02 smithi022 ceph-mon[40250]: pgmap v18702: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-12-16T04:22:04.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:22:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:22:04.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:04 smithi122 ceph-mon[32255]: pgmap v18703: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:22:05.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:04 smithi022 ceph-mon[31360]: pgmap v18703: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:22:05.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:04 smithi022 ceph-mon[40250]: pgmap v18703: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-12-16T04:22:05.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:05.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:05.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:05.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:05.844 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:05.845 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:05.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:06.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:06 smithi122 ceph-mon[32255]: pgmap v18704: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:06.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3976957523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:07.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:06 smithi022 ceph-mon[31360]: pgmap v18704: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:07.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3976957523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:06 smithi022 ceph-mon[40250]: pgmap v18704: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3976957523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:07.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:22:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:22:08.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:08 smithi122 ceph-mon[32255]: pgmap v18705: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:09.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:08 smithi022 ceph-mon[31360]: pgmap v18705: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:09.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:08 smithi022 ceph-mon[40250]: pgmap v18705: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:10.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:22:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:10] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:22:10.846 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:10.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:10 smithi122 ceph-mon[32255]: pgmap v18706: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:10 smithi022 ceph-mon[31360]: pgmap v18706: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:11.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:10 smithi022 ceph-mon[40250]: pgmap v18706: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:11.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:11.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:11.216 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:11.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3841353844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:12.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3841353844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:12.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3841353844' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:12.676 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:22:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:22:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:22:12.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:12 smithi122 ceph-mon[32255]: pgmap v18707: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:12 smithi022 ceph-mon[31360]: pgmap v18707: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:13.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:12 smithi022 ceph-mon[40250]: pgmap v18707: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:14.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:22:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:22:14.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:14 smithi122 ceph-mon[32255]: pgmap v18708: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:15.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:14 smithi022 ceph-mon[31360]: pgmap v18708: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:15.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:14 smithi022 ceph-mon[40250]: pgmap v18708: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:15.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:16.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:16.218 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:16.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:16.588 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:16.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:16.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:16 smithi122 ceph-mon[32255]: pgmap v18709: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:16.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3789314872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:17.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:16 smithi022 ceph-mon[31360]: pgmap v18709: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:17.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3789314872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:17.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:16 smithi022 ceph-mon[40250]: pgmap v18709: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:17.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3789314872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:17.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:22:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:22:19.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:18 smithi022 ceph-mon[31360]: pgmap v18710: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:19.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:18 smithi022 ceph-mon[40250]: pgmap v18710: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:19.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:18 smithi122 ceph-mon[32255]: pgmap v18710: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:22:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:20] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:22:21.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:20 smithi022 ceph-mon[31360]: pgmap v18711: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:21.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:20 smithi022 ceph-mon[40250]: pgmap v18711: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:20 smithi122 ceph-mon[32255]: pgmap v18711: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:21.590 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:21.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:21.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:21.962 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:22.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:22:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:22:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:22:23.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:22 smithi022 ceph-mon[31360]: pgmap v18712: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:23.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3937640954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:23.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:22 smithi022 ceph-mon[40250]: pgmap v18712: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:23.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3937640954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:23.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:22 smithi122 ceph-mon[32255]: pgmap v18712: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3937640954' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:24.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:22:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:22:25.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:24 smithi022 ceph-mon[31360]: pgmap v18713: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:25.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:24 smithi022 ceph-mon[40250]: pgmap v18713: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:25.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:24 smithi122 ceph-mon[32255]: pgmap v18713: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:26.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:26.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:26.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:26.963 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:26.964 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:26 smithi022 ceph-mon[31360]: pgmap v18714: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:26.964 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:26 smithi022 ceph-mon[40250]: pgmap v18714: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:27.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:26 smithi122 ceph-mon[32255]: pgmap v18714: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:27.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:27.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:27.337 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:27.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:22:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:22:28.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2423361907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:28.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2423361907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:28.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2423361907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:29.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:28 smithi022 ceph-mon[31360]: pgmap v18715: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:28 smithi022 ceph-mon[40250]: pgmap v18715: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:28 smithi122 ceph-mon[32255]: pgmap v18715: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:29.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:22:29.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:22:29.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:22:29.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:29 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:22:30.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:22:30.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:22:30.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:22:30.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:22:30.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:22:30.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:22:30.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:22:30.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:29 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:22:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:22:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:30] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:22:31.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:30 smithi022 ceph-mon[31360]: pgmap v18716: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:31.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:30 smithi022 ceph-mon[40250]: pgmap v18716: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:31.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:30 smithi122 ceph-mon[32255]: pgmap v18716: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:22:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:22:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:22:32.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:32.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:32.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:32 smithi022 ceph-mon[31360]: pgmap v18717: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/81152452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:33.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:32 smithi022 ceph-mon[40250]: pgmap v18717: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:33.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/81152452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:33.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:32 smithi122 ceph-mon[32255]: pgmap v18717: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/81152452' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:34.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:22:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:22:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[31360]: pgmap v18718: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:22:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:22:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:22:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:22:35.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[40250]: pgmap v18718: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:22:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:22:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:22:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:22:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:34 smithi122 ceph-mon[32255]: pgmap v18718: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:22:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:22:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:22:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:22:36.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:36.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:36.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:37.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:36 smithi022 ceph-mon[31360]: pgmap v18719: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:37.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:36 smithi022 ceph-mon[40250]: pgmap v18719: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:37.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:36 smithi122 ceph-mon[32255]: pgmap v18719: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:37.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:37.708 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:22:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:22:38.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:38.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:38.078 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:39.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:38 smithi022 ceph-mon[40250]: pgmap v18720: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:39.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1564652573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:38 smithi022 ceph-mon[31360]: pgmap v18720: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1564652573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:39.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:38 smithi122 ceph-mon[32255]: pgmap v18720: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:39.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1564652573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:40.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:39 smithi022 ceph-mon[31360]: pgmap v18721: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:40.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:39 smithi022 ceph-mon[40250]: pgmap v18721: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:40.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:39 smithi122 ceph-mon[32255]: pgmap v18721: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:22:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:40] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:22:42.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:42 smithi122 ceph-mon[32255]: pgmap v18722: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:22:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:22:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:22:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:42 smithi022 ceph-mon[31360]: pgmap v18722: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:42.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:42 smithi022 ceph-mon[40250]: pgmap v18722: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:43.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:43.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:43.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:43.454 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:43.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4179371069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:43.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4179371069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:43.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4179371069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:44.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:22:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:22:44.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:44 smithi122 ceph-mon[32255]: pgmap v18723: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:44 smithi022 ceph-mon[31360]: pgmap v18723: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:45.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:44 smithi022 ceph-mon[40250]: pgmap v18723: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:45.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:46.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:46.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:46 smithi122 ceph-mon[32255]: pgmap v18724: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:47.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:46 smithi022 ceph-mon[31360]: pgmap v18724: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:47.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:46 smithi022 ceph-mon[40250]: pgmap v18724: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:47.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:22:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:22:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:48.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:48 smithi022 ceph-mon[31360]: pgmap v18725: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:48.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:48 smithi022 ceph-mon[40250]: pgmap v18725: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:48.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:48.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:48.826 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:48.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:48 smithi122 ceph-mon[32255]: pgmap v18725: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:49.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3404917873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:50.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3404917873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:50.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3404917873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:22:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:50] "GET /metrics HTTP/1.1" 200 202489 "" "Prometheus/2.18.1" 2021-12-16T04:22:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:50 smithi122 ceph-mon[32255]: pgmap v18726: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:50 smithi022 ceph-mon[31360]: pgmap v18726: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:51.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:50 smithi022 ceph-mon[40250]: pgmap v18726: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:52.647 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:22:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:22:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:22:52.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:52 smithi122 ceph-mon[32255]: pgmap v18727: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:53.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:52 smithi022 ceph-mon[31360]: pgmap v18727: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:53.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:52 smithi022 ceph-mon[40250]: pgmap v18727: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:53.828 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:54.198 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:54.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:54.199 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:54.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:22:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:22:54.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:54 smithi122 ceph-mon[32255]: pgmap v18728: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:54.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3539465178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:55.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:54 smithi022 ceph-mon[31360]: pgmap v18728: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:55.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3539465178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:55.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:54 smithi022 ceph-mon[40250]: pgmap v18728: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:22:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3539465178' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:22:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:56.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:22:56.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:56 smithi122 ceph-mon[32255]: pgmap v18729: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:56 smithi022 ceph-mon[31360]: pgmap v18729: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:57.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:56 smithi022 ceph-mon[40250]: pgmap v18729: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:22:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:22:58.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:58 smithi122 ceph-mon[32255]: pgmap v18730: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:59.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:58 smithi022 ceph-mon[31360]: pgmap v18730: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:59.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:58 smithi022 ceph-mon[40250]: pgmap v18730: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:22:59.201 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:22:59.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:22:59.571 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:22:59.572 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:22:59.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:22:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3683939080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:00.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:22:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3683939080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:00.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:22:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3683939080' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:00.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:23:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:00] "GET /metrics HTTP/1.1" 200 202489 "" "Prometheus/2.18.1" 2021-12-16T04:23:00.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:00 smithi122 ceph-mon[32255]: pgmap v18731: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:01.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:00 smithi022 ceph-mon[31360]: pgmap v18731: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:00 smithi022 ceph-mon[40250]: pgmap v18731: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:02.667 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:23:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:23:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:23:02.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:02 smithi122 ceph-mon[32255]: pgmap v18732: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:02 smithi022 ceph-mon[31360]: pgmap v18732: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:02 smithi022 ceph-mon[40250]: pgmap v18732: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:04.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:23:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:23:04.573 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:04.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:04 smithi122 ceph-mon[32255]: pgmap v18733: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:04.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:04.942 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:04.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:04.943 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:04 smithi022 ceph-mon[31360]: pgmap v18733: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:04.944 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:04 smithi022 ceph-mon[40250]: pgmap v18733: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:05.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:05.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3677361698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:06.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:06.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3677361698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:06.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:06.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3677361698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:06.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:06 smithi122 ceph-mon[32255]: pgmap v18734: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:07.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:06 smithi022 ceph-mon[31360]: pgmap v18734: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:06 smithi022 ceph-mon[40250]: pgmap v18734: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:07.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:23:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:23:08.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:08 smithi122 ceph-mon[32255]: pgmap v18735: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:09.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:08 smithi022 ceph-mon[31360]: pgmap v18735: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:09.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:08 smithi022 ceph-mon[40250]: pgmap v18735: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:09.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:10.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:10.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:10.312 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:23:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:10] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:23:10.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:10 smithi122 ceph-mon[32255]: pgmap v18736: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:10.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2894062623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:10 smithi022 ceph-mon[31360]: pgmap v18736: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2894062623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:11.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:10 smithi022 ceph-mon[40250]: pgmap v18736: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:11.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2894062623' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:12.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:23:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:23:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:23:12.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:12 smithi122 ceph-mon[32255]: pgmap v18737: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:13.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:12 smithi022 ceph-mon[31360]: pgmap v18737: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:13.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:12 smithi022 ceph-mon[40250]: pgmap v18737: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:14.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:23:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:23:14.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:14 smithi122 ceph-mon[32255]: pgmap v18738: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:15.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:14 smithi022 ceph-mon[31360]: pgmap v18738: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:15.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:14 smithi022 ceph-mon[40250]: pgmap v18738: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:15.313 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:15.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:15.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:15.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:16.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/366712432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/366712432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:16.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:16.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/366712432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:17.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:16 smithi022 ceph-mon[31360]: pgmap v18739: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:17.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:16 smithi022 ceph-mon[40250]: pgmap v18739: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:17.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:16 smithi122 ceph-mon[32255]: pgmap v18739: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:17.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:23:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:23:19.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:18 smithi022 ceph-mon[31360]: pgmap v18740: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:19.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:18 smithi022 ceph-mon[40250]: pgmap v18740: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:19.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:18 smithi122 ceph-mon[32255]: pgmap v18740: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:23:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:20] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:23:20.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:21.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:20 smithi022 ceph-mon[31360]: pgmap v18741: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:21.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:20 smithi022 ceph-mon[40250]: pgmap v18741: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:21.052 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:21.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:21.053 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:20 smithi122 ceph-mon[32255]: pgmap v18741: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:22.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2020459941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:22.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2020459941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:22.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2020459941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:22.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:23:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:23:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:23:23.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:22 smithi022 ceph-mon[31360]: pgmap v18742: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:23.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:22 smithi022 ceph-mon[40250]: pgmap v18742: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:23.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:22 smithi122 ceph-mon[32255]: pgmap v18742: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:24.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:23:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:23:25.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:24 smithi022 ceph-mon[31360]: pgmap v18743: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:25.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:24 smithi022 ceph-mon[40250]: pgmap v18743: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:25.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:24 smithi122 ceph-mon[32255]: pgmap v18743: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:26.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:26.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:26.054 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:26.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:27.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:26 smithi022 ceph-mon[31360]: pgmap v18744: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:27.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/346806296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:27.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:26 smithi022 ceph-mon[40250]: pgmap v18744: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:27.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/346806296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:27.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:26 smithi122 ceph-mon[32255]: pgmap v18744: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:27.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/346806296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:27.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:23:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:23:29.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:28 smithi022 ceph-mon[31360]: pgmap v18745: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:29.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:28 smithi022 ceph-mon[40250]: pgmap v18745: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:29.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:28 smithi122 ceph-mon[32255]: pgmap v18745: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:29.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:23:29.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:23:29.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:29 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:23:30.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:23:30.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:23:30.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:29 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:23:30.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:23:30.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:23:30.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:29 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:23:30.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:23:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:30] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:23:31.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:30 smithi022 ceph-mon[31360]: pgmap v18746: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:31.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:23:31.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:30 smithi022 ceph-mon[40250]: pgmap v18746: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:31.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:23:31.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:30 smithi122 ceph-mon[32255]: pgmap v18746: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:31.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:23:31.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:31.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:31.799 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:31.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:23:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:23:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:23:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:32 smithi022 ceph-mon[31360]: pgmap v18747: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/146007912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:33.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:32 smithi022 ceph-mon[40250]: pgmap v18747: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:33.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/146007912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:32 smithi122 ceph-mon[32255]: pgmap v18747: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/146007912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:34.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:23:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:23:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[31360]: pgmap v18748: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:23:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:23:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:23:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:23:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[40250]: pgmap v18748: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:23:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:23:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:23:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:23:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:34 smithi122 ceph-mon[32255]: pgmap v18748: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:23:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:23:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:23:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:23:36.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:36.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:36.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:36.801 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:37.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:36 smithi022 ceph-mon[31360]: pgmap v18749: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:37.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:36 smithi022 ceph-mon[40250]: pgmap v18749: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:37.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:37.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:37.169 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:37.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:36 smithi122 ceph-mon[32255]: pgmap v18749: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:37.532 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:23:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:23:38.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1392833664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:38.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1392833664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:38.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1392833664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:39.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:38 smithi022 ceph-mon[31360]: pgmap v18750: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:39.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:38 smithi022 ceph-mon[40250]: pgmap v18750: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:39.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:38 smithi122 ceph-mon[32255]: pgmap v18750: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:40.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:39 smithi022 ceph-mon[31360]: pgmap v18751: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:40.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:39 smithi022 ceph-mon[40250]: pgmap v18751: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:40.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:39 smithi122 ceph-mon[32255]: pgmap v18751: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:23:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:40] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:23:42.170 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:42.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:42.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:42.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:42.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:42 smithi122 ceph-mon[32255]: pgmap v18752: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:23:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:23:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:23:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:42 smithi022 ceph-mon[31360]: pgmap v18752: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:42.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:42 smithi022 ceph-mon[40250]: pgmap v18752: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:43.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1756184674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:43.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1756184674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:43.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1756184674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:44.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:23:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:23:44.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:44 smithi122 ceph-mon[32255]: pgmap v18753: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:44 smithi022 ceph-mon[31360]: pgmap v18753: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:45.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:44 smithi022 ceph-mon[40250]: pgmap v18753: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:45.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:46.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:46 smithi122 ceph-mon[32255]: pgmap v18754: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:47.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:46 smithi022 ceph-mon[31360]: pgmap v18754: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:47.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:46 smithi022 ceph-mon[40250]: pgmap v18754: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:47.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:47.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:23:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:23:47.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:47.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:47.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:48.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:48 smithi122 ceph-mon[32255]: pgmap v18755: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:48.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2942326850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:49.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:48 smithi022 ceph-mon[31360]: pgmap v18755: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:49.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2942326850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:49.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:48 smithi022 ceph-mon[40250]: pgmap v18755: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:49.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2942326850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:23:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:50] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:23:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:50 smithi122 ceph-mon[32255]: pgmap v18756: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:51.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:50 smithi022 ceph-mon[31360]: pgmap v18756: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:50 smithi022 ceph-mon[40250]: pgmap v18756: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:52.664 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:23:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:23:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:23:52.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:52.911 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:52 smithi022 ceph-mon[31360]: pgmap v18757: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:52.912 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:52 smithi022 ceph-mon[40250]: pgmap v18757: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:52.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:52 smithi122 ceph-mon[32255]: pgmap v18757: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:23:53.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:53.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:53.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:53.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3591225251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:54.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3591225251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:54.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3591225251' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:54.439 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:23:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:23:54.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:54 smithi122 ceph-mon[32255]: pgmap v18758: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:55.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:54 smithi022 ceph-mon[31360]: pgmap v18758: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:55.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:54 smithi022 ceph-mon[40250]: pgmap v18758: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:23:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:56.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:23:56.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:56 smithi122 ceph-mon[32255]: pgmap v18759: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:56 smithi022 ceph-mon[31360]: pgmap v18759: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:57.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:56 smithi022 ceph-mon[40250]: pgmap v18759: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:23:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:23:58.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:23:58.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:23:58.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:23:58.657 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:23:58.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:58 smithi122 ceph-mon[32255]: pgmap v18760: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:58.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:23:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2653265376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:59.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:58 smithi022 ceph-mon[31360]: pgmap v18760: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:59.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:23:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2653265376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:23:59.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:58 smithi022 ceph-mon[40250]: pgmap v18760: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:23:59.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:23:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2653265376' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:00.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:24:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:00] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:24:00.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:00 smithi122 ceph-mon[32255]: pgmap v18761: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:01.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:00 smithi022 ceph-mon[31360]: pgmap v18761: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:01.446 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:00 smithi022 ceph-mon[40250]: pgmap v18761: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:02.681 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:24:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:24:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:24:02.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:02 smithi122 ceph-mon[32255]: pgmap v18762: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:02 smithi022 ceph-mon[31360]: pgmap v18762: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:02 smithi022 ceph-mon[40250]: pgmap v18762: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:03.658 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:04.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:04.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:04.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:04.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:24:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:24:04.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:04 smithi122 ceph-mon[32255]: pgmap v18763: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:04.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1628679840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:05.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:04 smithi022 ceph-mon[31360]: pgmap v18763: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:05.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1628679840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:05.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:04 smithi022 ceph-mon[40250]: pgmap v18763: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:05.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1628679840' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:05.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:06.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:07.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:06 smithi022 ceph-mon[31360]: pgmap v18764: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:06 smithi022 ceph-mon[40250]: pgmap v18764: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:07.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:06 smithi122 ceph-mon[32255]: pgmap v18764: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:07.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:24:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:24:09.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:09.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:08 smithi022 ceph-mon[31360]: pgmap v18765: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:09.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:08 smithi022 ceph-mon[40250]: pgmap v18765: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:09.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:08 smithi122 ceph-mon[32255]: pgmap v18765: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:09.405 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:09.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:09.406 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:10.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2384342533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:10.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2384342533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:10.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2384342533' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:24:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:10] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:24:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:10 smithi022 ceph-mon[31360]: pgmap v18766: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:11.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:10 smithi022 ceph-mon[40250]: pgmap v18766: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:11.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:10 smithi122 ceph-mon[32255]: pgmap v18766: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:24:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:24:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:24:13.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:12 smithi022 ceph-mon[31360]: pgmap v18767: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:13.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:12 smithi022 ceph-mon[40250]: pgmap v18767: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:13.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:12 smithi122 ceph-mon[32255]: pgmap v18767: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:14.407 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:14.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:24:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:24:14.781 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:14.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:14.782 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:15.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:14 smithi022 ceph-mon[31360]: pgmap v18768: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:15.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:14 smithi022 ceph-mon[40250]: pgmap v18768: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:15.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:14 smithi122 ceph-mon[32255]: pgmap v18768: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:16.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2412502162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2412502162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:16.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:16.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2412502162' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:17.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:16 smithi022 ceph-mon[31360]: pgmap v18769: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:17.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:16 smithi022 ceph-mon[40250]: pgmap v18769: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:17.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:16 smithi122 ceph-mon[32255]: pgmap v18769: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:17.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:24:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:24:19.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:18 smithi022 ceph-mon[31360]: pgmap v18770: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:19.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:18 smithi022 ceph-mon[40250]: pgmap v18770: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:18 smithi122 ceph-mon[32255]: pgmap v18770: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:19.783 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:20.154 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:20.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:20.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:24:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:20] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:24:21.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:20 smithi022 ceph-mon[31360]: pgmap v18771: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:21.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3997142957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:21.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:20 smithi022 ceph-mon[40250]: pgmap v18771: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:21.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3997142957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:20 smithi122 ceph-mon[32255]: pgmap v18771: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3997142957' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:22.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:24:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:24:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:24:23.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:22 smithi022 ceph-mon[31360]: pgmap v18772: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:23.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:22 smithi022 ceph-mon[40250]: pgmap v18772: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:23.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:22 smithi122 ceph-mon[32255]: pgmap v18772: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:24.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:24:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:24:25.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:24 smithi022 ceph-mon[31360]: pgmap v18773: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:25.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:24 smithi022 ceph-mon[40250]: pgmap v18773: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:25.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:25.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:24 smithi122 ceph-mon[32255]: pgmap v18773: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:25.524 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:25.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:25.525 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:25.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:25.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1152316814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:25.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:25.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1152316814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:26.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:26.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1152316814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:27.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:26 smithi022 ceph-mon[31360]: pgmap v18774: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:27.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:26 smithi022 ceph-mon[40250]: pgmap v18774: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:27.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:26 smithi122 ceph-mon[32255]: pgmap v18774: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:27.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:24:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:24:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:28 smithi022 ceph-mon[31360]: pgmap v18775: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:28 smithi022 ceph-mon[40250]: pgmap v18775: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:29.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:28 smithi122 ceph-mon[32255]: pgmap v18775: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:30.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:24:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:30] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:24:30.526 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:30.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[40250]: pgmap v18776: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:30.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:24:30.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:24:30.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:24:30.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:24:30.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[31360]: pgmap v18776: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:30.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:24:30.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:24:30.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:24:30.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:30 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:24:30.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:30.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:30.885 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:30.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:30 smithi122 ceph-mon[32255]: pgmap v18776: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:30.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:24:30.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:24:30.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:24:30.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:30 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:24:32.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2935031270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:32.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2935031270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:32.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2935031270' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:24:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:24:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:24:33.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:32 smithi022 ceph-mon[31360]: pgmap v18777: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:33.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:32 smithi022 ceph-mon[40250]: pgmap v18777: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:32 smithi122 ceph-mon[32255]: pgmap v18777: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:34.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:24:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:24:35.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[31360]: pgmap v18778: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:24:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:24:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:24:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:24:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[40250]: pgmap v18778: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:24:35.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:24:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:24:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:24:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:34 smithi122 ceph-mon[32255]: pgmap v18778: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:24:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:24:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:24:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:24:35.886 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:36.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:36.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:36.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:36.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:36.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:36.258 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:37.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:36 smithi022 ceph-mon[31360]: pgmap v18779: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:37.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2499595441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:37.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:36 smithi022 ceph-mon[40250]: pgmap v18779: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:37.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2499595441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:37.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:36 smithi122 ceph-mon[32255]: pgmap v18779: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:37.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2499595441' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:37.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:24:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:24:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:38 smithi022 ceph-mon[31360]: pgmap v18780: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:39.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:38 smithi022 ceph-mon[40250]: pgmap v18780: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:39.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:38 smithi122 ceph-mon[32255]: pgmap v18780: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:40.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:39 smithi122 ceph-mon[32255]: pgmap v18781: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:40.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:39 smithi022 ceph-mon[31360]: pgmap v18781: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:40.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:39 smithi022 ceph-mon[40250]: pgmap v18781: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:24:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:40] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:24:41.260 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:41.633 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:41.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:41.634 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:42.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:42 smithi122 ceph-mon[32255]: pgmap v18782: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:42.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2093841573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:24:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:24:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:24:42.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:42 smithi022 ceph-mon[31360]: pgmap v18782: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:42.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2093841573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:42.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:42 smithi022 ceph-mon[40250]: pgmap v18782: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:42.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2093841573' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:44.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:24:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:24:44.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:44 smithi122 ceph-mon[32255]: pgmap v18783: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:45.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:44 smithi022 ceph-mon[31360]: pgmap v18783: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:45.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:44 smithi022 ceph-mon[40250]: pgmap v18783: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:45.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:46.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:46.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:46.635 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:46.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:46 smithi122 ceph-mon[32255]: pgmap v18784: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:47.001 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:47.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:47.002 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:47.003 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:46 smithi022 ceph-mon[40250]: pgmap v18784: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:47.003 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:46 smithi022 ceph-mon[31360]: pgmap v18784: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:47.666 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:24:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:24:47.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3102611297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:48.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3102611297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:48.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3102611297' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:48.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:48 smithi122 ceph-mon[32255]: pgmap v18785: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:49.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:48 smithi022 ceph-mon[31360]: pgmap v18785: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:49.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:48 smithi022 ceph-mon[40250]: pgmap v18785: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:24:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:50] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:24:50.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:50 smithi122 ceph-mon[32255]: pgmap v18786: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:50 smithi022 ceph-mon[31360]: pgmap v18786: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:50 smithi022 ceph-mon[40250]: pgmap v18786: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:52.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:52.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:52.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:52.673 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:24:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:24:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:24:52.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:52 smithi122 ceph-mon[32255]: pgmap v18787: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:52.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/981057160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:53.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:52 smithi022 ceph-mon[31360]: pgmap v18787: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/981057160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:53.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:52 smithi022 ceph-mon[40250]: pgmap v18787: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:53.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/981057160' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:54.676 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:24:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:24:54.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:54 smithi122 ceph-mon[32255]: pgmap v18788: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:55.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:54 smithi022 ceph-mon[31360]: pgmap v18788: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:54 smithi022 ceph-mon[40250]: pgmap v18788: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:24:55.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:56.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:56.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:24:56.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:56 smithi122 ceph-mon[32255]: pgmap v18789: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:57.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:56 smithi022 ceph-mon[31360]: pgmap v18789: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:57.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:56 smithi022 ceph-mon[40250]: pgmap v18789: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:57.374 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:24:57.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:24:57.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:24:57.748 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:24:57.749 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:24:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:24:58.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:58 smithi122 ceph-mon[32255]: pgmap v18790: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:58.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:24:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1318821634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:59.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:58 smithi022 ceph-mon[31360]: pgmap v18790: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:59.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:24:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1318821634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:24:59.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:58 smithi022 ceph-mon[40250]: pgmap v18790: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:24:59.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:24:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1318821634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:00.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:25:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:00] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:25:00.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:00 smithi122 ceph-mon[32255]: pgmap v18791: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:01.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:00 smithi022 ceph-mon[31360]: pgmap v18791: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:00 smithi022 ceph-mon[40250]: pgmap v18791: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:02.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:25:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:25:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:25:02.750 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:02 smithi022 ceph-mon[31360]: pgmap v18792: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:02 smithi022 ceph-mon[40250]: pgmap v18792: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:03.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:03.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:03.119 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:03.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:02 smithi122 ceph-mon[32255]: pgmap v18792: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:04.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/784037451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:04.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/784037451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:04.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/784037451' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:04.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:25:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:25:05.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:04 smithi022 ceph-mon[31360]: pgmap v18793: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:05.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:04 smithi022 ceph-mon[40250]: pgmap v18793: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:05.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:04 smithi122 ceph-mon[32255]: pgmap v18793: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:06.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:06.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:07.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:06 smithi022 ceph-mon[31360]: pgmap v18794: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:06 smithi022 ceph-mon[40250]: pgmap v18794: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:07.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:06 smithi122 ceph-mon[32255]: pgmap v18794: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:07.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:25:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:25:08.121 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:08.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:08.490 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:08.491 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:08.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:08 smithi022 ceph-mon[31360]: pgmap v18795: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:08.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2286675370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:08.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:08 smithi022 ceph-mon[40250]: pgmap v18795: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:08.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2286675370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:09.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:08 smithi122 ceph-mon[32255]: pgmap v18795: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:09.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2286675370' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:25:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:10] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:25:11.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:10 smithi022 ceph-mon[31360]: pgmap v18796: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:11.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:10 smithi022 ceph-mon[40250]: pgmap v18796: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:11.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:10 smithi122 ceph-mon[32255]: pgmap v18796: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:25:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:25:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:25:13.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:12 smithi022 ceph-mon[31360]: pgmap v18797: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:13.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:12 smithi022 ceph-mon[40250]: pgmap v18797: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:13.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:12 smithi122 ceph-mon[32255]: pgmap v18797: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:13.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:13.863 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:13.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:13.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:14.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:25:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:25:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:14 smithi022 ceph-mon[31360]: pgmap v18798: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1500316152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:15.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:14 smithi022 ceph-mon[40250]: pgmap v18798: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:15.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1500316152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:15.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:14 smithi122 ceph-mon[32255]: pgmap v18798: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:15.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1500316152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:16.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:16.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:17.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:16 smithi022 ceph-mon[31360]: pgmap v18799: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:17.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:16 smithi022 ceph-mon[40250]: pgmap v18799: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:17.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:16 smithi122 ceph-mon[32255]: pgmap v18799: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:17.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:25:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:25:18.865 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:19.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:18 smithi022 ceph-mon[31360]: pgmap v18800: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:19.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:18 smithi022 ceph-mon[40250]: pgmap v18800: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:19.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:18 smithi122 ceph-mon[32255]: pgmap v18800: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:19.237 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:19.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:19.238 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:20.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3445075744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:20.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3445075744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:20.176 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3445075744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:25:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:20] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:25:21.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:20 smithi022 ceph-mon[31360]: pgmap v18801: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:21.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:20 smithi022 ceph-mon[40250]: pgmap v18801: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:20 smithi122 ceph-mon[32255]: pgmap v18801: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:22.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:25:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:25:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:25:23.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:22 smithi022 ceph-mon[31360]: pgmap v18802: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:23.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:22 smithi022 ceph-mon[40250]: pgmap v18802: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:23.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:22 smithi122 ceph-mon[32255]: pgmap v18802: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:24.239 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:24.611 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:24.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:24.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:24.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:25:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:25:25.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:24 smithi022 ceph-mon[31360]: pgmap v18803: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:25.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2007367372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:25.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:24 smithi022 ceph-mon[40250]: pgmap v18803: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:25.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2007367372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:25.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:24 smithi122 ceph-mon[32255]: pgmap v18803: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:25.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2007367372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:26.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:26.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:26.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:27.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:26 smithi022 ceph-mon[31360]: pgmap v18804: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:27.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:26 smithi022 ceph-mon[40250]: pgmap v18804: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:27.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:26 smithi122 ceph-mon[32255]: pgmap v18804: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:27.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:25:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:25:29.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:28 smithi022 ceph-mon[31360]: pgmap v18805: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:29.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:28 smithi022 ceph-mon[40250]: pgmap v18805: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:29.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:28 smithi122 ceph-mon[32255]: pgmap v18805: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:29.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:29.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:29.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:29.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:30.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:25:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:30] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:25:30.930 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:30 smithi122 ceph-mon[32255]: pgmap v18806: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:30.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1499543902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:30.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:25:30.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:25:30.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:30 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:25:30.948 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[40250]: pgmap v18806: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:30.948 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1499543902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:30.949 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:25:30.949 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:25:30.949 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:25:30.949 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[31360]: pgmap v18806: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:30.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1499543902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:30.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:25:30.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:25:30.950 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:30 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:25:32.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:31 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:31 smithi122 ceph-mon[32255]: pgmap v18807: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:32.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[31360]: pgmap v18807: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:25:32.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:31 smithi022 ceph-mon[40250]: pgmap v18807: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:25:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:25:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:25:34.668 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:25:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:25:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:34 smithi122 ceph-mon[32255]: pgmap v18808: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:25:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:25:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:25:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:25:34.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[31360]: pgmap v18808: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:34.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:25:34.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:25:34.990 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:25:34.990 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:25:34.990 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[40250]: pgmap v18808: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:34.991 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:25:34.991 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:25:34.991 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:25:34.991 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:25:34.992 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:35.355 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:35.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:35.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:35.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:35.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/118926149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:36.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:36.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/118926149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:36.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:36.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/118926149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:36.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:36 smithi122 ceph-mon[32255]: pgmap v18809: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:37.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:36 smithi022 ceph-mon[31360]: pgmap v18809: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:37.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:36 smithi022 ceph-mon[40250]: pgmap v18809: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:37.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:25:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:25:38.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:38 smithi122 ceph-mon[32255]: pgmap v18810: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:38 smithi022 ceph-mon[31360]: pgmap v18810: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:39.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:38 smithi022 ceph-mon[40250]: pgmap v18810: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:40.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:25:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:40] "GET /metrics HTTP/1.1" 200 202493 "" "Prometheus/2.18.1" 2021-12-16T04:25:40.679 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:40 smithi022 ceph-mon[31360]: pgmap v18811: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:40.680 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:40 smithi022 ceph-mon[40250]: pgmap v18811: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:40.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:40.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:40.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:40.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:40 smithi122 ceph-mon[32255]: pgmap v18811: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:41.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2066625745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:42.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2066625745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:42.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2066625745' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:42.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:25:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:25:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:25:42.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:42 smithi122 ceph-mon[32255]: pgmap v18812: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:43.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:42 smithi022 ceph-mon[31360]: pgmap v18812: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:43.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:42 smithi022 ceph-mon[40250]: pgmap v18812: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:44.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:25:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:25:44.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:44 smithi122 ceph-mon[32255]: pgmap v18813: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:44 smithi022 ceph-mon[31360]: pgmap v18813: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:45.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:44 smithi022 ceph-mon[40250]: pgmap v18813: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:45.731 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:46.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:46.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:46.102 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:46.103 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:46.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:47.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:46 smithi022 ceph-mon[31360]: pgmap v18814: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:47.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1022098664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:47.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:46 smithi022 ceph-mon[40250]: pgmap v18814: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:47.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1022098664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:47.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:46 smithi122 ceph-mon[32255]: pgmap v18814: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:47.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1022098664' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:25:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:25:49.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:48 smithi022 ceph-mon[31360]: pgmap v18815: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:49.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:48 smithi022 ceph-mon[40250]: pgmap v18815: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:49.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:48 smithi122 ceph-mon[32255]: pgmap v18815: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:25:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:50] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T04:25:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:50 smithi022 ceph-mon[31360]: pgmap v18816: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:50 smithi022 ceph-mon[40250]: pgmap v18816: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:51.104 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:51.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:50 smithi122 ceph-mon[32255]: pgmap v18816: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:51.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:51.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:51.472 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:51.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3919694608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:51.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3919694608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:52.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3919694608' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:52.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:25:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:25:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:25:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:52 smithi022 ceph-mon[31360]: pgmap v18817: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:53.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:52 smithi022 ceph-mon[40250]: pgmap v18817: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:53.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:52 smithi122 ceph-mon[32255]: pgmap v18817: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:54.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:25:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:25:55.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:54 smithi022 ceph-mon[31360]: pgmap v18818: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:55.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:54 smithi022 ceph-mon[40250]: pgmap v18818: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:55.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:54 smithi122 ceph-mon[32255]: pgmap v18818: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:25:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:56.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:56.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:25:56.473 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:25:56.782 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:56 smithi022 ceph-mon[31360]: pgmap v18819: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:56.783 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:56 smithi022 ceph-mon[40250]: pgmap v18819: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:56.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:25:56.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:25:56.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:25:57.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:56 smithi122 ceph-mon[32255]: pgmap v18819: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:57.719 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:25:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:25:58.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4053586206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:58.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4053586206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:58.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4053586206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:25:59.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:25:58 smithi022 ceph-mon[31360]: pgmap v18820: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:59.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:25:58 smithi022 ceph-mon[40250]: pgmap v18820: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:25:59.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:25:58 smithi122 ceph-mon[32255]: pgmap v18820: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:00.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:26:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:00] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T04:26:01.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:00 smithi022 ceph-mon[31360]: pgmap v18821: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:00 smithi022 ceph-mon[40250]: pgmap v18821: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:01.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:00 smithi122 ceph-mon[32255]: pgmap v18821: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:01.849 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:02.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:02.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:02.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:02.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:26:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:26:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:26:03.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:02 smithi022 ceph-mon[31360]: pgmap v18822: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:03.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/410392578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:02 smithi022 ceph-mon[40250]: pgmap v18822: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/410392578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:03.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:02 smithi122 ceph-mon[32255]: pgmap v18822: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:03.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/410392578' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:04.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:26:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:26:05.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:04 smithi022 ceph-mon[31360]: pgmap v18823: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:05.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:04 smithi022 ceph-mon[40250]: pgmap v18823: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:05.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:04 smithi122 ceph-mon[32255]: pgmap v18823: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:06.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:06.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:07.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:06 smithi022 ceph-mon[31360]: pgmap v18824: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:07.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:06 smithi022 ceph-mon[40250]: pgmap v18824: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:07.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:06 smithi122 ceph-mon[32255]: pgmap v18824: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:07.221 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:07.532 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:26:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:26:07.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:07.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:07.591 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:08.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/390591903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:08.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/390591903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:08.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/390591903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:09.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:08 smithi022 ceph-mon[31360]: pgmap v18825: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:09.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:08 smithi022 ceph-mon[40250]: pgmap v18825: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:09.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:08 smithi122 ceph-mon[32255]: pgmap v18825: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:26:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:10] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:26:11.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:10 smithi022 ceph-mon[31360]: pgmap v18826: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:11.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:10 smithi022 ceph-mon[40250]: pgmap v18826: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:11.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:10 smithi122 ceph-mon[32255]: pgmap v18826: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:12.593 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:26:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:26:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:26:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:12.967 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:12.967 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:12 smithi022 ceph-mon[31360]: pgmap v18827: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:12.967 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:12 smithi022 ceph-mon[40250]: pgmap v18827: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:13.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:12 smithi122 ceph-mon[32255]: pgmap v18827: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:14.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2161304672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:14.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2161304672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:14.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2161304672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:14.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:26:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:26:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:14 smithi022 ceph-mon[31360]: pgmap v18828: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:15.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:14 smithi022 ceph-mon[40250]: pgmap v18828: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:15.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:14 smithi122 ceph-mon[32255]: pgmap v18828: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:16.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:17.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:16 smithi022 ceph-mon[31360]: pgmap v18829: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:17.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:16 smithi022 ceph-mon[40250]: pgmap v18829: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:17.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:16 smithi122 ceph-mon[32255]: pgmap v18829: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:17.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:26:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:26:17.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:18.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:18.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:18.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:19.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:18 smithi022 ceph-mon[31360]: pgmap v18830: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:19.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/819628902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:19.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:18 smithi022 ceph-mon[40250]: pgmap v18830: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:19.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/819628902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:19.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:18 smithi122 ceph-mon[32255]: pgmap v18830: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/819628902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:26:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:20] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:26:21.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:20 smithi022 ceph-mon[31360]: pgmap v18831: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:21.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:20 smithi022 ceph-mon[40250]: pgmap v18831: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:20 smithi122 ceph-mon[32255]: pgmap v18831: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:22.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:21 smithi122 ceph-mon[32255]: pgmap v18832: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:22.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:21 smithi022 ceph-mon[31360]: pgmap v18832: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:22.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:21 smithi022 ceph-mon[40250]: pgmap v18832: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:22.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:26:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:26:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:26:23.342 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:23.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:23.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:23.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:24.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:26:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:26:24.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:24 smithi122 ceph-mon[32255]: pgmap v18833: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:24.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4148030262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:25.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:24 smithi022 ceph-mon[31360]: pgmap v18833: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:25.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4148030262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:25.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:24 smithi022 ceph-mon[40250]: pgmap v18833: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:25.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4148030262' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:25.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:26.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:26.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:26.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:26 smithi122 ceph-mon[32255]: pgmap v18834: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:27.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:26 smithi022 ceph-mon[31360]: pgmap v18834: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:27.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:26 smithi022 ceph-mon[40250]: pgmap v18834: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:27.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:26:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:26:28.714 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:28.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:28 smithi122 ceph-mon[32255]: pgmap v18835: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:29.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:28 smithi022 ceph-mon[40250]: pgmap v18835: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:28 smithi022 ceph-mon[31360]: pgmap v18835: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:29.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:29.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:29.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:30.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2628375587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:30.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2628375587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:30.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2628375587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:30.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:26:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:30] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:26:31.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:30 smithi022 ceph-mon[31360]: pgmap v18836: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:31.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:30 smithi022 ceph-mon[40250]: pgmap v18836: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:31.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:30 smithi122 ceph-mon[32255]: pgmap v18836: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:32.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:26:32.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:26:32.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:31 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:26:32.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:26:32.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:26:32.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:31 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:26:32.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:26:32.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:26:32.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:31 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:26:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:26:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:26:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:26:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:32 smithi022 ceph-mon[31360]: pgmap v18837: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:26:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:32 smithi022 ceph-mon[40250]: pgmap v18837: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:26:33.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:32 smithi122 ceph-mon[32255]: pgmap v18837: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:26:34.082 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:34.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:34.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:34.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:34.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:26:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:26:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[31360]: pgmap v18838: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/285673587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:26:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:26:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:26:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:26:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[40250]: pgmap v18838: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/285673587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:26:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:26:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:26:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:26:35.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:34 smithi122 ceph-mon[32255]: pgmap v18838: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/285673587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:26:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:26:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:26:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:26:36.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:36.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:36.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:37.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:36 smithi022 ceph-mon[31360]: pgmap v18839: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:37.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:36 smithi022 ceph-mon[40250]: pgmap v18839: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:36 smithi122 ceph-mon[32255]: pgmap v18839: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:37.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:26:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:26:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:38 smithi022 ceph-mon[31360]: pgmap v18840: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:39.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:38 smithi022 ceph-mon[40250]: pgmap v18840: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:39.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:38 smithi122 ceph-mon[32255]: pgmap v18840: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:39.454 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:39.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:39.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:39.824 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:26:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:40] "GET /metrics HTTP/1.1" 200 202492 "" "Prometheus/2.18.1" 2021-12-16T04:26:41.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:40 smithi022 ceph-mon[31360]: pgmap v18841: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:41.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/220986887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:41.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:40 smithi022 ceph-mon[40250]: pgmap v18841: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:41.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/220986887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:41.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:40 smithi122 ceph-mon[32255]: pgmap v18841: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:41.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/220986887' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:26:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:26:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:26:43.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:42 smithi022 ceph-mon[31360]: pgmap v18842: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:43.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:42 smithi022 ceph-mon[40250]: pgmap v18842: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:43.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:42 smithi122 ceph-mon[32255]: pgmap v18842: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:44.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:26:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:26:44.826 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:44 smithi022 ceph-mon[31360]: pgmap v18843: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:45.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:44 smithi022 ceph-mon[40250]: pgmap v18843: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:45.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:44 smithi122 ceph-mon[32255]: pgmap v18843: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:45.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:45.196 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:45.197 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3071589327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:46.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:46.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3071589327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:46.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3071589327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:47.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:46 smithi022 ceph-mon[31360]: pgmap v18844: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:47.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:46 smithi022 ceph-mon[40250]: pgmap v18844: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:47.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:46 smithi122 ceph-mon[32255]: pgmap v18844: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:26:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:26:49.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:48 smithi022 ceph-mon[31360]: pgmap v18845: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:49.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:48 smithi022 ceph-mon[40250]: pgmap v18845: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:49.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:48 smithi122 ceph-mon[32255]: pgmap v18845: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:50.198 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:26:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:50] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:26:50.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:50.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:50.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:50 smithi022 ceph-mon[31360]: pgmap v18846: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3941792272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:50 smithi022 ceph-mon[40250]: pgmap v18846: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3941792272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:51.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:50 smithi122 ceph-mon[32255]: pgmap v18846: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3941792272' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:26:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:26:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:26:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:52 smithi022 ceph-mon[31360]: pgmap v18847: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:53.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:52 smithi022 ceph-mon[40250]: pgmap v18847: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:52 smithi122 ceph-mon[32255]: pgmap v18847: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:54.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:26:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:26:55.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:54 smithi022 ceph-mon[31360]: pgmap v18848: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:55.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:54 smithi022 ceph-mon[40250]: pgmap v18848: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:55.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:54 smithi122 ceph-mon[32255]: pgmap v18848: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:26:55.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:26:55.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:26:55.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:26:55.943 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:26:55.944 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:55.944 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:56.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:26:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:56 smithi022 ceph-mon[31360]: pgmap v18849: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1702004795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:57.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:56 smithi022 ceph-mon[40250]: pgmap v18849: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:57.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1702004795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:56 smithi122 ceph-mon[32255]: pgmap v18849: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1702004795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:26:57.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:26:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:26:59.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:26:58 smithi022 ceph-mon[31360]: pgmap v18850: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:59.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:26:58 smithi022 ceph-mon[40250]: pgmap v18850: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:26:59.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:26:58 smithi122 ceph-mon[32255]: pgmap v18850: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:27:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:00] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:27:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:01.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:00 smithi022 ceph-mon[31360]: pgmap v18851: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:01.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:00 smithi022 ceph-mon[40250]: pgmap v18851: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:01.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:00 smithi122 ceph-mon[32255]: pgmap v18851: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:01.321 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:01.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:01.322 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:02.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2157006682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:02.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2157006682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:02.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2157006682' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:27:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:27:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:27:03.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:02 smithi022 ceph-mon[31360]: pgmap v18852: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:03.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:02 smithi022 ceph-mon[40250]: pgmap v18852: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:03.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:02 smithi122 ceph-mon[32255]: pgmap v18852: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:04.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:27:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:27:05.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:04 smithi022 ceph-mon[31360]: pgmap v18853: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:05.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:04 smithi022 ceph-mon[40250]: pgmap v18853: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:05.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:04 smithi122 ceph-mon[32255]: pgmap v18853: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:06.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:06.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:06.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:06.323 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:06.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:06.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:06.690 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:07.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:06 smithi022 ceph-mon[31360]: pgmap v18854: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:07.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2271114044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:07.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:06 smithi022 ceph-mon[40250]: pgmap v18854: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:07.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2271114044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:06 smithi122 ceph-mon[32255]: pgmap v18854: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2271114044' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:27:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:27:09.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:08 smithi022 ceph-mon[31360]: pgmap v18855: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:09.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:08 smithi022 ceph-mon[40250]: pgmap v18855: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:08 smithi122 ceph-mon[32255]: pgmap v18855: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:10.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:27:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:10] "GET /metrics HTTP/1.1" 200 202494 "" "Prometheus/2.18.1" 2021-12-16T04:27:11.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:10 smithi022 ceph-mon[31360]: pgmap v18856: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:11.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:10 smithi022 ceph-mon[40250]: pgmap v18856: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:11.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:10 smithi122 ceph-mon[32255]: pgmap v18856: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:11.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:12.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:11 smithi022 ceph-mon[31360]: pgmap v18857: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:12.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:11 smithi022 ceph-mon[40250]: pgmap v18857: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:12.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:12.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:12.069 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:12.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:11 smithi122 ceph-mon[32255]: pgmap v18857: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:27:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:27:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:27:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/583598225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:13.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/583598225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:13.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/583598225' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:14.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:27:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:27:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:14 smithi022 ceph-mon[31360]: pgmap v18858: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:15.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:14 smithi022 ceph-mon[40250]: pgmap v18858: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:15.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:14 smithi122 ceph-mon[32255]: pgmap v18858: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:16.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:16.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:17.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:16 smithi022 ceph-mon[31360]: pgmap v18859: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:17.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:16 smithi022 ceph-mon[40250]: pgmap v18859: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:17.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:16 smithi122 ceph-mon[32255]: pgmap v18859: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:17.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:17.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:17.441 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:17.701 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:27:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:27:18.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1626841465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:18.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1626841465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:18.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1626841465' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:19.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:18 smithi022 ceph-mon[31360]: pgmap v18860: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:19.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:18 smithi022 ceph-mon[40250]: pgmap v18860: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:19.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:18 smithi122 ceph-mon[32255]: pgmap v18860: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:27:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:20] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:27:21.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:20 smithi022 ceph-mon[31360]: pgmap v18861: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:21.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:20 smithi022 ceph-mon[40250]: pgmap v18861: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:21.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:20 smithi122 ceph-mon[32255]: pgmap v18861: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:22.442 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:22.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:27:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:27:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:27:22.708 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:22 smithi022 ceph-mon[31360]: pgmap v18862: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:22.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:22.815 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:22.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:23.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:22 smithi022 ceph-mon[40250]: pgmap v18862: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:22 smithi122 ceph-mon[32255]: pgmap v18862: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:24.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/246057662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:24.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/246057662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:24.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/246057662' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:24.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:27:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:27:25.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:24 smithi022 ceph-mon[31360]: pgmap v18863: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:25.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:24 smithi022 ceph-mon[40250]: pgmap v18863: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:25.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:24 smithi122 ceph-mon[32255]: pgmap v18863: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:26.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:26.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:27.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:26 smithi022 ceph-mon[31360]: pgmap v18864: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:27.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:26 smithi022 ceph-mon[40250]: pgmap v18864: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:27.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:26 smithi122 ceph-mon[32255]: pgmap v18864: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:27.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:27:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:27:27.817 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:28.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:28.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:28.190 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:28 smithi022 ceph-mon[31360]: pgmap v18865: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2897009522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:29.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:28 smithi022 ceph-mon[40250]: pgmap v18865: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:29.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2897009522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:29.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:28 smithi122 ceph-mon[32255]: pgmap v18865: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:29.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2897009522' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:27:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:30] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:27:31.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:30 smithi022 ceph-mon[31360]: pgmap v18866: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:31.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:30 smithi022 ceph-mon[40250]: pgmap v18866: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:31.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:30 smithi122 ceph-mon[32255]: pgmap v18866: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:27:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:27:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:27:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[31360]: pgmap v18867: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:27:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:27:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:27:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:27:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[40250]: pgmap v18867: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:27:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:27:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:27:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:32 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:27:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:32 smithi122 ceph-mon[32255]: pgmap v18867: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:27:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:27:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:27:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:32 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:27:33.192 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:33.562 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:33.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:33.563 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:34.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1696464810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:34.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1696464810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:34.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1696464810' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:34.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:27:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:27:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[31360]: pgmap v18868: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:27:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:27:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:27:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:27:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[40250]: pgmap v18868: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:27:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:27:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:27:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:27:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:34 smithi122 ceph-mon[32255]: pgmap v18868: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:27:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:27:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:27:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:27:36.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:36.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:36.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:37.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:36 smithi022 ceph-mon[31360]: pgmap v18869: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:37.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:36 smithi022 ceph-mon[40250]: pgmap v18869: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:36 smithi122 ceph-mon[32255]: pgmap v18869: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:37.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:27:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:27:38.564 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:38.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:38.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:38.936 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:38.937 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:38 smithi022 ceph-mon[31360]: pgmap v18870: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:38.937 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:38 smithi022 ceph-mon[40250]: pgmap v18870: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:39.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:38 smithi122 ceph-mon[32255]: pgmap v18870: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:40.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1380517766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:40.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1380517766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:40.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1380517766' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:27:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:40] "GET /metrics HTTP/1.1" 200 202491 "" "Prometheus/2.18.1" 2021-12-16T04:27:41.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:40 smithi022 ceph-mon[31360]: pgmap v18871: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:41.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:40 smithi022 ceph-mon[40250]: pgmap v18871: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:41.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:40 smithi122 ceph-mon[32255]: pgmap v18871: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:27:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:27:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:27:43.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:42 smithi022 ceph-mon[31360]: pgmap v18872: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:43.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:42 smithi022 ceph-mon[40250]: pgmap v18872: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:43.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:42 smithi122 ceph-mon[32255]: pgmap v18872: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:43.938 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:44.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:44.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:44.311 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:44.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:27:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:27:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:44 smithi022 ceph-mon[31360]: pgmap v18873: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1711173383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:45.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:44 smithi022 ceph-mon[40250]: pgmap v18873: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:45.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1711173383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:45.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:44 smithi122 ceph-mon[32255]: pgmap v18873: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:45.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1711173383' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:46.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:47.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:46 smithi022 ceph-mon[31360]: pgmap v18874: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:47.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:46 smithi022 ceph-mon[40250]: pgmap v18874: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:46 smithi122 ceph-mon[32255]: pgmap v18874: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:27:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:27:49.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:48 smithi022 ceph-mon[31360]: pgmap v18875: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:49.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:48 smithi022 ceph-mon[40250]: pgmap v18875: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:49.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:48 smithi122 ceph-mon[32255]: pgmap v18875: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:49.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:49.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:49.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:49.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:50.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/844768478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:50.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/844768478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:50.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/844768478' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:27:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:50] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:27:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:50 smithi022 ceph-mon[31360]: pgmap v18876: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:51.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:50 smithi022 ceph-mon[40250]: pgmap v18876: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:50 smithi122 ceph-mon[32255]: pgmap v18876: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:27:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:27:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:27:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:52 smithi022 ceph-mon[31360]: pgmap v18877: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:53.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:52 smithi022 ceph-mon[40250]: pgmap v18877: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:52 smithi122 ceph-mon[32255]: pgmap v18877: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:54.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:27:54.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:27:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:27:55.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:54 smithi022 ceph-mon[31360]: pgmap v18878: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:54 smithi022 ceph-mon[40250]: pgmap v18878: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:55.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:27:55.063 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:27:55.064 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:27:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:54 smithi122 ceph-mon[32255]: pgmap v18878: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:27:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3000614914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:56.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:56.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3000614914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:27:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3000614914' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:27:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:56 smithi022 ceph-mon[31360]: pgmap v18879: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:57.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:56 smithi022 ceph-mon[40250]: pgmap v18879: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:56 smithi122 ceph-mon[32255]: pgmap v18879: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:27:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:27:58.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:27:57 smithi122 ceph-mon[32255]: pgmap v18880: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:58.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:27:57 smithi022 ceph-mon[31360]: pgmap v18880: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:27:58.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:27:57 smithi022 ceph-mon[40250]: pgmap v18880: 132 pgs: 132 active+clean; 582 KiB data, 180 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:00.065 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:00.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:00.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:00.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:28:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:00] "GET /metrics HTTP/1.1" 200 202495 "" "Prometheus/2.18.1" 2021-12-16T04:28:01.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:00 smithi022 ceph-mon[31360]: pgmap v18881: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:01.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1157161724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:01.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:00 smithi022 ceph-mon[40250]: pgmap v18881: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:01.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1157161724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:00 smithi122 ceph-mon[32255]: pgmap v18881: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1157161724' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:28:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:28:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:28:03.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:02 smithi022 ceph-mon[31360]: pgmap v18882: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:02 smithi022 ceph-mon[40250]: pgmap v18882: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:03.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:02 smithi122 ceph-mon[32255]: pgmap v18882: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:04.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:28:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:28:05.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:04 smithi022 ceph-mon[31360]: pgmap v18883: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:05.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:04 smithi022 ceph-mon[40250]: pgmap v18883: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:05.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:04 smithi122 ceph-mon[32255]: pgmap v18883: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:05.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:05.712 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:05.800 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:05.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:05.801 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:06.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:06.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:07.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:06 smithi022 ceph-mon[31360]: pgmap v18884: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:07.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1410548184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:07.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:06 smithi022 ceph-mon[40250]: pgmap v18884: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:07.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1410548184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:06 smithi122 ceph-mon[32255]: pgmap v18884: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1410548184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:28:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:28:09.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:08 smithi022 ceph-mon[31360]: pgmap v18885: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:09.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:08 smithi022 ceph-mon[40250]: pgmap v18885: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:08 smithi122 ceph-mon[32255]: pgmap v18885: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:10.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:28:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:10] "GET /metrics HTTP/1.1" 200 202501 "" "Prometheus/2.18.1" 2021-12-16T04:28:10.802 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:11.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:10 smithi022 ceph-mon[31360]: pgmap v18886: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:11.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:10 smithi022 ceph-mon[40250]: pgmap v18886: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:11.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:11.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:11.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:11.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:10 smithi122 ceph-mon[32255]: pgmap v18886: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:28:12.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2139312260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:12.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2139312260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:12.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2139312260' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:28:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:28:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:28:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:12 smithi022 ceph-mon[31360]: pgmap v18887: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:13.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:12 smithi022 ceph-mon[40250]: pgmap v18887: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:13.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:12 smithi122 ceph-mon[32255]: pgmap v18887: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:14.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:28:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:28:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:14 smithi022 ceph-mon[31360]: pgmap v18888: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:15.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:14 smithi022 ceph-mon[40250]: pgmap v18888: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:15.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:14 smithi122 ceph-mon[32255]: pgmap v18888: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:16.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:16.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:16.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:16.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:16.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:16.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:17.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:16 smithi022 ceph-mon[40250]: pgmap v18889: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:17.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/481544638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:17.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:16 smithi022 ceph-mon[31360]: pgmap v18889: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:17.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/481544638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:17.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:16 smithi122 ceph-mon[32255]: pgmap v18889: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:17.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/481544638' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:17.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:28:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:28:19.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:18 smithi022 ceph-mon[31360]: pgmap v18890: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:19.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:18 smithi022 ceph-mon[40250]: pgmap v18890: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:18 smithi122 ceph-mon[32255]: pgmap v18890: 132 pgs: 132 active+clean; 582 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:28:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:20] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:28:21.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:20 smithi022 ceph-mon[31360]: pgmap v18891: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:21.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:20 smithi022 ceph-mon[40250]: pgmap v18891: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:21.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:20 smithi122 ceph-mon[32255]: pgmap v18891: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:21.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:21.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:21.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:21.919 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:22.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:28:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:28:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:28:23.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:22 smithi022 ceph-mon[31360]: pgmap v18892: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:23.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3272126211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:23.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:22 smithi022 ceph-mon[40250]: pgmap v18892: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:23.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3272126211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:22 smithi122 ceph-mon[32255]: pgmap v18892: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3272126211' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:24.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:28:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:28:25.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:24 smithi022 ceph-mon[31360]: pgmap v18893: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:25.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:24 smithi022 ceph-mon[40250]: pgmap v18893: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:25.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:24 smithi122 ceph-mon[32255]: pgmap v18893: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:26.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:26.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:26.920 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:27.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:26 smithi022 ceph-mon[31360]: pgmap v18894: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:27.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:26 smithi022 ceph-mon[40250]: pgmap v18894: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:27.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:26 smithi122 ceph-mon[32255]: pgmap v18894: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:27.291 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:27.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:27.292 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:27.752 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:28:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:28:28.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/108657056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:28.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/108657056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:28.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/108657056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:28 smithi022 ceph-mon[31360]: pgmap v18895: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:29.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:28 smithi022 ceph-mon[40250]: pgmap v18895: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:29.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:28 smithi122 ceph-mon[32255]: pgmap v18895: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:28:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:28:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:30] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:28:31.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:30 smithi022 ceph-mon[31360]: pgmap v18896: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:31.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:30 smithi022 ceph-mon[40250]: pgmap v18896: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:31.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:30 smithi122 ceph-mon[32255]: pgmap v18896: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:28:32.293 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:32.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:32.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:32.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:28:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:28:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:28:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[31360]: pgmap v18897: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:28:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:28:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:28:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/668546814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[40250]: pgmap v18897: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:28:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:28:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:28:33.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/668546814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:32 smithi122 ceph-mon[32255]: pgmap v18897: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:28:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:28:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:32 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:28:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/668546814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:34.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:33 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:28:34.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:33 smithi122 ceph-mon[32255]: pgmap v18898: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:33 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:28:34.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:33 smithi022 ceph-mon[31360]: pgmap v18898: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:33 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:28:34.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:33 smithi022 ceph-mon[40250]: pgmap v18898: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:34.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:28:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:28:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:34 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:28:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:28:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:28:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:28:35.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:35.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:28:35.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:28:35.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:28:35.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:28:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:28:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:28:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:28:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:28:36.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:35 smithi122 ceph-mon[32255]: pgmap v18899: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:36.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:35 smithi022 ceph-mon[31360]: pgmap v18899: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:36.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:35 smithi022 ceph-mon[40250]: pgmap v18899: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:37.680 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:28:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:28:37.681 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:38.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:38.049 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:38.050 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:38.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:38 smithi122 ceph-mon[32255]: pgmap v18900: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:38.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1399780980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:38 smithi022 ceph-mon[31360]: pgmap v18900: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1399780980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:39.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:38 smithi022 ceph-mon[40250]: pgmap v18900: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:39.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1399780980' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:28:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:40] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:28:41.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:40 smithi022 ceph-mon[31360]: pgmap v18901: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:41.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:40 smithi022 ceph-mon[40250]: pgmap v18901: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:41.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:40 smithi122 ceph-mon[32255]: pgmap v18901: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:28:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:28:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:28:43.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:42 smithi022 ceph-mon[31360]: pgmap v18902: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:43.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:42 smithi022 ceph-mon[40250]: pgmap v18902: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:43.051 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:43.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:42 smithi122 ceph-mon[32255]: pgmap v18902: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:43.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:43.426 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:44.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1186468995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:44.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1186468995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:44.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1186468995' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:44.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:28:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:28:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:44 smithi022 ceph-mon[31360]: pgmap v18903: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:45.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:44 smithi022 ceph-mon[40250]: pgmap v18903: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:45.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:44 smithi122 ceph-mon[32255]: pgmap v18903: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:46.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:47.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:46 smithi022 ceph-mon[31360]: pgmap v18904: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:47.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:46 smithi022 ceph-mon[40250]: pgmap v18904: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:46 smithi122 ceph-mon[32255]: pgmap v18904: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:28:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:28:48.427 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:48.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:48.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:48.797 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:49.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:48 smithi022 ceph-mon[31360]: pgmap v18905: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:49.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:48 smithi022 ceph-mon[40250]: pgmap v18905: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:49.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:48 smithi122 ceph-mon[32255]: pgmap v18905: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:50.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3141509104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:50.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3141509104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:50.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3141509104' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:28:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:50] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:28:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:50 smithi022 ceph-mon[31360]: pgmap v18906: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:50 smithi022 ceph-mon[40250]: pgmap v18906: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:50 smithi122 ceph-mon[32255]: pgmap v18906: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:28:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:28:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:28:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:52 smithi022 ceph-mon[31360]: pgmap v18907: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:53.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:52 smithi022 ceph-mon[40250]: pgmap v18907: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:52 smithi122 ceph-mon[32255]: pgmap v18907: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:53.798 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:54.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:54.167 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:54.168 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:28:54.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:28:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:28:55.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:54 smithi022 ceph-mon[31360]: pgmap v18908: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:55.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3515053172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:54 smithi022 ceph-mon[40250]: pgmap v18908: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3515053172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:54 smithi122 ceph-mon[32255]: pgmap v18908: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:28:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3515053172' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:28:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:56.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:28:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:56 smithi022 ceph-mon[31360]: pgmap v18909: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:57.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:56 smithi022 ceph-mon[40250]: pgmap v18909: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:56 smithi122 ceph-mon[32255]: pgmap v18909: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:57.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:28:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:28:59.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:58 smithi022 ceph-mon[31360]: pgmap v18910: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:59.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:58 smithi022 ceph-mon[40250]: pgmap v18910: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:59.169 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:28:59.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:58 smithi122 ceph-mon[32255]: pgmap v18910: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:28:59.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:28:59.541 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:28:59.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:00.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:28:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2361715488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:00.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:28:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2361715488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:00.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:28:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2361715488' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:29:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:00] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:29:01.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:00 smithi022 ceph-mon[31360]: pgmap v18911: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:01.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:00 smithi022 ceph-mon[40250]: pgmap v18911: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:00 smithi122 ceph-mon[32255]: pgmap v18911: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:29:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:29:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:29:03.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:02 smithi022 ceph-mon[31360]: pgmap v18912: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:02 smithi022 ceph-mon[40250]: pgmap v18912: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:03.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:02 smithi122 ceph-mon[32255]: pgmap v18912: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:04.543 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:04.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:29:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:29:04.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:04.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:04.913 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:04.914 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:04 smithi022 ceph-mon[40250]: pgmap v18913: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:04.914 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:04 smithi022 ceph-mon[31360]: pgmap v18913: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:05.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:04 smithi122 ceph-mon[32255]: pgmap v18913: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:06.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:06.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1329192253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:06.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:06.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1329192253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:06.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:06.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1329192253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:07.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:06 smithi022 ceph-mon[31360]: pgmap v18914: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:07.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:06 smithi022 ceph-mon[40250]: pgmap v18914: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:06 smithi122 ceph-mon[32255]: pgmap v18914: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:29:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:29:09.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:08 smithi022 ceph-mon[31360]: pgmap v18915: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:09.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:08 smithi022 ceph-mon[40250]: pgmap v18915: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:08 smithi122 ceph-mon[32255]: pgmap v18915: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:10.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:29:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:10] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:29:11.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:10 smithi022 ceph-mon[31360]: pgmap v18916: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:11.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3516971095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:11.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:10 smithi022 ceph-mon[40250]: pgmap v18916: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:11.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3516971095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:11.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:10 smithi122 ceph-mon[32255]: pgmap v18916: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:11.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3516971095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:29:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:29:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:29:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:12 smithi022 ceph-mon[31360]: pgmap v18917: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:13.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:12 smithi022 ceph-mon[40250]: pgmap v18917: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:13.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:12 smithi122 ceph-mon[32255]: pgmap v18917: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:14.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:29:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:29:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:14 smithi022 ceph-mon[31360]: pgmap v18918: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:15.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:14 smithi022 ceph-mon[40250]: pgmap v18918: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:15.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:14 smithi122 ceph-mon[32255]: pgmap v18918: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:15.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:15.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:15.654 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1297325431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:16.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:16.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1297325431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:16.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:16.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1297325431' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:17.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:16 smithi022 ceph-mon[31360]: pgmap v18919: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:17.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:16 smithi022 ceph-mon[40250]: pgmap v18919: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:17.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:16 smithi122 ceph-mon[32255]: pgmap v18919: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:29:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:29:19.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:18 smithi022 ceph-mon[31360]: pgmap v18920: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:19.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:18 smithi022 ceph-mon[40250]: pgmap v18920: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:18 smithi122 ceph-mon[32255]: pgmap v18920: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:29:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:20] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:29:20.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:21.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:21.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:21.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:21.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:20 smithi022 ceph-mon[31360]: pgmap v18921: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:21.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:20 smithi022 ceph-mon[40250]: pgmap v18921: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:21.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:20 smithi122 ceph-mon[32255]: pgmap v18921: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:22.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/828338348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:22.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/828338348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:22.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/828338348' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:22.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:29:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:29:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:29:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:22 smithi122 ceph-mon[32255]: pgmap v18922: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:23.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:22 smithi022 ceph-mon[31360]: pgmap v18922: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:23.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:22 smithi022 ceph-mon[40250]: pgmap v18922: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:24.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:29:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:29:25.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:24 smithi122 ceph-mon[32255]: pgmap v18923: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:25.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:24 smithi022 ceph-mon[31360]: pgmap v18923: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:25.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:24 smithi022 ceph-mon[40250]: pgmap v18923: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:26.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:26.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:26.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:26.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:26.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:27.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:26 smithi122 ceph-mon[32255]: pgmap v18924: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:27.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/156607181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:27.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:26 smithi022 ceph-mon[31360]: pgmap v18924: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/156607181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:27.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:26 smithi022 ceph-mon[40250]: pgmap v18924: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/156607181' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:27.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:29:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:29:28.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:27 smithi122 ceph-mon[32255]: pgmap v18925: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:28.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:27 smithi022 ceph-mon[31360]: pgmap v18925: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:28.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:27 smithi022 ceph-mon[40250]: pgmap v18925: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:29:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:30] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:29:31.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:30 smithi022 ceph-mon[31360]: pgmap v18926: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:31.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:30 smithi022 ceph-mon[40250]: pgmap v18926: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:31.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:30 smithi122 ceph-mon[32255]: pgmap v18926: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:31.399 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:31.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:31.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:31.765 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:29:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:29:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:29:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:32 smithi022 ceph-mon[31360]: pgmap v18927: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1311026271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:32 smithi022 ceph-mon[40250]: pgmap v18927: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1311026271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:32 smithi122 ceph-mon[32255]: pgmap v18927: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1311026271' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:34.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:29:34.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:29:34.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:33 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:29:34.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:29:34.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:29:34.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:33 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:29:34.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:29:34.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:29:34.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:33 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:29:34.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:29:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:29:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[31360]: pgmap v18928: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:29:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:29:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:29:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:29:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[40250]: pgmap v18928: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:29:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:29:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:29:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:29:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:34 smithi122 ceph-mon[32255]: pgmap v18928: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:29:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:29:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:29:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:29:36.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:36.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:36.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:36.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:37.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:36 smithi022 ceph-mon[31360]: pgmap v18929: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:37.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:36 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:37.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:36 smithi022 ceph-mon[40250]: pgmap v18929: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:37.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:36 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:37.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:37.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:37.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:36 smithi122 ceph-mon[32255]: pgmap v18929: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:36 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:37.738 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:29:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:29:37.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2828941177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:37.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:37.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:37 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:38.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2828941177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:38.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:38.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:37 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2828941177' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:38.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:37 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:29:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:38 smithi022 ceph-mon[31360]: pgmap v18930: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:39.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:38 smithi022 ceph-mon[40250]: pgmap v18930: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:39.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:38 smithi122 ceph-mon[32255]: pgmap v18930: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:29:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:40] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:29:41.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:40 smithi022 ceph-mon[31360]: pgmap v18931: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:41.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:40 smithi022 ceph-mon[40250]: pgmap v18931: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:41.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:40 smithi122 ceph-mon[32255]: pgmap v18931: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:42.134 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:42.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:42.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:42.502 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:29:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:29:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:29:42.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:42 smithi022 ceph-mon[31360]: pgmap v18932: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:42.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2257773744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:42.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:42 smithi022 ceph-mon[40250]: pgmap v18932: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:42.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2257773744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:43.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:42 smithi122 ceph-mon[32255]: pgmap v18932: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:43.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2257773744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:44.748 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:29:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:29:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:44 smithi022 ceph-mon[31360]: pgmap v18933: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:45.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:44 smithi022 ceph-mon[40250]: pgmap v18933: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:45.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:44 smithi122 ceph-mon[32255]: pgmap v18933: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:46.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:47.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:46 smithi022 ceph-mon[31360]: pgmap v18934: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:47.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:46 smithi022 ceph-mon[40250]: pgmap v18934: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:46 smithi122 ceph-mon[32255]: pgmap v18934: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:47.503 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:29:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:29:47.870 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:47.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:47.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:49.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:48 smithi022 ceph-mon[31360]: pgmap v18935: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:49.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3311068381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:49.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:48 smithi022 ceph-mon[40250]: pgmap v18935: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:49.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3311068381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:49.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:48 smithi122 ceph-mon[32255]: pgmap v18935: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:49.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3311068381' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:29:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:50] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:29:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:50 smithi022 ceph-mon[31360]: pgmap v18936: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:50 smithi022 ceph-mon[40250]: pgmap v18936: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:50 smithi122 ceph-mon[32255]: pgmap v18936: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:29:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:29:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:29:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:52 smithi022 ceph-mon[31360]: pgmap v18937: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:53.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:52 smithi022 ceph-mon[40250]: pgmap v18937: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:52 smithi122 ceph-mon[32255]: pgmap v18937: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:53.242 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:53.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:53.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:54.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1394453440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:54.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1394453440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:54.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1394453440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:54.767 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:29:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:29:55.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:54 smithi022 ceph-mon[31360]: pgmap v18938: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:54 smithi022 ceph-mon[40250]: pgmap v18938: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:54 smithi122 ceph-mon[32255]: pgmap v18938: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:29:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:56.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:29:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:56 smithi022 ceph-mon[31360]: pgmap v18939: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:57.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:56 smithi022 ceph-mon[40250]: pgmap v18939: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:56 smithi122 ceph-mon[32255]: pgmap v18939: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:57.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:29:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:29:58.244 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:29:58.616 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:29:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:29:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:29:59.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:58 smithi022 ceph-mon[31360]: pgmap v18940: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:59.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:29:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3278761941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:59.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:58 smithi022 ceph-mon[40250]: pgmap v18940: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:59.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:29:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3278761941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:29:59.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:58 smithi122 ceph-mon[32255]: pgmap v18940: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:29:59.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:29:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3278761941' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:30:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:00] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:30:01.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:00 smithi022 ceph-mon[31360]: pgmap v18941: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:01.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T04:30:01.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:00 smithi022 ceph-mon[40250]: pgmap v18941: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:01.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T04:30:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:00 smithi122 ceph-mon[32255]: pgmap v18941: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T04:30:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:30:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:30:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:30:03.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:02 smithi022 ceph-mon[31360]: pgmap v18942: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:02 smithi022 ceph-mon[40250]: pgmap v18942: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:03.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:02 smithi122 ceph-mon[32255]: pgmap v18942: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:03.618 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:03.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:03.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:03.988 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:04.783 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:30:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:30:05.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:04 smithi022 ceph-mon[31360]: pgmap v18943: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:05.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/855823894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:05.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:04 smithi022 ceph-mon[40250]: pgmap v18943: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:05.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/855823894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:05.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:04 smithi122 ceph-mon[32255]: pgmap v18943: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:05.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/855823894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:06.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:06.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:06.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:06 smithi122 ceph-mon[32255]: pgmap v18944: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:07.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:06 smithi022 ceph-mon[31360]: pgmap v18944: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:07.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:06 smithi022 ceph-mon[40250]: pgmap v18944: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:30:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:30:08.989 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:08 smithi122 ceph-mon[32255]: pgmap v18945: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:09.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:08 smithi022 ceph-mon[31360]: pgmap v18945: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:09.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:08 smithi022 ceph-mon[40250]: pgmap v18945: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:09.357 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:09.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:09.358 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:10.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/551064569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:10.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/551064569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:10.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/551064569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:10.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:30:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:10] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:30:11.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:10 smithi122 ceph-mon[32255]: pgmap v18946: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:11.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:10 smithi022 ceph-mon[31360]: pgmap v18946: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:11.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:10 smithi022 ceph-mon[40250]: pgmap v18946: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:30:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:30:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:30:13.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:12 smithi122 ceph-mon[32255]: pgmap v18947: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:13.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:12 smithi022 ceph-mon[31360]: pgmap v18947: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:13.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:12 smithi022 ceph-mon[40250]: pgmap v18947: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:14.359 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:14.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:14.728 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:14.729 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:14.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:30:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:30:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:14 smithi022 ceph-mon[31360]: pgmap v18948: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:15.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1987802819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:15.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:14 smithi022 ceph-mon[40250]: pgmap v18948: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:15.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1987802819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:15.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:14 smithi122 ceph-mon[32255]: pgmap v18948: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:15.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1987802819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:16.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:16.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:16.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:17.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:16 smithi122 ceph-mon[32255]: pgmap v18949: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:17.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:16 smithi022 ceph-mon[31360]: pgmap v18949: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:17.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:16 smithi022 ceph-mon[40250]: pgmap v18949: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:17.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:30:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:30:18.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:17 smithi122 ceph-mon[32255]: pgmap v18950: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:18.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:17 smithi022 ceph-mon[31360]: pgmap v18950: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:18.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:17 smithi022 ceph-mon[40250]: pgmap v18950: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:19.730 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:20.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:20.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:20.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:30:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:20] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:30:21.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:20 smithi022 ceph-mon[31360]: pgmap v18951: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:21.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2510834105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:21.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:20 smithi022 ceph-mon[40250]: pgmap v18951: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:21.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2510834105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:21.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:20 smithi122 ceph-mon[32255]: pgmap v18951: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2510834105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:22.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:30:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:30:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:30:23.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:22 smithi022 ceph-mon[31360]: pgmap v18952: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:23.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:22 smithi022 ceph-mon[40250]: pgmap v18952: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:22 smithi122 ceph-mon[32255]: pgmap v18952: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:24.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:30:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:30:25.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:24 smithi022 ceph-mon[31360]: pgmap v18953: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:25.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:24 smithi022 ceph-mon[40250]: pgmap v18953: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:25.101 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:25.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:24 smithi122 ceph-mon[32255]: pgmap v18953: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:25.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:25.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:25.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:26.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:26.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/440590549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:26.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:26.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/440590549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/440590549' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:27.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:26 smithi022 ceph-mon[31360]: pgmap v18954: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:27.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:26 smithi022 ceph-mon[40250]: pgmap v18954: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:27.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:26 smithi122 ceph-mon[32255]: pgmap v18954: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:30:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:30:29.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:28 smithi022 ceph-mon[31360]: pgmap v18955: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:29.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:28 smithi022 ceph-mon[40250]: pgmap v18955: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:29.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:28 smithi122 ceph-mon[32255]: pgmap v18955: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:30:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:30] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:30:30.475 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:30.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:30.848 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:30.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:31.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:30 smithi022 ceph-mon[31360]: pgmap v18956: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:31.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:30 smithi022 ceph-mon[40250]: pgmap v18956: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:31.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:30 smithi122 ceph-mon[32255]: pgmap v18956: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:32.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2978611221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:32.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2978611221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:32.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2978611221' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:30:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:30:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:30:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:32 smithi022 ceph-mon[31360]: pgmap v18957: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:32 smithi022 ceph-mon[40250]: pgmap v18957: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:32 smithi122 ceph-mon[32255]: pgmap v18957: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:34.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:30:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:30:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[31360]: pgmap v18958: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:30:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:30:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:30:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:30:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[40250]: pgmap v18958: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:30:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:30:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:30:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:30:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:34 smithi122 ceph-mon[32255]: pgmap v18958: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:30:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:30:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:30:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:30:35.850 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:36.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:36.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:36.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:36.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:36.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:36.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:37.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:36 smithi022 ceph-mon[31360]: pgmap v18959: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:37.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3302543713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:37.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:36 smithi022 ceph-mon[40250]: pgmap v18959: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:37.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3302543713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:36 smithi122 ceph-mon[32255]: pgmap v18959: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3302543713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:37.731 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:30:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:30:38.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:30:38.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:30:38.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:37 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:30:38.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:30:38.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:30:38.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:37 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:30:38.127 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:30:38.128 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:30:38.128 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:37 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:30:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:38 smithi022 ceph-mon[31360]: pgmap v18960: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:30:39.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:38 smithi022 ceph-mon[40250]: pgmap v18960: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:39.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:30:39.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:38 smithi122 ceph-mon[32255]: pgmap v18960: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:30:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:30:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:40] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:30:41.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:40 smithi022 ceph-mon[31360]: pgmap v18961: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:41.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:40 smithi022 ceph-mon[40250]: pgmap v18961: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:41.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:40 smithi122 ceph-mon[32255]: pgmap v18961: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:41.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:41.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:41.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:42.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3928333794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:42.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3928333794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:42.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3928333794' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:30:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:30:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:30:43.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:42 smithi022 ceph-mon[31360]: pgmap v18962: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:43.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:42 smithi022 ceph-mon[40250]: pgmap v18962: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:43.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:42 smithi122 ceph-mon[32255]: pgmap v18962: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:44.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:30:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:30:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:44 smithi022 ceph-mon[31360]: pgmap v18963: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:45.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:44 smithi022 ceph-mon[40250]: pgmap v18963: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:45.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:44 smithi122 ceph-mon[32255]: pgmap v18963: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:46.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:46.289 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:46.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:46.965 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:46.966 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:46.966 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:46 smithi022 ceph-mon[31360]: pgmap v18964: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:46.967 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:46 smithi022 ceph-mon[40250]: pgmap v18964: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:46 smithi122 ceph-mon[32255]: pgmap v18964: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:30:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:30:48.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1084503814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:48.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1084503814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:48.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1084503814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:49.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:48 smithi122 ceph-mon[32255]: pgmap v18965: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:49.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:48 smithi022 ceph-mon[31360]: pgmap v18965: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:49.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:48 smithi022 ceph-mon[40250]: pgmap v18965: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:30:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:50] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:30:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:50 smithi122 ceph-mon[32255]: pgmap v18966: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:51.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:50 smithi022 ceph-mon[31360]: pgmap v18966: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:51.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:50 smithi022 ceph-mon[40250]: pgmap v18966: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:51.968 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:52.339 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:52.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:52.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:30:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:30:52.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:30:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:52 smithi122 ceph-mon[32255]: pgmap v18967: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1243666046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:53.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:52 smithi022 ceph-mon[31360]: pgmap v18967: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1243666046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:53.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:52 smithi022 ceph-mon[40250]: pgmap v18967: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1243666046' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:54.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:30:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:30:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:54 smithi122 ceph-mon[32255]: pgmap v18968: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:55.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:54 smithi022 ceph-mon[31360]: pgmap v18968: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:55.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:54 smithi022 ceph-mon[40250]: pgmap v18968: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:30:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:56.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:56.372 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:30:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:56 smithi122 ceph-mon[32255]: pgmap v18969: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:57.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:56 smithi022 ceph-mon[31360]: pgmap v18969: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:56 smithi022 ceph-mon[40250]: pgmap v18969: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:57.337 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:30:57.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:30:57.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:30:57.706 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:30:57.707 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:30:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:30:58.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3206791380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:58.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3206791380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:58.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3206791380' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:30:59.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:58 smithi122 ceph-mon[32255]: pgmap v18970: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:59.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:58 smithi022 ceph-mon[31360]: pgmap v18970: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:30:59.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:58 smithi022 ceph-mon[40250]: pgmap v18970: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:00.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:30:59 smithi122 ceph-mon[32255]: pgmap v18971: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:00.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:30:59 smithi022 ceph-mon[40250]: pgmap v18971: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:00.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:30:59 smithi022 ceph-mon[31360]: pgmap v18971: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:31:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:00] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:31:02.631 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:31:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:31:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:31:02.707 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:02.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:02 smithi122 ceph-mon[32255]: pgmap v18972: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:03.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:02 smithi022 ceph-mon[31360]: pgmap v18972: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:02 smithi022 ceph-mon[40250]: pgmap v18972: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:03.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:03.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:03.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:03.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/901330141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:04.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/901330141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:04.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/901330141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:04.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:04 smithi122 ceph-mon[32255]: pgmap v18973: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:04.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:31:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:31:05.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:04 smithi022 ceph-mon[31360]: pgmap v18973: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:05.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:04 smithi022 ceph-mon[40250]: pgmap v18973: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:06.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:06.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:06.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:07.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:06 smithi022 ceph-mon[31360]: pgmap v18974: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:07.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:06 smithi022 ceph-mon[40250]: pgmap v18974: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:06 smithi122 ceph-mon[32255]: pgmap v18974: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:31:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:31:08.079 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:08.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:08.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:08.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:09.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:08 smithi022 ceph-mon[31360]: pgmap v18975: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:09.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1951174036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:09.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:08 smithi022 ceph-mon[40250]: pgmap v18975: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:09.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1951174036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:08 smithi122 ceph-mon[32255]: pgmap v18975: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1951174036' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:10.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:31:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:10] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:31:11.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:10 smithi022 ceph-mon[31360]: pgmap v18976: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:11.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:10 smithi022 ceph-mon[40250]: pgmap v18976: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:11.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:10 smithi122 ceph-mon[32255]: pgmap v18976: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:31:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:31:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:31:13.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:12 smithi022 ceph-mon[31360]: pgmap v18977: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:13.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:12 smithi022 ceph-mon[40250]: pgmap v18977: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:13.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:12 smithi122 ceph-mon[32255]: pgmap v18977: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:13.452 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:13.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:13.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:13.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:14.940 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:31:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:31:14.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:14 smithi122 ceph-mon[32255]: pgmap v18978: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:14.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3672888727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:15.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:14 smithi022 ceph-mon[31360]: pgmap v18978: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:15.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3672888727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:15.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:14 smithi022 ceph-mon[40250]: pgmap v18978: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:15.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3672888727' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:16.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:16.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:16.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:17.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:16 smithi022 ceph-mon[31360]: pgmap v18979: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:17.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:16 smithi022 ceph-mon[40250]: pgmap v18979: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:17.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:16 smithi122 ceph-mon[32255]: pgmap v18979: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:17.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:31:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:31:18.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:19.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:18 smithi022 ceph-mon[31360]: pgmap v18980: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:19.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:18 smithi022 ceph-mon[40250]: pgmap v18980: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:18 smithi122 ceph-mon[32255]: pgmap v18980: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:19.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:19.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:19.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:20.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/727529992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:20.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/727529992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:20.177 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/727529992' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:31:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:20] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:31:21.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:20 smithi022 ceph-mon[31360]: pgmap v18981: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:21.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:20 smithi022 ceph-mon[40250]: pgmap v18981: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:21.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:20 smithi122 ceph-mon[32255]: pgmap v18981: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:22.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:31:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:31:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:31:23.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:22 smithi022 ceph-mon[31360]: pgmap v18982: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:23.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:22 smithi022 ceph-mon[40250]: pgmap v18982: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:22 smithi122 ceph-mon[32255]: pgmap v18982: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:24.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:24.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:24.564 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:24.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:24.781 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:31:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:31:25.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:24 smithi022 ceph-mon[31360]: pgmap v18983: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:25.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2622952167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:25.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:24 smithi022 ceph-mon[40250]: pgmap v18983: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:25.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2622952167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:25.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:24 smithi122 ceph-mon[32255]: pgmap v18983: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:25.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2622952167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:26.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:27.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:26 smithi122 ceph-mon[32255]: pgmap v18984: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:26 smithi022 ceph-mon[31360]: pgmap v18984: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:27.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:26 smithi022 ceph-mon[40250]: pgmap v18984: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:27.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:31:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:31:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:28 smithi122 ceph-mon[32255]: pgmap v18985: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:29.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:28 smithi022 ceph-mon[31360]: pgmap v18985: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:29.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:28 smithi022 ceph-mon[40250]: pgmap v18985: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:29.566 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:29.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:29.939 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:29.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:31:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:30] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:31:31.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:30 smithi122 ceph-mon[32255]: pgmap v18986: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1669820527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:30 smithi022 ceph-mon[31360]: pgmap v18986: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1669820527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:31.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:30 smithi022 ceph-mon[40250]: pgmap v18986: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1669820527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:31:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:31:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:31:33.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:32 smithi122 ceph-mon[32255]: pgmap v18987: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:33.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:32 smithi022 ceph-mon[31360]: pgmap v18987: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:33.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:32 smithi022 ceph-mon[40250]: pgmap v18987: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:34.798 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:31:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:31:34.941 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:35.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:34 smithi122 ceph-mon[32255]: pgmap v18988: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:31:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:31:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:31:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:31:35.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[31360]: pgmap v18988: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:31:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:31:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:31:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:31:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[40250]: pgmap v18988: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:31:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:31:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:31:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:31:35.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:35.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:35.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:36.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2825026527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2825026527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:36.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2825026527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:36 smithi122 ceph-mon[32255]: pgmap v18989: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:37.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:36 smithi022 ceph-mon[31360]: pgmap v18989: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:36 smithi022 ceph-mon[40250]: pgmap v18989: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:37.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:31:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:31:38.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:38 smithi122 ceph-mon[32255]: pgmap v18990: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:38.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:31:38.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:31:38.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:31:38.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:38 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:31:39.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[31360]: pgmap v18990: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:31:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:31:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:31:39.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:31:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[40250]: pgmap v18990: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:31:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:31:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:31:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:38 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:31:40.312 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:31:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:40] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:31:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:41.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:40 smithi022 ceph-mon[31360]: pgmap v18991: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:41.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1737087007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:41.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:40 smithi022 ceph-mon[40250]: pgmap v18991: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:41.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1737087007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:41.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:40 smithi122 ceph-mon[32255]: pgmap v18991: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:41.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1737087007' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:31:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:31:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:31:43.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:42 smithi122 ceph-mon[32255]: pgmap v18992: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:42 smithi022 ceph-mon[31360]: pgmap v18992: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:43.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:42 smithi022 ceph-mon[40250]: pgmap v18992: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:44.815 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:31:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:31:45.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:44 smithi122 ceph-mon[32255]: pgmap v18993: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:45.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:44 smithi022 ceph-mon[31360]: pgmap v18993: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:45.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:44 smithi022 ceph-mon[40250]: pgmap v18993: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:45.684 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:46.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:46.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:46.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:46.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:46 smithi122 ceph-mon[32255]: pgmap v18994: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/464328550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:46 smithi022 ceph-mon[31360]: pgmap v18994: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/464328550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:47.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:46 smithi022 ceph-mon[40250]: pgmap v18994: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:47.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/464328550' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:31:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:31:48.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:47 smithi122 ceph-mon[32255]: pgmap v18995: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:48.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:47 smithi022 ceph-mon[31360]: pgmap v18995: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:48.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:47 smithi022 ceph-mon[40250]: pgmap v18995: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:31:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:50] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:31:51.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:50 smithi022 ceph-mon[31360]: pgmap v18996: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:50 smithi022 ceph-mon[40250]: pgmap v18996: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:51.056 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:50 smithi122 ceph-mon[32255]: pgmap v18996: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:51.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:51.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:52.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2860531790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:52.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2860531790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:52.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2860531790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:31:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:31:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:31:53.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:52 smithi022 ceph-mon[31360]: pgmap v18997: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:53.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:52 smithi022 ceph-mon[40250]: pgmap v18997: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:52 smithi122 ceph-mon[32255]: pgmap v18997: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:54.940 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:31:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:31:54.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:54 smithi122 ceph-mon[32255]: pgmap v18998: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:55.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:54 smithi022 ceph-mon[31360]: pgmap v18998: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:54 smithi022 ceph-mon[40250]: pgmap v18998: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:31:56.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:56.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:31:56.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:31:56.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:31:56.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:31:56.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:31:57.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:56 smithi022 ceph-mon[31360]: pgmap v18999: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:57.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:56 smithi022 ceph-mon[40250]: pgmap v18999: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:56 smithi122 ceph-mon[32255]: pgmap v18999: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:57.772 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:31:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:31:58.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2878148635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:58.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2878148635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:58.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2878148635' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:31:59.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:31:58 smithi022 ceph-mon[31360]: pgmap v19000: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:59.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:31:58 smithi022 ceph-mon[40250]: pgmap v19000: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:31:59.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:31:58 smithi122 ceph-mon[32255]: pgmap v19000: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:32:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:00] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:32:01.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:00 smithi022 ceph-mon[31360]: pgmap v19001: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:01.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:00 smithi022 ceph-mon[40250]: pgmap v19001: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:00 smithi122 ceph-mon[32255]: pgmap v19001: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:02.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:02.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:02.171 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:32:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:32:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:32:03.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:02 smithi022 ceph-mon[31360]: pgmap v19002: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:03.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/181439869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:03.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:02 smithi022 ceph-mon[40250]: pgmap v19002: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:03.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/181439869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:03.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:02 smithi122 ceph-mon[32255]: pgmap v19002: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/181439869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:04.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:04 smithi122 ceph-mon[32255]: pgmap v19003: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:04.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:32:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:32:05.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:04 smithi022 ceph-mon[31360]: pgmap v19003: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:05.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:04 smithi022 ceph-mon[40250]: pgmap v19003: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:06.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:06.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:06.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:07.173 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:07.173 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:06 smithi022 ceph-mon[31360]: pgmap v19004: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:07.174 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:06 smithi022 ceph-mon[40250]: pgmap v19004: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:06 smithi122 ceph-mon[32255]: pgmap v19004: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:07.533 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:32:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:32:07.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:07.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:07.542 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:07.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1488116060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:07.793 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1488116060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:08.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1488116060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:08 smithi122 ceph-mon[32255]: pgmap v19005: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:09.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:08 smithi022 ceph-mon[31360]: pgmap v19005: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:09.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:08 smithi022 ceph-mon[40250]: pgmap v19005: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:10.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:32:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:10] "GET /metrics HTTP/1.1" 200 202505 "" "Prometheus/2.18.1" 2021-12-16T04:32:11.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:10 smithi122 ceph-mon[32255]: pgmap v19006: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:11.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:10 smithi022 ceph-mon[31360]: pgmap v19006: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:11.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:10 smithi022 ceph-mon[40250]: pgmap v19006: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:12.544 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:32:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:32:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:32:12.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:12.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:12.911 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:13.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:12 smithi122 ceph-mon[32255]: pgmap v19007: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:13.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:12 smithi022 ceph-mon[31360]: pgmap v19007: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:13.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:12 smithi022 ceph-mon[40250]: pgmap v19007: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:14.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3358744899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:14.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3358744899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:14.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3358744899' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:14.806 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:32:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:32:15.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:14 smithi122 ceph-mon[32255]: pgmap v19008: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:14 smithi022 ceph-mon[31360]: pgmap v19008: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:15.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:14 smithi022 ceph-mon[40250]: pgmap v19008: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:16.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:16.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:17.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:16 smithi122 ceph-mon[32255]: pgmap v19009: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:17.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:16 smithi022 ceph-mon[31360]: pgmap v19009: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:17.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:16 smithi022 ceph-mon[40250]: pgmap v19009: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:32:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:32:17.913 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:18.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:18.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:18.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:18 smithi122 ceph-mon[32255]: pgmap v19010: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4037335099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:19.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:18 smithi022 ceph-mon[31360]: pgmap v19010: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:19.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4037335099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:19.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:18 smithi022 ceph-mon[40250]: pgmap v19010: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4037335099' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:32:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:20] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:32:21.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:20 smithi122 ceph-mon[32255]: pgmap v19011: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:20 smithi022 ceph-mon[31360]: pgmap v19011: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:20 smithi022 ceph-mon[40250]: pgmap v19011: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:32:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:32:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:32:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:22 smithi122 ceph-mon[32255]: pgmap v19012: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:23.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:22 smithi022 ceph-mon[31360]: pgmap v19012: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:23.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:22 smithi022 ceph-mon[40250]: pgmap v19012: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:23.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:23.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:23.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:23.651 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:24.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3213327785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:24.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3213327785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:24.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3213327785' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:24.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:32:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:32:25.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:24 smithi122 ceph-mon[32255]: pgmap v19013: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:25.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:24 smithi022 ceph-mon[31360]: pgmap v19013: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:25.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:24 smithi022 ceph-mon[40250]: pgmap v19013: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:25 smithi122 ceph-mon[32255]: pgmap v19014: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:25 smithi022 ceph-mon[31360]: pgmap v19014: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:25 smithi022 ceph-mon[40250]: pgmap v19014: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:27.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:32:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:32:28.653 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:28.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:28 smithi122 ceph-mon[32255]: pgmap v19015: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:29.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:29.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:29.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:29.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:28 smithi022 ceph-mon[31360]: pgmap v19015: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:29.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:28 smithi022 ceph-mon[40250]: pgmap v19015: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:30.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2768962216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:30.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2768962216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:30.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2768962216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:32:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:30] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:32:31.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:30 smithi022 ceph-mon[31360]: pgmap v19016: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:31.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:30 smithi022 ceph-mon[40250]: pgmap v19016: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:31.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:30 smithi122 ceph-mon[32255]: pgmap v19016: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:32:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:32:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:32:33.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:32 smithi022 ceph-mon[31360]: pgmap v19017: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:33.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:32 smithi022 ceph-mon[40250]: pgmap v19017: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:32 smithi122 ceph-mon[32255]: pgmap v19017: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:34.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:34.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:34.400 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:34.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:34.940 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:32:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:32:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:34 smithi122 ceph-mon[32255]: pgmap v19018: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1673457679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:32:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:32:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:32:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:32:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[31360]: pgmap v19018: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1673457679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:32:35.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:32:35.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:32:35.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:32:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[40250]: pgmap v19018: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1673457679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:32:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:32:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:32:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:32:36.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:36.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:36.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:37.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:36 smithi022 ceph-mon[31360]: pgmap v19019: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:37.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:36 smithi022 ceph-mon[40250]: pgmap v19019: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:36 smithi122 ceph-mon[32255]: pgmap v19019: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:32:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:32:38.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:38 smithi122 ceph-mon[32255]: pgmap v19020: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:38.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:32:38.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:32:38.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:38 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:32:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[31360]: pgmap v19020: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:32:39.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:32:39.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:32:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[40250]: pgmap v19020: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:32:39.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:32:39.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:38 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:32:39.403 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:39.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:39.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:39.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:32:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:40] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:32:40.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:32:40.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:40 smithi122 ceph-mon[32255]: pgmap v19021: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:40.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3491671223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:40.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:32:40.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:40 smithi022 ceph-mon[31360]: pgmap v19021: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:40.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3491671223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:40.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:32:40.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:40 smithi022 ceph-mon[40250]: pgmap v19021: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:40.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3491671223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:42.661 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:32:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:32:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:32:42.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:42 smithi122 ceph-mon[32255]: pgmap v19022: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:43.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:42 smithi022 ceph-mon[31360]: pgmap v19022: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:43.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:42 smithi022 ceph-mon[40250]: pgmap v19022: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:44.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:44.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:44 smithi122 ceph-mon[32255]: pgmap v19023: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:44.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:32:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:32:45.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:44 smithi022 ceph-mon[31360]: pgmap v19023: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:45.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:44 smithi022 ceph-mon[40250]: pgmap v19023: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:45.131 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:45.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:45.132 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:46.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:46.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/822958342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:46.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:46.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/822958342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/822958342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:47.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:46 smithi022 ceph-mon[40250]: pgmap v19024: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:47.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:46 smithi022 ceph-mon[31360]: pgmap v19024: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:46 smithi122 ceph-mon[32255]: pgmap v19024: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:32:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:32:49.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:48 smithi022 ceph-mon[31360]: pgmap v19025: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:49.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:48 smithi022 ceph-mon[40250]: pgmap v19025: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:49.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:48 smithi122 ceph-mon[32255]: pgmap v19025: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:50.133 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:32:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:50] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:32:50.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:50.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:50.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:51.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:50 smithi022 ceph-mon[31360]: pgmap v19026: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:51.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/754222529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:51.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:50 smithi022 ceph-mon[40250]: pgmap v19026: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:51.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/754222529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:51.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:50 smithi122 ceph-mon[32255]: pgmap v19026: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/754222529' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:32:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:32:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:32:53.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:52 smithi022 ceph-mon[31360]: pgmap v19027: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:53.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:52 smithi022 ceph-mon[40250]: pgmap v19027: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:53.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:52 smithi122 ceph-mon[32255]: pgmap v19027: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:54.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:32:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:32:54.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:54 smithi122 ceph-mon[32255]: pgmap v19028: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:55.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:54 smithi022 ceph-mon[31360]: pgmap v19028: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:55.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:54 smithi022 ceph-mon[40250]: pgmap v19028: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:32:55.503 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:32:55.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:32:55.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:32:55.876 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:32:56.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:56.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:56.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:32:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:56 smithi122 ceph-mon[32255]: pgmap v19029: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1715007409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:56 smithi022 ceph-mon[31360]: pgmap v19029: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1715007409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:56 smithi022 ceph-mon[40250]: pgmap v19029: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1715007409' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:32:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:32:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:32:59.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:32:58 smithi122 ceph-mon[32255]: pgmap v19030: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:59.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:32:58 smithi022 ceph-mon[40250]: pgmap v19030: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:32:59.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:32:58 smithi022 ceph-mon[31360]: pgmap v19030: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:33:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:00] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:33:00.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:00 smithi122 ceph-mon[32255]: pgmap v19031: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:01.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:01.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:01.250 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:01.250 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:00 smithi022 ceph-mon[31360]: pgmap v19031: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:01.250 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:00 smithi022 ceph-mon[40250]: pgmap v19031: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:02.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/305213347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:02.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/305213347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:02.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/305213347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:33:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:33:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:33:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:02 smithi122 ceph-mon[32255]: pgmap v19032: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:02 smithi022 ceph-mon[31360]: pgmap v19032: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:02 smithi022 ceph-mon[40250]: pgmap v19032: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:04.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:04 smithi122 ceph-mon[32255]: pgmap v19033: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:04.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:33:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:33:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:04 smithi022 ceph-mon[31360]: pgmap v19033: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:05.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:04 smithi022 ceph-mon[40250]: pgmap v19033: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:06.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:06.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:06.252 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:06.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:06.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:06.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:06.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:07.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:06 smithi022 ceph-mon[31360]: pgmap v19034: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:07.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3382709713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:07.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:06 smithi022 ceph-mon[40250]: pgmap v19034: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:07.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3382709713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:07.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:06 smithi122 ceph-mon[32255]: pgmap v19034: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3382709713' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:07.786 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:33:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:33:09.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:08 smithi122 ceph-mon[32255]: pgmap v19035: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:09.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:08 smithi022 ceph-mon[31360]: pgmap v19035: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:09.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:08 smithi022 ceph-mon[40250]: pgmap v19035: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:33:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:10] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:33:11.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:10 smithi122 ceph-mon[32255]: pgmap v19036: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:11.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:10 smithi022 ceph-mon[31360]: pgmap v19036: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:11.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:10 smithi022 ceph-mon[40250]: pgmap v19036: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:11.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:11.990 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:11.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:11.991 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:33:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:33:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:33:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:12 smithi122 ceph-mon[32255]: pgmap v19037: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3956617184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:12 smithi022 ceph-mon[31360]: pgmap v19037: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3956617184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:12 smithi022 ceph-mon[40250]: pgmap v19037: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3956617184' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:14.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:14 smithi122 ceph-mon[32255]: pgmap v19038: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:14.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:33:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:33:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:14 smithi022 ceph-mon[31360]: pgmap v19038: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:15.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:14 smithi022 ceph-mon[40250]: pgmap v19038: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:16.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:16.993 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:16 smithi122 ceph-mon[32255]: pgmap v19039: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:16 smithi022 ceph-mon[31360]: pgmap v19039: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:17.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:16 smithi022 ceph-mon[40250]: pgmap v19039: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:17.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:17.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:17.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:33:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:33:18.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2498814855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:18.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2498814855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:18.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2498814855' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:18 smithi122 ceph-mon[32255]: pgmap v19040: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:19.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:18 smithi022 ceph-mon[31360]: pgmap v19040: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:18 smithi022 ceph-mon[40250]: pgmap v19040: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:33:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:20] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:33:21.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:20 smithi122 ceph-mon[32255]: pgmap v19041: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:20 smithi022 ceph-mon[31360]: pgmap v19041: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:21.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:20 smithi022 ceph-mon[40250]: pgmap v19041: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:22.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:33:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:33:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:33:22.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:22.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:22.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:23.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:22 smithi022 ceph-mon[31360]: pgmap v19042: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:23.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2926593744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:23.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:22 smithi022 ceph-mon[40250]: pgmap v19042: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:23.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2926593744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:23.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:22 smithi122 ceph-mon[32255]: pgmap v19042: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2926593744' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:24.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:24 smithi122 ceph-mon[32255]: pgmap v19043: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:24.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:33:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:33:25.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:24 smithi022 ceph-mon[31360]: pgmap v19043: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:25.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:24 smithi022 ceph-mon[40250]: pgmap v19043: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:26.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:26.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:26.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:27.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:26 smithi022 ceph-mon[31360]: pgmap v19044: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:27.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:26 smithi022 ceph-mon[40250]: pgmap v19044: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:27.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:26 smithi122 ceph-mon[32255]: pgmap v19044: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:27.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:27.743 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:33:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:33:28.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:28.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:28.115 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:29.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:28 smithi022 ceph-mon[31360]: pgmap v19045: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:29.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/72692013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:29.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:28 smithi022 ceph-mon[40250]: pgmap v19045: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:29.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/72692013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:29.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:28 smithi122 ceph-mon[32255]: pgmap v19045: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/72692013' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:33:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:30] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:33:31.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:30 smithi122 ceph-mon[32255]: pgmap v19046: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:30 smithi022 ceph-mon[31360]: pgmap v19046: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:31.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:30 smithi022 ceph-mon[40250]: pgmap v19046: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:33:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:33:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:33:33.117 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:33.117 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:32 smithi022 ceph-mon[40250]: pgmap v19047: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:33.118 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:32 smithi022 ceph-mon[31360]: pgmap v19047: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:32 smithi122 ceph-mon[32255]: pgmap v19047: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:33.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:33.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:33.489 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:34.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/493270372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:34.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/493270372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/493270372' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:34.940 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:33:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:33:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:34 smithi122 ceph-mon[32255]: pgmap v19048: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:33:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:33:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:33:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:33:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[31360]: pgmap v19048: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:33:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:33:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:33:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:33:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[40250]: pgmap v19048: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:33:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:33:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:33:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:33:36.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:36.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:37.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:36 smithi122 ceph-mon[32255]: pgmap v19049: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:36 smithi022 ceph-mon[31360]: pgmap v19049: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:36 smithi022 ceph-mon[40250]: pgmap v19049: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:33:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:33:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:38.864 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:38.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:38.865 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:39.119 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:38 smithi022 ceph-mon[31360]: pgmap v19050: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:39.119 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:38 smithi022 ceph-mon[40250]: pgmap v19050: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:39.131 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:38 smithi122 ceph-mon[32255]: pgmap v19050: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:39.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2802566737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:39.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:33:39.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:33:39.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:33:39.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:39 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:33:40.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2802566737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:40.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:33:40.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:33:40.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:33:40.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:33:40.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2802566737' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:40.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:33:40.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:33:40.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:33:40.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:39 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:33:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:33:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:40] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:33:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:40 smithi122 ceph-mon[32255]: pgmap v19051: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:40 smithi022 ceph-mon[31360]: pgmap v19051: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:41.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:40 smithi022 ceph-mon[40250]: pgmap v19051: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:33:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:33:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:33:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:42 smithi122 ceph-mon[32255]: pgmap v19052: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:42 smithi022 ceph-mon[31360]: pgmap v19052: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:42 smithi022 ceph-mon[40250]: pgmap v19052: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:43.868 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:44.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:44.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:44.240 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:44.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:33:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:33:44.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:44 smithi122 ceph-mon[32255]: pgmap v19053: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:44.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/645458042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:44 smithi022 ceph-mon[31360]: pgmap v19053: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:45.302 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/645458042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:45.302 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:44 smithi022 ceph-mon[40250]: pgmap v19053: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:45.303 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/645458042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:46.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:47.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:46 smithi122 ceph-mon[32255]: pgmap v19054: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:46 smithi022 ceph-mon[31360]: pgmap v19054: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:47.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:46 smithi022 ceph-mon[40250]: pgmap v19054: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:33:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:33:49.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:48 smithi122 ceph-mon[32255]: pgmap v19055: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:49.242 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:49.243 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:48 smithi022 ceph-mon[31360]: pgmap v19055: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:49.243 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:48 smithi022 ceph-mon[40250]: pgmap v19055: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:49.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:49.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:49.612 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:50.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2460755472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:50.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2460755472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:50.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2460755472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:33:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:50] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:33:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:50 smithi122 ceph-mon[32255]: pgmap v19056: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:51.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:50 smithi022 ceph-mon[31360]: pgmap v19056: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:51.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:50 smithi022 ceph-mon[40250]: pgmap v19056: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:33:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:33:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:33:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:52 smithi122 ceph-mon[32255]: pgmap v19057: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:52 smithi022 ceph-mon[31360]: pgmap v19057: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:52 smithi022 ceph-mon[40250]: pgmap v19057: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:54.614 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:33:54.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:33:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:33:54.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:54 smithi122 ceph-mon[32255]: pgmap v19058: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:54.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:33:54.984 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:33:54.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:33:54.985 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:54 smithi022 ceph-mon[31360]: pgmap v19058: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:54.985 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:54 smithi022 ceph-mon[40250]: pgmap v19058: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:33:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1888941579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:56.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:56.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1888941579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:33:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1888941579' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:33:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:56 smithi122 ceph-mon[32255]: pgmap v19059: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:56 smithi022 ceph-mon[31360]: pgmap v19059: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:56 smithi022 ceph-mon[40250]: pgmap v19059: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:33:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:33:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:33:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:33:58 smithi122 ceph-mon[32255]: pgmap v19060: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:59.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:33:58 smithi022 ceph-mon[31360]: pgmap v19060: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:59.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:33:58 smithi022 ceph-mon[40250]: pgmap v19060: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:33:59.986 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:00.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:00.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:00.356 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:34:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:00] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:34:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:00 smithi122 ceph-mon[32255]: pgmap v19061: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2323491149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:00 smithi022 ceph-mon[31360]: pgmap v19061: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2323491149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:01.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:00 smithi022 ceph-mon[40250]: pgmap v19061: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2323491149' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:34:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:34:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:34:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:02 smithi122 ceph-mon[32255]: pgmap v19062: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:02 smithi022 ceph-mon[31360]: pgmap v19062: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:03.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:02 smithi022 ceph-mon[40250]: pgmap v19062: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:04.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:34:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:34:04.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:04 smithi122 ceph-mon[32255]: pgmap v19063: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:04 smithi022 ceph-mon[31360]: pgmap v19063: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:05.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:04 smithi022 ceph-mon[40250]: pgmap v19063: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:05.357 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:05.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:05.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:05.727 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:06.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:06.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:05 smithi022 ceph-mon[31360]: pgmap v19064: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:06.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2997055446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:06.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:06.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:05 smithi022 ceph-mon[40250]: pgmap v19064: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:06.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2997055446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:05 smithi122 ceph-mon[32255]: pgmap v19064: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2997055446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:34:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:34:08.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:08 smithi122 ceph-mon[32255]: pgmap v19065: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:09.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:08 smithi022 ceph-mon[31360]: pgmap v19065: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:09.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:08 smithi022 ceph-mon[40250]: pgmap v19065: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:34:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:34:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:10] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:34:10.729 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:11.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:10 smithi022 ceph-mon[31360]: pgmap v19066: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:11.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:10 smithi022 ceph-mon[40250]: pgmap v19066: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:11.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:10 smithi122 ceph-mon[32255]: pgmap v19066: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:12.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1287253890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:12.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1287253890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:12.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1287253890' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:34:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:34:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:34:13.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:12 smithi122 ceph-mon[32255]: pgmap v19067: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:12 smithi022 ceph-mon[31360]: pgmap v19067: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:13.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:12 smithi022 ceph-mon[40250]: pgmap v19067: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:14.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:14 smithi122 ceph-mon[32255]: pgmap v19068: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:14.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:34:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:34:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:14 smithi022 ceph-mon[31360]: pgmap v19068: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:15.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:14 smithi022 ceph-mon[40250]: pgmap v19068: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:16.102 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:16.103 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:16.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:16.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:16.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:16 smithi122 ceph-mon[32255]: pgmap v19069: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/880501888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:16 smithi022 ceph-mon[31360]: pgmap v19069: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/880501888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:16 smithi022 ceph-mon[40250]: pgmap v19069: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/880501888' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:34:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:34:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:18 smithi122 ceph-mon[32255]: pgmap v19070: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:19.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:18 smithi022 ceph-mon[31360]: pgmap v19070: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:19.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:18 smithi022 ceph-mon[40250]: pgmap v19070: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:34:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:20] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:34:21.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:20 smithi122 ceph-mon[32255]: pgmap v19071: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:20 smithi022 ceph-mon[31360]: pgmap v19071: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:21.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:20 smithi022 ceph-mon[40250]: pgmap v19071: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:21.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:21.838 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:21.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:21.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:34:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:34:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:34:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:22 smithi122 ceph-mon[32255]: pgmap v19072: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3010400720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:22 smithi022 ceph-mon[31360]: pgmap v19072: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3010400720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:22 smithi022 ceph-mon[40250]: pgmap v19072: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3010400720' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:24.940 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:24 smithi122 ceph-mon[32255]: pgmap v19073: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:24.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:34:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:34:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:24 smithi022 ceph-mon[31360]: pgmap v19073: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:24 smithi022 ceph-mon[40250]: pgmap v19073: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:26.841 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:26 smithi122 ceph-mon[32255]: pgmap v19074: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:27.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:27.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:27.210 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:27.211 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:26 smithi022 ceph-mon[40250]: pgmap v19074: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:27.211 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:26 smithi022 ceph-mon[31360]: pgmap v19074: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:27.534 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:34:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:34:28.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3270014224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:28.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3270014224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:28.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3270014224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:28 smithi122 ceph-mon[32255]: pgmap v19075: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:29.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:28 smithi022 ceph-mon[31360]: pgmap v19075: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:29.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:28 smithi022 ceph-mon[40250]: pgmap v19075: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:34:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:30] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:34:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:30 smithi122 ceph-mon[32255]: pgmap v19076: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:30 smithi022 ceph-mon[31360]: pgmap v19076: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:31.375 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:30 smithi022 ceph-mon[40250]: pgmap v19076: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:32.212 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:32.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:32.580 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:32.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:34:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:34:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:34:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:32 smithi022 ceph-mon[31360]: pgmap v19077: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:33.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/413994966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:32 smithi022 ceph-mon[40250]: pgmap v19077: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:33.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/413994966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:32 smithi122 ceph-mon[32255]: pgmap v19077: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:33.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/413994966' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:34.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:34:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:34:34.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:34 smithi122 ceph-mon[32255]: pgmap v19078: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:34.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:34:34.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:34:34.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:34:34.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:34:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[31360]: pgmap v19078: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:34:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:34:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:34:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:34:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[40250]: pgmap v19078: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:34:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:34:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:34:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:34:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:36.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:36 smithi122 ceph-mon[32255]: pgmap v19079: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:36 smithi022 ceph-mon[31360]: pgmap v19079: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:36 smithi022 ceph-mon[40250]: pgmap v19079: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:37.582 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:34:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:34:37.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:37.950 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:37.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:38 smithi122 ceph-mon[32255]: pgmap v19080: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/488913357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:38 smithi022 ceph-mon[31360]: pgmap v19080: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/488913357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:38 smithi022 ceph-mon[40250]: pgmap v19080: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/488913357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:39.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:34:39.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:34:39.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:39 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:34:40.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:34:40.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:34:40.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:39 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:34:40.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:34:40.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:34:40.021 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:39 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:34:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:34:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:40] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:34:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:40 smithi122 ceph-mon[32255]: pgmap v19081: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:34:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:40 smithi022 ceph-mon[31360]: pgmap v19081: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:34:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:40 smithi022 ceph-mon[40250]: pgmap v19081: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:34:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:34:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:34:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:34:42.952 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:43.195 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:42 smithi122 ceph-mon[32255]: pgmap v19082: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:43.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:42 smithi022 ceph-mon[40250]: pgmap v19082: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:42 smithi022 ceph-mon[31360]: pgmap v19082: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:43.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:43.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:43.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:44.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3704855534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:44.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3704855534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:44.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3704855534' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:44.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:34:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:34:44.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:44 smithi122 ceph-mon[32255]: pgmap v19083: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:44 smithi022 ceph-mon[31360]: pgmap v19083: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:45.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:44 smithi022 ceph-mon[40250]: pgmap v19083: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:46.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:46.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:46 smithi122 ceph-mon[32255]: pgmap v19084: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:46 smithi022 ceph-mon[31360]: pgmap v19084: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:47.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:46 smithi022 ceph-mon[40250]: pgmap v19084: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:34:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:34:48.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:47 smithi122 ceph-mon[32255]: pgmap v19085: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:48.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:47 smithi022 ceph-mon[31360]: pgmap v19085: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:48.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:47 smithi022 ceph-mon[40250]: pgmap v19085: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:48.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:48.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:48.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:48.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:49.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/839080592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:49.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/839080592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:49.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/839080592' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:50.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:49 smithi122 ceph-mon[32255]: pgmap v19086: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:50.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:49 smithi022 ceph-mon[31360]: pgmap v19086: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:50.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:49 smithi022 ceph-mon[40250]: pgmap v19086: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:34:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:50] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:34:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:34:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:34:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:34:53.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:52 smithi022 ceph-mon[31360]: pgmap v19087: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:53.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:52 smithi022 ceph-mon[40250]: pgmap v19087: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:52 smithi122 ceph-mon[32255]: pgmap v19087: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:53.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:54.072 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:54.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:54.073 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:34:54.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:34:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:34:54.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:54 smithi122 ceph-mon[32255]: pgmap v19088: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:54.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1433119462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:54 smithi022 ceph-mon[31360]: pgmap v19088: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1433119462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:54 smithi022 ceph-mon[40250]: pgmap v19088: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:34:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1433119462' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:34:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:56.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:34:57.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:56 smithi122 ceph-mon[32255]: pgmap v19089: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:56 smithi022 ceph-mon[31360]: pgmap v19089: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:56 smithi022 ceph-mon[40250]: pgmap v19089: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:34:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:34:59.075 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:34:59.075 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:58 smithi022 ceph-mon[31360]: pgmap v19090: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:59.076 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:58 smithi022 ceph-mon[40250]: pgmap v19090: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:58 smithi122 ceph-mon[32255]: pgmap v19090: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:34:59.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:34:59.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:34:59.446 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:00.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:34:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2337669011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:00.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:34:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2337669011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:00.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:34:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2337669011' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:35:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:00] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:35:01.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:00 smithi122 ceph-mon[32255]: pgmap v19091: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:00 smithi022 ceph-mon[31360]: pgmap v19091: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:01.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:00 smithi022 ceph-mon[40250]: pgmap v19091: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:35:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:35:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:35:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:02 smithi122 ceph-mon[32255]: pgmap v19092: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:02 smithi022 ceph-mon[31360]: pgmap v19092: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:02 smithi022 ceph-mon[40250]: pgmap v19092: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:04.448 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:04.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:04.816 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:04.817 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:04.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:04 smithi122 ceph-mon[32255]: pgmap v19093: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:04.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2907837743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:04.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:35:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:35:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:04 smithi022 ceph-mon[31360]: pgmap v19093: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2907837743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:05.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:04 smithi022 ceph-mon[40250]: pgmap v19093: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:05.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2907837743' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:06.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:06.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:07.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:06 smithi122 ceph-mon[32255]: pgmap v19094: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:07.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:06 smithi022 ceph-mon[31360]: pgmap v19094: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:07.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:06 smithi022 ceph-mon[40250]: pgmap v19094: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:35:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:35:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:08 smithi122 ceph-mon[32255]: pgmap v19095: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:09.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:08 smithi022 ceph-mon[31360]: pgmap v19095: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:09.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:08 smithi022 ceph-mon[40250]: pgmap v19095: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:10.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:10.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:10.188 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:35:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:10] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:35:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:10 smithi122 ceph-mon[32255]: pgmap v19096: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/253713144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:11.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:10 smithi022 ceph-mon[31360]: pgmap v19096: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/253713144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:10 smithi022 ceph-mon[40250]: pgmap v19096: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/253713144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:35:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:35:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:35:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:12 smithi122 ceph-mon[32255]: pgmap v19097: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:12 smithi022 ceph-mon[31360]: pgmap v19097: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:12 smithi022 ceph-mon[40250]: pgmap v19097: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:14.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:14 smithi122 ceph-mon[32255]: pgmap v19098: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:14.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:35:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:35:15.190 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:15.191 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:14 smithi022 ceph-mon[40250]: pgmap v19098: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:15.191 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:14 smithi022 ceph-mon[31360]: pgmap v19098: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:15.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:15.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4208808467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4208808467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4208808467' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:16 smithi122 ceph-mon[32255]: pgmap v19099: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:16 smithi022 ceph-mon[31360]: pgmap v19099: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:16 smithi022 ceph-mon[40250]: pgmap v19099: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:35:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:35:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:18 smithi122 ceph-mon[32255]: pgmap v19100: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:19.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:18 smithi022 ceph-mon[31360]: pgmap v19100: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:18 smithi022 ceph-mon[40250]: pgmap v19100: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:35:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:20] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:35:20.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:20.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:20.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:20.932 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:20 smithi122 ceph-mon[32255]: pgmap v19101: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:20 smithi022 ceph-mon[31360]: pgmap v19101: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:20 smithi022 ceph-mon[40250]: pgmap v19101: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:22.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/564929918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:22.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/564929918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:22.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/564929918' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:35:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:35:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:35:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:22 smithi122 ceph-mon[32255]: pgmap v19102: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:22 smithi022 ceph-mon[31360]: pgmap v19102: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:23.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:22 smithi022 ceph-mon[40250]: pgmap v19102: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:24.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:24 smithi122 ceph-mon[32255]: pgmap v19103: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:24.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:35:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:35:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:24 smithi022 ceph-mon[31360]: pgmap v19103: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:25.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:24 smithi022 ceph-mon[40250]: pgmap v19103: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:25.933 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:26.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:26.301 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:26.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:26.302 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:26 smithi122 ceph-mon[32255]: pgmap v19104: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3376745448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:26 smithi022 ceph-mon[31360]: pgmap v19104: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3376745448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:26 smithi022 ceph-mon[40250]: pgmap v19104: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3376745448' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:35:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:35:28.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:27 smithi122 ceph-mon[32255]: pgmap v19105: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:28.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:27 smithi022 ceph-mon[31360]: pgmap v19105: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:28.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:27 smithi022 ceph-mon[40250]: pgmap v19105: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:35:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:30] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:35:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:30 smithi122 ceph-mon[32255]: pgmap v19106: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:30 smithi022 ceph-mon[31360]: pgmap v19106: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:30 smithi022 ceph-mon[40250]: pgmap v19106: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:31.304 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:31.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:31.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:31.674 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:32.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/748704948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:32.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/748704948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:32.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/748704948' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:35:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:35:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:35:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:32 smithi122 ceph-mon[32255]: pgmap v19107: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:33.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:32 smithi022 ceph-mon[31360]: pgmap v19107: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:32 smithi022 ceph-mon[40250]: pgmap v19107: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:34.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:35:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:35:34.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:34 smithi122 ceph-mon[32255]: pgmap v19108: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:35:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:35:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:35:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:35:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[31360]: pgmap v19108: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:35:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:35:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:35:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:35:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[40250]: pgmap v19108: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:35:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:35:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:35:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:35:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:36.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:36.676 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:37.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:36 smithi022 ceph-mon[31360]: pgmap v19109: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:37.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:36 smithi022 ceph-mon[40250]: pgmap v19109: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:37.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:37.044 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:37.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:36 smithi122 ceph-mon[32255]: pgmap v19109: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:35:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:35:38.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/484439897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:38.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/484439897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:38.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/484439897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:38 smithi122 ceph-mon[32255]: pgmap v19110: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:38 smithi022 ceph-mon[31360]: pgmap v19110: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:38 smithi022 ceph-mon[40250]: pgmap v19110: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:40.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:35:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:40] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:35:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:40 smithi122 ceph-mon[32255]: pgmap v19111: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:35:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:35:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:40 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:35:41.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:41.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:40 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[31360]: pgmap v19111: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:35:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:35:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:35:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:41.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[40250]: pgmap v19111: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:41.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:35:41.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:35:41.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:35:41.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:41.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:40 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:42.047 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:41 smithi122 ceph-mon[32255]: pgmap v19112: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:42.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:42.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:41 smithi022 ceph-mon[31360]: pgmap v19112: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:42.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:35:42.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:41 smithi022 ceph-mon[40250]: pgmap v19112: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:42.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:42.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:42.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:35:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:35:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:35:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/451068808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/451068808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:43.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/451068808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:44.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:44 smithi122 ceph-mon[32255]: pgmap v19113: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:44.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:35:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:35:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:44 smithi022 ceph-mon[31360]: pgmap v19113: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:45.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:44 smithi022 ceph-mon[40250]: pgmap v19113: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:46 smithi122 ceph-mon[32255]: pgmap v19114: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:46 smithi022 ceph-mon[31360]: pgmap v19114: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:47.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:46 smithi022 ceph-mon[40250]: pgmap v19114: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:47.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:35:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:35:47.789 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:47.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:47.790 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:48.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2448299079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:48.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2448299079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:48.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2448299079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:49.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:48 smithi122 ceph-mon[32255]: pgmap v19115: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:49.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:48 smithi022 ceph-mon[31360]: pgmap v19115: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:49.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:48 smithi022 ceph-mon[40250]: pgmap v19115: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:35:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:50] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:35:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:50 smithi122 ceph-mon[32255]: pgmap v19116: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:51.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:50 smithi022 ceph-mon[31360]: pgmap v19116: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:51.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:50 smithi022 ceph-mon[40250]: pgmap v19116: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:35:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:35:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:35:52.792 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:53.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:53.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:53.163 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:53.164 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:52 smithi022 ceph-mon[40250]: pgmap v19117: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:53.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:52 smithi022 ceph-mon[31360]: pgmap v19117: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:52 smithi122 ceph-mon[32255]: pgmap v19117: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:54.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3818588141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:54.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3818588141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:54.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3818588141' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:54 smithi122 ceph-mon[32255]: pgmap v19118: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:55.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:35:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:35:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:54 smithi022 ceph-mon[31360]: pgmap v19118: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:54 smithi022 ceph-mon[40250]: pgmap v19118: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:35:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:56.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:35:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:56 smithi122 ceph-mon[32255]: pgmap v19119: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:56 smithi022 ceph-mon[40250]: pgmap v19119: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:56 smithi022 ceph-mon[31360]: pgmap v19119: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:35:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:35:58.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:35:58.534 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:35:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:35:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:35:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:58 smithi122 ceph-mon[32255]: pgmap v19120: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:35:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3718775122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:59.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:58 smithi022 ceph-mon[31360]: pgmap v19120: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:35:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3718775122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:35:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:58 smithi022 ceph-mon[40250]: pgmap v19120: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:35:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:35:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3718775122' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:36:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:00] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:36:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:00 smithi122 ceph-mon[32255]: pgmap v19121: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:00 smithi022 ceph-mon[31360]: pgmap v19121: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:00 smithi022 ceph-mon[40250]: pgmap v19121: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:36:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:36:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:36:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:02 smithi122 ceph-mon[32255]: pgmap v19122: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:02 smithi022 ceph-mon[31360]: pgmap v19122: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:02 smithi022 ceph-mon[40250]: pgmap v19122: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:03.536 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:03.906 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:03.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:03.907 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:05.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:36:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:36:05.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:04 smithi122 ceph-mon[32255]: pgmap v19123: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:05.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3535886798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:04 smithi022 ceph-mon[31360]: pgmap v19123: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3535886798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:04 smithi022 ceph-mon[40250]: pgmap v19123: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3535886798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:06.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:06.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:06 smithi122 ceph-mon[32255]: pgmap v19124: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:07.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:06 smithi022 ceph-mon[31360]: pgmap v19124: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:06 smithi022 ceph-mon[40250]: pgmap v19124: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:36:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:36:08.908 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:08 smithi122 ceph-mon[32255]: pgmap v19125: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:09.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:09.278 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:09.279 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:09.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:08 smithi022 ceph-mon[31360]: pgmap v19125: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:09.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:08 smithi022 ceph-mon[40250]: pgmap v19125: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:10.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3432052253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:10.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3432052253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:10.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3432052253' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:36:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:10] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:36:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:10 smithi122 ceph-mon[32255]: pgmap v19126: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:11.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:10 smithi022 ceph-mon[31360]: pgmap v19126: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:10 smithi022 ceph-mon[40250]: pgmap v19126: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:36:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:36:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:36:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:12 smithi122 ceph-mon[32255]: pgmap v19127: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:12 smithi022 ceph-mon[31360]: pgmap v19127: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:12 smithi022 ceph-mon[40250]: pgmap v19127: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:14.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:14.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:14.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:15.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:14 smithi022 ceph-mon[31360]: pgmap v19128: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:15.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1638853395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:15.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:14 smithi022 ceph-mon[40250]: pgmap v19128: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:15.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1638853395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:15.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:36:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:36:15.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:14 smithi122 ceph-mon[32255]: pgmap v19128: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:15.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1638853395' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:16 smithi122 ceph-mon[32255]: pgmap v19129: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:16 smithi022 ceph-mon[31360]: pgmap v19129: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:16 smithi022 ceph-mon[40250]: pgmap v19129: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:36:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:36:18.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:17 smithi122 ceph-mon[32255]: pgmap v19130: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:18.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:17 smithi022 ceph-mon[31360]: pgmap v19130: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:18.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:17 smithi022 ceph-mon[40250]: pgmap v19130: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:19.655 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:20.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:20.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:20.025 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:36:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:20] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:36:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:20 smithi122 ceph-mon[32255]: pgmap v19131: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2344095176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:20 smithi022 ceph-mon[31360]: pgmap v19131: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2344095176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:20 smithi022 ceph-mon[40250]: pgmap v19131: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2344095176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:36:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:36:22.319Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:36:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:22 smithi122 ceph-mon[32255]: pgmap v19132: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:22 smithi022 ceph-mon[31360]: pgmap v19132: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:23.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:22 smithi022 ceph-mon[40250]: pgmap v19132: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:25.026 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:25.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:24 smithi122 ceph-mon[32255]: pgmap v19133: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:25.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:36:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:36:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:24 smithi022 ceph-mon[31360]: pgmap v19133: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:25.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:24 smithi022 ceph-mon[40250]: pgmap v19133: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:25.398 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:25.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:25.399 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/97540527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/97540527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/97540527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:26 smithi122 ceph-mon[32255]: pgmap v19134: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:26 smithi022 ceph-mon[31360]: pgmap v19134: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:26 smithi022 ceph-mon[40250]: pgmap v19134: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:36:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:36:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:28 smithi122 ceph-mon[32255]: pgmap v19135: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:29.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:28 smithi022 ceph-mon[31360]: pgmap v19135: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:29.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:28 smithi022 ceph-mon[40250]: pgmap v19135: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:30.400 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:36:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:30] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:36:30.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:30.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:30.768 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:31.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:30 smithi022 ceph-mon[31360]: pgmap v19136: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:31.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/303598121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:31.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:30 smithi022 ceph-mon[40250]: pgmap v19136: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:31.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/303598121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:30 smithi122 ceph-mon[32255]: pgmap v19136: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/303598121' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:36:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:36:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:36:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:32 smithi122 ceph-mon[32255]: pgmap v19137: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:33.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:32 smithi022 ceph-mon[31360]: pgmap v19137: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:32 smithi022 ceph-mon[40250]: pgmap v19137: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:35.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:36:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:36:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:34 smithi122 ceph-mon[32255]: pgmap v19138: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:36:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:36:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:36:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:36:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[31360]: pgmap v19138: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:36:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:36:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:36:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:36:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[40250]: pgmap v19138: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:36:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:36:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:36:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:36:35.770 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:36.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:36.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:36.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:36.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:36.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:36 smithi122 ceph-mon[32255]: pgmap v19139: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3619730652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:36 smithi022 ceph-mon[31360]: pgmap v19139: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3619730652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:36 smithi022 ceph-mon[40250]: pgmap v19139: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3619730652' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:36:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:36:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:38 smithi122 ceph-mon[32255]: pgmap v19140: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:38 smithi022 ceph-mon[31360]: pgmap v19140: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:38 smithi022 ceph-mon[40250]: pgmap v19140: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:36:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:40] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:36:41.142 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:41.143 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:40 smithi022 ceph-mon[31360]: pgmap v19141: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:41.143 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:40 smithi022 ceph-mon[40250]: pgmap v19141: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:40 smithi122 ceph-mon[32255]: pgmap v19141: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:36:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:36:42.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:36:42.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:36:42.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1517436192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:42.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:36:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:36:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:36:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:36:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1517436192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:36:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:36:42.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:36:42.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:36:42.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1517436192' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:36:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:36:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:36:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:42 smithi122 ceph-mon[32255]: pgmap v19142: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:42 smithi022 ceph-mon[31360]: pgmap v19142: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:42 smithi022 ceph-mon[40250]: pgmap v19142: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:45.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:44 smithi122 ceph-mon[32255]: pgmap v19143: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:45.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:36:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:36:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:44 smithi022 ceph-mon[31360]: pgmap v19143: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:44 smithi022 ceph-mon[40250]: pgmap v19143: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:46.506 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:46.874 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:46.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:46.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:46 smithi122 ceph-mon[32255]: pgmap v19144: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:46 smithi022 ceph-mon[31360]: pgmap v19144: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:47.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:46 smithi022 ceph-mon[40250]: pgmap v19144: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:36:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:36:48.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1143235408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:48.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1143235408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:48.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1143235408' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:49.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:48 smithi122 ceph-mon[32255]: pgmap v19145: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:49.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:48 smithi022 ceph-mon[31360]: pgmap v19145: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:49.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:48 smithi022 ceph-mon[40250]: pgmap v19145: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:36:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:50] "GET /metrics HTTP/1.1" 200 202517 "" "Prometheus/2.18.1" 2021-12-16T04:36:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:50 smithi122 ceph-mon[32255]: pgmap v19146: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:51.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:50 smithi022 ceph-mon[31360]: pgmap v19146: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:51.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:50 smithi022 ceph-mon[40250]: pgmap v19146: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:51.876 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:52.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:36:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:36:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:36:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:52 smithi122 ceph-mon[32255]: pgmap v19147: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/53233220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:52 smithi022 ceph-mon[31360]: pgmap v19147: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/53233220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:52 smithi022 ceph-mon[40250]: pgmap v19147: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/53233220' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:54 smithi122 ceph-mon[32255]: pgmap v19148: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:55.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:36:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:36:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:54 smithi022 ceph-mon[31360]: pgmap v19148: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:54 smithi022 ceph-mon[40250]: pgmap v19148: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:36:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:56.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:36:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:56 smithi122 ceph-mon[32255]: pgmap v19149: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:57.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:36:57.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:56 smithi022 ceph-mon[31360]: pgmap v19149: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:57.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:56 smithi022 ceph-mon[40250]: pgmap v19149: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:57.534 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:36:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:36:57.622 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:36:57.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:36:57.623 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:36:58.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/439826610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:58.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/439826610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:58.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/439826610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:36:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:36:58 smithi122 ceph-mon[32255]: pgmap v19150: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:59.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:36:58 smithi022 ceph-mon[31360]: pgmap v19150: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:36:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:36:58 smithi022 ceph-mon[40250]: pgmap v19150: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:37:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:00] "GET /metrics HTTP/1.1" 200 202517 "" "Prometheus/2.18.1" 2021-12-16T04:37:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:00 smithi122 ceph-mon[32255]: pgmap v19151: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:00 smithi022 ceph-mon[31360]: pgmap v19151: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:00 smithi022 ceph-mon[40250]: pgmap v19151: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:02.624 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:37:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:37:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:37:02.994 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:02.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:02.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:02.996 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:02 smithi022 ceph-mon[40250]: pgmap v19152: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:02.996 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:02 smithi022 ceph-mon[31360]: pgmap v19152: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:02 smithi122 ceph-mon[32255]: pgmap v19152: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:04.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/785197148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:04.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/785197148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:04.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/785197148' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:05.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:37:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:37:05.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:04 smithi122 ceph-mon[32255]: pgmap v19153: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:04 smithi022 ceph-mon[31360]: pgmap v19153: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:04 smithi022 ceph-mon[40250]: pgmap v19153: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:06.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:06.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:06 smithi122 ceph-mon[32255]: pgmap v19154: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:07.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:06 smithi022 ceph-mon[31360]: pgmap v19154: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:06 smithi022 ceph-mon[40250]: pgmap v19154: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:37:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:37:07.996 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:08.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:08.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:08.365 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:08 smithi122 ceph-mon[32255]: pgmap v19155: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2713003981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:09.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:08 smithi022 ceph-mon[31360]: pgmap v19155: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:09.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2713003981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:09.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:08 smithi022 ceph-mon[40250]: pgmap v19155: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:09.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2713003981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:37:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:10] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:37:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:10 smithi122 ceph-mon[32255]: pgmap v19156: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:11.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:10 smithi022 ceph-mon[31360]: pgmap v19156: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:10 smithi022 ceph-mon[40250]: pgmap v19156: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:37:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:37:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:37:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:12 smithi122 ceph-mon[32255]: pgmap v19157: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:12 smithi022 ceph-mon[31360]: pgmap v19157: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:12 smithi022 ceph-mon[40250]: pgmap v19157: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:13.367 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:13.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:13.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:13.736 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:14.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1963798872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:14.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1963798872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:14.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1963798872' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:15.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:37:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:37:15.246 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:14 smithi122 ceph-mon[32255]: pgmap v19158: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:14 smithi022 ceph-mon[31360]: pgmap v19158: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:14 smithi022 ceph-mon[40250]: pgmap v19158: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:16 smithi122 ceph-mon[32255]: pgmap v19159: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:16 smithi022 ceph-mon[31360]: pgmap v19159: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:16 smithi022 ceph-mon[40250]: pgmap v19159: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:37:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:37:18.738 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:19.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:18 smithi022 ceph-mon[31360]: pgmap v19160: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:19.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:18 smithi022 ceph-mon[40250]: pgmap v19160: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:19.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:19.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:19.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:18 smithi122 ceph-mon[32255]: pgmap v19160: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:20.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1265522053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:20.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1265522053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:20.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1265522053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:37:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:20] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:37:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:20 smithi122 ceph-mon[32255]: pgmap v19161: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:20 smithi022 ceph-mon[31360]: pgmap v19161: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:20 smithi022 ceph-mon[40250]: pgmap v19161: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:22.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:21 smithi122 ceph-mon[32255]: pgmap v19162: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:22.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:21 smithi022 ceph-mon[31360]: pgmap v19162: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:22.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:21 smithi022 ceph-mon[40250]: pgmap v19162: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:37:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:37:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:37:24.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:24.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:24.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:24.478 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:25.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:37:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:37:25.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:24 smithi122 ceph-mon[32255]: pgmap v19163: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:25.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2259810306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:24 smithi022 ceph-mon[31360]: pgmap v19163: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2259810306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:24 smithi022 ceph-mon[40250]: pgmap v19163: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2259810306' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:26 smithi122 ceph-mon[32255]: pgmap v19164: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:26 smithi022 ceph-mon[31360]: pgmap v19164: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:26 smithi022 ceph-mon[40250]: pgmap v19164: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:37:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:37:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:28 smithi122 ceph-mon[32255]: pgmap v19165: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:29.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:28 smithi022 ceph-mon[31360]: pgmap v19165: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:29.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:28 smithi022 ceph-mon[40250]: pgmap v19165: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:29.480 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:29.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:29.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:29.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:37:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:30] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:37:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:30 smithi122 ceph-mon[32255]: pgmap v19166: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2596534869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:30 smithi022 ceph-mon[31360]: pgmap v19166: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2596534869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:30 smithi022 ceph-mon[40250]: pgmap v19166: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2596534869' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:37:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:37:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:37:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:32 smithi122 ceph-mon[32255]: pgmap v19167: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:33.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:32 smithi022 ceph-mon[31360]: pgmap v19167: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:32 smithi022 ceph-mon[40250]: pgmap v19167: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:34.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:35.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:37:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:37:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:34 smithi122 ceph-mon[32255]: pgmap v19168: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:37:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:37:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:37:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:37:35.218 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:35.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:35.219 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:35.219 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[31360]: pgmap v19168: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:35.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:37:35.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:37:35.220 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:37:35.221 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:37:35.221 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[40250]: pgmap v19168: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:35.222 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:37:35.222 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:37:35.223 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:37:35.223 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:37:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/817057777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:36.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/817057777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/817057777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:36 smithi122 ceph-mon[32255]: pgmap v19169: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:36 smithi022 ceph-mon[31360]: pgmap v19169: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:36 smithi022 ceph-mon[40250]: pgmap v19169: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:37:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:37:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:38 smithi122 ceph-mon[32255]: pgmap v19170: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:38 smithi022 ceph-mon[31360]: pgmap v19170: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:38 smithi022 ceph-mon[40250]: pgmap v19170: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:40.220 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:37:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:40] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:37:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:40.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:40.590 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:40 smithi122 ceph-mon[32255]: pgmap v19171: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2726104761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:40 smithi022 ceph-mon[31360]: pgmap v19171: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2726104761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:40 smithi022 ceph-mon[40250]: pgmap v19171: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2726104761' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:37:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:37:42.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:37:42.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:41 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:37:42.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:37:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:37:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:37:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:37:42.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:37:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:37:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:37:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:41 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:37:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:37:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:37:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:37:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:42 smithi122 ceph-mon[32255]: pgmap v19172: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:42 smithi022 ceph-mon[31360]: pgmap v19172: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:42 smithi022 ceph-mon[40250]: pgmap v19172: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:45.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:37:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:37:45.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:44 smithi122 ceph-mon[32255]: pgmap v19173: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:44 smithi022 ceph-mon[31360]: pgmap v19173: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:44 smithi022 ceph-mon[40250]: pgmap v19173: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:45.592 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:45.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:45.960 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:45.961 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:46 smithi122 ceph-mon[32255]: pgmap v19174: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3537455715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:46 smithi022 ceph-mon[31360]: pgmap v19174: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3537455715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:47.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:46 smithi022 ceph-mon[40250]: pgmap v19174: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:47.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3537455715' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:37:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:37:49.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:48 smithi122 ceph-mon[32255]: pgmap v19175: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:49.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:48 smithi022 ceph-mon[31360]: pgmap v19175: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:49.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:48 smithi022 ceph-mon[40250]: pgmap v19175: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:37:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:50] "GET /metrics HTTP/1.1" 200 202518 "" "Prometheus/2.18.1" 2021-12-16T04:37:50.962 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:50 smithi122 ceph-mon[32255]: pgmap v19176: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:51.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:50 smithi022 ceph-mon[31360]: pgmap v19176: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:51.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:50 smithi022 ceph-mon[40250]: pgmap v19176: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:51.329 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:51.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:51.330 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:52.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/710662907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:52.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/710662907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:52.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/710662907' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:37:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:37:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:37:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:52 smithi122 ceph-mon[32255]: pgmap v19177: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:52 smithi022 ceph-mon[31360]: pgmap v19177: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:52 smithi022 ceph-mon[40250]: pgmap v19177: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:54 smithi122 ceph-mon[32255]: pgmap v19178: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:55.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:37:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:37:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:54 smithi022 ceph-mon[31360]: pgmap v19178: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:54 smithi022 ceph-mon[40250]: pgmap v19178: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:37:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:56.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:37:56.332 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:37:56.700 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:37:56.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:37:56.701 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:37:57.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:56 smithi022 ceph-mon[31360]: pgmap v19179: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:57.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2963478917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:57.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:56 smithi022 ceph-mon[40250]: pgmap v19179: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:57.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2963478917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:56 smithi122 ceph-mon[32255]: pgmap v19179: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2963478917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:37:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:37:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:37:58.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:37:57 smithi122 ceph-mon[32255]: pgmap v19180: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:58.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:37:57 smithi022 ceph-mon[31360]: pgmap v19180: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:37:58.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:37:57 smithi022 ceph-mon[40250]: pgmap v19180: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:38:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:00] "GET /metrics HTTP/1.1" 200 202518 "" "Prometheus/2.18.1" 2021-12-16T04:38:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:00 smithi122 ceph-mon[32255]: pgmap v19181: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:00 smithi022 ceph-mon[31360]: pgmap v19181: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:00 smithi022 ceph-mon[40250]: pgmap v19181: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:01.702 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:02.076 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:02.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:02.077 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:38:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:38:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:38:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:02 smithi122 ceph-mon[32255]: pgmap v19182: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4092991797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:02 smithi022 ceph-mon[31360]: pgmap v19182: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4092991797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:02 smithi022 ceph-mon[40250]: pgmap v19182: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4092991797' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:05.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:38:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:38:05.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:04 smithi122 ceph-mon[32255]: pgmap v19183: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:04 smithi022 ceph-mon[31360]: pgmap v19183: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:04 smithi022 ceph-mon[40250]: pgmap v19183: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:05.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:06.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:06.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:07.078 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:07.199 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:06 smithi122 ceph-mon[32255]: pgmap v19184: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:07.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:06 smithi022 ceph-mon[31360]: pgmap v19184: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:06 smithi022 ceph-mon[40250]: pgmap v19184: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:07.451 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:38:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:38:08.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2397081906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:08.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2397081906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:08.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2397081906' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:08 smithi122 ceph-mon[32255]: pgmap v19185: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:09.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:08 smithi022 ceph-mon[31360]: pgmap v19185: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:09.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:08 smithi022 ceph-mon[40250]: pgmap v19185: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:38:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:10] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:38:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:10 smithi122 ceph-mon[32255]: pgmap v19186: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:11.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:10 smithi022 ceph-mon[31360]: pgmap v19186: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:10 smithi022 ceph-mon[40250]: pgmap v19186: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:38:12.452 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:38:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:38:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:38:12.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:12.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:12.822 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:12 smithi122 ceph-mon[32255]: pgmap v19187: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1411011222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:12 smithi022 ceph-mon[31360]: pgmap v19187: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:13.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1411011222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:12 smithi022 ceph-mon[40250]: pgmap v19187: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1411011222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:15.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:14 smithi122 ceph-mon[32255]: pgmap v19188: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:15.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:38:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:38:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:14 smithi022 ceph-mon[31360]: pgmap v19188: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:14 smithi022 ceph-mon[40250]: pgmap v19188: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:16 smithi122 ceph-mon[32255]: pgmap v19189: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:16 smithi022 ceph-mon[31360]: pgmap v19189: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:16 smithi022 ceph-mon[40250]: pgmap v19189: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:38:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:38:17.824 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:18.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:18.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:18 smithi122 ceph-mon[32255]: pgmap v19190: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3742663266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:18 smithi022 ceph-mon[31360]: pgmap v19190: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3742663266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:18 smithi022 ceph-mon[40250]: pgmap v19190: 132 pgs: 132 active+clean; 582 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3742663266' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:38:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:20] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:38:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:20 smithi122 ceph-mon[32255]: pgmap v19191: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:20 smithi022 ceph-mon[31360]: pgmap v19191: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:20 smithi022 ceph-mon[40250]: pgmap v19191: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:38:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:38:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:38:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:22 smithi122 ceph-mon[32255]: pgmap v19192: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:23.194 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:23.195 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:22 smithi022 ceph-mon[31360]: pgmap v19192: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:23.195 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:22 smithi022 ceph-mon[40250]: pgmap v19192: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:23.568 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:23.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:23.569 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:23.890 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1486811089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:23.891 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1486811089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:24.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1486811089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:25.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:38:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:38:25.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:24 smithi122 ceph-mon[32255]: pgmap v19193: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:25.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:24 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:24 smithi022 ceph-mon[31360]: pgmap v19193: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:25.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:24 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:24 smithi022 ceph-mon[40250]: pgmap v19193: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:24 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:25 smithi122 ceph-mon[32255]: pgmap v19194: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:25 smithi022 ceph-mon[31360]: pgmap v19194: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:25 smithi022 ceph-mon[40250]: pgmap v19194: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:38:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:38:28.570 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:28.940 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:28.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:28.941 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:28.942 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:28 smithi022 ceph-mon[40250]: pgmap v19195: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:28.942 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:28 smithi022 ceph-mon[31360]: pgmap v19195: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:28 smithi122 ceph-mon[32255]: pgmap v19195: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:38:30.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3801363357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:30.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3801363357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:30.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3801363357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:38:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:30] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:38:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:30 smithi122 ceph-mon[32255]: pgmap v19196: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:30 smithi022 ceph-mon[31360]: pgmap v19196: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:30 smithi022 ceph-mon[40250]: pgmap v19196: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:38:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:38:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:38:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:38:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:32 smithi122 ceph-mon[32255]: pgmap v19197: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:33.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:32 smithi022 ceph-mon[31360]: pgmap v19197: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:32 smithi022 ceph-mon[40250]: pgmap v19197: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:34.313 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:34.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:34.314 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:35.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:38:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:38:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:34 smithi122 ceph-mon[32255]: pgmap v19198: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3154325250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:38:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:38:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:38:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:38:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[31360]: pgmap v19198: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3154325250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:38:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:38:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:38:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:38:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[40250]: pgmap v19198: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3154325250' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:38:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:38:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:38:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:38:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:36 smithi122 ceph-mon[32255]: pgmap v19199: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:36 smithi022 ceph-mon[31360]: pgmap v19199: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:36 smithi022 ceph-mon[40250]: pgmap v19199: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:38:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:38:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:38 smithi122 ceph-mon[32255]: pgmap v19200: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:38 smithi022 ceph-mon[31360]: pgmap v19200: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:38 smithi022 ceph-mon[40250]: pgmap v19200: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:39.315 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:39.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:39.685 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:39.686 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:40.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/328162214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:40.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/328162214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:40.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/328162214' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:38:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:40] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:38:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:40 smithi122 ceph-mon[32255]: pgmap v19201: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:40 smithi022 ceph-mon[31360]: pgmap v19201: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:40 smithi022 ceph-mon[40250]: pgmap v19201: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:38:42.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:38:42.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:41 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:38:42.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:38:42.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:38:42.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:41 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:38:42.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:38:42.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:38:42.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:41 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:38:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:38:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:38:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:38:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:42 smithi122 ceph-mon[32255]: pgmap v19202: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:42 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:38:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:42 smithi022 ceph-mon[31360]: pgmap v19202: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:42 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:38:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:42 smithi022 ceph-mon[40250]: pgmap v19202: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:43.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:42 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:38:44.688 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:45.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:44 smithi022 ceph-mon[31360]: pgmap v19203: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:45.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:44 smithi022 ceph-mon[40250]: pgmap v19203: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:45.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:45.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:45.060 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:45.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:44 smithi122 ceph-mon[32255]: pgmap v19203: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:45.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:38:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:38:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/313853693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:46.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/313853693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/313853693' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:46 smithi122 ceph-mon[32255]: pgmap v19204: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:46 smithi022 ceph-mon[31360]: pgmap v19204: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:47.350 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:46 smithi022 ceph-mon[40250]: pgmap v19204: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:38:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:38:49.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:48 smithi122 ceph-mon[32255]: pgmap v19205: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:49.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:48 smithi022 ceph-mon[31360]: pgmap v19205: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:49.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:48 smithi022 ceph-mon[40250]: pgmap v19205: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:50.062 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:50.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:50.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:50.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:38:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:50] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:38:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:50 smithi122 ceph-mon[32255]: pgmap v19206: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2051420958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:51.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:50 smithi022 ceph-mon[31360]: pgmap v19206: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:51.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2051420958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:51.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:50 smithi022 ceph-mon[40250]: pgmap v19206: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:51.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2051420958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:38:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:38:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:38:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:52 smithi122 ceph-mon[32255]: pgmap v19207: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:52 smithi022 ceph-mon[31360]: pgmap v19207: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:52 smithi022 ceph-mon[40250]: pgmap v19207: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:54 smithi122 ceph-mon[32255]: pgmap v19208: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:55.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:38:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:38:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:54 smithi022 ceph-mon[31360]: pgmap v19208: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:54 smithi022 ceph-mon[40250]: pgmap v19208: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:38:55.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:38:55.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:38:55.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:38:55.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:38:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3780069402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:56.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:56.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3780069402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:38:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3780069402' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:38:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:56 smithi122 ceph-mon[32255]: pgmap v19209: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:56 smithi022 ceph-mon[31360]: pgmap v19209: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:56 smithi022 ceph-mon[40250]: pgmap v19209: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:38:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:38:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:38:58 smithi122 ceph-mon[32255]: pgmap v19210: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:59.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:38:58 smithi022 ceph-mon[31360]: pgmap v19210: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:38:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:38:58 smithi022 ceph-mon[40250]: pgmap v19210: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:39:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:00] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:39:00.806 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:01.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:01.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:01.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:01.177 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:00 smithi022 ceph-mon[31360]: pgmap v19211: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:01.177 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:00 smithi022 ceph-mon[40250]: pgmap v19211: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:00 smithi122 ceph-mon[32255]: pgmap v19211: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:02.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1748141870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:02.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1748141870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:02.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1748141870' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:39:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:39:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:39:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:02 smithi122 ceph-mon[32255]: pgmap v19212: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:02 smithi022 ceph-mon[31360]: pgmap v19212: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:02 smithi022 ceph-mon[40250]: pgmap v19212: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:05.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:04 smithi122 ceph-mon[32255]: pgmap v19213: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:05.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:39:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:39:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:04 smithi022 ceph-mon[31360]: pgmap v19213: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:04 smithi022 ceph-mon[40250]: pgmap v19213: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:06.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:06.178 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:06.179 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:06.544 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:06.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:06.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:06 smithi122 ceph-mon[32255]: pgmap v19214: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:07.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3819972601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:06 smithi022 ceph-mon[31360]: pgmap v19214: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3819972601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:06 smithi022 ceph-mon[40250]: pgmap v19214: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3819972601' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:39:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:39:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:08 smithi122 ceph-mon[32255]: pgmap v19215: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:09.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:08 smithi022 ceph-mon[31360]: pgmap v19215: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:09.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:08 smithi022 ceph-mon[40250]: pgmap v19215: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:39:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:10] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:39:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:10 smithi122 ceph-mon[32255]: pgmap v19216: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:10 smithi022 ceph-mon[31360]: pgmap v19216: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:10 smithi022 ceph-mon[40250]: pgmap v19216: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:11.546 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:11.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:11.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:11.918 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:39:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:39:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:39:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:12 smithi122 ceph-mon[32255]: pgmap v19217: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/911567457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:12 smithi022 ceph-mon[31360]: pgmap v19217: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:13.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/911567457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:12 smithi022 ceph-mon[40250]: pgmap v19217: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/911567457' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:15.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:14 smithi122 ceph-mon[32255]: pgmap v19218: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:15.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:39:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:39:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:14 smithi022 ceph-mon[31360]: pgmap v19218: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:14 smithi022 ceph-mon[40250]: pgmap v19218: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:16.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:16.921 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:16 smithi122 ceph-mon[32255]: pgmap v19219: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:16 smithi022 ceph-mon[31360]: pgmap v19219: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:16 smithi022 ceph-mon[40250]: pgmap v19219: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:17.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:17.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:17.289 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:39:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:39:18.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/778585284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:18.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/778585284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:18.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/778585284' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:18 smithi122 ceph-mon[32255]: pgmap v19220: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:19.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:18 smithi022 ceph-mon[31360]: pgmap v19220: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:18 smithi022 ceph-mon[40250]: pgmap v19220: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:20.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:19 smithi122 ceph-mon[32255]: pgmap v19221: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:20.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:19 smithi022 ceph-mon[31360]: pgmap v19221: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:20.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:19 smithi022 ceph-mon[40250]: pgmap v19221: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:20.443 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:39:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:20] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:39:22.291 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:22.658 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:22.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:22.659 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:39:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:39:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:39:23.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:22 smithi022 ceph-mon[31360]: pgmap v19222: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:23.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1383489018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:23.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:22 smithi022 ceph-mon[40250]: pgmap v19222: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:23.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1383489018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:22 smithi122 ceph-mon[32255]: pgmap v19222: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1383489018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:25.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:24 smithi122 ceph-mon[32255]: pgmap v19223: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:25.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:39:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:39:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:24 smithi022 ceph-mon[31360]: pgmap v19223: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:24 smithi022 ceph-mon[40250]: pgmap v19223: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:26.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:26 smithi122 ceph-mon[32255]: pgmap v19224: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:26 smithi022 ceph-mon[31360]: pgmap v19224: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:27.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:26 smithi022 ceph-mon[40250]: pgmap v19224: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:27.660 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:27.660 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:39:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:39:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:28 smithi122 ceph-mon[32255]: pgmap v19225: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:29.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2225128679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:29.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:28 smithi022 ceph-mon[40250]: pgmap v19225: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:29.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2225128679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:28 smithi022 ceph-mon[31360]: pgmap v19225: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2225128679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:39:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:30] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:39:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:30 smithi122 ceph-mon[32255]: pgmap v19226: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:30 smithi022 ceph-mon[31360]: pgmap v19226: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:30 smithi022 ceph-mon[40250]: pgmap v19226: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:32.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:39:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:39:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:39:33.030 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:33.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:32 smithi122 ceph-mon[32255]: pgmap v19227: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:33.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:32 smithi022 ceph-mon[31360]: pgmap v19227: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:32 smithi022 ceph-mon[40250]: pgmap v19227: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:34.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1372502997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:34.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1372502997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1372502997' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:34 smithi122 ceph-mon[32255]: pgmap v19228: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:35.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:39:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:39:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:39:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:39:35.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:39:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:39:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[31360]: pgmap v19228: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:39:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:39:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:39:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:39:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[40250]: pgmap v19228: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:39:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:39:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:39:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:39:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:36 smithi122 ceph-mon[32255]: pgmap v19229: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:36 smithi022 ceph-mon[31360]: pgmap v19229: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:37.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:36 smithi022 ceph-mon[40250]: pgmap v19229: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:37.786 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:39:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:39:38.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:38.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:38.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:38.775 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:39.039 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:38 smithi022 ceph-mon[31360]: pgmap v19230: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:39.040 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1007598305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:39.040 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:38 smithi022 ceph-mon[40250]: pgmap v19230: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:39.041 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1007598305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:38 smithi122 ceph-mon[32255]: pgmap v19230: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:39.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1007598305' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:39:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:40] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:39:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:40 smithi122 ceph-mon[32255]: pgmap v19231: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:40 smithi022 ceph-mon[31360]: pgmap v19231: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:41.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:40 smithi022 ceph-mon[40250]: pgmap v19231: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:39:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:39:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:39:43.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:42 smithi122 ceph-mon[32255]: pgmap v19232: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:43.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:39:43.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:39:43.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:42 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:39:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[31360]: pgmap v19232: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:39:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:39:43.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:39:43.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[40250]: pgmap v19232: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:43.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:39:43.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:39:43.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:42 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:39:43.774 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:44.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:44.143 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:44.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:45.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:39:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:39:45.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:44 smithi122 ceph-mon[32255]: pgmap v19233: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:45.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2143584227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:44 smithi022 ceph-mon[31360]: pgmap v19233: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:45.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2143584227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:44 smithi022 ceph-mon[40250]: pgmap v19233: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2143584227' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:46.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:45 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:46.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:45 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:45 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:47.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:46 smithi022 ceph-mon[31360]: pgmap v19234: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:47.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:47.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:46 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:47.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:46 smithi022 ceph-mon[40250]: pgmap v19234: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:47.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:47.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:46 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:47.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:46 smithi122 ceph-mon[32255]: pgmap v19234: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:47.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:47.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:46 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:47.534 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:39:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:39:48.509 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:48 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:48.510 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:48 smithi022 ceph-mon[40250]: pgmap v19235: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:48.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:48 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:48.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:48 smithi022 ceph-mon[31360]: pgmap v19235: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:48.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:48 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:48.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:48 smithi122 ceph-mon[32255]: pgmap v19235: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:49.145 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:50.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:39:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:50] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:39:50.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1624809450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:50.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:50 smithi122 ceph-mon[32255]: pgmap v19236: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:50.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1624809450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:50.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[31360]: pgmap v19236: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:50.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1624809450' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:50.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[40250]: pgmap v19236: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:50.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:50.787 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:39:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:39:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:39:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:39:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:52 smithi122 ceph-mon[32255]: pgmap v19237: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:52 smithi022 ceph-mon[31360]: pgmap v19237: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:52 smithi022 ceph-mon[40250]: pgmap v19237: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:54.511 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:39:54.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:39:54.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:39:54.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:39:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:54 smithi122 ceph-mon[32255]: pgmap v19238: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:55.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:39:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:39:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:54 smithi022 ceph-mon[31360]: pgmap v19238: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:54 smithi022 ceph-mon[40250]: pgmap v19238: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:39:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4184355460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:56.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:55 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:56.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4184355460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:56.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:55 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:56.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4184355460' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:39:56.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:55 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:39:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:56 smithi122 ceph-mon[32255]: pgmap v19239: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:56 smithi022 ceph-mon[31360]: pgmap v19239: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:56 smithi022 ceph-mon[40250]: pgmap v19239: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:39:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:39:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:39:58 smithi122 ceph-mon[32255]: pgmap v19240: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:39:58 smithi022 ceph-mon[31360]: pgmap v19240: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:39:58 smithi022 ceph-mon[40250]: pgmap v19240: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:39:59.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:00.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:00.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:00.269 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:40:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:00] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:40:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:00 smithi122 ceph-mon[32255]: pgmap v19241: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:01.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T04:40:01.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/691576434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:00 smithi022 ceph-mon[31360]: pgmap v19241: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T04:40:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/691576434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:01.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:00 smithi022 ceph-mon[40250]: pgmap v19241: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:01.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T04:40:01.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/691576434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:02.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:40:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:40:02.319Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:40:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:02 smithi122 ceph-mon[32255]: pgmap v19242: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:03.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:02 smithi022 ceph-mon[40250]: pgmap v19242: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:02 smithi022 ceph-mon[31360]: pgmap v19242: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:05.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:04 smithi122 ceph-mon[32255]: pgmap v19243: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:05.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:40:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:40:05.272 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:04 smithi022 ceph-mon[31360]: pgmap v19243: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:05.272 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:04 smithi022 ceph-mon[40250]: pgmap v19243: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:05.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:05.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:05.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:05.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:06.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:06.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2443439804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:06.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:06.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2443439804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:06.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2443439804' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:06 smithi122 ceph-mon[32255]: pgmap v19244: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:07.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:06 smithi022 ceph-mon[31360]: pgmap v19244: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:06 smithi022 ceph-mon[40250]: pgmap v19244: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:40:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:40:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:08 smithi122 ceph-mon[32255]: pgmap v19245: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:09.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:08 smithi022 ceph-mon[31360]: pgmap v19245: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:09.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:08 smithi022 ceph-mon[40250]: pgmap v19245: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:40:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:10] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:40:10.645 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:11.016 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:10 smithi022 ceph-mon[31360]: pgmap v19246: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:11.016 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:10 smithi022 ceph-mon[40250]: pgmap v19246: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:10 smithi122 ceph-mon[32255]: pgmap v19246: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:12.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2965929816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:12.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2965929816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:12.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2965929816' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:40:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:40:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:40:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:12 smithi122 ceph-mon[32255]: pgmap v19247: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:12 smithi022 ceph-mon[31360]: pgmap v19247: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:12 smithi022 ceph-mon[40250]: pgmap v19247: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:15.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:14 smithi122 ceph-mon[32255]: pgmap v19248: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:15.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:40:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:40:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:14 smithi022 ceph-mon[31360]: pgmap v19248: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:14 smithi022 ceph-mon[40250]: pgmap v19248: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:16.017 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:16.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:16.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:16.389 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:16 smithi122 ceph-mon[32255]: pgmap v19249: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:17.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/980468624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:16 smithi022 ceph-mon[31360]: pgmap v19249: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/980468624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:16 smithi022 ceph-mon[40250]: pgmap v19249: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/980468624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:40:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:40:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:18 smithi122 ceph-mon[32255]: pgmap v19250: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:19.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:18 smithi022 ceph-mon[31360]: pgmap v19250: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:18 smithi022 ceph-mon[40250]: pgmap v19250: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:20.178 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:19 smithi122 ceph-mon[32255]: pgmap v19251: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:20.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:19 smithi022 ceph-mon[31360]: pgmap v19251: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:20.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:19 smithi022 ceph-mon[40250]: pgmap v19251: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:40:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:20] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:40:21.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:21.759 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:21.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:21.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:22.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2263830881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:22.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2263830881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:22.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2263830881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:40:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:40:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:40:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:22 smithi122 ceph-mon[32255]: pgmap v19252: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:22 smithi022 ceph-mon[31360]: pgmap v19252: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:22 smithi022 ceph-mon[40250]: pgmap v19252: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:25.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:24 smithi122 ceph-mon[32255]: pgmap v19253: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:25.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:40:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:40:25.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:24 smithi022 ceph-mon[31360]: pgmap v19253: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:24 smithi022 ceph-mon[40250]: pgmap v19253: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:26.761 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:27.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:26 smithi022 ceph-mon[31360]: pgmap v19254: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:27.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:26 smithi022 ceph-mon[40250]: pgmap v19254: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:27.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:27.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:27.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:26 smithi122 ceph-mon[32255]: pgmap v19254: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:40:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:40:28.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3114928779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:28.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3114928779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:28.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3114928779' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:28 smithi122 ceph-mon[32255]: pgmap v19255: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:28 smithi022 ceph-mon[31360]: pgmap v19255: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:29.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:28 smithi022 ceph-mon[40250]: pgmap v19255: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:40:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:30] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:40:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:30 smithi122 ceph-mon[32255]: pgmap v19256: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:30 smithi022 ceph-mon[31360]: pgmap v19256: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:30 smithi022 ceph-mon[40250]: pgmap v19256: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:32.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:32.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:32.500 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:32.501 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:40:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:40:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:40:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:32 smithi122 ceph-mon[32255]: pgmap v19257: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:33.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/575452415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:33.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:32 smithi022 ceph-mon[31360]: pgmap v19257: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/575452415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:32 smithi022 ceph-mon[40250]: pgmap v19257: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/575452415' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:35.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:40:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:40:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:34 smithi122 ceph-mon[32255]: pgmap v19258: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:40:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:40:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:40:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:40:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[31360]: pgmap v19258: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:40:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:40:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:40:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:40:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[40250]: pgmap v19258: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:40:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:40:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:40:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:40:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:35 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:36.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:35 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:35 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:36 smithi122 ceph-mon[32255]: pgmap v19259: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:36 smithi022 ceph-mon[31360]: pgmap v19259: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:36 smithi022 ceph-mon[40250]: pgmap v19259: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:40:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:40:37.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:37.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:37.871 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:38.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3736432819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:38.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3736432819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:38.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3736432819' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:38 smithi122 ceph-mon[32255]: pgmap v19260: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:38 smithi022 ceph-mon[31360]: pgmap v19260: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:38 smithi022 ceph-mon[40250]: pgmap v19260: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:40:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:40] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:40:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:40 smithi122 ceph-mon[32255]: pgmap v19261: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:40 smithi022 ceph-mon[31360]: pgmap v19261: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:40 smithi022 ceph-mon[40250]: pgmap v19261: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:40:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:40:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:40:42.873 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:42 smithi122 ceph-mon[32255]: pgmap v19262: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:43.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:43.243 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:43.244 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:43.244 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:42 smithi022 ceph-mon[31360]: pgmap v19262: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:43.244 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:42 smithi022 ceph-mon[40250]: pgmap v19262: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:44.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3276089897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:44.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3276089897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:44.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3276089897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:45.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:44 smithi122 ceph-mon[32255]: pgmap v19263: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:45.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:40:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:40:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:44 smithi022 ceph-mon[31360]: pgmap v19263: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:44 smithi022 ceph-mon[40250]: pgmap v19263: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:46.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:46.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:45 smithi022 ceph-mon[31360]: pgmap v19264: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:46.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:45 smithi022 ceph-mon[40250]: pgmap v19264: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:46.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:46.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:45 smithi122 ceph-mon[32255]: pgmap v19264: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:40:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:40:48.245 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:48.613 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:48.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:48.614 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:49.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:48 smithi022 ceph-mon[31360]: pgmap v19265: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:49.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/892620274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:49.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:48 smithi022 ceph-mon[40250]: pgmap v19265: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:49.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/892620274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:49.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:48 smithi122 ceph-mon[32255]: pgmap v19265: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:49.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/892620274' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:40:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:50] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:40:50.941 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:50 smithi122 ceph-mon[32255]: pgmap v19266: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:50.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:40:50.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:40:50.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:40:50.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:50 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:40:51.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[31360]: pgmap v19266: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:51.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:40:51.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:40:51.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:40:51.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:40:51.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[40250]: pgmap v19266: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:51.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:40:51.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:40:51.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:40:51.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:50 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:40:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:40:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:40:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:40:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:52 smithi122 ceph-mon[32255]: pgmap v19267: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:52 smithi022 ceph-mon[31360]: pgmap v19267: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:52 smithi022 ceph-mon[40250]: pgmap v19267: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:53.616 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:53.985 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:53.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:53.986 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:40:55.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:40:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:40:55.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:54 smithi122 ceph-mon[32255]: pgmap v19268: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:55.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/353995188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:54 smithi022 ceph-mon[31360]: pgmap v19268: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/353995188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:54 smithi022 ceph-mon[40250]: pgmap v19268: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:40:55.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/353995188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:40:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:56 smithi122 ceph-mon[32255]: pgmap v19269: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:56 smithi022 ceph-mon[31360]: pgmap v19269: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:40:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:56 smithi022 ceph-mon[40250]: pgmap v19269: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:57.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:40:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:40:58.988 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:40:59.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:58 smithi122 ceph-mon[32255]: pgmap v19270: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:58 smithi022 ceph-mon[31360]: pgmap v19270: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:58 smithi022 ceph-mon[40250]: pgmap v19270: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:40:59.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:40:59.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:40:59.361 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:00.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:40:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/638430040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:00.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:40:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/638430040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:00.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:40:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/638430040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:41:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:00] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:41:01.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:00 smithi122 ceph-mon[32255]: pgmap v19271: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:00 smithi022 ceph-mon[31360]: pgmap v19271: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:00 smithi022 ceph-mon[40250]: pgmap v19271: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:41:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:41:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:41:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:02 smithi122 ceph-mon[32255]: pgmap v19272: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:02 smithi022 ceph-mon[31360]: pgmap v19272: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:02 smithi022 ceph-mon[40250]: pgmap v19272: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:04.362 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:04.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:04.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:04.735 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:05.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:04 smithi022 ceph-mon[31360]: pgmap v19273: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:05.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1130928672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:05.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:04 smithi022 ceph-mon[40250]: pgmap v19273: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:05.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1130928672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:05.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:04 smithi122 ceph-mon[32255]: pgmap v19273: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:05.194 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1130928672' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:05.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:41:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:41:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:07.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:06 smithi122 ceph-mon[32255]: pgmap v19274: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:06 smithi022 ceph-mon[31360]: pgmap v19274: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:06 smithi022 ceph-mon[40250]: pgmap v19274: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:41:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:41:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:08 smithi122 ceph-mon[32255]: pgmap v19275: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:09.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:08 smithi022 ceph-mon[31360]: pgmap v19275: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:09.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:08 smithi022 ceph-mon[40250]: pgmap v19275: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:09.737 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:10.107 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:10.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:10.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:41:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:10] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:41:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:10 smithi122 ceph-mon[32255]: pgmap v19276: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:11.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1285310347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:10 smithi022 ceph-mon[31360]: pgmap v19276: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1285310347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:10 smithi022 ceph-mon[40250]: pgmap v19276: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1285310347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:41:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:41:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:41:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:12 smithi122 ceph-mon[32255]: pgmap v19277: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:13.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:12 smithi022 ceph-mon[31360]: pgmap v19277: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:12 smithi022 ceph-mon[40250]: pgmap v19277: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:15.109 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:15.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:14 smithi122 ceph-mon[32255]: pgmap v19278: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:15.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:41:14 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:41:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:14 smithi022 ceph-mon[31360]: pgmap v19278: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:14 smithi022 ceph-mon[40250]: pgmap v19278: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:15.480 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:15.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:15.481 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:16.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/724812528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:16.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/724812528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:16.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/724812528' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:17.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:16 smithi122 ceph-mon[32255]: pgmap v19279: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:16 smithi022 ceph-mon[31360]: pgmap v19279: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:16 smithi022 ceph-mon[40250]: pgmap v19279: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:17.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:41:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:41:18.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:17 smithi122 ceph-mon[32255]: pgmap v19280: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:18.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:17 smithi022 ceph-mon[31360]: pgmap v19280: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:18.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:17 smithi022 ceph-mon[40250]: pgmap v19280: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:41:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:20] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:41:20.482 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:20.850 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:20 smithi122 ceph-mon[32255]: pgmap v19281: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:21.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/271426777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:20 smithi022 ceph-mon[31360]: pgmap v19281: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/271426777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:20 smithi022 ceph-mon[40250]: pgmap v19281: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:21.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/271426777' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:41:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:41:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:41:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:22 smithi122 ceph-mon[32255]: pgmap v19282: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:22 smithi022 ceph-mon[31360]: pgmap v19282: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:22 smithi022 ceph-mon[40250]: pgmap v19282: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:25.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:24 smithi122 ceph-mon[32255]: pgmap v19283: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:25.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:41:24 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:41:25.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:24 smithi022 ceph-mon[31360]: pgmap v19283: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:25.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:24 smithi022 ceph-mon[40250]: pgmap v19283: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:26.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:26.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:26.222 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:27.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:26 smithi122 ceph-mon[32255]: pgmap v19284: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:27.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3400116035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:27.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:26 smithi022 ceph-mon[31360]: pgmap v19284: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:27.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3400116035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:27.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:27.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:26 smithi022 ceph-mon[40250]: pgmap v19284: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:27.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3400116035' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:41:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:41:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:28 smithi122 ceph-mon[32255]: pgmap v19285: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:28 smithi022 ceph-mon[31360]: pgmap v19285: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:29.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:28 smithi022 ceph-mon[40250]: pgmap v19285: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:41:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:30] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:41:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:30 smithi122 ceph-mon[32255]: pgmap v19286: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:31.256 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:31.256 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:30 smithi022 ceph-mon[31360]: pgmap v19286: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:31.257 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:30 smithi022 ceph-mon[40250]: pgmap v19286: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:32.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3936419945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:32.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3936419945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:32.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3936419945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:41:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:41:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:41:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:32 smithi122 ceph-mon[32255]: pgmap v19287: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:32 smithi022 ceph-mon[31360]: pgmap v19287: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:32 smithi022 ceph-mon[40250]: pgmap v19287: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:35.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:41:34 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:41:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:34 smithi122 ceph-mon[32255]: pgmap v19288: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:41:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:41:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:41:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:41:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[31360]: pgmap v19288: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:41:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:41:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:41:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:41:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[40250]: pgmap v19288: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:41:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:41:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:41:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:41:36.594 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:37.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:36 smithi122 ceph-mon[32255]: pgmap v19289: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:37.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:36 smithi022 ceph-mon[31360]: pgmap v19289: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:36 smithi022 ceph-mon[40250]: pgmap v19289: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:37.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:41:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:41:38.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/362804786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:38.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/362804786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:38.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/362804786' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:38 smithi122 ceph-mon[32255]: pgmap v19290: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:38 smithi022 ceph-mon[40250]: pgmap v19290: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:39.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:38 smithi022 ceph-mon[31360]: pgmap v19290: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:41:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:40] "GET /metrics HTTP/1.1" 200 202501 "" "Prometheus/2.18.1" 2021-12-16T04:41:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:40 smithi122 ceph-mon[32255]: pgmap v19291: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:40 smithi022 ceph-mon[31360]: pgmap v19291: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:40 smithi022 ceph-mon[40250]: pgmap v19291: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:41.965 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:42.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:42.333 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:42.334 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:41:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:41:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:41:43.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:42 smithi122 ceph-mon[32255]: pgmap v19292: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:43.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2847809195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:42 smithi022 ceph-mon[31360]: pgmap v19292: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:43.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2847809195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:43.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:42 smithi022 ceph-mon[40250]: pgmap v19292: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:43.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2847809195' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:45.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:41:44 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:41:45.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:44 smithi122 ceph-mon[32255]: pgmap v19293: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:45.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:44 smithi022 ceph-mon[31360]: pgmap v19293: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:44 smithi022 ceph-mon[40250]: pgmap v19293: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:46 smithi122 ceph-mon[32255]: pgmap v19294: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:46 smithi022 ceph-mon[31360]: pgmap v19294: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:47.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:46 smithi022 ceph-mon[40250]: pgmap v19294: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:47.335 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:47.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:47.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:47.705 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:47.706 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:41:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:41:48.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1875817026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:48.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:47 smithi022 ceph-mon[31360]: pgmap v19295: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:48.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1875817026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:48.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:47 smithi022 ceph-mon[40250]: pgmap v19295: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:48.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1875817026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:48.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:47 smithi122 ceph-mon[32255]: pgmap v19295: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:41:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:50] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:41:51.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[31360]: pgmap v19296: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:51.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:41:51.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:41:51.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:41:51.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[40250]: pgmap v19296: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:51.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:41:51.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:41:51.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:50 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:41:51.093 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:50 smithi122 ceph-mon[32255]: pgmap v19296: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:51.093 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:41:51.093 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:41:51.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:50 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:41:52.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:41:52.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:52 smithi122 ceph-mon[32255]: pgmap v19297: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:52.442 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:41:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:41:52.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:41:52.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:41:52.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:52 smithi022 ceph-mon[31360]: pgmap v19297: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:52.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:41:52.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:52 smithi022 ceph-mon[40250]: pgmap v19297: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:52.708 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:53.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1080187669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:53.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1080187669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:53.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1080187669' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:54 smithi122 ceph-mon[32255]: pgmap v19298: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:55.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:41:54 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:41:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:54 smithi022 ceph-mon[31360]: pgmap v19298: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:54 smithi022 ceph-mon[40250]: pgmap v19298: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:41:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:56 smithi122 ceph-mon[32255]: pgmap v19299: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:56 smithi022 ceph-mon[31360]: pgmap v19299: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:41:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:56 smithi022 ceph-mon[40250]: pgmap v19299: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:57.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:41:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:41:58.081 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:41:58.449 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:41:58.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:41:58.450 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:41:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:58 smithi122 ceph-mon[32255]: pgmap v19300: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:59.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:41:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3346661507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:58 smithi022 ceph-mon[31360]: pgmap v19300: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:41:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3346661507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:41:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:58 smithi022 ceph-mon[40250]: pgmap v19300: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:41:59.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:41:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3346661507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:42:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:00] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:42:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:00 smithi122 ceph-mon[32255]: pgmap v19301: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:00 smithi022 ceph-mon[31360]: pgmap v19301: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:00 smithi022 ceph-mon[40250]: pgmap v19301: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:42:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:42:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:42:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:02 smithi122 ceph-mon[32255]: pgmap v19302: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:02 smithi022 ceph-mon[31360]: pgmap v19302: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:02 smithi022 ceph-mon[40250]: pgmap v19302: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:03.451 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:03.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:03.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:03.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:04.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/489693558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:04.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/489693558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:04.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/489693558' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:05.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:42:04 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:42:05.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:04 smithi122 ceph-mon[32255]: pgmap v19303: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:04 smithi022 ceph-mon[31360]: pgmap v19303: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:04 smithi022 ceph-mon[40250]: pgmap v19303: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:07.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:06 smithi122 ceph-mon[32255]: pgmap v19304: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:06 smithi022 ceph-mon[31360]: pgmap v19304: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:06 smithi022 ceph-mon[40250]: pgmap v19304: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:42:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:08 smithi122 ceph-mon[32255]: pgmap v19305: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:09.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:09.192 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:09.193 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:09.193 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:08 smithi022 ceph-mon[31360]: pgmap v19305: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:09.194 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:08 smithi022 ceph-mon[40250]: pgmap v19305: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:10.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/562409881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:10.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/562409881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:10.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/562409881' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:42:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:10] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:42:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:10 smithi122 ceph-mon[32255]: pgmap v19306: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:10 smithi022 ceph-mon[31360]: pgmap v19306: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:10 smithi022 ceph-mon[40250]: pgmap v19306: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:42:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:42:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:42:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:12 smithi122 ceph-mon[32255]: pgmap v19307: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:13.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:12 smithi022 ceph-mon[31360]: pgmap v19307: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:12 smithi022 ceph-mon[40250]: pgmap v19307: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:14.194 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:14.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:14.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:14.560 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:15.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:42:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:42:15.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:14 smithi122 ceph-mon[32255]: pgmap v19308: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:15.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4153300216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:14 smithi022 ceph-mon[31360]: pgmap v19308: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4153300216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:14 smithi022 ceph-mon[40250]: pgmap v19308: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:15.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4153300216' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:17.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:17.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:16 smithi122 ceph-mon[32255]: pgmap v19309: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:16 smithi022 ceph-mon[31360]: pgmap v19309: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:17.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:16 smithi022 ceph-mon[40250]: pgmap v19309: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:17.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:42:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:42:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:18 smithi122 ceph-mon[32255]: pgmap v19310: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:18 smithi022 ceph-mon[31360]: pgmap v19310: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:18 smithi022 ceph-mon[40250]: pgmap v19310: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:19.562 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:19.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:19.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:19.931 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:19.932 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1197721322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:20.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1197721322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:20.192 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:42:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:20] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:42:20.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1197721322' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:20 smithi122 ceph-mon[32255]: pgmap v19311: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:20 smithi022 ceph-mon[31360]: pgmap v19311: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:20 smithi022 ceph-mon[40250]: pgmap v19311: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:22.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:21 smithi122 ceph-mon[32255]: pgmap v19312: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:22.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:21 smithi022 ceph-mon[31360]: pgmap v19312: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:22.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:21 smithi022 ceph-mon[40250]: pgmap v19312: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:42:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:42:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:42:24.933 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:25.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:42:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:42:25.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:24 smithi122 ceph-mon[32255]: pgmap v19313: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:24 smithi022 ceph-mon[31360]: pgmap v19313: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:24 smithi022 ceph-mon[40250]: pgmap v19313: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:25.305 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:25.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:25.306 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:26.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/511469294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:26.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/511469294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/511469294' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:27.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:26 smithi122 ceph-mon[32255]: pgmap v19314: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:27.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:26 smithi022 ceph-mon[31360]: pgmap v19314: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:27.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:27.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:26 smithi022 ceph-mon[40250]: pgmap v19314: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:27.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:42:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:42:29.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:28 smithi122 ceph-mon[32255]: pgmap v19315: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:28 smithi022 ceph-mon[31360]: pgmap v19315: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:29.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:28 smithi022 ceph-mon[40250]: pgmap v19315: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:30.308 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:42:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:30] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:31.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:30 smithi022 ceph-mon[31360]: pgmap v19316: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:31.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1418688789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:31.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:30 smithi022 ceph-mon[40250]: pgmap v19316: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:31.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1418688789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:30 smithi122 ceph-mon[32255]: pgmap v19316: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:31.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1418688789' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:42:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:42:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:42:33.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:32 smithi122 ceph-mon[32255]: pgmap v19317: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:32 smithi022 ceph-mon[31360]: pgmap v19317: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:32 smithi022 ceph-mon[40250]: pgmap v19317: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:35.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:42:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:42:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:34 smithi122 ceph-mon[32255]: pgmap v19318: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:42:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:42:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:42:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:42:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[31360]: pgmap v19318: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:42:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:42:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:42:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:42:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[40250]: pgmap v19318: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:42:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:42:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:42:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:42:35.678 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:36.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:36.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:36.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:37.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:37.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:36 smithi122 ceph-mon[32255]: pgmap v19319: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:37.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/107845288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:36 smithi022 ceph-mon[31360]: pgmap v19319: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/107845288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:36 smithi022 ceph-mon[40250]: pgmap v19319: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/107845288' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:37.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:42:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:42:39.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:38 smithi122 ceph-mon[32255]: pgmap v19320: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:39.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:38 smithi022 ceph-mon[31360]: pgmap v19320: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:38 smithi022 ceph-mon[40250]: pgmap v19320: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:42:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:40] "GET /metrics HTTP/1.1" 200 202505 "" "Prometheus/2.18.1" 2021-12-16T04:42:41.049 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:41.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:40 smithi122 ceph-mon[32255]: pgmap v19321: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:40 smithi022 ceph-mon[31360]: pgmap v19321: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:40 smithi022 ceph-mon[40250]: pgmap v19321: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:41.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:41.421 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:41.422 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:42.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3966692850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3966692850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:42.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3966692850' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:42:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:42:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:42:43.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:42 smithi122 ceph-mon[32255]: pgmap v19322: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:42 smithi022 ceph-mon[31360]: pgmap v19322: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:43.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:42 smithi022 ceph-mon[40250]: pgmap v19322: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:45.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:44 smithi122 ceph-mon[32255]: pgmap v19323: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:45.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:42:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:42:45.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:44 smithi022 ceph-mon[31360]: pgmap v19323: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:45.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:44 smithi022 ceph-mon[40250]: pgmap v19323: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:46.423 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:46.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:46.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:46.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:46 smithi122 ceph-mon[32255]: pgmap v19324: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1649587432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:46 smithi022 ceph-mon[31360]: pgmap v19324: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:47.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1649587432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:46 smithi022 ceph-mon[40250]: pgmap v19324: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:47.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1649587432' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:42:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:42:48.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:47 smithi022 ceph-mon[31360]: pgmap v19325: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:48.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:47 smithi022 ceph-mon[40250]: pgmap v19325: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:48.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:47 smithi122 ceph-mon[32255]: pgmap v19325: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:50.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:42:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:50] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:42:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:50 smithi122 ceph-mon[32255]: pgmap v19326: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:51.173 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:50 smithi022 ceph-mon[31360]: pgmap v19326: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:51.173 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:50 smithi022 ceph-mon[40250]: pgmap v19326: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:52.155 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:52.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:42:52.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:42:52.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:42:52.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:42:52.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:42:52.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:42:52.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:42:52.158 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:51 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:42:52.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:42:52.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:42:52.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:42:52.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:51 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:42:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:42:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:42:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:42:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:52 smithi122 ceph-mon[32255]: pgmap v19327: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3221739484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:52 smithi022 ceph-mon[31360]: pgmap v19327: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:53.546 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3221739484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:53.547 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:52 smithi022 ceph-mon[40250]: pgmap v19327: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:53.547 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3221739484' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:54 smithi122 ceph-mon[32255]: pgmap v19328: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:55.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:42:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:42:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:54 smithi022 ceph-mon[31360]: pgmap v19328: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:54 smithi022 ceph-mon[40250]: pgmap v19328: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:42:57.157 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:42:57.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:56 smithi122 ceph-mon[32255]: pgmap v19329: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:56 smithi022 ceph-mon[31360]: pgmap v19329: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:42:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:56 smithi022 ceph-mon[40250]: pgmap v19329: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:57.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:42:57.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:42:57.526 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:42:57.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:42:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:42:58.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4089810684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:58.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4089810684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:58.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4089810684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:42:59.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:42:58 smithi122 ceph-mon[32255]: pgmap v19330: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:42:58 smithi022 ceph-mon[31360]: pgmap v19330: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:42:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:42:58 smithi022 ceph-mon[40250]: pgmap v19330: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:43:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:00] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:43:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:00 smithi122 ceph-mon[32255]: pgmap v19331: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:00 smithi022 ceph-mon[31360]: pgmap v19331: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:00 smithi022 ceph-mon[40250]: pgmap v19331: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:02.528 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:43:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:43:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:03.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:02 smithi122 ceph-mon[32255]: pgmap v19332: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:03.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3603925296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:02 smithi022 ceph-mon[40250]: pgmap v19332: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3603925296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:03.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:02 smithi022 ceph-mon[31360]: pgmap v19332: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:03.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3603925296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:05.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:04 smithi122 ceph-mon[32255]: pgmap v19333: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:05.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:43:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:43:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:04 smithi022 ceph-mon[31360]: pgmap v19333: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:05.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:04 smithi022 ceph-mon[40250]: pgmap v19333: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:07.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:07.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:06 smithi122 ceph-mon[32255]: pgmap v19334: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:06 smithi022 ceph-mon[31360]: pgmap v19334: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:06 smithi022 ceph-mon[40250]: pgmap v19334: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:07.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:43:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:43:07.901 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:08.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:08.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:08.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:09.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:08 smithi122 ceph-mon[32255]: pgmap v19335: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:09.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2315188151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:09.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:08 smithi022 ceph-mon[31360]: pgmap v19335: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:09.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2315188151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:09.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:08 smithi022 ceph-mon[40250]: pgmap v19335: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:09.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2315188151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:10.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:43:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:10] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:43:11.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:10 smithi122 ceph-mon[32255]: pgmap v19336: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:10 smithi022 ceph-mon[31360]: pgmap v19336: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:10 smithi022 ceph-mon[40250]: pgmap v19336: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:12.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:43:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:43:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:43:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:12 smithi122 ceph-mon[32255]: pgmap v19337: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:13.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:12 smithi022 ceph-mon[31360]: pgmap v19337: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:13.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:12 smithi022 ceph-mon[40250]: pgmap v19337: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:13.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:13.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:13.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:14.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/123830340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:14.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/123830340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:14.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/123830340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:15.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:14 smithi122 ceph-mon[32255]: pgmap v19338: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:15.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:43:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:43:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:14 smithi022 ceph-mon[31360]: pgmap v19338: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:14 smithi022 ceph-mon[40250]: pgmap v19338: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:16.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:15 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:16.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:15 smithi022 ceph-mon[31360]: pgmap v19339: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:16.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:15 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:16.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:15 smithi022 ceph-mon[40250]: pgmap v19339: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:16.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:15 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:16.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:15 smithi122 ceph-mon[32255]: pgmap v19339: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:17.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:43:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:43:18.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:19.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:19.019 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:19.020 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:18 smithi022 ceph-mon[40250]: pgmap v19340: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:19.020 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:18 smithi022 ceph-mon[31360]: pgmap v19340: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:18 smithi122 ceph-mon[32255]: pgmap v19340: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:20.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/56507926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:20.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/56507926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:20.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/56507926' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:43:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:20] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:43:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:20 smithi122 ceph-mon[32255]: pgmap v19341: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:20 smithi022 ceph-mon[31360]: pgmap v19341: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:20 smithi022 ceph-mon[40250]: pgmap v19341: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:22.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:43:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:43:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:43:23.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:22 smithi122 ceph-mon[32255]: pgmap v19342: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:22 smithi022 ceph-mon[31360]: pgmap v19342: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:22 smithi022 ceph-mon[40250]: pgmap v19342: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:24.020 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:24.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:24.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:24.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:25.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:43:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:43:25.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:24 smithi122 ceph-mon[32255]: pgmap v19343: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:25.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1928114945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:25.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:24 smithi022 ceph-mon[31360]: pgmap v19343: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:25.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1928114945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:25.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:24 smithi022 ceph-mon[40250]: pgmap v19343: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:25.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1928114945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:27.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:27.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:26 smithi122 ceph-mon[32255]: pgmap v19344: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:26 smithi022 ceph-mon[31360]: pgmap v19344: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:27.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:27.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:26 smithi022 ceph-mon[40250]: pgmap v19344: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:27.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:43:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:43:29.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:28 smithi122 ceph-mon[32255]: pgmap v19345: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:28 smithi022 ceph-mon[31360]: pgmap v19345: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:29.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:28 smithi022 ceph-mon[40250]: pgmap v19345: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:29.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:29.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:29.767 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:30.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/525933562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:30.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/525933562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:30.179 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/525933562' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:43:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:30] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:43:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:30 smithi122 ceph-mon[32255]: pgmap v19346: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:30 smithi022 ceph-mon[31360]: pgmap v19346: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:30 smithi022 ceph-mon[40250]: pgmap v19346: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:32.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:43:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:43:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:43:33.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:32 smithi122 ceph-mon[32255]: pgmap v19347: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:32 smithi022 ceph-mon[31360]: pgmap v19347: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:32 smithi022 ceph-mon[40250]: pgmap v19347: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:34.768 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:35.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[31360]: pgmap v19348: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:35.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:43:35.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:43:35.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:43:35.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:43:35.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[40250]: pgmap v19348: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:35.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:43:35.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:43:35.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:43:35.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:43:35.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:35.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:35.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:35.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:43:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:43:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:34 smithi122 ceph-mon[32255]: pgmap v19348: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:43:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:43:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:43:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:43:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2652728206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:36.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2652728206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2652728206' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:37.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:37.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:36 smithi122 ceph-mon[32255]: pgmap v19349: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:37.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:36 smithi022 ceph-mon[31360]: pgmap v19349: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:36 smithi022 ceph-mon[40250]: pgmap v19349: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:37.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:43:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:43:39.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:38 smithi022 ceph-mon[31360]: pgmap v19350: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:39.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:38 smithi022 ceph-mon[40250]: pgmap v19350: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:39.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:38 smithi122 ceph-mon[32255]: pgmap v19350: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:40.141 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:40.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:39 smithi022 ceph-mon[31360]: pgmap v19351: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:40.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:39 smithi022 ceph-mon[40250]: pgmap v19351: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:40.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:39 smithi122 ceph-mon[32255]: pgmap v19351: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:40.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:43:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:40] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:43:40.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:40.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:40.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1269431114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1269431114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:41.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1269431114' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:42.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:41 smithi022 ceph-mon[31360]: pgmap v19352: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:42.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:41 smithi022 ceph-mon[40250]: pgmap v19352: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:42.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:41 smithi122 ceph-mon[32255]: pgmap v19352: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:43:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:43:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:43:45.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:43:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:43:45.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:44 smithi122 ceph-mon[32255]: pgmap v19353: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:45.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:44 smithi022 ceph-mon[31360]: pgmap v19353: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:45.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:44 smithi022 ceph-mon[40250]: pgmap v19353: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:45.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:45.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:45.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:45.883 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:46.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2542801168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:46.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2542801168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:46.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2542801168' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:47.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:47.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:46 smithi122 ceph-mon[32255]: pgmap v19354: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:46 smithi022 ceph-mon[31360]: pgmap v19354: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:46 smithi022 ceph-mon[40250]: pgmap v19354: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:47.786 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:43:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:43:49.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:48 smithi122 ceph-mon[32255]: pgmap v19355: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:49.287 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:48 smithi022 ceph-mon[31360]: pgmap v19355: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:49.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:48 smithi022 ceph-mon[40250]: pgmap v19355: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:50.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:43:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:50] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:43:50.884 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:51.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:50 smithi122 ceph-mon[32255]: pgmap v19356: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:51.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:51.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:51.257 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:51.257 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:50 smithi022 ceph-mon[31360]: pgmap v19356: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:51.258 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:50 smithi022 ceph-mon[40250]: pgmap v19356: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:51.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1002518944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:51.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:43:51.929 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:43:51.930 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:43:52.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1002518944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:52.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:43:52.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:43:52.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:51 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:43:52.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1002518944' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:52.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:43:52.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:43:52.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:51 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:43:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:43:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:43:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:43:53.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:52 smithi122 ceph-mon[32255]: pgmap v19357: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:43:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:52 smithi022 ceph-mon[31360]: pgmap v19357: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:43:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:52 smithi022 ceph-mon[40250]: pgmap v19357: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:53.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:43:55.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:43:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:43:55.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:54 smithi122 ceph-mon[32255]: pgmap v19358: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:54 smithi022 ceph-mon[31360]: pgmap v19358: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:54 smithi022 ceph-mon[40250]: pgmap v19358: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:43:56.259 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:43:56.635 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:43:56.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:43:56.636 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:43:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:56 smithi122 ceph-mon[32255]: pgmap v19359: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4174154831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:56 smithi022 ceph-mon[31360]: pgmap v19359: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4174154831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:43:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:56 smithi022 ceph-mon[40250]: pgmap v19359: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4174154831' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:43:57.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:43:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:43:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:43:59.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:43:58 smithi122 ceph-mon[32255]: pgmap v19360: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:43:58 smithi022 ceph-mon[31360]: pgmap v19360: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:43:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:43:58 smithi022 ceph-mon[40250]: pgmap v19360: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:00.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:44:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:00] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:44:01.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:00 smithi122 ceph-mon[32255]: pgmap v19361: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:00 smithi022 ceph-mon[31360]: pgmap v19361: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:00 smithi022 ceph-mon[40250]: pgmap v19361: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:01.638 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:02.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:02.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:02.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:02.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:44:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:44:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:44:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:02 smithi022 ceph-mon[31360]: pgmap v19362: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1475357902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:03.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:02 smithi022 ceph-mon[40250]: pgmap v19362: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:03.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1475357902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:03.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:02 smithi122 ceph-mon[32255]: pgmap v19362: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:03.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1475357902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:04 smithi022 ceph-mon[31360]: pgmap v19363: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:04 smithi022 ceph-mon[40250]: pgmap v19363: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:05.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:04 smithi122 ceph-mon[32255]: pgmap v19363: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:05.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:44:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:44:06.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:06.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:05 smithi022 ceph-mon[31360]: pgmap v19364: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:06.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:06.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:05 smithi022 ceph-mon[40250]: pgmap v19364: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:06.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:06.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:05 smithi122 ceph-mon[32255]: pgmap v19364: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:07.008 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:07.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:07.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:07.379 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:07.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3622110110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:07.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3622110110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:07.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:44:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:44:07.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3622110110' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:08.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:08 smithi122 ceph-mon[32255]: pgmap v19365: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:08.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:08 smithi022 ceph-mon[31360]: pgmap v19365: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:08.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:08 smithi022 ceph-mon[40250]: pgmap v19365: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:44:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:10] "GET /metrics HTTP/1.1" 200 202505 "" "Prometheus/2.18.1" 2021-12-16T04:44:11.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:10 smithi122 ceph-mon[32255]: pgmap v19366: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:10 smithi022 ceph-mon[31360]: pgmap v19366: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:10 smithi022 ceph-mon[40250]: pgmap v19366: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:12.381 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:44:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:44:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:44:12.753 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:12.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:12.754 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:13.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:12 smithi022 ceph-mon[31360]: pgmap v19367: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:13.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2690337459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:13.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:12 smithi022 ceph-mon[40250]: pgmap v19367: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:13.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2690337459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:13.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:12 smithi122 ceph-mon[32255]: pgmap v19367: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:13.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2690337459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:15.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:14 smithi122 ceph-mon[32255]: pgmap v19368: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:15.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:44:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:44:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:14 smithi022 ceph-mon[31360]: pgmap v19368: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:14 smithi022 ceph-mon[40250]: pgmap v19368: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:17.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:17.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:16 smithi122 ceph-mon[32255]: pgmap v19369: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:16 smithi022 ceph-mon[31360]: pgmap v19369: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:17.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:17.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:16 smithi022 ceph-mon[40250]: pgmap v19369: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:17.755 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:17.756 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:44:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:44:18.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:18.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:18.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:19.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:18 smithi122 ceph-mon[32255]: pgmap v19370: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:19.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/540985056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:18 smithi022 ceph-mon[31360]: pgmap v19370: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/540985056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:18 smithi022 ceph-mon[40250]: pgmap v19370: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:44:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/540985056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:20.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:44:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:20] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:44:21.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:20 smithi122 ceph-mon[32255]: pgmap v19371: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:20 smithi022 ceph-mon[31360]: pgmap v19371: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:20 smithi022 ceph-mon[40250]: pgmap v19371: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:22.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:44:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:44:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:44:23.129 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:22 smithi022 ceph-mon[31360]: pgmap v19372: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:22 smithi022 ceph-mon[40250]: pgmap v19372: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:23.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:22 smithi122 ceph-mon[32255]: pgmap v19372: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:23.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:23.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:23.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:24.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4036549454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:24.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4036549454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:24.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4036549454' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:25.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:24 smithi022 ceph-mon[31360]: pgmap v19373: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:24 smithi022 ceph-mon[40250]: pgmap v19373: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:25.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:24 smithi122 ceph-mon[32255]: pgmap v19373: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:25.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:44:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:44:26.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:26.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:25 smithi022 ceph-mon[31360]: pgmap v19374: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:26.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:26.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:25 smithi022 ceph-mon[40250]: pgmap v19374: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:26.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:26.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:25 smithi122 ceph-mon[32255]: pgmap v19374: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:27.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:44:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:44:28.507 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:28.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:28.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:28.875 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:29.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:28 smithi122 ceph-mon[32255]: pgmap v19375: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:29.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1799826166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:28 smithi022 ceph-mon[31360]: pgmap v19375: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1799826166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:29.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:28 smithi022 ceph-mon[40250]: pgmap v19375: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:29.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1799826166' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:44:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:30] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:44:31.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:30 smithi122 ceph-mon[32255]: pgmap v19376: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:30 smithi022 ceph-mon[31360]: pgmap v19376: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:30 smithi022 ceph-mon[40250]: pgmap v19376: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:32.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:44:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:44:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:44:33.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:32 smithi122 ceph-mon[32255]: pgmap v19377: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:32 smithi022 ceph-mon[31360]: pgmap v19377: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:32 smithi022 ceph-mon[40250]: pgmap v19377: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:33.877 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:34.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:34.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:34.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:35.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:44:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:44:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:34 smithi122 ceph-mon[32255]: pgmap v19378: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:35.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3507913998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:44:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:44:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:44:35.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:44:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[31360]: pgmap v19378: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3507913998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:44:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:44:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:44:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:44:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[40250]: pgmap v19378: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3507913998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:44:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:44:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:44:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:44:37.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:37.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:36 smithi122 ceph-mon[32255]: pgmap v19379: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:36 smithi022 ceph-mon[31360]: pgmap v19379: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:36 smithi022 ceph-mon[40250]: pgmap v19379: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:37.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:44:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:44:39.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:38 smithi122 ceph-mon[32255]: pgmap v19380: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:39.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:39.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:38 smithi022 ceph-mon[31360]: pgmap v19380: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:39.252 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:38 smithi022 ceph-mon[40250]: pgmap v19380: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:39.619 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:39.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:39.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:40.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1303644247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:40.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1303644247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:40.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1303644247' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:40.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:44:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:40] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:44:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:40 smithi022 ceph-mon[31360]: pgmap v19381: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:40 smithi022 ceph-mon[40250]: pgmap v19381: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:41.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:40 smithi122 ceph-mon[32255]: pgmap v19381: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:44:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:44:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:44:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:42 smithi022 ceph-mon[31360]: pgmap v19382: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:42 smithi022 ceph-mon[40250]: pgmap v19382: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:43.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:42 smithi122 ceph-mon[32255]: pgmap v19382: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:44.621 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:44.951 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:44 smithi022 ceph-mon[31360]: pgmap v19383: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:44.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:44.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:44.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:44 smithi022 ceph-mon[40250]: pgmap v19383: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:45.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:44:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:44:45.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:44 smithi122 ceph-mon[32255]: pgmap v19383: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:46.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3615585088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3615585088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:46.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3615585088' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:47.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:46 smithi022 ceph-mon[31360]: pgmap v19384: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:46 smithi022 ceph-mon[40250]: pgmap v19384: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:47.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:47.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:46 smithi122 ceph-mon[32255]: pgmap v19384: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:47.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:44:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:44:48.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:47 smithi022 ceph-mon[31360]: pgmap v19385: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:48.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:47 smithi022 ceph-mon[40250]: pgmap v19385: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:48.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:47 smithi122 ceph-mon[32255]: pgmap v19385: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:49.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:50.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:50.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:50.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:50.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:44:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:50] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:44:51.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:50 smithi122 ceph-mon[32255]: pgmap v19386: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:51.343 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3159225340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:51.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:50 smithi022 ceph-mon[31360]: pgmap v19386: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:51.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3159225340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:51.345 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:50 smithi022 ceph-mon[40250]: pgmap v19386: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:51.345 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3159225340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:52.442 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:44:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:44:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:44:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:52 smithi122 ceph-mon[32255]: pgmap v19387: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:44:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:44:53.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:44:53.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:44:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[31360]: pgmap v19387: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:44:53.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:44:53.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:44:53.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:44:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[40250]: pgmap v19387: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:44:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:44:53.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:44:53.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:44:55.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:44:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:44:55.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:54 smithi122 ceph-mon[32255]: pgmap v19388: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:54 smithi022 ceph-mon[31360]: pgmap v19388: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:54 smithi022 ceph-mon[40250]: pgmap v19388: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:44:55.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:44:55.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:44:55.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:44:55.741 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:44:56.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1904469309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:56.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1904469309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:56.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1904469309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:44:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:56 smithi022 ceph-mon[31360]: pgmap v19389: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:56 smithi022 ceph-mon[40250]: pgmap v19389: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:57.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:44:57.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:56 smithi122 ceph-mon[32255]: pgmap v19389: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:57.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:44:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:44:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:44:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:44:58 smithi022 ceph-mon[31360]: pgmap v19390: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:44:58 smithi022 ceph-mon[40250]: pgmap v19390: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:44:59.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:44:58 smithi122 ceph-mon[32255]: pgmap v19390: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:00.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:45:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:00] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:45:00.743 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:01.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:00 smithi022 ceph-mon[31360]: pgmap v19391: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:01.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:00 smithi022 ceph-mon[40250]: pgmap v19391: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:01.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:01.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:01.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:01.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:00 smithi122 ceph-mon[32255]: pgmap v19391: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:02.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1239747951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:02.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1239747951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:02.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1239747951' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:45:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:45:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:45:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:02 smithi022 ceph-mon[31360]: pgmap v19392: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:02 smithi022 ceph-mon[40250]: pgmap v19392: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:03.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:02 smithi122 ceph-mon[32255]: pgmap v19392: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:05.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:04 smithi122 ceph-mon[32255]: pgmap v19393: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:05.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:45:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:45:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:04 smithi022 ceph-mon[31360]: pgmap v19393: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:04 smithi022 ceph-mon[40250]: pgmap v19393: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:06.113 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:06.482 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:06.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:06.483 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:07.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:07.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:06 smithi122 ceph-mon[32255]: pgmap v19394: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:07.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2559145469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:07.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:06 smithi022 ceph-mon[31360]: pgmap v19394: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:07.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2559145469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:07.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:07.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:06 smithi022 ceph-mon[40250]: pgmap v19394: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:07.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2559145469' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:07.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:45:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:45:09.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:08 smithi122 ceph-mon[32255]: pgmap v19395: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:09.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:08 smithi022 ceph-mon[31360]: pgmap v19395: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:09.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:08 smithi022 ceph-mon[40250]: pgmap v19395: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:45:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:10] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:45:11.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:10 smithi122 ceph-mon[32255]: pgmap v19396: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:10 smithi022 ceph-mon[31360]: pgmap v19396: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:10 smithi022 ceph-mon[40250]: pgmap v19396: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:11.484 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:11.851 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:11.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:11.852 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:12.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/436534193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:12.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/436534193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:12.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/436534193' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:45:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:45:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:45:13.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:12 smithi022 ceph-mon[31360]: pgmap v19397: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:12 smithi022 ceph-mon[40250]: pgmap v19397: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:13.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:12 smithi122 ceph-mon[32255]: pgmap v19397: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:14 smithi022 ceph-mon[31360]: pgmap v19398: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:14 smithi022 ceph-mon[40250]: pgmap v19398: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:15.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:45:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:45:15.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:14 smithi122 ceph-mon[32255]: pgmap v19398: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:16.854 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:17.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:17.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:17.225 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:17.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:17.226 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:16 smithi022 ceph-mon[31360]: pgmap v19399: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:17.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:17.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:16 smithi022 ceph-mon[40250]: pgmap v19399: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:17.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:17.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:16 smithi122 ceph-mon[32255]: pgmap v19399: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:17.535 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:45:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:45:18.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3653118624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:18.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3653118624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:18.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3653118624' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:18 smithi022 ceph-mon[31360]: pgmap v19400: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:19.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:18 smithi022 ceph-mon[40250]: pgmap v19400: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:19.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:18 smithi122 ceph-mon[32255]: pgmap v19400: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:20.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:45:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:20] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:45:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:20 smithi022 ceph-mon[31360]: pgmap v19401: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:21.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:20 smithi022 ceph-mon[40250]: pgmap v19401: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:21.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:20 smithi122 ceph-mon[32255]: pgmap v19401: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:22.227 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:22.227 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:21 smithi022 ceph-mon[40250]: pgmap v19402: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:22.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:21 smithi022 ceph-mon[31360]: pgmap v19402: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:22.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:21 smithi122 ceph-mon[32255]: pgmap v19402: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:22.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:22.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:22.597 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:22.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:45:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:45:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:45:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4234565751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4234565751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:23.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4234565751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:25.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:24 smithi122 ceph-mon[32255]: pgmap v19403: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:25.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:45:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:45:25.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:24 smithi022 ceph-mon[31360]: pgmap v19403: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:24 smithi022 ceph-mon[40250]: pgmap v19403: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:26 smithi022 ceph-mon[31360]: pgmap v19404: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:27.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:27.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:26 smithi022 ceph-mon[40250]: pgmap v19404: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:27.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:27.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:26 smithi122 ceph-mon[32255]: pgmap v19404: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:27.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:45:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:45:27.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:27.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:27.968 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:28 smithi022 ceph-mon[31360]: pgmap v19405: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2945510612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:29.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:28 smithi022 ceph-mon[40250]: pgmap v19405: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:29.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2945510612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:29.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:28 smithi122 ceph-mon[32255]: pgmap v19405: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:29.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2945510612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:30.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:45:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:30] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:45:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:30 smithi022 ceph-mon[31360]: pgmap v19406: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:31.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:30 smithi022 ceph-mon[40250]: pgmap v19406: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:31.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:30 smithi122 ceph-mon[32255]: pgmap v19406: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:32.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:45:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:45:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:45:32.970 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:32 smithi022 ceph-mon[31360]: pgmap v19407: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:32 smithi022 ceph-mon[40250]: pgmap v19407: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:33.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:33.341 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:33.342 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:33.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:32 smithi122 ceph-mon[32255]: pgmap v19407: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:34.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/544705915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:34.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/544705915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:34.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/544705915' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[31360]: pgmap v19408: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:45:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:45:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:45:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:45:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[40250]: pgmap v19408: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:45:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:45:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:45:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:45:35.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:45:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:45:35.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:34 smithi122 ceph-mon[32255]: pgmap v19408: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:35.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:45:35.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:45:35.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:45:35.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:45:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:36 smithi022 ceph-mon[31360]: pgmap v19409: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:36 smithi022 ceph-mon[40250]: pgmap v19409: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:37.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:37.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:36 smithi122 ceph-mon[32255]: pgmap v19409: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:37.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:45:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:45:38.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:38 smithi022 ceph-mon[31360]: pgmap v19410: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:38.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:38 smithi022 ceph-mon[40250]: pgmap v19410: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:38.343 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:38.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:38 smithi122 ceph-mon[32255]: pgmap v19410: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:38.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:38.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:38.712 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:39.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/962816026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/962816026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:39.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/962816026' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:40.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:40 smithi022 ceph-mon[31360]: pgmap v19411: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:40.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:40 smithi022 ceph-mon[40250]: pgmap v19411: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:40.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:45:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:40] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:45:40.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:40 smithi122 ceph-mon[32255]: pgmap v19411: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:42.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:45:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:45:42.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:45:43.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:42 smithi122 ceph-mon[32255]: pgmap v19412: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:42 smithi022 ceph-mon[31360]: pgmap v19412: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:42 smithi022 ceph-mon[40250]: pgmap v19412: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:43.713 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:45.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:45:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:45:45.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:44 smithi122 ceph-mon[32255]: pgmap v19413: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:45.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/602244806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:45.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:44 smithi022 ceph-mon[31360]: pgmap v19413: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:45.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/602244806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:44 smithi022 ceph-mon[40250]: pgmap v19413: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:45.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/602244806' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:46 smithi022 ceph-mon[31360]: pgmap v19414: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:46 smithi022 ceph-mon[40250]: pgmap v19414: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:47.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:47.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:46 smithi122 ceph-mon[32255]: pgmap v19414: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:47.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:45:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:45:49.088 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:49.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:48 smithi022 ceph-mon[31360]: pgmap v19415: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:49.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:48 smithi022 ceph-mon[40250]: pgmap v19415: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:49.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:48 smithi122 ceph-mon[32255]: pgmap v19415: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:49.463 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:50.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2729299414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:50.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2729299414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:50.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2729299414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:50.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:45:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:50] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:45:51.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:50 smithi022 ceph-mon[31360]: pgmap v19416: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:51.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:50 smithi022 ceph-mon[40250]: pgmap v19416: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:51.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:50 smithi122 ceph-mon[32255]: pgmap v19416: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:52.593 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:45:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:45:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:45:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:52 smithi122 ceph-mon[32255]: pgmap v19417: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:45:53.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:45:53.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:52 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:45:53.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:53.193 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:52 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[31360]: pgmap v19417: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:53.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:45:53.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:45:53.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:45:53.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:53.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[40250]: pgmap v19417: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:45:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:45:53.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:45:53.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:53.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:52 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:54.466 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:45:54.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:54.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:54 smithi122 ceph-mon[32255]: pgmap v19418: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:54.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:54.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:54 smithi022 ceph-mon[31360]: pgmap v19418: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:54.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:45:54.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:54 smithi022 ceph-mon[40250]: pgmap v19418: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:45:54.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:45:54.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:45:54.836 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:45:55.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/112306858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:55.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:45:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:45:55.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/112306858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:55.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/112306858' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:45:56.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:56.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:56 smithi122 ceph-mon[32255]: pgmap v19419: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:56.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:56.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:56 smithi022 ceph-mon[31360]: pgmap v19419: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:56.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:45:56.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:56 smithi022 ceph-mon[40250]: pgmap v19419: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:57.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:45:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:45:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:45:59.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:45:58 smithi122 ceph-mon[32255]: pgmap v19420: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:59.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:45:58 smithi022 ceph-mon[31360]: pgmap v19420: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:45:58 smithi022 ceph-mon[40250]: pgmap v19420: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:45:59.837 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:00.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:00.207 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:46:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:00] "GET /metrics HTTP/1.1" 200 202500 "" "Prometheus/2.18.1" 2021-12-16T04:46:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:00 smithi022 ceph-mon[31360]: pgmap v19421: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2610505042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:01.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:00 smithi022 ceph-mon[40250]: pgmap v19421: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:01.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2610505042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:01.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:00 smithi122 ceph-mon[32255]: pgmap v19421: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:01.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2610505042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:02.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:46:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:46:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:46:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:02 smithi022 ceph-mon[31360]: pgmap v19422: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:03.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:02 smithi022 ceph-mon[40250]: pgmap v19422: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:03.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:02 smithi122 ceph-mon[32255]: pgmap v19422: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:05.208 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:04 smithi022 ceph-mon[31360]: pgmap v19423: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:04 smithi022 ceph-mon[40250]: pgmap v19423: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:05.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:46:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:46:05.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:04 smithi122 ceph-mon[32255]: pgmap v19423: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:05.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:05.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:05.578 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:06.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1336266058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:06.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1336266058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:06.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1336266058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:07.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:06 smithi022 ceph-mon[31360]: pgmap v19424: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:06 smithi022 ceph-mon[40250]: pgmap v19424: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:07.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:07.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:06 smithi122 ceph-mon[32255]: pgmap v19424: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:07.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:46:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:46:08.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:07 smithi022 ceph-mon[31360]: pgmap v19425: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:08.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:07 smithi022 ceph-mon[40250]: pgmap v19425: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:08.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:07 smithi122 ceph-mon[32255]: pgmap v19425: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:10.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:46:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:10] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:46:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:10.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:10.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:10.947 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:10 smithi022 ceph-mon[31360]: pgmap v19426: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/140406636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:10 smithi022 ceph-mon[40250]: pgmap v19426: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/140406636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:11.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:10 smithi122 ceph-mon[32255]: pgmap v19426: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:11.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/140406636' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:12.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:46:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:46:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:46:13.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:12 smithi022 ceph-mon[31360]: pgmap v19427: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:13.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:12 smithi022 ceph-mon[40250]: pgmap v19427: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:13.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:12 smithi122 ceph-mon[32255]: pgmap v19427: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:14 smithi022 ceph-mon[31360]: pgmap v19428: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:15.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:14 smithi022 ceph-mon[40250]: pgmap v19428: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:15.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:14 smithi122 ceph-mon[32255]: pgmap v19428: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:15.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:46:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:46:15.949 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:16.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:16.319 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:16.320 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:17.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:16 smithi022 ceph-mon[31360]: pgmap v19429: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:17.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:16 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3936058795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:17.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:17.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:16 smithi022 ceph-mon[40250]: pgmap v19429: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:17.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:16 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3936058795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:17.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:17.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:16 smithi122 ceph-mon[32255]: pgmap v19429: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:17.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:16 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3936058795' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:17.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:46:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:46:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:18 smithi022 ceph-mon[31360]: pgmap v19430: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:18 smithi022 ceph-mon[40250]: pgmap v19430: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:19.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:18 smithi122 ceph-mon[32255]: pgmap v19430: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:20.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:46:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:20] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:46:21.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:20 smithi022 ceph-mon[31360]: pgmap v19431: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:21.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:20 smithi022 ceph-mon[40250]: pgmap v19431: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:21.321 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:21.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:20 smithi122 ceph-mon[32255]: pgmap v19431: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:21.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:21.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:21.689 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:22.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/30233595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:22.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/30233595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:22.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/30233595' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:22.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:46:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:46:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:46:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:22 smithi022 ceph-mon[31360]: pgmap v19432: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:23.292 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:22 smithi022 ceph-mon[40250]: pgmap v19432: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:23.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:22 smithi122 ceph-mon[32255]: pgmap v19432: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:25.214 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:24 smithi122 ceph-mon[32255]: pgmap v19433: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:25.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:24 smithi022 ceph-mon[31360]: pgmap v19433: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:24 smithi022 ceph-mon[40250]: pgmap v19433: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:25.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:46:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:46:26.691 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:27.056 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:27.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:27.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:27.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:26 smithi022 ceph-mon[31360]: pgmap v19434: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:27.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:27.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:26 smithi022 ceph-mon[40250]: pgmap v19434: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:27.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:27.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:26 smithi122 ceph-mon[32255]: pgmap v19434: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:27.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:46:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:46:28.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/75225634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:28.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/75225634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:28.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/75225634' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:29.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:28 smithi022 ceph-mon[31360]: pgmap v19435: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:29.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:28 smithi022 ceph-mon[40250]: pgmap v19435: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:29.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:28 smithi122 ceph-mon[32255]: pgmap v19435: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:30.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:46:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:30] "GET /metrics HTTP/1.1" 200 202506 "" "Prometheus/2.18.1" 2021-12-16T04:46:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:30 smithi022 ceph-mon[31360]: pgmap v19436: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:31.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:30 smithi022 ceph-mon[40250]: pgmap v19436: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:31.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:30 smithi122 ceph-mon[32255]: pgmap v19436: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:32.058 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:32.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:46:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:46:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:46:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:32 smithi022 ceph-mon[31360]: pgmap v19437: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:33.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1638813050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:33.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:32 smithi022 ceph-mon[40250]: pgmap v19437: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:33.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1638813050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:33.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:32 smithi122 ceph-mon[32255]: pgmap v19437: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:33.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1638813050' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:35.220 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:34 smithi122 ceph-mon[32255]: pgmap v19438: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:35.221 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:46:35.221 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:46:35.221 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:46:35.221 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:46:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[31360]: pgmap v19438: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:46:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:46:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:46:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:46:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[40250]: pgmap v19438: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:46:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:46:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:46:35.288 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:46:35.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:46:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:46:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:36 smithi022 ceph-mon[31360]: pgmap v19439: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:37.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:36 smithi022 ceph-mon[40250]: pgmap v19439: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:37.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:37.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:37.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:36 smithi122 ceph-mon[32255]: pgmap v19439: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:37.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:46:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:46:37.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:37.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:37.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:38.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3954240180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:38.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:37 smithi022 ceph-mon[40250]: pgmap v19440: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:38.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3954240180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:38.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:37 smithi022 ceph-mon[31360]: pgmap v19440: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:38.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3954240180' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:38.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:37 smithi122 ceph-mon[32255]: pgmap v19440: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:40.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:46:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:40] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:46:41.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:40 smithi022 ceph-mon[31360]: pgmap v19441: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:40 smithi022 ceph-mon[40250]: pgmap v19441: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:41.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:40 smithi122 ceph-mon[32255]: pgmap v19441: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:42.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:46:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:46:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:46:42.806 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:43.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:43.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:43.176 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:43.176 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:42 smithi022 ceph-mon[31360]: pgmap v19442: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:43.177 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:42 smithi022 ceph-mon[40250]: pgmap v19442: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:43.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:42 smithi122 ceph-mon[32255]: pgmap v19442: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:44.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1851449426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:44.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1851449426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:44.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1851449426' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:44 smithi122 ceph-mon[32255]: pgmap v19443: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:45.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:44 smithi022 ceph-mon[31360]: pgmap v19443: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:45.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:44 smithi022 ceph-mon[40250]: pgmap v19443: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:45.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:46:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:46:47.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:47.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:46 smithi022 ceph-mon[31360]: pgmap v19444: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:47.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:46 smithi022 ceph-mon[40250]: pgmap v19444: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:47.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:47.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:46 smithi122 ceph-mon[32255]: pgmap v19444: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:47.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:46:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:46:48.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:48.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:49.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:48 smithi022 ceph-mon[31360]: pgmap v19445: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:49.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/758043117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:49.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:48 smithi022 ceph-mon[40250]: pgmap v19445: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:49.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/758043117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:49.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:48 smithi122 ceph-mon[32255]: pgmap v19445: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:49.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/758043117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:50.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:46:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:50] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:46:51.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:50 smithi022 ceph-mon[31360]: pgmap v19446: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:51.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:50 smithi022 ceph-mon[40250]: pgmap v19446: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:51.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:50 smithi122 ceph-mon[32255]: pgmap v19446: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:52.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:52 smithi022 ceph-mon[31360]: pgmap v19447: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:52.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:52 smithi022 ceph-mon[40250]: pgmap v19447: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:52.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:52 smithi122 ceph-mon[32255]: pgmap v19447: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:52.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:46:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:46:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:46:53.547 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:53.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:46:53.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:46:53.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:46:53.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:46:53.785 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:46:53.786 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:46:53.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:46:53.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:46:53.786 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:46:53.908 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:54.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:54.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:55.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:46:55.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:54 smithi122 ceph-mon[32255]: pgmap v19448: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:55.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3815569612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:46:55.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:54 smithi022 ceph-mon[31360]: pgmap v19448: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:55.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3815569612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:55.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:46:55.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:54 smithi022 ceph-mon[40250]: pgmap v19448: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:46:55.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3815569612' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:46:55.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:46:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:46:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:57.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:56 smithi122 ceph-mon[32255]: pgmap v19449: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:56 smithi022 ceph-mon[31360]: pgmap v19449: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:46:57.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:56 smithi022 ceph-mon[40250]: pgmap v19449: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:57.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:46:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:46:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:46:58.910 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:46:59.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:58 smithi122 ceph-mon[32255]: pgmap v19450: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:59.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:46:59.282 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:46:59.283 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:46:59.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:58 smithi022 ceph-mon[40250]: pgmap v19450: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:46:59.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:58 smithi022 ceph-mon[31360]: pgmap v19450: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:00.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:46:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3050456281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:00.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:46:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3050456281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:46:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3050456281' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:00.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:47:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:00] "GET /metrics HTTP/1.1" 200 202497 "" "Prometheus/2.18.1" 2021-12-16T04:47:01.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:00 smithi022 ceph-mon[31360]: pgmap v19451: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:01.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:00 smithi022 ceph-mon[40250]: pgmap v19451: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:01.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:00 smithi122 ceph-mon[32255]: pgmap v19451: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:02.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:47:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:47:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:47:03.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:02 smithi022 ceph-mon[31360]: pgmap v19452: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:03.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:02 smithi022 ceph-mon[40250]: pgmap v19452: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:03.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:02 smithi122 ceph-mon[32255]: pgmap v19452: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:04.284 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:04.652 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:04.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:04.653 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:05.250 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:04 smithi122 ceph-mon[32255]: pgmap v19453: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:05.250 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1010082884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:04 smithi022 ceph-mon[31360]: pgmap v19453: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:05.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1010082884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:05.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:04 smithi022 ceph-mon[40250]: pgmap v19453: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:05.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1010082884' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:05.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:47:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:47:06.971 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:06.971 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:06 smithi022 ceph-mon[31360]: pgmap v19454: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:07.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:06 smithi022 ceph-mon[40250]: pgmap v19454: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:07.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:07.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:06 smithi122 ceph-mon[32255]: pgmap v19454: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:07.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:47:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:47:09.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:08 smithi022 ceph-mon[31360]: pgmap v19455: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:09.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:08 smithi022 ceph-mon[40250]: pgmap v19455: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:09.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:08 smithi122 ceph-mon[32255]: pgmap v19455: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:09.654 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:10.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:10.029 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:10.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:10.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:47:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:10] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:47:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:10 smithi022 ceph-mon[31360]: pgmap v19456: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:11.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1775855324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:10 smithi022 ceph-mon[40250]: pgmap v19456: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:11.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1775855324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:11.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:10 smithi122 ceph-mon[32255]: pgmap v19456: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:11.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1775855324' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:12.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:11 smithi022 ceph-mon[31360]: pgmap v19457: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:12.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:11 smithi022 ceph-mon[40250]: pgmap v19457: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:12.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:11 smithi122 ceph-mon[32255]: pgmap v19457: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:12.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:47:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:47:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:47:15.031 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:15.258 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:14 smithi122 ceph-mon[32255]: pgmap v19458: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:14 smithi022 ceph-mon[31360]: pgmap v19458: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:15.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:14 smithi022 ceph-mon[40250]: pgmap v19458: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:15.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:15.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:15.404 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:15.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:47:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:47:16.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1941913057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:16.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1941913057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:16.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1941913057' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:16 smithi022 ceph-mon[31360]: pgmap v19459: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:17.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:17.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:16 smithi022 ceph-mon[40250]: pgmap v19459: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:17.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:17.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:16 smithi122 ceph-mon[32255]: pgmap v19459: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:17.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:47:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:47:19.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:18 smithi022 ceph-mon[31360]: pgmap v19460: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:19.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:18 smithi022 ceph-mon[40250]: pgmap v19460: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:19.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:18 smithi122 ceph-mon[32255]: pgmap v19460: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:20.406 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:47:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:20] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:47:20.771 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:20.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:20.772 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:21.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:20 smithi022 ceph-mon[40250]: pgmap v19461: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:21.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/801726879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:21.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:20 smithi022 ceph-mon[31360]: pgmap v19461: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:21.036 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/801726879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:21.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:20 smithi122 ceph-mon[32255]: pgmap v19461: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:21.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/801726879' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:22.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:47:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:47:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:47:23.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:22 smithi022 ceph-mon[31360]: pgmap v19462: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:22 smithi022 ceph-mon[40250]: pgmap v19462: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:23.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:22 smithi122 ceph-mon[32255]: pgmap v19462: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:25.269 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:24 smithi122 ceph-mon[32255]: pgmap v19463: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:25.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:24 smithi022 ceph-mon[31360]: pgmap v19463: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:25.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:24 smithi022 ceph-mon[40250]: pgmap v19463: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:25.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:47:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:47:25.773 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:26.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:25 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:26.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:25 smithi022 ceph-mon[31360]: pgmap v19464: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:26.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:25 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:26.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:25 smithi022 ceph-mon[40250]: pgmap v19464: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:26.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:25 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:26.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:25 smithi122 ceph-mon[32255]: pgmap v19464: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3654134674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3654134674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:27.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3654134674' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:27.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:47:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:47:28.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:27 smithi022 ceph-mon[31360]: pgmap v19465: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:28.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:27 smithi022 ceph-mon[40250]: pgmap v19465: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:28.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:27 smithi122 ceph-mon[32255]: pgmap v19465: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:47:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:30] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:47:31.146 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:30 smithi022 ceph-mon[31360]: pgmap v19466: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:30 smithi022 ceph-mon[40250]: pgmap v19466: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:31.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:30 smithi122 ceph-mon[32255]: pgmap v19466: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:31.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:31.517 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:31.518 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:32.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2947032222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:32.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2947032222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:32.321 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2947032222' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:47:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:47:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:47:33.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:32 smithi022 ceph-mon[40250]: pgmap v19467: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:33.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:32 smithi022 ceph-mon[31360]: pgmap v19467: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:33.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:32 smithi122 ceph-mon[32255]: pgmap v19467: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:35.279 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:34 smithi122 ceph-mon[32255]: pgmap v19468: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:35.279 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:47:35.280 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:47:35.280 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:47:35.280 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:47:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[31360]: pgmap v19468: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:47:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:47:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:47:35.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:47:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[40250]: pgmap v19468: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:47:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:47:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:47:35.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:47:35.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:47:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:47:36.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:36.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:36 smithi022 ceph-mon[31360]: pgmap v19469: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:36.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:36.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:36 smithi022 ceph-mon[40250]: pgmap v19469: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:36.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:36.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:36 smithi122 ceph-mon[32255]: pgmap v19469: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:36.519 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:36.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:36.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:36.891 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:37.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/826722109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:37.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/826722109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:37.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/826722109' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:37.785 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:47:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:47:38.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:38 smithi122 ceph-mon[32255]: pgmap v19470: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:38.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:38 smithi022 ceph-mon[31360]: pgmap v19470: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:38.535 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:38 smithi022 ceph-mon[40250]: pgmap v19470: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:47:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:40] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:47:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:40 smithi022 ceph-mon[31360]: pgmap v19471: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:41.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:40 smithi022 ceph-mon[40250]: pgmap v19471: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:41.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:40 smithi122 ceph-mon[32255]: pgmap v19471: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:41.892 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:47:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:47:42.317Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:47:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:42 smithi022 ceph-mon[31360]: pgmap v19472: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:43.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2677801903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:43.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:42 smithi022 ceph-mon[40250]: pgmap v19472: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:43.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2677801903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:43.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:42 smithi122 ceph-mon[32255]: pgmap v19472: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:43.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2677801903' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:44 smithi022 ceph-mon[31360]: pgmap v19473: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:45.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:44 smithi022 ceph-mon[40250]: pgmap v19473: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:45.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:44 smithi122 ceph-mon[32255]: pgmap v19473: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:45.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:47:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:47:47.267 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:47.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:47.268 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:46 smithi022 ceph-mon[31360]: pgmap v19474: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:47.268 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:47.268 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:46 smithi022 ceph-mon[40250]: pgmap v19474: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:47.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:47.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:46 smithi122 ceph-mon[32255]: pgmap v19474: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:47.535 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:47:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:47:47.637 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:47.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:47.638 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:48.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/770704513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:48.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/770704513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:48.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/770704513' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:49.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:48 smithi022 ceph-mon[31360]: pgmap v19475: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:49.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:48 smithi022 ceph-mon[40250]: pgmap v19475: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:49.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:48 smithi122 ceph-mon[32255]: pgmap v19475: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:50.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:49 smithi022 ceph-mon[31360]: pgmap v19476: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:50.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:49 smithi022 ceph-mon[40250]: pgmap v19476: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:50.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:49 smithi122 ceph-mon[32255]: pgmap v19476: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:50.442 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:47:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:50] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:47:52.639 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:52.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:47:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:47:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:47:53.005 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:53.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:53.006 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:52 smithi022 ceph-mon[31360]: pgmap v19477: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:52 smithi022 ceph-mon[40250]: pgmap v19477: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:53.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:52 smithi122 ceph-mon[32255]: pgmap v19477: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:54.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/656126576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:54.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:47:54.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:47:54.192 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:53 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:47:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/656126576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:54.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:47:54.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:47:54.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:47:54.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/656126576' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:47:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:47:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:53 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:47:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:54 smithi022 ceph-mon[31360]: pgmap v19478: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:55.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:47:55.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:54 smithi022 ceph-mon[40250]: pgmap v19478: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:55.287 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:47:55.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:54 smithi122 ceph-mon[32255]: pgmap v19478: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:47:55.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:47:55.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:47:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:47:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:57.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:56 smithi022 ceph-mon[31360]: pgmap v19479: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:56 smithi022 ceph-mon[40250]: pgmap v19479: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:57.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:47:57.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:56 smithi122 ceph-mon[32255]: pgmap v19479: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:57.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:47:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:47:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:47:58.008 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:47:58.381 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:47:58.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:47:58.382 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:47:59.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:58 smithi022 ceph-mon[40250]: pgmap v19480: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:59.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:47:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/837467248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:59.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:58 smithi022 ceph-mon[31360]: pgmap v19480: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:59.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:47:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/837467248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:47:59.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:58 smithi122 ceph-mon[32255]: pgmap v19480: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:47:59.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:47:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/837467248' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:00.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:48:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:00] "GET /metrics HTTP/1.1" 200 202504 "" "Prometheus/2.18.1" 2021-12-16T04:48:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:00 smithi022 ceph-mon[31360]: pgmap v19481: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:01.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:00 smithi022 ceph-mon[40250]: pgmap v19481: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:01.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:00 smithi122 ceph-mon[32255]: pgmap v19481: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:02.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:48:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:48:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:48:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:02 smithi022 ceph-mon[31360]: pgmap v19482: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:02 smithi022 ceph-mon[40250]: pgmap v19482: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:03.383 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:03.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:02 smithi122 ceph-mon[32255]: pgmap v19482: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:03.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:03.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:03.752 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:04.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2992313566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:04.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2992313566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:04.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2992313566' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:04 smithi022 ceph-mon[31360]: pgmap v19483: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:05.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:04 smithi022 ceph-mon[40250]: pgmap v19483: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:05.301 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:04 smithi122 ceph-mon[32255]: pgmap v19483: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:05.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:48:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:48:06.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:05 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:06.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:05 smithi022 ceph-mon[31360]: pgmap v19484: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:06.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:05 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:06.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:05 smithi022 ceph-mon[40250]: pgmap v19484: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:06.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:05 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:06.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:05 smithi122 ceph-mon[32255]: pgmap v19484: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:07.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:48:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:48:08.753 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:09.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:08 smithi022 ceph-mon[31360]: pgmap v19485: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:09.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:08 smithi022 ceph-mon[40250]: pgmap v19485: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:09.125 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:09.126 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:09.127 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:09.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:08 smithi122 ceph-mon[32255]: pgmap v19485: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:10.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1489808315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:10.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1489808315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:48:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:10] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:48:10.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1489808315' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:11.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:10 smithi022 ceph-mon[31360]: pgmap v19486: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:11.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:10 smithi022 ceph-mon[40250]: pgmap v19486: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:11.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:10 smithi122 ceph-mon[32255]: pgmap v19486: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:48:12.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:48:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:48:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:48:13.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:12 smithi022 ceph-mon[31360]: pgmap v19487: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:12 smithi022 ceph-mon[40250]: pgmap v19487: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:13.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:12 smithi122 ceph-mon[32255]: pgmap v19487: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:14.127 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:14 smithi022 ceph-mon[31360]: pgmap v19488: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:15.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:14 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1948138344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:14 smithi022 ceph-mon[40250]: pgmap v19488: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:15.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:14 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1948138344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:15.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:14 smithi122 ceph-mon[32255]: pgmap v19488: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:15.309 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:14 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1948138344' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:15.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:48:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:48:17.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:16 smithi022 ceph-mon[40250]: pgmap v19489: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:17.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:16 smithi022 ceph-mon[31360]: pgmap v19489: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:17.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:17.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:16 smithi122 ceph-mon[32255]: pgmap v19489: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:48:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:48:18.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:18 smithi122 ceph-mon[32255]: pgmap v19490: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:18.534 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:18 smithi022 ceph-mon[40250]: pgmap v19490: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:18.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:18 smithi022 ceph-mon[31360]: pgmap v19490: 132 pgs: 132 active+clean; 582 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:19.524 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:19.894 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:19.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:19.895 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:20.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4070578932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4070578932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:20.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4070578932' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:48:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:20] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:48:21.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:20 smithi022 ceph-mon[31360]: pgmap v19491: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:21.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:20 smithi022 ceph-mon[40250]: pgmap v19491: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:21.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:20 smithi122 ceph-mon[32255]: pgmap v19491: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:22.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:48:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:48:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:48:23.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:22 smithi022 ceph-mon[31360]: pgmap v19492: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:23.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:22 smithi022 ceph-mon[40250]: pgmap v19492: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:23.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:22 smithi122 ceph-mon[32255]: pgmap v19492: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:24.896 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:25.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:25.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:25.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:25.273 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:24 smithi022 ceph-mon[31360]: pgmap v19493: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:25.273 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:24 smithi022 ceph-mon[40250]: pgmap v19493: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:25.311 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:24 smithi122 ceph-mon[32255]: pgmap v19493: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:25.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:48:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:48:26.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1213563194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:26.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1213563194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:26.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1213563194' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:27.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:26 smithi022 ceph-mon[31360]: pgmap v19494: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:27.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:26 smithi022 ceph-mon[40250]: pgmap v19494: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:27.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:27.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:26 smithi122 ceph-mon[32255]: pgmap v19494: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:27.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:48:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:48:28.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:27 smithi022 ceph-mon[31360]: pgmap v19495: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:28.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:27 smithi022 ceph-mon[40250]: pgmap v19495: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:28.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:27 smithi122 ceph-mon[32255]: pgmap v19495: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:48:30.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:48:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:30] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:48:30.645 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:30.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:30.646 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:31.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:30 smithi022 ceph-mon[31360]: pgmap v19496: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:31.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3249975655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:30 smithi022 ceph-mon[40250]: pgmap v19496: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:31.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3249975655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:31.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:30 smithi122 ceph-mon[32255]: pgmap v19496: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:48:31.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3249975655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:32.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:48:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:48:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:48:33.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:32 smithi022 ceph-mon[31360]: pgmap v19497: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:33.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:32 smithi022 ceph-mon[40250]: pgmap v19497: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:33.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:32 smithi122 ceph-mon[32255]: pgmap v19497: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[31360]: pgmap v19498: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:48:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:48:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:48:35.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:48:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[40250]: pgmap v19498: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:48:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:48:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:48:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:48:35.311 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:34 smithi122 ceph-mon[32255]: pgmap v19498: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:35.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:48:35.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:48:35.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:48:35.312 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:48:35.647 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:35.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:48:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:48:36.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:36.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:36.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:36.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:36 smithi022 ceph-mon[31360]: pgmap v19499: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:36.285 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3763652645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:36 smithi022 ceph-mon[40250]: pgmap v19499: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:36.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3763652645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:36.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:36.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:36 smithi122 ceph-mon[32255]: pgmap v19499: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:36.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3763652645' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:37.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:48:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:48:39.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:38 smithi022 ceph-mon[31360]: pgmap v19500: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:39.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:38 smithi022 ceph-mon[40250]: pgmap v19500: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:39.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:38 smithi122 ceph-mon[32255]: pgmap v19500: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:40.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:48:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:40] "GET /metrics HTTP/1.1" 200 202502 "" "Prometheus/2.18.1" 2021-12-16T04:48:41.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:40 smithi022 ceph-mon[31360]: pgmap v19501: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:41.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:40 smithi022 ceph-mon[40250]: pgmap v19501: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:41.392 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:41.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:41.393 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:41.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:40 smithi122 ceph-mon[32255]: pgmap v19501: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:42.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/254018319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:42.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/254018319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:42.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/254018319' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:42.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:48:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:48:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:48:43.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:42 smithi022 ceph-mon[31360]: pgmap v19502: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:43.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:42 smithi022 ceph-mon[40250]: pgmap v19502: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:43.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:42 smithi122 ceph-mon[32255]: pgmap v19502: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:45.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:44 smithi022 ceph-mon[31360]: pgmap v19503: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:45.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:44 smithi022 ceph-mon[40250]: pgmap v19503: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:45.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:44 smithi122 ceph-mon[32255]: pgmap v19503: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:45.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:48:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:48:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:45 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:46.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:45 smithi022 ceph-mon[31360]: pgmap v19504: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:45 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:46.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:45 smithi022 ceph-mon[40250]: pgmap v19504: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:46.394 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:46.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:45 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:46.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:45 smithi122 ceph-mon[32255]: pgmap v19504: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:47.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3701061366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:47.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3701061366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:47.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3701061366' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:47.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:48:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:48:48.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:47 smithi022 ceph-mon[31360]: pgmap v19505: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:48.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:47 smithi022 ceph-mon[40250]: pgmap v19505: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:48.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:47 smithi122 ceph-mon[32255]: pgmap v19505: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:48:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:50] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:48:51.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:50 smithi022 ceph-mon[31360]: pgmap v19506: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:51.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:50 smithi022 ceph-mon[40250]: pgmap v19506: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:51.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:50 smithi122 ceph-mon[32255]: pgmap v19506: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:51.766 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:52.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:52.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:52.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:48:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:48:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:48:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:52 smithi022 ceph-mon[31360]: pgmap v19507: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:53.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2748460438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:52 smithi022 ceph-mon[40250]: pgmap v19507: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:53.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2748460438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:53.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:52 smithi122 ceph-mon[32255]: pgmap v19507: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:53.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2748460438' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:55.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[31360]: pgmap v19508: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:55.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:48:55.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:48:55.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:48:55.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:48:55.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[40250]: pgmap v19508: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:55.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:48:55.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:48:55.036 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:48:55.037 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:54 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:48:55.328 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:54 smithi122 ceph-mon[32255]: pgmap v19508: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:48:55.328 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:48:55.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:48:55.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:48:55.329 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:54 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:48:55.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:48:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:48:57.143 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:48:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:57.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:56 smithi022 ceph-mon[31360]: pgmap v19509: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:57.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:56 smithi022 ceph-mon[40250]: pgmap v19509: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:57.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:48:57.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:56 smithi122 ceph-mon[32255]: pgmap v19509: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:57.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:48:57.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:48:57.511 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:48:57.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:48:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:48:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:48:58.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3060638087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:58.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:48:57 smithi022 ceph-mon[31360]: pgmap v19510: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:58.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3060638087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:58.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:48:57 smithi022 ceph-mon[40250]: pgmap v19510: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:48:58.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3060638087' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:48:58.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:48:57 smithi122 ceph-mon[32255]: pgmap v19510: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:49:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:00] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:49:01.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:00 smithi022 ceph-mon[31360]: pgmap v19511: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:01.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:00 smithi022 ceph-mon[40250]: pgmap v19511: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:01.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:00 smithi122 ceph-mon[32255]: pgmap v19511: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:02.513 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:49:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:49:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:49:02.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:02.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:02.880 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:02 smithi022 ceph-mon[31360]: pgmap v19512: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:03.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/937961001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:03.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:02 smithi022 ceph-mon[40250]: pgmap v19512: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:03.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/937961001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:03.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:02 smithi122 ceph-mon[32255]: pgmap v19512: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:03.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/937961001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:05.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:04 smithi022 ceph-mon[31360]: pgmap v19513: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:05.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:04 smithi022 ceph-mon[40250]: pgmap v19513: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:05.333 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:04 smithi122 ceph-mon[32255]: pgmap v19513: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:05.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:49:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:49:07.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:07.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:06 smithi022 ceph-mon[31360]: pgmap v19514: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:07.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:07.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:06 smithi022 ceph-mon[40250]: pgmap v19514: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:07.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:07.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:06 smithi122 ceph-mon[32255]: pgmap v19514: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:49:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:49:07.882 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:08.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:08.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:08.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:08.256 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:07 smithi022 ceph-mon[31360]: pgmap v19515: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:08.257 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:07 smithi022 ceph-mon[40250]: pgmap v19515: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:08.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:07 smithi122 ceph-mon[32255]: pgmap v19515: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:09.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1148822401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:09.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1148822401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:09.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1148822401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:10.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:09 smithi022 ceph-mon[31360]: pgmap v19516: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:10.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:09 smithi022 ceph-mon[40250]: pgmap v19516: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:10.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:09 smithi122 ceph-mon[32255]: pgmap v19516: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:10.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:49:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:10] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:49:12.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:49:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:49:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:49:13.257 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:13.258 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:12 smithi022 ceph-mon[31360]: pgmap v19517: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:13.258 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:12 smithi022 ceph-mon[40250]: pgmap v19517: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:13.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:12 smithi122 ceph-mon[32255]: pgmap v19517: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:13.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:13.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:13.630 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:13.970 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2767215235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:13.971 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2767215235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:14.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2767215235' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:15.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:14 smithi122 ceph-mon[32255]: pgmap v19518: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:15.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:14 smithi022 ceph-mon[31360]: pgmap v19518: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:15.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:14 smithi022 ceph-mon[40250]: pgmap v19518: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:15.440 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:49:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:49:17.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:17.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:16 smithi022 ceph-mon[31360]: pgmap v19519: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:17.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:17.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:16 smithi022 ceph-mon[40250]: pgmap v19519: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:17.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:17.313 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:16 smithi122 ceph-mon[32255]: pgmap v19519: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:17.784 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:49:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:49:18.631 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:18.989 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:18 smithi022 ceph-mon[31360]: pgmap v19520: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:19.003 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:19.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:19.004 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:19.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:18 smithi122 ceph-mon[32255]: pgmap v19520: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:19.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:18 smithi022 ceph-mon[40250]: pgmap v19520: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:20.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4008102998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4008102998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:20.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4008102998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:20.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:49:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:20] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:49:21.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:20 smithi022 ceph-mon[31360]: pgmap v19521: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:21.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:20 smithi022 ceph-mon[40250]: pgmap v19521: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:21.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:20 smithi122 ceph-mon[32255]: pgmap v19521: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:22.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:21 smithi022 ceph-mon[31360]: pgmap v19522: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:22.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:21 smithi022 ceph-mon[40250]: pgmap v19522: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:22.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:21 smithi122 ceph-mon[32255]: pgmap v19522: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:22.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:49:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:49:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:49:24.005 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:24.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:24.372 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:24.373 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:25.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:24 smithi022 ceph-mon[31360]: pgmap v19523: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:25.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2896979588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:24 smithi022 ceph-mon[40250]: pgmap v19523: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:25.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2896979588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:25.343 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:24 smithi122 ceph-mon[32255]: pgmap v19523: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:25.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2896979588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:25.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:49:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:49:27.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:27.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:26 smithi022 ceph-mon[31360]: pgmap v19524: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:27.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:27.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:26 smithi022 ceph-mon[40250]: pgmap v19524: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:27.393 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:27.393 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:26 smithi122 ceph-mon[32255]: pgmap v19524: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:27.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:49:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:49:29.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:28 smithi022 ceph-mon[31360]: pgmap v19525: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:29.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:28 smithi022 ceph-mon[40250]: pgmap v19525: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:29.374 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:29.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:28 smithi122 ceph-mon[32255]: pgmap v19525: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:29.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:29.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:29.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:30.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1799951790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:30.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1799951790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:30.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1799951790' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:30.441 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:49:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:30] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:49:31.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:30 smithi022 ceph-mon[31360]: pgmap v19526: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:31.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:30 smithi022 ceph-mon[40250]: pgmap v19526: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:31.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:30 smithi122 ceph-mon[32255]: pgmap v19526: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:32.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:31 smithi022 ceph-mon[31360]: pgmap v19527: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:32.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:31 smithi022 ceph-mon[40250]: pgmap v19527: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:32.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:32 smithi122 ceph-mon[32255]: pgmap v19527: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:32.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:49:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:49:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:49:34.745 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[31360]: pgmap v19528: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:35.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:49:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:49:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:49:35.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:49:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[40250]: pgmap v19528: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:49:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:49:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:49:35.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:49:35.113 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:35.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:35.114 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:35.352 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:34 smithi122 ceph-mon[32255]: pgmap v19528: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:35.352 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:49:35.353 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:49:35.353 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:49:35.353 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:49:35.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:49:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:49:36.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/928913255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:36.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/928913255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:36.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/928913255' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:37.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:37.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:36 smithi022 ceph-mon[31360]: pgmap v19529: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:36 smithi022 ceph-mon[40250]: pgmap v19529: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:37.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:37.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:36 smithi122 ceph-mon[32255]: pgmap v19529: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:37.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:49:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:49:38.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:37 smithi022 ceph-mon[31360]: pgmap v19530: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:38.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:37 smithi022 ceph-mon[40250]: pgmap v19530: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:38.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:37 smithi122 ceph-mon[32255]: pgmap v19530: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:40.115 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:40.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:49:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:40] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:49:40.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:40.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:40.485 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:41.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:40 smithi022 ceph-mon[31360]: pgmap v19531: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:41.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2882031338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:41.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:40 smithi022 ceph-mon[40250]: pgmap v19531: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:41.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2882031338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:41.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:40 smithi122 ceph-mon[32255]: pgmap v19531: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:41.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2882031338' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:42.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:49:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:49:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:49:43.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:42 smithi022 ceph-mon[31360]: pgmap v19532: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:43.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:42 smithi022 ceph-mon[40250]: pgmap v19532: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:43.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:42 smithi122 ceph-mon[32255]: pgmap v19532: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:45.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:44 smithi022 ceph-mon[31360]: pgmap v19533: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:45.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:44 smithi022 ceph-mon[40250]: pgmap v19533: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:45.362 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:44 smithi122 ceph-mon[32255]: pgmap v19533: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:45.487 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:45.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:49:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:49:45.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:45.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:45.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:46.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3150250556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:46.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3150250556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:46.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3150250556' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:47.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:47.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:46 smithi022 ceph-mon[31360]: pgmap v19534: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:47.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:47.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:46 smithi022 ceph-mon[40250]: pgmap v19534: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:47.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:47.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:46 smithi122 ceph-mon[32255]: pgmap v19534: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:47.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:49:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:49:48.286 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:47 smithi022 ceph-mon[31360]: pgmap v19535: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:48.373 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:47 smithi022 ceph-mon[40250]: pgmap v19535: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:48.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:47 smithi122 ceph-mon[32255]: pgmap v19535: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:50.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:49:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:50] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:49:50.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:51.229 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:51.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:51.230 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:51.231 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:50 smithi022 ceph-mon[40250]: pgmap v19536: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:51.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:50 smithi022 ceph-mon[31360]: pgmap v19536: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:51.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:50 smithi122 ceph-mon[32255]: pgmap v19536: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:52.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1919137880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:52.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1919137880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:52.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1919137880' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:52.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:49:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:49:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:49:53.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:52 smithi022 ceph-mon[31360]: pgmap v19537: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:53.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:52 smithi022 ceph-mon[40250]: pgmap v19537: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:53.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:52 smithi122 ceph-mon[32255]: pgmap v19537: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:54 smithi122 ceph-mon[32255]: pgmap v19538: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:49:55.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:49:55.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:54 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:49:55.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[31360]: pgmap v19538: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:49:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:49:55.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:49:55.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[40250]: pgmap v19538: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:49:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:49:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:49:55.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:54 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:49:55.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:49:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:49:56.232 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:49:56.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:49:56.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:49:56.610 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:49:57.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:57.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:56 smithi022 ceph-mon[31360]: pgmap v19539: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:57.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/635038089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:56 smithi022 ceph-mon[40250]: pgmap v19539: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:57.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/635038089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:57.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:49:57.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:56 smithi122 ceph-mon[32255]: pgmap v19539: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:57.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/635038089' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:49:57.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:49:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:49:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:49:59.280 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:58 smithi122 ceph-mon[32255]: pgmap v19540: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:59.280 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:49:58 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:49:59.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:58 smithi022 ceph-mon[31360]: pgmap v19540: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:59.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:49:58 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:49:59.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:58 smithi022 ceph-mon[40250]: pgmap v19540: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:49:59.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:49:58 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:50:00.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:50:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:00] "GET /metrics HTTP/1.1" 200 202509 "" "Prometheus/2.18.1" 2021-12-16T04:50:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:50:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:00 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:50:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:00 smithi122 ceph-mon[32255]: pgmap v19541: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T04:50:00.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:50:00.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:50:00.783 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[31360]: pgmap v19541: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:00.784 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T04:50:00.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:50:00.784 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:50:00.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[40250]: pgmap v19541: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:00.785 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T04:50:01.612 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:01.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:01.974 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:01.975 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:02.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:02 smithi122 ceph-mon[32255]: pgmap v19542: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:02.190 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/909462919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:02.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:02 smithi022 ceph-mon[31360]: pgmap v19542: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:02.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/909462919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:02.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:02 smithi022 ceph-mon[40250]: pgmap v19542: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:02.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/909462919' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:02.690 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:50:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:50:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:50:05.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:04 smithi022 ceph-mon[31360]: pgmap v19543: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:05.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:04 smithi022 ceph-mon[40250]: pgmap v19543: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:05.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:04 smithi122 ceph-mon[32255]: pgmap v19543: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:05.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:50:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:50:06.976 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:07.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:07.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:06 smithi022 ceph-mon[31360]: pgmap v19544: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:07.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:07.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:06 smithi022 ceph-mon[40250]: pgmap v19544: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:07.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:07.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:07.349 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:07.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:07.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:06 smithi122 ceph-mon[32255]: pgmap v19544: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:50:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:50:08.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1740140765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:08.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1740140765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:08.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1740140765' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:09.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:08 smithi022 ceph-mon[31360]: pgmap v19545: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:09.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:08 smithi022 ceph-mon[40250]: pgmap v19545: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:09.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:09 smithi122 ceph-mon[32255]: pgmap v19545: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:10.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:10 smithi022 ceph-mon[31360]: pgmap v19546: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:10.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:10 smithi022 ceph-mon[40250]: pgmap v19546: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:10.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:10 smithi122 ceph-mon[32255]: pgmap v19546: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:10.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:50:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:10] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:50:12.351 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:50:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:50:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:50:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:13.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:12 smithi022 ceph-mon[31360]: pgmap v19547: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:13.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3880011897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:13.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:12 smithi022 ceph-mon[40250]: pgmap v19547: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:13.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3880011897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:13.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:12 smithi122 ceph-mon[32255]: pgmap v19547: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:13.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3880011897' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:15.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:14 smithi022 ceph-mon[31360]: pgmap v19548: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:15.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:14 smithi022 ceph-mon[40250]: pgmap v19548: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:15.387 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:14 smithi122 ceph-mon[32255]: pgmap v19548: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:15.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:50:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:50:17.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:17.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:16 smithi022 ceph-mon[31360]: pgmap v19549: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:17.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:17.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:16 smithi022 ceph-mon[40250]: pgmap v19549: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:17.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:17.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:16 smithi122 ceph-mon[32255]: pgmap v19549: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:17.723 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:50:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:50:17.723 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:18.005 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:18 smithi022 ceph-mon[31360]: pgmap v19550: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:18.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:18.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:18.094 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:18 smithi022 ceph-mon[40250]: pgmap v19550: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:18.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:18 smithi122 ceph-mon[32255]: pgmap v19550: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:19.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2613735153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:19.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2613735153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:19.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2613735153' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:20.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:20 smithi022 ceph-mon[31360]: pgmap v19551: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:20.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:20 smithi022 ceph-mon[40250]: pgmap v19551: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:20.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:20 smithi122 ceph-mon[32255]: pgmap v19551: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:20.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:50:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:20] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:50:22.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:50:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:50:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:50:23.096 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:23.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:22 smithi022 ceph-mon[31360]: pgmap v19552: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:23.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:22 smithi022 ceph-mon[40250]: pgmap v19552: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:23.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:22 smithi122 ceph-mon[32255]: pgmap v19552: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:23.470 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:23.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:23.471 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:24.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1707727609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:24.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1707727609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:24.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1707727609' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:25.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:24 smithi022 ceph-mon[31360]: pgmap v19553: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:25.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:24 smithi022 ceph-mon[40250]: pgmap v19553: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:25.394 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:24 smithi122 ceph-mon[32255]: pgmap v19553: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:25.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:50:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:50:27.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:27.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:26 smithi022 ceph-mon[31360]: pgmap v19554: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:27.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:27.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:26 smithi022 ceph-mon[40250]: pgmap v19554: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:27.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:27.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:26 smithi122 ceph-mon[32255]: pgmap v19554: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:27.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:50:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:50:28.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:28 smithi022 ceph-mon[31360]: pgmap v19555: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:28.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:28 smithi022 ceph-mon[40250]: pgmap v19555: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:28.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:28 smithi122 ceph-mon[32255]: pgmap v19555: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:28.472 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:28.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:28.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:28.841 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:29.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/65910983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:29.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/65910983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:29.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/65910983' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:30.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:30 smithi022 ceph-mon[31360]: pgmap v19556: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:30.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:30 smithi022 ceph-mon[40250]: pgmap v19556: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:30.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:30 smithi122 ceph-mon[32255]: pgmap v19556: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:30.440 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:50:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:30] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:50:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:50:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:50:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:50:33.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:32 smithi022 ceph-mon[31360]: pgmap v19557: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:33.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:32 smithi022 ceph-mon[40250]: pgmap v19557: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:33.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:32 smithi122 ceph-mon[32255]: pgmap v19557: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:33.843 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:34.213 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:34.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:34.214 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:35.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[31360]: pgmap v19558: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:35.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3960542453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:50:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:50:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:50:35.284 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:50:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[40250]: pgmap v19558: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3960542453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:50:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:50:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:50:35.286 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:34 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:50:35.401 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:34 smithi122 ceph-mon[32255]: pgmap v19558: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:35.402 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3960542453' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:35.402 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:50:35.402 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:50:35.403 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:50:35.403 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:34 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:50:35.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:50:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:50:37.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:37.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:36 smithi022 ceph-mon[31360]: pgmap v19559: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:37.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:36 smithi022 ceph-mon[40250]: pgmap v19559: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:37.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:37 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:37.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:37 smithi122 ceph-mon[32255]: pgmap v19559: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:37.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:50:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:50:38.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:38 smithi022 ceph-mon[31360]: pgmap v19560: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:38.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:38 smithi022 ceph-mon[40250]: pgmap v19560: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:38.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:38 smithi122 ceph-mon[32255]: pgmap v19560: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:39.215 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:39.583 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:39.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:39.584 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:40.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/737647673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:40.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/737647673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:50:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:40] "GET /metrics HTTP/1.1" 200 202503 "" "Prometheus/2.18.1" 2021-12-16T04:50:40.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/737647673' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:41.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:41 smithi022 ceph-mon[31360]: pgmap v19561: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:41.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:41 smithi022 ceph-mon[40250]: pgmap v19561: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:41.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:41 smithi122 ceph-mon[32255]: pgmap v19561: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:42.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:42 smithi022 ceph-mon[31360]: pgmap v19562: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:42.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:42 smithi022 ceph-mon[40250]: pgmap v19562: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:42.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:42 smithi122 ceph-mon[32255]: pgmap v19562: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:42.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:50:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:50:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:50:44.585 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:44.956 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:44.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:44.957 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:45.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:44 smithi022 ceph-mon[31360]: pgmap v19563: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:45.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1187042569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:45.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:44 smithi022 ceph-mon[40250]: pgmap v19563: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:45.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1187042569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:45.410 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:44 smithi122 ceph-mon[32255]: pgmap v19563: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:45.411 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1187042569' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:45.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:50:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:50:46.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:46.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:46 smithi122 ceph-mon[32255]: pgmap v19564: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:46.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:46.533 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:46 smithi022 ceph-mon[31360]: pgmap v19564: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:46.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:46.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:46 smithi022 ceph-mon[40250]: pgmap v19564: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:47.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:50:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:50:49.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:48 smithi022 ceph-mon[31360]: pgmap v19565: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:49.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:48 smithi022 ceph-mon[40250]: pgmap v19565: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:49.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:48 smithi122 ceph-mon[32255]: pgmap v19565: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:49.958 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:50.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:50.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:50.328 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:50:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:50] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:50:51.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:51 smithi022 ceph-mon[31360]: pgmap v19566: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:51.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3681976223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:51.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:51 smithi022 ceph-mon[40250]: pgmap v19566: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:51.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3681976223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:51.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:51 smithi122 ceph-mon[32255]: pgmap v19566: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:51.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3681976223' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:52.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:52 smithi022 ceph-mon[31360]: pgmap v19567: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:52.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:52 smithi022 ceph-mon[40250]: pgmap v19567: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:52.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:52 smithi122 ceph-mon[32255]: pgmap v19567: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:52.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:50:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:50:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:50:55.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:55 smithi022 ceph-mon[31360]: pgmap v19568: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:55.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:55 smithi022 ceph-mon[40250]: pgmap v19568: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:55.329 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:50:55.414 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:55 smithi122 ceph-mon[32255]: pgmap v19568: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:50:55.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:50:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:50:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:50:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:50:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:50:56.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2691368275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2691368275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:56.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2691368275' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:50:57.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:57 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:57.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:57 smithi022 ceph-mon[31360]: pgmap v19569: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:57.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:57 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:57.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:57 smithi022 ceph-mon[40250]: pgmap v19569: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:57.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:57 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:50:57.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:57 smithi122 ceph-mon[32255]: pgmap v19569: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:50:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:50:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:50:58.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:50:58 smithi022 ceph-mon[31360]: pgmap v19570: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:58.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:50:58 smithi022 ceph-mon[40250]: pgmap v19570: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:50:58.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:50:58 smithi122 ceph-mon[32255]: pgmap v19570: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:00.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:51:00.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:51:00.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:00 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:51:00.033 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:51:00.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:51:00.034 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:00 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:51:00.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:51:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:00] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:51:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:51:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:51:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:00 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:51:00.701 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:01.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:01.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:01.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:01.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:01 smithi022 ceph-mon[40250]: pgmap v19571: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:01.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:51:01.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:01 smithi022 ceph-mon[31360]: pgmap v19571: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:01.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:51:01.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:01 smithi122 ceph-mon[32255]: pgmap v19571: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:01.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:51:02.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/152165901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:02.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:02 smithi022 ceph-mon[31360]: pgmap v19572: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:02.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/152165901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:02.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:02 smithi022 ceph-mon[40250]: pgmap v19572: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:02.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/152165901' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:02.319 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:02 smithi122 ceph-mon[32255]: pgmap v19572: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:02.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:51:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:51:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:51:05.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:05 smithi022 ceph-mon[31360]: pgmap v19573: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:05.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:05 smithi022 ceph-mon[40250]: pgmap v19573: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:05.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:05 smithi122 ceph-mon[32255]: pgmap v19573: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:05.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:51:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:51:06.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:06.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:06.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:06 smithi022 ceph-mon[31360]: pgmap v19574: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:06.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:06.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:06 smithi022 ceph-mon[40250]: pgmap v19574: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:06.435 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:06.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:06.436 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:06.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:06.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:06 smithi122 ceph-mon[32255]: pgmap v19574: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:07.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2146825151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:07.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2146825151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:07.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2146825151' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:07.783 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:51:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:51:08.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:08 smithi022 ceph-mon[31360]: pgmap v19575: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:08.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:08 smithi022 ceph-mon[40250]: pgmap v19575: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:08.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:08 smithi122 ceph-mon[32255]: pgmap v19575: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:10.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:10 smithi022 ceph-mon[31360]: pgmap v19576: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:10.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:10 smithi022 ceph-mon[40250]: pgmap v19576: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:10.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:10 smithi122 ceph-mon[32255]: pgmap v19576: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:51:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:10] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:51:11.437 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:11.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:12.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1424649588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:12.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1424649588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:12.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1424649588' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:12.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:51:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:51:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:51:13.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:12 smithi122 ceph-mon[32255]: pgmap v19577: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:13.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:12 smithi022 ceph-mon[31360]: pgmap v19577: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:13.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:12 smithi022 ceph-mon[40250]: pgmap v19577: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:15.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:14 smithi022 ceph-mon[31360]: pgmap v19578: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:15.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:14 smithi022 ceph-mon[40250]: pgmap v19578: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:14 smithi122 ceph-mon[32255]: pgmap v19578: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:15.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:51:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:51:16.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:17.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:17.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:17.181 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:17.182 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:17 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:17.182 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:17 smithi022 ceph-mon[40250]: pgmap v19579: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:17.182 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:17.183 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:16 smithi022 ceph-mon[31360]: pgmap v19579: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:17.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:17 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:17.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:17 smithi122 ceph-mon[32255]: pgmap v19579: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:17.532 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:51:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:51:18.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/973569483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:18.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/973569483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:18.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/973569483' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:19.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:19 smithi022 ceph-mon[31360]: pgmap v19580: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:19.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:19 smithi022 ceph-mon[40250]: pgmap v19580: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:19.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:19 smithi122 ceph-mon[32255]: pgmap v19580: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:20.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:20 smithi022 ceph-mon[31360]: pgmap v19581: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:20.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:20 smithi022 ceph-mon[40250]: pgmap v19581: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:51:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:20] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:51:20.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:20 smithi122 ceph-mon[32255]: pgmap v19581: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:22.183 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:22.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:22.551 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:22.552 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:22.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:51:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:51:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:51:23.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:22 smithi022 ceph-mon[31360]: pgmap v19582: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:23.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1903489414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:23.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:22 smithi022 ceph-mon[40250]: pgmap v19582: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:23.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1903489414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:23.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:22 smithi122 ceph-mon[32255]: pgmap v19582: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:23.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1903489414' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:25.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:25 smithi022 ceph-mon[31360]: pgmap v19583: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:25.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:25 smithi022 ceph-mon[40250]: pgmap v19583: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:25.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:25 smithi122 ceph-mon[32255]: pgmap v19583: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:25.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:51:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:51:27.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:27 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:27.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:27 smithi022 ceph-mon[31360]: pgmap v19584: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:27.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:27 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:27.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:27 smithi022 ceph-mon[40250]: pgmap v19584: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:27.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:27 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:27.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:27 smithi122 ceph-mon[32255]: pgmap v19584: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:27.554 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:27.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:51:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:51:27.921 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:27.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:27.922 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:28.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3750258459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:28.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3750258459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:28.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3750258459' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:29.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:29 smithi022 ceph-mon[31360]: pgmap v19585: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:29.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:29 smithi022 ceph-mon[40250]: pgmap v19585: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:29.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:29 smithi122 ceph-mon[32255]: pgmap v19585: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:30.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:30 smithi022 ceph-mon[31360]: pgmap v19586: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:30.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:30 smithi022 ceph-mon[40250]: pgmap v19586: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:30.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:51:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:30] "GET /metrics HTTP/1.1" 200 202499 "" "Prometheus/2.18.1" 2021-12-16T04:51:30.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:30 smithi122 ceph-mon[32255]: pgmap v19586: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:32.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:51:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:51:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:51:32.923 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:33.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:33 smithi022 ceph-mon[31360]: pgmap v19587: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:33.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:33 smithi022 ceph-mon[40250]: pgmap v19587: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:33.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:33.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:33.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:33.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:33 smithi122 ceph-mon[32255]: pgmap v19587: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:34.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1156761752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:34.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1156761752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:34.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1156761752' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:35.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[31360]: pgmap v19588: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:35.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:51:35.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:51:35.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:51:35.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:51:35.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[40250]: pgmap v19588: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:35.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:51:35.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:51:35.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:51:35.285 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:51:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:35 smithi122 ceph-mon[32255]: pgmap v19588: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:51:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:51:35.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:51:35.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:51:35.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:51:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:51:36.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:36.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:36 smithi022 ceph-mon[31360]: pgmap v19589: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:36.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:36.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:36 smithi022 ceph-mon[40250]: pgmap v19589: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:36.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:36.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:36 smithi122 ceph-mon[32255]: pgmap v19589: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:37.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:51:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:51:38.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:38.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:38.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:38.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:39.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:39 smithi022 ceph-mon[31360]: pgmap v19590: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:39.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1620074379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:39.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:39 smithi022 ceph-mon[40250]: pgmap v19590: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:39.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1620074379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:39.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:39 smithi122 ceph-mon[32255]: pgmap v19590: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:39.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1620074379' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:40.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:40 smithi022 ceph-mon[31360]: pgmap v19591: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:40.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:40 smithi022 ceph-mon[40250]: pgmap v19591: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:51:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:40] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:51:40.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:40 smithi122 ceph-mon[32255]: pgmap v19591: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:42.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:51:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:51:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:51:43.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:43 smithi022 ceph-mon[31360]: pgmap v19592: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:43.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:43 smithi022 ceph-mon[40250]: pgmap v19592: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:43.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:43 smithi122 ceph-mon[32255]: pgmap v19592: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:43.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:44.045 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:45.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:45 smithi022 ceph-mon[31360]: pgmap v19593: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:45.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1790040780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:45.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:45 smithi022 ceph-mon[40250]: pgmap v19593: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:45.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1790040780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:45.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:45 smithi122 ceph-mon[32255]: pgmap v19593: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:45.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1790040780' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:45.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:51:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:51:46.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:46.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:46 smithi122 ceph-mon[32255]: pgmap v19594: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:46.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:46.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:46 smithi022 ceph-mon[31360]: pgmap v19594: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:46.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:46 smithi022 ceph-mon[40250]: pgmap v19594: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:47.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:51:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:51:49.047 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:49.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:49 smithi022 ceph-mon[31360]: pgmap v19595: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:49.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:49 smithi022 ceph-mon[40250]: pgmap v19595: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:49.418 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:49.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:49.419 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:49.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:49 smithi122 ceph-mon[32255]: pgmap v19595: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:50.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1860434342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:50.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:50 smithi022 ceph-mon[31360]: pgmap v19596: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:50.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1860434342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:50.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:50 smithi022 ceph-mon[40250]: pgmap v19596: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:51:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:50] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:51:50.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1860434342' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:50.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:50 smithi122 ceph-mon[32255]: pgmap v19596: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:52.689 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:51:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:51:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:51:53.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:53 smithi022 ceph-mon[31360]: pgmap v19597: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:53.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:53 smithi022 ceph-mon[40250]: pgmap v19597: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:53.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:53 smithi122 ceph-mon[32255]: pgmap v19597: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:51:54.793 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:51:54.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:51:54.794 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:51:55.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:55 smithi022 ceph-mon[31360]: pgmap v19598: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:55.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4137062646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:55 smithi022 ceph-mon[40250]: pgmap v19598: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:55.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4137062646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:55.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:55 smithi122 ceph-mon[32255]: pgmap v19598: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:51:55.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4137062646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:51:55.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:51:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:51:56.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:56.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:56 smithi022 ceph-mon[31360]: pgmap v19599: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:56.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:56 smithi022 ceph-mon[40250]: pgmap v19599: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:56.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:51:56.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:56 smithi122 ceph-mon[32255]: pgmap v19599: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:51:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:51:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:51:59.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:51:59 smithi022 ceph-mon[31360]: pgmap v19600: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:59.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:51:59 smithi022 ceph-mon[40250]: pgmap v19600: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:59.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:51:59 smithi122 ceph-mon[32255]: pgmap v19600: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:51:59.795 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:00.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:00.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:00.162 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:00.163 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:00 smithi022 ceph-mon[40250]: pgmap v19601: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:00.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:00 smithi022 ceph-mon[31360]: pgmap v19601: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:52:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:00] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:52:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:00 smithi122 ceph-mon[32255]: pgmap v19601: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:01.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2207881933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:01.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:52:01.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:52:01.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:52:01.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:52:01.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2207881933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:01.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:52:01.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:52:01.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:52:01.284 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:01 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:52:01.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2207881933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:01.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:52:01.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:52:01.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:52:01.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:01 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:52:02.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:02 smithi022 ceph-mon[31360]: pgmap v19602: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:02.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:02 smithi022 ceph-mon[40250]: pgmap v19602: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:02.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:02 smithi122 ceph-mon[32255]: pgmap v19602: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:02.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:52:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:52:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:52:05.165 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:05.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:05 smithi022 ceph-mon[31360]: pgmap v19603: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:05.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:05 smithi022 ceph-mon[40250]: pgmap v19603: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:05.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:05 smithi122 ceph-mon[32255]: pgmap v19603: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:05.531 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:05.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:05.532 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:05.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:52:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:52:06.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:06.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1640457156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:06.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:06 smithi022 ceph-mon[31360]: pgmap v19604: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:06.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:06.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1640457156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:06.283 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:06 smithi022 ceph-mon[40250]: pgmap v19604: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:06.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:06.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1640457156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:06.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:06 smithi122 ceph-mon[32255]: pgmap v19604: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:52:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:52:09.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:09 smithi022 ceph-mon[31360]: pgmap v19605: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:09.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:09 smithi022 ceph-mon[40250]: pgmap v19605: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:09.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:09 smithi122 ceph-mon[32255]: pgmap v19605: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:10.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:10 smithi022 ceph-mon[31360]: pgmap v19606: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:10 smithi022 ceph-mon[40250]: pgmap v19606: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:10 smithi122 ceph-mon[32255]: pgmap v19606: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:52:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:10] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T04:52:10.533 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:10.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:10.903 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:10.904 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:11.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3089234330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:11.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3089234330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:11.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3089234330' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:12.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:12 smithi022 ceph-mon[31360]: pgmap v19607: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:12.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:12 smithi022 ceph-mon[40250]: pgmap v19607: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:12.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:12 smithi122 ceph-mon[32255]: pgmap v19607: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:12.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:52:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:52:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:52:14.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:14 smithi122 ceph-mon[32255]: pgmap v19608: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:14.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:14 smithi022 ceph-mon[31360]: pgmap v19608: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:14.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:14 smithi022 ceph-mon[40250]: pgmap v19608: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:15.905 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:15.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:52:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:52:16.275 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:16.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:16.276 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:16.276 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:16.277 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:16 smithi022 ceph-mon[31360]: pgmap v19609: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:16.277 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:16.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:16 smithi022 ceph-mon[40250]: pgmap v19609: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:16.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:16.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:16 smithi122 ceph-mon[32255]: pgmap v19609: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:17.396 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2360222311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:17.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2360222311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:17.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2360222311' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:52:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:52:18.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:18 smithi122 ceph-mon[32255]: pgmap v19610: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:18.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:18 smithi022 ceph-mon[31360]: pgmap v19610: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:18.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:18 smithi022 ceph-mon[40250]: pgmap v19610: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:20 smithi022 ceph-mon[31360]: pgmap v19611: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:20 smithi022 ceph-mon[40250]: pgmap v19611: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:20 smithi122 ceph-mon[32255]: pgmap v19611: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:20.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:52:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:20] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:52:21.277 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:21.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:21.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:21.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:21.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/900088229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:22.032 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/900088229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:22.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/900088229' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:22.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:52:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:52:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:52:23.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:22 smithi022 ceph-mon[31360]: pgmap v19612: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:23.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:22 smithi022 ceph-mon[40250]: pgmap v19612: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:23.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:22 smithi122 ceph-mon[32255]: pgmap v19612: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:24.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:24 smithi122 ceph-mon[32255]: pgmap v19613: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:24 smithi022 ceph-mon[31360]: pgmap v19613: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:24.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:24 smithi022 ceph-mon[40250]: pgmap v19613: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:25.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:52:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:52:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:26.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:26 smithi122 ceph-mon[32255]: pgmap v19614: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:26.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:26.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:26 smithi022 ceph-mon[31360]: pgmap v19614: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:26.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:26.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:26 smithi022 ceph-mon[40250]: pgmap v19614: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:26.649 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:27.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3248834784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:27.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3248834784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:27.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3248834784' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:52:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:52:28.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:28 smithi122 ceph-mon[32255]: pgmap v19615: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:28 smithi022 ceph-mon[31360]: pgmap v19615: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:28 smithi022 ceph-mon[40250]: pgmap v19615: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:30 smithi122 ceph-mon[32255]: pgmap v19616: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:30.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:52:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:30] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:52:30.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:30 smithi022 ceph-mon[31360]: pgmap v19616: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:30.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:30 smithi022 ceph-mon[40250]: pgmap v19616: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:32.022 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:32.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:32 smithi022 ceph-mon[31360]: pgmap v19617: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:32.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:32 smithi022 ceph-mon[40250]: pgmap v19617: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:32.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:32.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:32.388 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:32.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:32 smithi122 ceph-mon[32255]: pgmap v19617: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:32.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:52:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:52:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:52:33.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3445436455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:33.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3445436455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:33.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3445436455' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:34 smithi122 ceph-mon[32255]: pgmap v19618: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:34 smithi022 ceph-mon[31360]: pgmap v19618: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:34.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:34 smithi022 ceph-mon[40250]: pgmap v19618: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:52:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:52:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:52:35.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:52:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:52:35.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:52:35.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:52:35.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:52:35.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:52:35.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:52:35.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:52:35.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:52:35.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:52:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:52:36.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:36.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:36 smithi122 ceph-mon[32255]: pgmap v19619: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:36.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:36 smithi022 ceph-mon[31360]: pgmap v19619: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:36.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:36.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:36 smithi022 ceph-mon[40250]: pgmap v19619: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:37.390 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:37.760 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:37.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:37.761 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:37.761 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:52:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:52:38.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3497589349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:38.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3497589349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:38.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3497589349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:39.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:38 smithi122 ceph-mon[32255]: pgmap v19620: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:39.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:38 smithi022 ceph-mon[31360]: pgmap v19620: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:39.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:38 smithi022 ceph-mon[40250]: pgmap v19620: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:40.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:40 smithi122 ceph-mon[32255]: pgmap v19621: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:52:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:40] "GET /metrics HTTP/1.1" 200 202496 "" "Prometheus/2.18.1" 2021-12-16T04:52:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:40 smithi022 ceph-mon[40250]: pgmap v19621: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:40.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:40 smithi022 ceph-mon[31360]: pgmap v19621: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:42.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:42 smithi122 ceph-mon[32255]: pgmap v19622: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:42.439 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:52:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:52:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:52:42.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:42 smithi022 ceph-mon[31360]: pgmap v19622: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:42.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:42 smithi022 ceph-mon[40250]: pgmap v19622: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:42.762 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:43.133 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:43.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:43.134 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:43.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2712070016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:43.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2712070016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:43.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2712070016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:44.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:44 smithi122 ceph-mon[32255]: pgmap v19623: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:44.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:44 smithi022 ceph-mon[31360]: pgmap v19623: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:44.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:44 smithi022 ceph-mon[40250]: pgmap v19623: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:45.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:52:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:52:46.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:46.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:46 smithi122 ceph-mon[32255]: pgmap v19624: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:46.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:46.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:46 smithi022 ceph-mon[31360]: pgmap v19624: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:46 smithi022 ceph-mon[40250]: pgmap v19624: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:52:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:52:48.135 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:48.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:48 smithi122 ceph-mon[32255]: pgmap v19625: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:48.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:48.505 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:48.506 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:48.506 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:48 smithi022 ceph-mon[31360]: pgmap v19625: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:48.506 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:48 smithi022 ceph-mon[40250]: pgmap v19625: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:49.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1041655071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:49.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1041655071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:49.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1041655071' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:50 smithi122 ceph-mon[32255]: pgmap v19626: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:50.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:52:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:50] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:52:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:50 smithi022 ceph-mon[31360]: pgmap v19626: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:50.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:50 smithi022 ceph-mon[40250]: pgmap v19626: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:52.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:52 smithi122 ceph-mon[32255]: pgmap v19627: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:52.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:52:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:52:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:52:52.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:52 smithi022 ceph-mon[31360]: pgmap v19627: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:52.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:52 smithi022 ceph-mon[40250]: pgmap v19627: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:53.508 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:53.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:53.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:53.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:52:54.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/181134655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:54.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/181134655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:54.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/181134655' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:52:55.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:54 smithi122 ceph-mon[32255]: pgmap v19628: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:55.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:54 smithi022 ceph-mon[31360]: pgmap v19628: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:55.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:54 smithi022 ceph-mon[40250]: pgmap v19628: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:52:55.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:52:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:52:56.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:56.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:56 smithi122 ceph-mon[32255]: pgmap v19629: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:56.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:56.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:56 smithi022 ceph-mon[31360]: pgmap v19629: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:56.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:52:56.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:56 smithi022 ceph-mon[40250]: pgmap v19629: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:52:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:52:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:52:58.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:52:58 smithi122 ceph-mon[32255]: pgmap v19630: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:58.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:52:58 smithi022 ceph-mon[31360]: pgmap v19630: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:58.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:52:58 smithi022 ceph-mon[40250]: pgmap v19630: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:52:58.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:52:59.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:52:59.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:52:59.249 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:53:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:00] "GET /metrics HTTP/1.1" 200 202498 "" "Prometheus/2.18.1" 2021-12-16T04:53:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3042987353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:00 smithi122 ceph-mon[32255]: pgmap v19631: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:00.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3042987353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:00.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:00 smithi022 ceph-mon[31360]: pgmap v19631: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:00.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3042987353' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:00.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:00 smithi022 ceph-mon[40250]: pgmap v19631: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:01.037 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:53:01.038 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:53:01.038 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:53:01.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:53:01.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:53:01.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:53:01.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:53:01.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:53:01.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:53:02.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:53:02.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:02 smithi122 ceph-mon[32255]: pgmap v19632: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:02.439 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:53:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:53:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:53:02.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:53:02.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:02 smithi022 ceph-mon[31360]: pgmap v19632: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:02.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:53:02.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:02 smithi022 ceph-mon[40250]: pgmap v19632: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:04.251 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:04.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:04 smithi122 ceph-mon[32255]: pgmap v19633: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:04.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:04 smithi022 ceph-mon[31360]: pgmap v19633: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:04.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:04 smithi022 ceph-mon[40250]: pgmap v19633: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:04.620 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:04.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:04.621 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:05.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/864928040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:05.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/864928040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:05.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/864928040' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:05.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:53:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:53:06.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:06.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:06 smithi122 ceph-mon[32255]: pgmap v19634: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:06.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:06.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:06 smithi022 ceph-mon[31360]: pgmap v19634: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:06.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:06.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:06 smithi022 ceph-mon[40250]: pgmap v19634: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:53:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:53:08.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:08 smithi122 ceph-mon[32255]: pgmap v19635: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:08.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:08 smithi022 ceph-mon[31360]: pgmap v19635: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:08.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:08 smithi022 ceph-mon[40250]: pgmap v19635: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:09.622 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:09.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:09.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:09.995 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:10.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:10 smithi022 ceph-mon[31360]: pgmap v19636: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:10.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/262515610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:10 smithi022 ceph-mon[40250]: pgmap v19636: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:10.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/262515610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:10 smithi122 ceph-mon[32255]: pgmap v19636: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/262515610' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:10.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:53:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:10] "GET /metrics HTTP/1.1" 200 202508 "" "Prometheus/2.18.1" 2021-12-16T04:53:12.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:12 smithi122 ceph-mon[32255]: pgmap v19637: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:12.439 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:53:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:53:12.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:53:12.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:12 smithi022 ceph-mon[31360]: pgmap v19637: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:12.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:12 smithi022 ceph-mon[40250]: pgmap v19637: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:14.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:14 smithi122 ceph-mon[32255]: pgmap v19638: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:14.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:14 smithi022 ceph-mon[40250]: pgmap v19638: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:14.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:14 smithi022 ceph-mon[31360]: pgmap v19638: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:14.997 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:15.367 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:15.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:15.368 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:15.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:53:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:53:15.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2918940587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:15.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2918940587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:15.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2918940587' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:16.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:16.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:16 smithi122 ceph-mon[32255]: pgmap v19639: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:16.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:16.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:16 smithi022 ceph-mon[31360]: pgmap v19639: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:16.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:16.782 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:16 smithi022 ceph-mon[40250]: pgmap v19639: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:53:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:53:18.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:18 smithi122 ceph-mon[32255]: pgmap v19640: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:18 smithi022 ceph-mon[31360]: pgmap v19640: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:18.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:18 smithi022 ceph-mon[40250]: pgmap v19640: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:20.369 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:20.370 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:20 smithi022 ceph-mon[31360]: pgmap v19641: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:20.370 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:20 smithi022 ceph-mon[40250]: pgmap v19641: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:20 smithi122 ceph-mon[32255]: pgmap v19641: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:53:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:20] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:53:20.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:20.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:20.740 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:21.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2672593024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:21.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2672593024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:21.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2672593024' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:22.317 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:22 smithi122 ceph-mon[32255]: pgmap v19642: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:22.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:22 smithi022 ceph-mon[31360]: pgmap v19642: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:22.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:22 smithi022 ceph-mon[40250]: pgmap v19642: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:22.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:53:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:53:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:53:24.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:24 smithi122 ceph-mon[32255]: pgmap v19643: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:24.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:24 smithi022 ceph-mon[31360]: pgmap v19643: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:24 smithi022 ceph-mon[40250]: pgmap v19643: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:25.742 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:25.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:53:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:53:26.111 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:26.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:26.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:26.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:26 smithi122 ceph-mon[32255]: pgmap v19644: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:26 smithi022 ceph-mon[40250]: pgmap v19644: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:26.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:26.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:26 smithi022 ceph-mon[31360]: pgmap v19644: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:27.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3387321763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:27.396 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3387321763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:27.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3387321763' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:27.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:53:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:53:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:28 smithi122 ceph-mon[32255]: pgmap v19645: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:28 smithi022 ceph-mon[31360]: pgmap v19645: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:28.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:28 smithi022 ceph-mon[40250]: pgmap v19645: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:53:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:30] "GET /metrics HTTP/1.1" 200 202507 "" "Prometheus/2.18.1" 2021-12-16T04:53:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:30 smithi122 ceph-mon[32255]: pgmap v19646: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:30.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:30 smithi022 ceph-mon[31360]: pgmap v19646: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:30 smithi022 ceph-mon[40250]: pgmap v19646: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:31.112 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:31.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/125281986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:31.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/125281986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:31.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/125281986' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:32.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:53:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:53:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:53:32.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:32 smithi122 ceph-mon[32255]: pgmap v19647: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:32.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:32 smithi022 ceph-mon[31360]: pgmap v19647: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:32.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:32 smithi022 ceph-mon[40250]: pgmap v19647: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:34.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:34 smithi122 ceph-mon[32255]: pgmap v19648: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:34.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:34 smithi022 ceph-mon[31360]: pgmap v19648: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:34 smithi022 ceph-mon[40250]: pgmap v19648: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:53:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:53:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:53:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:53:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:53:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:53:35.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:53:35.532 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:53:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:53:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:53:35.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:53:35.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:53:35.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:53:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:53:36.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:36.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:36 smithi122 ceph-mon[32255]: pgmap v19649: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:36.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:36.488 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:36.489 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:36 smithi022 ceph-mon[40250]: pgmap v19649: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:36.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:36.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:36 smithi022 ceph-mon[31360]: pgmap v19649: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:36.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:36.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:36.855 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:37.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1804876291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:37.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1804876291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:37.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1804876291' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:37.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:53:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:53:38.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:38 smithi122 ceph-mon[32255]: pgmap v19650: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:38 smithi022 ceph-mon[31360]: pgmap v19650: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:38 smithi022 ceph-mon[40250]: pgmap v19650: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:40.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:40 smithi122 ceph-mon[32255]: pgmap v19651: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:40.439 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:53:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:40] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:53:40.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:40 smithi022 ceph-mon[31360]: pgmap v19651: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:40.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:40 smithi022 ceph-mon[40250]: pgmap v19651: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:41.857 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:42.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:42.227 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:42.228 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:42.228 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:42 smithi022 ceph-mon[31360]: pgmap v19652: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:42.228 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:42 smithi022 ceph-mon[40250]: pgmap v19652: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:42.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:42 smithi122 ceph-mon[32255]: pgmap v19652: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:42.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:53:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:53:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:53:43.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3125696981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:43.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3125696981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:43.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3125696981' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:44.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:44 smithi122 ceph-mon[32255]: pgmap v19653: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:44.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:44 smithi022 ceph-mon[31360]: pgmap v19653: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:44 smithi022 ceph-mon[40250]: pgmap v19653: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:53:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:53:46.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:46.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:46 smithi122 ceph-mon[32255]: pgmap v19654: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:46.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:46.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:46 smithi022 ceph-mon[31360]: pgmap v19654: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:46 smithi022 ceph-mon[40250]: pgmap v19654: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:47.229 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:47.531 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:53:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:53:47.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:47.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:47.599 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:47.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2646792309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:48.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2646792309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:48.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2646792309' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:48.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:48 smithi122 ceph-mon[32255]: pgmap v19655: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:49.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:48 smithi022 ceph-mon[31360]: pgmap v19655: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:49.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:48 smithi022 ceph-mon[40250]: pgmap v19655: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:50 smithi122 ceph-mon[32255]: pgmap v19656: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:50.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:53:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:50] "GET /metrics HTTP/1.1" 200 202519 "" "Prometheus/2.18.1" 2021-12-16T04:53:50.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:50 smithi022 ceph-mon[31360]: pgmap v19656: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:50 smithi022 ceph-mon[40250]: pgmap v19656: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:52.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:52 smithi122 ceph-mon[32255]: pgmap v19657: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:52.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:53:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:53:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:53:52.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:52 smithi022 ceph-mon[31360]: pgmap v19657: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:52 smithi022 ceph-mon[40250]: pgmap v19657: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:52.600 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:52.969 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:52.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:52.970 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:53.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3464908998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:53.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3464908998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:53.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3464908998' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:54 smithi122 ceph-mon[32255]: pgmap v19658: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:54.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:54 smithi022 ceph-mon[31360]: pgmap v19658: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:54 smithi022 ceph-mon[40250]: pgmap v19658: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:53:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:53:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:53:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:56.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:56 smithi122 ceph-mon[32255]: pgmap v19659: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:56.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:56.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:56 smithi022 ceph-mon[31360]: pgmap v19659: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:56.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:53:56.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:56 smithi022 ceph-mon[40250]: pgmap v19659: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:57.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:53:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:53:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:53:57.971 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:53:58.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:58 smithi022 ceph-mon[31360]: pgmap v19660: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:58.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:58 smithi022 ceph-mon[40250]: pgmap v19660: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:58.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:53:58.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:53:58.340 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:53:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:58 smithi122 ceph-mon[32255]: pgmap v19660: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:53:59.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:53:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2520296401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:59.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:53:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2520296401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:53:59.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:53:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2520296401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:00 smithi122 ceph-mon[32255]: pgmap v19661: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:54:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:00] "GET /metrics HTTP/1.1" 200 202519 "" "Prometheus/2.18.1" 2021-12-16T04:54:00.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:00 smithi022 ceph-mon[31360]: pgmap v19661: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:00 smithi022 ceph-mon[40250]: pgmap v19661: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:01.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:54:01.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:54:01.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:54:01.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:54:01.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:54:01.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:54:01.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:54:01.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:54:01.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:54:02.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:54:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:54:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:54:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:54:02.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:02 smithi122 ceph-mon[32255]: pgmap v19662: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:54:03.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:02 smithi022 ceph-mon[31360]: pgmap v19662: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:03.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:54:03.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:02 smithi022 ceph-mon[40250]: pgmap v19662: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:03.343 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:03.710 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:03.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:03.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:04.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/109814384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/109814384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:04.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/109814384' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:05.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:04 smithi022 ceph-mon[31360]: pgmap v19663: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:04 smithi022 ceph-mon[40250]: pgmap v19663: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:05.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:04 smithi122 ceph-mon[32255]: pgmap v19663: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:05.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:54:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:54:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:06.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:06 smithi122 ceph-mon[32255]: pgmap v19664: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:06.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:06.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:06 smithi022 ceph-mon[31360]: pgmap v19664: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:06.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:06.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:06 smithi022 ceph-mon[40250]: pgmap v19664: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:07.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:54:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:54:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:08 smithi122 ceph-mon[32255]: pgmap v19665: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:08.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:08 smithi022 ceph-mon[31360]: pgmap v19665: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:08 smithi022 ceph-mon[40250]: pgmap v19665: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:08.712 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:09.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:09.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:09.081 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:54:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:10] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:54:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2004569202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:10.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:10 smithi122 ceph-mon[32255]: pgmap v19666: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:10.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2004569202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:10.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:10 smithi022 ceph-mon[31360]: pgmap v19666: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2004569202' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:10.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:10 smithi022 ceph-mon[40250]: pgmap v19666: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:12 smithi122 ceph-mon[32255]: pgmap v19667: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:12.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:54:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:54:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:54:12.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:12 smithi022 ceph-mon[31360]: pgmap v19667: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:12 smithi022 ceph-mon[40250]: pgmap v19667: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:14.083 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:14.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:14 smithi122 ceph-mon[32255]: pgmap v19668: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:14.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:14.452 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:14.453 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:14.453 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:14 smithi022 ceph-mon[40250]: pgmap v19668: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:14.453 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:14 smithi022 ceph-mon[31360]: pgmap v19668: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:15.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1941058090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:15.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1941058090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1941058090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:15.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:54:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:54:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:16 smithi122 ceph-mon[32255]: pgmap v19669: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:16.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:16 smithi022 ceph-mon[31360]: pgmap v19669: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:16 smithi022 ceph-mon[40250]: pgmap v19669: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:17.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:54:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:54:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:18 smithi122 ceph-mon[32255]: pgmap v19670: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:18.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:18 smithi022 ceph-mon[31360]: pgmap v19670: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:18 smithi022 ceph-mon[40250]: pgmap v19670: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:19.455 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:19.821 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/671218686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:20.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:20 smithi022 ceph-mon[31360]: pgmap v19671: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/671218686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:20.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:20 smithi022 ceph-mon[40250]: pgmap v19671: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/671218686' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:20.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:20 smithi122 ceph-mon[32255]: pgmap v19671: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:54:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:20] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:54:22.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:22 smithi122 ceph-mon[32255]: pgmap v19672: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:22.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:54:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:54:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:54:22.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:22 smithi022 ceph-mon[31360]: pgmap v19672: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:22 smithi022 ceph-mon[40250]: pgmap v19672: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:24 smithi122 ceph-mon[32255]: pgmap v19673: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:24.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:24 smithi022 ceph-mon[31360]: pgmap v19673: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:24.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:24 smithi022 ceph-mon[40250]: pgmap v19673: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:24.822 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:25.194 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:25.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:25.195 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:25.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2417299856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2417299856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:25.582 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2417299856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:54:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:54:26.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:26.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:26 smithi022 ceph-mon[31360]: pgmap v19674: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:26.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:26 smithi022 ceph-mon[40250]: pgmap v19674: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:26.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:26.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:26 smithi122 ceph-mon[32255]: pgmap v19674: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:54:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:54:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:28 smithi122 ceph-mon[32255]: pgmap v19675: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:28.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:28 smithi022 ceph-mon[31360]: pgmap v19675: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:28 smithi022 ceph-mon[40250]: pgmap v19675: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:30.196 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:30 smithi122 ceph-mon[32255]: pgmap v19676: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:30.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:54:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:30] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:54:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:30 smithi022 ceph-mon[31360]: pgmap v19676: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:30 smithi022 ceph-mon[40250]: pgmap v19676: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:30.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:30.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:30.565 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:31.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4280818267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:31.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4280818267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:31.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4280818267' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:32.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:32 smithi122 ceph-mon[32255]: pgmap v19677: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:32.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:32 smithi022 ceph-mon[31360]: pgmap v19677: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:32 smithi022 ceph-mon[40250]: pgmap v19677: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:32.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:54:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:54:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:54:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:34 smithi122 ceph-mon[32255]: pgmap v19678: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:34 smithi022 ceph-mon[31360]: pgmap v19678: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:34 smithi022 ceph-mon[40250]: pgmap v19678: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:54:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:54:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:54:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:54:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:54:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:54:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:54:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:54:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:54:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:54:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:54:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:54:35.567 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:54:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:54:35.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:35.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:35.938 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:36.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:36.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2809312902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:36.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:36 smithi022 ceph-mon[31360]: pgmap v19679: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:36.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:36.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2809312902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:36.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:36 smithi022 ceph-mon[40250]: pgmap v19679: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2809312902' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:36.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:36 smithi122 ceph-mon[32255]: pgmap v19679: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:54:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:54:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:38 smithi122 ceph-mon[32255]: pgmap v19680: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:38.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:38 smithi022 ceph-mon[31360]: pgmap v19680: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:38 smithi022 ceph-mon[40250]: pgmap v19680: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:40 smithi122 ceph-mon[32255]: pgmap v19681: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:54:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:40] "GET /metrics HTTP/1.1" 200 202515 "" "Prometheus/2.18.1" 2021-12-16T04:54:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:40 smithi022 ceph-mon[31360]: pgmap v19681: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:40 smithi022 ceph-mon[40250]: pgmap v19681: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:40.939 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:41.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:41.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:41.310 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:41.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/449165822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:41.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/449165822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:41.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/449165822' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:42.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:42 smithi122 ceph-mon[32255]: pgmap v19682: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:42.688 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:54:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:54:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:54:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:42 smithi022 ceph-mon[31360]: pgmap v19682: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:42.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:42 smithi022 ceph-mon[40250]: pgmap v19682: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:44 smithi122 ceph-mon[32255]: pgmap v19683: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:44 smithi022 ceph-mon[31360]: pgmap v19683: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:44 smithi022 ceph-mon[40250]: pgmap v19683: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:54:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:54:46.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:46 smithi122 ceph-mon[32255]: pgmap v19684: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:46.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:46.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:46 smithi022 ceph-mon[31360]: pgmap v19684: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:46.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:46 smithi022 ceph-mon[40250]: pgmap v19684: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:46.682 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:46.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:46.683 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:47.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/466057909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:47.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/466057909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:47.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/466057909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:47.781 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:54:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:54:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:48 smithi122 ceph-mon[32255]: pgmap v19685: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:48 smithi022 ceph-mon[31360]: pgmap v19685: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:48 smithi022 ceph-mon[40250]: pgmap v19685: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:54:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:50] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:54:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:50 smithi122 ceph-mon[32255]: pgmap v19686: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:50 smithi022 ceph-mon[31360]: pgmap v19686: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:50 smithi022 ceph-mon[40250]: pgmap v19686: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:51.685 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:52.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:52.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:52.054 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:54:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:54:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:54:52.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:52 smithi122 ceph-mon[32255]: pgmap v19687: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:52.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2358578230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:52 smithi022 ceph-mon[31360]: pgmap v19687: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:52.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2358578230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:52 smithi022 ceph-mon[40250]: pgmap v19687: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2358578230' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:54 smithi122 ceph-mon[32255]: pgmap v19688: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:54 smithi022 ceph-mon[31360]: pgmap v19688: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:54 smithi022 ceph-mon[40250]: pgmap v19688: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:54:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:54:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:54:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:56 smithi122 ceph-mon[32255]: pgmap v19689: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:56.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:56 smithi022 ceph-mon[31360]: pgmap v19689: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:54:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:56 smithi022 ceph-mon[40250]: pgmap v19689: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:57.055 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:54:57.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:54:57.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:54:57.431 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:54:57.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1447357683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:57.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1447357683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:57.782 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:54:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:54:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:54:57.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1447357683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:54:58.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:54:58 smithi022 ceph-mon[31360]: pgmap v19690: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:58.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:54:58 smithi022 ceph-mon[40250]: pgmap v19690: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:54:58.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:54:58 smithi122 ceph-mon[32255]: pgmap v19690: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:00 smithi122 ceph-mon[32255]: pgmap v19691: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:00.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:55:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:00] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:55:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:00 smithi022 ceph-mon[31360]: pgmap v19691: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:00 smithi022 ceph-mon[40250]: pgmap v19691: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:55:01.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:55:01.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:01 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:55:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:55:02.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:55:02.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:01 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:55:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:55:02.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:55:02.032 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:01 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:55:02.432 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:02.650 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:55:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:55:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:55:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:02 smithi022 ceph-mon[31360]: pgmap v19692: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:02.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:55:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:02 smithi022 ceph-mon[40250]: pgmap v19692: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:02.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:55:02.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:02.795 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:02.796 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:02 smithi122 ceph-mon[32255]: pgmap v19692: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:02.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:55:03.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3023455296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:04.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3023455296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:04.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3023455296' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:04.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:04 smithi122 ceph-mon[32255]: pgmap v19693: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:05.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:04 smithi022 ceph-mon[31360]: pgmap v19693: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:05.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:04 smithi022 ceph-mon[40250]: pgmap v19693: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:05.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:55:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:55:06.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:06.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:06 smithi122 ceph-mon[32255]: pgmap v19694: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:06.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:06 smithi022 ceph-mon[31360]: pgmap v19694: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:06 smithi022 ceph-mon[40250]: pgmap v19694: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:55:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:55:07.797 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:08.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:08.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:08.170 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:08.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:08 smithi122 ceph-mon[32255]: pgmap v19695: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:08 smithi022 ceph-mon[31360]: pgmap v19695: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:08.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:08 smithi022 ceph-mon[40250]: pgmap v19695: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:09.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/589954490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:09.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/589954490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/589954490' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:10 smithi122 ceph-mon[32255]: pgmap v19696: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:55:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:10] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:55:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:10 smithi022 ceph-mon[31360]: pgmap v19696: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:10.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:10 smithi022 ceph-mon[40250]: pgmap v19696: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:12 smithi122 ceph-mon[32255]: pgmap v19697: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:55:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:55:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:55:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:12 smithi022 ceph-mon[31360]: pgmap v19697: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:12.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:12 smithi022 ceph-mon[40250]: pgmap v19697: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:13.172 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:13.546 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:13.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:13.547 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:13.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3960163893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:14.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3960163893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:14.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3960163893' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:14.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:14 smithi122 ceph-mon[32255]: pgmap v19698: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:15.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:14 smithi022 ceph-mon[31360]: pgmap v19698: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:15.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:14 smithi022 ceph-mon[40250]: pgmap v19698: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:15.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:55:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:55:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:16.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:16 smithi122 ceph-mon[32255]: pgmap v19699: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:16.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:16 smithi022 ceph-mon[31360]: pgmap v19699: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:16 smithi022 ceph-mon[40250]: pgmap v19699: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:55:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:55:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:18 smithi122 ceph-mon[32255]: pgmap v19700: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:18 smithi022 ceph-mon[31360]: pgmap v19700: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:18.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:18 smithi022 ceph-mon[40250]: pgmap v19700: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:18.548 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:18.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:18.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:18.917 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:19.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1641683683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:19.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1641683683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:19.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1641683683' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:20 smithi122 ceph-mon[32255]: pgmap v19701: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:20.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:55:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:20] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:55:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:20 smithi022 ceph-mon[31360]: pgmap v19701: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:20 smithi022 ceph-mon[40250]: pgmap v19701: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:55:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:55:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:55:22.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:22 smithi122 ceph-mon[32255]: pgmap v19702: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:22 smithi022 ceph-mon[31360]: pgmap v19702: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:22.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:22 smithi022 ceph-mon[40250]: pgmap v19702: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:23.919 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:24.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:24 smithi022 ceph-mon[31360]: pgmap v19703: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:24.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:24 smithi022 ceph-mon[40250]: pgmap v19703: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:24.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:24.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:24.298 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:24.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:24 smithi122 ceph-mon[32255]: pgmap v19703: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:25.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4279978446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:25.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4279978446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:25.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4279978446' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:25.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:55:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:55:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:26 smithi122 ceph-mon[32255]: pgmap v19704: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:26.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:26 smithi022 ceph-mon[31360]: pgmap v19704: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:26 smithi022 ceph-mon[40250]: pgmap v19704: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:55:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:55:28.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:28 smithi122 ceph-mon[32255]: pgmap v19705: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:28.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:28 smithi022 ceph-mon[31360]: pgmap v19705: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:28.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:28 smithi022 ceph-mon[40250]: pgmap v19705: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-12-16T04:55:29.300 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:29.669 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:55:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:30] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:55:30.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1065353347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:30.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:30 smithi122 ceph-mon[32255]: pgmap v19706: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1065353347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:30.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:30 smithi022 ceph-mon[31360]: pgmap v19706: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1065353347' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:30.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:30 smithi022 ceph-mon[40250]: pgmap v19706: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:32 smithi122 ceph-mon[32255]: pgmap v19707: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:32.438 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:55:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:55:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:55:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:32 smithi022 ceph-mon[31360]: pgmap v19707: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:32.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:32 smithi022 ceph-mon[40250]: pgmap v19707: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:34.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:34 smithi122 ceph-mon[32255]: pgmap v19708: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:34 smithi022 ceph-mon[31360]: pgmap v19708: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:34.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:34 smithi022 ceph-mon[40250]: pgmap v19708: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:34.671 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:35.036 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:35.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:35.037 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:55:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:55:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:55:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:55:35.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3563228207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:55:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:55:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:55:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:55:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3563228207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:55:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:55:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:55:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:55:35.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3563228207' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:35.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:55:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:55:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:36 smithi122 ceph-mon[32255]: pgmap v19709: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:36 smithi022 ceph-mon[31360]: pgmap v19709: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:36 smithi022 ceph-mon[40250]: pgmap v19709: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:55:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:55:38.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:38 smithi122 ceph-mon[32255]: pgmap v19710: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:38 smithi022 ceph-mon[31360]: pgmap v19710: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:38.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:38 smithi022 ceph-mon[40250]: pgmap v19710: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:40.039 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:40.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:40.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:40.409 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:40.410 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:40 smithi022 ceph-mon[40250]: pgmap v19711: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:40.410 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:40 smithi022 ceph-mon[31360]: pgmap v19711: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:40 smithi122 ceph-mon[32255]: pgmap v19711: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:55:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:40] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:55:41.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4147581485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:41.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4147581485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4147581485' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:42.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:42 smithi122 ceph-mon[32255]: pgmap v19712: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:55:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:55:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:55:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:42 smithi022 ceph-mon[31360]: pgmap v19712: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:42.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:42 smithi022 ceph-mon[40250]: pgmap v19712: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:44 smithi122 ceph-mon[32255]: pgmap v19713: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:44.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:44 smithi022 ceph-mon[31360]: pgmap v19713: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:44.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:44 smithi022 ceph-mon[40250]: pgmap v19713: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:45.411 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:45.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:45.786 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:45.787 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2446385494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:45.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:55:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:55:46.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2446385494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:46.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2446385494' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:47.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:46 smithi122 ceph-mon[32255]: pgmap v19714: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:47.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:46 smithi022 ceph-mon[31360]: pgmap v19714: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:47.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:47.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:46 smithi022 ceph-mon[40250]: pgmap v19714: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:55:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:55:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:48 smithi122 ceph-mon[32255]: pgmap v19715: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:48 smithi022 ceph-mon[31360]: pgmap v19715: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:48.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:48 smithi022 ceph-mon[40250]: pgmap v19715: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:50 smithi122 ceph-mon[32255]: pgmap v19716: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:55:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:50] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:55:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:50 smithi022 ceph-mon[31360]: pgmap v19716: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:50.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:50 smithi022 ceph-mon[40250]: pgmap v19716: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:50.788 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1501044947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:51.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1501044947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:51.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1501044947' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:52 smithi022 ceph-mon[31360]: pgmap v19717: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:52.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:52 smithi022 ceph-mon[40250]: pgmap v19717: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:52.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:52 smithi122 ceph-mon[32255]: pgmap v19717: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:52.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:55:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:55:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:55:54.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:54 smithi122 ceph-mon[32255]: pgmap v19718: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:54 smithi022 ceph-mon[31360]: pgmap v19718: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:54.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:54 smithi022 ceph-mon[40250]: pgmap v19718: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:55:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:55:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:55:56.162 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:55:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:56.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:56 smithi122 ceph-mon[32255]: pgmap v19719: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:56.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:55:56.528 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:55:56.529 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:55:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:56 smithi022 ceph-mon[31360]: pgmap v19719: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:55:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:56 smithi022 ceph-mon[40250]: pgmap v19719: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:57.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2935727614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:57.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2935727614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:57.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2935727614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:55:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:55:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:55:58.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:55:58 smithi122 ceph-mon[32255]: pgmap v19720: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:55:58 smithi022 ceph-mon[31360]: pgmap v19720: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:55:58.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:55:58 smithi022 ceph-mon[40250]: pgmap v19720: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:00 smithi122 ceph-mon[32255]: pgmap v19721: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:00.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:56:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:00] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:56:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:00 smithi022 ceph-mon[31360]: pgmap v19721: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:00 smithi022 ceph-mon[40250]: pgmap v19721: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:01.530 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:01.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:01.897 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:01.898 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:02.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1309186814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:02.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1309186814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:02.320 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1309186814' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:02.320 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:56:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:56:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:56:03.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:02 smithi122 ceph-mon[32255]: pgmap v19722: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:56:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:56:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:02 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:56:03.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.188 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:02 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[40250]: pgmap v19722: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:56:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:56:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:56:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.282 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[31360]: pgmap v19722: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:03.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:56:03.282 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:56:03.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:56:03.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:03.283 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:02 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:56:04.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:04 smithi122 ceph-mon[32255]: pgmap v19723: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:04.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:04 smithi022 ceph-mon[31360]: pgmap v19723: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:04 smithi022 ceph-mon[40250]: pgmap v19723: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:05.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:56:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:56:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:06 smithi122 ceph-mon[32255]: pgmap v19724: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:06.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:06 smithi022 ceph-mon[31360]: pgmap v19724: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:06.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:06 smithi022 ceph-mon[40250]: pgmap v19724: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:06.900 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:07.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:07.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:07.273 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:07.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2265751679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:07.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:56:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:56:07.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2265751679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2265751679' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:08.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:08 smithi122 ceph-mon[32255]: pgmap v19725: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:08.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:08 smithi022 ceph-mon[31360]: pgmap v19725: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:08.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:08 smithi022 ceph-mon[40250]: pgmap v19725: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:10 smithi122 ceph-mon[32255]: pgmap v19726: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:56:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:10] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:56:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:10 smithi022 ceph-mon[31360]: pgmap v19726: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:10 smithi022 ceph-mon[40250]: pgmap v19726: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:12.275 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:12.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:12 smithi122 ceph-mon[32255]: pgmap v19727: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:12.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:56:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:56:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:56:12.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:12 smithi022 ceph-mon[31360]: pgmap v19727: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:12 smithi022 ceph-mon[40250]: pgmap v19727: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:12.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:12.647 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:12.648 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:13.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/208390340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:13.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/208390340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:13.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/208390340' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:14 smithi122 ceph-mon[32255]: pgmap v19728: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:14.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:14 smithi022 ceph-mon[31360]: pgmap v19728: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:14 smithi022 ceph-mon[40250]: pgmap v19728: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:15.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:56:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:56:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:16.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:16 smithi122 ceph-mon[32255]: pgmap v19729: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:16 smithi022 ceph-mon[31360]: pgmap v19729: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:16.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:16 smithi022 ceph-mon[40250]: pgmap v19729: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:17.649 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:56:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:56:18.021 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:18.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:18.022 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2531334423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:18.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:18 smithi022 ceph-mon[31360]: pgmap v19730: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2531334423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:18.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:18 smithi022 ceph-mon[40250]: pgmap v19730: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:18.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2531334423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:18.439 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:18 smithi122 ceph-mon[32255]: pgmap v19730: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:56:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:20] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:56:20.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:20 smithi122 ceph-mon[32255]: pgmap v19731: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:20 smithi022 ceph-mon[31360]: pgmap v19731: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:20.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:20 smithi022 ceph-mon[40250]: pgmap v19731: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:22 smithi122 ceph-mon[32255]: pgmap v19732: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:56:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:56:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:56:22.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:22 smithi022 ceph-mon[31360]: pgmap v19732: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:22.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:22 smithi022 ceph-mon[40250]: pgmap v19732: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:23.023 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:23.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:23.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:23.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:23.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/632918171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:23.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/632918171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:23.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/632918171' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:24.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:24 smithi022 ceph-mon[31360]: pgmap v19733: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:24.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:24 smithi022 ceph-mon[40250]: pgmap v19733: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:24.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:24 smithi122 ceph-mon[32255]: pgmap v19733: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:26.107 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:56:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:56:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:26.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:26 smithi122 ceph-mon[32255]: pgmap v19734: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:26 smithi022 ceph-mon[31360]: pgmap v19734: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:26 smithi022 ceph-mon[40250]: pgmap v19734: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:56:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:56:28.398 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:28 smithi022 ceph-mon[31360]: pgmap v19735: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:28.399 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:28 smithi022 ceph-mon[40250]: pgmap v19735: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:28.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:28 smithi122 ceph-mon[32255]: pgmap v19735: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:28.769 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:28.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:28.770 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3676318868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3676318868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3676318868' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:30 smithi122 ceph-mon[32255]: pgmap v19736: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:56:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:30] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:56:30.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:30 smithi022 ceph-mon[31360]: pgmap v19736: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:30 smithi022 ceph-mon[40250]: pgmap v19736: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:32.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:56:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:56:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:56:32.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:32 smithi122 ceph-mon[32255]: pgmap v19737: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:32 smithi022 ceph-mon[31360]: pgmap v19737: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:32 smithi022 ceph-mon[40250]: pgmap v19737: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:33.771 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:34.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:34.140 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:34.141 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:34.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:34 smithi122 ceph-mon[32255]: pgmap v19738: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:34.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:34 smithi022 ceph-mon[31360]: pgmap v19738: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:34 smithi022 ceph-mon[40250]: pgmap v19738: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/774877327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:56:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:56:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:56:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:56:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/774877327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:56:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:56:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:56:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:56:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/774877327' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:56:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:56:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:56:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:56:36.119 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:56:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:56:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:36.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:36 smithi122 ceph-mon[32255]: pgmap v19739: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:36 smithi022 ceph-mon[31360]: pgmap v19739: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:36 smithi022 ceph-mon[40250]: pgmap v19739: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:37.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:56:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:56:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:38 smithi122 ceph-mon[32255]: pgmap v19740: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:38.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:38 smithi022 ceph-mon[31360]: pgmap v19740: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:38 smithi022 ceph-mon[40250]: pgmap v19740: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:39.142 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:39.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:39.516 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/706566933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:40.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:40 smithi122 ceph-mon[32255]: pgmap v19741: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:40.438 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:56:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:40] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:56:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/706566933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:40 smithi022 ceph-mon[31360]: pgmap v19741: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/706566933' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:40.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:40 smithi022 ceph-mon[40250]: pgmap v19741: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:42 smithi122 ceph-mon[32255]: pgmap v19742: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:56:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:56:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:56:42.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:42 smithi022 ceph-mon[31360]: pgmap v19742: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:42 smithi022 ceph-mon[40250]: pgmap v19742: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:44.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:44 smithi122 ceph-mon[32255]: pgmap v19743: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:44.515 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:44.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:44 smithi022 ceph-mon[31360]: pgmap v19743: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:44.516 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:44 smithi022 ceph-mon[40250]: pgmap v19743: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:44.881 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:44.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:44.882 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:45.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2889166769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2889166769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2889166769' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:46.115 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:56:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:56:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:46 smithi122 ceph-mon[32255]: pgmap v19744: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:46 smithi022 ceph-mon[31360]: pgmap v19744: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:46 smithi022 ceph-mon[40250]: pgmap v19744: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:47.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:56:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:56:48.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:48 smithi122 ceph-mon[32255]: pgmap v19745: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:48.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:48 smithi022 ceph-mon[31360]: pgmap v19745: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:48 smithi022 ceph-mon[40250]: pgmap v19745: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:49.883 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:50.255 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:50.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:50.256 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:50.256 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:50 smithi022 ceph-mon[31360]: pgmap v19746: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:50.257 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:50 smithi022 ceph-mon[40250]: pgmap v19746: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:50.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:50 smithi122 ceph-mon[32255]: pgmap v19746: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:50.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:56:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:50] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:56:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/653662287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:51.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/653662287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:51.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/653662287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:52 smithi122 ceph-mon[32255]: pgmap v19747: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:56:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:56:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:56:52.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:52 smithi022 ceph-mon[31360]: pgmap v19747: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:52 smithi022 ceph-mon[40250]: pgmap v19747: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:54 smithi122 ceph-mon[32255]: pgmap v19748: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:54.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:54 smithi022 ceph-mon[31360]: pgmap v19748: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:54 smithi022 ceph-mon[40250]: pgmap v19748: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:56:55.257 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:56:55.628 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:56:55.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:56:55.629 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:56:55.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3068024318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:55.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:56:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:56:56.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3068024318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:56.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3068024318' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:56:56.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:56.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:56 smithi122 ceph-mon[32255]: pgmap v19749: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:57.030 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:56 smithi022 ceph-mon[31360]: pgmap v19749: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:57.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:56:57.031 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:56 smithi022 ceph-mon[40250]: pgmap v19749: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:57.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:56:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:56:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:56:58 smithi122 ceph-mon[32255]: pgmap v19750: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:58.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:56:58 smithi022 ceph-mon[31360]: pgmap v19750: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:56:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:56:58 smithi022 ceph-mon[40250]: pgmap v19750: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:57:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:00] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:57:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:00 smithi122 ceph-mon[32255]: pgmap v19751: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:00 smithi022 ceph-mon[31360]: pgmap v19751: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:00 smithi022 ceph-mon[40250]: pgmap v19751: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:00.630 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:00.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:00.998 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:00.999 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:01.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1657549439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:01.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1657549439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1657549439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:02.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:57:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:57:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:57:02.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:02 smithi122 ceph-mon[32255]: pgmap v19752: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:02 smithi022 ceph-mon[31360]: pgmap v19752: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:02 smithi022 ceph-mon[40250]: pgmap v19752: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:57:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:57:03.187 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:57:03.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:57:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:57:03.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:57:03.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:57:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:57:03.281 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:57:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:57:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:04 smithi122 ceph-mon[32255]: pgmap v19753: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:57:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:04 smithi022 ceph-mon[31360]: pgmap v19753: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:57:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:04 smithi022 ceph-mon[40250]: pgmap v19753: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:06.001 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:06.116 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:57:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:57:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:06.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:06 smithi022 ceph-mon[31360]: pgmap v19754: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:06 smithi022 ceph-mon[40250]: pgmap v19754: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:06.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:06.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:06.370 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:06.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:06.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:06 smithi122 ceph-mon[32255]: pgmap v19754: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:07.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1144679586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:07.395 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1144679586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:07.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1144679586' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:57:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:57:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:08 smithi122 ceph-mon[32255]: pgmap v19755: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:08 smithi022 ceph-mon[31360]: pgmap v19755: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:08 smithi022 ceph-mon[40250]: pgmap v19755: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:10 smithi122 ceph-mon[32255]: pgmap v19756: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:10.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:57:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:10] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:57:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:10 smithi022 ceph-mon[31360]: pgmap v19756: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:10.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:10 smithi022 ceph-mon[40250]: pgmap v19756: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:11.371 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:11.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:11.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:11.739 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3768047276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3768047276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3768047276' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:12.687 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:57:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:57:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:57:13.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:12 smithi122 ceph-mon[32255]: pgmap v19757: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:13.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:12 smithi022 ceph-mon[31360]: pgmap v19757: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:13.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:12 smithi022 ceph-mon[40250]: pgmap v19757: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:14 smithi122 ceph-mon[32255]: pgmap v19758: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:14 smithi022 ceph-mon[31360]: pgmap v19758: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:14 smithi022 ceph-mon[40250]: pgmap v19758: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:16.114 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:57:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:57:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:16 smithi122 ceph-mon[32255]: pgmap v19759: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:16.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:16 smithi022 ceph-mon[31360]: pgmap v19759: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:16.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:16 smithi022 ceph-mon[40250]: pgmap v19759: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:16.740 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:17.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:17.108 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:17.109 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/320132971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/320132971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/320132971' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:57:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:57:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:18 smithi122 ceph-mon[32255]: pgmap v19760: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:18 smithi022 ceph-mon[40250]: pgmap v19760: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:18.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:18 smithi022 ceph-mon[31360]: pgmap v19760: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:20 smithi122 ceph-mon[32255]: pgmap v19761: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:20.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:57:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:20] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:57:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:20 smithi022 ceph-mon[40250]: pgmap v19761: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:20.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:20 smithi022 ceph-mon[31360]: pgmap v19761: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:22.110 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:22 smithi122 ceph-mon[32255]: pgmap v19762: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:22.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:57:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:57:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:57:22.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:22.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:22.473 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:22.474 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:22 smithi022 ceph-mon[31360]: pgmap v19762: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:22.474 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:22 smithi022 ceph-mon[40250]: pgmap v19762: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:23.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:23 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1095961397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:23 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1095961397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:23.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:23 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1095961397' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:24.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:24 smithi122 ceph-mon[32255]: pgmap v19763: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:24.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:24 smithi022 ceph-mon[31360]: pgmap v19763: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:24.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:24 smithi022 ceph-mon[40250]: pgmap v19763: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:26.116 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:57:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:57:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:26 smithi122 ceph-mon[32255]: pgmap v19764: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:26.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:26 smithi022 ceph-mon[31360]: pgmap v19764: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:26.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:26 smithi022 ceph-mon[40250]: pgmap v19764: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:27.475 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:57:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:57:27.846 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:27.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:27.847 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:28.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1468544042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:28.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1468544042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:28.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1468544042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:29.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:28 smithi122 ceph-mon[32255]: pgmap v19765: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:29.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:28 smithi022 ceph-mon[31360]: pgmap v19765: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:29.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:28 smithi022 ceph-mon[40250]: pgmap v19765: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:30 smithi122 ceph-mon[32255]: pgmap v19766: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:30.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:57:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:30] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:57:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:30 smithi022 ceph-mon[31360]: pgmap v19766: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:30 smithi022 ceph-mon[40250]: pgmap v19766: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:32.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:32 smithi122 ceph-mon[32255]: pgmap v19767: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:32.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:57:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:57:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:57:32.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:32 smithi022 ceph-mon[31360]: pgmap v19767: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:32.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:32 smithi022 ceph-mon[40250]: pgmap v19767: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:32.848 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:33.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:33.220 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:33.221 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3482993511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3482993511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:33.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3482993511' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:34.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:34 smithi022 ceph-mon[31360]: pgmap v19768: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:34.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:34 smithi022 ceph-mon[40250]: pgmap v19768: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:34.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:34 smithi122 ceph-mon[32255]: pgmap v19768: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:57:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:57:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:57:35.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:57:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:57:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:57:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:57:35.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:57:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:57:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:57:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:57:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:57:36.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:57:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:57:36.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:36.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:36 smithi022 ceph-mon[31360]: pgmap v19769: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:36.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:36.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:36 smithi022 ceph-mon[40250]: pgmap v19769: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:36 smithi122 ceph-mon[32255]: pgmap v19769: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:57:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:57:38.222 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:38 smithi122 ceph-mon[32255]: pgmap v19770: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:38 smithi022 ceph-mon[31360]: pgmap v19770: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:38 smithi022 ceph-mon[40250]: pgmap v19770: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:38.593 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:38.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:38.594 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:39 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4131578594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:39 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4131578594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:39.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:39 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4131578594' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:40 smithi122 ceph-mon[32255]: pgmap v19771: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:40.437 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:57:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:40] "GET /metrics HTTP/1.1" 200 202512 "" "Prometheus/2.18.1" 2021-12-16T04:57:40.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:40 smithi022 ceph-mon[31360]: pgmap v19771: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:40.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:40 smithi022 ceph-mon[40250]: pgmap v19771: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:42.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:42 smithi122 ceph-mon[32255]: pgmap v19772: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:42.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:57:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:57:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:57:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:42 smithi022 ceph-mon[31360]: pgmap v19772: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:42.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:42 smithi022 ceph-mon[40250]: pgmap v19772: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:43.595 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:43.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:43.963 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:43.964 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:44.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/469141917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:44.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/469141917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:44.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/469141917' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:45.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:45 smithi022 ceph-mon[31360]: pgmap v19773: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:45.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:45 smithi022 ceph-mon[40250]: pgmap v19773: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:45 smithi122 ceph-mon[32255]: pgmap v19773: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:46.119 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:57:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:57:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:46.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:46 smithi122 ceph-mon[32255]: pgmap v19774: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:46.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:46 smithi022 ceph-mon[31360]: pgmap v19774: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:46.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:46 smithi022 ceph-mon[40250]: pgmap v19774: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:57:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:57:48.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:48 smithi122 ceph-mon[32255]: pgmap v19775: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:48.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:48 smithi022 ceph-mon[31360]: pgmap v19775: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:48.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:48 smithi022 ceph-mon[40250]: pgmap v19775: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:48.965 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:49.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:49.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:49.338 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:57:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:50] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:57:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3847927461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:50.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:50 smithi122 ceph-mon[32255]: pgmap v19776: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:50.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3847927461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:50.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:50 smithi022 ceph-mon[31360]: pgmap v19776: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3847927461' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:50.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:50 smithi022 ceph-mon[40250]: pgmap v19776: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:52.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:52 smithi122 ceph-mon[32255]: pgmap v19777: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:52.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:57:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:57:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:57:52.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:52 smithi022 ceph-mon[31360]: pgmap v19777: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:52.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:52 smithi022 ceph-mon[40250]: pgmap v19777: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:54.339 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:57:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:54 smithi122 ceph-mon[32255]: pgmap v19778: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:54 smithi022 ceph-mon[31360]: pgmap v19778: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:54.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:54 smithi022 ceph-mon[40250]: pgmap v19778: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:57:54.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:57:54.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:57:54.711 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:57:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/385961107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/385961107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:55.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/385961107' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:57:56.127 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:57:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:57:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:56 smithi122 ceph-mon[32255]: pgmap v19779: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:56.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:56 smithi022 ceph-mon[31360]: pgmap v19779: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:56.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:57:56.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:56 smithi022 ceph-mon[40250]: pgmap v19779: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:57:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:57:58.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:57:58 smithi122 ceph-mon[32255]: pgmap v19780: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:57:58 smithi022 ceph-mon[31360]: pgmap v19780: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:58.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:57:58 smithi022 ceph-mon[40250]: pgmap v19780: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:57:59.713 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:00.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:00.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:00.084 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:58:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:00] "GET /metrics HTTP/1.1" 200 202513 "" "Prometheus/2.18.1" 2021-12-16T04:58:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:00 smithi122 ceph-mon[32255]: pgmap v19781: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2624652607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:00.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:00 smithi022 ceph-mon[31360]: pgmap v19781: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:00.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2624652607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:00.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:00 smithi022 ceph-mon[40250]: pgmap v19781: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:00.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2624652607' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:02.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:02 smithi122 ceph-mon[32255]: pgmap v19782: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:02.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:58:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:58:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:58:02.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:02 smithi022 ceph-mon[31360]: pgmap v19782: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:02.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:02 smithi022 ceph-mon[40250]: pgmap v19782: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:58:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:58:03.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:03 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:58:03.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:58:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:58:03.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:03 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:58:03.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:58:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:58:03.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:03 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:58:05.086 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:05.087 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:04 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:58:05.087 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:04 smithi022 ceph-mon[31360]: pgmap v19783: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:05.088 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:04 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:58:05.088 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:04 smithi022 ceph-mon[40250]: pgmap v19783: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:04 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:58:05.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:04 smithi122 ceph-mon[32255]: pgmap v19783: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:05.458 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:05.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:05.459 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:06.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/946209017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:06.220 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:58:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:58:06.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/946209017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:06.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/946209017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:07.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:06 smithi122 ceph-mon[32255]: pgmap v19784: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:07.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:07.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:06 smithi022 ceph-mon[31360]: pgmap v19784: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:07.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:06 smithi022 ceph-mon[40250]: pgmap v19784: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:58:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:58:08.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:08 smithi122 ceph-mon[32255]: pgmap v19785: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:08.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:08 smithi022 ceph-mon[31360]: pgmap v19785: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:08.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:08 smithi022 ceph-mon[40250]: pgmap v19785: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:58:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:10] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:58:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:10 smithi122 ceph-mon[32255]: pgmap v19786: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:10.460 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:10.461 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:10 smithi022 ceph-mon[31360]: pgmap v19786: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:10.461 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:10 smithi022 ceph-mon[40250]: pgmap v19786: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2021-12-16T04:58:10.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:10.828 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:10.829 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:11.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:11 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1987526101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:11.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:11 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1987526101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:11.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:11 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1987526101' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:12.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:12 smithi122 ceph-mon[32255]: pgmap v19787: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:12.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:58:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:58:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:58:12.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:12 smithi022 ceph-mon[31360]: pgmap v19787: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:12.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:12 smithi022 ceph-mon[40250]: pgmap v19787: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:14.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:14 smithi122 ceph-mon[32255]: pgmap v19788: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:14.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:14 smithi022 ceph-mon[31360]: pgmap v19788: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:14.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:14 smithi022 ceph-mon[40250]: pgmap v19788: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:16.127 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:58:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:58:16.128 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:16.128 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:16 smithi022 ceph-mon[40250]: pgmap v19789: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:16.128 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:16.128 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:16 smithi022 ceph-mon[31360]: pgmap v19789: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:16.201 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:16.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:16.202 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:16.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:16 smithi122 ceph-mon[32255]: pgmap v19789: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:17.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3220106084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:17.398 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3220106084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:17.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3220106084' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:58:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:58:18.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:18 smithi122 ceph-mon[32255]: pgmap v19790: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:18.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:18 smithi022 ceph-mon[31360]: pgmap v19790: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:18.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:18 smithi022 ceph-mon[40250]: pgmap v19790: 132 pgs: 132 active+clean; 582 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:58:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:20] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:58:21.203 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:21.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:21 smithi122 ceph-mon[32255]: pgmap v19791: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:21 smithi022 ceph-mon[31360]: pgmap v19791: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:21.677 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:21 smithi022 ceph-mon[40250]: pgmap v19791: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:21.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:21.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:21.678 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:22.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/32845937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:22.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:58:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:58:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:58:22.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/32845937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:22.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/32845937' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:23.440 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:23 smithi122 ceph-mon[32255]: pgmap v19792: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:23 smithi022 ceph-mon[31360]: pgmap v19792: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:23 smithi022 ceph-mon[40250]: pgmap v19792: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:25 smithi122 ceph-mon[32255]: pgmap v19793: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:25 smithi022 ceph-mon[31360]: pgmap v19793: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:25.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:25 smithi022 ceph-mon[40250]: pgmap v19793: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:26.119 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:58:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:58:26.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:26.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:26.944 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:26.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:26.945 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:27 smithi022 ceph-mon[31360]: pgmap v19794: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:27.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:27 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2192634711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:27 smithi022 ceph-mon[40250]: pgmap v19794: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:27.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:27 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2192634711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:27 smithi122 ceph-mon[32255]: pgmap v19794: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:27.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:27 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2192634711' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:27.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:58:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:58:29.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:29 smithi122 ceph-mon[32255]: pgmap v19795: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:29 smithi022 ceph-mon[31360]: pgmap v19795: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:29.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:29 smithi022 ceph-mon[40250]: pgmap v19795: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2021-12-16T04:58:30.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:58:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:30] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:58:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:31 smithi122 ceph-mon[32255]: pgmap v19796: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:31 smithi022 ceph-mon[31360]: pgmap v19796: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:31 smithi022 ceph-mon[40250]: pgmap v19796: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2021-12-16T04:58:31.946 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:32.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:32.317 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:32.318 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:58:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:58:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:58:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:33 smithi122 ceph-mon[32255]: pgmap v19797: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:33.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3803653540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:33 smithi022 ceph-mon[31360]: pgmap v19797: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:33.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3803653540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:33 smithi022 ceph-mon[40250]: pgmap v19797: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:33.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3803653540' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:35 smithi122 ceph-mon[32255]: pgmap v19798: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:58:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:58:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:58:35.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:58:35.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[31360]: pgmap v19798: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:58:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:58:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:58:35.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:58:35.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[40250]: pgmap v19798: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:35.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:58:35.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:58:35.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:58:35.537 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:58:36.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:58:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:58:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:36.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:37.319 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:37.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:37 smithi022 ceph-mon[40250]: pgmap v19799: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:37.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:37 smithi022 ceph-mon[31360]: pgmap v19799: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:37.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:37 smithi122 ceph-mon[32255]: pgmap v19799: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:37.687 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:37.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:37.688 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:37.689 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:58:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:58:38.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1179081851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:38.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1179081851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:38.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1179081851' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:39.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:39 smithi122 ceph-mon[32255]: pgmap v19800: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:39 smithi022 ceph-mon[31360]: pgmap v19800: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:39 smithi022 ceph-mon[40250]: pgmap v19800: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:40.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:58:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:40] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:58:41.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:41 smithi122 ceph-mon[32255]: pgmap v19801: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:41 smithi022 ceph-mon[31360]: pgmap v19801: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:41.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:41 smithi022 ceph-mon[40250]: pgmap v19801: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:58:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:58:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:58:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:43.057 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:43.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:43.058 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:43 smithi122 ceph-mon[32255]: pgmap v19802: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:43.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:43 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/820736690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:43 smithi022 ceph-mon[31360]: pgmap v19802: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:43.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:43 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/820736690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:43 smithi022 ceph-mon[40250]: pgmap v19802: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:43.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:43 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/820736690' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:45.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:45 smithi122 ceph-mon[32255]: pgmap v19803: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:45 smithi022 ceph-mon[31360]: pgmap v19803: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:45.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:45 smithi022 ceph-mon[40250]: pgmap v19803: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:46.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:58:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:58:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:46.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:47.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:47 smithi122 ceph-mon[32255]: pgmap v19804: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:47 smithi022 ceph-mon[31360]: pgmap v19804: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:47.530 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:58:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:58:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:47 smithi022 ceph-mon[40250]: pgmap v19804: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:48.059 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:48.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:48.432 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:48.433 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:49.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:49 smithi122 ceph-mon[32255]: pgmap v19805: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:49.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1597523873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:49 smithi022 ceph-mon[31360]: pgmap v19805: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:49.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1597523873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:49 smithi022 ceph-mon[40250]: pgmap v19805: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:49.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1597523873' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:50.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:58:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:50] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:58:51.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:51 smithi122 ceph-mon[32255]: pgmap v19806: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:51 smithi022 ceph-mon[31360]: pgmap v19806: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:51 smithi022 ceph-mon[40250]: pgmap v19806: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:58:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:58:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:58:53.434 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:53.434 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:53 smithi022 ceph-mon[31360]: pgmap v19807: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:53.435 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:53 smithi022 ceph-mon[40250]: pgmap v19807: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:53.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:53 smithi122 ceph-mon[32255]: pgmap v19807: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:53.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:53.802 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:53.803 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:54.168 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:54 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3528475389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:54 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3528475389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:54.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:54 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3528475389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:55.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:55 smithi122 ceph-mon[32255]: pgmap v19808: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:55.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:55 smithi022 ceph-mon[31360]: pgmap v19808: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:55 smithi022 ceph-mon[40250]: pgmap v19808: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:58:56.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:58:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:58:56.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:56.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:58:57.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:57 smithi122 ceph-mon[32255]: pgmap v19809: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:57 smithi022 ceph-mon[31360]: pgmap v19809: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:57.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:58:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:58:57.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:57 smithi022 ceph-mon[40250]: pgmap v19809: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:58.804 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:58:59.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:58:59.172 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:58:59.173 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:58:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:59 smithi122 ceph-mon[32255]: pgmap v19810: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:59.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:58:59 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2956796788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:59 smithi022 ceph-mon[31360]: pgmap v19810: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:58:59 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2956796788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:58:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:59 smithi022 ceph-mon[40250]: pgmap v19810: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:58:59.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:58:59 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2956796788' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:59:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:00] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T04:59:01.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:01 smithi122 ceph-mon[32255]: pgmap v19811: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:01 smithi022 ceph-mon[31360]: pgmap v19811: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:01.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:01 smithi022 ceph-mon[40250]: pgmap v19811: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:02.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:59:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:59:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:59:03.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:03 smithi122 ceph-mon[32255]: pgmap v19812: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:03 smithi022 ceph-mon[31360]: pgmap v19812: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:03 smithi022 ceph-mon[40250]: pgmap v19812: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:04.174 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:59:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:59:04.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:59:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:59:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:59:04.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:59:04.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:04.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:04.538 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T04:59:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T04:59:04.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T04:59:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:05 smithi022 ceph-mon[31360]: pgmap v19813: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:05 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:59:05.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:05 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1635992096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:05 smithi022 ceph-mon[40250]: pgmap v19813: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:05 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:59:05.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:05 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1635992096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:05 smithi122 ceph-mon[32255]: pgmap v19813: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:05 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T04:59:05.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:05 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1635992096' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:06.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:59:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:59:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:06.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:07.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:07 smithi122 ceph-mon[32255]: pgmap v19814: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:59:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:59:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:07 smithi022 ceph-mon[31360]: pgmap v19814: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:07 smithi022 ceph-mon[40250]: pgmap v19814: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:09.540 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:09.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:09 smithi022 ceph-mon[31360]: pgmap v19815: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:09.542 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:09 smithi022 ceph-mon[40250]: pgmap v19815: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:09.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:09 smithi122 ceph-mon[32255]: pgmap v19815: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:10 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1704167512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:10.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:59:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:10] "GET /metrics HTTP/1.1" 200 202511 "" "Prometheus/2.18.1" 2021-12-16T04:59:10.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:10 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1704167512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:10.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:10 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1704167512' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:11 smithi122 ceph-mon[32255]: pgmap v19816: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:11.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:11 smithi022 ceph-mon[31360]: pgmap v19816: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:11.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:11 smithi022 ceph-mon[40250]: pgmap v19816: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:12.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:59:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:59:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:59:13.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:13 smithi122 ceph-mon[32255]: pgmap v19817: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:13 smithi022 ceph-mon[31360]: pgmap v19817: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:13 smithi022 ceph-mon[40250]: pgmap v19817: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:14.911 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:15.284 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:15.285 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:15 smithi122 ceph-mon[32255]: pgmap v19818: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:15.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2378866224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:15 smithi022 ceph-mon[31360]: pgmap v19818: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:15.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2378866224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:15 smithi022 ceph-mon[40250]: pgmap v19818: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:15.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2378866224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:59:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:59:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:16.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:16.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:17 smithi122 ceph-mon[32255]: pgmap v19819: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:17.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:17 smithi022 ceph-mon[31360]: pgmap v19819: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:17.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:59:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:59:17.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:17 smithi022 ceph-mon[40250]: pgmap v19819: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:19.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:19 smithi122 ceph-mon[32255]: pgmap v19820: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:19.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:19 smithi022 ceph-mon[31360]: pgmap v19820: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:19.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:19 smithi022 ceph-mon[40250]: pgmap v19820: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:59:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:20] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:59:20.655 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:20.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:20.656 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:21 smithi122 ceph-mon[32255]: pgmap v19821: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:21.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:21 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2030908351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:21 smithi022 ceph-mon[31360]: pgmap v19821: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:21.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:21 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2030908351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:21 smithi022 ceph-mon[40250]: pgmap v19821: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:21.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:21 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2030908351' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:22.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:59:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:59:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:59:23.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:23 smithi122 ceph-mon[32255]: pgmap v19822: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:23.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:23 smithi022 ceph-mon[31360]: pgmap v19822: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:23.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:23 smithi022 ceph-mon[40250]: pgmap v19822: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:25.657 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:25.658 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:25 smithi022 ceph-mon[31360]: pgmap v19823: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:25.658 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:25 smithi022 ceph-mon[40250]: pgmap v19823: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:25 smithi122 ceph-mon[32255]: pgmap v19823: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:26.023 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:26.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:26.024 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:26.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:59:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:59:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:26.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:26 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/153815058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:26.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:26 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/153815058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:26.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:26 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/153815058' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:27.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:27 smithi122 ceph-mon[32255]: pgmap v19824: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:27 smithi022 ceph-mon[31360]: pgmap v19824: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:27.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:59:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:59:27.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:27 smithi022 ceph-mon[40250]: pgmap v19824: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:29.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:29 smithi122 ceph-mon[32255]: pgmap v19825: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:29.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:29 smithi022 ceph-mon[31360]: pgmap v19825: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:29.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:29 smithi022 ceph-mon[40250]: pgmap v19825: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:59:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:30] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T04:59:31.025 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:31.397 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:31 smithi122 ceph-mon[32255]: pgmap v19826: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:31.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:31 smithi022 ceph-mon[31360]: pgmap v19826: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:31.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:31 smithi022 ceph-mon[40250]: pgmap v19826: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:32.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:32 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3678788468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:32.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:59:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:59:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:59:32.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:32 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3678788468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:32.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:32 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3678788468' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:33.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:33 smithi122 ceph-mon[32255]: pgmap v19827: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:33.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:33 smithi022 ceph-mon[31360]: pgmap v19827: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:33.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:33 smithi022 ceph-mon[40250]: pgmap v19827: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:35 smithi122 ceph-mon[32255]: pgmap v19828: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:59:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:59:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:59:35.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:59:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[31360]: pgmap v19828: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:59:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:59:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:59:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:59:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[40250]: pgmap v19828: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:59:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:59:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T04:59:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T04:59:36.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:59:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:59:36.398 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:36.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:36.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:36.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:36.766 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:36.767 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:36.767 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:37 smithi122 ceph-mon[32255]: pgmap v19829: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:37.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:37 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3487236646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:59:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:59:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:37 smithi022 ceph-mon[40250]: pgmap v19829: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:37.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:37 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3487236646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:37 smithi022 ceph-mon[31360]: pgmap v19829: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:37.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:37 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3487236646' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:39.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:39 smithi122 ceph-mon[32255]: pgmap v19830: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:39.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:39 smithi022 ceph-mon[31360]: pgmap v19830: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:39.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:39 smithi022 ceph-mon[40250]: pgmap v19830: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:59:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:40] "GET /metrics HTTP/1.1" 200 202521 "" "Prometheus/2.18.1" 2021-12-16T04:59:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:41 smithi122 ceph-mon[32255]: pgmap v19831: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:41.768 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:41 smithi022 ceph-mon[31360]: pgmap v19831: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:41.768 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:41 smithi022 ceph-mon[40250]: pgmap v19831: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:42.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:42.138 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:42.139 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:42.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/254367853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/254367853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:42.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/254367853' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:42.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:59:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:59:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:59:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:43 smithi122 ceph-mon[32255]: pgmap v19832: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:43.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:43 smithi022 ceph-mon[31360]: pgmap v19832: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:43 smithi022 ceph-mon[40250]: pgmap v19832: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:45 smithi122 ceph-mon[32255]: pgmap v19833: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:45.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:45 smithi022 ceph-mon[31360]: pgmap v19833: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:45 smithi022 ceph-mon[40250]: pgmap v19833: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:59:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:59:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:46.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:47.140 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:47.392 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:47 smithi022 ceph-mon[31360]: pgmap v19834: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:47.393 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:47 smithi022 ceph-mon[40250]: pgmap v19834: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:47 smithi122 ceph-mon[32255]: pgmap v19834: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:59:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:59:48.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:48 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3782348856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:48.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:48 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3782348856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:48.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:48 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3782348856' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:49 smithi122 ceph-mon[32255]: pgmap v19835: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:49.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:49 smithi022 ceph-mon[31360]: pgmap v19835: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:49 smithi022 ceph-mon[40250]: pgmap v19835: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 04:59:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:50] "GET /metrics HTTP/1.1" 200 202520 "" "Prometheus/2.18.1" 2021-12-16T04:59:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:51 smithi122 ceph-mon[32255]: pgmap v19836: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:51.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:51 smithi022 ceph-mon[31360]: pgmap v19836: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:51 smithi022 ceph-mon[40250]: pgmap v19836: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:52.510 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 04:59:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T04:59:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T04:59:52.877 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:52.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:52.878 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:53 smithi122 ceph-mon[32255]: pgmap v19837: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:53.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:53 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/855036535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:53 smithi022 ceph-mon[31360]: pgmap v19837: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:53.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:53 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/855036535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:53.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:53 smithi022 ceph-mon[40250]: pgmap v19837: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:53.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:53 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/855036535' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:55 smithi122 ceph-mon[32255]: pgmap v19838: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:55.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:55 smithi022 ceph-mon[31360]: pgmap v19838: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:55.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:55 smithi022 ceph-mon[40250]: pgmap v19838: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 04:59:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T04:59:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:56.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T04:59:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:57 smithi122 ceph-mon[32255]: pgmap v19839: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:57.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:57 smithi022 ceph-mon[31360]: pgmap v19839: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 04:59:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:04:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T04:59:57.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:57 smithi022 ceph-mon[40250]: pgmap v19839: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T04:59:57.879 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T04:59:58.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T04:59:58.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T04:59:58.248 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T04:59:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3381976176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3381976176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3381976176' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T04:59:59.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 04:59:59 smithi122 ceph-mon[32255]: pgmap v19840: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:59.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 04:59:59 smithi022 ceph-mon[31360]: pgmap v19840: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T04:59:59.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 04:59:59 smithi022 ceph-mon[40250]: pgmap v19840: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:00 smithi122 ceph-mon[32255]: overall HEALTH_OK 2021-12-16T05:00:00.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:00:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:00] "GET /metrics HTTP/1.1" 200 202520 "" "Prometheus/2.18.1" 2021-12-16T05:00:00.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T05:00:00.228Z caller=compact.go:495 component=tsdb msg="write block" mint=1639620000000 maxt=1639627200000 ulid=01FQ0SS19JA8FN051H1Z7YM932 duration=50.464201ms 2021-12-16T05:00:00.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T05:00:00.238Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=2.663476ms 2021-12-16T05:00:00.437 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:00 smithi122 conmon[74079]: level=info ts=2021-12-16T05:00:00.415Z caller=head.go:734 component=tsdb msg="WAL checkpoint complete" first=2 last=3 duration=177.149923ms 2021-12-16T05:00:00.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:00 smithi022 ceph-mon[31360]: overall HEALTH_OK 2021-12-16T05:00:00.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:00 smithi022 ceph-mon[40250]: overall HEALTH_OK 2021-12-16T05:00:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:01 smithi122 ceph-mon[32255]: pgmap v19841: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:01.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:01 smithi022 ceph-mon[31360]: pgmap v19841: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:01.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:01 smithi022 ceph-mon[40250]: pgmap v19841: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:00:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:00:03.249 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:03 smithi022 ceph-mon[31360]: pgmap v19842: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:03 smithi022 ceph-mon[40250]: pgmap v19842: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:03.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:03.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:03.626 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:03 smithi122 ceph-mon[32255]: pgmap v19842: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:04.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:04 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2693305936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:00:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:00:04.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:04 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:00:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2693305936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:00:04.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:00:04.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:00:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2693305936' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:04.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:00:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:00:04.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:04 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:00:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:05 smithi122 ceph-mon[32255]: pgmap v19843: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:05.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:05 smithi022 ceph-mon[31360]: pgmap v19843: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:05.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:05 smithi022 ceph-mon[40250]: pgmap v19843: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:00:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:00:06.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:07 smithi122 ceph-mon[32255]: pgmap v19844: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:07.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:07 smithi022 ceph-mon[31360]: pgmap v19844: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:07.780 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:00:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:00:07.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:07 smithi022 ceph-mon[40250]: pgmap v19844: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:08.628 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:08.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:08.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:08.996 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:08.997 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:08.997 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:08 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:08.998 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:08.998 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:08 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:09.164 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:09.165 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:08 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:10.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:09 smithi122 ceph-mon[32255]: pgmap v19845: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:09 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2799085590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:10.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:09 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:09 smithi022 ceph-mon[31360]: pgmap v19845: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:10.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:09 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2799085590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:10.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:09 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:09 smithi022 ceph-mon[40250]: pgmap v19845: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:09 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2799085590' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:10.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:09 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:00:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:00:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:10] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T05:00:12.186 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:11 smithi122 ceph-mon[32255]: pgmap v19846: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:12.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:11 smithi022 ceph-mon[31360]: pgmap v19846: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:12.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:11 smithi022 ceph-mon[40250]: pgmap v19846: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:00:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:00:13.998 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:14.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:13 smithi122 ceph-mon[32255]: pgmap v19847: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:14.300 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:13 smithi022 ceph-mon[40250]: pgmap v19847: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:14.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:13 smithi022 ceph-mon[31360]: pgmap v19847: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:14.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:14.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:14.369 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:15.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:15 smithi122 ceph-mon[32255]: pgmap v19848: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:15.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:15 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1855812156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:15 smithi022 ceph-mon[31360]: pgmap v19848: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:15 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1855812156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:15 smithi022 ceph-mon[40250]: pgmap v19848: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:15.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:15 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1855812156' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:16.129 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:00:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:00:16.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:17.393 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:17 smithi022 ceph-mon[31360]: pgmap v19849: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:17.394 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:17 smithi022 ceph-mon[40250]: pgmap v19849: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:17.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:17 smithi122 ceph-mon[32255]: pgmap v19849: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:00:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:00:19.371 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:19.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:19 smithi122 ceph-mon[32255]: pgmap v19850: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:19.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:19 smithi022 ceph-mon[31360]: pgmap v19850: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:19 smithi022 ceph-mon[40250]: pgmap v19850: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:19.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:19.743 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:19.744 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:20.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:20 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3447398632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:20.436 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:00:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:20] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T05:00:20.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:20 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3447398632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:20.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:20 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3447398632' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:21.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:21 smithi122 ceph-mon[32255]: pgmap v19851: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:21.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:21 smithi022 ceph-mon[31360]: pgmap v19851: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:21.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:21 smithi022 ceph-mon[40250]: pgmap v19851: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:00:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:00:23.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:23 smithi122 ceph-mon[32255]: pgmap v19852: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:23.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:23 smithi022 ceph-mon[31360]: pgmap v19852: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:23.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:23 smithi022 ceph-mon[40250]: pgmap v19852: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:24.745 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:25 smithi122 ceph-mon[32255]: pgmap v19853: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:25.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2033988891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:25 smithi022 ceph-mon[31360]: pgmap v19853: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2033988891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:25 smithi022 ceph-mon[40250]: pgmap v19853: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2033988891' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:26.149 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:00:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:00:26.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:26.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:27 smithi122 ceph-mon[32255]: pgmap v19854: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:27 smithi022 ceph-mon[31360]: pgmap v19854: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:00:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:00:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:27 smithi022 ceph-mon[40250]: pgmap v19854: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:29 smithi122 ceph-mon[32255]: pgmap v19855: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:29.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:29 smithi022 ceph-mon[31360]: pgmap v19855: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:29 smithi022 ceph-mon[40250]: pgmap v19855: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:30.114 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:00:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:30] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T05:00:30.486 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:30.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:30.487 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:31 smithi122 ceph-mon[32255]: pgmap v19856: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:31.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:31 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/821043698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:31 smithi022 ceph-mon[31360]: pgmap v19856: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:31.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:31 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/821043698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:31 smithi022 ceph-mon[40250]: pgmap v19856: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:31.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:31 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/821043698' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:00:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:00:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:33 smithi122 ceph-mon[32255]: pgmap v19857: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:33.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:33 smithi022 ceph-mon[31360]: pgmap v19857: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:33 smithi022 ceph-mon[40250]: pgmap v19857: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:35 smithi122 ceph-mon[32255]: pgmap v19858: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:00:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:00:35.488 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:35.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[31360]: pgmap v19858: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:35.489 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:00:35.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:00:35.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:00:35.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:00:35.490 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[40250]: pgmap v19858: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:35.491 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:00:35.491 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:00:35.491 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:00:35.491 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:00:35.856 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:35.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:35.857 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:36.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:00:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:00:36.169 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:36 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3657501069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:36.169 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:36 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3657501069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:36.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:36.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:36 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3657501069' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:37 smithi122 ceph-mon[32255]: pgmap v19859: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:37 smithi022 ceph-mon[31360]: pgmap v19859: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:00:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:00:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:37 smithi022 ceph-mon[40250]: pgmap v19859: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:39 smithi122 ceph-mon[32255]: pgmap v19860: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:39 smithi022 ceph-mon[31360]: pgmap v19860: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:39 smithi022 ceph-mon[40250]: pgmap v19860: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:00:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:40] "GET /metrics HTTP/1.1" 200 202520 "" "Prometheus/2.18.1" 2021-12-16T05:00:40.859 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:41.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:41.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:41.231 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:41 smithi122 ceph-mon[32255]: pgmap v19861: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:41 smithi022 ceph-mon[31360]: pgmap v19861: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:41 smithi022 ceph-mon[40250]: pgmap v19861: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:42.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:42 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1009699349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:42.436 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:00:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:00:42.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:42 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1009699349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:42.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:42 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1009699349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:43 smithi122 ceph-mon[32255]: pgmap v19862: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:43 smithi022 ceph-mon[31360]: pgmap v19862: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:43 smithi022 ceph-mon[40250]: pgmap v19862: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:45 smithi122 ceph-mon[32255]: pgmap v19863: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:45.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:45 smithi022 ceph-mon[31360]: pgmap v19863: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:45.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:45 smithi022 ceph-mon[40250]: pgmap v19863: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:46.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:00:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:00:46.233 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:46.601 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:46.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:46.602 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:00:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:00:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:47 smithi022 ceph-mon[31360]: pgmap v19864: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:47.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:47 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3431135337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:47 smithi022 ceph-mon[40250]: pgmap v19864: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:47.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:47 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3431135337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:47 smithi122 ceph-mon[32255]: pgmap v19864: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:47.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:47 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3431135337' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:49.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:49 smithi022 ceph-mon[31360]: pgmap v19865: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:49.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:49 smithi022 ceph-mon[40250]: pgmap v19865: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:49 smithi122 ceph-mon[32255]: pgmap v19865: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:00:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:50] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T05:00:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:51 smithi022 ceph-mon[31360]: pgmap v19866: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:51 smithi022 ceph-mon[40250]: pgmap v19866: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:51.603 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:51 smithi122 ceph-mon[32255]: pgmap v19866: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:52.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:52 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/150318977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:52.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:52 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/150318977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:52.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:52 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/150318977' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:52.686 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:00:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:00:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:00:53.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:53 smithi022 ceph-mon[31360]: pgmap v19867: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:53.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:53 smithi022 ceph-mon[40250]: pgmap v19867: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:53 smithi122 ceph-mon[32255]: pgmap v19867: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:55 smithi022 ceph-mon[31360]: pgmap v19868: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:55 smithi022 ceph-mon[40250]: pgmap v19868: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:55 smithi122 ceph-mon[32255]: pgmap v19868: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:56.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:00:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:00:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:56.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:00:56.972 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:00:57.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:57 smithi022 ceph-mon[31360]: pgmap v19869: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:57.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:57 smithi022 ceph-mon[40250]: pgmap v19869: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:57.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:00:57.346 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:00:57.347 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:00:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:57 smithi122 ceph-mon[32255]: pgmap v19869: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:00:57.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:00:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:00:58.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:58 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/342325137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:58.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:58 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/342325137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:58.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:58 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/342325137' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:00:59.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:00:59 smithi022 ceph-mon[31360]: pgmap v19870: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:00:59 smithi022 ceph-mon[40250]: pgmap v19870: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:00:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:00:59 smithi122 ceph-mon[32255]: pgmap v19870: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:01:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:00] "GET /metrics HTTP/1.1" 200 202516 "" "Prometheus/2.18.1" 2021-12-16T05:01:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:01 smithi022 ceph-mon[31360]: pgmap v19871: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:01.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:01 smithi022 ceph-mon[40250]: pgmap v19871: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:01 smithi122 ceph-mon[32255]: pgmap v19871: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:02.348 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:01:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:01:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:01:02.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:02.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:02.719 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:03 smithi022 ceph-mon[31360]: pgmap v19872: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:03.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:03 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3354623958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:03 smithi022 ceph-mon[40250]: pgmap v19872: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:03.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:03 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3354623958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:03 smithi122 ceph-mon[32255]: pgmap v19872: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:03.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:03 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3354623958' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:05.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:05 smithi022 ceph-mon[31360]: pgmap v19873: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:05 smithi022 ceph-mon[40250]: pgmap v19873: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:05 smithi122 ceph-mon[32255]: pgmap v19873: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:06.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:01:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:01:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:07.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:07 smithi022 ceph-mon[31360]: pgmap v19874: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:07.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:01:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:01:07.536 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:07 smithi022 ceph-mon[40250]: pgmap v19874: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:07 smithi122 ceph-mon[32255]: pgmap v19874: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:07.720 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:08.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:08.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:08.091 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:08.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:08 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/625967472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:08.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:08 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/625967472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:08.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:08 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/625967472' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[31360]: pgmap v19875: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:01:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:01:09.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:01:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[40250]: pgmap v19875: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:01:09.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:01:09.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:09 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:01:09.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:09 smithi122 ceph-mon[32255]: pgmap v19875: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:09.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:01:09.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:01:09.623 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:09 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:01:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:01:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:10] "GET /metrics HTTP/1.1" 200 202520 "" "Prometheus/2.18.1" 2021-12-16T05:01:10.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:01:11.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:01:11.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:01:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:11 smithi122 ceph-mon[32255]: pgmap v19876: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:11 smithi022 ceph-mon[31360]: pgmap v19876: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:11 smithi022 ceph-mon[40250]: pgmap v19876: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:01:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:01:12.316Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:01:13.094 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:13.468 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:13.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:13.469 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:13.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:13 smithi022 ceph-mon[31360]: pgmap v19877: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:13.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2473964734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:13 smithi022 ceph-mon[40250]: pgmap v19877: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:13.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2473964734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:13 smithi122 ceph-mon[32255]: pgmap v19877: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:13.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2473964734' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:15.920 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:15 smithi122 ceph-mon[32255]: pgmap v19878: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:16.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:15 smithi022 ceph-mon[31360]: pgmap v19878: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:16.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:15 smithi022 ceph-mon[40250]: pgmap v19878: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:16.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:01:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:01:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:17.669 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:01:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:01:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:17 smithi122 ceph-mon[32255]: pgmap v19879: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:18.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:17 smithi022 ceph-mon[31360]: pgmap v19879: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:18.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:17 smithi022 ceph-mon[40250]: pgmap v19879: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:18.470 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:18.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:18.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:18.839 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:19 smithi122 ceph-mon[32255]: pgmap v19880: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:19.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:19 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2099266852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:19 smithi022 ceph-mon[31360]: pgmap v19880: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:20.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:19 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2099266852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:19 smithi022 ceph-mon[40250]: pgmap v19880: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:19 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2099266852' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:01:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:20] "GET /metrics HTTP/1.1" 200 202522 "" "Prometheus/2.18.1" 2021-12-16T05:01:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:21 smithi122 ceph-mon[32255]: pgmap v19881: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:22.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:21 smithi022 ceph-mon[31360]: pgmap v19881: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:22.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:21 smithi022 ceph-mon[40250]: pgmap v19881: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:01:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:01:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:01:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:23 smithi122 ceph-mon[32255]: pgmap v19882: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:23 smithi022 ceph-mon[31360]: pgmap v19882: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:23 smithi022 ceph-mon[40250]: pgmap v19882: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:24.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:24.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:24.208 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:25 smithi122 ceph-mon[32255]: pgmap v19883: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:25.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:25 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/749139126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:25 smithi022 ceph-mon[31360]: pgmap v19883: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:25.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:25 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/749139126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:25 smithi022 ceph-mon[40250]: pgmap v19883: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:25 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/749139126' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:26.185 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:26.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:01:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:01:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:27.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:27 smithi122 ceph-mon[32255]: pgmap v19884: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:27.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:01:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:01:27.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:27 smithi022 ceph-mon[31360]: pgmap v19884: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:27 smithi022 ceph-mon[40250]: pgmap v19884: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:29.209 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:29.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:29 smithi122 ceph-mon[32255]: pgmap v19885: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:29.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:29 smithi022 ceph-mon[31360]: pgmap v19885: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:29.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:29 smithi022 ceph-mon[40250]: pgmap v19885: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:29.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:29.581 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:29.582 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:01:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:30] "GET /metrics HTTP/1.1" 200 202522 "" "Prometheus/2.18.1" 2021-12-16T05:01:30.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:30 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1267695555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:30.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:30 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1267695555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:30.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:30 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1267695555' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:31.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:31 smithi122 ceph-mon[32255]: pgmap v19886: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:31 smithi022 ceph-mon[31360]: pgmap v19886: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:31 smithi022 ceph-mon[40250]: pgmap v19886: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:01:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:01:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:01:33.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:33 smithi122 ceph-mon[32255]: pgmap v19887: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:33 smithi022 ceph-mon[31360]: pgmap v19887: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:33 smithi022 ceph-mon[40250]: pgmap v19887: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:34.583 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:34.951 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:34.952 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:34.953 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:35.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[40250]: pgmap v19888: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:01:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:01:35.279 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:01:35.280 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/78687802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:35.280 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[31360]: pgmap v19888: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:01:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:01:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:01:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:01:35.281 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:35 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/78687802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:35 smithi122 ceph-mon[32255]: pgmap v19888: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:35.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:01:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:01:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:01:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:01:35.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:35 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/78687802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:36.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:36.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:01:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:01:36.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:36.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:37.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:37 smithi122 ceph-mon[32255]: pgmap v19889: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:37.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:37 smithi022 ceph-mon[31360]: pgmap v19889: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:37.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:01:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:01:37.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:37 smithi022 ceph-mon[40250]: pgmap v19889: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:39.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:39 smithi122 ceph-mon[32255]: pgmap v19890: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:39.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:39 smithi022 ceph-mon[31360]: pgmap v19890: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:39.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:39 smithi022 ceph-mon[40250]: pgmap v19890: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:39.953 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:01:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:40] "GET /metrics HTTP/1.1" 200 202514 "" "Prometheus/2.18.1" 2021-12-16T05:01:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:41 smithi122 ceph-mon[32255]: pgmap v19891: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:41.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:41 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/322347731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:41.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:41 smithi022 ceph-mon[31360]: pgmap v19891: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:41.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:41 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/322347731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:41 smithi022 ceph-mon[40250]: pgmap v19891: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:41.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:41 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/322347731' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:01:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:01:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:01:43.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:43 smithi122 ceph-mon[32255]: pgmap v19892: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:43.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:43 smithi022 ceph-mon[31360]: pgmap v19892: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:43.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:43 smithi022 ceph-mon[40250]: pgmap v19892: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:45.328 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:45.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:45 smithi122 ceph-mon[32255]: pgmap v19893: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:45.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:45 smithi022 ceph-mon[40250]: pgmap v19893: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:45.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:45 smithi022 ceph-mon[31360]: pgmap v19893: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:45.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:45.696 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:45.697 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:46.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:01:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:01:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:46 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2390673263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:46.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:46.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:46 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2390673263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:46.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:46 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2390673263' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:46.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:47.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:47 smithi122 ceph-mon[32255]: pgmap v19894: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:47.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:47 smithi022 ceph-mon[31360]: pgmap v19894: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:47.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:01:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:01:47.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:47 smithi022 ceph-mon[40250]: pgmap v19894: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:49.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:49 smithi022 ceph-mon[31360]: pgmap v19895: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:49.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:49 smithi022 ceph-mon[40250]: pgmap v19895: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:49 smithi122 ceph-mon[32255]: pgmap v19895: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:01:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:50] "GET /metrics HTTP/1.1" 200 202520 "" "Prometheus/2.18.1" 2021-12-16T05:01:50.698 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:51.066 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:51.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:51.067 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:51.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:51 smithi022 ceph-mon[31360]: pgmap v19896: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:51.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:51 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1107936598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:51 smithi022 ceph-mon[40250]: pgmap v19896: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:51.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:51 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1107936598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:51 smithi122 ceph-mon[32255]: pgmap v19896: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:51 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1107936598' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:01:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:01:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:01:53.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:53 smithi022 ceph-mon[31360]: pgmap v19897: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:53.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:53 smithi022 ceph-mon[40250]: pgmap v19897: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:53 smithi122 ceph-mon[32255]: pgmap v19897: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:55 smithi022 ceph-mon[31360]: pgmap v19898: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:55 smithi022 ceph-mon[40250]: pgmap v19898: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:55 smithi122 ceph-mon[32255]: pgmap v19898: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:56.068 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:01:56.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:56.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:01:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:01:56.439 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:01:56.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:01:56.440 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:01:56.441 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:56.441 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:01:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:01:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:01:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:57 smithi022 ceph-mon[31360]: pgmap v19899: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:57.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:57 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2494664447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:57 smithi022 ceph-mon[40250]: pgmap v19899: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:57.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:57 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2494664447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:57 smithi122 ceph-mon[32255]: pgmap v19899: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:01:57.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:57 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2494664447' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:01:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:01:59 smithi022 ceph-mon[31360]: pgmap v19900: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:59.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:01:59 smithi022 ceph-mon[40250]: pgmap v19900: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:01:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:01:59 smithi122 ceph-mon[32255]: pgmap v19900: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:02:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:00] "GET /metrics HTTP/1.1" 200 202520 "" "Prometheus/2.18.1" 2021-12-16T05:02:01.441 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:01 smithi022 ceph-mon[40250]: pgmap v19901: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:01.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:01 smithi022 ceph-mon[31360]: pgmap v19901: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:01 smithi122 ceph-mon[32255]: pgmap v19901: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:01.811 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:01.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:01.812 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:02.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:02 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1183364357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:02.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:02 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1183364357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:02.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:02 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1183364357' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:02:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:02:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:02:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:03 smithi022 ceph-mon[31360]: pgmap v19902: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:03.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:03 smithi022 ceph-mon[40250]: pgmap v19902: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:03 smithi122 ceph-mon[32255]: pgmap v19902: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:05.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:05 smithi022 ceph-mon[31360]: pgmap v19903: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:05.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:05 smithi022 ceph-mon[40250]: pgmap v19903: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:05 smithi122 ceph-mon[32255]: pgmap v19903: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:06.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:02:05 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:02:06.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:06.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:06.813 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:07.185 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:07.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:02:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:02:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:07 smithi022 ceph-mon[31360]: pgmap v19904: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:07.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:07 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3815821661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:07.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:07 smithi022 ceph-mon[40250]: pgmap v19904: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:07.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:07 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3815821661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:07 smithi122 ceph-mon[32255]: pgmap v19904: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:07 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3815821661' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:09.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:09 smithi022 ceph-mon[31360]: pgmap v19905: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:09.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:09 smithi022 ceph-mon[40250]: pgmap v19905: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:09 smithi122 ceph-mon[32255]: pgmap v19905: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:02:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:10] "GET /metrics HTTP/1.1" 200 202519 "" "Prometheus/2.18.1" 2021-12-16T05:02:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:02:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:02:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:02:10.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:10 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:02:10.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:02:10.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:02:10.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:02:10.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:02:10.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:02:10.532 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:02:10.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:02:10.533 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:10 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:02:11.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:11 smithi022 ceph-mon[31360]: pgmap v19906: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:11.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:11 smithi022 ceph-mon[40250]: pgmap v19906: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:11.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:11 smithi122 ceph-mon[32255]: pgmap v19906: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:12.186 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:12.558 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:12.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:12.559 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:12.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:02:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:02:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:02:13.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:13 smithi022 ceph-mon[31360]: pgmap v19907: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:13.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:13 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3234465188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:13 smithi022 ceph-mon[40250]: pgmap v19907: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:13.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:13 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3234465188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:13 smithi122 ceph-mon[32255]: pgmap v19907: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:13.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:13 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3234465188' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:15.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:15 smithi022 ceph-mon[31360]: pgmap v19908: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:15.530 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:15 smithi022 ceph-mon[40250]: pgmap v19908: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:15.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:15 smithi122 ceph-mon[32255]: pgmap v19908: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:16.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:02:15 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:02:16.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:16.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:16.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:17.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:17 smithi022 ceph-mon[31360]: pgmap v19909: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:17.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:02:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:02:17.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:17 smithi022 ceph-mon[40250]: pgmap v19909: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:17.561 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:17.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:17 smithi122 ceph-mon[32255]: pgmap v19909: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:17.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:17.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:17.935 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:18.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:18 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1520249894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:18.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:18 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1520249894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:18.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:18 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1520249894' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:19.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:19 smithi022 ceph-mon[31360]: pgmap v19910: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:19.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:19 smithi022 ceph-mon[40250]: pgmap v19910: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:19.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:19 smithi122 ceph-mon[32255]: pgmap v19910: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:02:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:20] "GET /metrics HTTP/1.1" 200 202521 "" "Prometheus/2.18.1" 2021-12-16T05:02:21.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:21 smithi022 ceph-mon[31360]: pgmap v19911: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:21.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:21 smithi022 ceph-mon[40250]: pgmap v19911: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:21.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:21 smithi122 ceph-mon[32255]: pgmap v19911: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:22.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:02:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:02:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:02:22.937 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:23.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:23.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:23.309 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:23.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:23 smithi022 ceph-mon[31360]: pgmap v19912: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:23.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:23 smithi022 ceph-mon[40250]: pgmap v19912: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:23.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:23 smithi122 ceph-mon[32255]: pgmap v19912: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:24.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:24 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3110268365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:24.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:24 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3110268365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:24.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:24 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3110268365' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:25.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:25 smithi022 ceph-mon[31360]: pgmap v19913: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:25.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:25 smithi022 ceph-mon[40250]: pgmap v19913: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:25.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:25 smithi122 ceph-mon[32255]: pgmap v19913: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:26.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:02:25 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:02:26.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:26.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:26.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:27.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:27 smithi022 ceph-mon[31360]: pgmap v19914: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:27.529 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:02:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:02:27.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:27 smithi022 ceph-mon[40250]: pgmap v19914: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:27.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:27 smithi122 ceph-mon[32255]: pgmap v19914: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:28.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:28.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:28.673 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:29.530 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:29 smithi022 ceph-mon[31360]: pgmap v19915: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:29.531 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:29 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4004516283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:29 smithi022 ceph-mon[40250]: pgmap v19915: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:29.531 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:29 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4004516283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:29 smithi122 ceph-mon[32255]: pgmap v19915: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:29.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:29 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4004516283' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:30.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:02:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:30] "GET /metrics HTTP/1.1" 200 202521 "" "Prometheus/2.18.1" 2021-12-16T05:02:31.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:31 smithi022 ceph-mon[31360]: pgmap v19916: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:31.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:31 smithi022 ceph-mon[40250]: pgmap v19916: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:31.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:31 smithi122 ceph-mon[32255]: pgmap v19916: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:02:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:02:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:02:33.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:33 smithi022 ceph-mon[31360]: pgmap v19917: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:33.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:33 smithi022 ceph-mon[40250]: pgmap v19917: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:33.675 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:33.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:33 smithi122 ceph-mon[32255]: pgmap v19917: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:34.047 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:34.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:34.048 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:34.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:34 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/775617061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:34.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:34 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/775617061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:34.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:34 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/775617061' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:35 smithi122 ceph-mon[32255]: pgmap v19918: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:35.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:02:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:02:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:02:35.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:02:35.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[31360]: pgmap v19918: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:02:35.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:02:35.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:02:35.780 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[40250]: pgmap v19918: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:02:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:02:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:02:35.781 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:02:36.285 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:02:35 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:02:36.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:36.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:36.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:37.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:37 smithi122 ceph-mon[32255]: pgmap v19919: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:37.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:37 smithi022 ceph-mon[31360]: pgmap v19919: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:37.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:02:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:02:37.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:37 smithi022 ceph-mon[40250]: pgmap v19919: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:39.049 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:39.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:39.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:39.430 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:39.431 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:39 smithi022 ceph-mon[31360]: pgmap v19920: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:39.431 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:39 smithi022 ceph-mon[40250]: pgmap v19920: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:39.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:39 smithi122 ceph-mon[32255]: pgmap v19920: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:40.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:02:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:40] "GET /metrics HTTP/1.1" 200 202515 "" "Prometheus/2.18.1" 2021-12-16T05:02:40.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:40 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/4184556507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:40.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:40 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/4184556507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:40.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:40 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/4184556507' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:41.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:41 smithi122 ceph-mon[32255]: pgmap v19921: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:41.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:41 smithi022 ceph-mon[31360]: pgmap v19921: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:41.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:41 smithi022 ceph-mon[40250]: pgmap v19921: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:02:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:02:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:02:43.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:43 smithi122 ceph-mon[32255]: pgmap v19922: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:43 smithi022 ceph-mon[31360]: pgmap v19922: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:43.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:43 smithi022 ceph-mon[40250]: pgmap v19922: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:44.431 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:44.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:44.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:44.805 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:45 smithi122 ceph-mon[32255]: pgmap v19923: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:45.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:45 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3232344445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:45 smithi022 ceph-mon[31360]: pgmap v19923: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:45.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:45 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3232344445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:45 smithi022 ceph-mon[40250]: pgmap v19923: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:45.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:45 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3232344445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:46.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:02:45 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:02:46.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:46.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:46.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:47.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:47 smithi122 ceph-mon[32255]: pgmap v19924: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:47.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:47 smithi022 ceph-mon[31360]: pgmap v19924: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:47.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:02:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:02:47.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:47 smithi022 ceph-mon[40250]: pgmap v19924: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:49.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:49 smithi122 ceph-mon[32255]: pgmap v19925: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:49.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:49 smithi022 ceph-mon[31360]: pgmap v19925: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:49.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:49 smithi022 ceph-mon[40250]: pgmap v19925: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:49.807 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:50.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:50.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:50.175 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:50.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:50 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/793121079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:02:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:50] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T05:02:50.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:50 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/793121079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:50.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:50 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/793121079' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:51.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:51 smithi122 ceph-mon[32255]: pgmap v19926: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:51.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:51 smithi022 ceph-mon[31360]: pgmap v19926: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:51.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:51 smithi022 ceph-mon[40250]: pgmap v19926: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:02:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:02:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:02:53.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:53 smithi122 ceph-mon[32255]: pgmap v19927: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:53.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:53 smithi022 ceph-mon[31360]: pgmap v19927: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:53.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:53 smithi022 ceph-mon[40250]: pgmap v19927: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:55.177 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:02:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:55 smithi022 ceph-mon[31360]: pgmap v19928: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:55 smithi022 ceph-mon[40250]: pgmap v19928: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:55.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:02:55.549 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:02:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:02:55.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:55 smithi122 ceph-mon[32255]: pgmap v19928: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:56.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:02:55 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:02:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:56 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3362657389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:56.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:56 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3362657389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:56.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:56 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3362657389' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:02:56.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:02:57.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:57 smithi122 ceph-mon[32255]: pgmap v19929: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:57.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:57 smithi022 ceph-mon[31360]: pgmap v19929: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:57.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:02:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:02:57.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:57 smithi022 ceph-mon[40250]: pgmap v19929: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:02:59.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:02:59 smithi122 ceph-mon[32255]: pgmap v19930: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:59.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:02:59 smithi022 ceph-mon[31360]: pgmap v19930: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:02:59.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:02:59 smithi022 ceph-mon[40250]: pgmap v19930: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:03:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:00] "GET /metrics HTTP/1.1" 200 202510 "" "Prometheus/2.18.1" 2021-12-16T05:03:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:00.914 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:00.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:00.915 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:01 smithi122 ceph-mon[32255]: pgmap v19931: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:01 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2187565082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:01 smithi022 ceph-mon[31360]: pgmap v19931: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:01.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:01 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2187565082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:01 smithi022 ceph-mon[40250]: pgmap v19931: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:01.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:01 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2187565082' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:02.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:03:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:03:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:03:03.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:03 smithi122 ceph-mon[32255]: pgmap v19932: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:03.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:03 smithi022 ceph-mon[31360]: pgmap v19932: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:03.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:03 smithi022 ceph-mon[40250]: pgmap v19932: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:05.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:05 smithi122 ceph-mon[32255]: pgmap v19933: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:05.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:05 smithi022 ceph-mon[31360]: pgmap v19933: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:05.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:05 smithi022 ceph-mon[40250]: pgmap v19933: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:05.917 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:06.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:06.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:06.286 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:06.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:03:06 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:03:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:06 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:06.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:06 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1344178712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:06 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:06.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:06 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1344178712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:06 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:06.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:06 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1344178712' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:07.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:07 smithi122 ceph-mon[32255]: pgmap v19934: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:07.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:07 smithi022 ceph-mon[31360]: pgmap v19934: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:07.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:03:07 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:03:07.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:07 smithi022 ceph-mon[40250]: pgmap v19934: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:09.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:09 smithi122 ceph-mon[32255]: pgmap v19935: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:09.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:09 smithi022 ceph-mon[31360]: pgmap v19935: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:09.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:09 smithi022 ceph-mon[40250]: pgmap v19935: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:10.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:03:10 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:10] "GET /metrics HTTP/1.1" 200 202517 "" "Prometheus/2.18.1" 2021-12-16T05:03:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:03:10.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:03:10.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:10 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:03:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:03:10.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:03:10.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:10 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:03:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-12-16T05:03:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-12-16T05:03:10.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:10 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2021-12-16T05:03:11.289 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:11.661 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:11.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:11.662 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:11 smithi122 ceph-mon[32255]: pgmap v19936: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:11.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:11 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:03:12.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:11 smithi022 ceph-mon[31360]: pgmap v19936: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:12.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:11 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:03:12.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:11 smithi022 ceph-mon[40250]: pgmap v19936: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:12.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:11 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' 2021-12-16T05:03:12.593 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:03:12 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:03:12.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:03:12.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:12 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/552960445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:13.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:12 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/552960445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:13.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:12 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/552960445' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:13.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:13 smithi122 ceph-mon[32255]: pgmap v19937: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:14.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:13 smithi022 ceph-mon[31360]: pgmap v19937: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:14.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:13 smithi022 ceph-mon[40250]: pgmap v19937: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:15.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:15 smithi122 ceph-mon[32255]: pgmap v19938: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:16.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:15 smithi022 ceph-mon[31360]: pgmap v19938: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:16.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:15 smithi022 ceph-mon[40250]: pgmap v19938: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:16.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:03:16 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:03:16.664 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:16.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:16 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:17.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:16 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:17.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:16 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:17.030 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:17 smithi022 ceph-mon[31360]: pgmap v19939: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:17.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:17 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1711978912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:17.779 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:03:17 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:03:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:17 smithi022 ceph-mon[40250]: pgmap v19939: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:17.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:17 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1711978912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:17 smithi122 ceph-mon[32255]: pgmap v19939: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:17.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:17 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1711978912' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:19.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:19 smithi122 ceph-mon[32255]: pgmap v19940: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:20.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:19 smithi022 ceph-mon[31360]: pgmap v19940: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:20.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:19 smithi022 ceph-mon[40250]: pgmap v19940: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:20.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:03:20 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:20] "GET /metrics HTTP/1.1" 200 202517 "" "Prometheus/2.18.1" 2021-12-16T05:03:21.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:21 smithi122 ceph-mon[32255]: pgmap v19941: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:22.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:21 smithi022 ceph-mon[31360]: pgmap v19941: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:22.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:21 smithi022 ceph-mon[40250]: pgmap v19941: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:22.032 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:22.402 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:22.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:22.403 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:22.615 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:03:22 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:03:22.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:03:22.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:22 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/514375152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:22.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:22 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/514375152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:22.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:22 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/514375152' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:23.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:23 smithi122 ceph-mon[32255]: pgmap v19942: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:24.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:23 smithi022 ceph-mon[31360]: pgmap v19942: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:24.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:23 smithi022 ceph-mon[40250]: pgmap v19942: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:25.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:25 smithi122 ceph-mon[32255]: pgmap v19943: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:26.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:25 smithi022 ceph-mon[31360]: pgmap v19943: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:26.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:25 smithi022 ceph-mon[40250]: pgmap v19943: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:26.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:03:26 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:03:26.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:26 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:27.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:26 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:27.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:26 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:27.404 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:27.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:27.777 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:27.778 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:27.778 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:03:27 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:03:27.779 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:27 smithi022 ceph-mon[31360]: pgmap v19944: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:27.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:27 smithi022 ceph-mon[40250]: pgmap v19944: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:27.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:27 smithi122 ceph-mon[32255]: pgmap v19944: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:28.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:28 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2458960509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:29.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:28 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2458960509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:29.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:28 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2458960509' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:29.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:29 smithi122 ceph-mon[32255]: pgmap v19945: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:30.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:29 smithi022 ceph-mon[31360]: pgmap v19945: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:30.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:29 smithi022 ceph-mon[40250]: pgmap v19945: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:30.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:03:30 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:30] "GET /metrics HTTP/1.1" 200 202517 "" "Prometheus/2.18.1" 2021-12-16T05:03:31.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:31 smithi122 ceph-mon[32255]: pgmap v19946: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:32.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:31 smithi022 ceph-mon[31360]: pgmap v19946: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:32.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:31 smithi022 ceph-mon[40250]: pgmap v19946: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:32.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:03:32 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:03:32.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:03:32.779 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:33.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:33 smithi122 ceph-mon[32255]: pgmap v19947: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:33.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:33 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3708930017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:33 smithi022 ceph-mon[31360]: pgmap v19947: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:34.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:33 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3708930017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:33 smithi022 ceph-mon[40250]: pgmap v19947: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:34.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:33 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3708930017' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:35 smithi122 ceph-mon[32255]: pgmap v19948: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:03:35.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:03:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:03:35.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:35 smithi122 ceph-mon[32255]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:03:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[31360]: pgmap v19948: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:36.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:03:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:03:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:03:36.029 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[31360]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:03:36.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[40250]: pgmap v19948: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:03:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 v1:172.21.15.122:0/3108874494' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:03:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2021-12-16T05:03:36.030 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:35 smithi022 ceph-mon[40250]: from='mgr.34556 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2021-12-16T05:03:36.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:03:36 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:03:36.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:36 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:37.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:36 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:37.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:36 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:37.646 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:03:37 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:03:37.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:37 smithi122 ceph-mon[32255]: pgmap v19949: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:38.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:37 smithi022 ceph-mon[31360]: pgmap v19949: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:38.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:37 smithi022 ceph-mon[40250]: pgmap v19949: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:38.153 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:38.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:38.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:38.523 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:38.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:38 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2562733407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:38.779 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:38 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2562733407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:38.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:38 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2562733407' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:39.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:39 smithi122 ceph-mon[32255]: pgmap v19950: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:40.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:39 smithi022 ceph-mon[31360]: pgmap v19950: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:40.029 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:39 smithi022 ceph-mon[40250]: pgmap v19950: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:40.434 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:03:40 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:40] "GET /metrics HTTP/1.1" 200 202521 "" "Prometheus/2.18.1" 2021-12-16T05:03:41.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:41 smithi122 ceph-mon[32255]: pgmap v19951: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:42.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:41 smithi022 ceph-mon[31360]: pgmap v19951: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:42.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:41 smithi022 ceph-mon[40250]: pgmap v19951: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:42.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:03:42 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:03:42.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:03:43.525 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:43.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:43 smithi022 ceph-mon[31360]: pgmap v19952: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:43.778 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:43 smithi022 ceph-mon[40250]: pgmap v19952: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:43.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:43.899 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:43.900 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:43.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:43 smithi122 ceph-mon[32255]: pgmap v19952: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:44.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:44 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/1317268312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:45.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:44 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/1317268312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:45.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:44 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/1317268312' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:45.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:45 smithi122 ceph-mon[32255]: pgmap v19953: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:46.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:45 smithi022 ceph-mon[31360]: pgmap v19953: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:46.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:45 smithi022 ceph-mon[40250]: pgmap v19953: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:46.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:03:46 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:03:46.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:46 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:47.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:46 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:47.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:46 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:47.669 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:03:47 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:03:47.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:47 smithi122 ceph-mon[32255]: pgmap v19954: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:48.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:47 smithi022 ceph-mon[31360]: pgmap v19954: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:48.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:47 smithi022 ceph-mon[40250]: pgmap v19954: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:48.901 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:49.271 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:49.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:49.272 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:49 smithi122 ceph-mon[32255]: pgmap v19955: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:49.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:49 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/790649299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:50.034 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:49 smithi022 ceph-mon[31360]: pgmap v19955: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:50.035 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:49 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/790649299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:50.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:49 smithi022 ceph-mon[40250]: pgmap v19955: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:50.035 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:49 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/790649299' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:50.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:03:50 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:50] "GET /metrics HTTP/1.1" 200 202519 "" "Prometheus/2.18.1" 2021-12-16T05:03:51.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:51 smithi122 ceph-mon[32255]: pgmap v19956: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:52.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:51 smithi022 ceph-mon[31360]: pgmap v19956: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:52.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:51 smithi022 ceph-mon[40250]: pgmap v19956: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:52.685 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:03:52 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:03:52.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:03:53.934 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:53 smithi122 ceph-mon[32255]: pgmap v19957: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:54.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:53 smithi022 ceph-mon[31360]: pgmap v19957: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:54.028 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:53 smithi022 ceph-mon[40250]: pgmap v19957: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:54.273 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:54.643 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:03:54.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:03:54.644 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:03:55.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:55 smithi122 ceph-mon[32255]: pgmap v19958: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:55.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:55 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/2494069165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:55 smithi022 ceph-mon[31360]: pgmap v19958: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:55.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:55 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/2494069165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:55 smithi022 ceph-mon[40250]: pgmap v19958: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:55.529 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:55 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/2494069165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:03:56.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:56 smithi122 ceph-mon[32255]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:56.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi122.stdout:Dec 16 05:03:56 smithi122 conmon[89367]: debug there is no tcmu-runner data available 2021-12-16T05:03:56.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:56 smithi022 ceph-mon[31360]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:56.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:56 smithi022 ceph-mon[40250]: from='client.17766 v1:172.21.15.122:0/2252059031' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-12-16T05:03:57.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:57 smithi122 ceph-mon[32255]: pgmap v19959: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:57.528 INFO:journalctl@ceph.mgr.y.smithi022.stdout:Dec 16 05:03:57 smithi022 conmon[31781]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.18.1" 2021-12-16T05:03:57.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:57 smithi022 ceph-mon[40250]: pgmap v19959: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:57.529 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:57 smithi022 ceph-mon[31360]: pgmap v19959: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:03:59.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:03:59 smithi022 ceph-mon[40250]: pgmap v19960: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:59.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:03:59 smithi022 ceph-mon[31360]: pgmap v19960: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:03:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:03:59.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:03:59 smithi122 ceph-mon[32255]: pgmap v19960: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-12-16T05:04:00.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ NK=AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== 2021-12-16T05:04:00.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ '[' AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== == AQDqIbphxMyNMBAAYj1fwpEjhZ7zPmdkXj/Bhw== ']' 2021-12-16T05:04:00.011 INFO:teuthology.orchestra.run.smithi022.stderr:+ sleep 5 2021-12-16T05:04:00.278 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:04:00 smithi022 ceph-mon[31360]: from='client.? v1:172.21.15.22:0/3133915056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:04:00.278 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:04:00 smithi022 ceph-mon[40250]: from='client.? v1:172.21.15.22:0/3133915056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:04:00.435 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Dec 16 05:04:00 smithi122 conmon[33927]: ::ffff:172.21.15.122 - - [16/Dec/2021:05:04:00] "GET /metrics HTTP/1.1" 200 202519 "" "Prometheus/2.18.1" 2021-12-16T05:04:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:04:00 smithi122 ceph-mon[32255]: from='client.? v1:172.21.15.22:0/3133915056' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2021-12-16T05:04:01.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:04:01 smithi022 ceph-mon[31360]: pgmap v19961: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:04:01.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:04:01 smithi022 ceph-mon[40250]: pgmap v19961: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:04:01.685 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:04:01 smithi122 ceph-mon[32255]: pgmap v19961: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:04:02.684 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Dec 16 05:04:02 smithi122 conmon[74079]: level=warn ts=2021-12-16T05:04:02.315Z caller=manager.go:577 component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.22:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2021-12-16T05:04:03.528 INFO:journalctl@ceph.mon.a.smithi022.stdout:Dec 16 05:04:03 smithi022 ceph-mon[31360]: pgmap v19962: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:04:03.528 INFO:journalctl@ceph.mon.c.smithi022.stdout:Dec 16 05:04:03 smithi022 ceph-mon[40250]: pgmap v19962: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:04:03.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Dec 16 05:04:03 smithi122 ceph-mon[32255]: pgmap v19962: 132 pgs: 132 active+clean; 582 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-12-16T05:04:05.012 INFO:teuthology.orchestra.run.smithi022.stderr:++ ceph auth get-key mgr.y 2021-12-16T05:04:05.128 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-12-16T05:04:05.130 DEBUG:teuthology.task.console_log:Killing console logger for smithi022 2021-12-16T05:04:05.131 DEBUG:teuthology.task.console_log:Killing console logger for smithi122 2021-12-16T05:04:05.132 DEBUG:teuthology.exit:Finished running handlers